Nov 03 00:40:17.422626 ip-10-0-133-217 systemd[1]: Starting Kubernetes Kubelet... Nov 03 00:40:17.884334 ip-10-0-133-217 kubenswrapper[2575]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 00:40:17.884334 ip-10-0-133-217 kubenswrapper[2575]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 03 00:40:17.884334 ip-10-0-133-217 kubenswrapper[2575]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 00:40:17.884334 ip-10-0-133-217 kubenswrapper[2575]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 00:40:17.887765 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.887682 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 00:40:17.887765 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.887764 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 00:40:17.887765 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.887769 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 00:40:17.887869 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.887772 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 00:40:17.887869 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.887775 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 00:40:17.887869 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.887778 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 00:40:17.887869 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.887781 2575 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 00:40:17.887869 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.887784 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 00:40:17.887869 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.887787 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 00:40:17.887869 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.887790 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 00:40:17.887869 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.887792 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 00:40:17.887869 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.887795 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 00:40:17.887869 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.887798 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 00:40:17.887869 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.887800 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 00:40:17.887869 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.887803 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 00:40:17.887869 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.887807 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 00:40:17.887869 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.887810 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 00:40:17.887869 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.887813 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 00:40:17.887869 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.887815 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 00:40:17.887869 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.887818 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 00:40:17.887869 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.887821 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 00:40:17.887869 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.887824 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 00:40:17.887869 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.887826 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 00:40:17.888349 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.887829 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 00:40:17.888349 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.887832 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 00:40:17.888349 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.887835 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 00:40:17.888349 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.887837 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 00:40:17.888349 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.887840 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 00:40:17.888349 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.887843 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 00:40:17.888349 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.887845 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 00:40:17.888349 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.887848 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 00:40:17.888349 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.887850 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 00:40:17.888349 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.887853 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 00:40:17.888349 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.887856 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 00:40:17.888349 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.887858 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 00:40:17.888349 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.887861 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 00:40:17.888349 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.887864 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 00:40:17.888349 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.887866 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 00:40:17.888349 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.887869 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 00:40:17.888349 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.887872 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 00:40:17.888349 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.887875 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 00:40:17.888349 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.887878 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 00:40:17.888796 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.887881 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 00:40:17.888796 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.887885 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 00:40:17.888796 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.887914 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 00:40:17.888796 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.887917 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 00:40:17.888796 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.887921 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 00:40:17.888796 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.887924 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 00:40:17.888796 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.887927 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 00:40:17.888796 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.887930 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 00:40:17.888796 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.887933 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 00:40:17.888796 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.887936 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 00:40:17.888796 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.887938 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 00:40:17.888796 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.887941 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 00:40:17.888796 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.887943 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 00:40:17.888796 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.887946 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 00:40:17.888796 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.887949 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 00:40:17.888796 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.887951 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 00:40:17.888796 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.887954 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 00:40:17.888796 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.887957 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 00:40:17.888796 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.887960 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 00:40:17.889251 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.887963 2575 feature_gate.go:386] unrecognized feature gate: Example Nov 03 00:40:17.889251 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.887965 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 00:40:17.889251 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.887968 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 00:40:17.889251 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.887971 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 00:40:17.889251 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.887973 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 00:40:17.889251 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.887976 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 00:40:17.889251 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.887979 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 00:40:17.889251 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.887982 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 00:40:17.889251 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.887984 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 00:40:17.889251 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.887987 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 00:40:17.889251 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.887990 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 00:40:17.889251 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.887993 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 00:40:17.889251 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.887996 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 00:40:17.889251 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.887999 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 00:40:17.889251 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.888004 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 00:40:17.889251 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.888008 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 00:40:17.889251 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.888012 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 00:40:17.889251 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.888014 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 00:40:17.889251 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.888017 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 00:40:17.889799 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.888019 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 00:40:17.889799 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.888022 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 00:40:17.889799 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.888025 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 00:40:17.889799 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.888028 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 00:40:17.889799 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.888030 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 00:40:17.889799 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.888033 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 00:40:17.889799 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.888035 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 00:40:17.889799 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.888039 2575 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 00:40:17.889799 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.888041 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 00:40:17.889799 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.888044 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 00:40:17.889799 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.888047 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 00:40:17.889799 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.888050 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 00:40:17.889799 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.888052 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 00:40:17.889799 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.888055 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 00:40:17.889799 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.888058 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 00:40:17.889799 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.888060 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 00:40:17.889799 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.888063 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 00:40:17.889799 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.888066 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 00:40:17.889799 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.888068 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 00:40:17.889799 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.888071 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 00:40:17.890283 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.888074 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 00:40:17.890283 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.888076 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 00:40:17.890283 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.888080 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 00:40:17.890283 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.888083 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 00:40:17.890283 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.888086 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 00:40:17.890283 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.888090 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 00:40:17.890283 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.888914 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 00:40:17.890283 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.888922 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 00:40:17.890283 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.888926 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 00:40:17.890283 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.888929 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 00:40:17.890283 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.888931 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 00:40:17.890283 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.888934 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 00:40:17.890283 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.888939 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 00:40:17.890283 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.888943 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 00:40:17.890283 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.888946 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 00:40:17.890283 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.888950 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 00:40:17.890283 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.888955 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 00:40:17.890283 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.888958 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 00:40:17.890283 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.888961 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 00:40:17.890769 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.888964 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 00:40:17.890769 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.888967 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 00:40:17.890769 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.888969 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 00:40:17.890769 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.888973 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 00:40:17.890769 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.888976 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 00:40:17.890769 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.888979 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 00:40:17.890769 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.888982 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 00:40:17.890769 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.888984 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 00:40:17.890769 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.888987 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 00:40:17.890769 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.888990 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 00:40:17.890769 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.888992 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 00:40:17.890769 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.888996 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 00:40:17.890769 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.888999 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 00:40:17.890769 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.889001 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 00:40:17.890769 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.889004 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 00:40:17.890769 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.889006 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 00:40:17.890769 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.889009 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 00:40:17.890769 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.889012 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 00:40:17.890769 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.889014 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 00:40:17.890769 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.889018 2575 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 00:40:17.891250 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.889020 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 00:40:17.891250 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.889023 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 00:40:17.891250 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.889025 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 00:40:17.891250 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.889028 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 00:40:17.891250 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.889031 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 00:40:17.891250 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.889034 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 00:40:17.891250 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.889036 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 00:40:17.891250 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.889039 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 00:40:17.891250 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.889042 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 00:40:17.891250 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.889045 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 00:40:17.891250 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.889048 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 00:40:17.891250 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.889050 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 00:40:17.891250 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.889053 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 00:40:17.891250 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.889056 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 00:40:17.891250 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.889058 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 00:40:17.891250 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.889061 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 00:40:17.891250 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.889064 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 00:40:17.891250 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.889066 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 00:40:17.891250 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.889069 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 00:40:17.891250 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.889072 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 00:40:17.891738 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.889074 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 00:40:17.891738 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.889077 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 00:40:17.891738 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.889080 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 00:40:17.891738 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.889082 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 00:40:17.891738 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.889086 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 00:40:17.891738 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.889088 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 00:40:17.891738 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.889091 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 00:40:17.891738 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.889093 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 00:40:17.891738 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.889096 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 00:40:17.891738 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.889098 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 00:40:17.891738 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.889101 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 00:40:17.891738 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.889104 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 00:40:17.891738 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.889107 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 00:40:17.891738 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.889109 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 00:40:17.891738 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.889111 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 00:40:17.891738 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.889115 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 00:40:17.891738 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.889118 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 00:40:17.891738 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.889120 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 00:40:17.891738 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.889123 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 00:40:17.892202 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.889125 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 00:40:17.892202 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.889128 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 00:40:17.892202 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.889130 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 00:40:17.892202 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.889133 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 00:40:17.892202 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.889136 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 00:40:17.892202 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.889138 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 00:40:17.892202 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.889141 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 00:40:17.892202 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.889144 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 00:40:17.892202 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.889146 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 00:40:17.892202 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.889149 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 00:40:17.892202 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.889151 2575 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 00:40:17.892202 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.889154 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 00:40:17.892202 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.889156 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 00:40:17.892202 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.889159 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 00:40:17.892202 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.889161 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 00:40:17.892202 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.889164 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 00:40:17.892202 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.889167 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 00:40:17.892202 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.889169 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 00:40:17.892202 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.889172 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 00:40:17.892202 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.889175 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 00:40:17.892707 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.889177 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 00:40:17.892707 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.889180 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 00:40:17.892707 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.889182 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 00:40:17.892707 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.889185 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 00:40:17.892707 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.889187 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 00:40:17.892707 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.889190 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 00:40:17.892707 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.889193 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 00:40:17.892707 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.889196 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 00:40:17.892707 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.889198 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 00:40:17.892707 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.889201 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 00:40:17.892707 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.889205 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 00:40:17.892707 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.889207 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 00:40:17.892707 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.889210 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 00:40:17.892707 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.889213 2575 feature_gate.go:386] unrecognized feature gate: Example Nov 03 00:40:17.892707 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.890474 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 00:40:17.892707 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.890482 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 00:40:17.892707 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.890486 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 00:40:17.892707 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.890490 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 00:40:17.892707 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.890493 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 00:40:17.892707 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.890495 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 00:40:17.893192 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.890498 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 00:40:17.893192 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.890502 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 00:40:17.893192 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.890507 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 00:40:17.893192 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.890510 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 00:40:17.893192 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.890513 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 00:40:17.893192 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.890516 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 00:40:17.893192 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.890519 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 00:40:17.893192 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.890522 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 00:40:17.893192 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.890525 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 00:40:17.893192 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.890528 2575 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 00:40:17.893192 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.890530 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 00:40:17.893192 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.890533 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 00:40:17.893192 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.890536 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 00:40:17.893192 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.890538 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 00:40:17.893192 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.890541 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 00:40:17.893192 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.890544 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 00:40:17.893192 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.890547 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 00:40:17.893192 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.890549 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 00:40:17.893192 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.890557 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 00:40:17.893663 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.890561 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 00:40:17.893663 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.890565 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 00:40:17.893663 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.890568 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 00:40:17.893663 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.890570 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 00:40:17.893663 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.890573 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 00:40:17.893663 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.890576 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 00:40:17.893663 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.890578 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 00:40:17.893663 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.890581 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 00:40:17.893663 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.890584 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 00:40:17.893663 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.890586 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 00:40:17.893663 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.890589 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 00:40:17.893663 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.890592 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 00:40:17.893663 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.890594 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 00:40:17.893663 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.890597 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 00:40:17.893663 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.890599 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 00:40:17.893663 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.890603 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 00:40:17.893663 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.890606 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 00:40:17.893663 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.890609 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 00:40:17.893663 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.890611 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 00:40:17.894103 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.890614 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 00:40:17.894103 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.890616 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 00:40:17.894103 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.890619 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 00:40:17.894103 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.890621 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 00:40:17.894103 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.890624 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 00:40:17.894103 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.890626 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 00:40:17.894103 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.890629 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 00:40:17.894103 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.890632 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 00:40:17.894103 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.890634 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 00:40:17.894103 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.890637 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 00:40:17.894103 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.890639 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 00:40:17.894103 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.890642 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 00:40:17.894103 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.890647 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 00:40:17.894103 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.890650 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 00:40:17.894103 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.890653 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 00:40:17.894103 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.890655 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 00:40:17.894103 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.890658 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 00:40:17.894103 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.890661 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 00:40:17.894103 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.890664 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 00:40:17.894103 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.890666 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 00:40:17.894590 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.890669 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 00:40:17.894590 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.890671 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 00:40:17.894590 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.890674 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 00:40:17.894590 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.890676 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 00:40:17.894590 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.890679 2575 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 00:40:17.894590 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.890682 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 00:40:17.894590 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.890684 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 00:40:17.894590 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.890687 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 00:40:17.894590 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.890690 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 00:40:17.894590 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.890692 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 00:40:17.894590 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.890695 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 00:40:17.894590 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.890697 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 00:40:17.894590 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.890700 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 00:40:17.894590 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.890703 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 00:40:17.894590 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.890707 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 00:40:17.894590 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.890710 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 00:40:17.894590 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.890712 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 00:40:17.894590 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.890715 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 00:40:17.894590 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.890718 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 00:40:17.894590 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.890721 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 00:40:17.895106 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.890723 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 00:40:17.895106 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.890726 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 00:40:17.895106 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.890729 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 00:40:17.895106 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.890732 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 00:40:17.895106 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.890736 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 00:40:17.895106 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.890738 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 00:40:17.895106 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.890742 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 00:40:17.895106 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.890744 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 00:40:17.895106 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.890747 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 00:40:17.895106 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.890749 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 00:40:17.895106 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.890752 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 00:40:17.895106 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.890755 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 00:40:17.895106 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.890757 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 00:40:17.895106 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.890761 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 00:40:17.895106 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.890764 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 00:40:17.895106 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.890766 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 00:40:17.895106 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.890769 2575 feature_gate.go:386] unrecognized feature gate: Example Nov 03 00:40:17.895106 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.890772 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 00:40:17.895106 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.890775 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 00:40:17.895589 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.890777 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 00:40:17.895589 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.890780 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 00:40:17.895589 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.890785 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 00:40:17.895589 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.891774 2575 flags.go:64] FLAG: --address="0.0.0.0" Nov 03 00:40:17.895589 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.891785 2575 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 03 00:40:17.895589 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.891792 2575 flags.go:64] FLAG: --anonymous-auth="true" Nov 03 00:40:17.895589 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.891797 2575 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 03 00:40:17.895589 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.891803 2575 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 03 00:40:17.895589 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.891806 2575 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 03 00:40:17.895589 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.891812 2575 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 03 00:40:17.895589 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.891817 2575 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 03 00:40:17.895589 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.891820 2575 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 03 00:40:17.895589 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.891823 2575 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 03 00:40:17.895589 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.891827 2575 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 03 00:40:17.895589 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.891830 2575 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 03 00:40:17.895589 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.891833 2575 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 03 00:40:17.895589 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.891836 2575 flags.go:64] FLAG: --cgroup-root="" Nov 03 00:40:17.895589 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.891840 2575 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 03 00:40:17.895589 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.891843 2575 flags.go:64] FLAG: --client-ca-file="" Nov 03 00:40:17.895589 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.891846 2575 flags.go:64] FLAG: --cloud-provider="external" Nov 03 00:40:17.895589 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.891853 2575 flags.go:64] FLAG: --cluster-dns="[]" Nov 03 00:40:17.895589 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.891859 2575 flags.go:64] FLAG: --cluster-domain="" Nov 03 00:40:17.895589 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.891862 2575 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 03 00:40:17.895589 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.891865 2575 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 03 00:40:17.896154 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.891868 2575 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 03 00:40:17.896154 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.891871 2575 flags.go:64] FLAG: --container-log-max-files="5" Nov 03 00:40:17.896154 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.891878 2575 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 03 00:40:17.896154 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.891882 2575 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 03 00:40:17.896154 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.891885 2575 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 03 00:40:17.896154 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.891889 2575 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 03 00:40:17.896154 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.891892 2575 flags.go:64] FLAG: --contention-profiling="false" Nov 03 00:40:17.896154 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.891895 2575 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 03 00:40:17.896154 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.891899 2575 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 03 00:40:17.896154 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.891902 2575 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 03 00:40:17.896154 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.891905 2575 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 03 00:40:17.896154 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.891909 2575 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 03 00:40:17.896154 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.891912 2575 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 03 00:40:17.896154 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.891916 2575 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 03 00:40:17.896154 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.891919 2575 flags.go:64] FLAG: --enable-load-reader="false" Nov 03 00:40:17.896154 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.891922 2575 flags.go:64] FLAG: --enable-server="true" Nov 03 00:40:17.896154 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.891925 2575 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 03 00:40:17.896154 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.891930 2575 flags.go:64] FLAG: --event-burst="100" Nov 03 00:40:17.896154 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.891933 2575 flags.go:64] FLAG: --event-qps="50" Nov 03 00:40:17.896154 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.891936 2575 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 03 00:40:17.896154 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.891939 2575 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 03 00:40:17.896154 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.891942 2575 flags.go:64] FLAG: --eviction-hard="" Nov 03 00:40:17.896154 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.891947 2575 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 03 00:40:17.896154 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.891950 2575 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 03 00:40:17.897331 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.891954 2575 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 03 00:40:17.897331 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.891957 2575 flags.go:64] FLAG: --eviction-soft="" Nov 03 00:40:17.897331 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.891960 2575 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 03 00:40:17.897331 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.891964 2575 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 03 00:40:17.897331 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.891967 2575 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 03 00:40:17.897331 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.891971 2575 flags.go:64] FLAG: --experimental-mounter-path="" Nov 03 00:40:17.897331 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.891974 2575 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 03 00:40:17.897331 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.891977 2575 flags.go:64] FLAG: --fail-swap-on="true" Nov 03 00:40:17.897331 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.891980 2575 flags.go:64] FLAG: --feature-gates="" Nov 03 00:40:17.897331 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.891985 2575 flags.go:64] FLAG: --file-check-frequency="20s" Nov 03 00:40:17.897331 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.891988 2575 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 03 00:40:17.897331 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.891991 2575 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 03 00:40:17.897331 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.891995 2575 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 03 00:40:17.897331 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.891998 2575 flags.go:64] FLAG: --healthz-port="10248" Nov 03 00:40:17.897331 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.892001 2575 flags.go:64] FLAG: --help="false" Nov 03 00:40:17.897331 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.892005 2575 flags.go:64] FLAG: --hostname-override="ip-10-0-133-217.us-east-2.compute.internal" Nov 03 00:40:17.897331 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.892008 2575 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 03 00:40:17.897331 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.892011 2575 flags.go:64] FLAG: --http-check-frequency="20s" Nov 03 00:40:17.897331 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.892014 2575 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 03 00:40:17.897331 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.892018 2575 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 03 00:40:17.897331 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.892021 2575 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 03 00:40:17.897331 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.892024 2575 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 03 00:40:17.897331 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.892027 2575 flags.go:64] FLAG: --image-service-endpoint="" Nov 03 00:40:17.897872 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.892030 2575 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 03 00:40:17.897872 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.892033 2575 flags.go:64] FLAG: --kube-api-burst="100" Nov 03 00:40:17.897872 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.892036 2575 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 03 00:40:17.897872 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.892039 2575 flags.go:64] FLAG: --kube-api-qps="50" Nov 03 00:40:17.897872 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.892042 2575 flags.go:64] FLAG: --kube-reserved="" Nov 03 00:40:17.897872 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.892045 2575 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 03 00:40:17.897872 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.892048 2575 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 03 00:40:17.897872 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.892051 2575 flags.go:64] FLAG: --kubelet-cgroups="" Nov 03 00:40:17.897872 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.892054 2575 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 03 00:40:17.897872 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.892058 2575 flags.go:64] FLAG: --lock-file="" Nov 03 00:40:17.897872 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.892060 2575 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 03 00:40:17.897872 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.892063 2575 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 03 00:40:17.897872 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.892067 2575 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 03 00:40:17.897872 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.892073 2575 flags.go:64] FLAG: --log-json-split-stream="false" Nov 03 00:40:17.897872 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.892076 2575 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 03 00:40:17.897872 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.892079 2575 flags.go:64] FLAG: --log-text-split-stream="false" Nov 03 00:40:17.897872 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.892082 2575 flags.go:64] FLAG: --logging-format="text" Nov 03 00:40:17.897872 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.892085 2575 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 03 00:40:17.897872 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.892089 2575 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 03 00:40:17.897872 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.892092 2575 flags.go:64] FLAG: --manifest-url="" Nov 03 00:40:17.897872 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.892095 2575 flags.go:64] FLAG: --manifest-url-header="" Nov 03 00:40:17.897872 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.892100 2575 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 03 00:40:17.897872 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.892103 2575 flags.go:64] FLAG: --max-open-files="1000000" Nov 03 00:40:17.897872 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.892107 2575 flags.go:64] FLAG: --max-pods="110" Nov 03 00:40:17.897872 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.892110 2575 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 03 00:40:17.898493 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.892113 2575 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 03 00:40:17.898493 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.892116 2575 flags.go:64] FLAG: --memory-manager-policy="None" Nov 03 00:40:17.898493 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.892120 2575 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 03 00:40:17.898493 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.892122 2575 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 03 00:40:17.898493 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.892126 2575 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 03 00:40:17.898493 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.892129 2575 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 03 00:40:17.898493 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.892137 2575 flags.go:64] FLAG: --node-status-max-images="50" Nov 03 00:40:17.898493 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.892140 2575 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 03 00:40:17.898493 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.892144 2575 flags.go:64] FLAG: --oom-score-adj="-999" Nov 03 00:40:17.898493 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.892147 2575 flags.go:64] FLAG: --pod-cidr="" Nov 03 00:40:17.898493 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.892150 2575 flags.go:64] FLAG: --pod-manifest-path="" Nov 03 00:40:17.898493 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.892153 2575 flags.go:64] FLAG: --pod-max-pids="-1" Nov 03 00:40:17.898493 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.892156 2575 flags.go:64] FLAG: --pods-per-core="0" Nov 03 00:40:17.898493 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.892159 2575 flags.go:64] FLAG: --port="10250" Nov 03 00:40:17.898493 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.892162 2575 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 03 00:40:17.898493 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.892165 2575 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-0836f34c7363c1256" Nov 03 00:40:17.898493 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.892169 2575 flags.go:64] FLAG: --qos-reserved="" Nov 03 00:40:17.898493 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.892172 2575 flags.go:64] FLAG: --read-only-port="10255" Nov 03 00:40:17.898493 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.892175 2575 flags.go:64] FLAG: --register-node="true" Nov 03 00:40:17.898493 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.892178 2575 flags.go:64] FLAG: --register-with-taints="" Nov 03 00:40:17.898493 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.892182 2575 flags.go:64] FLAG: --registry-burst="10" Nov 03 00:40:17.898493 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.892185 2575 flags.go:64] FLAG: --registry-qps="5" Nov 03 00:40:17.898493 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.892188 2575 flags.go:64] FLAG: --reserved-cpus="" Nov 03 00:40:17.898493 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.892192 2575 flags.go:64] FLAG: --reserved-memory="" Nov 03 00:40:17.898493 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.892196 2575 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 03 00:40:17.898493 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.892199 2575 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 03 00:40:17.899224 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.892202 2575 flags.go:64] FLAG: --rotate-certificates="false" Nov 03 00:40:17.899224 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.892205 2575 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 03 00:40:17.899224 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.892208 2575 flags.go:64] FLAG: --runonce="false" Nov 03 00:40:17.899224 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.892211 2575 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 03 00:40:17.899224 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.892217 2575 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 03 00:40:17.899224 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.892220 2575 flags.go:64] FLAG: --seccomp-default="false" Nov 03 00:40:17.899224 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.892223 2575 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 03 00:40:17.899224 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.892226 2575 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 03 00:40:17.899224 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.892230 2575 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 03 00:40:17.899224 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.892233 2575 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 03 00:40:17.899224 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.892236 2575 flags.go:64] FLAG: --storage-driver-password="root" Nov 03 00:40:17.899224 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.892239 2575 flags.go:64] FLAG: --storage-driver-secure="false" Nov 03 00:40:17.899224 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.892243 2575 flags.go:64] FLAG: --storage-driver-table="stats" Nov 03 00:40:17.899224 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.892246 2575 flags.go:64] FLAG: --storage-driver-user="root" Nov 03 00:40:17.899224 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.892250 2575 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 03 00:40:17.899224 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.892253 2575 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 03 00:40:17.899224 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.892256 2575 flags.go:64] FLAG: --system-cgroups="" Nov 03 00:40:17.899224 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.892259 2575 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 03 00:40:17.899224 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.892264 2575 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 03 00:40:17.899224 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.892266 2575 flags.go:64] FLAG: --tls-cert-file="" Nov 03 00:40:17.899224 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.892280 2575 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 03 00:40:17.899224 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.892287 2575 flags.go:64] FLAG: --tls-min-version="" Nov 03 00:40:17.899224 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.892290 2575 flags.go:64] FLAG: --tls-private-key-file="" Nov 03 00:40:17.899224 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.892293 2575 flags.go:64] FLAG: --topology-manager-policy="none" Nov 03 00:40:17.899224 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.892296 2575 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 03 00:40:17.899835 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.892299 2575 flags.go:64] FLAG: --topology-manager-scope="container" Nov 03 00:40:17.899835 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.892302 2575 flags.go:64] FLAG: --v="2" Nov 03 00:40:17.899835 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.892307 2575 flags.go:64] FLAG: --version="false" Nov 03 00:40:17.899835 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.892311 2575 flags.go:64] FLAG: --vmodule="" Nov 03 00:40:17.899835 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.892315 2575 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 03 00:40:17.899835 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.892319 2575 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 03 00:40:17.899835 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.892461 2575 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 00:40:17.899835 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.892465 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 00:40:17.899835 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.892468 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 00:40:17.899835 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.892471 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 00:40:17.899835 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.892474 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 00:40:17.899835 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.892480 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 00:40:17.899835 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.892484 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 00:40:17.899835 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.892487 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 00:40:17.899835 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.892490 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 00:40:17.899835 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.892493 2575 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 00:40:17.899835 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.892496 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 00:40:17.899835 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.892499 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 00:40:17.899835 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.892502 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 00:40:17.899835 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.892504 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 00:40:17.900327 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.892507 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 00:40:17.900327 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.892510 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 00:40:17.900327 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.892513 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 00:40:17.900327 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.892516 2575 feature_gate.go:386] unrecognized feature gate: Example Nov 03 00:40:17.900327 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.892518 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 00:40:17.900327 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.892521 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 00:40:17.900327 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.892524 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 00:40:17.900327 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.892527 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 00:40:17.900327 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.892531 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 00:40:17.900327 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.892533 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 00:40:17.900327 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.892536 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 00:40:17.900327 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.892539 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 00:40:17.900327 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.892541 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 00:40:17.900327 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.892544 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 00:40:17.900327 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.892546 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 00:40:17.900327 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.892549 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 00:40:17.900327 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.892552 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 00:40:17.900327 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.892554 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 00:40:17.900327 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.892558 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 00:40:17.900327 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.892560 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 00:40:17.900809 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.892563 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 00:40:17.900809 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.892566 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 00:40:17.900809 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.892568 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 00:40:17.900809 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.892573 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 00:40:17.900809 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.892575 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 00:40:17.900809 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.892578 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 00:40:17.900809 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.892581 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 00:40:17.900809 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.892585 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 00:40:17.900809 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.892589 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 00:40:17.900809 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.892592 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 00:40:17.900809 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.892595 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 00:40:17.900809 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.892598 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 00:40:17.900809 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.892600 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 00:40:17.900809 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.892603 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 00:40:17.900809 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.892606 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 00:40:17.900809 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.892609 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 00:40:17.900809 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.892612 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 00:40:17.900809 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.892614 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 00:40:17.900809 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.892617 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 00:40:17.901285 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.892620 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 00:40:17.901285 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.892624 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 00:40:17.901285 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.892627 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 00:40:17.901285 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.892630 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 00:40:17.901285 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.892632 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 00:40:17.901285 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.892635 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 00:40:17.901285 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.892637 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 00:40:17.901285 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.892640 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 00:40:17.901285 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.892643 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 00:40:17.901285 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.892645 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 00:40:17.901285 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.892648 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 00:40:17.901285 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.892650 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 00:40:17.901285 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.892653 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 00:40:17.901285 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.892656 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 00:40:17.901285 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.892659 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 00:40:17.901285 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.892662 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 00:40:17.901285 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.892666 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 00:40:17.901285 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.892669 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 00:40:17.901285 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.892672 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 00:40:17.901285 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.892674 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 00:40:17.901781 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.892677 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 00:40:17.901781 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.892680 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 00:40:17.901781 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.892682 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 00:40:17.901781 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.892685 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 00:40:17.901781 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.892688 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 00:40:17.901781 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.892690 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 00:40:17.901781 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.892693 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 00:40:17.901781 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.892695 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 00:40:17.901781 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.892698 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 00:40:17.901781 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.892701 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 00:40:17.901781 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.892703 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 00:40:17.901781 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.892707 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 00:40:17.901781 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.892710 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 00:40:17.901781 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.892715 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 00:40:17.901781 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.892717 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 00:40:17.901781 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.892720 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 00:40:17.901781 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.892723 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 00:40:17.901781 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.892726 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 00:40:17.901781 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.892728 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 00:40:17.902230 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.892731 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 00:40:17.902230 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.892734 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 00:40:17.902230 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.892737 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 00:40:17.902230 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.892739 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 00:40:17.902230 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.892742 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 00:40:17.902230 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.892744 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 00:40:17.902230 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.892748 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 00:40:17.902230 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.892750 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 00:40:17.902230 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.892753 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 00:40:17.902230 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.892765 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 00:40:17.902230 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.892768 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 00:40:17.902230 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.892771 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 00:40:17.902230 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.892773 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 00:40:17.902230 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.892776 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 00:40:17.902230 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.892803 2575 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 00:40:17.902230 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.892839 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 00:40:17.902630 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.892844 2575 feature_gate.go:386] unrecognized feature gate: Example Nov 03 00:40:17.902630 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.892848 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 00:40:17.902630 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.892851 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 00:40:17.902630 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.892854 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 00:40:17.902630 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.892856 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 00:40:17.902630 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.892859 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 00:40:17.902630 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.892862 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 00:40:17.902630 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.892864 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 00:40:17.902630 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.892868 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 00:40:17.902630 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.892871 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 00:40:17.902630 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.892875 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 00:40:17.902630 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.892878 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 00:40:17.902630 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.892881 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 00:40:17.902630 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.892883 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 00:40:17.902630 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.892886 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 00:40:17.902630 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.892888 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 00:40:17.902630 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.892891 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 00:40:17.902630 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.892894 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 00:40:17.902630 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.892896 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 00:40:17.902630 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.892899 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 00:40:17.903092 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.892901 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 00:40:17.903092 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.892904 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 00:40:17.903092 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.892907 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 00:40:17.903092 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.892910 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 00:40:17.903092 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.892914 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 00:40:17.903092 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.892918 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 00:40:17.903092 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.892922 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 00:40:17.903092 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.892924 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 00:40:17.903092 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.892927 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 00:40:17.903092 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.892930 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 00:40:17.903092 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.892932 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 00:40:17.903092 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.892936 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 00:40:17.903092 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.892939 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 00:40:17.903092 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.892941 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 00:40:17.903092 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.892944 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 00:40:17.903092 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.892947 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 00:40:17.903092 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.892949 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 00:40:17.903092 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.892952 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 00:40:17.903092 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.892955 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 00:40:17.903092 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.892957 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 00:40:17.903598 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.892960 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 00:40:17.903598 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.892963 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 00:40:17.903598 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.892967 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 00:40:17.903598 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.892970 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 00:40:17.903598 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.892973 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 00:40:17.903598 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.892975 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 00:40:17.903598 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.892977 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 00:40:17.903598 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.892980 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 00:40:17.903598 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.892983 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 00:40:17.903598 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.892985 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 00:40:17.903598 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.892988 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 00:40:17.903598 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.892991 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 00:40:17.903598 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.892994 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 00:40:17.903598 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.892996 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 00:40:17.903598 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.892999 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 00:40:17.903598 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.893002 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 00:40:17.903598 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.893005 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 00:40:17.903598 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.893007 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 00:40:17.903598 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.893011 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 00:40:17.904058 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.893014 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 00:40:17.904058 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.893016 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 00:40:17.904058 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.893019 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 00:40:17.904058 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.893021 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 00:40:17.904058 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.893024 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 00:40:17.904058 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.893027 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 00:40:17.904058 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.893029 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 00:40:17.904058 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.893032 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 00:40:17.904058 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.893034 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 00:40:17.904058 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.893037 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 00:40:17.904058 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.893040 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 00:40:17.904058 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.893042 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 00:40:17.904058 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.893045 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 00:40:17.904058 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.893048 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 00:40:17.904058 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.893051 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 00:40:17.904058 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.893055 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 00:40:17.904058 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.893058 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 00:40:17.904058 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.893060 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 00:40:17.904058 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.893063 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 00:40:17.904058 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.893065 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 00:40:17.904693 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.893068 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 00:40:17.904693 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.893070 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 00:40:17.904693 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.893073 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 00:40:17.904693 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.893075 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 00:40:17.904693 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.893078 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 00:40:17.904693 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.893081 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 00:40:17.904693 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.893083 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 00:40:17.904693 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.893086 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 00:40:17.904693 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.893088 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 00:40:17.904693 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.893091 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 00:40:17.904693 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.893094 2575 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 00:40:17.904693 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.893098 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 00:40:17.904693 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.893101 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 00:40:17.904693 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.893104 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 00:40:17.904693 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.893106 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 00:40:17.904693 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.893110 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 00:40:17.904693 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.893114 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 00:40:17.904693 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.893118 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 00:40:17.904693 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.893122 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 00:40:17.905186 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.893125 2575 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 00:40:17.905186 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.893128 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 00:40:17.905186 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.893130 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 00:40:17.905186 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.893133 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 00:40:17.905186 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.893135 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 00:40:17.905186 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.893138 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 00:40:17.905186 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.893141 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 00:40:17.905186 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.900684 2575 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 03 00:40:17.905186 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.900708 2575 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 03 00:40:17.905186 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.900727 2575 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 03 00:40:17.905186 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.900732 2575 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 03 00:40:17.905186 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.900810 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 00:40:17.905186 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.900816 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 00:40:17.905186 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.900820 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 00:40:17.905186 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.900824 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 00:40:17.905186 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.900827 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 00:40:17.905186 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.900830 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 00:40:17.905186 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.900833 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 00:40:17.905186 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.900836 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 00:40:17.905672 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.900839 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 00:40:17.905672 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.900842 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 00:40:17.905672 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.900845 2575 feature_gate.go:386] unrecognized feature gate: Example Nov 03 00:40:17.905672 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.900848 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 00:40:17.905672 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.900851 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 00:40:17.905672 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.900853 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 00:40:17.905672 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.900856 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 00:40:17.905672 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.900859 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 00:40:17.905672 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.900862 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 00:40:17.905672 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.900865 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 00:40:17.905672 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.900867 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 00:40:17.905672 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.900870 2575 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 00:40:17.905672 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.900873 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 00:40:17.905672 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.900875 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 00:40:17.905672 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.900878 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 00:40:17.905672 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.900880 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 00:40:17.905672 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.900883 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 00:40:17.905672 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.900886 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 00:40:17.905672 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.900889 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 00:40:17.905672 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.900891 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 00:40:17.906149 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.900895 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 00:40:17.906149 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.900906 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 00:40:17.906149 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.900909 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 00:40:17.906149 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.900912 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 00:40:17.906149 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.900914 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 00:40:17.906149 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.900918 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 00:40:17.906149 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.900920 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 00:40:17.906149 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.900923 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 00:40:17.906149 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.900926 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 00:40:17.906149 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.900928 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 00:40:17.906149 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.900931 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 00:40:17.906149 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.900934 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 00:40:17.906149 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.900937 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 00:40:17.906149 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.900940 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 00:40:17.906149 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.900942 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 00:40:17.906149 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.900945 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 00:40:17.906149 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.900948 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 00:40:17.906149 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.900950 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 00:40:17.906149 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.900953 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 00:40:17.906629 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.900956 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 00:40:17.906629 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.900958 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 00:40:17.906629 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.900961 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 00:40:17.906629 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.900963 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 00:40:17.906629 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.900966 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 00:40:17.906629 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.900969 2575 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 00:40:17.906629 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.900971 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 00:40:17.906629 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.900974 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 00:40:17.906629 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.900977 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 00:40:17.906629 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.900980 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 00:40:17.906629 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.900983 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 00:40:17.906629 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.900985 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 00:40:17.906629 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.900988 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 00:40:17.906629 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.900991 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 00:40:17.906629 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.900995 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 00:40:17.906629 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901004 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 00:40:17.906629 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901007 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 00:40:17.906629 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901011 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 00:40:17.906629 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901016 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 00:40:17.907072 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901019 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 00:40:17.907072 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901022 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 00:40:17.907072 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901031 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 00:40:17.907072 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901035 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 00:40:17.907072 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901037 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 00:40:17.907072 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901040 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 00:40:17.907072 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901042 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 00:40:17.907072 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901045 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 00:40:17.907072 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901048 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 00:40:17.907072 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901052 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 00:40:17.907072 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901055 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 00:40:17.907072 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901058 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 00:40:17.907072 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901061 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 00:40:17.907072 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901064 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 00:40:17.907072 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901066 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 00:40:17.907072 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901069 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 00:40:17.907072 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901072 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 00:40:17.907072 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901074 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 00:40:17.907072 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901077 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 00:40:17.907558 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901080 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 00:40:17.907558 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901083 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 00:40:17.907558 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901085 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 00:40:17.907558 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901088 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 00:40:17.907558 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901091 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 00:40:17.907558 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901094 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 00:40:17.907558 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901096 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 00:40:17.907558 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901099 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 00:40:17.907558 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901102 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 00:40:17.907558 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901105 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 00:40:17.907558 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901114 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 00:40:17.907558 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901117 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 00:40:17.907558 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901120 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 00:40:17.907558 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901122 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 00:40:17.907558 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901125 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 00:40:17.907558 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901128 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 00:40:17.907558 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901131 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 00:40:17.907558 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901134 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 00:40:17.907558 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901136 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 00:40:17.908032 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901139 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 00:40:17.908032 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901142 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 00:40:17.908032 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.901170 2575 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 00:40:17.908032 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901377 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 00:40:17.908032 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901382 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 00:40:17.908032 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901385 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 00:40:17.908032 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901389 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 00:40:17.908032 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901392 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 00:40:17.908032 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901395 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 00:40:17.908032 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901397 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 00:40:17.908032 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901400 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 00:40:17.908032 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901402 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 00:40:17.908032 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901405 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 00:40:17.908032 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901408 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 00:40:17.908032 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901411 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 00:40:17.908032 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901414 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 00:40:17.908472 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901416 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 00:40:17.908472 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901419 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 00:40:17.908472 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901422 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 00:40:17.908472 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901424 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 00:40:17.908472 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901427 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 00:40:17.908472 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901430 2575 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 00:40:17.908472 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901433 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 00:40:17.908472 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901436 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 00:40:17.908472 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901446 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 00:40:17.908472 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901449 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 00:40:17.908472 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901452 2575 feature_gate.go:386] unrecognized feature gate: Example Nov 03 00:40:17.908472 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901455 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 00:40:17.908472 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901458 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 00:40:17.908472 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901461 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 00:40:17.908472 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901463 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 00:40:17.908472 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901466 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 00:40:17.908472 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901469 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 00:40:17.908472 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901471 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 00:40:17.908472 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901474 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 00:40:17.908472 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901477 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 00:40:17.908947 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901479 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 00:40:17.908947 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901482 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 00:40:17.908947 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901485 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 00:40:17.908947 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901487 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 00:40:17.908947 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901490 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 00:40:17.908947 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901493 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 00:40:17.908947 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901495 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 00:40:17.908947 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901498 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 00:40:17.908947 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901501 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 00:40:17.908947 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901503 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 00:40:17.908947 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901506 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 00:40:17.908947 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901509 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 00:40:17.908947 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901512 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 00:40:17.908947 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901514 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 00:40:17.908947 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901517 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 00:40:17.908947 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901520 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 00:40:17.908947 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901522 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 00:40:17.908947 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901525 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 00:40:17.908947 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901527 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 00:40:17.908947 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901530 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 00:40:17.909545 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901533 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 00:40:17.909545 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901542 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 00:40:17.909545 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901545 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 00:40:17.909545 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901548 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 00:40:17.909545 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901550 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 00:40:17.909545 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901553 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 00:40:17.909545 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901556 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 00:40:17.909545 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901558 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 00:40:17.909545 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901561 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 00:40:17.909545 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901564 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 00:40:17.909545 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901566 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 00:40:17.909545 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901569 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 00:40:17.909545 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901572 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 00:40:17.909545 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901574 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 00:40:17.909545 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901577 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 00:40:17.909545 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901580 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 00:40:17.909545 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901583 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 00:40:17.909545 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901586 2575 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 00:40:17.909545 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901588 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 00:40:17.910300 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901592 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 00:40:17.910300 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901596 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 00:40:17.910300 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901599 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 00:40:17.910300 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901602 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 00:40:17.910300 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901605 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 00:40:17.910300 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901608 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 00:40:17.910300 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901611 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 00:40:17.910300 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901613 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 00:40:17.910300 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901616 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 00:40:17.910300 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901618 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 00:40:17.910300 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901621 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 00:40:17.910300 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901624 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 00:40:17.910300 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901627 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 00:40:17.910300 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901629 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 00:40:17.910300 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901632 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 00:40:17.910300 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901641 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 00:40:17.910300 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901644 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 00:40:17.910300 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901646 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 00:40:17.910300 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901651 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 00:40:17.910792 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901654 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 00:40:17.910792 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901657 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 00:40:17.910792 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901660 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 00:40:17.910792 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901662 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 00:40:17.910792 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901665 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 00:40:17.910792 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901667 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 00:40:17.910792 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901670 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 00:40:17.910792 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901673 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 00:40:17.910792 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901677 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 00:40:17.910792 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901681 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 00:40:17.910792 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901684 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 00:40:17.910792 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901688 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 00:40:17.910792 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901691 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 00:40:17.910792 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901693 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 00:40:17.910792 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901696 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 00:40:17.911182 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.901721 2575 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 00:40:17.911182 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901758 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 00:40:17.911182 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901763 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 00:40:17.911182 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901766 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 00:40:17.911182 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901769 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 00:40:17.911182 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901772 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 00:40:17.911182 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901775 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 00:40:17.911182 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901778 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 00:40:17.911182 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901780 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 00:40:17.911182 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901783 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 00:40:17.911182 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901786 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 00:40:17.911182 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901789 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 00:40:17.911182 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901792 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 00:40:17.911182 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901794 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 00:40:17.911182 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901803 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 00:40:17.911560 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901806 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 00:40:17.911560 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901811 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 00:40:17.911560 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901814 2575 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 00:40:17.911560 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901816 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 00:40:17.911560 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901820 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 00:40:17.911560 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901824 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 00:40:17.911560 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901827 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 00:40:17.911560 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901830 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 00:40:17.911560 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901833 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 00:40:17.911560 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901836 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 00:40:17.911560 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901839 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 00:40:17.911560 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901842 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 00:40:17.911560 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901845 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 00:40:17.911560 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901847 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 00:40:17.911560 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901850 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 00:40:17.911560 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901852 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 00:40:17.911560 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901855 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 00:40:17.911560 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901858 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 00:40:17.911560 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901860 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 00:40:17.911560 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901863 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 00:40:17.912097 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901866 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 00:40:17.912097 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901868 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 00:40:17.912097 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901871 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 00:40:17.912097 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901873 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 00:40:17.912097 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901876 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 00:40:17.912097 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901879 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 00:40:17.912097 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901881 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 00:40:17.912097 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901884 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 00:40:17.912097 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901887 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 00:40:17.912097 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901889 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 00:40:17.912097 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901892 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 00:40:17.912097 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901896 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 00:40:17.912097 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901905 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 00:40:17.912097 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901909 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 00:40:17.912097 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901912 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 00:40:17.912097 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901915 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 00:40:17.912097 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901918 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 00:40:17.912097 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901920 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 00:40:17.912097 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901923 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 00:40:17.912573 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901926 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 00:40:17.912573 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901929 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 00:40:17.912573 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901932 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 00:40:17.912573 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901934 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 00:40:17.912573 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901937 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 00:40:17.912573 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901940 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 00:40:17.912573 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901943 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 00:40:17.912573 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901945 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 00:40:17.912573 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901948 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 00:40:17.912573 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901951 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 00:40:17.912573 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901953 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 00:40:17.912573 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901956 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 00:40:17.912573 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901958 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 00:40:17.912573 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901961 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 00:40:17.912573 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901964 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 00:40:17.912573 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901966 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 00:40:17.912573 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901969 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 00:40:17.912573 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901972 2575 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 00:40:17.912573 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901974 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 00:40:17.912573 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901977 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 00:40:17.913067 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901980 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 00:40:17.913067 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901982 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 00:40:17.913067 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901985 2575 feature_gate.go:386] unrecognized feature gate: Example Nov 03 00:40:17.913067 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901987 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 00:40:17.913067 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901990 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 00:40:17.913067 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.901993 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 00:40:17.913067 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.902013 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 00:40:17.913067 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.902016 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 00:40:17.913067 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.902019 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 00:40:17.913067 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.902022 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 00:40:17.913067 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.902024 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 00:40:17.913067 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.902027 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 00:40:17.913067 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.902030 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 00:40:17.913067 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.902032 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 00:40:17.913067 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.902035 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 00:40:17.913067 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.902038 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 00:40:17.913067 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.902041 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 00:40:17.913067 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.902044 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 00:40:17.913067 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.902046 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 00:40:17.913067 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.902049 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 00:40:17.913584 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.902052 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 00:40:17.913584 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.902055 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 00:40:17.913584 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.902057 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 00:40:17.913584 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.902061 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 00:40:17.913584 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.902063 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 00:40:17.913584 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.902066 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 00:40:17.913584 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.902069 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 00:40:17.913584 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.902071 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 00:40:17.913584 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.902074 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 00:40:17.913584 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.902077 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 00:40:17.913584 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.902079 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 00:40:17.913584 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.902082 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 00:40:17.913584 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:17.902085 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 00:40:17.913584 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.902145 2575 server.go:957] "Client rotation is on, will bootstrap in background" Nov 03 00:40:17.913584 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.904416 2575 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Nov 03 00:40:17.913584 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.905446 2575 server.go:1014] "Starting client certificate rotation" Nov 03 00:40:17.913584 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.905547 2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 00:40:17.913584 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.906282 2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 00:40:17.914133 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.907799 2575 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 00:40:17.915188 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.915166 2575 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 00:40:17.931688 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.931663 2575 log.go:25] "Validated CRI v1 runtime API" Nov 03 00:40:17.934770 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.934752 2575 log.go:25] "Validated CRI v1 image API" Nov 03 00:40:17.935672 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.935647 2575 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 03 00:40:17.937880 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.937852 2575 fs.go:135] Filesystem UUIDs: map[66ae21c0-dbe3-4b2c-bc86-d0ecb8c85e81:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 9947e40b-3366-48ec-a10d-96375a6b0879:/dev/nvme0n1p3] Nov 03 00:40:17.937943 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.937879 2575 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Nov 03 00:40:17.938236 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.938220 2575 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 03 00:40:17.943141 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.943021 2575 manager.go:217] Machine: {Timestamp:2026-11-03 00:40:17.941713807 +0000 UTC m=+0.386771263 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100245 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec28b1067fd68b362a334eddcbc934b3 SystemUUID:ec28b106-7fd6-8b36-2a33-4eddcbc934b3 BootID:ea9d839b-74ae-4252-b09f-dd36e6790996 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:ce:16:47:ef Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:ce:16:47:ef Speed:0 Mtu:9001} {Name:ovs-system MacAddress:a6:39:2f:1b:f7:08 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 03 00:40:17.943141 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.943138 2575 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 03 00:40:17.943260 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.943234 2575 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 03 00:40:17.946252 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.946225 2575 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 03 00:40:17.946401 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.946254 2575 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-217.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 03 00:40:17.946442 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.946403 2575 topology_manager.go:143] "Creating topology manager with none policy" Nov 03 00:40:17.946442 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.946412 2575 container_manager_linux.go:311] "Creating device plugin manager" Nov 03 00:40:17.946442 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.946437 2575 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 00:40:17.946524 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.946451 2575 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 00:40:17.947457 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.947446 2575 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 03 00:40:17.951140 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.951127 2575 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 03 00:40:17.951260 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.951251 2575 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 03 00:40:17.951309 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.951296 2575 kubelet.go:493] "Attempting to sync node with API server" Nov 03 00:40:17.951309 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.951307 2575 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 03 00:40:17.951399 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.951321 2575 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 03 00:40:17.951399 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.951332 2575 kubelet.go:405] "Adding apiserver pod source" Nov 03 00:40:17.951399 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.951340 2575 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 03 00:40:17.951767 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.951755 2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 00:40:17.954130 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.954112 2575 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 03 00:40:17.959945 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.959920 2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 00:40:17.960837 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.960817 2575 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 03 00:40:17.960941 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.960859 2575 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 03 00:40:17.961098 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.961084 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 03 00:40:17.961163 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.961105 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 03 00:40:17.961163 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.961122 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 03 00:40:17.961163 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.961133 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 03 00:40:17.961163 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.961144 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 03 00:40:17.961163 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.961153 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 03 00:40:17.961732 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.961719 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 03 00:40:17.961789 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.961735 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 03 00:40:17.961789 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.961749 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 03 00:40:17.961789 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.961759 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 03 00:40:17.961789 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.961782 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 03 00:40:17.961941 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.961797 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 03 00:40:17.963494 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.963464 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 03 00:40:17.963582 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.963500 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 03 00:40:17.969830 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.969805 2575 server.go:1263] "Started kubelet" Nov 03 00:40:17.969917 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.969861 2575 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 03 00:40:17.970357 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.969990 2575 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 03 00:40:17.970357 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.970048 2575 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 03 00:40:17.970684 ip-10-0-133-217 systemd[1]: Started Kubernetes Kubelet. Nov 03 00:40:17.971479 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.971459 2575 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 03 00:40:17.972192 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.972177 2575 server.go:317] "Adding debug handlers to kubelet server" Nov 03 00:40:17.976434 ip-10-0-133-217 kubenswrapper[2575]: E1103 00:40:17.976406 2575 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Nov 03 00:40:17.976700 ip-10-0-133-217 kubenswrapper[2575]: E1103 00:40:17.976673 2575 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Nov 03 00:40:17.976784 ip-10-0-133-217 kubenswrapper[2575]: E1103 00:40:17.976747 2575 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-133-217.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Nov 03 00:40:17.977099 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.977084 2575 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-217.us-east-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Nov 03 00:40:17.978070 ip-10-0-133-217 kubenswrapper[2575]: E1103 00:40:17.977134 2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-217.us-east-2.compute.internal.18e4621a98a2ab83 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-217.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-133-217.us-east-2.compute.internal,},FirstTimestamp:2026-11-03 00:40:17.969769347 +0000 UTC m=+0.414826808,LastTimestamp:2026-11-03 00:40:17.969769347 +0000 UTC m=+0.414826808,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-217.us-east-2.compute.internal,}" Nov 03 00:40:17.980990 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.980972 2575 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 03 00:40:17.981070 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.980986 2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 03 00:40:17.981199 ip-10-0-133-217 kubenswrapper[2575]: E1103 00:40:17.981160 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-217.us-east-2.compute.internal\" not found" Nov 03 00:40:17.981616 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.981602 2575 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 03 00:40:17.981674 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.981620 2575 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 03 00:40:17.981715 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.981600 2575 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 03 00:40:17.981745 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.981720 2575 reconstruct.go:97] "Volume reconstruction finished" Nov 03 00:40:17.981745 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.981727 2575 reconciler.go:29] "Reconciler: start to sync state" Nov 03 00:40:17.983233 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.983044 2575 factory.go:55] Registering systemd factory Nov 03 00:40:17.983233 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.983063 2575 factory.go:223] Registration of the systemd container factory successfully Nov 03 00:40:17.983490 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.983465 2575 factory.go:153] Registering CRI-O factory Nov 03 00:40:17.983490 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.983491 2575 factory.go:223] Registration of the crio container factory successfully Nov 03 00:40:17.983598 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.983549 2575 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 03 00:40:17.983598 ip-10-0-133-217 kubenswrapper[2575]: E1103 00:40:17.983437 2575 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-133-217.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Nov 03 00:40:17.983598 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.983569 2575 factory.go:103] Registering Raw factory Nov 03 00:40:17.983723 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.983601 2575 manager.go:1199] Started watching for new ooms in manager Nov 03 00:40:17.984620 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.984602 2575 manager.go:319] Starting recovery of all containers Nov 03 00:40:17.986806 ip-10-0-133-217 kubenswrapper[2575]: E1103 00:40:17.986772 2575 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Nov 03 00:40:17.994468 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.994440 2575 manager.go:324] Recovery completed Nov 03 00:40:17.999805 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:17.999786 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 00:40:18.001786 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:18.001762 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-217.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 00:40:18.001890 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:18.001805 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-217.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 00:40:18.001890 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:18.001821 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-217.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 03 00:40:18.002491 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:18.002473 2575 cpu_manager.go:226] "Starting" policy="none" Nov 03 00:40:18.002491 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:18.002489 2575 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 03 00:40:18.002491 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:18.002493 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 00:40:18.002654 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:18.002507 2575 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 03 00:40:18.003211 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:18.003197 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-217.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 00:40:18.003329 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:18.003233 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-217.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 00:40:18.003329 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:18.003248 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-217.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 03 00:40:18.004567 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:18.004554 2575 policy_none.go:50] "Start" Nov 03 00:40:18.004640 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:18.004571 2575 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 03 00:40:18.004640 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:18.004581 2575 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 03 00:40:18.006376 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:18.006362 2575 policy_none.go:44] "Start" Nov 03 00:40:18.014624 ip-10-0-133-217 kubenswrapper[2575]: E1103 00:40:18.014146 2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-217.us-east-2.compute.internal.18e4621a9a8b3625 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-217.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-133-217.us-east-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-133-217.us-east-2.compute.internal,},FirstTimestamp:2026-11-03 00:40:18.001786405 +0000 UTC m=+0.446843862,LastTimestamp:2026-11-03 00:40:18.001786405 +0000 UTC m=+0.446843862,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-217.us-east-2.compute.internal,}" Nov 03 00:40:18.024678 ip-10-0-133-217 kubenswrapper[2575]: E1103 00:40:18.023783 2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-217.us-east-2.compute.internal.18e4621a9a8ba0cd default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-217.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-133-217.us-east-2.compute.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-133-217.us-east-2.compute.internal,},FirstTimestamp:2026-11-03 00:40:18.001813709 +0000 UTC m=+0.446871171,LastTimestamp:2026-11-03 00:40:18.001813709 +0000 UTC m=+0.446871171,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-217.us-east-2.compute.internal,}" Nov 03 00:40:18.034320 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:18.034212 2575 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-299vh" Nov 03 00:40:18.035062 ip-10-0-133-217 kubenswrapper[2575]: E1103 00:40:18.034977 2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-217.us-east-2.compute.internal.18e4621a9a8bcd18 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-217.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-133-217.us-east-2.compute.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-133-217.us-east-2.compute.internal,},FirstTimestamp:2026-11-03 00:40:18.001825048 +0000 UTC m=+0.446882502,LastTimestamp:2026-11-03 00:40:18.001825048 +0000 UTC m=+0.446882502,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-217.us-east-2.compute.internal,}" Nov 03 00:40:18.062751 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:18.044708 2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-299vh" Nov 03 00:40:18.062751 ip-10-0-133-217 kubenswrapper[2575]: E1103 00:40:18.045077 2575 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-133-217.us-east-2.compute.internal.18e4621a9a8b3625\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-217.us-east-2.compute.internal.18e4621a9a8b3625 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-217.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-133-217.us-east-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-133-217.us-east-2.compute.internal,},FirstTimestamp:2026-11-03 00:40:18.001786405 +0000 UTC m=+0.446843862,LastTimestamp:2026-11-03 00:40:18.003211097 +0000 UTC m=+0.448268561,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-217.us-east-2.compute.internal,}" Nov 03 00:40:18.062751 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:18.052966 2575 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 03 00:40:18.062751 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:18.055767 2575 manager.go:341] "Starting Device Plugin manager" Nov 03 00:40:18.062751 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:18.055769 2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 03 00:40:18.062751 ip-10-0-133-217 kubenswrapper[2575]: E1103 00:40:18.056663 2575 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 03 00:40:18.062751 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:18.056679 2575 server.go:87] "Starting device plugin registration server" Nov 03 00:40:18.062751 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:18.056897 2575 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 03 00:40:18.062751 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:18.056909 2575 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 03 00:40:18.062751 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:18.057023 2575 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 03 00:40:18.062751 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:18.057097 2575 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 03 00:40:18.062751 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:18.057105 2575 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 03 00:40:18.062751 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:18.057764 2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 03 00:40:18.062751 ip-10-0-133-217 kubenswrapper[2575]: E1103 00:40:18.057773 2575 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 03 00:40:18.062751 ip-10-0-133-217 kubenswrapper[2575]: E1103 00:40:18.057825 2575 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-133-217.us-east-2.compute.internal\" not found" Nov 03 00:40:18.062751 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:18.057779 2575 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 03 00:40:18.062751 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:18.057864 2575 kubelet.go:2521] "Starting kubelet main sync loop" Nov 03 00:40:18.062751 ip-10-0-133-217 kubenswrapper[2575]: E1103 00:40:18.057907 2575 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Nov 03 00:40:18.062751 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:18.062503 2575 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 00:40:18.157481 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:18.157222 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 00:40:18.158166 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:18.158144 2575 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-217.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-133-217.us-east-2.compute.internal"] Nov 03 00:40:18.158303 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:18.158215 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 00:40:18.158510 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:18.158493 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-217.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 00:40:18.158587 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:18.158526 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-217.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 00:40:18.158587 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:18.158539 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-217.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 03 00:40:18.158587 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:18.158564 2575 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-133-217.us-east-2.compute.internal" Nov 03 00:40:18.159059 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:18.159043 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-217.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 00:40:18.159137 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:18.159067 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-217.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 00:40:18.159137 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:18.159076 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-217.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 03 00:40:18.161251 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:18.161238 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 00:40:18.161419 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:18.161403 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-217.us-east-2.compute.internal" Nov 03 00:40:18.161467 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:18.161434 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 00:40:18.162007 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:18.161990 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-217.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 00:40:18.162092 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:18.162012 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-217.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 00:40:18.162092 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:18.162024 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-217.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 00:40:18.162092 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:18.162036 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-217.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 00:40:18.162092 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:18.162040 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-217.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 03 00:40:18.162092 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:18.162047 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-217.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 03 00:40:18.164086 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:18.164070 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-217.us-east-2.compute.internal" Nov 03 00:40:18.164169 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:18.164099 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 00:40:18.164860 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:18.164845 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-217.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 00:40:18.164942 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:18.164873 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-217.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 00:40:18.164942 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:18.164887 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-217.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 03 00:40:18.167432 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:18.167417 2575 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-133-217.us-east-2.compute.internal" Nov 03 00:40:18.167517 ip-10-0-133-217 kubenswrapper[2575]: E1103 00:40:18.167437 2575 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-133-217.us-east-2.compute.internal\": node \"ip-10-0-133-217.us-east-2.compute.internal\" not found" Nov 03 00:40:18.179903 ip-10-0-133-217 kubenswrapper[2575]: E1103 00:40:18.179883 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-217.us-east-2.compute.internal\" not found" Nov 03 00:40:18.195025 ip-10-0-133-217 kubenswrapper[2575]: E1103 00:40:18.195000 2575 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-217.us-east-2.compute.internal\" not found" node="ip-10-0-133-217.us-east-2.compute.internal" Nov 03 00:40:18.198422 ip-10-0-133-217 kubenswrapper[2575]: E1103 00:40:18.198403 2575 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-217.us-east-2.compute.internal\" not found" node="ip-10-0-133-217.us-east-2.compute.internal" Nov 03 00:40:18.280827 ip-10-0-133-217 kubenswrapper[2575]: E1103 00:40:18.280794 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-217.us-east-2.compute.internal\" not found" Nov 03 00:40:18.283017 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:18.282996 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fdeeb94bfd911d86834e9d29bdbbadab-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-217.us-east-2.compute.internal\" (UID: \"fdeeb94bfd911d86834e9d29bdbbadab\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-217.us-east-2.compute.internal" Nov 03 00:40:18.283124 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:18.283026 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fdeeb94bfd911d86834e9d29bdbbadab-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-217.us-east-2.compute.internal\" (UID: \"fdeeb94bfd911d86834e9d29bdbbadab\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-217.us-east-2.compute.internal" Nov 03 00:40:18.283124 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:18.283044 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a1171317f26f09b75b7b789f796f1384-config\") pod \"kube-apiserver-proxy-ip-10-0-133-217.us-east-2.compute.internal\" (UID: \"a1171317f26f09b75b7b789f796f1384\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-217.us-east-2.compute.internal" Nov 03 00:40:18.381252 ip-10-0-133-217 kubenswrapper[2575]: E1103 00:40:18.381205 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-217.us-east-2.compute.internal\" not found" Nov 03 00:40:18.383469 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:18.383447 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fdeeb94bfd911d86834e9d29bdbbadab-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-217.us-east-2.compute.internal\" (UID: \"fdeeb94bfd911d86834e9d29bdbbadab\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-217.us-east-2.compute.internal" Nov 03 00:40:18.383528 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:18.383492 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fdeeb94bfd911d86834e9d29bdbbadab-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-217.us-east-2.compute.internal\" (UID: \"fdeeb94bfd911d86834e9d29bdbbadab\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-217.us-east-2.compute.internal" Nov 03 00:40:18.383528 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:18.383492 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fdeeb94bfd911d86834e9d29bdbbadab-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-217.us-east-2.compute.internal\" (UID: \"fdeeb94bfd911d86834e9d29bdbbadab\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-217.us-east-2.compute.internal" Nov 03 00:40:18.383606 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:18.383528 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fdeeb94bfd911d86834e9d29bdbbadab-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-217.us-east-2.compute.internal\" (UID: \"fdeeb94bfd911d86834e9d29bdbbadab\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-217.us-east-2.compute.internal" Nov 03 00:40:18.383606 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:18.383533 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a1171317f26f09b75b7b789f796f1384-config\") pod \"kube-apiserver-proxy-ip-10-0-133-217.us-east-2.compute.internal\" (UID: \"a1171317f26f09b75b7b789f796f1384\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-217.us-east-2.compute.internal" Nov 03 00:40:18.383606 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:18.383571 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a1171317f26f09b75b7b789f796f1384-config\") pod \"kube-apiserver-proxy-ip-10-0-133-217.us-east-2.compute.internal\" (UID: \"a1171317f26f09b75b7b789f796f1384\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-217.us-east-2.compute.internal" Nov 03 00:40:18.481976 ip-10-0-133-217 kubenswrapper[2575]: E1103 00:40:18.481903 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-217.us-east-2.compute.internal\" not found" Nov 03 00:40:18.499242 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:18.499211 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-217.us-east-2.compute.internal" Nov 03 00:40:18.502392 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:18.502372 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-217.us-east-2.compute.internal" Nov 03 00:40:18.582728 ip-10-0-133-217 kubenswrapper[2575]: E1103 00:40:18.582695 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-217.us-east-2.compute.internal\" not found" Nov 03 00:40:18.683099 ip-10-0-133-217 kubenswrapper[2575]: E1103 00:40:18.683057 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-217.us-east-2.compute.internal\" not found" Nov 03 00:40:18.783603 ip-10-0-133-217 kubenswrapper[2575]: E1103 00:40:18.783562 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-217.us-east-2.compute.internal\" not found" Nov 03 00:40:18.883986 ip-10-0-133-217 kubenswrapper[2575]: E1103 00:40:18.883953 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-217.us-east-2.compute.internal\" not found" Nov 03 00:40:18.905176 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:18.905145 2575 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Nov 03 00:40:18.905620 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:18.905306 2575 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 03 00:40:18.984907 ip-10-0-133-217 kubenswrapper[2575]: E1103 00:40:18.984879 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-217.us-east-2.compute.internal\" not found" Nov 03 00:40:19.014446 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:19.014419 2575 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 00:40:19.050635 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:19.050541 2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 00:35:17 +0000 UTC" deadline="2028-04-09 14:07:52.603859895 +0000 UTC" Nov 03 00:40:19.050635 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:19.050589 2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12565h27m33.553275699s" Nov 03 00:40:19.085125 ip-10-0-133-217 kubenswrapper[2575]: E1103 00:40:19.085095 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-217.us-east-2.compute.internal\" not found" Nov 03 00:40:19.153465 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:19.153427 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdeeb94bfd911d86834e9d29bdbbadab.slice/crio-d26b7bbc18a12c91a4663e8aabaac3278b33104993a34ea7f7e9c75dfeb63adb WatchSource:0}: Error finding container d26b7bbc18a12c91a4663e8aabaac3278b33104993a34ea7f7e9c75dfeb63adb: Status 404 returned error can't find the container with id d26b7bbc18a12c91a4663e8aabaac3278b33104993a34ea7f7e9c75dfeb63adb Nov 03 00:40:19.154836 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:19.154812 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1171317f26f09b75b7b789f796f1384.slice/crio-baf995ad5d0a338d5cab169fe531fe16fa1263a06c4429391a64689fce408d2a WatchSource:0}: Error finding container baf995ad5d0a338d5cab169fe531fe16fa1263a06c4429391a64689fce408d2a: Status 404 returned error can't find the container with id baf995ad5d0a338d5cab169fe531fe16fa1263a06c4429391a64689fce408d2a Nov 03 00:40:19.159620 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:19.159604 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 00:40:19.167970 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:19.167950 2575 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 00:40:19.181777 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:19.181753 2575 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-217.us-east-2.compute.internal" Nov 03 00:40:19.194019 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:19.193994 2575 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 00:40:19.195914 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:19.195900 2575 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-217.us-east-2.compute.internal" Nov 03 00:40:19.205643 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:19.205625 2575 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 00:40:19.547057 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:19.545734 2575 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 00:40:19.881879 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:19.881787 2575 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 00:40:19.952656 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:19.952622 2575 apiserver.go:52] "Watching apiserver" Nov 03 00:40:19.962082 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:19.961296 2575 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 03 00:40:19.962082 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:19.961686 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-jltt2","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-trgdf","openshift-image-registry/node-ca-w8k5t","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-217.us-east-2.compute.internal","openshift-multus/multus-lb46r","openshift-ovn-kubernetes/ovnkube-node-zq57z","kube-system/global-pull-secret-syncer-262mm","kube-system/kube-apiserver-proxy-ip-10-0-133-217.us-east-2.compute.internal","openshift-multus/multus-additional-cni-plugins-8rxnt","openshift-multus/network-metrics-daemon-5v4k7","openshift-cluster-node-tuning-operator/tuned-n7ksd","openshift-dns/node-resolver-cphks","openshift-network-diagnostics/network-check-target-6lvpt","openshift-network-operator/iptables-alerter-9w99c"] Nov 03 00:40:19.964923 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:19.964394 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5v4k7" Nov 03 00:40:19.964923 ip-10-0-133-217 kubenswrapper[2575]: E1103 00:40:19.964476 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5v4k7" podUID="b6abcc85-61c0-42ea-b868-0d71a46a4514" Nov 03 00:40:19.966703 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:19.966569 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6lvpt" Nov 03 00:40:19.966703 ip-10-0-133-217 kubenswrapper[2575]: E1103 00:40:19.966643 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6lvpt" podUID="3dcbff01-09fb-458e-b3a5-594d6b271133" Nov 03 00:40:19.970941 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:19.970309 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-w8k5t" Nov 03 00:40:19.971765 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:19.971137 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-9w99c" Nov 03 00:40:19.974072 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:19.973361 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-lb46r" Nov 03 00:40:19.975268 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:19.975240 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 03 00:40:19.975364 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:19.975304 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 03 00:40:19.975364 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:19.975317 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-d54vs\"" Nov 03 00:40:19.975364 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:19.975316 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 03 00:40:19.975727 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:19.975702 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-zq57z" Nov 03 00:40:19.978672 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:19.978046 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-8rxnt" Nov 03 00:40:19.980644 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:19.980410 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-262mm" Nov 03 00:40:19.980644 ip-10-0-133-217 kubenswrapper[2575]: E1103 00:40:19.980469 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-262mm" podUID="92341144-23c4-4cf0-a0c8-26e1891397e9" Nov 03 00:40:19.982170 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:19.981581 2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Nov 03 00:40:19.983373 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:19.983352 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-trgdf" Nov 03 00:40:19.985301 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:19.985262 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-jltt2" Nov 03 00:40:19.988252 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:19.988019 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-n7ksd" Nov 03 00:40:19.989748 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:19.989661 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dt25z\" (UniqueName: \"kubernetes.io/projected/3dcbff01-09fb-458e-b3a5-594d6b271133-kube-api-access-dt25z\") pod \"network-check-target-6lvpt\" (UID: \"3dcbff01-09fb-458e-b3a5-594d6b271133\") " pod="openshift-network-diagnostics/network-check-target-6lvpt" Nov 03 00:40:19.989748 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:19.989701 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b2d13929-be23-4396-8445-c3eee12e9b9d-host-run-netns\") pod \"multus-lb46r\" (UID: \"b2d13929-be23-4396-8445-c3eee12e9b9d\") " pod="openshift-multus/multus-lb46r" Nov 03 00:40:19.989748 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:19.989727 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/92341144-23c4-4cf0-a0c8-26e1891397e9-original-pull-secret\") pod \"global-pull-secret-syncer-262mm\" (UID: \"92341144-23c4-4cf0-a0c8-26e1891397e9\") " pod="kube-system/global-pull-secret-syncer-262mm" Nov 03 00:40:19.990043 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:19.989770 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/18afc5dd-bc60-4794-91ab-b4019be56a61-system-cni-dir\") pod \"multus-additional-cni-plugins-8rxnt\" (UID: \"18afc5dd-bc60-4794-91ab-b4019be56a61\") " pod="openshift-multus/multus-additional-cni-plugins-8rxnt" Nov 03 00:40:19.990043 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:19.989799 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/174d7fa2-7163-47a8-a047-8727cdb3cb9f-iptables-alerter-script\") pod \"iptables-alerter-9w99c\" (UID: \"174d7fa2-7163-47a8-a047-8727cdb3cb9f\") " pod="openshift-network-operator/iptables-alerter-9w99c" Nov 03 00:40:19.990043 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:19.989840 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b2d13929-be23-4396-8445-c3eee12e9b9d-host-var-lib-cni-bin\") pod \"multus-lb46r\" (UID: \"b2d13929-be23-4396-8445-c3eee12e9b9d\") " pod="openshift-multus/multus-lb46r" Nov 03 00:40:19.990043 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:19.989854 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b2d13929-be23-4396-8445-c3eee12e9b9d-hostroot\") pod \"multus-lb46r\" (UID: \"b2d13929-be23-4396-8445-c3eee12e9b9d\") " pod="openshift-multus/multus-lb46r" Nov 03 00:40:19.990043 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:19.989886 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/29de0f7e-9c96-46dd-8fec-434e3689a047-run-systemd\") pod \"ovnkube-node-zq57z\" (UID: \"29de0f7e-9c96-46dd-8fec-434e3689a047\") " pod="openshift-ovn-kubernetes/ovnkube-node-zq57z" Nov 03 00:40:19.990043 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:19.989902 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/29de0f7e-9c96-46dd-8fec-434e3689a047-ovnkube-config\") pod \"ovnkube-node-zq57z\" (UID: \"29de0f7e-9c96-46dd-8fec-434e3689a047\") " pod="openshift-ovn-kubernetes/ovnkube-node-zq57z" Nov 03 00:40:19.990043 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:19.989938 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b2d13929-be23-4396-8445-c3eee12e9b9d-multus-conf-dir\") pod \"multus-lb46r\" (UID: \"b2d13929-be23-4396-8445-c3eee12e9b9d\") " pod="openshift-multus/multus-lb46r" Nov 03 00:40:19.990043 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:19.989961 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/29de0f7e-9c96-46dd-8fec-434e3689a047-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zq57z\" (UID: \"29de0f7e-9c96-46dd-8fec-434e3689a047\") " pod="openshift-ovn-kubernetes/ovnkube-node-zq57z" Nov 03 00:40:19.990043 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:19.989983 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6ab5d408-02a2-4284-bf59-5827c6f84643-kubelet-dir\") pod \"aws-ebs-csi-driver-node-trgdf\" (UID: \"6ab5d408-02a2-4284-bf59-5827c6f84643\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-trgdf" Nov 03 00:40:19.990043 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:19.989998 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b2d13929-be23-4396-8445-c3eee12e9b9d-host-var-lib-cni-multus\") pod \"multus-lb46r\" (UID: \"b2d13929-be23-4396-8445-c3eee12e9b9d\") " pod="openshift-multus/multus-lb46r" Nov 03 00:40:19.990043 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:19.990012 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/29de0f7e-9c96-46dd-8fec-434e3689a047-systemd-units\") pod \"ovnkube-node-zq57z\" (UID: \"29de0f7e-9c96-46dd-8fec-434e3689a047\") " pod="openshift-ovn-kubernetes/ovnkube-node-zq57z" Nov 03 00:40:19.990043 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:19.990026 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/29de0f7e-9c96-46dd-8fec-434e3689a047-host-run-ovn-kubernetes\") pod \"ovnkube-node-zq57z\" (UID: \"29de0f7e-9c96-46dd-8fec-434e3689a047\") " pod="openshift-ovn-kubernetes/ovnkube-node-zq57z" Nov 03 00:40:19.990738 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:19.990077 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/29de0f7e-9c96-46dd-8fec-434e3689a047-ovn-node-metrics-cert\") pod \"ovnkube-node-zq57z\" (UID: \"29de0f7e-9c96-46dd-8fec-434e3689a047\") " pod="openshift-ovn-kubernetes/ovnkube-node-zq57z" Nov 03 00:40:19.990738 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:19.990098 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5844f274-ad38-42a3-9900-b051b01c9953-host\") pod \"node-ca-w8k5t\" (UID: \"5844f274-ad38-42a3-9900-b051b01c9953\") " pod="openshift-image-registry/node-ca-w8k5t" Nov 03 00:40:19.990738 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:19.990118 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5844f274-ad38-42a3-9900-b051b01c9953-serviceca\") pod \"node-ca-w8k5t\" (UID: \"5844f274-ad38-42a3-9900-b051b01c9953\") " pod="openshift-image-registry/node-ca-w8k5t" Nov 03 00:40:19.990738 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:19.990150 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b2d13929-be23-4396-8445-c3eee12e9b9d-system-cni-dir\") pod \"multus-lb46r\" (UID: \"b2d13929-be23-4396-8445-c3eee12e9b9d\") " pod="openshift-multus/multus-lb46r" Nov 03 00:40:19.990738 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:19.990180 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/29de0f7e-9c96-46dd-8fec-434e3689a047-run-openvswitch\") pod \"ovnkube-node-zq57z\" (UID: \"29de0f7e-9c96-46dd-8fec-434e3689a047\") " pod="openshift-ovn-kubernetes/ovnkube-node-zq57z" Nov 03 00:40:19.990738 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:19.990202 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/29de0f7e-9c96-46dd-8fec-434e3689a047-run-ovn\") pod \"ovnkube-node-zq57z\" (UID: \"29de0f7e-9c96-46dd-8fec-434e3689a047\") " pod="openshift-ovn-kubernetes/ovnkube-node-zq57z" Nov 03 00:40:19.990738 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:19.990224 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-cphks" Nov 03 00:40:19.990738 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:19.990225 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/29de0f7e-9c96-46dd-8fec-434e3689a047-host-cni-bin\") pod \"ovnkube-node-zq57z\" (UID: \"29de0f7e-9c96-46dd-8fec-434e3689a047\") " pod="openshift-ovn-kubernetes/ovnkube-node-zq57z" Nov 03 00:40:19.990738 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:19.990627 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/18afc5dd-bc60-4794-91ab-b4019be56a61-cnibin\") pod \"multus-additional-cni-plugins-8rxnt\" (UID: \"18afc5dd-bc60-4794-91ab-b4019be56a61\") " pod="openshift-multus/multus-additional-cni-plugins-8rxnt" Nov 03 00:40:19.990738 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:19.990679 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/18afc5dd-bc60-4794-91ab-b4019be56a61-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8rxnt\" (UID: \"18afc5dd-bc60-4794-91ab-b4019be56a61\") " pod="openshift-multus/multus-additional-cni-plugins-8rxnt" Nov 03 00:40:19.990738 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:19.990708 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6ab5d408-02a2-4284-bf59-5827c6f84643-etc-selinux\") pod \"aws-ebs-csi-driver-node-trgdf\" (UID: \"6ab5d408-02a2-4284-bf59-5827c6f84643\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-trgdf" Nov 03 00:40:19.990738 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:19.990735 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b2d13929-be23-4396-8445-c3eee12e9b9d-host-var-lib-kubelet\") pod \"multus-lb46r\" (UID: \"b2d13929-be23-4396-8445-c3eee12e9b9d\") " pod="openshift-multus/multus-lb46r" Nov 03 00:40:19.991367 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:19.990795 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/18afc5dd-bc60-4794-91ab-b4019be56a61-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8rxnt\" (UID: \"18afc5dd-bc60-4794-91ab-b4019be56a61\") " pod="openshift-multus/multus-additional-cni-plugins-8rxnt" Nov 03 00:40:19.991367 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:19.990834 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b6abcc85-61c0-42ea-b868-0d71a46a4514-metrics-certs\") pod \"network-metrics-daemon-5v4k7\" (UID: \"b6abcc85-61c0-42ea-b868-0d71a46a4514\") " pod="openshift-multus/network-metrics-daemon-5v4k7" Nov 03 00:40:19.991367 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:19.990870 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5fls\" (UniqueName: \"kubernetes.io/projected/174d7fa2-7163-47a8-a047-8727cdb3cb9f-kube-api-access-c5fls\") pod \"iptables-alerter-9w99c\" (UID: \"174d7fa2-7163-47a8-a047-8727cdb3cb9f\") " pod="openshift-network-operator/iptables-alerter-9w99c" Nov 03 00:40:19.991367 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:19.990945 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b2d13929-be23-4396-8445-c3eee12e9b9d-multus-socket-dir-parent\") pod \"multus-lb46r\" (UID: \"b2d13929-be23-4396-8445-c3eee12e9b9d\") " pod="openshift-multus/multus-lb46r" Nov 03 00:40:19.991367 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:19.990973 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b2d13929-be23-4396-8445-c3eee12e9b9d-multus-daemon-config\") pod \"multus-lb46r\" (UID: \"b2d13929-be23-4396-8445-c3eee12e9b9d\") " pod="openshift-multus/multus-lb46r" Nov 03 00:40:19.991367 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:19.991011 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b2d13929-be23-4396-8445-c3eee12e9b9d-host-run-multus-certs\") pod \"multus-lb46r\" (UID: \"b2d13929-be23-4396-8445-c3eee12e9b9d\") " pod="openshift-multus/multus-lb46r" Nov 03 00:40:19.991367 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:19.991044 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/29de0f7e-9c96-46dd-8fec-434e3689a047-log-socket\") pod \"ovnkube-node-zq57z\" (UID: \"29de0f7e-9c96-46dd-8fec-434e3689a047\") " pod="openshift-ovn-kubernetes/ovnkube-node-zq57z" Nov 03 00:40:19.991367 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:19.991064 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/29de0f7e-9c96-46dd-8fec-434e3689a047-env-overrides\") pod \"ovnkube-node-zq57z\" (UID: \"29de0f7e-9c96-46dd-8fec-434e3689a047\") " pod="openshift-ovn-kubernetes/ovnkube-node-zq57z" Nov 03 00:40:19.991367 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:19.991079 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/174d7fa2-7163-47a8-a047-8727cdb3cb9f-host-slash\") pod \"iptables-alerter-9w99c\" (UID: \"174d7fa2-7163-47a8-a047-8727cdb3cb9f\") " pod="openshift-network-operator/iptables-alerter-9w99c" Nov 03 00:40:19.991367 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:19.991119 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/29de0f7e-9c96-46dd-8fec-434e3689a047-host-slash\") pod \"ovnkube-node-zq57z\" (UID: \"29de0f7e-9c96-46dd-8fec-434e3689a047\") " pod="openshift-ovn-kubernetes/ovnkube-node-zq57z" Nov 03 00:40:19.991367 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:19.991157 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/29de0f7e-9c96-46dd-8fec-434e3689a047-node-log\") pod \"ovnkube-node-zq57z\" (UID: \"29de0f7e-9c96-46dd-8fec-434e3689a047\") " pod="openshift-ovn-kubernetes/ovnkube-node-zq57z" Nov 03 00:40:19.991367 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:19.991181 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/29de0f7e-9c96-46dd-8fec-434e3689a047-ovnkube-script-lib\") pod \"ovnkube-node-zq57z\" (UID: \"29de0f7e-9c96-46dd-8fec-434e3689a047\") " pod="openshift-ovn-kubernetes/ovnkube-node-zq57z" Nov 03 00:40:19.991367 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:19.991215 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45qm2\" (UniqueName: \"kubernetes.io/projected/29de0f7e-9c96-46dd-8fec-434e3689a047-kube-api-access-45qm2\") pod \"ovnkube-node-zq57z\" (UID: \"29de0f7e-9c96-46dd-8fec-434e3689a047\") " pod="openshift-ovn-kubernetes/ovnkube-node-zq57z" Nov 03 00:40:19.991367 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:19.991258 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/18afc5dd-bc60-4794-91ab-b4019be56a61-cni-binary-copy\") pod \"multus-additional-cni-plugins-8rxnt\" (UID: \"18afc5dd-bc60-4794-91ab-b4019be56a61\") " pod="openshift-multus/multus-additional-cni-plugins-8rxnt" Nov 03 00:40:19.991367 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:19.991298 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6ab5d408-02a2-4284-bf59-5827c6f84643-registration-dir\") pod \"aws-ebs-csi-driver-node-trgdf\" (UID: \"6ab5d408-02a2-4284-bf59-5827c6f84643\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-trgdf" Nov 03 00:40:19.991367 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:19.991339 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4c5b\" (UniqueName: \"kubernetes.io/projected/b2d13929-be23-4396-8445-c3eee12e9b9d-kube-api-access-h4c5b\") pod \"multus-lb46r\" (UID: \"b2d13929-be23-4396-8445-c3eee12e9b9d\") " pod="openshift-multus/multus-lb46r" Nov 03 00:40:19.992179 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:19.991366 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/29de0f7e-9c96-46dd-8fec-434e3689a047-host-run-netns\") pod \"ovnkube-node-zq57z\" (UID: \"29de0f7e-9c96-46dd-8fec-434e3689a047\") " pod="openshift-ovn-kubernetes/ovnkube-node-zq57z" Nov 03 00:40:19.992179 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:19.991386 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/29de0f7e-9c96-46dd-8fec-434e3689a047-var-lib-openvswitch\") pod \"ovnkube-node-zq57z\" (UID: \"29de0f7e-9c96-46dd-8fec-434e3689a047\") " pod="openshift-ovn-kubernetes/ovnkube-node-zq57z" Nov 03 00:40:19.992179 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:19.991414 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6ab5d408-02a2-4284-bf59-5827c6f84643-socket-dir\") pod \"aws-ebs-csi-driver-node-trgdf\" (UID: \"6ab5d408-02a2-4284-bf59-5827c6f84643\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-trgdf" Nov 03 00:40:19.992179 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:19.991491 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b2d13929-be23-4396-8445-c3eee12e9b9d-multus-cni-dir\") pod \"multus-lb46r\" (UID: \"b2d13929-be23-4396-8445-c3eee12e9b9d\") " pod="openshift-multus/multus-lb46r" Nov 03 00:40:19.992179 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:19.991517 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b2d13929-be23-4396-8445-c3eee12e9b9d-host-run-k8s-cni-cncf-io\") pod \"multus-lb46r\" (UID: \"b2d13929-be23-4396-8445-c3eee12e9b9d\") " pod="openshift-multus/multus-lb46r" Nov 03 00:40:19.992179 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:19.991548 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/92341144-23c4-4cf0-a0c8-26e1891397e9-dbus\") pod \"global-pull-secret-syncer-262mm\" (UID: \"92341144-23c4-4cf0-a0c8-26e1891397e9\") " pod="kube-system/global-pull-secret-syncer-262mm" Nov 03 00:40:19.992179 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:19.991605 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b2d13929-be23-4396-8445-c3eee12e9b9d-cni-binary-copy\") pod \"multus-lb46r\" (UID: \"b2d13929-be23-4396-8445-c3eee12e9b9d\") " pod="openshift-multus/multus-lb46r" Nov 03 00:40:19.992179 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:19.991644 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qn5dn\" (UniqueName: \"kubernetes.io/projected/5844f274-ad38-42a3-9900-b051b01c9953-kube-api-access-qn5dn\") pod \"node-ca-w8k5t\" (UID: \"5844f274-ad38-42a3-9900-b051b01c9953\") " pod="openshift-image-registry/node-ca-w8k5t" Nov 03 00:40:19.992179 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:19.991692 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2n98j\" (UniqueName: \"kubernetes.io/projected/6ab5d408-02a2-4284-bf59-5827c6f84643-kube-api-access-2n98j\") pod \"aws-ebs-csi-driver-node-trgdf\" (UID: \"6ab5d408-02a2-4284-bf59-5827c6f84643\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-trgdf" Nov 03 00:40:19.992179 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:19.991723 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b2d13929-be23-4396-8445-c3eee12e9b9d-cnibin\") pod \"multus-lb46r\" (UID: \"b2d13929-be23-4396-8445-c3eee12e9b9d\") " pod="openshift-multus/multus-lb46r" Nov 03 00:40:19.992179 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:19.991773 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/18afc5dd-bc60-4794-91ab-b4019be56a61-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8rxnt\" (UID: \"18afc5dd-bc60-4794-91ab-b4019be56a61\") " pod="openshift-multus/multus-additional-cni-plugins-8rxnt" Nov 03 00:40:19.992179 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:19.991810 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsbnr\" (UniqueName: \"kubernetes.io/projected/18afc5dd-bc60-4794-91ab-b4019be56a61-kube-api-access-wsbnr\") pod \"multus-additional-cni-plugins-8rxnt\" (UID: \"18afc5dd-bc60-4794-91ab-b4019be56a61\") " pod="openshift-multus/multus-additional-cni-plugins-8rxnt" Nov 03 00:40:19.992179 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:19.991830 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b2d13929-be23-4396-8445-c3eee12e9b9d-etc-kubernetes\") pod \"multus-lb46r\" (UID: \"b2d13929-be23-4396-8445-c3eee12e9b9d\") " pod="openshift-multus/multus-lb46r" Nov 03 00:40:19.992179 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:19.991859 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/29de0f7e-9c96-46dd-8fec-434e3689a047-etc-openvswitch\") pod \"ovnkube-node-zq57z\" (UID: \"29de0f7e-9c96-46dd-8fec-434e3689a047\") " pod="openshift-ovn-kubernetes/ovnkube-node-zq57z" Nov 03 00:40:19.992179 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:19.991889 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/92341144-23c4-4cf0-a0c8-26e1891397e9-kubelet-config\") pod \"global-pull-secret-syncer-262mm\" (UID: \"92341144-23c4-4cf0-a0c8-26e1891397e9\") " pod="kube-system/global-pull-secret-syncer-262mm" Nov 03 00:40:19.992179 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:19.991923 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6ab5d408-02a2-4284-bf59-5827c6f84643-sys-fs\") pod \"aws-ebs-csi-driver-node-trgdf\" (UID: \"6ab5d408-02a2-4284-bf59-5827c6f84643\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-trgdf" Nov 03 00:40:19.993081 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:19.991949 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzb4d\" (UniqueName: \"kubernetes.io/projected/b6abcc85-61c0-42ea-b868-0d71a46a4514-kube-api-access-lzb4d\") pod \"network-metrics-daemon-5v4k7\" (UID: \"b6abcc85-61c0-42ea-b868-0d71a46a4514\") " pod="openshift-multus/network-metrics-daemon-5v4k7" Nov 03 00:40:19.993081 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:19.992004 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6ab5d408-02a2-4284-bf59-5827c6f84643-device-dir\") pod \"aws-ebs-csi-driver-node-trgdf\" (UID: \"6ab5d408-02a2-4284-bf59-5827c6f84643\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-trgdf" Nov 03 00:40:19.993081 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:19.992030 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/29de0f7e-9c96-46dd-8fec-434e3689a047-host-kubelet\") pod \"ovnkube-node-zq57z\" (UID: \"29de0f7e-9c96-46dd-8fec-434e3689a047\") " pod="openshift-ovn-kubernetes/ovnkube-node-zq57z" Nov 03 00:40:19.993081 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:19.992076 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/29de0f7e-9c96-46dd-8fec-434e3689a047-host-cni-netd\") pod \"ovnkube-node-zq57z\" (UID: \"29de0f7e-9c96-46dd-8fec-434e3689a047\") " pod="openshift-ovn-kubernetes/ovnkube-node-zq57z" Nov 03 00:40:19.998333 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:19.998308 2575 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 03 00:40:20.030389 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.030360 2575 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-bc8bl" Nov 03 00:40:20.040620 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.040588 2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-bc8bl" Nov 03 00:40:20.064234 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.064180 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-217.us-east-2.compute.internal" event={"ID":"fdeeb94bfd911d86834e9d29bdbbadab","Type":"ContainerStarted","Data":"d26b7bbc18a12c91a4663e8aabaac3278b33104993a34ea7f7e9c75dfeb63adb"} Nov 03 00:40:20.065585 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.065552 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-217.us-east-2.compute.internal" event={"ID":"a1171317f26f09b75b7b789f796f1384","Type":"ContainerStarted","Data":"baf995ad5d0a338d5cab169fe531fe16fa1263a06c4429391a64689fce408d2a"} Nov 03 00:40:20.077378 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.077349 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 03 00:40:20.077518 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.077438 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-9v27r\"" Nov 03 00:40:20.077663 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.077647 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-7mmgk\"" Nov 03 00:40:20.078445 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.078416 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 03 00:40:20.078445 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.078431 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-w9l2l\"" Nov 03 00:40:20.078616 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.078532 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 03 00:40:20.078616 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.078581 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 03 00:40:20.078717 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.078660 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 03 00:40:20.079012 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.078994 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-229lw\"" Nov 03 00:40:20.079472 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.079150 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 03 00:40:20.079472 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.079189 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 03 00:40:20.079472 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.079309 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 03 00:40:20.079472 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.079322 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 03 00:40:20.079472 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.079385 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 03 00:40:20.079472 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.079406 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 03 00:40:20.079472 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.079423 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 03 00:40:20.079472 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.079432 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 03 00:40:20.079472 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.079449 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 03 00:40:20.079472 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.079465 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 03 00:40:20.079874 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.079524 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-kp9k4\"" Nov 03 00:40:20.079874 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.079406 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 03 00:40:20.079874 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.079580 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 03 00:40:20.079874 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.079410 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 03 00:40:20.079874 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.079647 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-97hfk\"" Nov 03 00:40:20.079874 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.079651 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-tq982\"" Nov 03 00:40:20.080140 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.080115 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 03 00:40:20.080140 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.080138 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 03 00:40:20.080322 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.080192 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 03 00:40:20.080322 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.080227 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-4zb6t\"" Nov 03 00:40:20.080483 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.080463 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 03 00:40:20.080581 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.080493 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 03 00:40:20.080688 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.080666 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 03 00:40:20.083179 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.083158 2575 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 03 00:40:20.092921 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.092884 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b2d13929-be23-4396-8445-c3eee12e9b9d-cni-binary-copy\") pod \"multus-lb46r\" (UID: \"b2d13929-be23-4396-8445-c3eee12e9b9d\") " pod="openshift-multus/multus-lb46r" Nov 03 00:40:20.093081 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.092932 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7f1faa5-fc0e-4f46-9208-b1e58f50fcf9-tmp-dir\") pod \"node-resolver-cphks\" (UID: \"e7f1faa5-fc0e-4f46-9208-b1e58f50fcf9\") " pod="openshift-dns/node-resolver-cphks" Nov 03 00:40:20.093081 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.092958 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qn5dn\" (UniqueName: \"kubernetes.io/projected/5844f274-ad38-42a3-9900-b051b01c9953-kube-api-access-qn5dn\") pod \"node-ca-w8k5t\" (UID: \"5844f274-ad38-42a3-9900-b051b01c9953\") " pod="openshift-image-registry/node-ca-w8k5t" Nov 03 00:40:20.093204 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.093084 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2n98j\" (UniqueName: \"kubernetes.io/projected/6ab5d408-02a2-4284-bf59-5827c6f84643-kube-api-access-2n98j\") pod \"aws-ebs-csi-driver-node-trgdf\" (UID: \"6ab5d408-02a2-4284-bf59-5827c6f84643\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-trgdf" Nov 03 00:40:20.093346 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.093321 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b2d13929-be23-4396-8445-c3eee12e9b9d-cnibin\") pod \"multus-lb46r\" (UID: \"b2d13929-be23-4396-8445-c3eee12e9b9d\") " pod="openshift-multus/multus-lb46r" Nov 03 00:40:20.093470 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.093355 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/cd47e545-4089-44ad-bc4d-0545061c7b11-agent-certs\") pod \"konnectivity-agent-jltt2\" (UID: \"cd47e545-4089-44ad-bc4d-0545061c7b11\") " pod="kube-system/konnectivity-agent-jltt2" Nov 03 00:40:20.093470 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.093374 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/476ff8a0-f214-4e5e-aa90-8a505974f963-etc-systemd\") pod \"tuned-n7ksd\" (UID: \"476ff8a0-f214-4e5e-aa90-8a505974f963\") " pod="openshift-cluster-node-tuning-operator/tuned-n7ksd" Nov 03 00:40:20.093470 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.093390 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/476ff8a0-f214-4e5e-aa90-8a505974f963-sys\") pod \"tuned-n7ksd\" (UID: \"476ff8a0-f214-4e5e-aa90-8a505974f963\") " pod="openshift-cluster-node-tuning-operator/tuned-n7ksd" Nov 03 00:40:20.093470 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.093411 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/476ff8a0-f214-4e5e-aa90-8a505974f963-host\") pod \"tuned-n7ksd\" (UID: \"476ff8a0-f214-4e5e-aa90-8a505974f963\") " pod="openshift-cluster-node-tuning-operator/tuned-n7ksd" Nov 03 00:40:20.093470 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.093439 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/18afc5dd-bc60-4794-91ab-b4019be56a61-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8rxnt\" (UID: \"18afc5dd-bc60-4794-91ab-b4019be56a61\") " pod="openshift-multus/multus-additional-cni-plugins-8rxnt" Nov 03 00:40:20.093470 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.093457 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsbnr\" (UniqueName: \"kubernetes.io/projected/18afc5dd-bc60-4794-91ab-b4019be56a61-kube-api-access-wsbnr\") pod \"multus-additional-cni-plugins-8rxnt\" (UID: \"18afc5dd-bc60-4794-91ab-b4019be56a61\") " pod="openshift-multus/multus-additional-cni-plugins-8rxnt" Nov 03 00:40:20.093761 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.093537 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b2d13929-be23-4396-8445-c3eee12e9b9d-cnibin\") pod \"multus-lb46r\" (UID: \"b2d13929-be23-4396-8445-c3eee12e9b9d\") " pod="openshift-multus/multus-lb46r" Nov 03 00:40:20.093761 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.093581 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b2d13929-be23-4396-8445-c3eee12e9b9d-etc-kubernetes\") pod \"multus-lb46r\" (UID: \"b2d13929-be23-4396-8445-c3eee12e9b9d\") " pod="openshift-multus/multus-lb46r" Nov 03 00:40:20.093761 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.093620 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/29de0f7e-9c96-46dd-8fec-434e3689a047-etc-openvswitch\") pod \"ovnkube-node-zq57z\" (UID: \"29de0f7e-9c96-46dd-8fec-434e3689a047\") " pod="openshift-ovn-kubernetes/ovnkube-node-zq57z" Nov 03 00:40:20.093761 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.093680 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/92341144-23c4-4cf0-a0c8-26e1891397e9-kubelet-config\") pod \"global-pull-secret-syncer-262mm\" (UID: \"92341144-23c4-4cf0-a0c8-26e1891397e9\") " pod="kube-system/global-pull-secret-syncer-262mm" Nov 03 00:40:20.093761 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.093709 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/476ff8a0-f214-4e5e-aa90-8a505974f963-etc-sysconfig\") pod \"tuned-n7ksd\" (UID: \"476ff8a0-f214-4e5e-aa90-8a505974f963\") " pod="openshift-cluster-node-tuning-operator/tuned-n7ksd" Nov 03 00:40:20.093761 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.093740 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/476ff8a0-f214-4e5e-aa90-8a505974f963-etc-sysctl-d\") pod \"tuned-n7ksd\" (UID: \"476ff8a0-f214-4e5e-aa90-8a505974f963\") " pod="openshift-cluster-node-tuning-operator/tuned-n7ksd" Nov 03 00:40:20.094047 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.093803 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6ab5d408-02a2-4284-bf59-5827c6f84643-sys-fs\") pod \"aws-ebs-csi-driver-node-trgdf\" (UID: \"6ab5d408-02a2-4284-bf59-5827c6f84643\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-trgdf" Nov 03 00:40:20.094047 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.093829 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzb4d\" (UniqueName: \"kubernetes.io/projected/b6abcc85-61c0-42ea-b868-0d71a46a4514-kube-api-access-lzb4d\") pod \"network-metrics-daemon-5v4k7\" (UID: \"b6abcc85-61c0-42ea-b868-0d71a46a4514\") " pod="openshift-multus/network-metrics-daemon-5v4k7" Nov 03 00:40:20.094047 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.093860 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/476ff8a0-f214-4e5e-aa90-8a505974f963-var-lib-kubelet\") pod \"tuned-n7ksd\" (UID: \"476ff8a0-f214-4e5e-aa90-8a505974f963\") " pod="openshift-cluster-node-tuning-operator/tuned-n7ksd" Nov 03 00:40:20.094047 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.093871 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b2d13929-be23-4396-8445-c3eee12e9b9d-etc-kubernetes\") pod \"multus-lb46r\" (UID: \"b2d13929-be23-4396-8445-c3eee12e9b9d\") " pod="openshift-multus/multus-lb46r" Nov 03 00:40:20.094047 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.093883 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6ab5d408-02a2-4284-bf59-5827c6f84643-device-dir\") pod \"aws-ebs-csi-driver-node-trgdf\" (UID: \"6ab5d408-02a2-4284-bf59-5827c6f84643\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-trgdf" Nov 03 00:40:20.094047 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.093944 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6ab5d408-02a2-4284-bf59-5827c6f84643-device-dir\") pod \"aws-ebs-csi-driver-node-trgdf\" (UID: \"6ab5d408-02a2-4284-bf59-5827c6f84643\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-trgdf" Nov 03 00:40:20.094047 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.093979 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6ab5d408-02a2-4284-bf59-5827c6f84643-sys-fs\") pod \"aws-ebs-csi-driver-node-trgdf\" (UID: \"6ab5d408-02a2-4284-bf59-5827c6f84643\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-trgdf" Nov 03 00:40:20.094047 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.093984 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/29de0f7e-9c96-46dd-8fec-434e3689a047-etc-openvswitch\") pod \"ovnkube-node-zq57z\" (UID: \"29de0f7e-9c96-46dd-8fec-434e3689a047\") " pod="openshift-ovn-kubernetes/ovnkube-node-zq57z" Nov 03 00:40:20.094047 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.094019 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/29de0f7e-9c96-46dd-8fec-434e3689a047-host-kubelet\") pod \"ovnkube-node-zq57z\" (UID: \"29de0f7e-9c96-46dd-8fec-434e3689a047\") " pod="openshift-ovn-kubernetes/ovnkube-node-zq57z" Nov 03 00:40:20.094047 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.094048 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/29de0f7e-9c96-46dd-8fec-434e3689a047-host-cni-netd\") pod \"ovnkube-node-zq57z\" (UID: \"29de0f7e-9c96-46dd-8fec-434e3689a047\") " pod="openshift-ovn-kubernetes/ovnkube-node-zq57z" Nov 03 00:40:20.094561 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.094089 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dt25z\" (UniqueName: \"kubernetes.io/projected/3dcbff01-09fb-458e-b3a5-594d6b271133-kube-api-access-dt25z\") pod \"network-check-target-6lvpt\" (UID: \"3dcbff01-09fb-458e-b3a5-594d6b271133\") " pod="openshift-network-diagnostics/network-check-target-6lvpt" Nov 03 00:40:20.094561 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.094116 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b2d13929-be23-4396-8445-c3eee12e9b9d-host-run-netns\") pod \"multus-lb46r\" (UID: \"b2d13929-be23-4396-8445-c3eee12e9b9d\") " pod="openshift-multus/multus-lb46r" Nov 03 00:40:20.094561 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.094146 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/92341144-23c4-4cf0-a0c8-26e1891397e9-original-pull-secret\") pod \"global-pull-secret-syncer-262mm\" (UID: \"92341144-23c4-4cf0-a0c8-26e1891397e9\") " pod="kube-system/global-pull-secret-syncer-262mm" Nov 03 00:40:20.094561 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.094175 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/cd47e545-4089-44ad-bc4d-0545061c7b11-konnectivity-ca\") pod \"konnectivity-agent-jltt2\" (UID: \"cd47e545-4089-44ad-bc4d-0545061c7b11\") " pod="kube-system/konnectivity-agent-jltt2" Nov 03 00:40:20.094561 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.094214 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/92341144-23c4-4cf0-a0c8-26e1891397e9-kubelet-config\") pod \"global-pull-secret-syncer-262mm\" (UID: \"92341144-23c4-4cf0-a0c8-26e1891397e9\") " pod="kube-system/global-pull-secret-syncer-262mm" Nov 03 00:40:20.094561 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.094232 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b2d13929-be23-4396-8445-c3eee12e9b9d-cni-binary-copy\") pod \"multus-lb46r\" (UID: \"b2d13929-be23-4396-8445-c3eee12e9b9d\") " pod="openshift-multus/multus-lb46r" Nov 03 00:40:20.094561 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.094262 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/29de0f7e-9c96-46dd-8fec-434e3689a047-host-kubelet\") pod \"ovnkube-node-zq57z\" (UID: \"29de0f7e-9c96-46dd-8fec-434e3689a047\") " pod="openshift-ovn-kubernetes/ovnkube-node-zq57z" Nov 03 00:40:20.094561 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.094236 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/18afc5dd-bc60-4794-91ab-b4019be56a61-system-cni-dir\") pod \"multus-additional-cni-plugins-8rxnt\" (UID: \"18afc5dd-bc60-4794-91ab-b4019be56a61\") " pod="openshift-multus/multus-additional-cni-plugins-8rxnt" Nov 03 00:40:20.094561 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.094304 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b2d13929-be23-4396-8445-c3eee12e9b9d-host-run-netns\") pod \"multus-lb46r\" (UID: \"b2d13929-be23-4396-8445-c3eee12e9b9d\") " pod="openshift-multus/multus-lb46r" Nov 03 00:40:20.094561 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.094309 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/18afc5dd-bc60-4794-91ab-b4019be56a61-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8rxnt\" (UID: \"18afc5dd-bc60-4794-91ab-b4019be56a61\") " pod="openshift-multus/multus-additional-cni-plugins-8rxnt" Nov 03 00:40:20.094561 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.094315 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/174d7fa2-7163-47a8-a047-8727cdb3cb9f-iptables-alerter-script\") pod \"iptables-alerter-9w99c\" (UID: \"174d7fa2-7163-47a8-a047-8727cdb3cb9f\") " pod="openshift-network-operator/iptables-alerter-9w99c" Nov 03 00:40:20.094561 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.094321 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/18afc5dd-bc60-4794-91ab-b4019be56a61-system-cni-dir\") pod \"multus-additional-cni-plugins-8rxnt\" (UID: \"18afc5dd-bc60-4794-91ab-b4019be56a61\") " pod="openshift-multus/multus-additional-cni-plugins-8rxnt" Nov 03 00:40:20.094561 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.094371 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b2d13929-be23-4396-8445-c3eee12e9b9d-host-var-lib-cni-bin\") pod \"multus-lb46r\" (UID: \"b2d13929-be23-4396-8445-c3eee12e9b9d\") " pod="openshift-multus/multus-lb46r" Nov 03 00:40:20.094561 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.094390 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b2d13929-be23-4396-8445-c3eee12e9b9d-hostroot\") pod \"multus-lb46r\" (UID: \"b2d13929-be23-4396-8445-c3eee12e9b9d\") " pod="openshift-multus/multus-lb46r" Nov 03 00:40:20.094561 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.094406 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/29de0f7e-9c96-46dd-8fec-434e3689a047-run-systemd\") pod \"ovnkube-node-zq57z\" (UID: \"29de0f7e-9c96-46dd-8fec-434e3689a047\") " pod="openshift-ovn-kubernetes/ovnkube-node-zq57z" Nov 03 00:40:20.094561 ip-10-0-133-217 kubenswrapper[2575]: E1103 00:40:20.094472 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 00:40:20.094561 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.094497 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b2d13929-be23-4396-8445-c3eee12e9b9d-host-var-lib-cni-bin\") pod \"multus-lb46r\" (UID: \"b2d13929-be23-4396-8445-c3eee12e9b9d\") " pod="openshift-multus/multus-lb46r" Nov 03 00:40:20.095355 ip-10-0-133-217 kubenswrapper[2575]: E1103 00:40:20.094543 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92341144-23c4-4cf0-a0c8-26e1891397e9-original-pull-secret podName:92341144-23c4-4cf0-a0c8-26e1891397e9 nodeName:}" failed. No retries permitted until 2026-11-03 00:40:20.594504211 +0000 UTC m=+3.039561683 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/92341144-23c4-4cf0-a0c8-26e1891397e9-original-pull-secret") pod "global-pull-secret-syncer-262mm" (UID: "92341144-23c4-4cf0-a0c8-26e1891397e9") : object "kube-system"/"original-pull-secret" not registered Nov 03 00:40:20.095355 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.094543 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/29de0f7e-9c96-46dd-8fec-434e3689a047-host-cni-netd\") pod \"ovnkube-node-zq57z\" (UID: \"29de0f7e-9c96-46dd-8fec-434e3689a047\") " pod="openshift-ovn-kubernetes/ovnkube-node-zq57z" Nov 03 00:40:20.095355 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.094575 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/29de0f7e-9c96-46dd-8fec-434e3689a047-run-systemd\") pod \"ovnkube-node-zq57z\" (UID: \"29de0f7e-9c96-46dd-8fec-434e3689a047\") " pod="openshift-ovn-kubernetes/ovnkube-node-zq57z" Nov 03 00:40:20.095355 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.094598 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/29de0f7e-9c96-46dd-8fec-434e3689a047-ovnkube-config\") pod \"ovnkube-node-zq57z\" (UID: \"29de0f7e-9c96-46dd-8fec-434e3689a047\") " pod="openshift-ovn-kubernetes/ovnkube-node-zq57z" Nov 03 00:40:20.095355 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.094623 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b2d13929-be23-4396-8445-c3eee12e9b9d-multus-conf-dir\") pod \"multus-lb46r\" (UID: \"b2d13929-be23-4396-8445-c3eee12e9b9d\") " pod="openshift-multus/multus-lb46r" Nov 03 00:40:20.095355 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.094639 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/29de0f7e-9c96-46dd-8fec-434e3689a047-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zq57z\" (UID: \"29de0f7e-9c96-46dd-8fec-434e3689a047\") " pod="openshift-ovn-kubernetes/ovnkube-node-zq57z" Nov 03 00:40:20.095355 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.094661 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/476ff8a0-f214-4e5e-aa90-8a505974f963-etc-sysctl-conf\") pod \"tuned-n7ksd\" (UID: \"476ff8a0-f214-4e5e-aa90-8a505974f963\") " pod="openshift-cluster-node-tuning-operator/tuned-n7ksd" Nov 03 00:40:20.095355 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.094691 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/476ff8a0-f214-4e5e-aa90-8a505974f963-lib-modules\") pod \"tuned-n7ksd\" (UID: \"476ff8a0-f214-4e5e-aa90-8a505974f963\") " pod="openshift-cluster-node-tuning-operator/tuned-n7ksd" Nov 03 00:40:20.095355 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.094708 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6ab5d408-02a2-4284-bf59-5827c6f84643-kubelet-dir\") pod \"aws-ebs-csi-driver-node-trgdf\" (UID: \"6ab5d408-02a2-4284-bf59-5827c6f84643\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-trgdf" Nov 03 00:40:20.095355 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.094725 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b2d13929-be23-4396-8445-c3eee12e9b9d-host-var-lib-cni-multus\") pod \"multus-lb46r\" (UID: \"b2d13929-be23-4396-8445-c3eee12e9b9d\") " pod="openshift-multus/multus-lb46r" Nov 03 00:40:20.095355 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.094740 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/29de0f7e-9c96-46dd-8fec-434e3689a047-systemd-units\") pod \"ovnkube-node-zq57z\" (UID: \"29de0f7e-9c96-46dd-8fec-434e3689a047\") " pod="openshift-ovn-kubernetes/ovnkube-node-zq57z" Nov 03 00:40:20.095355 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.094782 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/29de0f7e-9c96-46dd-8fec-434e3689a047-host-run-ovn-kubernetes\") pod \"ovnkube-node-zq57z\" (UID: \"29de0f7e-9c96-46dd-8fec-434e3689a047\") " pod="openshift-ovn-kubernetes/ovnkube-node-zq57z" Nov 03 00:40:20.095355 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.094811 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/29de0f7e-9c96-46dd-8fec-434e3689a047-ovn-node-metrics-cert\") pod \"ovnkube-node-zq57z\" (UID: \"29de0f7e-9c96-46dd-8fec-434e3689a047\") " pod="openshift-ovn-kubernetes/ovnkube-node-zq57z" Nov 03 00:40:20.095355 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.094826 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5844f274-ad38-42a3-9900-b051b01c9953-host\") pod \"node-ca-w8k5t\" (UID: \"5844f274-ad38-42a3-9900-b051b01c9953\") " pod="openshift-image-registry/node-ca-w8k5t" Nov 03 00:40:20.095355 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.094840 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5844f274-ad38-42a3-9900-b051b01c9953-serviceca\") pod \"node-ca-w8k5t\" (UID: \"5844f274-ad38-42a3-9900-b051b01c9953\") " pod="openshift-image-registry/node-ca-w8k5t" Nov 03 00:40:20.095355 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.094857 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b2d13929-be23-4396-8445-c3eee12e9b9d-system-cni-dir\") pod \"multus-lb46r\" (UID: \"b2d13929-be23-4396-8445-c3eee12e9b9d\") " pod="openshift-multus/multus-lb46r" Nov 03 00:40:20.095926 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.094855 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/174d7fa2-7163-47a8-a047-8727cdb3cb9f-iptables-alerter-script\") pod \"iptables-alerter-9w99c\" (UID: \"174d7fa2-7163-47a8-a047-8727cdb3cb9f\") " pod="openshift-network-operator/iptables-alerter-9w99c" Nov 03 00:40:20.095926 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.094873 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/29de0f7e-9c96-46dd-8fec-434e3689a047-run-openvswitch\") pod \"ovnkube-node-zq57z\" (UID: \"29de0f7e-9c96-46dd-8fec-434e3689a047\") " pod="openshift-ovn-kubernetes/ovnkube-node-zq57z" Nov 03 00:40:20.095926 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.094888 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/29de0f7e-9c96-46dd-8fec-434e3689a047-run-ovn\") pod \"ovnkube-node-zq57z\" (UID: \"29de0f7e-9c96-46dd-8fec-434e3689a047\") " pod="openshift-ovn-kubernetes/ovnkube-node-zq57z" Nov 03 00:40:20.095926 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.094911 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/29de0f7e-9c96-46dd-8fec-434e3689a047-host-cni-bin\") pod \"ovnkube-node-zq57z\" (UID: \"29de0f7e-9c96-46dd-8fec-434e3689a047\") " pod="openshift-ovn-kubernetes/ovnkube-node-zq57z" Nov 03 00:40:20.095926 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.094918 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/29de0f7e-9c96-46dd-8fec-434e3689a047-host-run-ovn-kubernetes\") pod \"ovnkube-node-zq57z\" (UID: \"29de0f7e-9c96-46dd-8fec-434e3689a047\") " pod="openshift-ovn-kubernetes/ovnkube-node-zq57z" Nov 03 00:40:20.095926 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.094927 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e7f1faa5-fc0e-4f46-9208-b1e58f50fcf9-hosts-file\") pod \"node-resolver-cphks\" (UID: \"e7f1faa5-fc0e-4f46-9208-b1e58f50fcf9\") " pod="openshift-dns/node-resolver-cphks" Nov 03 00:40:20.095926 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.095069 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/29de0f7e-9c96-46dd-8fec-434e3689a047-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zq57z\" (UID: \"29de0f7e-9c96-46dd-8fec-434e3689a047\") " pod="openshift-ovn-kubernetes/ovnkube-node-zq57z" Nov 03 00:40:20.095926 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.095097 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b2d13929-be23-4396-8445-c3eee12e9b9d-hostroot\") pod \"multus-lb46r\" (UID: \"b2d13929-be23-4396-8445-c3eee12e9b9d\") " pod="openshift-multus/multus-lb46r" Nov 03 00:40:20.095926 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.095110 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6ab5d408-02a2-4284-bf59-5827c6f84643-kubelet-dir\") pod \"aws-ebs-csi-driver-node-trgdf\" (UID: \"6ab5d408-02a2-4284-bf59-5827c6f84643\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-trgdf" Nov 03 00:40:20.095926 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.095099 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/18afc5dd-bc60-4794-91ab-b4019be56a61-cnibin\") pod \"multus-additional-cni-plugins-8rxnt\" (UID: \"18afc5dd-bc60-4794-91ab-b4019be56a61\") " pod="openshift-multus/multus-additional-cni-plugins-8rxnt" Nov 03 00:40:20.095926 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.095137 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b2d13929-be23-4396-8445-c3eee12e9b9d-multus-conf-dir\") pod \"multus-lb46r\" (UID: \"b2d13929-be23-4396-8445-c3eee12e9b9d\") " pod="openshift-multus/multus-lb46r" Nov 03 00:40:20.095926 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.095152 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b2d13929-be23-4396-8445-c3eee12e9b9d-system-cni-dir\") pod \"multus-lb46r\" (UID: \"b2d13929-be23-4396-8445-c3eee12e9b9d\") " pod="openshift-multus/multus-lb46r" Nov 03 00:40:20.095926 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.095191 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5844f274-ad38-42a3-9900-b051b01c9953-host\") pod \"node-ca-w8k5t\" (UID: \"5844f274-ad38-42a3-9900-b051b01c9953\") " pod="openshift-image-registry/node-ca-w8k5t" Nov 03 00:40:20.095926 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.095203 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b2d13929-be23-4396-8445-c3eee12e9b9d-host-var-lib-cni-multus\") pod \"multus-lb46r\" (UID: \"b2d13929-be23-4396-8445-c3eee12e9b9d\") " pod="openshift-multus/multus-lb46r" Nov 03 00:40:20.095926 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.095229 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/29de0f7e-9c96-46dd-8fec-434e3689a047-run-ovn\") pod \"ovnkube-node-zq57z\" (UID: \"29de0f7e-9c96-46dd-8fec-434e3689a047\") " pod="openshift-ovn-kubernetes/ovnkube-node-zq57z" Nov 03 00:40:20.095926 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.095214 2575 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 03 00:40:20.095926 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.095270 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/29de0f7e-9c96-46dd-8fec-434e3689a047-host-cni-bin\") pod \"ovnkube-node-zq57z\" (UID: \"29de0f7e-9c96-46dd-8fec-434e3689a047\") " pod="openshift-ovn-kubernetes/ovnkube-node-zq57z" Nov 03 00:40:20.095926 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.095299 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/18afc5dd-bc60-4794-91ab-b4019be56a61-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8rxnt\" (UID: \"18afc5dd-bc60-4794-91ab-b4019be56a61\") " pod="openshift-multus/multus-additional-cni-plugins-8rxnt" Nov 03 00:40:20.096588 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.095330 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6ab5d408-02a2-4284-bf59-5827c6f84643-etc-selinux\") pod \"aws-ebs-csi-driver-node-trgdf\" (UID: \"6ab5d408-02a2-4284-bf59-5827c6f84643\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-trgdf" Nov 03 00:40:20.096588 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.095334 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/29de0f7e-9c96-46dd-8fec-434e3689a047-systemd-units\") pod \"ovnkube-node-zq57z\" (UID: \"29de0f7e-9c96-46dd-8fec-434e3689a047\") " pod="openshift-ovn-kubernetes/ovnkube-node-zq57z" Nov 03 00:40:20.096588 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.095389 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b2d13929-be23-4396-8445-c3eee12e9b9d-host-var-lib-kubelet\") pod \"multus-lb46r\" (UID: \"b2d13929-be23-4396-8445-c3eee12e9b9d\") " pod="openshift-multus/multus-lb46r" Nov 03 00:40:20.096588 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.095419 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/18afc5dd-bc60-4794-91ab-b4019be56a61-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8rxnt\" (UID: \"18afc5dd-bc60-4794-91ab-b4019be56a61\") " pod="openshift-multus/multus-additional-cni-plugins-8rxnt" Nov 03 00:40:20.096588 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.095447 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b6abcc85-61c0-42ea-b868-0d71a46a4514-metrics-certs\") pod \"network-metrics-daemon-5v4k7\" (UID: \"b6abcc85-61c0-42ea-b868-0d71a46a4514\") " pod="openshift-multus/network-metrics-daemon-5v4k7" Nov 03 00:40:20.096588 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.095462 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6ab5d408-02a2-4284-bf59-5827c6f84643-etc-selinux\") pod \"aws-ebs-csi-driver-node-trgdf\" (UID: \"6ab5d408-02a2-4284-bf59-5827c6f84643\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-trgdf" Nov 03 00:40:20.096588 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.095480 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5844f274-ad38-42a3-9900-b051b01c9953-serviceca\") pod \"node-ca-w8k5t\" (UID: \"5844f274-ad38-42a3-9900-b051b01c9953\") " pod="openshift-image-registry/node-ca-w8k5t" Nov 03 00:40:20.096588 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.095536 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/29de0f7e-9c96-46dd-8fec-434e3689a047-ovnkube-config\") pod \"ovnkube-node-zq57z\" (UID: \"29de0f7e-9c96-46dd-8fec-434e3689a047\") " pod="openshift-ovn-kubernetes/ovnkube-node-zq57z" Nov 03 00:40:20.096588 ip-10-0-133-217 kubenswrapper[2575]: E1103 00:40:20.095539 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 00:40:20.096588 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.095564 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5fls\" (UniqueName: \"kubernetes.io/projected/174d7fa2-7163-47a8-a047-8727cdb3cb9f-kube-api-access-c5fls\") pod \"iptables-alerter-9w99c\" (UID: \"174d7fa2-7163-47a8-a047-8727cdb3cb9f\") " pod="openshift-network-operator/iptables-alerter-9w99c" Nov 03 00:40:20.096588 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.095572 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/18afc5dd-bc60-4794-91ab-b4019be56a61-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8rxnt\" (UID: \"18afc5dd-bc60-4794-91ab-b4019be56a61\") " pod="openshift-multus/multus-additional-cni-plugins-8rxnt" Nov 03 00:40:20.096588 ip-10-0-133-217 kubenswrapper[2575]: E1103 00:40:20.095598 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b6abcc85-61c0-42ea-b868-0d71a46a4514-metrics-certs podName:b6abcc85-61c0-42ea-b868-0d71a46a4514 nodeName:}" failed. No retries permitted until 2026-11-03 00:40:20.595581226 +0000 UTC m=+3.040638683 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b6abcc85-61c0-42ea-b868-0d71a46a4514-metrics-certs") pod "network-metrics-daemon-5v4k7" (UID: "b6abcc85-61c0-42ea-b868-0d71a46a4514") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 00:40:20.096588 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.095597 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b2d13929-be23-4396-8445-c3eee12e9b9d-host-var-lib-kubelet\") pod \"multus-lb46r\" (UID: \"b2d13929-be23-4396-8445-c3eee12e9b9d\") " pod="openshift-multus/multus-lb46r" Nov 03 00:40:20.096588 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.095616 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b2d13929-be23-4396-8445-c3eee12e9b9d-multus-socket-dir-parent\") pod \"multus-lb46r\" (UID: \"b2d13929-be23-4396-8445-c3eee12e9b9d\") " pod="openshift-multus/multus-lb46r" Nov 03 00:40:20.096588 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.095625 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/29de0f7e-9c96-46dd-8fec-434e3689a047-run-openvswitch\") pod \"ovnkube-node-zq57z\" (UID: \"29de0f7e-9c96-46dd-8fec-434e3689a047\") " pod="openshift-ovn-kubernetes/ovnkube-node-zq57z" Nov 03 00:40:20.096588 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.095672 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/18afc5dd-bc60-4794-91ab-b4019be56a61-cnibin\") pod \"multus-additional-cni-plugins-8rxnt\" (UID: \"18afc5dd-bc60-4794-91ab-b4019be56a61\") " pod="openshift-multus/multus-additional-cni-plugins-8rxnt" Nov 03 00:40:20.096588 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.095696 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b2d13929-be23-4396-8445-c3eee12e9b9d-multus-daemon-config\") pod \"multus-lb46r\" (UID: \"b2d13929-be23-4396-8445-c3eee12e9b9d\") " pod="openshift-multus/multus-lb46r" Nov 03 00:40:20.097371 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.095728 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b2d13929-be23-4396-8445-c3eee12e9b9d-multus-socket-dir-parent\") pod \"multus-lb46r\" (UID: \"b2d13929-be23-4396-8445-c3eee12e9b9d\") " pod="openshift-multus/multus-lb46r" Nov 03 00:40:20.097371 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.095770 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b2d13929-be23-4396-8445-c3eee12e9b9d-host-run-multus-certs\") pod \"multus-lb46r\" (UID: \"b2d13929-be23-4396-8445-c3eee12e9b9d\") " pod="openshift-multus/multus-lb46r" Nov 03 00:40:20.097371 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.096388 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/29de0f7e-9c96-46dd-8fec-434e3689a047-log-socket\") pod \"ovnkube-node-zq57z\" (UID: \"29de0f7e-9c96-46dd-8fec-434e3689a047\") " pod="openshift-ovn-kubernetes/ovnkube-node-zq57z" Nov 03 00:40:20.097371 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.096437 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/18afc5dd-bc60-4794-91ab-b4019be56a61-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8rxnt\" (UID: \"18afc5dd-bc60-4794-91ab-b4019be56a61\") " pod="openshift-multus/multus-additional-cni-plugins-8rxnt" Nov 03 00:40:20.097371 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.095868 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b2d13929-be23-4396-8445-c3eee12e9b9d-host-run-multus-certs\") pod \"multus-lb46r\" (UID: \"b2d13929-be23-4396-8445-c3eee12e9b9d\") " pod="openshift-multus/multus-lb46r" Nov 03 00:40:20.097371 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.096854 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/29de0f7e-9c96-46dd-8fec-434e3689a047-env-overrides\") pod \"ovnkube-node-zq57z\" (UID: \"29de0f7e-9c96-46dd-8fec-434e3689a047\") " pod="openshift-ovn-kubernetes/ovnkube-node-zq57z" Nov 03 00:40:20.097371 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.096875 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/29de0f7e-9c96-46dd-8fec-434e3689a047-log-socket\") pod \"ovnkube-node-zq57z\" (UID: \"29de0f7e-9c96-46dd-8fec-434e3689a047\") " pod="openshift-ovn-kubernetes/ovnkube-node-zq57z" Nov 03 00:40:20.097371 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.096907 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b2d13929-be23-4396-8445-c3eee12e9b9d-multus-daemon-config\") pod \"multus-lb46r\" (UID: \"b2d13929-be23-4396-8445-c3eee12e9b9d\") " pod="openshift-multus/multus-lb46r" Nov 03 00:40:20.097371 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.096920 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/174d7fa2-7163-47a8-a047-8727cdb3cb9f-host-slash\") pod \"iptables-alerter-9w99c\" (UID: \"174d7fa2-7163-47a8-a047-8727cdb3cb9f\") " pod="openshift-network-operator/iptables-alerter-9w99c" Nov 03 00:40:20.097371 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.096965 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/29de0f7e-9c96-46dd-8fec-434e3689a047-host-slash\") pod \"ovnkube-node-zq57z\" (UID: \"29de0f7e-9c96-46dd-8fec-434e3689a047\") " pod="openshift-ovn-kubernetes/ovnkube-node-zq57z" Nov 03 00:40:20.097371 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.096988 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/29de0f7e-9c96-46dd-8fec-434e3689a047-node-log\") pod \"ovnkube-node-zq57z\" (UID: \"29de0f7e-9c96-46dd-8fec-434e3689a047\") " pod="openshift-ovn-kubernetes/ovnkube-node-zq57z" Nov 03 00:40:20.097371 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.097082 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/29de0f7e-9c96-46dd-8fec-434e3689a047-ovnkube-script-lib\") pod \"ovnkube-node-zq57z\" (UID: \"29de0f7e-9c96-46dd-8fec-434e3689a047\") " pod="openshift-ovn-kubernetes/ovnkube-node-zq57z" Nov 03 00:40:20.097371 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.097114 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45qm2\" (UniqueName: \"kubernetes.io/projected/29de0f7e-9c96-46dd-8fec-434e3689a047-kube-api-access-45qm2\") pod \"ovnkube-node-zq57z\" (UID: \"29de0f7e-9c96-46dd-8fec-434e3689a047\") " pod="openshift-ovn-kubernetes/ovnkube-node-zq57z" Nov 03 00:40:20.097371 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.097180 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/476ff8a0-f214-4e5e-aa90-8a505974f963-run\") pod \"tuned-n7ksd\" (UID: \"476ff8a0-f214-4e5e-aa90-8a505974f963\") " pod="openshift-cluster-node-tuning-operator/tuned-n7ksd" Nov 03 00:40:20.097371 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.097220 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/476ff8a0-f214-4e5e-aa90-8a505974f963-etc-tuned\") pod \"tuned-n7ksd\" (UID: \"476ff8a0-f214-4e5e-aa90-8a505974f963\") " pod="openshift-cluster-node-tuning-operator/tuned-n7ksd" Nov 03 00:40:20.098006 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.097317 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/29de0f7e-9c96-46dd-8fec-434e3689a047-env-overrides\") pod \"ovnkube-node-zq57z\" (UID: \"29de0f7e-9c96-46dd-8fec-434e3689a047\") " pod="openshift-ovn-kubernetes/ovnkube-node-zq57z" Nov 03 00:40:20.098006 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.097500 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqbgf\" (UniqueName: \"kubernetes.io/projected/476ff8a0-f214-4e5e-aa90-8a505974f963-kube-api-access-lqbgf\") pod \"tuned-n7ksd\" (UID: \"476ff8a0-f214-4e5e-aa90-8a505974f963\") " pod="openshift-cluster-node-tuning-operator/tuned-n7ksd" Nov 03 00:40:20.098006 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.097544 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/18afc5dd-bc60-4794-91ab-b4019be56a61-cni-binary-copy\") pod \"multus-additional-cni-plugins-8rxnt\" (UID: \"18afc5dd-bc60-4794-91ab-b4019be56a61\") " pod="openshift-multus/multus-additional-cni-plugins-8rxnt" Nov 03 00:40:20.098006 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.097623 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6ab5d408-02a2-4284-bf59-5827c6f84643-registration-dir\") pod \"aws-ebs-csi-driver-node-trgdf\" (UID: \"6ab5d408-02a2-4284-bf59-5827c6f84643\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-trgdf" Nov 03 00:40:20.098006 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.097665 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4c5b\" (UniqueName: \"kubernetes.io/projected/b2d13929-be23-4396-8445-c3eee12e9b9d-kube-api-access-h4c5b\") pod \"multus-lb46r\" (UID: \"b2d13929-be23-4396-8445-c3eee12e9b9d\") " pod="openshift-multus/multus-lb46r" Nov 03 00:40:20.098006 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.097701 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/29de0f7e-9c96-46dd-8fec-434e3689a047-host-run-netns\") pod \"ovnkube-node-zq57z\" (UID: \"29de0f7e-9c96-46dd-8fec-434e3689a047\") " pod="openshift-ovn-kubernetes/ovnkube-node-zq57z" Nov 03 00:40:20.098006 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.097733 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/29de0f7e-9c96-46dd-8fec-434e3689a047-var-lib-openvswitch\") pod \"ovnkube-node-zq57z\" (UID: \"29de0f7e-9c96-46dd-8fec-434e3689a047\") " pod="openshift-ovn-kubernetes/ovnkube-node-zq57z" Nov 03 00:40:20.098006 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.097774 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/476ff8a0-f214-4e5e-aa90-8a505974f963-etc-modprobe-d\") pod \"tuned-n7ksd\" (UID: \"476ff8a0-f214-4e5e-aa90-8a505974f963\") " pod="openshift-cluster-node-tuning-operator/tuned-n7ksd" Nov 03 00:40:20.098365 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.098134 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/29de0f7e-9c96-46dd-8fec-434e3689a047-ovnkube-script-lib\") pod \"ovnkube-node-zq57z\" (UID: \"29de0f7e-9c96-46dd-8fec-434e3689a047\") " pod="openshift-ovn-kubernetes/ovnkube-node-zq57z" Nov 03 00:40:20.098365 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.098219 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/476ff8a0-f214-4e5e-aa90-8a505974f963-etc-kubernetes\") pod \"tuned-n7ksd\" (UID: \"476ff8a0-f214-4e5e-aa90-8a505974f963\") " pod="openshift-cluster-node-tuning-operator/tuned-n7ksd" Nov 03 00:40:20.098365 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.098267 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6ab5d408-02a2-4284-bf59-5827c6f84643-socket-dir\") pod \"aws-ebs-csi-driver-node-trgdf\" (UID: \"6ab5d408-02a2-4284-bf59-5827c6f84643\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-trgdf" Nov 03 00:40:20.098365 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.098308 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/29de0f7e-9c96-46dd-8fec-434e3689a047-var-lib-openvswitch\") pod \"ovnkube-node-zq57z\" (UID: \"29de0f7e-9c96-46dd-8fec-434e3689a047\") " pod="openshift-ovn-kubernetes/ovnkube-node-zq57z" Nov 03 00:40:20.098365 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.098323 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b2d13929-be23-4396-8445-c3eee12e9b9d-multus-cni-dir\") pod \"multus-lb46r\" (UID: \"b2d13929-be23-4396-8445-c3eee12e9b9d\") " pod="openshift-multus/multus-lb46r" Nov 03 00:40:20.098365 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.098363 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b2d13929-be23-4396-8445-c3eee12e9b9d-host-run-k8s-cni-cncf-io\") pod \"multus-lb46r\" (UID: \"b2d13929-be23-4396-8445-c3eee12e9b9d\") " pod="openshift-multus/multus-lb46r" Nov 03 00:40:20.098624 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.098369 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6ab5d408-02a2-4284-bf59-5827c6f84643-registration-dir\") pod \"aws-ebs-csi-driver-node-trgdf\" (UID: \"6ab5d408-02a2-4284-bf59-5827c6f84643\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-trgdf" Nov 03 00:40:20.098756 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.098733 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/92341144-23c4-4cf0-a0c8-26e1891397e9-dbus\") pod \"global-pull-secret-syncer-262mm\" (UID: \"92341144-23c4-4cf0-a0c8-26e1891397e9\") " pod="kube-system/global-pull-secret-syncer-262mm" Nov 03 00:40:20.098820 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.098778 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/18afc5dd-bc60-4794-91ab-b4019be56a61-cni-binary-copy\") pod \"multus-additional-cni-plugins-8rxnt\" (UID: \"18afc5dd-bc60-4794-91ab-b4019be56a61\") " pod="openshift-multus/multus-additional-cni-plugins-8rxnt" Nov 03 00:40:20.098820 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.098781 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/476ff8a0-f214-4e5e-aa90-8a505974f963-tmp\") pod \"tuned-n7ksd\" (UID: \"476ff8a0-f214-4e5e-aa90-8a505974f963\") " pod="openshift-cluster-node-tuning-operator/tuned-n7ksd" Nov 03 00:40:20.098912 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.098842 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8t7dh\" (UniqueName: \"kubernetes.io/projected/e7f1faa5-fc0e-4f46-9208-b1e58f50fcf9-kube-api-access-8t7dh\") pod \"node-resolver-cphks\" (UID: \"e7f1faa5-fc0e-4f46-9208-b1e58f50fcf9\") " pod="openshift-dns/node-resolver-cphks" Nov 03 00:40:20.098912 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.098878 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b2d13929-be23-4396-8445-c3eee12e9b9d-host-run-k8s-cni-cncf-io\") pod \"multus-lb46r\" (UID: \"b2d13929-be23-4396-8445-c3eee12e9b9d\") " pod="openshift-multus/multus-lb46r" Nov 03 00:40:20.099011 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.098918 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/29de0f7e-9c96-46dd-8fec-434e3689a047-node-log\") pod \"ovnkube-node-zq57z\" (UID: \"29de0f7e-9c96-46dd-8fec-434e3689a047\") " pod="openshift-ovn-kubernetes/ovnkube-node-zq57z" Nov 03 00:40:20.099011 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.098960 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/174d7fa2-7163-47a8-a047-8727cdb3cb9f-host-slash\") pod \"iptables-alerter-9w99c\" (UID: \"174d7fa2-7163-47a8-a047-8727cdb3cb9f\") " pod="openshift-network-operator/iptables-alerter-9w99c" Nov 03 00:40:20.099011 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.098977 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6ab5d408-02a2-4284-bf59-5827c6f84643-socket-dir\") pod \"aws-ebs-csi-driver-node-trgdf\" (UID: \"6ab5d408-02a2-4284-bf59-5827c6f84643\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-trgdf" Nov 03 00:40:20.099011 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.098844 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/29de0f7e-9c96-46dd-8fec-434e3689a047-host-run-netns\") pod \"ovnkube-node-zq57z\" (UID: \"29de0f7e-9c96-46dd-8fec-434e3689a047\") " pod="openshift-ovn-kubernetes/ovnkube-node-zq57z" Nov 03 00:40:20.099011 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.098997 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/29de0f7e-9c96-46dd-8fec-434e3689a047-host-slash\") pod \"ovnkube-node-zq57z\" (UID: \"29de0f7e-9c96-46dd-8fec-434e3689a047\") " pod="openshift-ovn-kubernetes/ovnkube-node-zq57z" Nov 03 00:40:20.099229 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.099103 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b2d13929-be23-4396-8445-c3eee12e9b9d-multus-cni-dir\") pod \"multus-lb46r\" (UID: \"b2d13929-be23-4396-8445-c3eee12e9b9d\") " pod="openshift-multus/multus-lb46r" Nov 03 00:40:20.099363 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.099334 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/92341144-23c4-4cf0-a0c8-26e1891397e9-dbus\") pod \"global-pull-secret-syncer-262mm\" (UID: \"92341144-23c4-4cf0-a0c8-26e1891397e9\") " pod="kube-system/global-pull-secret-syncer-262mm" Nov 03 00:40:20.099632 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.099608 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/29de0f7e-9c96-46dd-8fec-434e3689a047-ovn-node-metrics-cert\") pod \"ovnkube-node-zq57z\" (UID: \"29de0f7e-9c96-46dd-8fec-434e3689a047\") " pod="openshift-ovn-kubernetes/ovnkube-node-zq57z" Nov 03 00:40:20.107201 ip-10-0-133-217 kubenswrapper[2575]: E1103 00:40:20.106566 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 00:40:20.107201 ip-10-0-133-217 kubenswrapper[2575]: E1103 00:40:20.106594 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 00:40:20.107201 ip-10-0-133-217 kubenswrapper[2575]: E1103 00:40:20.106609 2575 projected.go:196] Error preparing data for projected volume kube-api-access-dt25z for pod openshift-network-diagnostics/network-check-target-6lvpt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 00:40:20.107201 ip-10-0-133-217 kubenswrapper[2575]: E1103 00:40:20.106679 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3dcbff01-09fb-458e-b3a5-594d6b271133-kube-api-access-dt25z podName:3dcbff01-09fb-458e-b3a5-594d6b271133 nodeName:}" failed. No retries permitted until 2026-11-03 00:40:20.606657648 +0000 UTC m=+3.051715109 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-dt25z" (UniqueName: "kubernetes.io/projected/3dcbff01-09fb-458e-b3a5-594d6b271133-kube-api-access-dt25z") pod "network-check-target-6lvpt" (UID: "3dcbff01-09fb-458e-b3a5-594d6b271133") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 00:40:20.108653 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.108555 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsbnr\" (UniqueName: \"kubernetes.io/projected/18afc5dd-bc60-4794-91ab-b4019be56a61-kube-api-access-wsbnr\") pod \"multus-additional-cni-plugins-8rxnt\" (UID: \"18afc5dd-bc60-4794-91ab-b4019be56a61\") " pod="openshift-multus/multus-additional-cni-plugins-8rxnt" Nov 03 00:40:20.109940 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.109880 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5fls\" (UniqueName: \"kubernetes.io/projected/174d7fa2-7163-47a8-a047-8727cdb3cb9f-kube-api-access-c5fls\") pod \"iptables-alerter-9w99c\" (UID: \"174d7fa2-7163-47a8-a047-8727cdb3cb9f\") " pod="openshift-network-operator/iptables-alerter-9w99c" Nov 03 00:40:20.110337 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.110313 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qn5dn\" (UniqueName: \"kubernetes.io/projected/5844f274-ad38-42a3-9900-b051b01c9953-kube-api-access-qn5dn\") pod \"node-ca-w8k5t\" (UID: \"5844f274-ad38-42a3-9900-b051b01c9953\") " pod="openshift-image-registry/node-ca-w8k5t" Nov 03 00:40:20.110874 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.110854 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2n98j\" (UniqueName: \"kubernetes.io/projected/6ab5d408-02a2-4284-bf59-5827c6f84643-kube-api-access-2n98j\") pod \"aws-ebs-csi-driver-node-trgdf\" (UID: \"6ab5d408-02a2-4284-bf59-5827c6f84643\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-trgdf" Nov 03 00:40:20.114356 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.114268 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzb4d\" (UniqueName: \"kubernetes.io/projected/b6abcc85-61c0-42ea-b868-0d71a46a4514-kube-api-access-lzb4d\") pod \"network-metrics-daemon-5v4k7\" (UID: \"b6abcc85-61c0-42ea-b868-0d71a46a4514\") " pod="openshift-multus/network-metrics-daemon-5v4k7" Nov 03 00:40:20.115859 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.115836 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4c5b\" (UniqueName: \"kubernetes.io/projected/b2d13929-be23-4396-8445-c3eee12e9b9d-kube-api-access-h4c5b\") pod \"multus-lb46r\" (UID: \"b2d13929-be23-4396-8445-c3eee12e9b9d\") " pod="openshift-multus/multus-lb46r" Nov 03 00:40:20.116382 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.116359 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45qm2\" (UniqueName: \"kubernetes.io/projected/29de0f7e-9c96-46dd-8fec-434e3689a047-kube-api-access-45qm2\") pod \"ovnkube-node-zq57z\" (UID: \"29de0f7e-9c96-46dd-8fec-434e3689a047\") " pod="openshift-ovn-kubernetes/ovnkube-node-zq57z" Nov 03 00:40:20.199244 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.199147 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/476ff8a0-f214-4e5e-aa90-8a505974f963-etc-sysctl-conf\") pod \"tuned-n7ksd\" (UID: \"476ff8a0-f214-4e5e-aa90-8a505974f963\") " pod="openshift-cluster-node-tuning-operator/tuned-n7ksd" Nov 03 00:40:20.199244 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.199178 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/476ff8a0-f214-4e5e-aa90-8a505974f963-lib-modules\") pod \"tuned-n7ksd\" (UID: \"476ff8a0-f214-4e5e-aa90-8a505974f963\") " pod="openshift-cluster-node-tuning-operator/tuned-n7ksd" Nov 03 00:40:20.199244 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.199215 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e7f1faa5-fc0e-4f46-9208-b1e58f50fcf9-hosts-file\") pod \"node-resolver-cphks\" (UID: \"e7f1faa5-fc0e-4f46-9208-b1e58f50fcf9\") " pod="openshift-dns/node-resolver-cphks" Nov 03 00:40:20.199591 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.199259 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/476ff8a0-f214-4e5e-aa90-8a505974f963-run\") pod \"tuned-n7ksd\" (UID: \"476ff8a0-f214-4e5e-aa90-8a505974f963\") " pod="openshift-cluster-node-tuning-operator/tuned-n7ksd" Nov 03 00:40:20.199591 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.199289 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/476ff8a0-f214-4e5e-aa90-8a505974f963-etc-tuned\") pod \"tuned-n7ksd\" (UID: \"476ff8a0-f214-4e5e-aa90-8a505974f963\") " pod="openshift-cluster-node-tuning-operator/tuned-n7ksd" Nov 03 00:40:20.199591 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.199310 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqbgf\" (UniqueName: \"kubernetes.io/projected/476ff8a0-f214-4e5e-aa90-8a505974f963-kube-api-access-lqbgf\") pod \"tuned-n7ksd\" (UID: \"476ff8a0-f214-4e5e-aa90-8a505974f963\") " pod="openshift-cluster-node-tuning-operator/tuned-n7ksd" Nov 03 00:40:20.199591 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.199341 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/476ff8a0-f214-4e5e-aa90-8a505974f963-etc-modprobe-d\") pod \"tuned-n7ksd\" (UID: \"476ff8a0-f214-4e5e-aa90-8a505974f963\") " pod="openshift-cluster-node-tuning-operator/tuned-n7ksd" Nov 03 00:40:20.199591 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.199365 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/476ff8a0-f214-4e5e-aa90-8a505974f963-etc-kubernetes\") pod \"tuned-n7ksd\" (UID: \"476ff8a0-f214-4e5e-aa90-8a505974f963\") " pod="openshift-cluster-node-tuning-operator/tuned-n7ksd" Nov 03 00:40:20.199591 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.199388 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/476ff8a0-f214-4e5e-aa90-8a505974f963-tmp\") pod \"tuned-n7ksd\" (UID: \"476ff8a0-f214-4e5e-aa90-8a505974f963\") " pod="openshift-cluster-node-tuning-operator/tuned-n7ksd" Nov 03 00:40:20.199591 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.199425 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8t7dh\" (UniqueName: \"kubernetes.io/projected/e7f1faa5-fc0e-4f46-9208-b1e58f50fcf9-kube-api-access-8t7dh\") pod \"node-resolver-cphks\" (UID: \"e7f1faa5-fc0e-4f46-9208-b1e58f50fcf9\") " pod="openshift-dns/node-resolver-cphks" Nov 03 00:40:20.199591 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.199454 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7f1faa5-fc0e-4f46-9208-b1e58f50fcf9-tmp-dir\") pod \"node-resolver-cphks\" (UID: \"e7f1faa5-fc0e-4f46-9208-b1e58f50fcf9\") " pod="openshift-dns/node-resolver-cphks" Nov 03 00:40:20.199591 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.199480 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/cd47e545-4089-44ad-bc4d-0545061c7b11-agent-certs\") pod \"konnectivity-agent-jltt2\" (UID: \"cd47e545-4089-44ad-bc4d-0545061c7b11\") " pod="kube-system/konnectivity-agent-jltt2" Nov 03 00:40:20.199591 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.199502 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/476ff8a0-f214-4e5e-aa90-8a505974f963-etc-systemd\") pod \"tuned-n7ksd\" (UID: \"476ff8a0-f214-4e5e-aa90-8a505974f963\") " pod="openshift-cluster-node-tuning-operator/tuned-n7ksd" Nov 03 00:40:20.199591 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.199524 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/476ff8a0-f214-4e5e-aa90-8a505974f963-sys\") pod \"tuned-n7ksd\" (UID: \"476ff8a0-f214-4e5e-aa90-8a505974f963\") " pod="openshift-cluster-node-tuning-operator/tuned-n7ksd" Nov 03 00:40:20.199591 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.199545 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/476ff8a0-f214-4e5e-aa90-8a505974f963-host\") pod \"tuned-n7ksd\" (UID: \"476ff8a0-f214-4e5e-aa90-8a505974f963\") " pod="openshift-cluster-node-tuning-operator/tuned-n7ksd" Nov 03 00:40:20.199591 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.199583 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/476ff8a0-f214-4e5e-aa90-8a505974f963-etc-sysconfig\") pod \"tuned-n7ksd\" (UID: \"476ff8a0-f214-4e5e-aa90-8a505974f963\") " pod="openshift-cluster-node-tuning-operator/tuned-n7ksd" Nov 03 00:40:20.200127 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.199605 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/476ff8a0-f214-4e5e-aa90-8a505974f963-etc-sysctl-d\") pod \"tuned-n7ksd\" (UID: \"476ff8a0-f214-4e5e-aa90-8a505974f963\") " pod="openshift-cluster-node-tuning-operator/tuned-n7ksd" Nov 03 00:40:20.200127 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.199636 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/476ff8a0-f214-4e5e-aa90-8a505974f963-var-lib-kubelet\") pod \"tuned-n7ksd\" (UID: \"476ff8a0-f214-4e5e-aa90-8a505974f963\") " pod="openshift-cluster-node-tuning-operator/tuned-n7ksd" Nov 03 00:40:20.200127 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.199681 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/cd47e545-4089-44ad-bc4d-0545061c7b11-konnectivity-ca\") pod \"konnectivity-agent-jltt2\" (UID: \"cd47e545-4089-44ad-bc4d-0545061c7b11\") " pod="kube-system/konnectivity-agent-jltt2" Nov 03 00:40:20.200270 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.200245 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/cd47e545-4089-44ad-bc4d-0545061c7b11-konnectivity-ca\") pod \"konnectivity-agent-jltt2\" (UID: \"cd47e545-4089-44ad-bc4d-0545061c7b11\") " pod="kube-system/konnectivity-agent-jltt2" Nov 03 00:40:20.201666 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.200366 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/476ff8a0-f214-4e5e-aa90-8a505974f963-etc-sysconfig\") pod \"tuned-n7ksd\" (UID: \"476ff8a0-f214-4e5e-aa90-8a505974f963\") " pod="openshift-cluster-node-tuning-operator/tuned-n7ksd" Nov 03 00:40:20.201666 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.200378 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/476ff8a0-f214-4e5e-aa90-8a505974f963-sys\") pod \"tuned-n7ksd\" (UID: \"476ff8a0-f214-4e5e-aa90-8a505974f963\") " pod="openshift-cluster-node-tuning-operator/tuned-n7ksd" Nov 03 00:40:20.201666 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.200444 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/476ff8a0-f214-4e5e-aa90-8a505974f963-host\") pod \"tuned-n7ksd\" (UID: \"476ff8a0-f214-4e5e-aa90-8a505974f963\") " pod="openshift-cluster-node-tuning-operator/tuned-n7ksd" Nov 03 00:40:20.201666 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.200476 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/476ff8a0-f214-4e5e-aa90-8a505974f963-etc-sysctl-d\") pod \"tuned-n7ksd\" (UID: \"476ff8a0-f214-4e5e-aa90-8a505974f963\") " pod="openshift-cluster-node-tuning-operator/tuned-n7ksd" Nov 03 00:40:20.201666 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.200522 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/476ff8a0-f214-4e5e-aa90-8a505974f963-var-lib-kubelet\") pod \"tuned-n7ksd\" (UID: \"476ff8a0-f214-4e5e-aa90-8a505974f963\") " pod="openshift-cluster-node-tuning-operator/tuned-n7ksd" Nov 03 00:40:20.201666 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.200588 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/476ff8a0-f214-4e5e-aa90-8a505974f963-run\") pod \"tuned-n7ksd\" (UID: \"476ff8a0-f214-4e5e-aa90-8a505974f963\") " pod="openshift-cluster-node-tuning-operator/tuned-n7ksd" Nov 03 00:40:20.201666 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.200714 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/476ff8a0-f214-4e5e-aa90-8a505974f963-etc-sysctl-conf\") pod \"tuned-n7ksd\" (UID: \"476ff8a0-f214-4e5e-aa90-8a505974f963\") " pod="openshift-cluster-node-tuning-operator/tuned-n7ksd" Nov 03 00:40:20.201666 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.200730 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7f1faa5-fc0e-4f46-9208-b1e58f50fcf9-tmp-dir\") pod \"node-resolver-cphks\" (UID: \"e7f1faa5-fc0e-4f46-9208-b1e58f50fcf9\") " pod="openshift-dns/node-resolver-cphks" Nov 03 00:40:20.201666 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.200811 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/476ff8a0-f214-4e5e-aa90-8a505974f963-lib-modules\") pod \"tuned-n7ksd\" (UID: \"476ff8a0-f214-4e5e-aa90-8a505974f963\") " pod="openshift-cluster-node-tuning-operator/tuned-n7ksd" Nov 03 00:40:20.201666 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.200870 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e7f1faa5-fc0e-4f46-9208-b1e58f50fcf9-hosts-file\") pod \"node-resolver-cphks\" (UID: \"e7f1faa5-fc0e-4f46-9208-b1e58f50fcf9\") " pod="openshift-dns/node-resolver-cphks" Nov 03 00:40:20.201666 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.201289 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/476ff8a0-f214-4e5e-aa90-8a505974f963-etc-modprobe-d\") pod \"tuned-n7ksd\" (UID: \"476ff8a0-f214-4e5e-aa90-8a505974f963\") " pod="openshift-cluster-node-tuning-operator/tuned-n7ksd" Nov 03 00:40:20.201666 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.201306 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/476ff8a0-f214-4e5e-aa90-8a505974f963-etc-systemd\") pod \"tuned-n7ksd\" (UID: \"476ff8a0-f214-4e5e-aa90-8a505974f963\") " pod="openshift-cluster-node-tuning-operator/tuned-n7ksd" Nov 03 00:40:20.201666 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.201392 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/476ff8a0-f214-4e5e-aa90-8a505974f963-etc-kubernetes\") pod \"tuned-n7ksd\" (UID: \"476ff8a0-f214-4e5e-aa90-8a505974f963\") " pod="openshift-cluster-node-tuning-operator/tuned-n7ksd" Nov 03 00:40:20.203938 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.203909 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/476ff8a0-f214-4e5e-aa90-8a505974f963-etc-tuned\") pod \"tuned-n7ksd\" (UID: \"476ff8a0-f214-4e5e-aa90-8a505974f963\") " pod="openshift-cluster-node-tuning-operator/tuned-n7ksd" Nov 03 00:40:20.204543 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.204517 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/476ff8a0-f214-4e5e-aa90-8a505974f963-tmp\") pod \"tuned-n7ksd\" (UID: \"476ff8a0-f214-4e5e-aa90-8a505974f963\") " pod="openshift-cluster-node-tuning-operator/tuned-n7ksd" Nov 03 00:40:20.204721 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.204701 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/cd47e545-4089-44ad-bc4d-0545061c7b11-agent-certs\") pod \"konnectivity-agent-jltt2\" (UID: \"cd47e545-4089-44ad-bc4d-0545061c7b11\") " pod="kube-system/konnectivity-agent-jltt2" Nov 03 00:40:20.210208 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.210179 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8t7dh\" (UniqueName: \"kubernetes.io/projected/e7f1faa5-fc0e-4f46-9208-b1e58f50fcf9-kube-api-access-8t7dh\") pod \"node-resolver-cphks\" (UID: \"e7f1faa5-fc0e-4f46-9208-b1e58f50fcf9\") " pod="openshift-dns/node-resolver-cphks" Nov 03 00:40:20.210441 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.210413 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqbgf\" (UniqueName: \"kubernetes.io/projected/476ff8a0-f214-4e5e-aa90-8a505974f963-kube-api-access-lqbgf\") pod \"tuned-n7ksd\" (UID: \"476ff8a0-f214-4e5e-aa90-8a505974f963\") " pod="openshift-cluster-node-tuning-operator/tuned-n7ksd" Nov 03 00:40:20.287384 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.287340 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-w8k5t" Nov 03 00:40:20.297962 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:20.297928 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5844f274_ad38_42a3_9900_b051b01c9953.slice/crio-f9e348c6d21aff71e0966f83a17915b5fb972bb78f9804902ac570b8049d91d3 WatchSource:0}: Error finding container f9e348c6d21aff71e0966f83a17915b5fb972bb78f9804902ac570b8049d91d3: Status 404 returned error can't find the container with id f9e348c6d21aff71e0966f83a17915b5fb972bb78f9804902ac570b8049d91d3 Nov 03 00:40:20.303930 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.303902 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-9w99c" Nov 03 00:40:20.313478 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:20.313444 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod174d7fa2_7163_47a8_a047_8727cdb3cb9f.slice/crio-ec1b91b125fcf362dae30d04e6bd5f32dc1c657adf0aebc80e5ae74c4d7c32e0 WatchSource:0}: Error finding container ec1b91b125fcf362dae30d04e6bd5f32dc1c657adf0aebc80e5ae74c4d7c32e0: Status 404 returned error can't find the container with id ec1b91b125fcf362dae30d04e6bd5f32dc1c657adf0aebc80e5ae74c4d7c32e0 Nov 03 00:40:20.316802 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.316779 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-lb46r" Nov 03 00:40:20.326463 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:20.326430 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2d13929_be23_4396_8445_c3eee12e9b9d.slice/crio-8da6814d8364a7135cd62e8752fc9462c556c73a806f3d2c8968892be5ee923e WatchSource:0}: Error finding container 8da6814d8364a7135cd62e8752fc9462c556c73a806f3d2c8968892be5ee923e: Status 404 returned error can't find the container with id 8da6814d8364a7135cd62e8752fc9462c556c73a806f3d2c8968892be5ee923e Nov 03 00:40:20.342889 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.342857 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-zq57z" Nov 03 00:40:20.349289 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.349245 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-jltt2" Nov 03 00:40:20.353159 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:20.353127 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29de0f7e_9c96_46dd_8fec_434e3689a047.slice/crio-b4b5234880d4312f754619ef4e1d8f7b3b0cc1d1de0d269baa813a9d050048d7 WatchSource:0}: Error finding container b4b5234880d4312f754619ef4e1d8f7b3b0cc1d1de0d269baa813a9d050048d7: Status 404 returned error can't find the container with id b4b5234880d4312f754619ef4e1d8f7b3b0cc1d1de0d269baa813a9d050048d7 Nov 03 00:40:20.356225 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.356203 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-trgdf" Nov 03 00:40:20.359300 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:20.359251 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd47e545_4089_44ad_bc4d_0545061c7b11.slice/crio-eaba454a940bc816d97e79a2e397bdb6ba69fdf2c1c1691c7e89f4f0832939e7 WatchSource:0}: Error finding container eaba454a940bc816d97e79a2e397bdb6ba69fdf2c1c1691c7e89f4f0832939e7: Status 404 returned error can't find the container with id eaba454a940bc816d97e79a2e397bdb6ba69fdf2c1c1691c7e89f4f0832939e7 Nov 03 00:40:20.362667 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.362467 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-n7ksd" Nov 03 00:40:20.366745 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.366707 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-8rxnt" Nov 03 00:40:20.367354 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:20.367326 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ab5d408_02a2_4284_bf59_5827c6f84643.slice/crio-d00486ba1df0fa7cdbd2b3470a55384be63a1cb897beeb9e8e00b20c30b94823 WatchSource:0}: Error finding container d00486ba1df0fa7cdbd2b3470a55384be63a1cb897beeb9e8e00b20c30b94823: Status 404 returned error can't find the container with id d00486ba1df0fa7cdbd2b3470a55384be63a1cb897beeb9e8e00b20c30b94823 Nov 03 00:40:20.368670 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.368647 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-cphks" Nov 03 00:40:20.374736 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:20.374697 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod476ff8a0_f214_4e5e_aa90_8a505974f963.slice/crio-32de8796ec42699867c6f9bbef833eff854667f5a77f714ebd3474e303f47d2a WatchSource:0}: Error finding container 32de8796ec42699867c6f9bbef833eff854667f5a77f714ebd3474e303f47d2a: Status 404 returned error can't find the container with id 32de8796ec42699867c6f9bbef833eff854667f5a77f714ebd3474e303f47d2a Nov 03 00:40:20.380918 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:20.380892 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18afc5dd_bc60_4794_91ab_b4019be56a61.slice/crio-0c5dc40a50488cf5e29018d41a984f698d723f309f28c83c293d048ad0ba09dd WatchSource:0}: Error finding container 0c5dc40a50488cf5e29018d41a984f698d723f309f28c83c293d048ad0ba09dd: Status 404 returned error can't find the container with id 0c5dc40a50488cf5e29018d41a984f698d723f309f28c83c293d048ad0ba09dd Nov 03 00:40:20.383087 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:20.383063 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7f1faa5_fc0e_4f46_9208_b1e58f50fcf9.slice/crio-7fe87d05e5214377cd693e21d77c16e252b8ad1eff614e3bd29a94d0463a701b WatchSource:0}: Error finding container 7fe87d05e5214377cd693e21d77c16e252b8ad1eff614e3bd29a94d0463a701b: Status 404 returned error can't find the container with id 7fe87d05e5214377cd693e21d77c16e252b8ad1eff614e3bd29a94d0463a701b Nov 03 00:40:20.602856 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.602816 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b6abcc85-61c0-42ea-b868-0d71a46a4514-metrics-certs\") pod \"network-metrics-daemon-5v4k7\" (UID: \"b6abcc85-61c0-42ea-b868-0d71a46a4514\") " pod="openshift-multus/network-metrics-daemon-5v4k7" Nov 03 00:40:20.603028 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.602907 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/92341144-23c4-4cf0-a0c8-26e1891397e9-original-pull-secret\") pod \"global-pull-secret-syncer-262mm\" (UID: \"92341144-23c4-4cf0-a0c8-26e1891397e9\") " pod="kube-system/global-pull-secret-syncer-262mm" Nov 03 00:40:20.603096 ip-10-0-133-217 kubenswrapper[2575]: E1103 00:40:20.603031 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 00:40:20.603146 ip-10-0-133-217 kubenswrapper[2575]: E1103 00:40:20.603095 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92341144-23c4-4cf0-a0c8-26e1891397e9-original-pull-secret podName:92341144-23c4-4cf0-a0c8-26e1891397e9 nodeName:}" failed. No retries permitted until 2026-11-03 00:40:21.603077483 +0000 UTC m=+4.048134941 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/92341144-23c4-4cf0-a0c8-26e1891397e9-original-pull-secret") pod "global-pull-secret-syncer-262mm" (UID: "92341144-23c4-4cf0-a0c8-26e1891397e9") : object "kube-system"/"original-pull-secret" not registered Nov 03 00:40:20.603533 ip-10-0-133-217 kubenswrapper[2575]: E1103 00:40:20.603511 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 00:40:20.603653 ip-10-0-133-217 kubenswrapper[2575]: E1103 00:40:20.603567 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b6abcc85-61c0-42ea-b868-0d71a46a4514-metrics-certs podName:b6abcc85-61c0-42ea-b868-0d71a46a4514 nodeName:}" failed. No retries permitted until 2026-11-03 00:40:21.603552574 +0000 UTC m=+4.048610033 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b6abcc85-61c0-42ea-b868-0d71a46a4514-metrics-certs") pod "network-metrics-daemon-5v4k7" (UID: "b6abcc85-61c0-42ea-b868-0d71a46a4514") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 00:40:20.703630 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:20.703589 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dt25z\" (UniqueName: \"kubernetes.io/projected/3dcbff01-09fb-458e-b3a5-594d6b271133-kube-api-access-dt25z\") pod \"network-check-target-6lvpt\" (UID: \"3dcbff01-09fb-458e-b3a5-594d6b271133\") " pod="openshift-network-diagnostics/network-check-target-6lvpt" Nov 03 00:40:20.703827 ip-10-0-133-217 kubenswrapper[2575]: E1103 00:40:20.703805 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 00:40:20.703827 ip-10-0-133-217 kubenswrapper[2575]: E1103 00:40:20.703827 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 00:40:20.703827 ip-10-0-133-217 kubenswrapper[2575]: E1103 00:40:20.703836 2575 projected.go:196] Error preparing data for projected volume kube-api-access-dt25z for pod openshift-network-diagnostics/network-check-target-6lvpt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 00:40:20.704026 ip-10-0-133-217 kubenswrapper[2575]: E1103 00:40:20.703883 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3dcbff01-09fb-458e-b3a5-594d6b271133-kube-api-access-dt25z podName:3dcbff01-09fb-458e-b3a5-594d6b271133 nodeName:}" failed. No retries permitted until 2026-11-03 00:40:21.703871372 +0000 UTC m=+4.148928827 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-dt25z" (UniqueName: "kubernetes.io/projected/3dcbff01-09fb-458e-b3a5-594d6b271133-kube-api-access-dt25z") pod "network-check-target-6lvpt" (UID: "3dcbff01-09fb-458e-b3a5-594d6b271133") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 00:40:21.043965 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:21.043625 2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 00:35:20 +0000 UTC" deadline="2028-06-16 06:12:40.498138463 +0000 UTC" Nov 03 00:40:21.043965 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:21.043851 2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14189h32m19.454292341s" Nov 03 00:40:21.058678 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:21.058126 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-262mm" Nov 03 00:40:21.058678 ip-10-0-133-217 kubenswrapper[2575]: E1103 00:40:21.058258 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-262mm" podUID="92341144-23c4-4cf0-a0c8-26e1891397e9" Nov 03 00:40:21.083076 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:21.077837 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-217.us-east-2.compute.internal" event={"ID":"a1171317f26f09b75b7b789f796f1384","Type":"ContainerStarted","Data":"b0a2b773a641306f4e9959431d9faacbd9461bea3824a5c6ceae00068ed04ddc"} Nov 03 00:40:21.083363 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:21.083335 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-cphks" event={"ID":"e7f1faa5-fc0e-4f46-9208-b1e58f50fcf9","Type":"ContainerStarted","Data":"7fe87d05e5214377cd693e21d77c16e252b8ad1eff614e3bd29a94d0463a701b"} Nov 03 00:40:21.097336 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:21.093940 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-trgdf" event={"ID":"6ab5d408-02a2-4284-bf59-5827c6f84643","Type":"ContainerStarted","Data":"d00486ba1df0fa7cdbd2b3470a55384be63a1cb897beeb9e8e00b20c30b94823"} Nov 03 00:40:21.097553 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:21.097518 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zq57z" event={"ID":"29de0f7e-9c96-46dd-8fec-434e3689a047","Type":"ContainerStarted","Data":"b4b5234880d4312f754619ef4e1d8f7b3b0cc1d1de0d269baa813a9d050048d7"} Nov 03 00:40:21.103240 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:21.103164 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-lb46r" event={"ID":"b2d13929-be23-4396-8445-c3eee12e9b9d","Type":"ContainerStarted","Data":"8da6814d8364a7135cd62e8752fc9462c556c73a806f3d2c8968892be5ee923e"} Nov 03 00:40:21.106852 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:21.106771 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-w8k5t" event={"ID":"5844f274-ad38-42a3-9900-b051b01c9953","Type":"ContainerStarted","Data":"f9e348c6d21aff71e0966f83a17915b5fb972bb78f9804902ac570b8049d91d3"} Nov 03 00:40:21.120877 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:21.120828 2575 generic.go:356] "Generic (PLEG): container finished" podID="fdeeb94bfd911d86834e9d29bdbbadab" containerID="00557970e2b0aac26fb1fe305bc704b99b3e296fcc7310e2a74186b8102140c1" exitCode=0 Nov 03 00:40:21.121110 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:21.121090 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-217.us-east-2.compute.internal" event={"ID":"fdeeb94bfd911d86834e9d29bdbbadab","Type":"ContainerDied","Data":"00557970e2b0aac26fb1fe305bc704b99b3e296fcc7310e2a74186b8102140c1"} Nov 03 00:40:21.131876 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:21.131794 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8rxnt" event={"ID":"18afc5dd-bc60-4794-91ab-b4019be56a61","Type":"ContainerStarted","Data":"0c5dc40a50488cf5e29018d41a984f698d723f309f28c83c293d048ad0ba09dd"} Nov 03 00:40:21.137341 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:21.136615 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-217.us-east-2.compute.internal" podStartSLOduration=2.136600605 podStartE2EDuration="2.136600605s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 00:40:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:40:21.094136474 +0000 UTC m=+3.539193930" watchObservedRunningTime="2026-11-03 00:40:21.136600605 +0000 UTC m=+3.581658071" Nov 03 00:40:21.138530 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:21.138454 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-n7ksd" event={"ID":"476ff8a0-f214-4e5e-aa90-8a505974f963","Type":"ContainerStarted","Data":"32de8796ec42699867c6f9bbef833eff854667f5a77f714ebd3474e303f47d2a"} Nov 03 00:40:21.142792 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:21.142729 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-jltt2" event={"ID":"cd47e545-4089-44ad-bc4d-0545061c7b11","Type":"ContainerStarted","Data":"eaba454a940bc816d97e79a2e397bdb6ba69fdf2c1c1691c7e89f4f0832939e7"} Nov 03 00:40:21.145913 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:21.145882 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-9w99c" event={"ID":"174d7fa2-7163-47a8-a047-8727cdb3cb9f","Type":"ContainerStarted","Data":"ec1b91b125fcf362dae30d04e6bd5f32dc1c657adf0aebc80e5ae74c4d7c32e0"} Nov 03 00:40:21.613083 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:21.612237 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b6abcc85-61c0-42ea-b868-0d71a46a4514-metrics-certs\") pod \"network-metrics-daemon-5v4k7\" (UID: \"b6abcc85-61c0-42ea-b868-0d71a46a4514\") " pod="openshift-multus/network-metrics-daemon-5v4k7" Nov 03 00:40:21.613083 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:21.612349 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/92341144-23c4-4cf0-a0c8-26e1891397e9-original-pull-secret\") pod \"global-pull-secret-syncer-262mm\" (UID: \"92341144-23c4-4cf0-a0c8-26e1891397e9\") " pod="kube-system/global-pull-secret-syncer-262mm" Nov 03 00:40:21.613083 ip-10-0-133-217 kubenswrapper[2575]: E1103 00:40:21.612475 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 00:40:21.613083 ip-10-0-133-217 kubenswrapper[2575]: E1103 00:40:21.612539 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92341144-23c4-4cf0-a0c8-26e1891397e9-original-pull-secret podName:92341144-23c4-4cf0-a0c8-26e1891397e9 nodeName:}" failed. No retries permitted until 2026-11-03 00:40:23.612519569 +0000 UTC m=+6.057577014 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/92341144-23c4-4cf0-a0c8-26e1891397e9-original-pull-secret") pod "global-pull-secret-syncer-262mm" (UID: "92341144-23c4-4cf0-a0c8-26e1891397e9") : object "kube-system"/"original-pull-secret" not registered Nov 03 00:40:21.613083 ip-10-0-133-217 kubenswrapper[2575]: E1103 00:40:21.612939 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 00:40:21.613083 ip-10-0-133-217 kubenswrapper[2575]: E1103 00:40:21.612995 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b6abcc85-61c0-42ea-b868-0d71a46a4514-metrics-certs podName:b6abcc85-61c0-42ea-b868-0d71a46a4514 nodeName:}" failed. No retries permitted until 2026-11-03 00:40:23.612979421 +0000 UTC m=+6.058036870 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b6abcc85-61c0-42ea-b868-0d71a46a4514-metrics-certs") pod "network-metrics-daemon-5v4k7" (UID: "b6abcc85-61c0-42ea-b868-0d71a46a4514") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 00:40:21.713686 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:21.713644 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dt25z\" (UniqueName: \"kubernetes.io/projected/3dcbff01-09fb-458e-b3a5-594d6b271133-kube-api-access-dt25z\") pod \"network-check-target-6lvpt\" (UID: \"3dcbff01-09fb-458e-b3a5-594d6b271133\") " pod="openshift-network-diagnostics/network-check-target-6lvpt" Nov 03 00:40:21.713877 ip-10-0-133-217 kubenswrapper[2575]: E1103 00:40:21.713827 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 00:40:21.713877 ip-10-0-133-217 kubenswrapper[2575]: E1103 00:40:21.713847 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 00:40:21.713877 ip-10-0-133-217 kubenswrapper[2575]: E1103 00:40:21.713860 2575 projected.go:196] Error preparing data for projected volume kube-api-access-dt25z for pod openshift-network-diagnostics/network-check-target-6lvpt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 00:40:21.714029 ip-10-0-133-217 kubenswrapper[2575]: E1103 00:40:21.713912 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3dcbff01-09fb-458e-b3a5-594d6b271133-kube-api-access-dt25z podName:3dcbff01-09fb-458e-b3a5-594d6b271133 nodeName:}" failed. No retries permitted until 2026-11-03 00:40:23.713899187 +0000 UTC m=+6.158956630 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-dt25z" (UniqueName: "kubernetes.io/projected/3dcbff01-09fb-458e-b3a5-594d6b271133-kube-api-access-dt25z") pod "network-check-target-6lvpt" (UID: "3dcbff01-09fb-458e-b3a5-594d6b271133") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 00:40:22.044565 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:22.044462 2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 00:35:20 +0000 UTC" deadline="2028-06-15 03:38:06.801341583 +0000 UTC" Nov 03 00:40:22.044565 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:22.044531 2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14162h57m44.756815179s" Nov 03 00:40:22.062768 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:22.061991 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5v4k7" Nov 03 00:40:22.062768 ip-10-0-133-217 kubenswrapper[2575]: E1103 00:40:22.062143 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5v4k7" podUID="b6abcc85-61c0-42ea-b868-0d71a46a4514" Nov 03 00:40:22.062768 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:22.062588 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6lvpt" Nov 03 00:40:22.062768 ip-10-0-133-217 kubenswrapper[2575]: E1103 00:40:22.062681 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6lvpt" podUID="3dcbff01-09fb-458e-b3a5-594d6b271133" Nov 03 00:40:22.178869 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:22.177985 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-217.us-east-2.compute.internal" event={"ID":"fdeeb94bfd911d86834e9d29bdbbadab","Type":"ContainerStarted","Data":"a5317886b2cb85855082058b7d59790632950ccefa1a6205312a5ad79a0a2f39"} Nov 03 00:40:23.058761 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:23.058706 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-262mm" Nov 03 00:40:23.059256 ip-10-0-133-217 kubenswrapper[2575]: E1103 00:40:23.058902 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-262mm" podUID="92341144-23c4-4cf0-a0c8-26e1891397e9" Nov 03 00:40:23.631504 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:23.630130 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b6abcc85-61c0-42ea-b868-0d71a46a4514-metrics-certs\") pod \"network-metrics-daemon-5v4k7\" (UID: \"b6abcc85-61c0-42ea-b868-0d71a46a4514\") " pod="openshift-multus/network-metrics-daemon-5v4k7" Nov 03 00:40:23.631504 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:23.630229 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/92341144-23c4-4cf0-a0c8-26e1891397e9-original-pull-secret\") pod \"global-pull-secret-syncer-262mm\" (UID: \"92341144-23c4-4cf0-a0c8-26e1891397e9\") " pod="kube-system/global-pull-secret-syncer-262mm" Nov 03 00:40:23.631504 ip-10-0-133-217 kubenswrapper[2575]: E1103 00:40:23.630412 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 00:40:23.631504 ip-10-0-133-217 kubenswrapper[2575]: E1103 00:40:23.630447 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 00:40:23.631504 ip-10-0-133-217 kubenswrapper[2575]: E1103 00:40:23.630469 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b6abcc85-61c0-42ea-b868-0d71a46a4514-metrics-certs podName:b6abcc85-61c0-42ea-b868-0d71a46a4514 nodeName:}" failed. No retries permitted until 2026-11-03 00:40:27.630455796 +0000 UTC m=+10.075513253 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b6abcc85-61c0-42ea-b868-0d71a46a4514-metrics-certs") pod "network-metrics-daemon-5v4k7" (UID: "b6abcc85-61c0-42ea-b868-0d71a46a4514") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 00:40:23.631504 ip-10-0-133-217 kubenswrapper[2575]: E1103 00:40:23.630511 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92341144-23c4-4cf0-a0c8-26e1891397e9-original-pull-secret podName:92341144-23c4-4cf0-a0c8-26e1891397e9 nodeName:}" failed. No retries permitted until 2026-11-03 00:40:27.630494861 +0000 UTC m=+10.075552318 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/92341144-23c4-4cf0-a0c8-26e1891397e9-original-pull-secret") pod "global-pull-secret-syncer-262mm" (UID: "92341144-23c4-4cf0-a0c8-26e1891397e9") : object "kube-system"/"original-pull-secret" not registered Nov 03 00:40:23.731268 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:23.731224 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dt25z\" (UniqueName: \"kubernetes.io/projected/3dcbff01-09fb-458e-b3a5-594d6b271133-kube-api-access-dt25z\") pod \"network-check-target-6lvpt\" (UID: \"3dcbff01-09fb-458e-b3a5-594d6b271133\") " pod="openshift-network-diagnostics/network-check-target-6lvpt" Nov 03 00:40:23.731461 ip-10-0-133-217 kubenswrapper[2575]: E1103 00:40:23.731433 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 00:40:23.731461 ip-10-0-133-217 kubenswrapper[2575]: E1103 00:40:23.731460 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 00:40:23.731553 ip-10-0-133-217 kubenswrapper[2575]: E1103 00:40:23.731473 2575 projected.go:196] Error preparing data for projected volume kube-api-access-dt25z for pod openshift-network-diagnostics/network-check-target-6lvpt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 00:40:23.731553 ip-10-0-133-217 kubenswrapper[2575]: E1103 00:40:23.731533 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3dcbff01-09fb-458e-b3a5-594d6b271133-kube-api-access-dt25z podName:3dcbff01-09fb-458e-b3a5-594d6b271133 nodeName:}" failed. No retries permitted until 2026-11-03 00:40:27.731515703 +0000 UTC m=+10.176573150 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-dt25z" (UniqueName: "kubernetes.io/projected/3dcbff01-09fb-458e-b3a5-594d6b271133-kube-api-access-dt25z") pod "network-check-target-6lvpt" (UID: "3dcbff01-09fb-458e-b3a5-594d6b271133") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 00:40:24.059418 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:24.059386 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6lvpt" Nov 03 00:40:24.059890 ip-10-0-133-217 kubenswrapper[2575]: E1103 00:40:24.059518 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6lvpt" podUID="3dcbff01-09fb-458e-b3a5-594d6b271133" Nov 03 00:40:24.060029 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:24.059991 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5v4k7" Nov 03 00:40:24.060133 ip-10-0-133-217 kubenswrapper[2575]: E1103 00:40:24.060104 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5v4k7" podUID="b6abcc85-61c0-42ea-b868-0d71a46a4514" Nov 03 00:40:24.199545 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:24.199504 2575 generic.go:356] "Generic (PLEG): container finished" podID="18afc5dd-bc60-4794-91ab-b4019be56a61" containerID="ffc09ec0336dc44907202879030aa74d8d9a837c7717d78202167e7799d0f8d8" exitCode=0 Nov 03 00:40:24.199718 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:24.199566 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8rxnt" event={"ID":"18afc5dd-bc60-4794-91ab-b4019be56a61","Type":"ContainerDied","Data":"ffc09ec0336dc44907202879030aa74d8d9a837c7717d78202167e7799d0f8d8"} Nov 03 00:40:24.226031 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:24.223060 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-217.us-east-2.compute.internal" podStartSLOduration=5.223043671 podStartE2EDuration="5.223043671s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 00:40:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:40:22.192619357 +0000 UTC m=+4.637676823" watchObservedRunningTime="2026-11-03 00:40:24.223043671 +0000 UTC m=+6.668101135" Nov 03 00:40:25.058818 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:25.058736 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-262mm" Nov 03 00:40:25.059044 ip-10-0-133-217 kubenswrapper[2575]: E1103 00:40:25.058878 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-262mm" podUID="92341144-23c4-4cf0-a0c8-26e1891397e9" Nov 03 00:40:25.208119 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:25.208078 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-jltt2" event={"ID":"cd47e545-4089-44ad-bc4d-0545061c7b11","Type":"ContainerStarted","Data":"25b0803e3e99d86834f3048cfaad40c0300e6c319a8d0b12c18a81b5a9b56784"} Nov 03 00:40:26.061414 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:26.061380 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6lvpt" Nov 03 00:40:26.061414 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:26.061395 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5v4k7" Nov 03 00:40:26.061654 ip-10-0-133-217 kubenswrapper[2575]: E1103 00:40:26.061516 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6lvpt" podUID="3dcbff01-09fb-458e-b3a5-594d6b271133" Nov 03 00:40:26.061963 ip-10-0-133-217 kubenswrapper[2575]: E1103 00:40:26.061934 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5v4k7" podUID="b6abcc85-61c0-42ea-b868-0d71a46a4514" Nov 03 00:40:26.217054 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:26.217004 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-trgdf" event={"ID":"6ab5d408-02a2-4284-bf59-5827c6f84643","Type":"ContainerStarted","Data":"841327c5579ea1f7fee2553328c8d2dbe22a9552f9a488a6bbd2511a60f6a4f1"} Nov 03 00:40:26.225989 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:26.225952 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-w8k5t" event={"ID":"5844f274-ad38-42a3-9900-b051b01c9953","Type":"ContainerStarted","Data":"2aaafe31f14758b282b09e3b38c041ab7715d9c67354c30c2f65bf8814e7d3b5"} Nov 03 00:40:26.241949 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:26.241897 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-jltt2" podStartSLOduration=4.253636865 podStartE2EDuration="8.24188058s" totalImagesPullingTime="3.988243715s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:40:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:40:25.225179965 +0000 UTC m=+7.670237429" watchObservedRunningTime="2026-11-03 00:40:26.24188058 +0000 UTC m=+8.686938045" Nov 03 00:40:26.779256 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:26.778837 2575 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-jltt2" Nov 03 00:40:26.779561 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:26.779432 2575 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-jltt2" Nov 03 00:40:26.797979 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:26.797903 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-w8k5t" podStartSLOduration=3.957393477 podStartE2EDuration="8.797884332s" totalImagesPullingTime="4.840490855s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 00:40:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:40:26.242458839 +0000 UTC m=+8.687516315" watchObservedRunningTime="2026-11-03 00:40:26.797884332 +0000 UTC m=+9.242941798" Nov 03 00:40:27.058394 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:27.058362 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-262mm" Nov 03 00:40:27.058574 ip-10-0-133-217 kubenswrapper[2575]: E1103 00:40:27.058514 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-262mm" podUID="92341144-23c4-4cf0-a0c8-26e1891397e9" Nov 03 00:40:27.675638 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:27.675600 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/92341144-23c4-4cf0-a0c8-26e1891397e9-original-pull-secret\") pod \"global-pull-secret-syncer-262mm\" (UID: \"92341144-23c4-4cf0-a0c8-26e1891397e9\") " pod="kube-system/global-pull-secret-syncer-262mm" Nov 03 00:40:27.676084 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:27.675665 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b6abcc85-61c0-42ea-b868-0d71a46a4514-metrics-certs\") pod \"network-metrics-daemon-5v4k7\" (UID: \"b6abcc85-61c0-42ea-b868-0d71a46a4514\") " pod="openshift-multus/network-metrics-daemon-5v4k7" Nov 03 00:40:27.676084 ip-10-0-133-217 kubenswrapper[2575]: E1103 00:40:27.675789 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 00:40:27.676084 ip-10-0-133-217 kubenswrapper[2575]: E1103 00:40:27.675846 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b6abcc85-61c0-42ea-b868-0d71a46a4514-metrics-certs podName:b6abcc85-61c0-42ea-b868-0d71a46a4514 nodeName:}" failed. No retries permitted until 2026-11-03 00:40:35.675827418 +0000 UTC m=+18.120884880 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b6abcc85-61c0-42ea-b868-0d71a46a4514-metrics-certs") pod "network-metrics-daemon-5v4k7" (UID: "b6abcc85-61c0-42ea-b868-0d71a46a4514") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 00:40:27.676290 ip-10-0-133-217 kubenswrapper[2575]: E1103 00:40:27.676235 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 00:40:27.676357 ip-10-0-133-217 kubenswrapper[2575]: E1103 00:40:27.676304 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92341144-23c4-4cf0-a0c8-26e1891397e9-original-pull-secret podName:92341144-23c4-4cf0-a0c8-26e1891397e9 nodeName:}" failed. No retries permitted until 2026-11-03 00:40:35.676287313 +0000 UTC m=+18.121344772 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/92341144-23c4-4cf0-a0c8-26e1891397e9-original-pull-secret") pod "global-pull-secret-syncer-262mm" (UID: "92341144-23c4-4cf0-a0c8-26e1891397e9") : object "kube-system"/"original-pull-secret" not registered Nov 03 00:40:27.776314 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:27.776257 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dt25z\" (UniqueName: \"kubernetes.io/projected/3dcbff01-09fb-458e-b3a5-594d6b271133-kube-api-access-dt25z\") pod \"network-check-target-6lvpt\" (UID: \"3dcbff01-09fb-458e-b3a5-594d6b271133\") " pod="openshift-network-diagnostics/network-check-target-6lvpt" Nov 03 00:40:27.776623 ip-10-0-133-217 kubenswrapper[2575]: E1103 00:40:27.776520 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 00:40:27.776623 ip-10-0-133-217 kubenswrapper[2575]: E1103 00:40:27.776547 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 00:40:27.776623 ip-10-0-133-217 kubenswrapper[2575]: E1103 00:40:27.776561 2575 projected.go:196] Error preparing data for projected volume kube-api-access-dt25z for pod openshift-network-diagnostics/network-check-target-6lvpt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 00:40:27.776623 ip-10-0-133-217 kubenswrapper[2575]: E1103 00:40:27.776624 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3dcbff01-09fb-458e-b3a5-594d6b271133-kube-api-access-dt25z podName:3dcbff01-09fb-458e-b3a5-594d6b271133 nodeName:}" failed. No retries permitted until 2026-11-03 00:40:35.776605719 +0000 UTC m=+18.221663184 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-dt25z" (UniqueName: "kubernetes.io/projected/3dcbff01-09fb-458e-b3a5-594d6b271133-kube-api-access-dt25z") pod "network-check-target-6lvpt" (UID: "3dcbff01-09fb-458e-b3a5-594d6b271133") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 00:40:27.806407 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:27.805511 2575 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 03 00:40:28.059261 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:28.059234 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6lvpt" Nov 03 00:40:28.059390 ip-10-0-133-217 kubenswrapper[2575]: E1103 00:40:28.059346 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6lvpt" podUID="3dcbff01-09fb-458e-b3a5-594d6b271133" Nov 03 00:40:28.060544 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:28.060514 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5v4k7" Nov 03 00:40:28.060660 ip-10-0-133-217 kubenswrapper[2575]: E1103 00:40:28.060637 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5v4k7" podUID="b6abcc85-61c0-42ea-b868-0d71a46a4514" Nov 03 00:40:28.066299 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:28.065488 2575 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T00:40:27.805538424Z","UUID":"cb01c4c6-4bc4-4c6b-8175-d648cb64688e","Handler":null,"Name":"","Endpoint":""} Nov 03 00:40:28.069071 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:28.069050 2575 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 03 00:40:28.069176 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:28.069082 2575 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 03 00:40:28.233150 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:28.233063 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-trgdf" event={"ID":"6ab5d408-02a2-4284-bf59-5827c6f84643","Type":"ContainerStarted","Data":"adae1bc8f1a384618b0b5f96d3129425238db9c295d191883502167ff451ac06"} Nov 03 00:40:28.235569 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:28.235544 2575 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 00:40:28.235716 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:28.235577 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-lb46r" event={"ID":"b2d13929-be23-4396-8445-c3eee12e9b9d","Type":"ContainerStarted","Data":"824fabd19239be622cafadd1d1bdef67f5a8a226d0fcb594c92a9249b125d568"} Nov 03 00:40:28.253255 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:28.253179 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-lb46r" podStartSLOduration=2.598129501 podStartE2EDuration="10.253158827s" totalImagesPullingTime="7.655029326s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 00:40:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:40:28.251992107 +0000 UTC m=+10.697049576" watchObservedRunningTime="2026-11-03 00:40:28.253158827 +0000 UTC m=+10.698216293" Nov 03 00:40:29.058919 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:29.058659 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-262mm" Nov 03 00:40:29.059364 ip-10-0-133-217 kubenswrapper[2575]: E1103 00:40:29.059026 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-262mm" podUID="92341144-23c4-4cf0-a0c8-26e1891397e9" Nov 03 00:40:29.240191 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:29.240100 2575 generic.go:356] "Generic (PLEG): container finished" podID="18afc5dd-bc60-4794-91ab-b4019be56a61" containerID="215e994a15e19bbe833dfbd044681089a2a6f8903d5b490a867da6eb03308072" exitCode=0 Nov 03 00:40:29.240383 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:29.240232 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8rxnt" event={"ID":"18afc5dd-bc60-4794-91ab-b4019be56a61","Type":"ContainerDied","Data":"215e994a15e19bbe833dfbd044681089a2a6f8903d5b490a867da6eb03308072"} Nov 03 00:40:29.242524 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:29.242497 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-cphks" event={"ID":"e7f1faa5-fc0e-4f46-9208-b1e58f50fcf9","Type":"ContainerStarted","Data":"de4133b5de0a77366c4e1deaa6e4f8a06590ba02fc263c8bcf044f487df65d8f"} Nov 03 00:40:29.251846 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:29.251813 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-trgdf" event={"ID":"6ab5d408-02a2-4284-bf59-5827c6f84643","Type":"ContainerStarted","Data":"beda41154fc238d677b6391e7c71faafb013f61dc67c1689f757dc2c69e7db3b"} Nov 03 00:40:29.286080 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:29.286015 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-trgdf" podStartSLOduration=3.114908373 podStartE2EDuration="11.285997712s" totalImagesPullingTime="8.171089339s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 00:40:18 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:40:29.285726051 +0000 UTC m=+11.730783517" watchObservedRunningTime="2026-11-03 00:40:29.285997712 +0000 UTC m=+11.731055178" Nov 03 00:40:30.058472 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:30.058436 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5v4k7" Nov 03 00:40:30.058618 ip-10-0-133-217 kubenswrapper[2575]: E1103 00:40:30.058578 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5v4k7" podUID="b6abcc85-61c0-42ea-b868-0d71a46a4514" Nov 03 00:40:30.059124 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:30.059002 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6lvpt" Nov 03 00:40:30.059124 ip-10-0-133-217 kubenswrapper[2575]: E1103 00:40:30.059087 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6lvpt" podUID="3dcbff01-09fb-458e-b3a5-594d6b271133" Nov 03 00:40:30.256997 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:30.256961 2575 generic.go:356] "Generic (PLEG): container finished" podID="18afc5dd-bc60-4794-91ab-b4019be56a61" containerID="a4878089ef5451a280992f04f3ea2b52c9653d62c3b1d176161d514c0ca26c37" exitCode=0 Nov 03 00:40:30.257202 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:30.257058 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8rxnt" event={"ID":"18afc5dd-bc60-4794-91ab-b4019be56a61","Type":"ContainerDied","Data":"a4878089ef5451a280992f04f3ea2b52c9653d62c3b1d176161d514c0ca26c37"} Nov 03 00:40:30.259540 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:30.259512 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-9w99c" event={"ID":"174d7fa2-7163-47a8-a047-8727cdb3cb9f","Type":"ContainerStarted","Data":"017c9b5a03f7cf8ce367f41fd04df127d9cf8c40aadfa5e13725ae7cbafa6f74"} Nov 03 00:40:30.276617 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:30.276542 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-cphks" podStartSLOduration=4.523576434 podStartE2EDuration="12.276522323s" totalImagesPullingTime="7.752945889s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 00:40:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:40:29.298443051 +0000 UTC m=+11.743500519" watchObservedRunningTime="2026-11-03 00:40:30.276522323 +0000 UTC m=+12.721579789" Nov 03 00:40:31.058827 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:31.058801 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-262mm" Nov 03 00:40:31.058953 ip-10-0-133-217 kubenswrapper[2575]: E1103 00:40:31.058916 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-262mm" podUID="92341144-23c4-4cf0-a0c8-26e1891397e9" Nov 03 00:40:31.264219 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:31.264179 2575 generic.go:356] "Generic (PLEG): container finished" podID="18afc5dd-bc60-4794-91ab-b4019be56a61" containerID="05834d884916df322d6a53853036fd2808cc38de59a9542ebb91dd53f5900496" exitCode=0 Nov 03 00:40:31.264924 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:31.264260 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8rxnt" event={"ID":"18afc5dd-bc60-4794-91ab-b4019be56a61","Type":"ContainerDied","Data":"05834d884916df322d6a53853036fd2808cc38de59a9542ebb91dd53f5900496"} Nov 03 00:40:31.268672 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:31.268642 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-n7ksd" event={"ID":"476ff8a0-f214-4e5e-aa90-8a505974f963","Type":"ContainerStarted","Data":"dcc337b75addc5ef3d82c5ead634300a7c44e58d371b44bdf49af649455fa74a"} Nov 03 00:40:31.285690 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:31.285590 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-9w99c" podStartSLOduration=5.1514963 podStartE2EDuration="13.28557777s" totalImagesPullingTime="8.13408147s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 00:40:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:40:30.289636571 +0000 UTC m=+12.734694036" watchObservedRunningTime="2026-11-03 00:40:31.28557777 +0000 UTC m=+13.730635235" Nov 03 00:40:32.059941 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:32.059908 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6lvpt" Nov 03 00:40:32.060108 ip-10-0-133-217 kubenswrapper[2575]: E1103 00:40:32.060017 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6lvpt" podUID="3dcbff01-09fb-458e-b3a5-594d6b271133" Nov 03 00:40:32.060478 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:32.060460 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5v4k7" Nov 03 00:40:32.060561 ip-10-0-133-217 kubenswrapper[2575]: E1103 00:40:32.060545 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5v4k7" podUID="b6abcc85-61c0-42ea-b868-0d71a46a4514" Nov 03 00:40:33.058207 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:33.058166 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-262mm" Nov 03 00:40:33.058775 ip-10-0-133-217 kubenswrapper[2575]: E1103 00:40:33.058329 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-262mm" podUID="92341144-23c4-4cf0-a0c8-26e1891397e9" Nov 03 00:40:34.058434 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:34.058196 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5v4k7" Nov 03 00:40:34.058914 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:34.058210 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6lvpt" Nov 03 00:40:34.058914 ip-10-0-133-217 kubenswrapper[2575]: E1103 00:40:34.058564 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5v4k7" podUID="b6abcc85-61c0-42ea-b868-0d71a46a4514" Nov 03 00:40:34.058914 ip-10-0-133-217 kubenswrapper[2575]: E1103 00:40:34.058625 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6lvpt" podUID="3dcbff01-09fb-458e-b3a5-594d6b271133" Nov 03 00:40:34.278534 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:34.278494 2575 generic.go:356] "Generic (PLEG): container finished" podID="18afc5dd-bc60-4794-91ab-b4019be56a61" containerID="62473d6087e26fc4e344ba92bc31c92f99655f2dad9487b90cdf0df87946436c" exitCode=0 Nov 03 00:40:34.278687 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:34.278563 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8rxnt" event={"ID":"18afc5dd-bc60-4794-91ab-b4019be56a61","Type":"ContainerDied","Data":"62473d6087e26fc4e344ba92bc31c92f99655f2dad9487b90cdf0df87946436c"} Nov 03 00:40:34.303944 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:34.303401 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-n7ksd" podStartSLOduration=6.031759383 podStartE2EDuration="16.303382024s" totalImagesPullingTime="10.271622641s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 00:40:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:40:31.301261514 +0000 UTC m=+13.746318992" watchObservedRunningTime="2026-11-03 00:40:34.303382024 +0000 UTC m=+16.748439490" Nov 03 00:40:35.059038 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:35.059005 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-262mm" Nov 03 00:40:35.059468 ip-10-0-133-217 kubenswrapper[2575]: E1103 00:40:35.059112 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-262mm" podUID="92341144-23c4-4cf0-a0c8-26e1891397e9" Nov 03 00:40:35.284269 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:35.284060 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zq57z" event={"ID":"29de0f7e-9c96-46dd-8fec-434e3689a047","Type":"ContainerStarted","Data":"b55bc3f5af7ab01e934144fa3db19943740fdff82b3333d21cbf11dcf3c1ee88"} Nov 03 00:40:35.284472 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:35.284358 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zq57z" event={"ID":"29de0f7e-9c96-46dd-8fec-434e3689a047","Type":"ContainerStarted","Data":"7baa993438831af753ccb334e03aab093146306e17276148aa2cfebf76461a9d"} Nov 03 00:40:35.284472 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:35.284416 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zq57z" event={"ID":"29de0f7e-9c96-46dd-8fec-434e3689a047","Type":"ContainerStarted","Data":"778526076e0a2b218c80a04b622d41f4d237e0eaa6296f333e85d595852654b4"} Nov 03 00:40:35.284472 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:35.284452 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zq57z" event={"ID":"29de0f7e-9c96-46dd-8fec-434e3689a047","Type":"ContainerStarted","Data":"99a8ec574a55d8831a3d4906b41c51f84e53463f5d8251c10a951d9d7679d354"} Nov 03 00:40:35.284609 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:35.284481 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zq57z" event={"ID":"29de0f7e-9c96-46dd-8fec-434e3689a047","Type":"ContainerStarted","Data":"52d53003f9844c8358768c1183603151ac0627619c002cb771ddc874f87c47fc"} Nov 03 00:40:35.284609 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:35.284521 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zq57z" event={"ID":"29de0f7e-9c96-46dd-8fec-434e3689a047","Type":"ContainerStarted","Data":"b37907637df96738ae09079bde6dffcf9310f0b4c720ac5f37b5c16952d6719f"} Nov 03 00:40:35.287397 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:35.287368 2575 generic.go:356] "Generic (PLEG): container finished" podID="18afc5dd-bc60-4794-91ab-b4019be56a61" containerID="1c783e52f0e82ca5675d8fae07f7a6289f582d17d5239a45061d389c89bd300b" exitCode=0 Nov 03 00:40:35.287524 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:35.287424 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8rxnt" event={"ID":"18afc5dd-bc60-4794-91ab-b4019be56a61","Type":"ContainerDied","Data":"1c783e52f0e82ca5675d8fae07f7a6289f582d17d5239a45061d389c89bd300b"} Nov 03 00:40:35.742742 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:35.742659 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b6abcc85-61c0-42ea-b868-0d71a46a4514-metrics-certs\") pod \"network-metrics-daemon-5v4k7\" (UID: \"b6abcc85-61c0-42ea-b868-0d71a46a4514\") " pod="openshift-multus/network-metrics-daemon-5v4k7" Nov 03 00:40:35.742742 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:35.742710 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/92341144-23c4-4cf0-a0c8-26e1891397e9-original-pull-secret\") pod \"global-pull-secret-syncer-262mm\" (UID: \"92341144-23c4-4cf0-a0c8-26e1891397e9\") " pod="kube-system/global-pull-secret-syncer-262mm" Nov 03 00:40:35.742975 ip-10-0-133-217 kubenswrapper[2575]: E1103 00:40:35.742796 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 00:40:35.742975 ip-10-0-133-217 kubenswrapper[2575]: E1103 00:40:35.742797 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 00:40:35.742975 ip-10-0-133-217 kubenswrapper[2575]: E1103 00:40:35.742842 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92341144-23c4-4cf0-a0c8-26e1891397e9-original-pull-secret podName:92341144-23c4-4cf0-a0c8-26e1891397e9 nodeName:}" failed. No retries permitted until 2026-11-03 00:40:51.742830435 +0000 UTC m=+34.187887879 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/92341144-23c4-4cf0-a0c8-26e1891397e9-original-pull-secret") pod "global-pull-secret-syncer-262mm" (UID: "92341144-23c4-4cf0-a0c8-26e1891397e9") : object "kube-system"/"original-pull-secret" not registered Nov 03 00:40:35.742975 ip-10-0-133-217 kubenswrapper[2575]: E1103 00:40:35.742855 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b6abcc85-61c0-42ea-b868-0d71a46a4514-metrics-certs podName:b6abcc85-61c0-42ea-b868-0d71a46a4514 nodeName:}" failed. No retries permitted until 2026-11-03 00:40:51.742849051 +0000 UTC m=+34.187906494 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b6abcc85-61c0-42ea-b868-0d71a46a4514-metrics-certs") pod "network-metrics-daemon-5v4k7" (UID: "b6abcc85-61c0-42ea-b868-0d71a46a4514") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 00:40:35.843499 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:35.843469 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dt25z\" (UniqueName: \"kubernetes.io/projected/3dcbff01-09fb-458e-b3a5-594d6b271133-kube-api-access-dt25z\") pod \"network-check-target-6lvpt\" (UID: \"3dcbff01-09fb-458e-b3a5-594d6b271133\") " pod="openshift-network-diagnostics/network-check-target-6lvpt" Nov 03 00:40:35.843665 ip-10-0-133-217 kubenswrapper[2575]: E1103 00:40:35.843586 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 00:40:35.843665 ip-10-0-133-217 kubenswrapper[2575]: E1103 00:40:35.843600 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 00:40:35.843665 ip-10-0-133-217 kubenswrapper[2575]: E1103 00:40:35.843608 2575 projected.go:196] Error preparing data for projected volume kube-api-access-dt25z for pod openshift-network-diagnostics/network-check-target-6lvpt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 00:40:35.843665 ip-10-0-133-217 kubenswrapper[2575]: E1103 00:40:35.843653 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3dcbff01-09fb-458e-b3a5-594d6b271133-kube-api-access-dt25z podName:3dcbff01-09fb-458e-b3a5-594d6b271133 nodeName:}" failed. No retries permitted until 2026-11-03 00:40:51.843640931 +0000 UTC m=+34.288698375 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-dt25z" (UniqueName: "kubernetes.io/projected/3dcbff01-09fb-458e-b3a5-594d6b271133-kube-api-access-dt25z") pod "network-check-target-6lvpt" (UID: "3dcbff01-09fb-458e-b3a5-594d6b271133") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 00:40:36.058252 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:36.058218 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6lvpt" Nov 03 00:40:36.058481 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:36.058229 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5v4k7" Nov 03 00:40:36.058481 ip-10-0-133-217 kubenswrapper[2575]: E1103 00:40:36.058340 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6lvpt" podUID="3dcbff01-09fb-458e-b3a5-594d6b271133" Nov 03 00:40:36.058481 ip-10-0-133-217 kubenswrapper[2575]: E1103 00:40:36.058400 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5v4k7" podUID="b6abcc85-61c0-42ea-b868-0d71a46a4514" Nov 03 00:40:36.292929 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:36.292890 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8rxnt" event={"ID":"18afc5dd-bc60-4794-91ab-b4019be56a61","Type":"ContainerStarted","Data":"fbeae70ec538d8bcfd23d18c23a9073951eb6db769eb745f42faa48af4a2e07f"} Nov 03 00:40:36.332652 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:36.332555 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-8rxnt" podStartSLOduration=8.064876177 podStartE2EDuration="18.332543885s" totalImagesPullingTime="10.267667708s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 00:40:18 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:40:36.331073697 +0000 UTC m=+18.776131162" watchObservedRunningTime="2026-11-03 00:40:36.332543885 +0000 UTC m=+18.777601347" Nov 03 00:40:37.058007 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:37.057988 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-262mm" Nov 03 00:40:37.058131 ip-10-0-133-217 kubenswrapper[2575]: E1103 00:40:37.058109 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-262mm" podUID="92341144-23c4-4cf0-a0c8-26e1891397e9" Nov 03 00:40:37.298055 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:37.297972 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zq57z" event={"ID":"29de0f7e-9c96-46dd-8fec-434e3689a047","Type":"ContainerStarted","Data":"b334a27a518a6a5fd88f479da67e1de1f9a3fe67b14b2d2a8c1c369e3a1125ae"} Nov 03 00:40:38.059046 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:38.059016 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5v4k7" Nov 03 00:40:38.059288 ip-10-0-133-217 kubenswrapper[2575]: E1103 00:40:38.059130 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5v4k7" podUID="b6abcc85-61c0-42ea-b868-0d71a46a4514" Nov 03 00:40:38.059288 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:38.059177 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6lvpt" Nov 03 00:40:38.059288 ip-10-0-133-217 kubenswrapper[2575]: E1103 00:40:38.059254 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6lvpt" podUID="3dcbff01-09fb-458e-b3a5-594d6b271133" Nov 03 00:40:39.059103 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:39.059043 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-262mm" Nov 03 00:40:39.059613 ip-10-0-133-217 kubenswrapper[2575]: E1103 00:40:39.059183 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-262mm" podUID="92341144-23c4-4cf0-a0c8-26e1891397e9" Nov 03 00:40:39.305920 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:39.305881 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zq57z" event={"ID":"29de0f7e-9c96-46dd-8fec-434e3689a047","Type":"ContainerStarted","Data":"70e860964ad64a722fbdc31d58549e426a36686e500d209d7e9e21165372de52"} Nov 03 00:40:39.306204 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:39.306187 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zq57z" Nov 03 00:40:39.321691 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:39.321633 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zq57z" Nov 03 00:40:39.339588 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:39.339527 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-zq57z" podStartSLOduration=7.016599673 podStartE2EDuration="21.33951522s" totalImagesPullingTime="14.322915547s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 00:40:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:40:39.33879876 +0000 UTC m=+21.783856225" watchObservedRunningTime="2026-11-03 00:40:39.33951522 +0000 UTC m=+21.784572684" Nov 03 00:40:40.058120 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:40.058082 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6lvpt" Nov 03 00:40:40.058360 ip-10-0-133-217 kubenswrapper[2575]: E1103 00:40:40.058327 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6lvpt" podUID="3dcbff01-09fb-458e-b3a5-594d6b271133" Nov 03 00:40:40.058417 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:40.058352 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5v4k7" Nov 03 00:40:40.058490 ip-10-0-133-217 kubenswrapper[2575]: E1103 00:40:40.058466 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5v4k7" podUID="b6abcc85-61c0-42ea-b868-0d71a46a4514" Nov 03 00:40:40.308872 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:40.308786 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zq57z" Nov 03 00:40:40.309445 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:40.308876 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zq57z" Nov 03 00:40:40.325088 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:40.325059 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zq57z" Nov 03 00:40:41.058529 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:41.058264 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-262mm" Nov 03 00:40:41.058685 ip-10-0-133-217 kubenswrapper[2575]: E1103 00:40:41.058557 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-262mm" podUID="92341144-23c4-4cf0-a0c8-26e1891397e9" Nov 03 00:40:41.066663 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:41.066292 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-6lvpt"] Nov 03 00:40:41.066663 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:41.066400 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6lvpt" Nov 03 00:40:41.066663 ip-10-0-133-217 kubenswrapper[2575]: E1103 00:40:41.066500 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6lvpt" podUID="3dcbff01-09fb-458e-b3a5-594d6b271133" Nov 03 00:40:41.067054 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:41.067033 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-262mm"] Nov 03 00:40:41.068359 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:41.068331 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-5v4k7"] Nov 03 00:40:41.068510 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:41.068496 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5v4k7" Nov 03 00:40:41.068666 ip-10-0-133-217 kubenswrapper[2575]: E1103 00:40:41.068644 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5v4k7" podUID="b6abcc85-61c0-42ea-b868-0d71a46a4514" Nov 03 00:40:41.311308 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:41.311220 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-262mm" Nov 03 00:40:41.311655 ip-10-0-133-217 kubenswrapper[2575]: E1103 00:40:41.311334 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-262mm" podUID="92341144-23c4-4cf0-a0c8-26e1891397e9" Nov 03 00:40:43.058253 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:43.058222 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6lvpt" Nov 03 00:40:43.058823 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:43.058267 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-262mm" Nov 03 00:40:43.058823 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:43.058325 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5v4k7" Nov 03 00:40:43.058823 ip-10-0-133-217 kubenswrapper[2575]: E1103 00:40:43.058348 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6lvpt" podUID="3dcbff01-09fb-458e-b3a5-594d6b271133" Nov 03 00:40:43.058823 ip-10-0-133-217 kubenswrapper[2575]: E1103 00:40:43.058440 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5v4k7" podUID="b6abcc85-61c0-42ea-b868-0d71a46a4514" Nov 03 00:40:43.058823 ip-10-0-133-217 kubenswrapper[2575]: E1103 00:40:43.058530 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-262mm" podUID="92341144-23c4-4cf0-a0c8-26e1891397e9" Nov 03 00:40:45.058700 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:45.058661 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5v4k7" Nov 03 00:40:45.059137 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:45.058661 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6lvpt" Nov 03 00:40:45.059137 ip-10-0-133-217 kubenswrapper[2575]: E1103 00:40:45.058786 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5v4k7" podUID="b6abcc85-61c0-42ea-b868-0d71a46a4514" Nov 03 00:40:45.059137 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:45.058661 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-262mm" Nov 03 00:40:45.059137 ip-10-0-133-217 kubenswrapper[2575]: E1103 00:40:45.058845 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6lvpt" podUID="3dcbff01-09fb-458e-b3a5-594d6b271133" Nov 03 00:40:45.059137 ip-10-0-133-217 kubenswrapper[2575]: E1103 00:40:45.058949 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-262mm" podUID="92341144-23c4-4cf0-a0c8-26e1891397e9" Nov 03 00:40:47.058733 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:47.058685 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5v4k7" Nov 03 00:40:47.059112 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:47.058691 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6lvpt" Nov 03 00:40:47.059112 ip-10-0-133-217 kubenswrapper[2575]: E1103 00:40:47.058822 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5v4k7" podUID="b6abcc85-61c0-42ea-b868-0d71a46a4514" Nov 03 00:40:47.059112 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:47.058706 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-262mm" Nov 03 00:40:47.059112 ip-10-0-133-217 kubenswrapper[2575]: E1103 00:40:47.058885 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6lvpt" podUID="3dcbff01-09fb-458e-b3a5-594d6b271133" Nov 03 00:40:47.059112 ip-10-0-133-217 kubenswrapper[2575]: E1103 00:40:47.058962 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-262mm" podUID="92341144-23c4-4cf0-a0c8-26e1891397e9" Nov 03 00:40:47.810255 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:47.810225 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-217.us-east-2.compute.internal" event="NodeReady" Nov 03 00:40:47.810434 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:47.810378 2575 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Nov 03 00:40:47.876296 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:47.876255 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-rx2xs"] Nov 03 00:40:47.882809 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:47.882772 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-rx2xs" Nov 03 00:40:47.885659 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:47.885628 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-zskmb"] Nov 03 00:40:47.890634 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:47.890607 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-zskmb" Nov 03 00:40:47.891706 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:47.891682 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 03 00:40:47.893002 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:47.892979 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 03 00:40:47.893101 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:47.893012 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-vz52v\"" Nov 03 00:40:47.901680 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:47.901653 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-rx2xs"] Nov 03 00:40:47.906954 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:47.906923 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-whbrp"] Nov 03 00:40:47.910136 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:47.910112 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-whbrp" Nov 03 00:40:47.910716 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:47.910684 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-zskmb"] Nov 03 00:40:47.929079 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:47.929053 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-whbrp"] Nov 03 00:40:47.990880 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:47.990841 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 03 00:40:47.990880 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:47.990841 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 03 00:40:47.991096 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:47.990900 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 03 00:40:47.991096 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:47.990942 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 03 00:40:47.991096 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:47.990936 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 03 00:40:47.991806 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:47.991792 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-42dxl\"" Nov 03 00:40:47.991892 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:47.991832 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-k7bjp\"" Nov 03 00:40:47.991892 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:47.991849 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 03 00:40:47.991892 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:47.991884 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 03 00:40:48.021745 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:48.021709 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86gz6\" (UniqueName: \"kubernetes.io/projected/bac578ef-a88b-4a74-9a65-bebc8ea1d00e-kube-api-access-86gz6\") pod \"insights-runtime-extractor-zskmb\" (UID: \"bac578ef-a88b-4a74-9a65-bebc8ea1d00e\") " pod="openshift-insights/insights-runtime-extractor-zskmb" Nov 03 00:40:48.021874 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:48.021772 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4svt5\" (UniqueName: \"kubernetes.io/projected/9c505674-e66b-4573-bbf5-7b6acf6684e6-kube-api-access-4svt5\") pod \"dns-default-rx2xs\" (UID: \"9c505674-e66b-4573-bbf5-7b6acf6684e6\") " pod="openshift-dns/dns-default-rx2xs" Nov 03 00:40:48.021874 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:48.021809 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bac578ef-a88b-4a74-9a65-bebc8ea1d00e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zskmb\" (UID: \"bac578ef-a88b-4a74-9a65-bebc8ea1d00e\") " pod="openshift-insights/insights-runtime-extractor-zskmb" Nov 03 00:40:48.021941 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:48.021926 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bac578ef-a88b-4a74-9a65-bebc8ea1d00e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zskmb\" (UID: \"bac578ef-a88b-4a74-9a65-bebc8ea1d00e\") " pod="openshift-insights/insights-runtime-extractor-zskmb" Nov 03 00:40:48.021970 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:48.021959 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/bac578ef-a88b-4a74-9a65-bebc8ea1d00e-crio-socket\") pod \"insights-runtime-extractor-zskmb\" (UID: \"bac578ef-a88b-4a74-9a65-bebc8ea1d00e\") " pod="openshift-insights/insights-runtime-extractor-zskmb" Nov 03 00:40:48.021998 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:48.021980 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/bac578ef-a88b-4a74-9a65-bebc8ea1d00e-data-volume\") pod \"insights-runtime-extractor-zskmb\" (UID: \"bac578ef-a88b-4a74-9a65-bebc8ea1d00e\") " pod="openshift-insights/insights-runtime-extractor-zskmb" Nov 03 00:40:48.022031 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:48.021995 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9c505674-e66b-4573-bbf5-7b6acf6684e6-metrics-tls\") pod \"dns-default-rx2xs\" (UID: \"9c505674-e66b-4573-bbf5-7b6acf6684e6\") " pod="openshift-dns/dns-default-rx2xs" Nov 03 00:40:48.022031 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:48.022017 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c505674-e66b-4573-bbf5-7b6acf6684e6-tmp-dir\") pod \"dns-default-rx2xs\" (UID: \"9c505674-e66b-4573-bbf5-7b6acf6684e6\") " pod="openshift-dns/dns-default-rx2xs" Nov 03 00:40:48.022131 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:48.022119 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/76ccef27-d056-402e-a7e1-ee524d63bf35-cert\") pod \"ingress-canary-whbrp\" (UID: \"76ccef27-d056-402e-a7e1-ee524d63bf35\") " pod="openshift-ingress-canary/ingress-canary-whbrp" Nov 03 00:40:48.022163 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:48.022138 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjg6m\" (UniqueName: \"kubernetes.io/projected/76ccef27-d056-402e-a7e1-ee524d63bf35-kube-api-access-rjg6m\") pod \"ingress-canary-whbrp\" (UID: \"76ccef27-d056-402e-a7e1-ee524d63bf35\") " pod="openshift-ingress-canary/ingress-canary-whbrp" Nov 03 00:40:48.022163 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:48.022157 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9c505674-e66b-4573-bbf5-7b6acf6684e6-config-volume\") pod \"dns-default-rx2xs\" (UID: \"9c505674-e66b-4573-bbf5-7b6acf6684e6\") " pod="openshift-dns/dns-default-rx2xs" Nov 03 00:40:48.122510 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:48.122421 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86gz6\" (UniqueName: \"kubernetes.io/projected/bac578ef-a88b-4a74-9a65-bebc8ea1d00e-kube-api-access-86gz6\") pod \"insights-runtime-extractor-zskmb\" (UID: \"bac578ef-a88b-4a74-9a65-bebc8ea1d00e\") " pod="openshift-insights/insights-runtime-extractor-zskmb" Nov 03 00:40:48.122510 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:48.122487 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4svt5\" (UniqueName: \"kubernetes.io/projected/9c505674-e66b-4573-bbf5-7b6acf6684e6-kube-api-access-4svt5\") pod \"dns-default-rx2xs\" (UID: \"9c505674-e66b-4573-bbf5-7b6acf6684e6\") " pod="openshift-dns/dns-default-rx2xs" Nov 03 00:40:48.122971 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:48.122604 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bac578ef-a88b-4a74-9a65-bebc8ea1d00e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zskmb\" (UID: \"bac578ef-a88b-4a74-9a65-bebc8ea1d00e\") " pod="openshift-insights/insights-runtime-extractor-zskmb" Nov 03 00:40:48.122971 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:48.122638 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bac578ef-a88b-4a74-9a65-bebc8ea1d00e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zskmb\" (UID: \"bac578ef-a88b-4a74-9a65-bebc8ea1d00e\") " pod="openshift-insights/insights-runtime-extractor-zskmb" Nov 03 00:40:48.122971 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:48.122668 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/bac578ef-a88b-4a74-9a65-bebc8ea1d00e-crio-socket\") pod \"insights-runtime-extractor-zskmb\" (UID: \"bac578ef-a88b-4a74-9a65-bebc8ea1d00e\") " pod="openshift-insights/insights-runtime-extractor-zskmb" Nov 03 00:40:48.122971 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:48.122695 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/bac578ef-a88b-4a74-9a65-bebc8ea1d00e-data-volume\") pod \"insights-runtime-extractor-zskmb\" (UID: \"bac578ef-a88b-4a74-9a65-bebc8ea1d00e\") " pod="openshift-insights/insights-runtime-extractor-zskmb" Nov 03 00:40:48.122971 ip-10-0-133-217 kubenswrapper[2575]: E1103 00:40:48.122759 2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 03 00:40:48.122971 ip-10-0-133-217 kubenswrapper[2575]: E1103 00:40:48.122830 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bac578ef-a88b-4a74-9a65-bebc8ea1d00e-kube-rbac-proxy-cm podName:bac578ef-a88b-4a74-9a65-bebc8ea1d00e nodeName:}" failed. No retries permitted until 2026-11-03 00:40:48.622810817 +0000 UTC m=+31.067868280 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/bac578ef-a88b-4a74-9a65-bebc8ea1d00e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zskmb" (UID: "bac578ef-a88b-4a74-9a65-bebc8ea1d00e") : configmap "kube-rbac-proxy" not found Nov 03 00:40:48.122971 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:48.122876 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9c505674-e66b-4573-bbf5-7b6acf6684e6-metrics-tls\") pod \"dns-default-rx2xs\" (UID: \"9c505674-e66b-4573-bbf5-7b6acf6684e6\") " pod="openshift-dns/dns-default-rx2xs" Nov 03 00:40:48.122971 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:48.122901 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c505674-e66b-4573-bbf5-7b6acf6684e6-tmp-dir\") pod \"dns-default-rx2xs\" (UID: \"9c505674-e66b-4573-bbf5-7b6acf6684e6\") " pod="openshift-dns/dns-default-rx2xs" Nov 03 00:40:48.122971 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:48.122897 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/bac578ef-a88b-4a74-9a65-bebc8ea1d00e-crio-socket\") pod \"insights-runtime-extractor-zskmb\" (UID: \"bac578ef-a88b-4a74-9a65-bebc8ea1d00e\") " pod="openshift-insights/insights-runtime-extractor-zskmb" Nov 03 00:40:48.122971 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:48.122958 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/76ccef27-d056-402e-a7e1-ee524d63bf35-cert\") pod \"ingress-canary-whbrp\" (UID: \"76ccef27-d056-402e-a7e1-ee524d63bf35\") " pod="openshift-ingress-canary/ingress-canary-whbrp" Nov 03 00:40:48.123441 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:48.122991 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjg6m\" (UniqueName: \"kubernetes.io/projected/76ccef27-d056-402e-a7e1-ee524d63bf35-kube-api-access-rjg6m\") pod \"ingress-canary-whbrp\" (UID: \"76ccef27-d056-402e-a7e1-ee524d63bf35\") " pod="openshift-ingress-canary/ingress-canary-whbrp" Nov 03 00:40:48.123441 ip-10-0-133-217 kubenswrapper[2575]: E1103 00:40:48.123018 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 00:40:48.123441 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:48.123027 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9c505674-e66b-4573-bbf5-7b6acf6684e6-config-volume\") pod \"dns-default-rx2xs\" (UID: \"9c505674-e66b-4573-bbf5-7b6acf6684e6\") " pod="openshift-dns/dns-default-rx2xs" Nov 03 00:40:48.123441 ip-10-0-133-217 kubenswrapper[2575]: E1103 00:40:48.123076 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bac578ef-a88b-4a74-9a65-bebc8ea1d00e-insights-runtime-extractor-tls podName:bac578ef-a88b-4a74-9a65-bebc8ea1d00e nodeName:}" failed. No retries permitted until 2026-11-03 00:40:48.623057916 +0000 UTC m=+31.068115370 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/bac578ef-a88b-4a74-9a65-bebc8ea1d00e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zskmb" (UID: "bac578ef-a88b-4a74-9a65-bebc8ea1d00e") : secret "insights-runtime-extractor-tls" not found Nov 03 00:40:48.123441 ip-10-0-133-217 kubenswrapper[2575]: E1103 00:40:48.123115 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 00:40:48.123441 ip-10-0-133-217 kubenswrapper[2575]: E1103 00:40:48.123174 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c505674-e66b-4573-bbf5-7b6acf6684e6-metrics-tls podName:9c505674-e66b-4573-bbf5-7b6acf6684e6 nodeName:}" failed. No retries permitted until 2026-11-03 00:40:48.623160217 +0000 UTC m=+31.068217672 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9c505674-e66b-4573-bbf5-7b6acf6684e6-metrics-tls") pod "dns-default-rx2xs" (UID: "9c505674-e66b-4573-bbf5-7b6acf6684e6") : secret "dns-default-metrics-tls" not found Nov 03 00:40:48.123441 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:48.123209 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c505674-e66b-4573-bbf5-7b6acf6684e6-tmp-dir\") pod \"dns-default-rx2xs\" (UID: \"9c505674-e66b-4573-bbf5-7b6acf6684e6\") " pod="openshift-dns/dns-default-rx2xs" Nov 03 00:40:48.123441 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:48.123314 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/bac578ef-a88b-4a74-9a65-bebc8ea1d00e-data-volume\") pod \"insights-runtime-extractor-zskmb\" (UID: \"bac578ef-a88b-4a74-9a65-bebc8ea1d00e\") " pod="openshift-insights/insights-runtime-extractor-zskmb" Nov 03 00:40:48.123734 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:48.123702 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9c505674-e66b-4573-bbf5-7b6acf6684e6-config-volume\") pod \"dns-default-rx2xs\" (UID: \"9c505674-e66b-4573-bbf5-7b6acf6684e6\") " pod="openshift-dns/dns-default-rx2xs" Nov 03 00:40:48.126671 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:48.126654 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/76ccef27-d056-402e-a7e1-ee524d63bf35-cert\") pod \"ingress-canary-whbrp\" (UID: \"76ccef27-d056-402e-a7e1-ee524d63bf35\") " pod="openshift-ingress-canary/ingress-canary-whbrp" Nov 03 00:40:48.144285 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:48.144253 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86gz6\" (UniqueName: \"kubernetes.io/projected/bac578ef-a88b-4a74-9a65-bebc8ea1d00e-kube-api-access-86gz6\") pod \"insights-runtime-extractor-zskmb\" (UID: \"bac578ef-a88b-4a74-9a65-bebc8ea1d00e\") " pod="openshift-insights/insights-runtime-extractor-zskmb" Nov 03 00:40:48.145596 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:48.145569 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4svt5\" (UniqueName: \"kubernetes.io/projected/9c505674-e66b-4573-bbf5-7b6acf6684e6-kube-api-access-4svt5\") pod \"dns-default-rx2xs\" (UID: \"9c505674-e66b-4573-bbf5-7b6acf6684e6\") " pod="openshift-dns/dns-default-rx2xs" Nov 03 00:40:48.146147 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:48.146130 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjg6m\" (UniqueName: \"kubernetes.io/projected/76ccef27-d056-402e-a7e1-ee524d63bf35-kube-api-access-rjg6m\") pod \"ingress-canary-whbrp\" (UID: \"76ccef27-d056-402e-a7e1-ee524d63bf35\") " pod="openshift-ingress-canary/ingress-canary-whbrp" Nov 03 00:40:48.222099 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:48.222060 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-whbrp" Nov 03 00:40:48.583665 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:48.583619 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-whbrp"] Nov 03 00:40:48.587415 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:48.587385 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76ccef27_d056_402e_a7e1_ee524d63bf35.slice/crio-6cda35d070db858a1d3a28f63c2c222c2f5ab534fabe1c579c5ec311ad94f1c5 WatchSource:0}: Error finding container 6cda35d070db858a1d3a28f63c2c222c2f5ab534fabe1c579c5ec311ad94f1c5: Status 404 returned error can't find the container with id 6cda35d070db858a1d3a28f63c2c222c2f5ab534fabe1c579c5ec311ad94f1c5 Nov 03 00:40:48.626260 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:48.626223 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bac578ef-a88b-4a74-9a65-bebc8ea1d00e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zskmb\" (UID: \"bac578ef-a88b-4a74-9a65-bebc8ea1d00e\") " pod="openshift-insights/insights-runtime-extractor-zskmb" Nov 03 00:40:48.626260 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:48.626264 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bac578ef-a88b-4a74-9a65-bebc8ea1d00e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zskmb\" (UID: \"bac578ef-a88b-4a74-9a65-bebc8ea1d00e\") " pod="openshift-insights/insights-runtime-extractor-zskmb" Nov 03 00:40:48.626488 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:48.626315 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9c505674-e66b-4573-bbf5-7b6acf6684e6-metrics-tls\") pod \"dns-default-rx2xs\" (UID: \"9c505674-e66b-4573-bbf5-7b6acf6684e6\") " pod="openshift-dns/dns-default-rx2xs" Nov 03 00:40:48.626488 ip-10-0-133-217 kubenswrapper[2575]: E1103 00:40:48.626348 2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 03 00:40:48.626488 ip-10-0-133-217 kubenswrapper[2575]: E1103 00:40:48.626412 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bac578ef-a88b-4a74-9a65-bebc8ea1d00e-kube-rbac-proxy-cm podName:bac578ef-a88b-4a74-9a65-bebc8ea1d00e nodeName:}" failed. No retries permitted until 2026-11-03 00:40:49.62639514 +0000 UTC m=+32.071452606 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/bac578ef-a88b-4a74-9a65-bebc8ea1d00e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zskmb" (UID: "bac578ef-a88b-4a74-9a65-bebc8ea1d00e") : configmap "kube-rbac-proxy" not found Nov 03 00:40:48.626488 ip-10-0-133-217 kubenswrapper[2575]: E1103 00:40:48.626444 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 00:40:48.626488 ip-10-0-133-217 kubenswrapper[2575]: E1103 00:40:48.626486 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 00:40:48.626636 ip-10-0-133-217 kubenswrapper[2575]: E1103 00:40:48.626501 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bac578ef-a88b-4a74-9a65-bebc8ea1d00e-insights-runtime-extractor-tls podName:bac578ef-a88b-4a74-9a65-bebc8ea1d00e nodeName:}" failed. No retries permitted until 2026-11-03 00:40:49.626489281 +0000 UTC m=+32.071546723 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/bac578ef-a88b-4a74-9a65-bebc8ea1d00e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zskmb" (UID: "bac578ef-a88b-4a74-9a65-bebc8ea1d00e") : secret "insights-runtime-extractor-tls" not found Nov 03 00:40:48.626636 ip-10-0-133-217 kubenswrapper[2575]: E1103 00:40:48.626523 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c505674-e66b-4573-bbf5-7b6acf6684e6-metrics-tls podName:9c505674-e66b-4573-bbf5-7b6acf6684e6 nodeName:}" failed. No retries permitted until 2026-11-03 00:40:49.626511809 +0000 UTC m=+32.071569277 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9c505674-e66b-4573-bbf5-7b6acf6684e6-metrics-tls") pod "dns-default-rx2xs" (UID: "9c505674-e66b-4573-bbf5-7b6acf6684e6") : secret "dns-default-metrics-tls" not found Nov 03 00:40:49.058521 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:49.058480 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5v4k7" Nov 03 00:40:49.058683 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:49.058533 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6lvpt" Nov 03 00:40:49.058683 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:49.058624 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-262mm" Nov 03 00:40:49.064500 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:49.064478 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-28tgg\"" Nov 03 00:40:49.064646 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:49.064562 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 03 00:40:49.064827 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:49.064813 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 03 00:40:49.065650 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:49.065612 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-7trdq\"" Nov 03 00:40:49.065723 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:49.065706 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 03 00:40:49.065759 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:49.065718 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 03 00:40:49.065759 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:49.065727 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-vmfsq\"" Nov 03 00:40:49.330430 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:49.330353 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-whbrp" event={"ID":"76ccef27-d056-402e-a7e1-ee524d63bf35","Type":"ContainerStarted","Data":"6cda35d070db858a1d3a28f63c2c222c2f5ab534fabe1c579c5ec311ad94f1c5"} Nov 03 00:40:49.540835 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:49.540791 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-jltt2" Nov 03 00:40:49.541021 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:49.540940 2575 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 00:40:49.541461 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:49.541437 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-jltt2" Nov 03 00:40:49.634394 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:49.634282 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bac578ef-a88b-4a74-9a65-bebc8ea1d00e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zskmb\" (UID: \"bac578ef-a88b-4a74-9a65-bebc8ea1d00e\") " pod="openshift-insights/insights-runtime-extractor-zskmb" Nov 03 00:40:49.634394 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:49.634329 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bac578ef-a88b-4a74-9a65-bebc8ea1d00e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zskmb\" (UID: \"bac578ef-a88b-4a74-9a65-bebc8ea1d00e\") " pod="openshift-insights/insights-runtime-extractor-zskmb" Nov 03 00:40:49.634394 ip-10-0-133-217 kubenswrapper[2575]: E1103 00:40:49.634376 2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 03 00:40:49.634686 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:49.634388 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9c505674-e66b-4573-bbf5-7b6acf6684e6-metrics-tls\") pod \"dns-default-rx2xs\" (UID: \"9c505674-e66b-4573-bbf5-7b6acf6684e6\") " pod="openshift-dns/dns-default-rx2xs" Nov 03 00:40:49.634686 ip-10-0-133-217 kubenswrapper[2575]: E1103 00:40:49.634460 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bac578ef-a88b-4a74-9a65-bebc8ea1d00e-kube-rbac-proxy-cm podName:bac578ef-a88b-4a74-9a65-bebc8ea1d00e nodeName:}" failed. No retries permitted until 2026-11-03 00:40:51.634439188 +0000 UTC m=+34.079496654 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/bac578ef-a88b-4a74-9a65-bebc8ea1d00e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zskmb" (UID: "bac578ef-a88b-4a74-9a65-bebc8ea1d00e") : configmap "kube-rbac-proxy" not found Nov 03 00:40:49.634686 ip-10-0-133-217 kubenswrapper[2575]: E1103 00:40:49.634621 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 00:40:49.634686 ip-10-0-133-217 kubenswrapper[2575]: E1103 00:40:49.634672 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c505674-e66b-4573-bbf5-7b6acf6684e6-metrics-tls podName:9c505674-e66b-4573-bbf5-7b6acf6684e6 nodeName:}" failed. No retries permitted until 2026-11-03 00:40:51.63465741 +0000 UTC m=+34.079714866 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9c505674-e66b-4573-bbf5-7b6acf6684e6-metrics-tls") pod "dns-default-rx2xs" (UID: "9c505674-e66b-4573-bbf5-7b6acf6684e6") : secret "dns-default-metrics-tls" not found Nov 03 00:40:49.634875 ip-10-0-133-217 kubenswrapper[2575]: E1103 00:40:49.634728 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 00:40:49.634875 ip-10-0-133-217 kubenswrapper[2575]: E1103 00:40:49.634761 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bac578ef-a88b-4a74-9a65-bebc8ea1d00e-insights-runtime-extractor-tls podName:bac578ef-a88b-4a74-9a65-bebc8ea1d00e nodeName:}" failed. No retries permitted until 2026-11-03 00:40:51.634748968 +0000 UTC m=+34.079806410 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/bac578ef-a88b-4a74-9a65-bebc8ea1d00e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zskmb" (UID: "bac578ef-a88b-4a74-9a65-bebc8ea1d00e") : secret "insights-runtime-extractor-tls" not found Nov 03 00:40:51.652081 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:51.652047 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bac578ef-a88b-4a74-9a65-bebc8ea1d00e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zskmb\" (UID: \"bac578ef-a88b-4a74-9a65-bebc8ea1d00e\") " pod="openshift-insights/insights-runtime-extractor-zskmb" Nov 03 00:40:51.652524 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:51.652088 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bac578ef-a88b-4a74-9a65-bebc8ea1d00e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zskmb\" (UID: \"bac578ef-a88b-4a74-9a65-bebc8ea1d00e\") " pod="openshift-insights/insights-runtime-extractor-zskmb" Nov 03 00:40:51.652524 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:51.652114 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9c505674-e66b-4573-bbf5-7b6acf6684e6-metrics-tls\") pod \"dns-default-rx2xs\" (UID: \"9c505674-e66b-4573-bbf5-7b6acf6684e6\") " pod="openshift-dns/dns-default-rx2xs" Nov 03 00:40:51.652524 ip-10-0-133-217 kubenswrapper[2575]: E1103 00:40:51.652194 2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 03 00:40:51.652524 ip-10-0-133-217 kubenswrapper[2575]: E1103 00:40:51.652239 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 00:40:51.652524 ip-10-0-133-217 kubenswrapper[2575]: E1103 00:40:51.652239 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 00:40:51.652524 ip-10-0-133-217 kubenswrapper[2575]: E1103 00:40:51.652309 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bac578ef-a88b-4a74-9a65-bebc8ea1d00e-kube-rbac-proxy-cm podName:bac578ef-a88b-4a74-9a65-bebc8ea1d00e nodeName:}" failed. No retries permitted until 2026-11-03 00:40:55.652266769 +0000 UTC m=+38.097324226 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/bac578ef-a88b-4a74-9a65-bebc8ea1d00e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zskmb" (UID: "bac578ef-a88b-4a74-9a65-bebc8ea1d00e") : configmap "kube-rbac-proxy" not found Nov 03 00:40:51.652524 ip-10-0-133-217 kubenswrapper[2575]: E1103 00:40:51.652331 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c505674-e66b-4573-bbf5-7b6acf6684e6-metrics-tls podName:9c505674-e66b-4573-bbf5-7b6acf6684e6 nodeName:}" failed. No retries permitted until 2026-11-03 00:40:55.652320796 +0000 UTC m=+38.097378242 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9c505674-e66b-4573-bbf5-7b6acf6684e6-metrics-tls") pod "dns-default-rx2xs" (UID: "9c505674-e66b-4573-bbf5-7b6acf6684e6") : secret "dns-default-metrics-tls" not found Nov 03 00:40:51.652524 ip-10-0-133-217 kubenswrapper[2575]: E1103 00:40:51.652347 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bac578ef-a88b-4a74-9a65-bebc8ea1d00e-insights-runtime-extractor-tls podName:bac578ef-a88b-4a74-9a65-bebc8ea1d00e nodeName:}" failed. No retries permitted until 2026-11-03 00:40:55.652337619 +0000 UTC m=+38.097395064 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/bac578ef-a88b-4a74-9a65-bebc8ea1d00e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zskmb" (UID: "bac578ef-a88b-4a74-9a65-bebc8ea1d00e") : secret "insights-runtime-extractor-tls" not found Nov 03 00:40:51.752577 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:51.752541 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b6abcc85-61c0-42ea-b868-0d71a46a4514-metrics-certs\") pod \"network-metrics-daemon-5v4k7\" (UID: \"b6abcc85-61c0-42ea-b868-0d71a46a4514\") " pod="openshift-multus/network-metrics-daemon-5v4k7" Nov 03 00:40:51.752739 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:51.752641 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/92341144-23c4-4cf0-a0c8-26e1891397e9-original-pull-secret\") pod \"global-pull-secret-syncer-262mm\" (UID: \"92341144-23c4-4cf0-a0c8-26e1891397e9\") " pod="kube-system/global-pull-secret-syncer-262mm" Nov 03 00:40:51.752739 ip-10-0-133-217 kubenswrapper[2575]: E1103 00:40:51.752719 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 03 00:40:51.752815 ip-10-0-133-217 kubenswrapper[2575]: E1103 00:40:51.752803 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b6abcc85-61c0-42ea-b868-0d71a46a4514-metrics-certs podName:b6abcc85-61c0-42ea-b868-0d71a46a4514 nodeName:}" failed. No retries permitted until 2026-11-03 00:41:23.752782131 +0000 UTC m=+66.197839589 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b6abcc85-61c0-42ea-b868-0d71a46a4514-metrics-certs") pod "network-metrics-daemon-5v4k7" (UID: "b6abcc85-61c0-42ea-b868-0d71a46a4514") : secret "metrics-daemon-secret" not found Nov 03 00:40:51.754955 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:51.754930 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/92341144-23c4-4cf0-a0c8-26e1891397e9-original-pull-secret\") pod \"global-pull-secret-syncer-262mm\" (UID: \"92341144-23c4-4cf0-a0c8-26e1891397e9\") " pod="kube-system/global-pull-secret-syncer-262mm" Nov 03 00:40:51.853245 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:51.853142 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dt25z\" (UniqueName: \"kubernetes.io/projected/3dcbff01-09fb-458e-b3a5-594d6b271133-kube-api-access-dt25z\") pod \"network-check-target-6lvpt\" (UID: \"3dcbff01-09fb-458e-b3a5-594d6b271133\") " pod="openshift-network-diagnostics/network-check-target-6lvpt" Nov 03 00:40:51.855736 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:51.855712 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dt25z\" (UniqueName: \"kubernetes.io/projected/3dcbff01-09fb-458e-b3a5-594d6b271133-kube-api-access-dt25z\") pod \"network-check-target-6lvpt\" (UID: \"3dcbff01-09fb-458e-b3a5-594d6b271133\") " pod="openshift-network-diagnostics/network-check-target-6lvpt" Nov 03 00:40:51.857440 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:51.857422 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-262mm" Nov 03 00:40:52.141068 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:52.140991 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6lvpt" Nov 03 00:40:52.152458 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:52.152427 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-262mm"] Nov 03 00:40:52.155650 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:52.155621 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92341144_23c4_4cf0_a0c8_26e1891397e9.slice/crio-b70feb1275eaf5ff25046474ea2e0d8794dfc882f99b257fb699089bc3010cab WatchSource:0}: Error finding container b70feb1275eaf5ff25046474ea2e0d8794dfc882f99b257fb699089bc3010cab: Status 404 returned error can't find the container with id b70feb1275eaf5ff25046474ea2e0d8794dfc882f99b257fb699089bc3010cab Nov 03 00:40:52.337889 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:52.337843 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-whbrp" event={"ID":"76ccef27-d056-402e-a7e1-ee524d63bf35","Type":"ContainerStarted","Data":"90f87cf98fea14337866c35c02071fadcb064e52ffbb656fa18a707bd049cf67"} Nov 03 00:40:52.338997 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:52.338972 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-262mm" event={"ID":"92341144-23c4-4cf0-a0c8-26e1891397e9","Type":"ContainerStarted","Data":"b70feb1275eaf5ff25046474ea2e0d8794dfc882f99b257fb699089bc3010cab"} Nov 03 00:40:52.357844 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:52.357798 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-whbrp" podStartSLOduration=2.4282107010000002 podStartE2EDuration="5.357784283s" totalImagesPullingTime="2.929573582s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:40:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:40:52.356991693 +0000 UTC m=+34.802049158" watchObservedRunningTime="2026-11-03 00:40:52.357784283 +0000 UTC m=+34.802841748" Nov 03 00:40:52.421878 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:52.421806 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-6lvpt"] Nov 03 00:40:52.425490 ip-10-0-133-217 kubenswrapper[2575]: W1103 00:40:52.425466 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3dcbff01_09fb_458e_b3a5_594d6b271133.slice/crio-b0df856bc6a0c41be88663d8b0a01bf05381d52cb6db6b6e164570d4e5f5e555 WatchSource:0}: Error finding container b0df856bc6a0c41be88663d8b0a01bf05381d52cb6db6b6e164570d4e5f5e555: Status 404 returned error can't find the container with id b0df856bc6a0c41be88663d8b0a01bf05381d52cb6db6b6e164570d4e5f5e555 Nov 03 00:40:53.342699 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:53.342659 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-6lvpt" event={"ID":"3dcbff01-09fb-458e-b3a5-594d6b271133","Type":"ContainerStarted","Data":"b0df856bc6a0c41be88663d8b0a01bf05381d52cb6db6b6e164570d4e5f5e555"} Nov 03 00:40:53.583350 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:53.583320 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-cphks_e7f1faa5-fc0e-4f46-9208-b1e58f50fcf9/dns-node-resolver/0.log" Nov 03 00:40:54.783509 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:54.783484 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-w8k5t_5844f274-ad38-42a3-9900-b051b01c9953/node-ca/0.log" Nov 03 00:40:55.584794 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:55.584747 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-whbrp_76ccef27-d056-402e-a7e1-ee524d63bf35/serve-healthcheck-canary/0.log" Nov 03 00:40:55.685077 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:55.685035 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bac578ef-a88b-4a74-9a65-bebc8ea1d00e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zskmb\" (UID: \"bac578ef-a88b-4a74-9a65-bebc8ea1d00e\") " pod="openshift-insights/insights-runtime-extractor-zskmb" Nov 03 00:40:55.685263 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:55.685087 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bac578ef-a88b-4a74-9a65-bebc8ea1d00e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zskmb\" (UID: \"bac578ef-a88b-4a74-9a65-bebc8ea1d00e\") " pod="openshift-insights/insights-runtime-extractor-zskmb" Nov 03 00:40:55.685263 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:55.685124 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9c505674-e66b-4573-bbf5-7b6acf6684e6-metrics-tls\") pod \"dns-default-rx2xs\" (UID: \"9c505674-e66b-4573-bbf5-7b6acf6684e6\") " pod="openshift-dns/dns-default-rx2xs" Nov 03 00:40:55.685416 ip-10-0-133-217 kubenswrapper[2575]: E1103 00:40:55.685313 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 00:40:55.685416 ip-10-0-133-217 kubenswrapper[2575]: E1103 00:40:55.685395 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bac578ef-a88b-4a74-9a65-bebc8ea1d00e-insights-runtime-extractor-tls podName:bac578ef-a88b-4a74-9a65-bebc8ea1d00e nodeName:}" failed. No retries permitted until 2026-11-03 00:41:03.685373124 +0000 UTC m=+46.130430588 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/bac578ef-a88b-4a74-9a65-bebc8ea1d00e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zskmb" (UID: "bac578ef-a88b-4a74-9a65-bebc8ea1d00e") : secret "insights-runtime-extractor-tls" not found Nov 03 00:40:55.685720 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:55.685695 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bac578ef-a88b-4a74-9a65-bebc8ea1d00e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zskmb\" (UID: \"bac578ef-a88b-4a74-9a65-bebc8ea1d00e\") " pod="openshift-insights/insights-runtime-extractor-zskmb" Nov 03 00:40:55.689916 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:55.689889 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9c505674-e66b-4573-bbf5-7b6acf6684e6-metrics-tls\") pod \"dns-default-rx2xs\" (UID: \"9c505674-e66b-4573-bbf5-7b6acf6684e6\") " pod="openshift-dns/dns-default-rx2xs" Nov 03 00:40:55.696465 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:55.696445 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-rx2xs" Nov 03 00:40:56.014301 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:56.013659 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-rx2xs"] Nov 03 00:40:56.352825 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:56.352740 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rx2xs" event={"ID":"9c505674-e66b-4573-bbf5-7b6acf6684e6","Type":"ContainerStarted","Data":"70058e8eadc84f83ec6ce1887b5a390c133578193406ac6dfc14abc706dfb8d9"} Nov 03 00:40:57.357008 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:57.356972 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-6lvpt" event={"ID":"3dcbff01-09fb-458e-b3a5-594d6b271133","Type":"ContainerStarted","Data":"820466609ced0b9066abc277f26d419c07bfc78d6d1a2dee03bc281c067dc326"} Nov 03 00:40:57.357805 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:57.357121 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-6lvpt" Nov 03 00:40:57.376607 ip-10-0-133-217 kubenswrapper[2575]: I1103 00:40:57.376542 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-6lvpt" podStartSLOduration=35.294350206 podStartE2EDuration="39.376522121s" totalImagesPullingTime="4.082171915s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:40:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:40:57.375335206 +0000 UTC m=+39.820392671" watchObservedRunningTime="2026-11-03 00:40:57.376522121 +0000 UTC m=+39.821579587" Nov 03 00:40:57.773310 ip-10-0-133-217 systemd[1]: Stopping Kubernetes Kubelet... Nov 03 00:40:57.786660 ip-10-0-133-217 systemd[1]: kubelet.service: Deactivated successfully. Nov 03 00:40:57.786835 ip-10-0-133-217 systemd[1]: Stopped Kubernetes Kubelet. Nov 03 00:40:57.786886 ip-10-0-133-217 systemd[1]: kubelet.service: Consumed 2.353s CPU time. Nov 03 00:40:57.794965 ip-10-0-133-217 systemd[1]: Starting Kubernetes Kubelet... Nov 03 00:40:57.875625 ip-10-0-133-217 kubenswrapper[4421]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 00:40:57.875625 ip-10-0-133-217 kubenswrapper[4421]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 03 00:40:57.875625 ip-10-0-133-217 kubenswrapper[4421]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 00:40:57.875625 ip-10-0-133-217 kubenswrapper[4421]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 00:40:57.878106 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.878015 4421 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 00:40:57.878106 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.878104 4421 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 00:40:57.878106 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.878108 4421 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 00:40:57.878106 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.878111 4421 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 00:40:57.878106 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.878114 4421 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 00:40:57.878345 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.878118 4421 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 00:40:57.878345 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.878121 4421 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 00:40:57.878345 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.878124 4421 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 00:40:57.878345 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.878129 4421 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 00:40:57.878345 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.878134 4421 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 00:40:57.878345 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.878139 4421 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 00:40:57.878345 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.878146 4421 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 00:40:57.878345 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.878151 4421 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 00:40:57.878345 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.878156 4421 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 00:40:57.878345 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.878161 4421 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 00:40:57.878345 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.878167 4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 00:40:57.878345 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.878172 4421 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 00:40:57.878345 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.878179 4421 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 00:40:57.878345 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.878186 4421 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 00:40:57.878345 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.878190 4421 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 00:40:57.878345 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.878193 4421 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 00:40:57.878345 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.878204 4421 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 00:40:57.878345 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.878208 4421 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 00:40:57.878345 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.878211 4421 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 00:40:57.878853 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.878214 4421 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 00:40:57.878853 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.878217 4421 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 00:40:57.878853 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.878220 4421 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 00:40:57.878853 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.878223 4421 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 00:40:57.878853 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.878226 4421 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 00:40:57.878853 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.878228 4421 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 00:40:57.878853 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.878231 4421 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 00:40:57.878853 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.878234 4421 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 00:40:57.878853 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.878236 4421 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 00:40:57.878853 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.878239 4421 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 00:40:57.878853 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.878243 4421 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 00:40:57.878853 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.878248 4421 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 00:40:57.878853 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.878253 4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 00:40:57.878853 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.878257 4421 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 00:40:57.878853 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.878262 4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 00:40:57.878853 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.878264 4421 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 00:40:57.878853 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.878267 4421 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 00:40:57.878853 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.878282 4421 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 00:40:57.878853 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.878285 4421 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 00:40:57.879447 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.878287 4421 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 00:40:57.879447 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.878290 4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 00:40:57.879447 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.878293 4421 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 00:40:57.879447 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.878295 4421 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 00:40:57.879447 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.878298 4421 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 00:40:57.879447 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.878300 4421 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 00:40:57.879447 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.878303 4421 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 00:40:57.879447 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.878306 4421 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 00:40:57.879447 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.878309 4421 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 00:40:57.879447 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.878312 4421 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 00:40:57.879447 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.878323 4421 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 00:40:57.879447 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.878329 4421 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 00:40:57.879447 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.878335 4421 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 00:40:57.879447 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.878340 4421 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 00:40:57.879447 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.878343 4421 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 00:40:57.879447 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.878346 4421 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 00:40:57.879447 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.878348 4421 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 00:40:57.879447 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.878352 4421 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 00:40:57.879447 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.878355 4421 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 00:40:57.879972 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.878357 4421 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 00:40:57.879972 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.878360 4421 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 00:40:57.879972 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.878363 4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 00:40:57.879972 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.878365 4421 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 00:40:57.879972 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.878368 4421 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 00:40:57.879972 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.878371 4421 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 00:40:57.879972 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.878373 4421 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 00:40:57.879972 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.878376 4421 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 00:40:57.879972 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.878378 4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 00:40:57.879972 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.878382 4421 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 00:40:57.879972 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.878386 4421 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 00:40:57.879972 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.878388 4421 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 00:40:57.879972 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.878391 4421 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 00:40:57.879972 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.878394 4421 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 00:40:57.879972 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.878398 4421 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 00:40:57.879972 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.878403 4421 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 00:40:57.879972 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.878410 4421 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 00:40:57.879972 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.878415 4421 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 00:40:57.879972 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.878420 4421 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 00:40:57.880539 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.878425 4421 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 00:40:57.880539 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.878430 4421 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 00:40:57.880539 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.878435 4421 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 00:40:57.880539 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.878438 4421 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 00:40:57.880539 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.878441 4421 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 00:40:57.880539 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.878445 4421 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 00:40:57.880539 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.878448 4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 00:40:57.880539 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.878452 4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 00:40:57.880539 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.878455 4421 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 00:40:57.880539 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.878458 4421 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 00:40:57.880539 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.878460 4421 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 00:40:57.880539 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.878463 4421 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 00:40:57.880539 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.878466 4421 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 00:40:57.880539 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.878476 4421 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 00:40:57.880539 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.878480 4421 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 00:40:57.880539 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.878482 4421 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 00:40:57.880539 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.878486 4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 00:40:57.880539 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.878488 4421 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 00:40:57.880539 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.878491 4421 feature_gate.go:386] unrecognized feature gate: Example Nov 03 00:40:57.880539 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.878494 4421 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 00:40:57.881124 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.878498 4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 00:40:57.881124 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.878503 4421 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 00:40:57.881124 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.878507 4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 00:40:57.881124 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.878514 4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 00:40:57.881124 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.878518 4421 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 00:40:57.881124 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.879573 4421 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 00:40:57.881124 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.879584 4421 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 00:40:57.881124 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.879587 4421 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 00:40:57.881124 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.879590 4421 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 00:40:57.881124 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.879593 4421 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 00:40:57.881124 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.879596 4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 00:40:57.881124 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.879599 4421 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 00:40:57.881124 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.879602 4421 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 00:40:57.881124 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.879605 4421 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 00:40:57.881124 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.879608 4421 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 00:40:57.881124 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.879611 4421 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 00:40:57.881124 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.879614 4421 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 00:40:57.881124 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.879617 4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 00:40:57.881124 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.879621 4421 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 00:40:57.881663 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.879624 4421 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 00:40:57.881663 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.879627 4421 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 00:40:57.881663 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.879629 4421 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 00:40:57.881663 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.879632 4421 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 00:40:57.881663 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.879634 4421 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 00:40:57.881663 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.879637 4421 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 00:40:57.881663 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.879639 4421 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 00:40:57.881663 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.879642 4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 00:40:57.881663 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.879645 4421 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 00:40:57.881663 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.879647 4421 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 00:40:57.881663 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.879650 4421 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 00:40:57.881663 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.879653 4421 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 00:40:57.881663 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.879656 4421 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 00:40:57.881663 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.879659 4421 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 00:40:57.881663 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.879661 4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 00:40:57.881663 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.879664 4421 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 00:40:57.881663 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.879667 4421 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 00:40:57.881663 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.879669 4421 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 00:40:57.881663 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.879672 4421 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 00:40:57.881663 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.879674 4421 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 00:40:57.882243 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.879677 4421 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 00:40:57.882243 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.879681 4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 00:40:57.882243 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.879684 4421 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 00:40:57.882243 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.879687 4421 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 00:40:57.882243 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.879690 4421 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 00:40:57.882243 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.879693 4421 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 00:40:57.882243 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.879696 4421 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 00:40:57.882243 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.879698 4421 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 00:40:57.882243 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.879701 4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 00:40:57.882243 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.879704 4421 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 00:40:57.882243 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.879707 4421 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 00:40:57.882243 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.879709 4421 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 00:40:57.882243 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.879718 4421 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 00:40:57.882243 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.879721 4421 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 00:40:57.882243 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.879724 4421 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 00:40:57.882243 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.879727 4421 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 00:40:57.882243 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.879730 4421 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 00:40:57.882243 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.879733 4421 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 00:40:57.882243 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.879735 4421 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 00:40:57.882243 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.879738 4421 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 00:40:57.882862 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.879741 4421 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 00:40:57.882862 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.879743 4421 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 00:40:57.882862 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.879746 4421 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 00:40:57.882862 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.879748 4421 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 00:40:57.882862 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.879751 4421 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 00:40:57.882862 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.879756 4421 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 00:40:57.882862 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.879764 4421 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 00:40:57.882862 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.879770 4421 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 00:40:57.882862 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.879774 4421 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 00:40:57.882862 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.879776 4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 00:40:57.882862 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.879779 4421 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 00:40:57.882862 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.879782 4421 feature_gate.go:386] unrecognized feature gate: Example Nov 03 00:40:57.882862 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.879787 4421 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 00:40:57.882862 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.879791 4421 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 00:40:57.882862 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.879795 4421 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 00:40:57.882862 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.879798 4421 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 00:40:57.882862 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.879801 4421 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 00:40:57.882862 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.879803 4421 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 00:40:57.882862 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.879806 4421 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 00:40:57.883479 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.879808 4421 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 00:40:57.883479 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.879811 4421 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 00:40:57.883479 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.879814 4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 00:40:57.883479 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.879817 4421 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 00:40:57.883479 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.879820 4421 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 00:40:57.883479 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.879824 4421 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 00:40:57.883479 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.879830 4421 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 00:40:57.883479 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.879835 4421 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 00:40:57.883479 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.879843 4421 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 00:40:57.883479 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.879849 4421 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 00:40:57.883479 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.879854 4421 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 00:40:57.883479 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.879860 4421 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 00:40:57.883479 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.879865 4421 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 00:40:57.883479 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.879870 4421 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 00:40:57.883479 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.879873 4421 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 00:40:57.883479 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.879875 4421 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 00:40:57.883479 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.879878 4421 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 00:40:57.883479 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.879881 4421 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 00:40:57.883479 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.879883 4421 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 00:40:57.883479 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.879886 4421 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 00:40:57.884057 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.879888 4421 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 00:40:57.884057 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.879891 4421 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 00:40:57.884057 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.879893 4421 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 00:40:57.884057 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.879896 4421 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 00:40:57.884057 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.879898 4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 00:40:57.884057 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.879901 4421 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 00:40:57.884057 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.879904 4421 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 00:40:57.884057 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.879907 4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 00:40:57.884057 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.879909 4421 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 00:40:57.884057 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.879912 4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 00:40:57.884057 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.879914 4421 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 00:40:57.884057 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.879917 4421 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 00:40:57.884057 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.879920 4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 00:40:57.884057 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.880536 4421 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 00:40:57.884057 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.880547 4421 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 00:40:57.884057 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.880553 4421 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 00:40:57.884057 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.880558 4421 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 00:40:57.884057 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.880564 4421 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 00:40:57.884057 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.880569 4421 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 00:40:57.884617 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.880572 4421 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 00:40:57.884617 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.880576 4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 00:40:57.884617 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.880578 4421 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 00:40:57.884617 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.880581 4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 00:40:57.884617 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.880584 4421 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 00:40:57.884617 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.880587 4421 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 00:40:57.884617 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.880590 4421 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 00:40:57.884617 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.880592 4421 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 00:40:57.884617 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.880595 4421 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 00:40:57.884617 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.880598 4421 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 00:40:57.884617 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.880600 4421 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 00:40:57.884617 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.880603 4421 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 00:40:57.884617 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.880606 4421 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 00:40:57.884617 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.880608 4421 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 00:40:57.884617 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.880611 4421 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 00:40:57.884617 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.880614 4421 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 00:40:57.884617 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.880616 4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 00:40:57.884617 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.880619 4421 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 00:40:57.884617 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.880622 4421 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 00:40:57.885158 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.880624 4421 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 00:40:57.885158 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.880627 4421 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 00:40:57.885158 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.880630 4421 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 00:40:57.885158 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.880632 4421 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 00:40:57.885158 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.880636 4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 00:40:57.885158 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.880640 4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 00:40:57.885158 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.880645 4421 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 00:40:57.885158 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.880649 4421 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 00:40:57.885158 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.880655 4421 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 00:40:57.885158 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.880661 4421 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 00:40:57.885158 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.880668 4421 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 00:40:57.885158 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.880674 4421 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 00:40:57.885158 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.880679 4421 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 00:40:57.885158 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.880683 4421 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 00:40:57.885158 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.880686 4421 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 00:40:57.885158 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.880689 4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 00:40:57.885158 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.880692 4421 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 00:40:57.885158 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.880694 4421 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 00:40:57.885158 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.880697 4421 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 00:40:57.885158 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.880699 4421 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 00:40:57.885751 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.880702 4421 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 00:40:57.885751 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.880704 4421 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 00:40:57.885751 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.880707 4421 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 00:40:57.885751 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.880710 4421 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 00:40:57.885751 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.880712 4421 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 00:40:57.885751 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.880717 4421 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 00:40:57.885751 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.880721 4421 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 00:40:57.885751 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.880727 4421 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 00:40:57.885751 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.880732 4421 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 00:40:57.885751 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.880736 4421 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 00:40:57.885751 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.880741 4421 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 00:40:57.885751 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.880747 4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 00:40:57.885751 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.880752 4421 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 00:40:57.885751 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.880757 4421 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 00:40:57.885751 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.880762 4421 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 00:40:57.885751 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.880767 4421 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 00:40:57.885751 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.880772 4421 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 00:40:57.885751 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.880776 4421 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 00:40:57.885751 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.880779 4421 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 00:40:57.885751 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.880783 4421 feature_gate.go:386] unrecognized feature gate: Example Nov 03 00:40:57.886370 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.880786 4421 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 00:40:57.886370 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.880792 4421 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 00:40:57.886370 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.880795 4421 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 00:40:57.886370 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.880817 4421 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 00:40:57.886370 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.880821 4421 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 00:40:57.886370 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.880825 4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 00:40:57.886370 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.880829 4421 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 00:40:57.886370 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.880832 4421 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 00:40:57.886370 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.880835 4421 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 00:40:57.886370 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.880838 4421 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 00:40:57.886370 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.880840 4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 00:40:57.886370 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.880844 4421 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 00:40:57.886370 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.880847 4421 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 00:40:57.886370 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.880850 4421 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 00:40:57.886370 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.880852 4421 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 00:40:57.886370 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.880855 4421 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 00:40:57.886370 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.880858 4421 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 00:40:57.886370 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.880861 4421 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 00:40:57.886370 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.880863 4421 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 00:40:57.886370 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.880866 4421 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 00:40:57.886961 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.880869 4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 00:40:57.886961 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.880872 4421 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 00:40:57.886961 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.880876 4421 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 00:40:57.886961 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.880879 4421 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 00:40:57.886961 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.880882 4421 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 00:40:57.886961 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.880885 4421 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 00:40:57.886961 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.880888 4421 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 00:40:57.886961 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.880890 4421 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 00:40:57.886961 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.880893 4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 00:40:57.886961 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.880895 4421 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 00:40:57.886961 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.880898 4421 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 00:40:57.886961 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.880901 4421 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 00:40:57.886961 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.880904 4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 00:40:57.886961 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.880908 4421 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 00:40:57.886961 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.880910 4421 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 00:40:57.886961 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.880916 4421 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 00:40:57.886961 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.880919 4421 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 00:40:57.886961 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.880922 4421 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 00:40:57.886961 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.880924 4421 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 00:40:57.887542 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.880928 4421 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 00:40:57.887542 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.880930 4421 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 00:40:57.887542 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.881049 4421 flags.go:64] FLAG: --address="0.0.0.0" Nov 03 00:40:57.887542 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.881061 4421 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 03 00:40:57.887542 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.881072 4421 flags.go:64] FLAG: --anonymous-auth="true" Nov 03 00:40:57.887542 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.881077 4421 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 03 00:40:57.887542 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.881083 4421 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 03 00:40:57.887542 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.881086 4421 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 03 00:40:57.887542 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.881094 4421 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 03 00:40:57.887542 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.881099 4421 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 03 00:40:57.887542 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.881102 4421 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 03 00:40:57.887542 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.881105 4421 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 03 00:40:57.887542 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.881109 4421 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 03 00:40:57.887542 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.881113 4421 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 03 00:40:57.887542 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.881116 4421 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 03 00:40:57.887542 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.881119 4421 flags.go:64] FLAG: --cgroup-root="" Nov 03 00:40:57.887542 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.881122 4421 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 03 00:40:57.887542 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.881126 4421 flags.go:64] FLAG: --client-ca-file="" Nov 03 00:40:57.887542 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.881129 4421 flags.go:64] FLAG: --cloud-provider="external" Nov 03 00:40:57.887542 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.881132 4421 flags.go:64] FLAG: --cluster-dns="[]" Nov 03 00:40:57.887542 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.881136 4421 flags.go:64] FLAG: --cluster-domain="" Nov 03 00:40:57.887542 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.881139 4421 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 03 00:40:57.887542 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.881142 4421 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 03 00:40:57.888231 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.881146 4421 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 03 00:40:57.888231 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.881150 4421 flags.go:64] FLAG: --container-log-max-files="5" Nov 03 00:40:57.888231 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.881154 4421 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 03 00:40:57.888231 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.881159 4421 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 03 00:40:57.888231 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.881162 4421 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 03 00:40:57.888231 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.881165 4421 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 03 00:40:57.888231 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.881168 4421 flags.go:64] FLAG: --contention-profiling="false" Nov 03 00:40:57.888231 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.881172 4421 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 03 00:40:57.888231 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.881175 4421 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 03 00:40:57.888231 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.881178 4421 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 03 00:40:57.888231 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.881182 4421 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 03 00:40:57.888231 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.881187 4421 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 03 00:40:57.888231 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.881190 4421 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 03 00:40:57.888231 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.881193 4421 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 03 00:40:57.888231 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.881196 4421 flags.go:64] FLAG: --enable-load-reader="false" Nov 03 00:40:57.888231 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.881199 4421 flags.go:64] FLAG: --enable-server="true" Nov 03 00:40:57.888231 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.881203 4421 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 03 00:40:57.888231 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.881211 4421 flags.go:64] FLAG: --event-burst="100" Nov 03 00:40:57.888231 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.881217 4421 flags.go:64] FLAG: --event-qps="50" Nov 03 00:40:57.888231 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.881222 4421 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 03 00:40:57.888231 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.881229 4421 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 03 00:40:57.888231 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.881234 4421 flags.go:64] FLAG: --eviction-hard="" Nov 03 00:40:57.888231 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.881243 4421 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 03 00:40:57.888231 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.881248 4421 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 03 00:40:57.888911 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.881254 4421 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 03 00:40:57.888911 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.881260 4421 flags.go:64] FLAG: --eviction-soft="" Nov 03 00:40:57.888911 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.881266 4421 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 03 00:40:57.888911 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.881284 4421 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 03 00:40:57.888911 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.881290 4421 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 03 00:40:57.888911 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.881296 4421 flags.go:64] FLAG: --experimental-mounter-path="" Nov 03 00:40:57.888911 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.881300 4421 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 03 00:40:57.888911 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.881303 4421 flags.go:64] FLAG: --fail-swap-on="true" Nov 03 00:40:57.888911 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.881306 4421 flags.go:64] FLAG: --feature-gates="" Nov 03 00:40:57.888911 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.881311 4421 flags.go:64] FLAG: --file-check-frequency="20s" Nov 03 00:40:57.888911 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.881314 4421 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 03 00:40:57.888911 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.881319 4421 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 03 00:40:57.888911 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.881323 4421 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 03 00:40:57.888911 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.881326 4421 flags.go:64] FLAG: --healthz-port="10248" Nov 03 00:40:57.888911 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.881329 4421 flags.go:64] FLAG: --help="false" Nov 03 00:40:57.888911 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.881332 4421 flags.go:64] FLAG: --hostname-override="ip-10-0-133-217.us-east-2.compute.internal" Nov 03 00:40:57.888911 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.881336 4421 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 03 00:40:57.888911 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.881339 4421 flags.go:64] FLAG: --http-check-frequency="20s" Nov 03 00:40:57.888911 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.881342 4421 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 03 00:40:57.888911 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.881347 4421 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 03 00:40:57.888911 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.881351 4421 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 03 00:40:57.888911 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.881354 4421 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 03 00:40:57.888911 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.881359 4421 flags.go:64] FLAG: --image-service-endpoint="" Nov 03 00:40:57.889679 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.881364 4421 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 03 00:40:57.889679 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.881370 4421 flags.go:64] FLAG: --kube-api-burst="100" Nov 03 00:40:57.889679 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.881377 4421 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 03 00:40:57.889679 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.881381 4421 flags.go:64] FLAG: --kube-api-qps="50" Nov 03 00:40:57.889679 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.881384 4421 flags.go:64] FLAG: --kube-reserved="" Nov 03 00:40:57.889679 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.881387 4421 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 03 00:40:57.889679 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.881390 4421 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 03 00:40:57.889679 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.881394 4421 flags.go:64] FLAG: --kubelet-cgroups="" Nov 03 00:40:57.889679 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.881397 4421 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 03 00:40:57.889679 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.881400 4421 flags.go:64] FLAG: --lock-file="" Nov 03 00:40:57.889679 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.881403 4421 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 03 00:40:57.889679 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.881406 4421 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 03 00:40:57.889679 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.881409 4421 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 03 00:40:57.889679 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.881417 4421 flags.go:64] FLAG: --log-json-split-stream="false" Nov 03 00:40:57.889679 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.881422 4421 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 03 00:40:57.889679 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.881428 4421 flags.go:64] FLAG: --log-text-split-stream="false" Nov 03 00:40:57.889679 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.881431 4421 flags.go:64] FLAG: --logging-format="text" Nov 03 00:40:57.889679 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.881434 4421 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 03 00:40:57.889679 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.881439 4421 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 03 00:40:57.889679 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.881442 4421 flags.go:64] FLAG: --manifest-url="" Nov 03 00:40:57.889679 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.881446 4421 flags.go:64] FLAG: --manifest-url-header="" Nov 03 00:40:57.889679 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.881452 4421 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 03 00:40:57.889679 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.881455 4421 flags.go:64] FLAG: --max-open-files="1000000" Nov 03 00:40:57.889679 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.881459 4421 flags.go:64] FLAG: --max-pods="110" Nov 03 00:40:57.889679 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.881462 4421 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 03 00:40:57.890389 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.881465 4421 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 03 00:40:57.890389 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.881469 4421 flags.go:64] FLAG: --memory-manager-policy="None" Nov 03 00:40:57.890389 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.881472 4421 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 03 00:40:57.890389 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.881475 4421 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 03 00:40:57.890389 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.881479 4421 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 03 00:40:57.890389 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.881483 4421 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 03 00:40:57.890389 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.881494 4421 flags.go:64] FLAG: --node-status-max-images="50" Nov 03 00:40:57.890389 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.881500 4421 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 03 00:40:57.890389 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.881506 4421 flags.go:64] FLAG: --oom-score-adj="-999" Nov 03 00:40:57.890389 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.881511 4421 flags.go:64] FLAG: --pod-cidr="" Nov 03 00:40:57.890389 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.881515 4421 flags.go:64] FLAG: --pod-manifest-path="" Nov 03 00:40:57.890389 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.881518 4421 flags.go:64] FLAG: --pod-max-pids="-1" Nov 03 00:40:57.890389 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.881521 4421 flags.go:64] FLAG: --pods-per-core="0" Nov 03 00:40:57.890389 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.881524 4421 flags.go:64] FLAG: --port="10250" Nov 03 00:40:57.890389 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.881527 4421 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 03 00:40:57.890389 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.881531 4421 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-0836f34c7363c1256" Nov 03 00:40:57.890389 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.881534 4421 flags.go:64] FLAG: --qos-reserved="" Nov 03 00:40:57.890389 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.881537 4421 flags.go:64] FLAG: --read-only-port="10255" Nov 03 00:40:57.890389 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.881541 4421 flags.go:64] FLAG: --register-node="true" Nov 03 00:40:57.890389 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.881544 4421 flags.go:64] FLAG: --register-with-taints="" Nov 03 00:40:57.890389 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.881548 4421 flags.go:64] FLAG: --registry-burst="10" Nov 03 00:40:57.890389 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.881551 4421 flags.go:64] FLAG: --registry-qps="5" Nov 03 00:40:57.890389 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.881555 4421 flags.go:64] FLAG: --reserved-cpus="" Nov 03 00:40:57.890389 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.881558 4421 flags.go:64] FLAG: --reserved-memory="" Nov 03 00:40:57.890389 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.881562 4421 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 03 00:40:57.890389 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.881565 4421 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 03 00:40:57.891149 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.881570 4421 flags.go:64] FLAG: --rotate-certificates="false" Nov 03 00:40:57.891149 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.881579 4421 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 03 00:40:57.891149 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.881585 4421 flags.go:64] FLAG: --runonce="false" Nov 03 00:40:57.891149 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.881589 4421 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 03 00:40:57.891149 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.881592 4421 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 03 00:40:57.891149 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.881595 4421 flags.go:64] FLAG: --seccomp-default="false" Nov 03 00:40:57.891149 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.881598 4421 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 03 00:40:57.891149 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.881601 4421 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 03 00:40:57.891149 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.881605 4421 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 03 00:40:57.891149 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.881608 4421 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 03 00:40:57.891149 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.881611 4421 flags.go:64] FLAG: --storage-driver-password="root" Nov 03 00:40:57.891149 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.881615 4421 flags.go:64] FLAG: --storage-driver-secure="false" Nov 03 00:40:57.891149 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.881619 4421 flags.go:64] FLAG: --storage-driver-table="stats" Nov 03 00:40:57.891149 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.881622 4421 flags.go:64] FLAG: --storage-driver-user="root" Nov 03 00:40:57.891149 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.881625 4421 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 03 00:40:57.891149 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.881630 4421 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 03 00:40:57.891149 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.881633 4421 flags.go:64] FLAG: --system-cgroups="" Nov 03 00:40:57.891149 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.881636 4421 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 03 00:40:57.891149 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.881643 4421 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 03 00:40:57.891149 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.881646 4421 flags.go:64] FLAG: --tls-cert-file="" Nov 03 00:40:57.891149 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.881649 4421 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 03 00:40:57.891149 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.881653 4421 flags.go:64] FLAG: --tls-min-version="" Nov 03 00:40:57.891149 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.881656 4421 flags.go:64] FLAG: --tls-private-key-file="" Nov 03 00:40:57.891149 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.881659 4421 flags.go:64] FLAG: --topology-manager-policy="none" Nov 03 00:40:57.891149 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.881662 4421 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 03 00:40:57.892055 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.881666 4421 flags.go:64] FLAG: --topology-manager-scope="container" Nov 03 00:40:57.892055 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.881669 4421 flags.go:64] FLAG: --v="2" Nov 03 00:40:57.892055 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.881674 4421 flags.go:64] FLAG: --version="false" Nov 03 00:40:57.892055 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.881679 4421 flags.go:64] FLAG: --vmodule="" Nov 03 00:40:57.892055 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.881684 4421 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 03 00:40:57.892055 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.881687 4421 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 03 00:40:57.892055 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.881848 4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 00:40:57.892055 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.881853 4421 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 00:40:57.892055 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.881858 4421 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 00:40:57.892055 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.881861 4421 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 00:40:57.892055 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.881864 4421 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 00:40:57.892055 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.881866 4421 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 00:40:57.892055 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.881869 4421 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 00:40:57.892055 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.881872 4421 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 00:40:57.892055 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.881874 4421 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 00:40:57.892055 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.881877 4421 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 00:40:57.892055 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.881880 4421 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 00:40:57.892055 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.881882 4421 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 00:40:57.892055 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.881885 4421 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 00:40:57.892055 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.881889 4421 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 00:40:57.892055 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.881891 4421 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 00:40:57.892685 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.881894 4421 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 00:40:57.892685 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.881898 4421 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 00:40:57.892685 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.881901 4421 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 00:40:57.892685 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.881905 4421 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 00:40:57.892685 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.881909 4421 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 00:40:57.892685 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.881914 4421 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 00:40:57.892685 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.881919 4421 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 00:40:57.892685 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.881924 4421 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 00:40:57.892685 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.881928 4421 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 00:40:57.892685 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.881933 4421 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 00:40:57.892685 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.881937 4421 feature_gate.go:386] unrecognized feature gate: Example Nov 03 00:40:57.892685 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.881940 4421 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 00:40:57.892685 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.881943 4421 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 00:40:57.892685 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.881946 4421 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 00:40:57.892685 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.881948 4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 00:40:57.892685 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.881951 4421 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 00:40:57.892685 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.881956 4421 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 00:40:57.892685 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.881961 4421 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 00:40:57.892685 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.881965 4421 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 00:40:57.893283 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.881969 4421 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 00:40:57.893283 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.881972 4421 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 00:40:57.893283 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.881975 4421 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 00:40:57.893283 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.881977 4421 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 00:40:57.893283 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.881980 4421 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 00:40:57.893283 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.881984 4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 00:40:57.893283 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.881988 4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 00:40:57.893283 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.881993 4421 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 00:40:57.893283 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.881997 4421 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 00:40:57.893283 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.882002 4421 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 00:40:57.893283 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.882007 4421 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 00:40:57.893283 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.882009 4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 00:40:57.893283 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.882013 4421 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 00:40:57.893283 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.882016 4421 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 00:40:57.893283 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.882020 4421 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 00:40:57.893283 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.882023 4421 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 00:40:57.893283 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.882025 4421 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 00:40:57.893283 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.882028 4421 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 00:40:57.893283 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.882031 4421 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 00:40:57.893283 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.882034 4421 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 00:40:57.893904 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.882038 4421 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 00:40:57.893904 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.882042 4421 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 00:40:57.893904 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.882045 4421 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 00:40:57.893904 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.882047 4421 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 00:40:57.893904 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.882050 4421 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 00:40:57.893904 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.882053 4421 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 00:40:57.893904 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.882055 4421 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 00:40:57.893904 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.882058 4421 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 00:40:57.893904 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.882061 4421 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 00:40:57.893904 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.882063 4421 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 00:40:57.893904 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.882067 4421 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 00:40:57.893904 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.882069 4421 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 00:40:57.893904 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.882073 4421 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 00:40:57.893904 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.882076 4421 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 00:40:57.893904 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.882078 4421 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 00:40:57.893904 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.882081 4421 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 00:40:57.893904 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.882084 4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 00:40:57.893904 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.882086 4421 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 00:40:57.893904 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.882089 4421 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 00:40:57.894506 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.882092 4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 00:40:57.894506 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.882095 4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 00:40:57.894506 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.882097 4421 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 00:40:57.894506 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.882100 4421 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 00:40:57.894506 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.882103 4421 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 00:40:57.894506 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.882105 4421 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 00:40:57.894506 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.882108 4421 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 00:40:57.894506 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.882129 4421 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 00:40:57.894506 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.882133 4421 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 00:40:57.894506 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.882136 4421 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 00:40:57.894506 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.882139 4421 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 00:40:57.894506 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.882142 4421 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 00:40:57.894506 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.882145 4421 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 00:40:57.894506 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.882148 4421 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 00:40:57.894506 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.882150 4421 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 00:40:57.894506 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.882153 4421 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 00:40:57.894506 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.882156 4421 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 00:40:57.894506 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.882160 4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 00:40:57.894506 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.882167 4421 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 00:40:57.894506 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.882173 4421 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 00:40:57.895243 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.882177 4421 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 00:40:57.895243 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.882181 4421 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 00:40:57.895243 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.882184 4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 00:40:57.895243 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.882187 4421 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 00:40:57.895243 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.882190 4421 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 00:40:57.895243 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.882194 4421 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 00:40:57.895243 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.882198 4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 00:40:57.895243 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.882203 4421 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 00:40:57.895243 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.882208 4421 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 00:40:57.895243 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.882214 4421 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 00:40:57.895243 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.882219 4421 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 00:40:57.895243 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.882223 4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 00:40:57.895243 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.882228 4421 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 00:40:57.895243 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.882283 4421 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 00:40:57.895243 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.882572 4421 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 00:40:57.895243 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.882586 4421 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 00:40:57.895781 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.882592 4421 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 00:40:57.895781 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.882597 4421 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 00:40:57.895781 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.882602 4421 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 00:40:57.895781 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.882609 4421 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 00:40:57.895781 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.884890 4421 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 00:40:57.895781 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.884910 4421 feature_gate.go:386] unrecognized feature gate: Example Nov 03 00:40:57.895781 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.884916 4421 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 00:40:57.895781 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.884921 4421 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 00:40:57.895781 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.884929 4421 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 00:40:57.895781 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.884934 4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 00:40:57.895781 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.884939 4421 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 00:40:57.895781 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.884944 4421 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 00:40:57.895781 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.884948 4421 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 00:40:57.895781 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.884952 4421 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 00:40:57.895781 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.884957 4421 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 00:40:57.895781 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.884961 4421 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 00:40:57.895781 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.884966 4421 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 00:40:57.895781 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.884970 4421 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 00:40:57.895781 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.884974 4421 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 00:40:57.896450 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.884979 4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 00:40:57.896450 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.884983 4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 00:40:57.896450 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.884988 4421 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 00:40:57.896450 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.884992 4421 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 00:40:57.896450 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.884996 4421 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 00:40:57.896450 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.885001 4421 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 00:40:57.896450 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.885005 4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 00:40:57.896450 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.885009 4421 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 00:40:57.896450 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.885013 4421 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 00:40:57.896450 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.885017 4421 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 00:40:57.896450 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.885023 4421 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 00:40:57.896450 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.885027 4421 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 00:40:57.896450 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.885032 4421 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 00:40:57.896450 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.885036 4421 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 00:40:57.896450 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.885040 4421 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 00:40:57.896450 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.885048 4421 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 00:40:57.896450 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.885055 4421 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 00:40:57.896450 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.885064 4421 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 00:40:57.896450 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.885069 4421 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 00:40:57.897082 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.885076 4421 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 00:40:57.897082 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.885080 4421 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 00:40:57.897082 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.885085 4421 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 00:40:57.897082 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.885090 4421 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 00:40:57.897082 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.885095 4421 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 00:40:57.897082 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.885099 4421 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 00:40:57.897082 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.885104 4421 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 00:40:57.897082 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.885108 4421 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 00:40:57.897082 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.885112 4421 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 00:40:57.897082 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.885117 4421 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 00:40:57.897082 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.885121 4421 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 00:40:57.897082 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.885126 4421 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 00:40:57.897082 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.885130 4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 00:40:57.897082 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.885134 4421 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 00:40:57.897082 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.885138 4421 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 00:40:57.897082 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.885146 4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 00:40:57.897082 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.885150 4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 00:40:57.897082 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.885155 4421 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 00:40:57.897082 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.885159 4421 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 00:40:57.897082 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.885164 4421 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 00:40:57.897673 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.885168 4421 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 00:40:57.897673 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.885172 4421 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 00:40:57.897673 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.885176 4421 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 00:40:57.897673 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.885181 4421 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 00:40:57.897673 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.885185 4421 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 00:40:57.897673 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.885189 4421 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 00:40:57.897673 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.885193 4421 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 00:40:57.897673 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.885197 4421 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 00:40:57.897673 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.885201 4421 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 00:40:57.897673 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.885206 4421 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 00:40:57.897673 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.885210 4421 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 00:40:57.897673 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.885216 4421 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 00:40:57.897673 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.885220 4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 00:40:57.897673 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.885227 4421 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 00:40:57.897673 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.885233 4421 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 00:40:57.897673 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.885238 4421 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 00:40:57.897673 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.885242 4421 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 00:40:57.897673 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.885247 4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 00:40:57.897673 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.885252 4421 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 00:40:57.897673 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.885256 4421 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 00:40:57.898220 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.885260 4421 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 00:40:57.898220 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.885264 4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 00:40:57.898220 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.885268 4421 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 00:40:57.898220 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.885285 4421 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 00:40:57.898220 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.885292 4421 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 00:40:57.898220 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.885297 4421 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 00:40:57.898220 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.885301 4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 00:40:57.898220 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.885305 4421 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 00:40:57.898220 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.885309 4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 00:40:57.898220 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.885314 4421 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 00:40:57.898220 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.885319 4421 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 00:40:57.898220 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.885323 4421 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 00:40:57.898220 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.885327 4421 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 00:40:57.898220 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.885331 4421 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 00:40:57.898220 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.885335 4421 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 00:40:57.898220 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.885340 4421 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 00:40:57.898220 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.885344 4421 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 00:40:57.898220 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.885348 4421 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 00:40:57.898220 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.885352 4421 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 00:40:57.898220 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.885356 4421 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 00:40:57.899025 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.885360 4421 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 00:40:57.899025 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.885364 4421 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 00:40:57.899025 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.885368 4421 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 00:40:57.899025 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.885373 4421 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 00:40:57.899025 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.885377 4421 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 00:40:57.899025 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.885382 4421 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 00:40:57.899025 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.889758 4421 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 03 00:40:57.899025 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.889772 4421 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 03 00:40:57.899025 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.889791 4421 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 03 00:40:57.899025 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.889796 4421 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 03 00:40:57.899025 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.889852 4421 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 00:40:57.899025 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.889858 4421 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 00:40:57.899025 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.889861 4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 00:40:57.899025 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.889865 4421 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 00:40:57.899025 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.889868 4421 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 00:40:57.899025 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.889871 4421 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 00:40:57.899025 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.889874 4421 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 00:40:57.899025 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.889877 4421 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 00:40:57.899025 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.889880 4421 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 00:40:57.899691 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.889883 4421 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 00:40:57.899691 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.889887 4421 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 00:40:57.899691 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.889890 4421 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 00:40:57.899691 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.889893 4421 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 00:40:57.899691 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.889896 4421 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 00:40:57.899691 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.889899 4421 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 00:40:57.899691 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.889902 4421 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 00:40:57.899691 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.889905 4421 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 00:40:57.899691 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.889908 4421 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 00:40:57.899691 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.889912 4421 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 00:40:57.899691 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.889917 4421 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 00:40:57.899691 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.889921 4421 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 00:40:57.899691 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.889926 4421 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 00:40:57.899691 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.889930 4421 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 00:40:57.899691 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.889933 4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 00:40:57.899691 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.889936 4421 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 00:40:57.899691 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.889939 4421 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 00:40:57.899691 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.889942 4421 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 00:40:57.899691 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.889945 4421 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 00:40:57.899691 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.889948 4421 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 00:40:57.900318 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.889951 4421 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 00:40:57.900318 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.889953 4421 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 00:40:57.900318 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.889956 4421 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 00:40:57.900318 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.889959 4421 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 00:40:57.900318 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.889961 4421 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 00:40:57.900318 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.889964 4421 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 00:40:57.900318 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.889967 4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 00:40:57.900318 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.889971 4421 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 00:40:57.900318 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.889974 4421 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 00:40:57.900318 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.889978 4421 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 00:40:57.900318 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.889983 4421 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 00:40:57.900318 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.889987 4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 00:40:57.900318 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.889991 4421 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 00:40:57.900318 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.889996 4421 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 00:40:57.900318 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890000 4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 00:40:57.900318 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890006 4421 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 00:40:57.900318 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890010 4421 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 00:40:57.900318 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890015 4421 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 00:40:57.900318 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890019 4421 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 00:40:57.900880 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890022 4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 00:40:57.900880 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890024 4421 feature_gate.go:386] unrecognized feature gate: Example Nov 03 00:40:57.900880 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890027 4421 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 00:40:57.900880 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890030 4421 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 00:40:57.900880 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890032 4421 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 00:40:57.900880 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890035 4421 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 00:40:57.900880 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890038 4421 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 00:40:57.900880 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890041 4421 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 00:40:57.900880 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890044 4421 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 00:40:57.900880 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890047 4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 00:40:57.900880 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890050 4421 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 00:40:57.900880 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890052 4421 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 00:40:57.900880 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890055 4421 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 00:40:57.900880 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890058 4421 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 00:40:57.900880 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890060 4421 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 00:40:57.900880 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890063 4421 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 00:40:57.900880 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890066 4421 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 00:40:57.900880 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890068 4421 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 00:40:57.900880 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890071 4421 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 00:40:57.900880 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890074 4421 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 00:40:57.901466 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890077 4421 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 00:40:57.901466 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890081 4421 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 00:40:57.901466 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890085 4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 00:40:57.901466 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890089 4421 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 00:40:57.901466 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890092 4421 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 00:40:57.901466 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890095 4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 00:40:57.901466 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890098 4421 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 00:40:57.901466 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890101 4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 00:40:57.901466 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890104 4421 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 00:40:57.901466 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890106 4421 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 00:40:57.901466 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890109 4421 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 00:40:57.901466 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890112 4421 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 00:40:57.901466 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890114 4421 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 00:40:57.901466 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890117 4421 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 00:40:57.901466 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890120 4421 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 00:40:57.901466 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890122 4421 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 00:40:57.901466 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890125 4421 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 00:40:57.901466 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890128 4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 00:40:57.901466 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890130 4421 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 00:40:57.902035 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890133 4421 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 00:40:57.902035 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890136 4421 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 00:40:57.902035 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890139 4421 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 00:40:57.902035 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890141 4421 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 00:40:57.902035 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890144 4421 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 00:40:57.902035 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890146 4421 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 00:40:57.902035 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890149 4421 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 00:40:57.902035 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890152 4421 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 00:40:57.902035 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890155 4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 00:40:57.902035 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890158 4421 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 00:40:57.902035 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890160 4421 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 00:40:57.902035 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890163 4421 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 00:40:57.902035 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890166 4421 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 00:40:57.902035 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890169 4421 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 00:40:57.902035 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890171 4421 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 00:40:57.902035 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890174 4421 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 00:40:57.902035 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890177 4421 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 00:40:57.902035 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890181 4421 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 00:40:57.902035 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890183 4421 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 00:40:57.902632 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.890211 4421 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 00:40:57.902632 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890374 4421 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 00:40:57.902632 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890380 4421 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 00:40:57.902632 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890384 4421 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 00:40:57.902632 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890389 4421 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 00:40:57.902632 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890393 4421 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 00:40:57.902632 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890398 4421 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 00:40:57.902632 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890403 4421 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 00:40:57.902632 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890408 4421 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 00:40:57.902632 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890413 4421 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 00:40:57.902632 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890418 4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 00:40:57.902632 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890422 4421 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 00:40:57.902632 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890425 4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 00:40:57.902632 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890429 4421 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 00:40:57.902632 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890433 4421 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 00:40:57.903053 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890438 4421 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 00:40:57.903053 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890443 4421 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 00:40:57.903053 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890447 4421 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 00:40:57.903053 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890450 4421 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 00:40:57.903053 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890455 4421 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 00:40:57.903053 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890459 4421 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 00:40:57.903053 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890463 4421 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 00:40:57.903053 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890473 4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 00:40:57.903053 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890478 4421 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 00:40:57.903053 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890482 4421 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 00:40:57.903053 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890485 4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 00:40:57.903053 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890487 4421 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 00:40:57.903053 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890490 4421 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 00:40:57.903053 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890493 4421 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 00:40:57.903053 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890495 4421 feature_gate.go:386] unrecognized feature gate: Example Nov 03 00:40:57.903053 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890498 4421 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 00:40:57.903053 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890502 4421 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 00:40:57.903053 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890505 4421 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 00:40:57.903053 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890509 4421 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 00:40:57.903053 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890513 4421 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 00:40:57.903620 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890516 4421 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 00:40:57.903620 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890519 4421 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 00:40:57.903620 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890522 4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 00:40:57.903620 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890525 4421 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 00:40:57.903620 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890528 4421 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 00:40:57.903620 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890531 4421 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 00:40:57.903620 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890534 4421 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 00:40:57.903620 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890536 4421 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 00:40:57.903620 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890539 4421 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 00:40:57.903620 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890542 4421 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 00:40:57.903620 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890544 4421 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 00:40:57.903620 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890547 4421 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 00:40:57.903620 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890550 4421 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 00:40:57.903620 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890553 4421 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 00:40:57.903620 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890555 4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 00:40:57.903620 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890558 4421 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 00:40:57.903620 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890561 4421 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 00:40:57.903620 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890563 4421 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 00:40:57.903620 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890566 4421 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 00:40:57.903620 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890568 4421 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 00:40:57.904159 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890571 4421 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 00:40:57.904159 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890573 4421 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 00:40:57.904159 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890576 4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 00:40:57.904159 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890578 4421 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 00:40:57.904159 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890581 4421 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 00:40:57.904159 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890584 4421 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 00:40:57.904159 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890586 4421 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 00:40:57.904159 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890589 4421 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 00:40:57.904159 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890591 4421 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 00:40:57.904159 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890595 4421 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 00:40:57.904159 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890597 4421 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 00:40:57.904159 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890600 4421 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 00:40:57.904159 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890602 4421 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 00:40:57.904159 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890605 4421 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 00:40:57.904159 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890607 4421 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 00:40:57.904159 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890610 4421 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 00:40:57.904159 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890613 4421 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 00:40:57.904159 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890615 4421 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 00:40:57.904159 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890617 4421 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 00:40:57.904159 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890620 4421 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 00:40:57.904865 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890623 4421 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 00:40:57.904865 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890625 4421 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 00:40:57.904865 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890628 4421 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 00:40:57.904865 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890633 4421 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 00:40:57.904865 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890636 4421 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 00:40:57.904865 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890639 4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 00:40:57.904865 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890642 4421 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 00:40:57.904865 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890644 4421 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 00:40:57.904865 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890647 4421 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 00:40:57.904865 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890649 4421 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 00:40:57.904865 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890652 4421 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 00:40:57.904865 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890655 4421 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 00:40:57.904865 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890657 4421 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 00:40:57.904865 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890660 4421 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 00:40:57.904865 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890662 4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 00:40:57.904865 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890665 4421 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 00:40:57.904865 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890667 4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 00:40:57.904865 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890670 4421 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 00:40:57.904865 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890672 4421 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 00:40:57.904865 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890675 4421 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 00:40:57.905491 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890677 4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 00:40:57.905491 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890681 4421 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 00:40:57.905491 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890687 4421 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 00:40:57.905491 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890693 4421 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 00:40:57.905491 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890700 4421 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 00:40:57.905491 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890706 4421 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 00:40:57.905491 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890711 4421 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 00:40:57.905491 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890716 4421 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 00:40:57.905491 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890721 4421 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 00:40:57.905491 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890725 4421 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 00:40:57.905491 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890730 4421 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 00:40:57.905491 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890735 4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 00:40:57.905491 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.890770 4421 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 00:40:57.905491 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890819 4421 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 00:40:57.905491 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890826 4421 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 00:40:57.905993 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890831 4421 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 00:40:57.905993 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890834 4421 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 00:40:57.905993 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890837 4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 00:40:57.905993 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890840 4421 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 00:40:57.905993 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890843 4421 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 00:40:57.905993 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890845 4421 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 00:40:57.905993 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890848 4421 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 00:40:57.905993 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890851 4421 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 00:40:57.905993 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890853 4421 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 00:40:57.905993 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890856 4421 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 00:40:57.905993 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890859 4421 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 00:40:57.905993 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890862 4421 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 00:40:57.905993 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890864 4421 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 00:40:57.905993 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890867 4421 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 00:40:57.905993 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890870 4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 00:40:57.905993 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890873 4421 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 00:40:57.905993 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890875 4421 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 00:40:57.905993 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890878 4421 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 00:40:57.905993 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890881 4421 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 00:40:57.905993 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890883 4421 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 00:40:57.906646 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890887 4421 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 00:40:57.906646 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890890 4421 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 00:40:57.906646 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890892 4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 00:40:57.906646 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890895 4421 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 00:40:57.906646 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890898 4421 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 00:40:57.906646 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890900 4421 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 00:40:57.906646 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890903 4421 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 00:40:57.906646 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890906 4421 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 00:40:57.906646 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890908 4421 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 00:40:57.906646 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890911 4421 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 00:40:57.906646 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890913 4421 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 00:40:57.906646 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890916 4421 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 00:40:57.906646 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890920 4421 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 00:40:57.906646 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890923 4421 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 00:40:57.906646 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890925 4421 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 00:40:57.906646 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890928 4421 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 00:40:57.906646 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890931 4421 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 00:40:57.906646 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890933 4421 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 00:40:57.906646 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890938 4421 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 00:40:57.906646 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890941 4421 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 00:40:57.907257 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890943 4421 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 00:40:57.907257 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890946 4421 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 00:40:57.907257 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890949 4421 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 00:40:57.907257 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890952 4421 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 00:40:57.907257 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890954 4421 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 00:40:57.907257 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890959 4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 00:40:57.907257 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890963 4421 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 00:40:57.907257 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890967 4421 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 00:40:57.907257 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890971 4421 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 00:40:57.907257 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890976 4421 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 00:40:57.907257 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890980 4421 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 00:40:57.907257 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890984 4421 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 00:40:57.907257 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890990 4421 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 00:40:57.907257 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890995 4421 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 00:40:57.907257 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.890999 4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 00:40:57.907257 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.891003 4421 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 00:40:57.907257 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.891007 4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 00:40:57.907257 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.891011 4421 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 00:40:57.907257 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.891016 4421 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 00:40:57.907257 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.891020 4421 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 00:40:57.907962 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.891023 4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 00:40:57.907962 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.891025 4421 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 00:40:57.907962 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.891028 4421 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 00:40:57.907962 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.891032 4421 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 00:40:57.907962 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.891039 4421 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 00:40:57.907962 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.891042 4421 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 00:40:57.907962 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.891045 4421 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 00:40:57.907962 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.891048 4421 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 00:40:57.907962 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.891051 4421 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 00:40:57.907962 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.891054 4421 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 00:40:57.907962 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.891057 4421 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 00:40:57.907962 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.891060 4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 00:40:57.907962 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.891064 4421 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 00:40:57.907962 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.891067 4421 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 00:40:57.907962 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.891069 4421 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 00:40:57.907962 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.891072 4421 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 00:40:57.907962 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.891075 4421 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 00:40:57.907962 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.891077 4421 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 00:40:57.908684 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.891080 4421 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 00:40:57.908684 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.891083 4421 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 00:40:57.908684 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.891086 4421 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 00:40:57.908684 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.891089 4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 00:40:57.908684 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.891091 4421 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 00:40:57.908684 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.891094 4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 00:40:57.908684 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.891096 4421 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 00:40:57.908684 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.891099 4421 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 00:40:57.908684 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.891110 4421 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 00:40:57.908684 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.891113 4421 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 00:40:57.908684 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.891116 4421 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 00:40:57.908684 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.891118 4421 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 00:40:57.908684 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.891121 4421 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 00:40:57.908684 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.891123 4421 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 00:40:57.908684 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.891126 4421 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 00:40:57.908684 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.891129 4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 00:40:57.908684 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.891131 4421 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 00:40:57.908684 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.891134 4421 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 00:40:57.908684 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.891138 4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 00:40:57.909261 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.891141 4421 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 00:40:57.909261 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.891144 4421 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 00:40:57.909261 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.891148 4421 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 00:40:57.909261 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.891153 4421 feature_gate.go:386] unrecognized feature gate: Example Nov 03 00:40:57.909261 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.891158 4421 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 00:40:57.909261 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.891166 4421 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 00:40:57.909261 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:57.891175 4421 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 00:40:57.909261 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.891240 4421 server.go:957] "Client rotation is on, will bootstrap in background" Nov 03 00:40:57.909261 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.892427 4421 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Nov 03 00:40:57.909261 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.892484 4421 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Nov 03 00:40:57.909261 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.892678 4421 server.go:1014] "Starting client certificate rotation" Nov 03 00:40:57.909261 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.892840 4421 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 00:40:57.909261 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.892905 4421 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 00:35:17 +0000 UTC" deadline="2028-08-12 10:55:47.05799585 +0000 UTC" Nov 03 00:40:57.909261 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.892948 4421 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15562h14m49.165051342s" Nov 03 00:40:57.909261 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.893317 4421 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 00:40:57.909261 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.894639 4421 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 00:40:57.909261 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.896538 4421 log.go:25] "Validated CRI v1 runtime API" Nov 03 00:40:57.909891 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.898522 4421 log.go:25] "Validated CRI v1 image API" Nov 03 00:40:57.909891 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.899408 4421 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 03 00:40:57.909891 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.901516 4421 fs.go:135] Filesystem UUIDs: map[66ae21c0-dbe3-4b2c-bc86-d0ecb8c85e81:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 9947e40b-3366-48ec-a10d-96375a6b0879:/dev/nvme0n1p3] Nov 03 00:40:57.910018 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.901541 4421 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0c5dc40a50488cf5e29018d41a984f698d723f309f28c83c293d048ad0ba09dd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0c5dc40a50488cf5e29018d41a984f698d723f309f28c83c293d048ad0ba09dd/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6cda35d070db858a1d3a28f63c2c222c2f5ab534fabe1c579c5ec311ad94f1c5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6cda35d070db858a1d3a28f63c2c222c2f5ab534fabe1c579c5ec311ad94f1c5/userdata/shm major:0 minor:203 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/70058e8eadc84f83ec6ce1887b5a390c133578193406ac6dfc14abc706dfb8d9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/70058e8eadc84f83ec6ce1887b5a390c133578193406ac6dfc14abc706dfb8d9/userdata/shm major:0 minor:224 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7fe87d05e5214377cd693e21d77c16e252b8ad1eff614e3bd29a94d0463a701b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7fe87d05e5214377cd693e21d77c16e252b8ad1eff614e3bd29a94d0463a701b/userdata/shm major:0 minor:96 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8da6814d8364a7135cd62e8752fc9462c556c73a806f3d2c8968892be5ee923e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8da6814d8364a7135cd62e8752fc9462c556c73a806f3d2c8968892be5ee923e/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b0df856bc6a0c41be88663d8b0a01bf05381d52cb6db6b6e164570d4e5f5e555/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b0df856bc6a0c41be88663d8b0a01bf05381d52cb6db6b6e164570d4e5f5e555/userdata/shm major:0 minor:219 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b4b5234880d4312f754619ef4e1d8f7b3b0cc1d1de0d269baa813a9d050048d7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b4b5234880d4312f754619ef4e1d8f7b3b0cc1d1de0d269baa813a9d050048d7/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b70feb1275eaf5ff25046474ea2e0d8794dfc882f99b257fb699089bc3010cab/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b70feb1275eaf5ff25046474ea2e0d8794dfc882f99b257fb699089bc3010cab/userdata/shm major:0 minor:215 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/baf995ad5d0a338d5cab169fe531fe16fa1263a06c4429391a64689fce408d2a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/baf995ad5d0a338d5cab169fe531fe16fa1263a06c4429391a64689fce408d2a/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d00486ba1df0fa7cdbd2b3470a55384be63a1cb897beeb9e8e00b20c30b94823/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d00486ba1df0fa7cdbd2b3470a55384be63a1cb897beeb9e8e00b20c30b94823/userdata/shm major:0 minor:88 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d26b7bbc18a12c91a4663e8aabaac3278b33104993a34ea7f7e9c75dfeb63adb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d26b7bbc18a12c91a4663e8aabaac3278b33104993a34ea7f7e9c75dfeb63adb/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/eaba454a940bc816d97e79a2e397bdb6ba69fdf2c1c1691c7e89f4f0832939e7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/eaba454a940bc816d97e79a2e397bdb6ba69fdf2c1c1691c7e89f4f0832939e7/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ec1b91b125fcf362dae30d04e6bd5f32dc1c657adf0aebc80e5ae74c4d7c32e0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ec1b91b125fcf362dae30d04e6bd5f32dc1c657adf0aebc80e5ae74c4d7c32e0/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f9e348c6d21aff71e0966f83a17915b5fb972bb78f9804902ac570b8049d91d3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f9e348c6d21aff71e0966f83a17915b5fb972bb78f9804902ac570b8049d91d3/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/174d7fa2-7163-47a8-a047-8727cdb3cb9f/volumes/kubernetes.io~projected/kube-api-access-c5fls:{mountpoint:/var/lib/kubelet/pods/174d7fa2-7163-47a8-a047-8727cdb3cb9f/volumes/kubernetes.io~projected/kube-api-access-c5fls major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/18afc5dd-bc60-4794-91ab-b4019be56a61/volumes/kubernetes.io~projected/kube-api-access-wsbnr:{mountpoint:/var/lib/kubelet/pods/18afc5dd-bc60-4794-91ab-b4019be56a61/volumes/kubernetes.io~projected/kube-api-access-wsbnr major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/29de0f7e-9c96-46dd-8fec-434e3689a047/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/29de0f7e-9c96-46dd-8fec-434e3689a047/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/29de0f7e-9c96-46dd-8fec-434e3689a047/volumes/kubernetes.io~projected/kube-api-access-45qm2:{mountpoint:/var/lib/kubelet/pods/29de0f7e-9c96-46dd-8fec-434e3689a047/volumes/kubernetes.io~projected/kube-api-access-45qm2 major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/29de0f7e-9c96-46dd-8fec-434e3689a047/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/29de0f7e-9c96-46dd-8fec-434e3689a047/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3dcbff01-09fb-458e-b3a5-594d6b271133/volumes/kubernetes.io~projected/kube-api-access-dt25z:{mountpoint:/var/lib/kubelet/pods/3dcbff01-09fb-458e-b3a5-594d6b271133/volumes/kubernetes.io~projected/kube-api-access-dt25z major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/476ff8a0-f214-4e5e-aa90-8a505974f963/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/476ff8a0-f214-4e5e-aa90-8a505974f963/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/476ff8a0-f214-4e5e-aa90-8a505974f963/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/476ff8a0-f214-4e5e-aa90-8a505974f963/volumes/kubernetes.io~empty-dir/tmp major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/476ff8a0-f214-4e5e-aa90-8a505974f963/volumes/kubernetes.io~projected/kube-api-access-lqbgf:{mountpoint:/var/lib/kubelet/pods/476ff8a0-f214-4e5e-aa90-8a505974f963/volumes/kubernetes.io~projected/kube-api-access-lqbgf major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5844f274-ad38-42a3-9900-b051b01c9953/volumes/kubernetes.io~projected/kube-api-access-qn5dn:{mountpoint:/var/lib/kubelet/pods/5844f274-ad38-42a3-9900-b051b01c9953/volumes/kubernetes.io~projected/kube-api-access-qn5dn major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6ab5d408-02a2-4284-bf59-5827c6f84643/volumes/kubernetes.io~projected/kube-api-access-2n98j:{mountpoint:/var/lib/kubelet/pods/6ab5d408-02a2-4284-bf59-5827c6f84643/volumes/kubernetes.io~projected/kube-api-access-2n98j major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/76ccef27-d056-402e-a7e1-ee524d63bf35/volumes/kubernetes.io~projected/kube-api-access-rjg6m:{mountpoint:/var/lib/kubelet/pods/76ccef27-d056-402e-a7e1-ee524d63bf35/volumes/kubernetes.io~projected/kube-api-access-rjg6m major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/76ccef27-d056-402e-a7e1-ee524d63bf35/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/76ccef27-d056-402e-a7e1-ee524d63bf35/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/92341144-23c4-4cf0-a0c8-26e1891397e9/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/92341144-23c4-4cf0-a0c8-26e1891397e9/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9c505674-e66b-4573-bbf5-7b6acf6684e6/volumes/kubernetes.io~projected/kube-api-access-4svt5:{mountpoint:/var/lib/kubelet/pods/9c505674-e66b-4573-bbf5-7b6acf6684e6/volumes/kubernetes.io~projected/kube-api-access-4svt5 major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9c505674-e66b-4573-bbf5-7b6acf6684e6/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/9c505674-e66b-4573-bbf5-7b6acf6684e6/volumes/kubernetes.io~secret/metrics-tls major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b2d13929-be23-4396-8445-c3eee12e9b9d/volumes/kubernetes.io~projected/kube-api-access-h4c5b:{mountpoint:/var/lib/kubelet/pods/b2d13929-be23-4396-8445-c3eee12e9b9d/volumes/kubernetes.io~projected/kube-api-access-h4c5b major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b6abcc85-61c0-42ea-b868-0d71a46a4514/volumes/kubernetes.io~projected/kube-api-access-lzb4d:{mountpoint:/var/lib/kubelet/pods/b6abcc85-61c0-42ea-b868-0d71a46a4514/volumes/kubernetes.io~projected/kube-api-access-lzb4d major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bac578ef-a88b-4a74-9a65-bebc8ea1d00e/volumes/kubernetes.io~projected/kube-api-access-86gz6:{mountpoint:/var/lib/kubelet/pods/bac578ef-a88b-4a74-9a65-bebc8ea1d00e/volumes/kubernetes.io~projected/kube-api-access-86gz6 major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cd47e545-4089-44ad-bc4d-0545061c7b11/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/cd47e545-4089-44ad-bc4d-0545061c7b11/volumes/kubernetes.io~secret/agent-certs major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e7f1faa5-fc0e-4f46-9208-b1e58f50fcf9/volumes/kubernetes.io~projected/kube-api-access-8t7dh:{mountpoint:/var/lib/kubelet/pods/e7f1faa5-fc0e-4f46-9208-b1e58f50fcf9/volumes/kubernetes.io~projected/kube-api-access-8t7dh major:0 minor:66 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/4f336df4f362ae8285f9c084d6a035e47544aca8a46534a18c0dca95ffaafeff/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/f724e86d588f4d23554390a7e4244d976edfd7672b7b0bf42e5031e9e32750ab/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/1dd547dd2d363e1bfd1313be56e9224e4774a189345e8b05b9cbfae50760d8df/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/25dc9ee0205e2a91e86be850c9fc9362456227c4d34338ed04efa6fec1210c52/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/276b0acdfde902e290856d552ea1e26829f6d0649e568a00c08b2314e20aa68b/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/4afce0f486f7002aac1f399c6c205cd115916aa97b3febb59c6001ebf1de06f3/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/70980df1e3d4ef2822cc561fd0247fe3369f80779f5e9d470931bf06fe8f8ae2/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/9662f35f788bcdf1b88128b58f0d058f92745bccd41705744d0ef5fdab81e6ed/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/54d70fab7993b3d2732375957aae370339fdac7301f4156183643f292db39f45/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/75e59217568a1904f531a009233f99e399fb641feb37bdd2324ae820c0336911/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-142:{mountpoint:/var/lib/containers/storage/overlay/f6d2227cf097416d36dca94c73fce7028b0f2aeb215cd23b49080183e6686927/merged major:0 minor:142 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/e2bbec91230ef8d4e80fe013a2555c184da83509688ca3a136ce16c769f86972/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/9575420f730f621cc7c220075526231c3fb9a9b81b89fb06dc5498cd000e5bc1/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/c9b68b53f1b3d3e54d81891a3ba3cb58fa3625b463520b9c91a2944a581166c6/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/203952692b710ef6cdd48036655903569e96c32d3c1f6468f090f1d47b95ddfc/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/5bb1b4b649fc48b49ce09579db3e69ff52142cc7bfba47fb2c4ddae6bdfa84f6/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/12e574389ae4125d450381e140f2391195898ae68589d0c9bb94c15af588d8b3/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/db7682f6daa61ec342cdaeaccfdacff047adee379766f1e819b0b9a2b3df29bd/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/2f403dddb593dbb53dd1f6027197a7f2622feec201057cc875b5513b27c7d44b/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/d97c5455a3d74217b46dd9c0658813f0f541a737d5a9ef17b08676f7f4eee29a/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/ae85ceae8aea72a9830b71e2c22cd6e7e0613f78f923b59663a5c55c18a8e80b/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-205:{mountpoint:/var/lib/containers/storage/overlay/3c24619651314c321be1d24d58b70c7d13ec1d5cf5cd60b11dcdd0535d7f181f/merged major:0 minor:205 fsType:overlay blockSize:0} overlay_0-207:{mountpoint:/var/lib/containers/storage/overlay/130b1fb17fbe4c0cea766cfb68a27e7eb486e137810158e65adec76b994a2a1f/merged major:0 minor:207 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/0557b66ce06c884592b3be68f994b5f06636bcd9836c677de8e9e21a4080e01b/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-221:{mountpoint:/var/lib/containers/storage/overlay/0d01d59bf5687873c8fa22bec4c5a302034bfb74f8791985f9290e0e1765cc8e/merged major:0 minor:221 fsType:overlay blockSize:0} overlay_0-226:{mountpoint:/var/lib/containers/storage/overlay/b16a8423673ca0519327d9814cc0860f6a6a838d14f836f567551228cf10912e/merged major:0 minor:226 fsType:overlay blockSize:0} overlay_0-228:{mountpoint:/var/lib/containers/storage/overlay/f3b0a9348f0518a5c30040fb614b92bb1438d1a691811cda799ab96bd472be59/merged major:0 minor:228 fsType:overlay blockSize:0} overlay_0-230:{mountpoint:/var/lib/containers/storage/overlay/d6624d0a9576d77e1b8b4e26b28dbab1889703fe612c37d4b1413ccda8656172/merged major:0 minor:230 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/3ccbd65c55d679b4cffbdc644f94408a48e0ba8701ebd1df9f99ac6c0e070dbe/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/9733e261d44048b6911755173619cbac5e9e45607ed81fc401514457ec93c8a2/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/e03b08101caff41c73b95995dca202d766c746c00dccd866f7e57368e54515ce/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/95531677174ae7f8e10c3ec56d4849af4e62775c76802b7f0444c717a7d25d29/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/0089bcfc6bfb4938a319966c2cb402bd1279e2dbfd466ad8ad71bb2bcc023e38/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/f4e6722179ae8d6b269581cead407488f7a1fea8f51aaa436c9fb3efc5481648/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/9da27f6c4f74af46a5848543688b36f178d1d71f0f0dc4e0931764f6a0b63f16/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/04af0b45861407c91ca61fb518a70192570be7aea10640af45af4a7514bd1820/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/5c5385d851a6e5cad92ae8c17d058d19adcfbeb5e217aecbc5453fa359aa04f6/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-97:{mountpoint:/var/lib/containers/storage/overlay/6540fabc4a2c1f28cd11215ca8428fa09a6fc34143b2dec59bdca672fe0e16e8/merged major:0 minor:97 fsType:overlay blockSize:0}] Nov 03 00:40:57.911121 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.910845 4421 manager.go:217] Machine: {Timestamp:2026-11-03 00:40:57.910243646 +0000 UTC m=+0.081246733 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100178 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec28b1067fd68b362a334eddcbc934b3 SystemUUID:ec28b106-7fd6-8b36-2a33-4eddcbc934b3 BootID:ea9d839b-74ae-4252-b09f-dd36e6790996 Filesystems:[{Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/29de0f7e-9c96-46dd-8fec-434e3689a047/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:55 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/18afc5dd-bc60-4794-91ab-b4019be56a61/volumes/kubernetes.io~projected/kube-api-access-wsbnr DeviceMajor:0 DeviceMinor:57 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b2d13929-be23-4396-8445-c3eee12e9b9d/volumes/kubernetes.io~projected/kube-api-access-h4c5b DeviceMajor:0 DeviceMinor:61 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/476ff8a0-f214-4e5e-aa90-8a505974f963/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:63 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/eaba454a940bc816d97e79a2e397bdb6ba69fdf2c1c1691c7e89f4f0832939e7/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cd47e545-4089-44ad-bc4d-0545061c7b11/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:64 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/29de0f7e-9c96-46dd-8fec-434e3689a047/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/92341144-23c4-4cf0-a0c8-26e1891397e9/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:209 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3dcbff01-09fb-458e-b3a5-594d6b271133/volumes/kubernetes.io~projected/kube-api-access-dt25z DeviceMajor:0 DeviceMinor:214 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/70058e8eadc84f83ec6ce1887b5a390c133578193406ac6dfc14abc706dfb8d9/userdata/shm DeviceMajor:0 DeviceMinor:224 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bac578ef-a88b-4a74-9a65-bebc8ea1d00e/volumes/kubernetes.io~projected/kube-api-access-86gz6 DeviceMajor:0 DeviceMinor:197 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-207 DeviceMajor:0 DeviceMinor:207 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b70feb1275eaf5ff25046474ea2e0d8794dfc882f99b257fb699089bc3010cab/userdata/shm DeviceMajor:0 DeviceMinor:215 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/174d7fa2-7163-47a8-a047-8727cdb3cb9f/volumes/kubernetes.io~projected/kube-api-access-c5fls DeviceMajor:0 DeviceMinor:56 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/76ccef27-d056-402e-a7e1-ee524d63bf35/volumes/kubernetes.io~projected/kube-api-access-rjg6m DeviceMajor:0 DeviceMinor:202 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f9e348c6d21aff71e0966f83a17915b5fb972bb78f9804902ac570b8049d91d3/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b6abcc85-61c0-42ea-b868-0d71a46a4514/volumes/kubernetes.io~projected/kube-api-access-lzb4d DeviceMajor:0 DeviceMinor:60 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b4b5234880d4312f754619ef4e1d8f7b3b0cc1d1de0d269baa813a9d050048d7/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9c505674-e66b-4573-bbf5-7b6acf6684e6/volumes/kubernetes.io~projected/kube-api-access-4svt5 DeviceMajor:0 DeviceMinor:201 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/476ff8a0-f214-4e5e-aa90-8a505974f963/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:65 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7fe87d05e5214377cd693e21d77c16e252b8ad1eff614e3bd29a94d0463a701b/userdata/shm DeviceMajor:0 DeviceMinor:96 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d00486ba1df0fa7cdbd2b3470a55384be63a1cb897beeb9e8e00b20c30b94823/userdata/shm DeviceMajor:0 DeviceMinor:88 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-205 DeviceMajor:0 DeviceMinor:205 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e7f1faa5-fc0e-4f46-9208-b1e58f50fcf9/volumes/kubernetes.io~projected/kube-api-access-8t7dh DeviceMajor:0 DeviceMinor:66 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-142 DeviceMajor:0 DeviceMinor:142 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/baf995ad5d0a338d5cab169fe531fe16fa1263a06c4429391a64689fce408d2a/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d26b7bbc18a12c91a4663e8aabaac3278b33104993a34ea7f7e9c75dfeb63adb/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/5844f274-ad38-42a3-9900-b051b01c9953/volumes/kubernetes.io~projected/kube-api-access-qn5dn DeviceMajor:0 DeviceMinor:59 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-226 DeviceMajor:0 DeviceMinor:226 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/76ccef27-d056-402e-a7e1-ee524d63bf35/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-221 DeviceMajor:0 DeviceMinor:221 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-230 DeviceMajor:0 DeviceMinor:230 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/476ff8a0-f214-4e5e-aa90-8a505974f963/volumes/kubernetes.io~projected/kube-api-access-lqbgf DeviceMajor:0 DeviceMinor:67 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ec1b91b125fcf362dae30d04e6bd5f32dc1c657adf0aebc80e5ae74c4d7c32e0/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0c5dc40a50488cf5e29018d41a984f698d723f309f28c83c293d048ad0ba09dd/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8da6814d8364a7135cd62e8752fc9462c556c73a806f3d2c8968892be5ee923e/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-97 DeviceMajor:0 DeviceMinor:97 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6cda35d070db858a1d3a28f63c2c222c2f5ab534fabe1c579c5ec311ad94f1c5/userdata/shm DeviceMajor:0 DeviceMinor:203 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b0df856bc6a0c41be88663d8b0a01bf05381d52cb6db6b6e164570d4e5f5e555/userdata/shm DeviceMajor:0 DeviceMinor:219 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-228 DeviceMajor:0 DeviceMinor:228 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/29de0f7e-9c96-46dd-8fec-434e3689a047/volumes/kubernetes.io~projected/kube-api-access-45qm2 DeviceMajor:0 DeviceMinor:62 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6ab5d408-02a2-4284-bf59-5827c6f84643/volumes/kubernetes.io~projected/kube-api-access-2n98j DeviceMajor:0 DeviceMinor:58 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9c505674-e66b-4573-bbf5-7b6acf6684e6/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:223 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:6cda35d070db858 MacAddress:1e:34:b3:1d:fb:9b Speed:10000 Mtu:8901} {Name:70058e8eadc84f8 MacAddress:fe:7b:d1:a9:04:8c Speed:10000 Mtu:8901} {Name:b0df856bc6a0c41 MacAddress:82:52:54:3c:8a:7e Speed:10000 Mtu:8901} {Name:b70feb1275eaf5f MacAddress:d2:ac:f8:cf:6a:9c Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:ce:16:47:ef Speed:0 Mtu:9001} {Name:br-int MacAddress:f2:f2:59:94:f0:39 Speed:0 Mtu:8901} {Name:ens5 MacAddress:02:ff:ce:16:47:ef Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:9e:c0:89:18:06:63 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:a6:39:2f:1b:f7:08 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 03 00:40:57.911380 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.911126 4421 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 03 00:40:57.911380 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.911181 4421 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 03 00:40:57.911642 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.911606 4421 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 03 00:40:57.911826 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.911644 4421 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-217.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 03 00:40:57.911826 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.911829 4421 topology_manager.go:143] "Creating topology manager with none policy" Nov 03 00:40:57.911826 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.911840 4421 container_manager_linux.go:311] "Creating device plugin manager" Nov 03 00:40:57.912018 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.911855 4421 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 00:40:57.912018 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.911875 4421 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 00:40:57.912018 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.911996 4421 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 03 00:40:57.912211 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.912198 4421 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 03 00:40:57.912366 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.912355 4421 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 03 00:40:57.912404 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.912400 4421 kubelet.go:493] "Attempting to sync node with API server" Nov 03 00:40:57.912432 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.912418 4421 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 03 00:40:57.912432 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.912430 4421 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 03 00:40:57.912510 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.912439 4421 kubelet.go:405] "Adding apiserver pod source" Nov 03 00:40:57.912510 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.912447 4421 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 03 00:40:57.913188 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.913169 4421 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 03 00:40:57.913188 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.913189 4421 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 00:40:57.913938 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.913918 4421 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 03 00:40:57.922390 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.922374 4421 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 03 00:40:57.922390 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.922389 4421 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 00:40:57.922560 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.922545 4421 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Nov 03 00:40:57.922830 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.922807 4421 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 03 00:40:57.922931 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.922850 4421 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 03 00:40:57.923002 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.922991 4421 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 03 00:40:57.923055 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.923012 4421 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 03 00:40:57.923055 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.923022 4421 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 03 00:40:57.923055 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.923033 4421 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 03 00:40:57.923055 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.923043 4421 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 03 00:40:57.923055 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.923052 4421 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 03 00:40:57.923372 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.923062 4421 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 03 00:40:57.923372 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.923071 4421 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 03 00:40:57.923372 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.923083 4421 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 03 00:40:57.923372 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.923103 4421 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 03 00:40:57.923372 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.923125 4421 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 03 00:40:57.923372 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.923150 4421 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 03 00:40:57.923372 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.923186 4421 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 03 00:40:57.923372 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.923197 4421 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 03 00:40:57.927515 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.927497 4421 server.go:1263] "Started kubelet" Nov 03 00:40:57.927975 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.927918 4421 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 03 00:40:57.928063 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.927998 4421 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 03 00:40:57.928133 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.928071 4421 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 03 00:40:57.928261 ip-10-0-133-217 systemd[1]: Started Kubernetes Kubelet. Nov 03 00:40:57.931425 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.928772 4421 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 03 00:40:57.936372 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.936350 4421 server.go:317] "Adding debug handlers to kubelet server" Nov 03 00:40:57.939250 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.937766 4421 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 03 00:40:57.939250 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.938128 4421 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 03 00:40:57.939250 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.938309 4421 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 00:40:57.939250 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.938466 4421 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 03 00:40:57.939250 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.938484 4421 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 03 00:40:57.939250 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.938558 4421 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 03 00:40:57.939250 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.938578 4421 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 00:40:57.939250 ip-10-0-133-217 kubenswrapper[4421]: E1103 00:40:57.938627 4421 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-217.us-east-2.compute.internal\" not found" Nov 03 00:40:57.939250 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.938921 4421 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 00:35:20 +0000 UTC" deadline="2028-08-05 07:56:36.899782494 +0000 UTC" Nov 03 00:40:57.939250 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.938943 4421 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15391h15m38.960843928s" Nov 03 00:40:57.939250 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.939174 4421 factory.go:55] Registering systemd factory Nov 03 00:40:57.939250 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.939189 4421 factory.go:223] Registration of the systemd container factory successfully Nov 03 00:40:57.941831 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.939404 4421 factory.go:153] Registering CRI-O factory Nov 03 00:40:57.941831 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.939422 4421 factory.go:223] Registration of the crio container factory successfully Nov 03 00:40:57.941831 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.939479 4421 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 03 00:40:57.941831 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.939500 4421 factory.go:103] Registering Raw factory Nov 03 00:40:57.941831 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.939516 4421 manager.go:1199] Started watching for new ooms in manager Nov 03 00:40:57.941831 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.939867 4421 manager.go:319] Starting recovery of all containers Nov 03 00:40:57.941831 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.939976 4421 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="29de0f7e-9c96-46dd-8fec-434e3689a047" volumeName="kubernetes.io/configmap/29de0f7e-9c96-46dd-8fec-434e3689a047-ovnkube-script-lib" seLinuxMountContext="" Nov 03 00:40:57.941831 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.940014 4421 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="29de0f7e-9c96-46dd-8fec-434e3689a047" volumeName="kubernetes.io/projected/29de0f7e-9c96-46dd-8fec-434e3689a047-kube-api-access-45qm2" seLinuxMountContext="" Nov 03 00:40:57.941831 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.940028 4421 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="92341144-23c4-4cf0-a0c8-26e1891397e9" volumeName="kubernetes.io/secret/92341144-23c4-4cf0-a0c8-26e1891397e9-original-pull-secret" seLinuxMountContext="" Nov 03 00:40:57.941831 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.940040 4421 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9c505674-e66b-4573-bbf5-7b6acf6684e6" volumeName="kubernetes.io/secret/9c505674-e66b-4573-bbf5-7b6acf6684e6-metrics-tls" seLinuxMountContext="" Nov 03 00:40:57.941831 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.940052 4421 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bac578ef-a88b-4a74-9a65-bebc8ea1d00e" volumeName="kubernetes.io/configmap/bac578ef-a88b-4a74-9a65-bebc8ea1d00e-kube-rbac-proxy-cm" seLinuxMountContext="" Nov 03 00:40:57.941831 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.940063 4421 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e7f1faa5-fc0e-4f46-9208-b1e58f50fcf9" volumeName="kubernetes.io/projected/e7f1faa5-fc0e-4f46-9208-b1e58f50fcf9-kube-api-access-8t7dh" seLinuxMountContext="" Nov 03 00:40:57.941831 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.940093 4421 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="174d7fa2-7163-47a8-a047-8727cdb3cb9f" volumeName="kubernetes.io/configmap/174d7fa2-7163-47a8-a047-8727cdb3cb9f-iptables-alerter-script" seLinuxMountContext="" Nov 03 00:40:57.941831 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.940105 4421 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="476ff8a0-f214-4e5e-aa90-8a505974f963" volumeName="kubernetes.io/empty-dir/476ff8a0-f214-4e5e-aa90-8a505974f963-etc-tuned" seLinuxMountContext="" Nov 03 00:40:57.941831 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.940119 4421 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="76ccef27-d056-402e-a7e1-ee524d63bf35" volumeName="kubernetes.io/projected/76ccef27-d056-402e-a7e1-ee524d63bf35-kube-api-access-rjg6m" seLinuxMountContext="" Nov 03 00:40:57.941831 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.940131 4421 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e7f1faa5-fc0e-4f46-9208-b1e58f50fcf9" volumeName="kubernetes.io/empty-dir/e7f1faa5-fc0e-4f46-9208-b1e58f50fcf9-tmp-dir" seLinuxMountContext="" Nov 03 00:40:57.941831 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.940143 4421 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="18afc5dd-bc60-4794-91ab-b4019be56a61" volumeName="kubernetes.io/projected/18afc5dd-bc60-4794-91ab-b4019be56a61-kube-api-access-wsbnr" seLinuxMountContext="" Nov 03 00:40:57.941831 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.940157 4421 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="476ff8a0-f214-4e5e-aa90-8a505974f963" volumeName="kubernetes.io/projected/476ff8a0-f214-4e5e-aa90-8a505974f963-kube-api-access-lqbgf" seLinuxMountContext="" Nov 03 00:40:57.941831 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.940169 4421 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bac578ef-a88b-4a74-9a65-bebc8ea1d00e" volumeName="kubernetes.io/projected/bac578ef-a88b-4a74-9a65-bebc8ea1d00e-kube-api-access-86gz6" seLinuxMountContext="" Nov 03 00:40:57.941831 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.940199 4421 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b2d13929-be23-4396-8445-c3eee12e9b9d" volumeName="kubernetes.io/projected/b2d13929-be23-4396-8445-c3eee12e9b9d-kube-api-access-h4c5b" seLinuxMountContext="" Nov 03 00:40:57.941831 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.941302 4421 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="174d7fa2-7163-47a8-a047-8727cdb3cb9f" volumeName="kubernetes.io/projected/174d7fa2-7163-47a8-a047-8727cdb3cb9f-kube-api-access-c5fls" seLinuxMountContext="" Nov 03 00:40:57.941831 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.941329 4421 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="29de0f7e-9c96-46dd-8fec-434e3689a047" volumeName="kubernetes.io/configmap/29de0f7e-9c96-46dd-8fec-434e3689a047-ovnkube-config" seLinuxMountContext="" Nov 03 00:40:57.941831 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.941364 4421 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="29de0f7e-9c96-46dd-8fec-434e3689a047" volumeName="kubernetes.io/secret/29de0f7e-9c96-46dd-8fec-434e3689a047-ovn-node-metrics-cert" seLinuxMountContext="" Nov 03 00:40:57.941831 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.941379 4421 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3dcbff01-09fb-458e-b3a5-594d6b271133" volumeName="kubernetes.io/projected/3dcbff01-09fb-458e-b3a5-594d6b271133-kube-api-access-dt25z" seLinuxMountContext="" Nov 03 00:40:57.941831 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.941393 4421 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="476ff8a0-f214-4e5e-aa90-8a505974f963" volumeName="kubernetes.io/empty-dir/476ff8a0-f214-4e5e-aa90-8a505974f963-tmp" seLinuxMountContext="" Nov 03 00:40:57.941831 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.941407 4421 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="76ccef27-d056-402e-a7e1-ee524d63bf35" volumeName="kubernetes.io/secret/76ccef27-d056-402e-a7e1-ee524d63bf35-cert" seLinuxMountContext="" Nov 03 00:40:57.941831 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.941454 4421 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bac578ef-a88b-4a74-9a65-bebc8ea1d00e" volumeName="kubernetes.io/empty-dir/bac578ef-a88b-4a74-9a65-bebc8ea1d00e-data-volume" seLinuxMountContext="" Nov 03 00:40:57.941831 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.941468 4421 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="18afc5dd-bc60-4794-91ab-b4019be56a61" volumeName="kubernetes.io/configmap/18afc5dd-bc60-4794-91ab-b4019be56a61-cni-sysctl-allowlist" seLinuxMountContext="" Nov 03 00:40:57.941831 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.941481 4421 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9c505674-e66b-4573-bbf5-7b6acf6684e6" volumeName="kubernetes.io/empty-dir/9c505674-e66b-4573-bbf5-7b6acf6684e6-tmp-dir" seLinuxMountContext="" Nov 03 00:40:57.941831 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.941521 4421 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cd47e545-4089-44ad-bc4d-0545061c7b11" volumeName="kubernetes.io/secret/cd47e545-4089-44ad-bc4d-0545061c7b11-agent-certs" seLinuxMountContext="" Nov 03 00:40:57.941831 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.941535 4421 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="29de0f7e-9c96-46dd-8fec-434e3689a047" volumeName="kubernetes.io/configmap/29de0f7e-9c96-46dd-8fec-434e3689a047-env-overrides" seLinuxMountContext="" Nov 03 00:40:57.941831 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.941559 4421 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5844f274-ad38-42a3-9900-b051b01c9953" volumeName="kubernetes.io/projected/5844f274-ad38-42a3-9900-b051b01c9953-kube-api-access-qn5dn" seLinuxMountContext="" Nov 03 00:40:57.941831 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.941577 4421 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b6abcc85-61c0-42ea-b868-0d71a46a4514" volumeName="kubernetes.io/projected/b6abcc85-61c0-42ea-b868-0d71a46a4514-kube-api-access-lzb4d" seLinuxMountContext="" Nov 03 00:40:57.941831 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.941599 4421 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="18afc5dd-bc60-4794-91ab-b4019be56a61" volumeName="kubernetes.io/configmap/18afc5dd-bc60-4794-91ab-b4019be56a61-whereabouts-flatfile-configmap" seLinuxMountContext="" Nov 03 00:40:57.941831 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.941613 4421 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6ab5d408-02a2-4284-bf59-5827c6f84643" volumeName="kubernetes.io/projected/6ab5d408-02a2-4284-bf59-5827c6f84643-kube-api-access-2n98j" seLinuxMountContext="" Nov 03 00:40:57.941831 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.941633 4421 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9c505674-e66b-4573-bbf5-7b6acf6684e6" volumeName="kubernetes.io/projected/9c505674-e66b-4573-bbf5-7b6acf6684e6-kube-api-access-4svt5" seLinuxMountContext="" Nov 03 00:40:57.941831 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.941641 4421 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b2d13929-be23-4396-8445-c3eee12e9b9d" volumeName="kubernetes.io/configmap/b2d13929-be23-4396-8445-c3eee12e9b9d-multus-daemon-config" seLinuxMountContext="" Nov 03 00:40:57.941831 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.941648 4421 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="18afc5dd-bc60-4794-91ab-b4019be56a61" volumeName="kubernetes.io/configmap/18afc5dd-bc60-4794-91ab-b4019be56a61-cni-binary-copy" seLinuxMountContext="" Nov 03 00:40:57.941831 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.941657 4421 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5844f274-ad38-42a3-9900-b051b01c9953" volumeName="kubernetes.io/configmap/5844f274-ad38-42a3-9900-b051b01c9953-serviceca" seLinuxMountContext="" Nov 03 00:40:57.941831 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.941676 4421 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9c505674-e66b-4573-bbf5-7b6acf6684e6" volumeName="kubernetes.io/configmap/9c505674-e66b-4573-bbf5-7b6acf6684e6-config-volume" seLinuxMountContext="" Nov 03 00:40:57.941831 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.941688 4421 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b2d13929-be23-4396-8445-c3eee12e9b9d" volumeName="kubernetes.io/configmap/b2d13929-be23-4396-8445-c3eee12e9b9d-cni-binary-copy" seLinuxMountContext="" Nov 03 00:40:57.941831 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.941696 4421 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cd47e545-4089-44ad-bc4d-0545061c7b11" volumeName="kubernetes.io/configmap/cd47e545-4089-44ad-bc4d-0545061c7b11-konnectivity-ca" seLinuxMountContext="" Nov 03 00:40:57.941831 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.941705 4421 reconstruct.go:97] "Volume reconstruction finished" Nov 03 00:40:57.941831 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.941710 4421 reconciler.go:29] "Reconciler: start to sync state" Nov 03 00:40:57.943828 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.943069 4421 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 00:40:57.946239 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.946219 4421 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Nov 03 00:40:57.966594 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.966549 4421 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 03 00:40:57.970534 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.970502 4421 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 03 00:40:57.970534 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.970532 4421 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 03 00:40:57.970739 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.970553 4421 kubelet.go:2521] "Starting kubelet main sync loop" Nov 03 00:40:57.970739 ip-10-0-133-217 kubenswrapper[4421]: E1103 00:40:57.970594 4421 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 03 00:40:57.976417 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.976394 4421 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 00:40:57.990756 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.990716 4421 generic.go:356] "Generic (PLEG): container finished" podID="fdeeb94bfd911d86834e9d29bdbbadab" containerID="00557970e2b0aac26fb1fe305bc704b99b3e296fcc7310e2a74186b8102140c1" exitCode=0 Nov 03 00:40:57.998940 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.998906 4421 generic.go:356] "Generic (PLEG): container finished" podID="18afc5dd-bc60-4794-91ab-b4019be56a61" containerID="1c783e52f0e82ca5675d8fae07f7a6289f582d17d5239a45061d389c89bd300b" exitCode=0 Nov 03 00:40:57.998940 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.998937 4421 generic.go:356] "Generic (PLEG): container finished" podID="18afc5dd-bc60-4794-91ab-b4019be56a61" containerID="62473d6087e26fc4e344ba92bc31c92f99655f2dad9487b90cdf0df87946436c" exitCode=0 Nov 03 00:40:57.998940 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.998946 4421 generic.go:356] "Generic (PLEG): container finished" podID="18afc5dd-bc60-4794-91ab-b4019be56a61" containerID="05834d884916df322d6a53853036fd2808cc38de59a9542ebb91dd53f5900496" exitCode=0 Nov 03 00:40:57.999161 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.998954 4421 generic.go:356] "Generic (PLEG): container finished" podID="18afc5dd-bc60-4794-91ab-b4019be56a61" containerID="a4878089ef5451a280992f04f3ea2b52c9653d62c3b1d176161d514c0ca26c37" exitCode=0 Nov 03 00:40:57.999161 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.998962 4421 generic.go:356] "Generic (PLEG): container finished" podID="18afc5dd-bc60-4794-91ab-b4019be56a61" containerID="215e994a15e19bbe833dfbd044681089a2a6f8903d5b490a867da6eb03308072" exitCode=0 Nov 03 00:40:57.999161 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:57.998970 4421 generic.go:356] "Generic (PLEG): container finished" podID="18afc5dd-bc60-4794-91ab-b4019be56a61" containerID="ffc09ec0336dc44907202879030aa74d8d9a837c7717d78202167e7799d0f8d8" exitCode=0 Nov 03 00:40:58.011102 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.011072 4421 manager.go:324] Recovery completed Nov 03 00:40:58.031609 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.031525 4421 cpu_manager.go:226] "Starting" policy="none" Nov 03 00:40:58.031609 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.031546 4421 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 03 00:40:58.031609 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.031572 4421 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 03 00:40:58.031837 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.031774 4421 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Nov 03 00:40:58.031837 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.031792 4421 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Nov 03 00:40:58.031837 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.031819 4421 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Nov 03 00:40:58.031963 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.031848 4421 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Nov 03 00:40:58.031963 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.031856 4421 policy_none.go:50] "Start" Nov 03 00:40:58.031963 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.031869 4421 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 03 00:40:58.031963 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.031883 4421 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 03 00:40:58.032105 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.032066 4421 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Nov 03 00:40:58.032105 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.032078 4421 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Nov 03 00:40:58.032105 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.032086 4421 policy_none.go:44] "Start" Nov 03 00:40:58.038718 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.038033 4421 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 03 00:40:58.045750 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.045726 4421 manager.go:341] "Starting Device Plugin manager" Nov 03 00:40:58.045891 ip-10-0-133-217 kubenswrapper[4421]: E1103 00:40:58.045760 4421 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 03 00:40:58.045891 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.045773 4421 server.go:87] "Starting device plugin registration server" Nov 03 00:40:58.046118 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.046099 4421 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 03 00:40:58.046181 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.046115 4421 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 03 00:40:58.046385 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.046366 4421 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 03 00:40:58.046514 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.046460 4421 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 03 00:40:58.046514 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.046511 4421 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 03 00:40:58.046624 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.046518 4421 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 03 00:40:58.046678 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.046593 4421 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T00:40:58.046499563Z","UUID":"4c359f18-9a8e-4e53-97ab-7b7bf1d61d74","Handler":null,"Name":"","Endpoint":""} Nov 03 00:40:58.047696 ip-10-0-133-217 kubenswrapper[4421]: E1103 00:40:58.047673 4421 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 03 00:40:58.048096 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.048077 4421 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 03 00:40:58.048194 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.048112 4421 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 03 00:40:58.073433 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.073368 4421 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-217.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-133-217.us-east-2.compute.internal"] Nov 03 00:40:58.073993 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.073951 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-217.us-east-2.compute.internal" event={"ID":"fdeeb94bfd911d86834e9d29bdbbadab","Type":"ContainerStarted","Data":"a5317886b2cb85855082058b7d59790632950ccefa1a6205312a5ad79a0a2f39"} Nov 03 00:40:58.074065 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.074006 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-217.us-east-2.compute.internal" event={"ID":"fdeeb94bfd911d86834e9d29bdbbadab","Type":"ContainerDied","Data":"00557970e2b0aac26fb1fe305bc704b99b3e296fcc7310e2a74186b8102140c1"} Nov 03 00:40:58.074065 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.074035 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-217.us-east-2.compute.internal" event={"ID":"fdeeb94bfd911d86834e9d29bdbbadab","Type":"ContainerStarted","Data":"d26b7bbc18a12c91a4663e8aabaac3278b33104993a34ea7f7e9c75dfeb63adb"} Nov 03 00:40:58.074196 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.074167 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-217.us-east-2.compute.internal" event={"ID":"a1171317f26f09b75b7b789f796f1384","Type":"ContainerStarted","Data":"b0a2b773a641306f4e9959431d9faacbd9461bea3824a5c6ceae00068ed04ddc"} Nov 03 00:40:58.074196 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.074191 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-217.us-east-2.compute.internal" event={"ID":"a1171317f26f09b75b7b789f796f1384","Type":"ContainerStarted","Data":"baf995ad5d0a338d5cab169fe531fe16fa1263a06c4429391a64689fce408d2a"} Nov 03 00:40:58.074530 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.074509 4421 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-217.us-east-2.compute.internal" Nov 03 00:40:58.074602 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.074518 4421 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-217.us-east-2.compute.internal" Nov 03 00:40:58.092011 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.091984 4421 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 00:40:58.092132 ip-10-0-133-217 kubenswrapper[4421]: E1103 00:40:58.092043 4421 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-133-217.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-217.us-east-2.compute.internal" Nov 03 00:40:58.092132 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.091991 4421 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 00:40:58.092132 ip-10-0-133-217 kubenswrapper[4421]: E1103 00:40:58.092107 4421 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-133-217.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-217.us-east-2.compute.internal" Nov 03 00:40:58.146296 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.146248 4421 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 00:40:58.146784 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.146760 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fdeeb94bfd911d86834e9d29bdbbadab-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-217.us-east-2.compute.internal\" (UID: \"fdeeb94bfd911d86834e9d29bdbbadab\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-217.us-east-2.compute.internal" Nov 03 00:40:58.146850 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.146793 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a1171317f26f09b75b7b789f796f1384-config\") pod \"kube-apiserver-proxy-ip-10-0-133-217.us-east-2.compute.internal\" (UID: \"a1171317f26f09b75b7b789f796f1384\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-217.us-east-2.compute.internal" Nov 03 00:40:58.146850 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.146811 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fdeeb94bfd911d86834e9d29bdbbadab-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-217.us-east-2.compute.internal\" (UID: \"fdeeb94bfd911d86834e9d29bdbbadab\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-217.us-east-2.compute.internal" Nov 03 00:40:58.147607 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.147587 4421 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-217.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 00:40:58.147699 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.147624 4421 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-217.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 00:40:58.147699 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.147638 4421 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-217.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 03 00:40:58.147699 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.147682 4421 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-133-217.us-east-2.compute.internal" Nov 03 00:40:58.163548 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.163528 4421 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-133-217.us-east-2.compute.internal" Nov 03 00:40:58.163600 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.163591 4421 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-133-217.us-east-2.compute.internal" Nov 03 00:40:58.247751 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.247710 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fdeeb94bfd911d86834e9d29bdbbadab-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-217.us-east-2.compute.internal\" (UID: \"fdeeb94bfd911d86834e9d29bdbbadab\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-217.us-east-2.compute.internal" Nov 03 00:40:58.247751 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.247754 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fdeeb94bfd911d86834e9d29bdbbadab-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-217.us-east-2.compute.internal\" (UID: \"fdeeb94bfd911d86834e9d29bdbbadab\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-217.us-east-2.compute.internal" Nov 03 00:40:58.247945 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.247782 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a1171317f26f09b75b7b789f796f1384-config\") pod \"kube-apiserver-proxy-ip-10-0-133-217.us-east-2.compute.internal\" (UID: \"a1171317f26f09b75b7b789f796f1384\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-217.us-east-2.compute.internal" Nov 03 00:40:58.247945 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.247832 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a1171317f26f09b75b7b789f796f1384-config\") pod \"kube-apiserver-proxy-ip-10-0-133-217.us-east-2.compute.internal\" (UID: \"a1171317f26f09b75b7b789f796f1384\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-217.us-east-2.compute.internal" Nov 03 00:40:58.247945 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.247834 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fdeeb94bfd911d86834e9d29bdbbadab-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-217.us-east-2.compute.internal\" (UID: \"fdeeb94bfd911d86834e9d29bdbbadab\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-217.us-east-2.compute.internal" Nov 03 00:40:58.247945 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.247838 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fdeeb94bfd911d86834e9d29bdbbadab-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-217.us-east-2.compute.internal\" (UID: \"fdeeb94bfd911d86834e9d29bdbbadab\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-217.us-east-2.compute.internal" Nov 03 00:40:58.913136 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.913093 4421 apiserver.go:52] "Watching apiserver" Nov 03 00:40:58.923557 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.923530 4421 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 03 00:40:58.923989 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.923967 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-zskmb","kube-system/global-pull-secret-syncer-262mm","openshift-dns/node-resolver-cphks","openshift-ingress-canary/ingress-canary-whbrp","kube-system/kube-apiserver-proxy-ip-10-0-133-217.us-east-2.compute.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-trgdf","openshift-image-registry/node-ca-w8k5t","openshift-multus/network-metrics-daemon-5v4k7","openshift-ovn-kubernetes/ovnkube-node-zq57z","openshift-dns/dns-default-rx2xs","kube-system/konnectivity-agent-jltt2","openshift-cluster-node-tuning-operator/tuned-n7ksd","openshift-multus/multus-additional-cni-plugins-8rxnt","openshift-network-operator/iptables-alerter-9w99c","openshift-network-diagnostics/network-check-target-6lvpt","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-217.us-east-2.compute.internal","openshift-multus/multus-lb46r"] Nov 03 00:40:58.928971 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.928642 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5v4k7" Nov 03 00:40:58.930857 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.930833 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-zskmb" Nov 03 00:40:58.931875 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.931835 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 03 00:40:58.931875 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.931864 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-7trdq\"" Nov 03 00:40:58.932073 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.932058 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 03 00:40:58.932179 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.932161 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 03 00:40:58.932179 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.932153 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 03 00:40:58.932316 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.932268 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 03 00:40:58.932388 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.932335 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 03 00:40:58.932388 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.932355 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-97hfk\"" Nov 03 00:40:58.932388 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.932373 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 03 00:40:58.932388 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.932376 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-7mmgk\"" Nov 03 00:40:58.932544 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.932393 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 03 00:40:58.932544 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.932410 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-4zb6t\"" Nov 03 00:40:58.932544 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.932336 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 03 00:40:58.932544 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.932413 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 03 00:40:58.932544 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.932437 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 03 00:40:58.932544 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.932456 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 03 00:40:58.932544 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.932508 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 03 00:40:58.932544 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.932539 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 03 00:40:58.932784 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.932720 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 03 00:40:58.932784 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.932762 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 03 00:40:58.933016 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.933002 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-w9l2l\"" Nov 03 00:40:58.933100 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.933024 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-229lw\"" Nov 03 00:40:58.936447 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.936433 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 03 00:40:58.936506 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.936467 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 03 00:40:58.936506 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.936494 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 03 00:40:58.936506 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.936501 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 03 00:40:58.936629 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.936517 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 03 00:40:58.936629 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.936536 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 03 00:40:58.936726 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.936666 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 03 00:40:58.936776 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.936757 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-28tgg\"" Nov 03 00:40:58.936776 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.936767 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-9v27r\"" Nov 03 00:40:58.936873 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.936776 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 03 00:40:58.936967 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.936954 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 03 00:40:58.937254 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.937224 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 03 00:40:58.937422 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.937295 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-kp9k4\"" Nov 03 00:40:58.937422 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.937336 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-42dxl\"" Nov 03 00:40:58.937580 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.937462 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-k7bjp\"" Nov 03 00:40:58.937580 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.937463 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 03 00:40:58.937580 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.937474 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 03 00:40:58.937580 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.937508 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 03 00:40:58.937698 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.937633 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 03 00:40:58.937698 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.937676 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 03 00:40:58.937824 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.937791 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 03 00:40:58.937880 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.937851 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 03 00:40:58.937912 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.937908 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 03 00:40:58.937945 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.937913 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 03 00:40:58.937945 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.937936 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 03 00:40:58.938004 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.937950 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-tq982\"" Nov 03 00:40:58.938004 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.937979 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-vz52v\"" Nov 03 00:40:58.938004 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.937982 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-d54vs\"" Nov 03 00:40:58.938004 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.937990 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 03 00:40:58.938177 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.938009 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 03 00:40:58.938177 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.938014 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 03 00:40:58.938177 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.938047 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-vmfsq\"" Nov 03 00:40:58.938269 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.938187 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 03 00:40:58.939668 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.939655 4421 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 03 00:40:58.952030 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.952002 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e7f1faa5-fc0e-4f46-9208-b1e58f50fcf9-hosts-file\") pod \"node-resolver-cphks\" (UID: \"e7f1faa5-fc0e-4f46-9208-b1e58f50fcf9\") " pod="openshift-dns/node-resolver-cphks" Nov 03 00:40:58.952156 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.952037 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/29de0f7e-9c96-46dd-8fec-434e3689a047-host-slash\") pod \"ovnkube-node-zq57z\" (UID: \"29de0f7e-9c96-46dd-8fec-434e3689a047\") " pod="openshift-ovn-kubernetes/ovnkube-node-zq57z" Nov 03 00:40:58.952156 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.952067 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b2d13929-be23-4396-8445-c3eee12e9b9d-cni-binary-copy\") pod \"multus-lb46r\" (UID: \"b2d13929-be23-4396-8445-c3eee12e9b9d\") " pod="openshift-multus/multus-lb46r" Nov 03 00:40:58.952156 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.952122 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b2d13929-be23-4396-8445-c3eee12e9b9d-hostroot\") pod \"multus-lb46r\" (UID: \"b2d13929-be23-4396-8445-c3eee12e9b9d\") " pod="openshift-multus/multus-lb46r" Nov 03 00:40:58.952338 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.952209 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5844f274-ad38-42a3-9900-b051b01c9953-host\") pod \"node-ca-w8k5t\" (UID: \"5844f274-ad38-42a3-9900-b051b01c9953\") " pod="openshift-image-registry/node-ca-w8k5t" Nov 03 00:40:58.952428 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.952411 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/bac578ef-a88b-4a74-9a65-bebc8ea1d00e-crio-socket\") pod \"insights-runtime-extractor-zskmb\" (UID: \"bac578ef-a88b-4a74-9a65-bebc8ea1d00e\") " pod="openshift-insights/insights-runtime-extractor-zskmb" Nov 03 00:40:58.952509 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.952461 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/476ff8a0-f214-4e5e-aa90-8a505974f963-etc-kubernetes\") pod \"tuned-n7ksd\" (UID: \"476ff8a0-f214-4e5e-aa90-8a505974f963\") " pod="openshift-cluster-node-tuning-operator/tuned-n7ksd" Nov 03 00:40:58.952560 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.952505 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/18afc5dd-bc60-4794-91ab-b4019be56a61-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8rxnt\" (UID: \"18afc5dd-bc60-4794-91ab-b4019be56a61\") " pod="openshift-multus/multus-additional-cni-plugins-8rxnt" Nov 03 00:40:58.952560 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.952523 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b2d13929-be23-4396-8445-c3eee12e9b9d-cni-binary-copy\") pod \"multus-lb46r\" (UID: \"b2d13929-be23-4396-8445-c3eee12e9b9d\") " pod="openshift-multus/multus-lb46r" Nov 03 00:40:58.952644 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.952561 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6ab5d408-02a2-4284-bf59-5827c6f84643-socket-dir\") pod \"aws-ebs-csi-driver-node-trgdf\" (UID: \"6ab5d408-02a2-4284-bf59-5827c6f84643\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-trgdf" Nov 03 00:40:58.952830 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.952805 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/29de0f7e-9c96-46dd-8fec-434e3689a047-etc-openvswitch\") pod \"ovnkube-node-zq57z\" (UID: \"29de0f7e-9c96-46dd-8fec-434e3689a047\") " pod="openshift-ovn-kubernetes/ovnkube-node-zq57z" Nov 03 00:40:58.952907 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.952844 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b2d13929-be23-4396-8445-c3eee12e9b9d-system-cni-dir\") pod \"multus-lb46r\" (UID: \"b2d13929-be23-4396-8445-c3eee12e9b9d\") " pod="openshift-multus/multus-lb46r" Nov 03 00:40:58.952907 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.952872 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5844f274-ad38-42a3-9900-b051b01c9953-serviceca\") pod \"node-ca-w8k5t\" (UID: \"5844f274-ad38-42a3-9900-b051b01c9953\") " pod="openshift-image-registry/node-ca-w8k5t" Nov 03 00:40:58.952907 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.952872 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/18afc5dd-bc60-4794-91ab-b4019be56a61-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8rxnt\" (UID: \"18afc5dd-bc60-4794-91ab-b4019be56a61\") " pod="openshift-multus/multus-additional-cni-plugins-8rxnt" Nov 03 00:40:58.953016 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.952911 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bac578ef-a88b-4a74-9a65-bebc8ea1d00e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zskmb\" (UID: \"bac578ef-a88b-4a74-9a65-bebc8ea1d00e\") " pod="openshift-insights/insights-runtime-extractor-zskmb" Nov 03 00:40:58.953016 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.952937 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c505674-e66b-4573-bbf5-7b6acf6684e6-tmp-dir\") pod \"dns-default-rx2xs\" (UID: \"9c505674-e66b-4573-bbf5-7b6acf6684e6\") " pod="openshift-dns/dns-default-rx2xs" Nov 03 00:40:58.953088 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.953048 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/476ff8a0-f214-4e5e-aa90-8a505974f963-etc-sysctl-d\") pod \"tuned-n7ksd\" (UID: \"476ff8a0-f214-4e5e-aa90-8a505974f963\") " pod="openshift-cluster-node-tuning-operator/tuned-n7ksd" Nov 03 00:40:58.953088 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.953080 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/18afc5dd-bc60-4794-91ab-b4019be56a61-system-cni-dir\") pod \"multus-additional-cni-plugins-8rxnt\" (UID: \"18afc5dd-bc60-4794-91ab-b4019be56a61\") " pod="openshift-multus/multus-additional-cni-plugins-8rxnt" Nov 03 00:40:58.953203 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.953105 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b2d13929-be23-4396-8445-c3eee12e9b9d-multus-conf-dir\") pod \"multus-lb46r\" (UID: \"b2d13929-be23-4396-8445-c3eee12e9b9d\") " pod="openshift-multus/multus-lb46r" Nov 03 00:40:58.953203 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.953124 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c505674-e66b-4573-bbf5-7b6acf6684e6-tmp-dir\") pod \"dns-default-rx2xs\" (UID: \"9c505674-e66b-4573-bbf5-7b6acf6684e6\") " pod="openshift-dns/dns-default-rx2xs" Nov 03 00:40:58.953203 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.953134 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/cd47e545-4089-44ad-bc4d-0545061c7b11-konnectivity-ca\") pod \"konnectivity-agent-jltt2\" (UID: \"cd47e545-4089-44ad-bc4d-0545061c7b11\") " pod="kube-system/konnectivity-agent-jltt2" Nov 03 00:40:58.953352 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.953255 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bac578ef-a88b-4a74-9a65-bebc8ea1d00e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zskmb\" (UID: \"bac578ef-a88b-4a74-9a65-bebc8ea1d00e\") " pod="openshift-insights/insights-runtime-extractor-zskmb" Nov 03 00:40:58.953352 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.953319 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/29de0f7e-9c96-46dd-8fec-434e3689a047-run-openvswitch\") pod \"ovnkube-node-zq57z\" (UID: \"29de0f7e-9c96-46dd-8fec-434e3689a047\") " pod="openshift-ovn-kubernetes/ovnkube-node-zq57z" Nov 03 00:40:58.953598 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.953580 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5844f274-ad38-42a3-9900-b051b01c9953-serviceca\") pod \"node-ca-w8k5t\" (UID: \"5844f274-ad38-42a3-9900-b051b01c9953\") " pod="openshift-image-registry/node-ca-w8k5t" Nov 03 00:40:58.953780 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.953389 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzb4d\" (UniqueName: \"kubernetes.io/projected/b6abcc85-61c0-42ea-b868-0d71a46a4514-kube-api-access-lzb4d\") pod \"network-metrics-daemon-5v4k7\" (UID: \"b6abcc85-61c0-42ea-b868-0d71a46a4514\") " pod="openshift-multus/network-metrics-daemon-5v4k7" Nov 03 00:40:58.953820 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.953802 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/476ff8a0-f214-4e5e-aa90-8a505974f963-etc-modprobe-d\") pod \"tuned-n7ksd\" (UID: \"476ff8a0-f214-4e5e-aa90-8a505974f963\") " pod="openshift-cluster-node-tuning-operator/tuned-n7ksd" Nov 03 00:40:58.953850 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.953826 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/29de0f7e-9c96-46dd-8fec-434e3689a047-host-kubelet\") pod \"ovnkube-node-zq57z\" (UID: \"29de0f7e-9c96-46dd-8fec-434e3689a047\") " pod="openshift-ovn-kubernetes/ovnkube-node-zq57z" Nov 03 00:40:58.953886 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.953856 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8t7dh\" (UniqueName: \"kubernetes.io/projected/e7f1faa5-fc0e-4f46-9208-b1e58f50fcf9-kube-api-access-8t7dh\") pod \"node-resolver-cphks\" (UID: \"e7f1faa5-fc0e-4f46-9208-b1e58f50fcf9\") " pod="openshift-dns/node-resolver-cphks" Nov 03 00:40:58.953933 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.953885 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6ab5d408-02a2-4284-bf59-5827c6f84643-registration-dir\") pod \"aws-ebs-csi-driver-node-trgdf\" (UID: \"6ab5d408-02a2-4284-bf59-5827c6f84643\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-trgdf" Nov 03 00:40:58.953933 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.953911 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/cd47e545-4089-44ad-bc4d-0545061c7b11-konnectivity-ca\") pod \"konnectivity-agent-jltt2\" (UID: \"cd47e545-4089-44ad-bc4d-0545061c7b11\") " pod="kube-system/konnectivity-agent-jltt2" Nov 03 00:40:58.953933 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.953922 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6ab5d408-02a2-4284-bf59-5827c6f84643-sys-fs\") pod \"aws-ebs-csi-driver-node-trgdf\" (UID: \"6ab5d408-02a2-4284-bf59-5827c6f84643\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-trgdf" Nov 03 00:40:58.954053 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.953946 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45qm2\" (UniqueName: \"kubernetes.io/projected/29de0f7e-9c96-46dd-8fec-434e3689a047-kube-api-access-45qm2\") pod \"ovnkube-node-zq57z\" (UID: \"29de0f7e-9c96-46dd-8fec-434e3689a047\") " pod="openshift-ovn-kubernetes/ovnkube-node-zq57z" Nov 03 00:40:58.954053 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.953962 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5fls\" (UniqueName: \"kubernetes.io/projected/174d7fa2-7163-47a8-a047-8727cdb3cb9f-kube-api-access-c5fls\") pod \"iptables-alerter-9w99c\" (UID: \"174d7fa2-7163-47a8-a047-8727cdb3cb9f\") " pod="openshift-network-operator/iptables-alerter-9w99c" Nov 03 00:40:58.954053 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.953995 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bac578ef-a88b-4a74-9a65-bebc8ea1d00e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zskmb\" (UID: \"bac578ef-a88b-4a74-9a65-bebc8ea1d00e\") " pod="openshift-insights/insights-runtime-extractor-zskmb" Nov 03 00:40:58.954198 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.954051 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86gz6\" (UniqueName: \"kubernetes.io/projected/bac578ef-a88b-4a74-9a65-bebc8ea1d00e-kube-api-access-86gz6\") pod \"insights-runtime-extractor-zskmb\" (UID: \"bac578ef-a88b-4a74-9a65-bebc8ea1d00e\") " pod="openshift-insights/insights-runtime-extractor-zskmb" Nov 03 00:40:58.954198 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.954082 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4c5b\" (UniqueName: \"kubernetes.io/projected/b2d13929-be23-4396-8445-c3eee12e9b9d-kube-api-access-h4c5b\") pod \"multus-lb46r\" (UID: \"b2d13929-be23-4396-8445-c3eee12e9b9d\") " pod="openshift-multus/multus-lb46r" Nov 03 00:40:58.954198 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.954110 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/29de0f7e-9c96-46dd-8fec-434e3689a047-run-systemd\") pod \"ovnkube-node-zq57z\" (UID: \"29de0f7e-9c96-46dd-8fec-434e3689a047\") " pod="openshift-ovn-kubernetes/ovnkube-node-zq57z" Nov 03 00:40:58.954198 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.954137 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/29de0f7e-9c96-46dd-8fec-434e3689a047-run-ovn\") pod \"ovnkube-node-zq57z\" (UID: \"29de0f7e-9c96-46dd-8fec-434e3689a047\") " pod="openshift-ovn-kubernetes/ovnkube-node-zq57z" Nov 03 00:40:58.954198 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.954162 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/174d7fa2-7163-47a8-a047-8727cdb3cb9f-host-slash\") pod \"iptables-alerter-9w99c\" (UID: \"174d7fa2-7163-47a8-a047-8727cdb3cb9f\") " pod="openshift-network-operator/iptables-alerter-9w99c" Nov 03 00:40:58.954198 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.954186 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b2d13929-be23-4396-8445-c3eee12e9b9d-host-var-lib-cni-multus\") pod \"multus-lb46r\" (UID: \"b2d13929-be23-4396-8445-c3eee12e9b9d\") " pod="openshift-multus/multus-lb46r" Nov 03 00:40:58.954640 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.954254 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b2d13929-be23-4396-8445-c3eee12e9b9d-multus-daemon-config\") pod \"multus-lb46r\" (UID: \"b2d13929-be23-4396-8445-c3eee12e9b9d\") " pod="openshift-multus/multus-lb46r" Nov 03 00:40:58.954640 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.954289 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/cd47e545-4089-44ad-bc4d-0545061c7b11-agent-certs\") pod \"konnectivity-agent-jltt2\" (UID: \"cd47e545-4089-44ad-bc4d-0545061c7b11\") " pod="kube-system/konnectivity-agent-jltt2" Nov 03 00:40:58.954640 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.954304 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9c505674-e66b-4573-bbf5-7b6acf6684e6-config-volume\") pod \"dns-default-rx2xs\" (UID: \"9c505674-e66b-4573-bbf5-7b6acf6684e6\") " pod="openshift-dns/dns-default-rx2xs" Nov 03 00:40:58.954640 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.954319 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6ab5d408-02a2-4284-bf59-5827c6f84643-device-dir\") pod \"aws-ebs-csi-driver-node-trgdf\" (UID: \"6ab5d408-02a2-4284-bf59-5827c6f84643\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-trgdf" Nov 03 00:40:58.954640 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.954334 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/29de0f7e-9c96-46dd-8fec-434e3689a047-env-overrides\") pod \"ovnkube-node-zq57z\" (UID: \"29de0f7e-9c96-46dd-8fec-434e3689a047\") " pod="openshift-ovn-kubernetes/ovnkube-node-zq57z" Nov 03 00:40:58.954640 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.954365 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b2d13929-be23-4396-8445-c3eee12e9b9d-host-run-netns\") pod \"multus-lb46r\" (UID: \"b2d13929-be23-4396-8445-c3eee12e9b9d\") " pod="openshift-multus/multus-lb46r" Nov 03 00:40:58.954640 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.954389 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9c505674-e66b-4573-bbf5-7b6acf6684e6-metrics-tls\") pod \"dns-default-rx2xs\" (UID: \"9c505674-e66b-4573-bbf5-7b6acf6684e6\") " pod="openshift-dns/dns-default-rx2xs" Nov 03 00:40:58.954640 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.954429 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/92341144-23c4-4cf0-a0c8-26e1891397e9-kubelet-config\") pod \"global-pull-secret-syncer-262mm\" (UID: \"92341144-23c4-4cf0-a0c8-26e1891397e9\") " pod="kube-system/global-pull-secret-syncer-262mm" Nov 03 00:40:58.954640 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.954457 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b2d13929-be23-4396-8445-c3eee12e9b9d-multus-daemon-config\") pod \"multus-lb46r\" (UID: \"b2d13929-be23-4396-8445-c3eee12e9b9d\") " pod="openshift-multus/multus-lb46r" Nov 03 00:40:58.954640 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.954476 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/92341144-23c4-4cf0-a0c8-26e1891397e9-dbus\") pod \"global-pull-secret-syncer-262mm\" (UID: \"92341144-23c4-4cf0-a0c8-26e1891397e9\") " pod="kube-system/global-pull-secret-syncer-262mm" Nov 03 00:40:58.954640 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.954535 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/76ccef27-d056-402e-a7e1-ee524d63bf35-cert\") pod \"ingress-canary-whbrp\" (UID: \"76ccef27-d056-402e-a7e1-ee524d63bf35\") " pod="openshift-ingress-canary/ingress-canary-whbrp" Nov 03 00:40:58.954640 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.954571 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/29de0f7e-9c96-46dd-8fec-434e3689a047-env-overrides\") pod \"ovnkube-node-zq57z\" (UID: \"29de0f7e-9c96-46dd-8fec-434e3689a047\") " pod="openshift-ovn-kubernetes/ovnkube-node-zq57z" Nov 03 00:40:58.954640 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.954574 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9c505674-e66b-4573-bbf5-7b6acf6684e6-config-volume\") pod \"dns-default-rx2xs\" (UID: \"9c505674-e66b-4573-bbf5-7b6acf6684e6\") " pod="openshift-dns/dns-default-rx2xs" Nov 03 00:40:58.954640 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.954630 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/cd47e545-4089-44ad-bc4d-0545061c7b11-agent-certs\") pod \"konnectivity-agent-jltt2\" (UID: \"cd47e545-4089-44ad-bc4d-0545061c7b11\") " pod="kube-system/konnectivity-agent-jltt2" Nov 03 00:40:58.955247 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.954695 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/476ff8a0-f214-4e5e-aa90-8a505974f963-etc-sysctl-conf\") pod \"tuned-n7ksd\" (UID: \"476ff8a0-f214-4e5e-aa90-8a505974f963\") " pod="openshift-cluster-node-tuning-operator/tuned-n7ksd" Nov 03 00:40:58.955247 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.954719 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9c505674-e66b-4573-bbf5-7b6acf6684e6-metrics-tls\") pod \"dns-default-rx2xs\" (UID: \"9c505674-e66b-4573-bbf5-7b6acf6684e6\") " pod="openshift-dns/dns-default-rx2xs" Nov 03 00:40:58.955247 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.954729 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6ab5d408-02a2-4284-bf59-5827c6f84643-kubelet-dir\") pod \"aws-ebs-csi-driver-node-trgdf\" (UID: \"6ab5d408-02a2-4284-bf59-5827c6f84643\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-trgdf" Nov 03 00:40:58.955247 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.954770 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/76ccef27-d056-402e-a7e1-ee524d63bf35-cert\") pod \"ingress-canary-whbrp\" (UID: \"76ccef27-d056-402e-a7e1-ee524d63bf35\") " pod="openshift-ingress-canary/ingress-canary-whbrp" Nov 03 00:40:58.955247 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.954786 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/29de0f7e-9c96-46dd-8fec-434e3689a047-host-cni-netd\") pod \"ovnkube-node-zq57z\" (UID: \"29de0f7e-9c96-46dd-8fec-434e3689a047\") " pod="openshift-ovn-kubernetes/ovnkube-node-zq57z" Nov 03 00:40:58.955247 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.954837 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/29de0f7e-9c96-46dd-8fec-434e3689a047-ovnkube-config\") pod \"ovnkube-node-zq57z\" (UID: \"29de0f7e-9c96-46dd-8fec-434e3689a047\") " pod="openshift-ovn-kubernetes/ovnkube-node-zq57z" Nov 03 00:40:58.955247 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.954880 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b2d13929-be23-4396-8445-c3eee12e9b9d-multus-socket-dir-parent\") pod \"multus-lb46r\" (UID: \"b2d13929-be23-4396-8445-c3eee12e9b9d\") " pod="openshift-multus/multus-lb46r" Nov 03 00:40:58.955247 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.954972 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b2d13929-be23-4396-8445-c3eee12e9b9d-etc-kubernetes\") pod \"multus-lb46r\" (UID: \"b2d13929-be23-4396-8445-c3eee12e9b9d\") " pod="openshift-multus/multus-lb46r" Nov 03 00:40:58.955247 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.954990 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/476ff8a0-f214-4e5e-aa90-8a505974f963-etc-sysconfig\") pod \"tuned-n7ksd\" (UID: \"476ff8a0-f214-4e5e-aa90-8a505974f963\") " pod="openshift-cluster-node-tuning-operator/tuned-n7ksd" Nov 03 00:40:58.955247 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.955013 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/476ff8a0-f214-4e5e-aa90-8a505974f963-etc-systemd\") pod \"tuned-n7ksd\" (UID: \"476ff8a0-f214-4e5e-aa90-8a505974f963\") " pod="openshift-cluster-node-tuning-operator/tuned-n7ksd" Nov 03 00:40:58.955247 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.955037 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2n98j\" (UniqueName: \"kubernetes.io/projected/6ab5d408-02a2-4284-bf59-5827c6f84643-kube-api-access-2n98j\") pod \"aws-ebs-csi-driver-node-trgdf\" (UID: \"6ab5d408-02a2-4284-bf59-5827c6f84643\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-trgdf" Nov 03 00:40:58.955247 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.955089 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/29de0f7e-9c96-46dd-8fec-434e3689a047-ovnkube-config\") pod \"ovnkube-node-zq57z\" (UID: \"29de0f7e-9c96-46dd-8fec-434e3689a047\") " pod="openshift-ovn-kubernetes/ovnkube-node-zq57z" Nov 03 00:40:58.955247 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.955110 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/29de0f7e-9c96-46dd-8fec-434e3689a047-log-socket\") pod \"ovnkube-node-zq57z\" (UID: \"29de0f7e-9c96-46dd-8fec-434e3689a047\") " pod="openshift-ovn-kubernetes/ovnkube-node-zq57z" Nov 03 00:40:58.955247 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.955162 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b2d13929-be23-4396-8445-c3eee12e9b9d-multus-cni-dir\") pod \"multus-lb46r\" (UID: \"b2d13929-be23-4396-8445-c3eee12e9b9d\") " pod="openshift-multus/multus-lb46r" Nov 03 00:40:58.955247 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.955181 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b2d13929-be23-4396-8445-c3eee12e9b9d-host-var-lib-cni-bin\") pod \"multus-lb46r\" (UID: \"b2d13929-be23-4396-8445-c3eee12e9b9d\") " pod="openshift-multus/multus-lb46r" Nov 03 00:40:58.955247 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.955194 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/476ff8a0-f214-4e5e-aa90-8a505974f963-sys\") pod \"tuned-n7ksd\" (UID: \"476ff8a0-f214-4e5e-aa90-8a505974f963\") " pod="openshift-cluster-node-tuning-operator/tuned-n7ksd" Nov 03 00:40:58.955247 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.955208 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/476ff8a0-f214-4e5e-aa90-8a505974f963-lib-modules\") pod \"tuned-n7ksd\" (UID: \"476ff8a0-f214-4e5e-aa90-8a505974f963\") " pod="openshift-cluster-node-tuning-operator/tuned-n7ksd" Nov 03 00:40:58.955247 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.955253 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dt25z\" (UniqueName: \"kubernetes.io/projected/3dcbff01-09fb-458e-b3a5-594d6b271133-kube-api-access-dt25z\") pod \"network-check-target-6lvpt\" (UID: \"3dcbff01-09fb-458e-b3a5-594d6b271133\") " pod="openshift-network-diagnostics/network-check-target-6lvpt" Nov 03 00:40:58.955950 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.955304 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/18afc5dd-bc60-4794-91ab-b4019be56a61-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8rxnt\" (UID: \"18afc5dd-bc60-4794-91ab-b4019be56a61\") " pod="openshift-multus/multus-additional-cni-plugins-8rxnt" Nov 03 00:40:58.955950 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.955423 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/29de0f7e-9c96-46dd-8fec-434e3689a047-systemd-units\") pod \"ovnkube-node-zq57z\" (UID: \"29de0f7e-9c96-46dd-8fec-434e3689a047\") " pod="openshift-ovn-kubernetes/ovnkube-node-zq57z" Nov 03 00:40:58.955950 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.955457 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/29de0f7e-9c96-46dd-8fec-434e3689a047-ovnkube-script-lib\") pod \"ovnkube-node-zq57z\" (UID: \"29de0f7e-9c96-46dd-8fec-434e3689a047\") " pod="openshift-ovn-kubernetes/ovnkube-node-zq57z" Nov 03 00:40:58.955950 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.955484 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/29de0f7e-9c96-46dd-8fec-434e3689a047-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zq57z\" (UID: \"29de0f7e-9c96-46dd-8fec-434e3689a047\") " pod="openshift-ovn-kubernetes/ovnkube-node-zq57z" Nov 03 00:40:58.955950 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.955518 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/bac578ef-a88b-4a74-9a65-bebc8ea1d00e-data-volume\") pod \"insights-runtime-extractor-zskmb\" (UID: \"bac578ef-a88b-4a74-9a65-bebc8ea1d00e\") " pod="openshift-insights/insights-runtime-extractor-zskmb" Nov 03 00:40:58.955950 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.955551 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/476ff8a0-f214-4e5e-aa90-8a505974f963-var-lib-kubelet\") pod \"tuned-n7ksd\" (UID: \"476ff8a0-f214-4e5e-aa90-8a505974f963\") " pod="openshift-cluster-node-tuning-operator/tuned-n7ksd" Nov 03 00:40:58.955950 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.955593 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/476ff8a0-f214-4e5e-aa90-8a505974f963-tmp\") pod \"tuned-n7ksd\" (UID: \"476ff8a0-f214-4e5e-aa90-8a505974f963\") " pod="openshift-cluster-node-tuning-operator/tuned-n7ksd" Nov 03 00:40:58.955950 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.955622 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/29de0f7e-9c96-46dd-8fec-434e3689a047-var-lib-openvswitch\") pod \"ovnkube-node-zq57z\" (UID: \"29de0f7e-9c96-46dd-8fec-434e3689a047\") " pod="openshift-ovn-kubernetes/ovnkube-node-zq57z" Nov 03 00:40:58.955950 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.955645 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/bac578ef-a88b-4a74-9a65-bebc8ea1d00e-data-volume\") pod \"insights-runtime-extractor-zskmb\" (UID: \"bac578ef-a88b-4a74-9a65-bebc8ea1d00e\") " pod="openshift-insights/insights-runtime-extractor-zskmb" Nov 03 00:40:58.955950 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.955683 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/174d7fa2-7163-47a8-a047-8727cdb3cb9f-iptables-alerter-script\") pod \"iptables-alerter-9w99c\" (UID: \"174d7fa2-7163-47a8-a047-8727cdb3cb9f\") " pod="openshift-network-operator/iptables-alerter-9w99c" Nov 03 00:40:58.955950 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.955748 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/476ff8a0-f214-4e5e-aa90-8a505974f963-tmp\") pod \"tuned-n7ksd\" (UID: \"476ff8a0-f214-4e5e-aa90-8a505974f963\") " pod="openshift-cluster-node-tuning-operator/tuned-n7ksd" Nov 03 00:40:58.955950 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.955802 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qn5dn\" (UniqueName: \"kubernetes.io/projected/5844f274-ad38-42a3-9900-b051b01c9953-kube-api-access-qn5dn\") pod \"node-ca-w8k5t\" (UID: \"5844f274-ad38-42a3-9900-b051b01c9953\") " pod="openshift-image-registry/node-ca-w8k5t" Nov 03 00:40:58.955950 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.955801 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/29de0f7e-9c96-46dd-8fec-434e3689a047-ovnkube-script-lib\") pod \"ovnkube-node-zq57z\" (UID: \"29de0f7e-9c96-46dd-8fec-434e3689a047\") " pod="openshift-ovn-kubernetes/ovnkube-node-zq57z" Nov 03 00:40:58.955950 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.955899 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/476ff8a0-f214-4e5e-aa90-8a505974f963-etc-tuned\") pod \"tuned-n7ksd\" (UID: \"476ff8a0-f214-4e5e-aa90-8a505974f963\") " pod="openshift-cluster-node-tuning-operator/tuned-n7ksd" Nov 03 00:40:58.955950 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.955951 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/476ff8a0-f214-4e5e-aa90-8a505974f963-etc-tuned\") pod \"tuned-n7ksd\" (UID: \"476ff8a0-f214-4e5e-aa90-8a505974f963\") " pod="openshift-cluster-node-tuning-operator/tuned-n7ksd" Nov 03 00:40:58.956662 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.955967 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/18afc5dd-bc60-4794-91ab-b4019be56a61-cni-binary-copy\") pod \"multus-additional-cni-plugins-8rxnt\" (UID: \"18afc5dd-bc60-4794-91ab-b4019be56a61\") " pod="openshift-multus/multus-additional-cni-plugins-8rxnt" Nov 03 00:40:58.956662 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.955994 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4svt5\" (UniqueName: \"kubernetes.io/projected/9c505674-e66b-4573-bbf5-7b6acf6684e6-kube-api-access-4svt5\") pod \"dns-default-rx2xs\" (UID: \"9c505674-e66b-4573-bbf5-7b6acf6684e6\") " pod="openshift-dns/dns-default-rx2xs" Nov 03 00:40:58.956662 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.956020 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/18afc5dd-bc60-4794-91ab-b4019be56a61-cnibin\") pod \"multus-additional-cni-plugins-8rxnt\" (UID: \"18afc5dd-bc60-4794-91ab-b4019be56a61\") " pod="openshift-multus/multus-additional-cni-plugins-8rxnt" Nov 03 00:40:58.956662 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.956059 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/174d7fa2-7163-47a8-a047-8727cdb3cb9f-iptables-alerter-script\") pod \"iptables-alerter-9w99c\" (UID: \"174d7fa2-7163-47a8-a047-8727cdb3cb9f\") " pod="openshift-network-operator/iptables-alerter-9w99c" Nov 03 00:40:58.956662 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.956061 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b2d13929-be23-4396-8445-c3eee12e9b9d-host-run-multus-certs\") pod \"multus-lb46r\" (UID: \"b2d13929-be23-4396-8445-c3eee12e9b9d\") " pod="openshift-multus/multus-lb46r" Nov 03 00:40:58.956662 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.956105 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6ab5d408-02a2-4284-bf59-5827c6f84643-etc-selinux\") pod \"aws-ebs-csi-driver-node-trgdf\" (UID: \"6ab5d408-02a2-4284-bf59-5827c6f84643\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-trgdf" Nov 03 00:40:58.956662 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.956157 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/29de0f7e-9c96-46dd-8fec-434e3689a047-node-log\") pod \"ovnkube-node-zq57z\" (UID: \"29de0f7e-9c96-46dd-8fec-434e3689a047\") " pod="openshift-ovn-kubernetes/ovnkube-node-zq57z" Nov 03 00:40:58.956662 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.956230 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/18afc5dd-bc60-4794-91ab-b4019be56a61-cni-binary-copy\") pod \"multus-additional-cni-plugins-8rxnt\" (UID: \"18afc5dd-bc60-4794-91ab-b4019be56a61\") " pod="openshift-multus/multus-additional-cni-plugins-8rxnt" Nov 03 00:40:58.956662 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.956239 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/29de0f7e-9c96-46dd-8fec-434e3689a047-host-cni-bin\") pod \"ovnkube-node-zq57z\" (UID: \"29de0f7e-9c96-46dd-8fec-434e3689a047\") " pod="openshift-ovn-kubernetes/ovnkube-node-zq57z" Nov 03 00:40:58.956662 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.956348 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b2d13929-be23-4396-8445-c3eee12e9b9d-cnibin\") pod \"multus-lb46r\" (UID: \"b2d13929-be23-4396-8445-c3eee12e9b9d\") " pod="openshift-multus/multus-lb46r" Nov 03 00:40:58.956662 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.956383 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b2d13929-be23-4396-8445-c3eee12e9b9d-host-run-k8s-cni-cncf-io\") pod \"multus-lb46r\" (UID: \"b2d13929-be23-4396-8445-c3eee12e9b9d\") " pod="openshift-multus/multus-lb46r" Nov 03 00:40:58.956662 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.956435 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b2d13929-be23-4396-8445-c3eee12e9b9d-host-var-lib-kubelet\") pod \"multus-lb46r\" (UID: \"b2d13929-be23-4396-8445-c3eee12e9b9d\") " pod="openshift-multus/multus-lb46r" Nov 03 00:40:58.956662 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.956485 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/92341144-23c4-4cf0-a0c8-26e1891397e9-original-pull-secret\") pod \"global-pull-secret-syncer-262mm\" (UID: \"92341144-23c4-4cf0-a0c8-26e1891397e9\") " pod="kube-system/global-pull-secret-syncer-262mm" Nov 03 00:40:58.956662 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.956527 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7f1faa5-fc0e-4f46-9208-b1e58f50fcf9-tmp-dir\") pod \"node-resolver-cphks\" (UID: \"e7f1faa5-fc0e-4f46-9208-b1e58f50fcf9\") " pod="openshift-dns/node-resolver-cphks" Nov 03 00:40:58.956662 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.956565 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/29de0f7e-9c96-46dd-8fec-434e3689a047-ovn-node-metrics-cert\") pod \"ovnkube-node-zq57z\" (UID: \"29de0f7e-9c96-46dd-8fec-434e3689a047\") " pod="openshift-ovn-kubernetes/ovnkube-node-zq57z" Nov 03 00:40:58.956662 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.956632 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjg6m\" (UniqueName: \"kubernetes.io/projected/76ccef27-d056-402e-a7e1-ee524d63bf35-kube-api-access-rjg6m\") pod \"ingress-canary-whbrp\" (UID: \"76ccef27-d056-402e-a7e1-ee524d63bf35\") " pod="openshift-ingress-canary/ingress-canary-whbrp" Nov 03 00:40:58.957310 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.956672 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7f1faa5-fc0e-4f46-9208-b1e58f50fcf9-tmp-dir\") pod \"node-resolver-cphks\" (UID: \"e7f1faa5-fc0e-4f46-9208-b1e58f50fcf9\") " pod="openshift-dns/node-resolver-cphks" Nov 03 00:40:58.957310 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.956721 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/476ff8a0-f214-4e5e-aa90-8a505974f963-host\") pod \"tuned-n7ksd\" (UID: \"476ff8a0-f214-4e5e-aa90-8a505974f963\") " pod="openshift-cluster-node-tuning-operator/tuned-n7ksd" Nov 03 00:40:58.957310 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.956766 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/18afc5dd-bc60-4794-91ab-b4019be56a61-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8rxnt\" (UID: \"18afc5dd-bc60-4794-91ab-b4019be56a61\") " pod="openshift-multus/multus-additional-cni-plugins-8rxnt" Nov 03 00:40:58.957310 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.956793 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsbnr\" (UniqueName: \"kubernetes.io/projected/18afc5dd-bc60-4794-91ab-b4019be56a61-kube-api-access-wsbnr\") pod \"multus-additional-cni-plugins-8rxnt\" (UID: \"18afc5dd-bc60-4794-91ab-b4019be56a61\") " pod="openshift-multus/multus-additional-cni-plugins-8rxnt" Nov 03 00:40:58.957310 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.956819 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/29de0f7e-9c96-46dd-8fec-434e3689a047-host-run-netns\") pod \"ovnkube-node-zq57z\" (UID: \"29de0f7e-9c96-46dd-8fec-434e3689a047\") " pod="openshift-ovn-kubernetes/ovnkube-node-zq57z" Nov 03 00:40:58.957310 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.956821 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/29de0f7e-9c96-46dd-8fec-434e3689a047-ovn-node-metrics-cert\") pod \"ovnkube-node-zq57z\" (UID: \"29de0f7e-9c96-46dd-8fec-434e3689a047\") " pod="openshift-ovn-kubernetes/ovnkube-node-zq57z" Nov 03 00:40:58.957310 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.956847 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/29de0f7e-9c96-46dd-8fec-434e3689a047-host-run-ovn-kubernetes\") pod \"ovnkube-node-zq57z\" (UID: \"29de0f7e-9c96-46dd-8fec-434e3689a047\") " pod="openshift-ovn-kubernetes/ovnkube-node-zq57z" Nov 03 00:40:58.957310 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.957159 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/92341144-23c4-4cf0-a0c8-26e1891397e9-original-pull-secret\") pod \"global-pull-secret-syncer-262mm\" (UID: \"92341144-23c4-4cf0-a0c8-26e1891397e9\") " pod="kube-system/global-pull-secret-syncer-262mm" Nov 03 00:40:58.957310 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.957225 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b6abcc85-61c0-42ea-b868-0d71a46a4514-metrics-certs\") pod \"network-metrics-daemon-5v4k7\" (UID: \"b6abcc85-61c0-42ea-b868-0d71a46a4514\") " pod="openshift-multus/network-metrics-daemon-5v4k7" Nov 03 00:40:58.957310 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.957296 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/476ff8a0-f214-4e5e-aa90-8a505974f963-run\") pod \"tuned-n7ksd\" (UID: \"476ff8a0-f214-4e5e-aa90-8a505974f963\") " pod="openshift-cluster-node-tuning-operator/tuned-n7ksd" Nov 03 00:40:58.957728 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.957351 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqbgf\" (UniqueName: \"kubernetes.io/projected/476ff8a0-f214-4e5e-aa90-8a505974f963-kube-api-access-lqbgf\") pod \"tuned-n7ksd\" (UID: \"476ff8a0-f214-4e5e-aa90-8a505974f963\") " pod="openshift-cluster-node-tuning-operator/tuned-n7ksd" Nov 03 00:40:58.957836 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.957733 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/18afc5dd-bc60-4794-91ab-b4019be56a61-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8rxnt\" (UID: \"18afc5dd-bc60-4794-91ab-b4019be56a61\") " pod="openshift-multus/multus-additional-cni-plugins-8rxnt" Nov 03 00:40:58.962581 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.962559 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8t7dh\" (UniqueName: \"kubernetes.io/projected/e7f1faa5-fc0e-4f46-9208-b1e58f50fcf9-kube-api-access-8t7dh\") pod \"node-resolver-cphks\" (UID: \"e7f1faa5-fc0e-4f46-9208-b1e58f50fcf9\") " pod="openshift-dns/node-resolver-cphks" Nov 03 00:40:58.963561 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.963535 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2n98j\" (UniqueName: \"kubernetes.io/projected/6ab5d408-02a2-4284-bf59-5827c6f84643-kube-api-access-2n98j\") pod \"aws-ebs-csi-driver-node-trgdf\" (UID: \"6ab5d408-02a2-4284-bf59-5827c6f84643\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-trgdf" Nov 03 00:40:58.963633 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.963601 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dt25z\" (UniqueName: \"kubernetes.io/projected/3dcbff01-09fb-458e-b3a5-594d6b271133-kube-api-access-dt25z\") pod \"network-check-target-6lvpt\" (UID: \"3dcbff01-09fb-458e-b3a5-594d6b271133\") " pod="openshift-network-diagnostics/network-check-target-6lvpt" Nov 03 00:40:58.963707 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.963689 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzb4d\" (UniqueName: \"kubernetes.io/projected/b6abcc85-61c0-42ea-b868-0d71a46a4514-kube-api-access-lzb4d\") pod \"network-metrics-daemon-5v4k7\" (UID: \"b6abcc85-61c0-42ea-b868-0d71a46a4514\") " pod="openshift-multus/network-metrics-daemon-5v4k7" Nov 03 00:40:58.964371 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.964354 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86gz6\" (UniqueName: \"kubernetes.io/projected/bac578ef-a88b-4a74-9a65-bebc8ea1d00e-kube-api-access-86gz6\") pod \"insights-runtime-extractor-zskmb\" (UID: \"bac578ef-a88b-4a74-9a65-bebc8ea1d00e\") " pod="openshift-insights/insights-runtime-extractor-zskmb" Nov 03 00:40:58.965843 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.965798 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45qm2\" (UniqueName: \"kubernetes.io/projected/29de0f7e-9c96-46dd-8fec-434e3689a047-kube-api-access-45qm2\") pod \"ovnkube-node-zq57z\" (UID: \"29de0f7e-9c96-46dd-8fec-434e3689a047\") " pod="openshift-ovn-kubernetes/ovnkube-node-zq57z" Nov 03 00:40:58.965926 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.965900 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjg6m\" (UniqueName: \"kubernetes.io/projected/76ccef27-d056-402e-a7e1-ee524d63bf35-kube-api-access-rjg6m\") pod \"ingress-canary-whbrp\" (UID: \"76ccef27-d056-402e-a7e1-ee524d63bf35\") " pod="openshift-ingress-canary/ingress-canary-whbrp" Nov 03 00:40:58.966070 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.966054 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qn5dn\" (UniqueName: \"kubernetes.io/projected/5844f274-ad38-42a3-9900-b051b01c9953-kube-api-access-qn5dn\") pod \"node-ca-w8k5t\" (UID: \"5844f274-ad38-42a3-9900-b051b01c9953\") " pod="openshift-image-registry/node-ca-w8k5t" Nov 03 00:40:58.966116 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.966103 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4c5b\" (UniqueName: \"kubernetes.io/projected/b2d13929-be23-4396-8445-c3eee12e9b9d-kube-api-access-h4c5b\") pod \"multus-lb46r\" (UID: \"b2d13929-be23-4396-8445-c3eee12e9b9d\") " pod="openshift-multus/multus-lb46r" Nov 03 00:40:58.966267 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.966251 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5fls\" (UniqueName: \"kubernetes.io/projected/174d7fa2-7163-47a8-a047-8727cdb3cb9f-kube-api-access-c5fls\") pod \"iptables-alerter-9w99c\" (UID: \"174d7fa2-7163-47a8-a047-8727cdb3cb9f\") " pod="openshift-network-operator/iptables-alerter-9w99c" Nov 03 00:40:58.966836 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.966785 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4svt5\" (UniqueName: \"kubernetes.io/projected/9c505674-e66b-4573-bbf5-7b6acf6684e6-kube-api-access-4svt5\") pod \"dns-default-rx2xs\" (UID: \"9c505674-e66b-4573-bbf5-7b6acf6684e6\") " pod="openshift-dns/dns-default-rx2xs" Nov 03 00:40:58.966969 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.966952 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqbgf\" (UniqueName: \"kubernetes.io/projected/476ff8a0-f214-4e5e-aa90-8a505974f963-kube-api-access-lqbgf\") pod \"tuned-n7ksd\" (UID: \"476ff8a0-f214-4e5e-aa90-8a505974f963\") " pod="openshift-cluster-node-tuning-operator/tuned-n7ksd" Nov 03 00:40:58.967489 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:58.967474 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsbnr\" (UniqueName: \"kubernetes.io/projected/18afc5dd-bc60-4794-91ab-b4019be56a61-kube-api-access-wsbnr\") pod \"multus-additional-cni-plugins-8rxnt\" (UID: \"18afc5dd-bc60-4794-91ab-b4019be56a61\") " pod="openshift-multus/multus-additional-cni-plugins-8rxnt" Nov 03 00:40:59.057816 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:59.057775 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/29de0f7e-9c96-46dd-8fec-434e3689a047-host-slash\") pod \"ovnkube-node-zq57z\" (UID: \"29de0f7e-9c96-46dd-8fec-434e3689a047\") " pod="openshift-ovn-kubernetes/ovnkube-node-zq57z" Nov 03 00:40:59.057816 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:59.057822 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b2d13929-be23-4396-8445-c3eee12e9b9d-hostroot\") pod \"multus-lb46r\" (UID: \"b2d13929-be23-4396-8445-c3eee12e9b9d\") " pod="openshift-multus/multus-lb46r" Nov 03 00:40:59.058010 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:59.057845 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5844f274-ad38-42a3-9900-b051b01c9953-host\") pod \"node-ca-w8k5t\" (UID: \"5844f274-ad38-42a3-9900-b051b01c9953\") " pod="openshift-image-registry/node-ca-w8k5t" Nov 03 00:40:59.058010 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:59.057912 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b2d13929-be23-4396-8445-c3eee12e9b9d-hostroot\") pod \"multus-lb46r\" (UID: \"b2d13929-be23-4396-8445-c3eee12e9b9d\") " pod="openshift-multus/multus-lb46r" Nov 03 00:40:59.058010 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:59.057982 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5844f274-ad38-42a3-9900-b051b01c9953-host\") pod \"node-ca-w8k5t\" (UID: \"5844f274-ad38-42a3-9900-b051b01c9953\") " pod="openshift-image-registry/node-ca-w8k5t" Nov 03 00:40:59.058151 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:59.058025 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/bac578ef-a88b-4a74-9a65-bebc8ea1d00e-crio-socket\") pod \"insights-runtime-extractor-zskmb\" (UID: \"bac578ef-a88b-4a74-9a65-bebc8ea1d00e\") " pod="openshift-insights/insights-runtime-extractor-zskmb" Nov 03 00:40:59.058151 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:59.058045 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/476ff8a0-f214-4e5e-aa90-8a505974f963-etc-kubernetes\") pod \"tuned-n7ksd\" (UID: \"476ff8a0-f214-4e5e-aa90-8a505974f963\") " pod="openshift-cluster-node-tuning-operator/tuned-n7ksd" Nov 03 00:40:59.058151 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:59.058063 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6ab5d408-02a2-4284-bf59-5827c6f84643-socket-dir\") pod \"aws-ebs-csi-driver-node-trgdf\" (UID: \"6ab5d408-02a2-4284-bf59-5827c6f84643\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-trgdf" Nov 03 00:40:59.058151 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:59.058077 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/29de0f7e-9c96-46dd-8fec-434e3689a047-etc-openvswitch\") pod \"ovnkube-node-zq57z\" (UID: \"29de0f7e-9c96-46dd-8fec-434e3689a047\") " pod="openshift-ovn-kubernetes/ovnkube-node-zq57z" Nov 03 00:40:59.058151 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:59.058091 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b2d13929-be23-4396-8445-c3eee12e9b9d-system-cni-dir\") pod \"multus-lb46r\" (UID: \"b2d13929-be23-4396-8445-c3eee12e9b9d\") " pod="openshift-multus/multus-lb46r" Nov 03 00:40:59.058151 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:59.058107 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/476ff8a0-f214-4e5e-aa90-8a505974f963-etc-sysctl-d\") pod \"tuned-n7ksd\" (UID: \"476ff8a0-f214-4e5e-aa90-8a505974f963\") " pod="openshift-cluster-node-tuning-operator/tuned-n7ksd" Nov 03 00:40:59.058151 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:59.058116 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/29de0f7e-9c96-46dd-8fec-434e3689a047-host-slash\") pod \"ovnkube-node-zq57z\" (UID: \"29de0f7e-9c96-46dd-8fec-434e3689a047\") " pod="openshift-ovn-kubernetes/ovnkube-node-zq57z" Nov 03 00:40:59.058151 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:59.058129 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/bac578ef-a88b-4a74-9a65-bebc8ea1d00e-crio-socket\") pod \"insights-runtime-extractor-zskmb\" (UID: \"bac578ef-a88b-4a74-9a65-bebc8ea1d00e\") " pod="openshift-insights/insights-runtime-extractor-zskmb" Nov 03 00:40:59.058151 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:59.058154 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/18afc5dd-bc60-4794-91ab-b4019be56a61-system-cni-dir\") pod \"multus-additional-cni-plugins-8rxnt\" (UID: \"18afc5dd-bc60-4794-91ab-b4019be56a61\") " pod="openshift-multus/multus-additional-cni-plugins-8rxnt" Nov 03 00:40:59.058607 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:59.058173 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b2d13929-be23-4396-8445-c3eee12e9b9d-multus-conf-dir\") pod \"multus-lb46r\" (UID: \"b2d13929-be23-4396-8445-c3eee12e9b9d\") " pod="openshift-multus/multus-lb46r" Nov 03 00:40:59.058607 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:59.058190 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/29de0f7e-9c96-46dd-8fec-434e3689a047-run-openvswitch\") pod \"ovnkube-node-zq57z\" (UID: \"29de0f7e-9c96-46dd-8fec-434e3689a047\") " pod="openshift-ovn-kubernetes/ovnkube-node-zq57z" Nov 03 00:40:59.058607 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:59.058188 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6ab5d408-02a2-4284-bf59-5827c6f84643-socket-dir\") pod \"aws-ebs-csi-driver-node-trgdf\" (UID: \"6ab5d408-02a2-4284-bf59-5827c6f84643\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-trgdf" Nov 03 00:40:59.058607 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:59.058209 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/476ff8a0-f214-4e5e-aa90-8a505974f963-etc-modprobe-d\") pod \"tuned-n7ksd\" (UID: \"476ff8a0-f214-4e5e-aa90-8a505974f963\") " pod="openshift-cluster-node-tuning-operator/tuned-n7ksd" Nov 03 00:40:59.058607 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:59.058246 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/18afc5dd-bc60-4794-91ab-b4019be56a61-system-cni-dir\") pod \"multus-additional-cni-plugins-8rxnt\" (UID: \"18afc5dd-bc60-4794-91ab-b4019be56a61\") " pod="openshift-multus/multus-additional-cni-plugins-8rxnt" Nov 03 00:40:59.058607 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:59.058252 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/29de0f7e-9c96-46dd-8fec-434e3689a047-host-kubelet\") pod \"ovnkube-node-zq57z\" (UID: \"29de0f7e-9c96-46dd-8fec-434e3689a047\") " pod="openshift-ovn-kubernetes/ovnkube-node-zq57z" Nov 03 00:40:59.058607 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:59.058292 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/476ff8a0-f214-4e5e-aa90-8a505974f963-etc-sysctl-d\") pod \"tuned-n7ksd\" (UID: \"476ff8a0-f214-4e5e-aa90-8a505974f963\") " pod="openshift-cluster-node-tuning-operator/tuned-n7ksd" Nov 03 00:40:59.058607 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:59.058305 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/29de0f7e-9c96-46dd-8fec-434e3689a047-host-kubelet\") pod \"ovnkube-node-zq57z\" (UID: \"29de0f7e-9c96-46dd-8fec-434e3689a047\") " pod="openshift-ovn-kubernetes/ovnkube-node-zq57z" Nov 03 00:40:59.058607 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:59.058314 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6ab5d408-02a2-4284-bf59-5827c6f84643-registration-dir\") pod \"aws-ebs-csi-driver-node-trgdf\" (UID: \"6ab5d408-02a2-4284-bf59-5827c6f84643\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-trgdf" Nov 03 00:40:59.058607 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:59.058321 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/476ff8a0-f214-4e5e-aa90-8a505974f963-etc-modprobe-d\") pod \"tuned-n7ksd\" (UID: \"476ff8a0-f214-4e5e-aa90-8a505974f963\") " pod="openshift-cluster-node-tuning-operator/tuned-n7ksd" Nov 03 00:40:59.058607 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:59.058262 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/29de0f7e-9c96-46dd-8fec-434e3689a047-run-openvswitch\") pod \"ovnkube-node-zq57z\" (UID: \"29de0f7e-9c96-46dd-8fec-434e3689a047\") " pod="openshift-ovn-kubernetes/ovnkube-node-zq57z" Nov 03 00:40:59.058607 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:59.058322 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b2d13929-be23-4396-8445-c3eee12e9b9d-multus-conf-dir\") pod \"multus-lb46r\" (UID: \"b2d13929-be23-4396-8445-c3eee12e9b9d\") " pod="openshift-multus/multus-lb46r" Nov 03 00:40:59.058607 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:59.058297 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/29de0f7e-9c96-46dd-8fec-434e3689a047-etc-openvswitch\") pod \"ovnkube-node-zq57z\" (UID: \"29de0f7e-9c96-46dd-8fec-434e3689a047\") " pod="openshift-ovn-kubernetes/ovnkube-node-zq57z" Nov 03 00:40:59.058607 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:59.058343 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6ab5d408-02a2-4284-bf59-5827c6f84643-sys-fs\") pod \"aws-ebs-csi-driver-node-trgdf\" (UID: \"6ab5d408-02a2-4284-bf59-5827c6f84643\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-trgdf" Nov 03 00:40:59.058607 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:59.058352 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/476ff8a0-f214-4e5e-aa90-8a505974f963-etc-kubernetes\") pod \"tuned-n7ksd\" (UID: \"476ff8a0-f214-4e5e-aa90-8a505974f963\") " pod="openshift-cluster-node-tuning-operator/tuned-n7ksd" Nov 03 00:40:59.058607 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:59.058366 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6ab5d408-02a2-4284-bf59-5827c6f84643-registration-dir\") pod \"aws-ebs-csi-driver-node-trgdf\" (UID: \"6ab5d408-02a2-4284-bf59-5827c6f84643\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-trgdf" Nov 03 00:40:59.058607 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:59.058403 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bac578ef-a88b-4a74-9a65-bebc8ea1d00e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zskmb\" (UID: \"bac578ef-a88b-4a74-9a65-bebc8ea1d00e\") " pod="openshift-insights/insights-runtime-extractor-zskmb" Nov 03 00:40:59.058607 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:59.058419 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6ab5d408-02a2-4284-bf59-5827c6f84643-sys-fs\") pod \"aws-ebs-csi-driver-node-trgdf\" (UID: \"6ab5d408-02a2-4284-bf59-5827c6f84643\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-trgdf" Nov 03 00:40:59.058607 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:59.058464 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/29de0f7e-9c96-46dd-8fec-434e3689a047-run-systemd\") pod \"ovnkube-node-zq57z\" (UID: \"29de0f7e-9c96-46dd-8fec-434e3689a047\") " pod="openshift-ovn-kubernetes/ovnkube-node-zq57z" Nov 03 00:40:59.058607 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:59.058430 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b2d13929-be23-4396-8445-c3eee12e9b9d-system-cni-dir\") pod \"multus-lb46r\" (UID: \"b2d13929-be23-4396-8445-c3eee12e9b9d\") " pod="openshift-multus/multus-lb46r" Nov 03 00:40:59.058607 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:59.058502 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/29de0f7e-9c96-46dd-8fec-434e3689a047-run-systemd\") pod \"ovnkube-node-zq57z\" (UID: \"29de0f7e-9c96-46dd-8fec-434e3689a047\") " pod="openshift-ovn-kubernetes/ovnkube-node-zq57z" Nov 03 00:40:59.058607 ip-10-0-133-217 kubenswrapper[4421]: E1103 00:40:59.058506 4421 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 00:40:59.058607 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:59.058497 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/29de0f7e-9c96-46dd-8fec-434e3689a047-run-ovn\") pod \"ovnkube-node-zq57z\" (UID: \"29de0f7e-9c96-46dd-8fec-434e3689a047\") " pod="openshift-ovn-kubernetes/ovnkube-node-zq57z" Nov 03 00:40:59.058607 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:59.058538 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/29de0f7e-9c96-46dd-8fec-434e3689a047-run-ovn\") pod \"ovnkube-node-zq57z\" (UID: \"29de0f7e-9c96-46dd-8fec-434e3689a047\") " pod="openshift-ovn-kubernetes/ovnkube-node-zq57z" Nov 03 00:40:59.058607 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:59.058609 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/174d7fa2-7163-47a8-a047-8727cdb3cb9f-host-slash\") pod \"iptables-alerter-9w99c\" (UID: \"174d7fa2-7163-47a8-a047-8727cdb3cb9f\") " pod="openshift-network-operator/iptables-alerter-9w99c" Nov 03 00:40:59.059459 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:59.058638 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b2d13929-be23-4396-8445-c3eee12e9b9d-host-var-lib-cni-multus\") pod \"multus-lb46r\" (UID: \"b2d13929-be23-4396-8445-c3eee12e9b9d\") " pod="openshift-multus/multus-lb46r" Nov 03 00:40:59.059459 ip-10-0-133-217 kubenswrapper[4421]: E1103 00:40:59.058678 4421 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bac578ef-a88b-4a74-9a65-bebc8ea1d00e-insights-runtime-extractor-tls podName:bac578ef-a88b-4a74-9a65-bebc8ea1d00e nodeName:}" failed. No retries permitted until 2026-11-03 00:40:59.558653917 +0000 UTC m=+1.729657014 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/bac578ef-a88b-4a74-9a65-bebc8ea1d00e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zskmb" (UID: "bac578ef-a88b-4a74-9a65-bebc8ea1d00e") : secret "insights-runtime-extractor-tls" not found Nov 03 00:40:59.059459 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:59.058681 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b2d13929-be23-4396-8445-c3eee12e9b9d-host-var-lib-cni-multus\") pod \"multus-lb46r\" (UID: \"b2d13929-be23-4396-8445-c3eee12e9b9d\") " pod="openshift-multus/multus-lb46r" Nov 03 00:40:59.059459 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:59.058692 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/174d7fa2-7163-47a8-a047-8727cdb3cb9f-host-slash\") pod \"iptables-alerter-9w99c\" (UID: \"174d7fa2-7163-47a8-a047-8727cdb3cb9f\") " pod="openshift-network-operator/iptables-alerter-9w99c" Nov 03 00:40:59.059459 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:59.058697 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6ab5d408-02a2-4284-bf59-5827c6f84643-device-dir\") pod \"aws-ebs-csi-driver-node-trgdf\" (UID: \"6ab5d408-02a2-4284-bf59-5827c6f84643\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-trgdf" Nov 03 00:40:59.059459 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:59.058728 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6ab5d408-02a2-4284-bf59-5827c6f84643-device-dir\") pod \"aws-ebs-csi-driver-node-trgdf\" (UID: \"6ab5d408-02a2-4284-bf59-5827c6f84643\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-trgdf" Nov 03 00:40:59.059459 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:59.058732 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b2d13929-be23-4396-8445-c3eee12e9b9d-host-run-netns\") pod \"multus-lb46r\" (UID: \"b2d13929-be23-4396-8445-c3eee12e9b9d\") " pod="openshift-multus/multus-lb46r" Nov 03 00:40:59.059459 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:59.058765 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b2d13929-be23-4396-8445-c3eee12e9b9d-host-run-netns\") pod \"multus-lb46r\" (UID: \"b2d13929-be23-4396-8445-c3eee12e9b9d\") " pod="openshift-multus/multus-lb46r" Nov 03 00:40:59.059459 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:59.058764 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/92341144-23c4-4cf0-a0c8-26e1891397e9-kubelet-config\") pod \"global-pull-secret-syncer-262mm\" (UID: \"92341144-23c4-4cf0-a0c8-26e1891397e9\") " pod="kube-system/global-pull-secret-syncer-262mm" Nov 03 00:40:59.059459 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:59.058788 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/92341144-23c4-4cf0-a0c8-26e1891397e9-dbus\") pod \"global-pull-secret-syncer-262mm\" (UID: \"92341144-23c4-4cf0-a0c8-26e1891397e9\") " pod="kube-system/global-pull-secret-syncer-262mm" Nov 03 00:40:59.059459 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:59.058804 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/476ff8a0-f214-4e5e-aa90-8a505974f963-etc-sysctl-conf\") pod \"tuned-n7ksd\" (UID: \"476ff8a0-f214-4e5e-aa90-8a505974f963\") " pod="openshift-cluster-node-tuning-operator/tuned-n7ksd" Nov 03 00:40:59.059459 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:59.058809 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/92341144-23c4-4cf0-a0c8-26e1891397e9-kubelet-config\") pod \"global-pull-secret-syncer-262mm\" (UID: \"92341144-23c4-4cf0-a0c8-26e1891397e9\") " pod="kube-system/global-pull-secret-syncer-262mm" Nov 03 00:40:59.059459 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:59.058822 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6ab5d408-02a2-4284-bf59-5827c6f84643-kubelet-dir\") pod \"aws-ebs-csi-driver-node-trgdf\" (UID: \"6ab5d408-02a2-4284-bf59-5827c6f84643\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-trgdf" Nov 03 00:40:59.059459 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:59.058848 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/29de0f7e-9c96-46dd-8fec-434e3689a047-host-cni-netd\") pod \"ovnkube-node-zq57z\" (UID: \"29de0f7e-9c96-46dd-8fec-434e3689a047\") " pod="openshift-ovn-kubernetes/ovnkube-node-zq57z" Nov 03 00:40:59.059459 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:59.058869 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/92341144-23c4-4cf0-a0c8-26e1891397e9-dbus\") pod \"global-pull-secret-syncer-262mm\" (UID: \"92341144-23c4-4cf0-a0c8-26e1891397e9\") " pod="kube-system/global-pull-secret-syncer-262mm" Nov 03 00:40:59.059459 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:59.058882 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b2d13929-be23-4396-8445-c3eee12e9b9d-multus-socket-dir-parent\") pod \"multus-lb46r\" (UID: \"b2d13929-be23-4396-8445-c3eee12e9b9d\") " pod="openshift-multus/multus-lb46r" Nov 03 00:40:59.059459 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:59.058912 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b2d13929-be23-4396-8445-c3eee12e9b9d-etc-kubernetes\") pod \"multus-lb46r\" (UID: \"b2d13929-be23-4396-8445-c3eee12e9b9d\") " pod="openshift-multus/multus-lb46r" Nov 03 00:40:59.059459 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:59.058928 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6ab5d408-02a2-4284-bf59-5827c6f84643-kubelet-dir\") pod \"aws-ebs-csi-driver-node-trgdf\" (UID: \"6ab5d408-02a2-4284-bf59-5827c6f84643\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-trgdf" Nov 03 00:40:59.059459 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:59.058935 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/476ff8a0-f214-4e5e-aa90-8a505974f963-etc-sysconfig\") pod \"tuned-n7ksd\" (UID: \"476ff8a0-f214-4e5e-aa90-8a505974f963\") " pod="openshift-cluster-node-tuning-operator/tuned-n7ksd" Nov 03 00:40:59.059459 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:59.058961 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b2d13929-be23-4396-8445-c3eee12e9b9d-multus-socket-dir-parent\") pod \"multus-lb46r\" (UID: \"b2d13929-be23-4396-8445-c3eee12e9b9d\") " pod="openshift-multus/multus-lb46r" Nov 03 00:40:59.059459 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:59.058971 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/476ff8a0-f214-4e5e-aa90-8a505974f963-etc-systemd\") pod \"tuned-n7ksd\" (UID: \"476ff8a0-f214-4e5e-aa90-8a505974f963\") " pod="openshift-cluster-node-tuning-operator/tuned-n7ksd" Nov 03 00:40:59.059459 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:59.058977 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/476ff8a0-f214-4e5e-aa90-8a505974f963-etc-sysconfig\") pod \"tuned-n7ksd\" (UID: \"476ff8a0-f214-4e5e-aa90-8a505974f963\") " pod="openshift-cluster-node-tuning-operator/tuned-n7ksd" Nov 03 00:40:59.059459 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:59.058999 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/29de0f7e-9c96-46dd-8fec-434e3689a047-log-socket\") pod \"ovnkube-node-zq57z\" (UID: \"29de0f7e-9c96-46dd-8fec-434e3689a047\") " pod="openshift-ovn-kubernetes/ovnkube-node-zq57z" Nov 03 00:40:59.059459 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:59.059009 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/29de0f7e-9c96-46dd-8fec-434e3689a047-host-cni-netd\") pod \"ovnkube-node-zq57z\" (UID: \"29de0f7e-9c96-46dd-8fec-434e3689a047\") " pod="openshift-ovn-kubernetes/ovnkube-node-zq57z" Nov 03 00:40:59.059459 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:59.059011 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b2d13929-be23-4396-8445-c3eee12e9b9d-etc-kubernetes\") pod \"multus-lb46r\" (UID: \"b2d13929-be23-4396-8445-c3eee12e9b9d\") " pod="openshift-multus/multus-lb46r" Nov 03 00:40:59.059459 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:59.059023 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b2d13929-be23-4396-8445-c3eee12e9b9d-multus-cni-dir\") pod \"multus-lb46r\" (UID: \"b2d13929-be23-4396-8445-c3eee12e9b9d\") " pod="openshift-multus/multus-lb46r" Nov 03 00:40:59.059459 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:59.058942 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/476ff8a0-f214-4e5e-aa90-8a505974f963-etc-sysctl-conf\") pod \"tuned-n7ksd\" (UID: \"476ff8a0-f214-4e5e-aa90-8a505974f963\") " pod="openshift-cluster-node-tuning-operator/tuned-n7ksd" Nov 03 00:40:59.059459 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:59.059012 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/476ff8a0-f214-4e5e-aa90-8a505974f963-etc-systemd\") pod \"tuned-n7ksd\" (UID: \"476ff8a0-f214-4e5e-aa90-8a505974f963\") " pod="openshift-cluster-node-tuning-operator/tuned-n7ksd" Nov 03 00:40:59.059459 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:59.059042 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/29de0f7e-9c96-46dd-8fec-434e3689a047-log-socket\") pod \"ovnkube-node-zq57z\" (UID: \"29de0f7e-9c96-46dd-8fec-434e3689a047\") " pod="openshift-ovn-kubernetes/ovnkube-node-zq57z" Nov 03 00:40:59.059459 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:59.059055 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b2d13929-be23-4396-8445-c3eee12e9b9d-host-var-lib-cni-bin\") pod \"multus-lb46r\" (UID: \"b2d13929-be23-4396-8445-c3eee12e9b9d\") " pod="openshift-multus/multus-lb46r" Nov 03 00:40:59.059459 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:59.059082 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b2d13929-be23-4396-8445-c3eee12e9b9d-multus-cni-dir\") pod \"multus-lb46r\" (UID: \"b2d13929-be23-4396-8445-c3eee12e9b9d\") " pod="openshift-multus/multus-lb46r" Nov 03 00:40:59.059459 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:59.059082 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/476ff8a0-f214-4e5e-aa90-8a505974f963-sys\") pod \"tuned-n7ksd\" (UID: \"476ff8a0-f214-4e5e-aa90-8a505974f963\") " pod="openshift-cluster-node-tuning-operator/tuned-n7ksd" Nov 03 00:40:59.059459 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:59.059106 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b2d13929-be23-4396-8445-c3eee12e9b9d-host-var-lib-cni-bin\") pod \"multus-lb46r\" (UID: \"b2d13929-be23-4396-8445-c3eee12e9b9d\") " pod="openshift-multus/multus-lb46r" Nov 03 00:40:59.059459 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:59.059114 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/476ff8a0-f214-4e5e-aa90-8a505974f963-lib-modules\") pod \"tuned-n7ksd\" (UID: \"476ff8a0-f214-4e5e-aa90-8a505974f963\") " pod="openshift-cluster-node-tuning-operator/tuned-n7ksd" Nov 03 00:40:59.059459 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:59.059136 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/476ff8a0-f214-4e5e-aa90-8a505974f963-sys\") pod \"tuned-n7ksd\" (UID: \"476ff8a0-f214-4e5e-aa90-8a505974f963\") " pod="openshift-cluster-node-tuning-operator/tuned-n7ksd" Nov 03 00:40:59.059459 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:59.059135 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/18afc5dd-bc60-4794-91ab-b4019be56a61-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8rxnt\" (UID: \"18afc5dd-bc60-4794-91ab-b4019be56a61\") " pod="openshift-multus/multus-additional-cni-plugins-8rxnt" Nov 03 00:40:59.059459 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:59.059181 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/29de0f7e-9c96-46dd-8fec-434e3689a047-systemd-units\") pod \"ovnkube-node-zq57z\" (UID: \"29de0f7e-9c96-46dd-8fec-434e3689a047\") " pod="openshift-ovn-kubernetes/ovnkube-node-zq57z" Nov 03 00:40:59.059459 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:59.059198 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/29de0f7e-9c96-46dd-8fec-434e3689a047-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zq57z\" (UID: \"29de0f7e-9c96-46dd-8fec-434e3689a047\") " pod="openshift-ovn-kubernetes/ovnkube-node-zq57z" Nov 03 00:40:59.059459 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:59.059220 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/476ff8a0-f214-4e5e-aa90-8a505974f963-var-lib-kubelet\") pod \"tuned-n7ksd\" (UID: \"476ff8a0-f214-4e5e-aa90-8a505974f963\") " pod="openshift-cluster-node-tuning-operator/tuned-n7ksd" Nov 03 00:40:59.059459 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:59.059232 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/29de0f7e-9c96-46dd-8fec-434e3689a047-systemd-units\") pod \"ovnkube-node-zq57z\" (UID: \"29de0f7e-9c96-46dd-8fec-434e3689a047\") " pod="openshift-ovn-kubernetes/ovnkube-node-zq57z" Nov 03 00:40:59.059459 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:59.059234 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/18afc5dd-bc60-4794-91ab-b4019be56a61-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8rxnt\" (UID: \"18afc5dd-bc60-4794-91ab-b4019be56a61\") " pod="openshift-multus/multus-additional-cni-plugins-8rxnt" Nov 03 00:40:59.059459 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:59.059245 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/29de0f7e-9c96-46dd-8fec-434e3689a047-var-lib-openvswitch\") pod \"ovnkube-node-zq57z\" (UID: \"29de0f7e-9c96-46dd-8fec-434e3689a047\") " pod="openshift-ovn-kubernetes/ovnkube-node-zq57z" Nov 03 00:40:59.059459 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:59.059256 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/29de0f7e-9c96-46dd-8fec-434e3689a047-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zq57z\" (UID: \"29de0f7e-9c96-46dd-8fec-434e3689a047\") " pod="openshift-ovn-kubernetes/ovnkube-node-zq57z" Nov 03 00:40:59.059459 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:59.059263 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/476ff8a0-f214-4e5e-aa90-8a505974f963-lib-modules\") pod \"tuned-n7ksd\" (UID: \"476ff8a0-f214-4e5e-aa90-8a505974f963\") " pod="openshift-cluster-node-tuning-operator/tuned-n7ksd" Nov 03 00:40:59.059459 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:59.059294 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/29de0f7e-9c96-46dd-8fec-434e3689a047-var-lib-openvswitch\") pod \"ovnkube-node-zq57z\" (UID: \"29de0f7e-9c96-46dd-8fec-434e3689a047\") " pod="openshift-ovn-kubernetes/ovnkube-node-zq57z" Nov 03 00:40:59.059459 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:59.059293 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/18afc5dd-bc60-4794-91ab-b4019be56a61-cnibin\") pod \"multus-additional-cni-plugins-8rxnt\" (UID: \"18afc5dd-bc60-4794-91ab-b4019be56a61\") " pod="openshift-multus/multus-additional-cni-plugins-8rxnt" Nov 03 00:40:59.059459 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:59.059300 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/476ff8a0-f214-4e5e-aa90-8a505974f963-var-lib-kubelet\") pod \"tuned-n7ksd\" (UID: \"476ff8a0-f214-4e5e-aa90-8a505974f963\") " pod="openshift-cluster-node-tuning-operator/tuned-n7ksd" Nov 03 00:40:59.059459 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:59.059318 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/18afc5dd-bc60-4794-91ab-b4019be56a61-cnibin\") pod \"multus-additional-cni-plugins-8rxnt\" (UID: \"18afc5dd-bc60-4794-91ab-b4019be56a61\") " pod="openshift-multus/multus-additional-cni-plugins-8rxnt" Nov 03 00:40:59.059459 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:59.059318 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b2d13929-be23-4396-8445-c3eee12e9b9d-host-run-multus-certs\") pod \"multus-lb46r\" (UID: \"b2d13929-be23-4396-8445-c3eee12e9b9d\") " pod="openshift-multus/multus-lb46r" Nov 03 00:40:59.059459 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:59.059336 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b2d13929-be23-4396-8445-c3eee12e9b9d-host-run-multus-certs\") pod \"multus-lb46r\" (UID: \"b2d13929-be23-4396-8445-c3eee12e9b9d\") " pod="openshift-multus/multus-lb46r" Nov 03 00:40:59.059459 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:59.059343 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6ab5d408-02a2-4284-bf59-5827c6f84643-etc-selinux\") pod \"aws-ebs-csi-driver-node-trgdf\" (UID: \"6ab5d408-02a2-4284-bf59-5827c6f84643\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-trgdf" Nov 03 00:40:59.059459 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:59.059363 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/29de0f7e-9c96-46dd-8fec-434e3689a047-node-log\") pod \"ovnkube-node-zq57z\" (UID: \"29de0f7e-9c96-46dd-8fec-434e3689a047\") " pod="openshift-ovn-kubernetes/ovnkube-node-zq57z" Nov 03 00:40:59.059459 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:59.059383 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/29de0f7e-9c96-46dd-8fec-434e3689a047-host-cni-bin\") pod \"ovnkube-node-zq57z\" (UID: \"29de0f7e-9c96-46dd-8fec-434e3689a047\") " pod="openshift-ovn-kubernetes/ovnkube-node-zq57z" Nov 03 00:40:59.059459 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:59.059397 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b2d13929-be23-4396-8445-c3eee12e9b9d-cnibin\") pod \"multus-lb46r\" (UID: \"b2d13929-be23-4396-8445-c3eee12e9b9d\") " pod="openshift-multus/multus-lb46r" Nov 03 00:40:59.059459 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:59.059421 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b2d13929-be23-4396-8445-c3eee12e9b9d-host-run-k8s-cni-cncf-io\") pod \"multus-lb46r\" (UID: \"b2d13929-be23-4396-8445-c3eee12e9b9d\") " pod="openshift-multus/multus-lb46r" Nov 03 00:40:59.059459 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:59.059423 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6ab5d408-02a2-4284-bf59-5827c6f84643-etc-selinux\") pod \"aws-ebs-csi-driver-node-trgdf\" (UID: \"6ab5d408-02a2-4284-bf59-5827c6f84643\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-trgdf" Nov 03 00:40:59.059459 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:59.059442 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b2d13929-be23-4396-8445-c3eee12e9b9d-host-var-lib-kubelet\") pod \"multus-lb46r\" (UID: \"b2d13929-be23-4396-8445-c3eee12e9b9d\") " pod="openshift-multus/multus-lb46r" Nov 03 00:40:59.059459 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:59.059448 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/29de0f7e-9c96-46dd-8fec-434e3689a047-host-cni-bin\") pod \"ovnkube-node-zq57z\" (UID: \"29de0f7e-9c96-46dd-8fec-434e3689a047\") " pod="openshift-ovn-kubernetes/ovnkube-node-zq57z" Nov 03 00:40:59.059459 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:59.059468 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/476ff8a0-f214-4e5e-aa90-8a505974f963-host\") pod \"tuned-n7ksd\" (UID: \"476ff8a0-f214-4e5e-aa90-8a505974f963\") " pod="openshift-cluster-node-tuning-operator/tuned-n7ksd" Nov 03 00:40:59.059459 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:59.059478 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/29de0f7e-9c96-46dd-8fec-434e3689a047-node-log\") pod \"ovnkube-node-zq57z\" (UID: \"29de0f7e-9c96-46dd-8fec-434e3689a047\") " pod="openshift-ovn-kubernetes/ovnkube-node-zq57z" Nov 03 00:40:59.059459 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:59.059481 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b2d13929-be23-4396-8445-c3eee12e9b9d-host-run-k8s-cni-cncf-io\") pod \"multus-lb46r\" (UID: \"b2d13929-be23-4396-8445-c3eee12e9b9d\") " pod="openshift-multus/multus-lb46r" Nov 03 00:40:59.059459 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:59.059484 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b2d13929-be23-4396-8445-c3eee12e9b9d-cnibin\") pod \"multus-lb46r\" (UID: \"b2d13929-be23-4396-8445-c3eee12e9b9d\") " pod="openshift-multus/multus-lb46r" Nov 03 00:40:59.061161 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:59.059498 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/29de0f7e-9c96-46dd-8fec-434e3689a047-host-run-netns\") pod \"ovnkube-node-zq57z\" (UID: \"29de0f7e-9c96-46dd-8fec-434e3689a047\") " pod="openshift-ovn-kubernetes/ovnkube-node-zq57z" Nov 03 00:40:59.061161 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:59.059526 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b2d13929-be23-4396-8445-c3eee12e9b9d-host-var-lib-kubelet\") pod \"multus-lb46r\" (UID: \"b2d13929-be23-4396-8445-c3eee12e9b9d\") " pod="openshift-multus/multus-lb46r" Nov 03 00:40:59.061161 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:59.059542 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/29de0f7e-9c96-46dd-8fec-434e3689a047-host-run-ovn-kubernetes\") pod \"ovnkube-node-zq57z\" (UID: \"29de0f7e-9c96-46dd-8fec-434e3689a047\") " pod="openshift-ovn-kubernetes/ovnkube-node-zq57z" Nov 03 00:40:59.061161 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:59.059545 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/476ff8a0-f214-4e5e-aa90-8a505974f963-host\") pod \"tuned-n7ksd\" (UID: \"476ff8a0-f214-4e5e-aa90-8a505974f963\") " pod="openshift-cluster-node-tuning-operator/tuned-n7ksd" Nov 03 00:40:59.061161 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:59.059567 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b6abcc85-61c0-42ea-b868-0d71a46a4514-metrics-certs\") pod \"network-metrics-daemon-5v4k7\" (UID: \"b6abcc85-61c0-42ea-b868-0d71a46a4514\") " pod="openshift-multus/network-metrics-daemon-5v4k7" Nov 03 00:40:59.061161 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:59.059584 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/476ff8a0-f214-4e5e-aa90-8a505974f963-run\") pod \"tuned-n7ksd\" (UID: \"476ff8a0-f214-4e5e-aa90-8a505974f963\") " pod="openshift-cluster-node-tuning-operator/tuned-n7ksd" Nov 03 00:40:59.061161 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:59.059613 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/29de0f7e-9c96-46dd-8fec-434e3689a047-host-run-ovn-kubernetes\") pod \"ovnkube-node-zq57z\" (UID: \"29de0f7e-9c96-46dd-8fec-434e3689a047\") " pod="openshift-ovn-kubernetes/ovnkube-node-zq57z" Nov 03 00:40:59.061161 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:59.059612 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/29de0f7e-9c96-46dd-8fec-434e3689a047-host-run-netns\") pod \"ovnkube-node-zq57z\" (UID: \"29de0f7e-9c96-46dd-8fec-434e3689a047\") " pod="openshift-ovn-kubernetes/ovnkube-node-zq57z" Nov 03 00:40:59.061161 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:59.059620 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e7f1faa5-fc0e-4f46-9208-b1e58f50fcf9-hosts-file\") pod \"node-resolver-cphks\" (UID: \"e7f1faa5-fc0e-4f46-9208-b1e58f50fcf9\") " pod="openshift-dns/node-resolver-cphks" Nov 03 00:40:59.061161 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:59.059640 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/476ff8a0-f214-4e5e-aa90-8a505974f963-run\") pod \"tuned-n7ksd\" (UID: \"476ff8a0-f214-4e5e-aa90-8a505974f963\") " pod="openshift-cluster-node-tuning-operator/tuned-n7ksd" Nov 03 00:40:59.061161 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:59.059666 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e7f1faa5-fc0e-4f46-9208-b1e58f50fcf9-hosts-file\") pod \"node-resolver-cphks\" (UID: \"e7f1faa5-fc0e-4f46-9208-b1e58f50fcf9\") " pod="openshift-dns/node-resolver-cphks" Nov 03 00:40:59.061161 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:59.059900 4421 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 03 00:40:59.061941 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:59.061923 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b6abcc85-61c0-42ea-b868-0d71a46a4514-metrics-certs\") pod \"network-metrics-daemon-5v4k7\" (UID: \"b6abcc85-61c0-42ea-b868-0d71a46a4514\") " pod="openshift-multus/network-metrics-daemon-5v4k7" Nov 03 00:40:59.230084 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:59.230045 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5v4k7" Nov 03 00:40:59.233331 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:59.233313 4421 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 00:40:59.546006 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:59.545976 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-5v4k7"] Nov 03 00:40:59.549308 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:40:59.549254 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6abcc85_61c0_42ea_b868_0d71a46a4514.slice/crio-82a96994ad3b05302c72cfd72a5754f2a4416d989db7f7c5b283991e770dfdb2 WatchSource:0}: Error finding container 82a96994ad3b05302c72cfd72a5754f2a4416d989db7f7c5b283991e770dfdb2: Status 404 returned error can't find the container with id 82a96994ad3b05302c72cfd72a5754f2a4416d989db7f7c5b283991e770dfdb2 Nov 03 00:40:59.565193 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:40:59.565163 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bac578ef-a88b-4a74-9a65-bebc8ea1d00e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zskmb\" (UID: \"bac578ef-a88b-4a74-9a65-bebc8ea1d00e\") " pod="openshift-insights/insights-runtime-extractor-zskmb" Nov 03 00:40:59.565403 ip-10-0-133-217 kubenswrapper[4421]: E1103 00:40:59.565348 4421 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 00:40:59.565544 ip-10-0-133-217 kubenswrapper[4421]: E1103 00:40:59.565435 4421 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bac578ef-a88b-4a74-9a65-bebc8ea1d00e-insights-runtime-extractor-tls podName:bac578ef-a88b-4a74-9a65-bebc8ea1d00e nodeName:}" failed. No retries permitted until 2026-11-03 00:41:00.565411702 +0000 UTC m=+2.736414792 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/bac578ef-a88b-4a74-9a65-bebc8ea1d00e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zskmb" (UID: "bac578ef-a88b-4a74-9a65-bebc8ea1d00e") : secret "insights-runtime-extractor-tls" not found Nov 03 00:41:00.023361 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:00.023319 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rx2xs" event={"ID":"9c505674-e66b-4573-bbf5-7b6acf6684e6","Type":"ContainerStarted","Data":"b89559261ce708e153dff0f1310a721ec4999018e9474e9a8fbf90ce944f0c9a"} Nov 03 00:41:00.024461 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:00.024434 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5v4k7" event={"ID":"b6abcc85-61c0-42ea-b868-0d71a46a4514","Type":"ContainerStarted","Data":"82a96994ad3b05302c72cfd72a5754f2a4416d989db7f7c5b283991e770dfdb2"} Nov 03 00:41:00.573072 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:00.573035 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bac578ef-a88b-4a74-9a65-bebc8ea1d00e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zskmb\" (UID: \"bac578ef-a88b-4a74-9a65-bebc8ea1d00e\") " pod="openshift-insights/insights-runtime-extractor-zskmb" Nov 03 00:41:00.573222 ip-10-0-133-217 kubenswrapper[4421]: E1103 00:41:00.573158 4421 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 00:41:00.573222 ip-10-0-133-217 kubenswrapper[4421]: E1103 00:41:00.573213 4421 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bac578ef-a88b-4a74-9a65-bebc8ea1d00e-insights-runtime-extractor-tls podName:bac578ef-a88b-4a74-9a65-bebc8ea1d00e nodeName:}" failed. No retries permitted until 2026-11-03 00:41:02.573199268 +0000 UTC m=+4.744202342 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/bac578ef-a88b-4a74-9a65-bebc8ea1d00e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zskmb" (UID: "bac578ef-a88b-4a74-9a65-bebc8ea1d00e") : secret "insights-runtime-extractor-tls" not found Nov 03 00:41:01.029732 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:01.029704 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rx2xs" event={"ID":"9c505674-e66b-4573-bbf5-7b6acf6684e6","Type":"ContainerStarted","Data":"0b028603aaa624fb9f68c814ea21d79a4b674f3bbe3bb094f53734388e2051e8"} Nov 03 00:41:01.030127 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:01.029805 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-rx2xs" Nov 03 00:41:01.031481 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:01.031395 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5v4k7" event={"ID":"b6abcc85-61c0-42ea-b868-0d71a46a4514","Type":"ContainerStarted","Data":"5586f5e677d3b9d86aa19f65c51805878cdd43d0e207d6974a51df87ab03cf66"} Nov 03 00:41:01.545637 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:01.545601 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-6lvpt" Nov 03 00:41:02.036865 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:02.036822 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5v4k7" event={"ID":"b6abcc85-61c0-42ea-b868-0d71a46a4514","Type":"ContainerStarted","Data":"2d47e4d28b70b80f61dc5894b159b9e7d8876a02d0143bd7689fe40422cc31ce"} Nov 03 00:41:02.585870 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:02.585828 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bac578ef-a88b-4a74-9a65-bebc8ea1d00e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zskmb\" (UID: \"bac578ef-a88b-4a74-9a65-bebc8ea1d00e\") " pod="openshift-insights/insights-runtime-extractor-zskmb" Nov 03 00:41:02.588228 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:02.588191 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bac578ef-a88b-4a74-9a65-bebc8ea1d00e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zskmb\" (UID: \"bac578ef-a88b-4a74-9a65-bebc8ea1d00e\") " pod="openshift-insights/insights-runtime-extractor-zskmb" Nov 03 00:41:02.831731 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:02.831690 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-zskmb" Nov 03 00:41:03.101716 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:03.101643 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-zskmb"] Nov 03 00:41:03.104947 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:41:03.104922 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbac578ef_a88b_4a74_9a65_bebc8ea1d00e.slice/crio-01b605e530cd1e46e5e2590fa4a823bd52ba3b86ce22463be13c46ba2ee2520c WatchSource:0}: Error finding container 01b605e530cd1e46e5e2590fa4a823bd52ba3b86ce22463be13c46ba2ee2520c: Status 404 returned error can't find the container with id 01b605e530cd1e46e5e2590fa4a823bd52ba3b86ce22463be13c46ba2ee2520c Nov 03 00:41:03.290236 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:03.290210 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zq57z" Nov 03 00:41:04.046524 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:04.046427 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zskmb" event={"ID":"bac578ef-a88b-4a74-9a65-bebc8ea1d00e","Type":"ContainerStarted","Data":"5816eb48cfad902933bea861a47adc1d7eb40ed79c1d73074e430f5a24e2ceb7"} Nov 03 00:41:04.046524 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:04.046499 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zskmb" event={"ID":"bac578ef-a88b-4a74-9a65-bebc8ea1d00e","Type":"ContainerStarted","Data":"dbf12d96bcc2be44356805e2543185575715dd76c6cb1fd0e4b4511d9bfb7a2a"} Nov 03 00:41:04.046524 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:04.046522 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zskmb" event={"ID":"bac578ef-a88b-4a74-9a65-bebc8ea1d00e","Type":"ContainerStarted","Data":"01b605e530cd1e46e5e2590fa4a823bd52ba3b86ce22463be13c46ba2ee2520c"} Nov 03 00:41:06.054876 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:06.054840 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zskmb" event={"ID":"bac578ef-a88b-4a74-9a65-bebc8ea1d00e","Type":"ContainerStarted","Data":"2d27de39c4f3420716ebd3d8914fb0cf792cbd5d8cdcfc1ab7b1b8b5ac330e2d"} Nov 03 00:41:07.613202 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:07.613168 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zq57z" Nov 03 00:41:07.613620 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:07.613419 4421 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 00:41:09.894135 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:09.894103 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-bzcwg"] Nov 03 00:41:09.896039 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:09.896015 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-bzcwg" Nov 03 00:41:09.899266 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:09.899240 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-2496n"] Nov 03 00:41:09.901476 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:09.901442 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-2496n" Nov 03 00:41:09.921334 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:09.921224 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Nov 03 00:41:09.921334 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:09.921296 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Nov 03 00:41:09.922513 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:09.922471 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Nov 03 00:41:09.924464 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:09.924377 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-2496n"] Nov 03 00:41:09.924464 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:09.924437 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-bzcwg"] Nov 03 00:41:09.924808 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:09.924793 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-nxf4k\"" Nov 03 00:41:09.925754 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:09.925736 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-hwsvf\"" Nov 03 00:41:09.933102 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:09.933075 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w76fq\" (UniqueName: \"kubernetes.io/projected/a145389b-418b-4ef0-9c66-29c9d7786081-kube-api-access-w76fq\") pod \"downloads-6dd789c8b7-bzcwg\" (UID: \"a145389b-418b-4ef0-9c66-29c9d7786081\") " pod="openshift-console/downloads-6dd789c8b7-bzcwg" Nov 03 00:41:09.933348 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:09.933326 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a145389b-418b-4ef0-9c66-29c9d7786081-tmp\") pod \"downloads-6dd789c8b7-bzcwg\" (UID: \"a145389b-418b-4ef0-9c66-29c9d7786081\") " pod="openshift-console/downloads-6dd789c8b7-bzcwg" Nov 03 00:41:09.991970 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:09.991937 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5f895c599b-f8f2x"] Nov 03 00:41:09.994989 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:09.994970 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5f895c599b-f8f2x" Nov 03 00:41:10.005288 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:10.005247 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Nov 03 00:41:10.017534 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:10.017502 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5f895c599b-f8f2x"] Nov 03 00:41:10.022332 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:10.022304 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Nov 03 00:41:10.022332 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:10.022312 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-grbz8\"" Nov 03 00:41:10.022540 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:10.022379 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Nov 03 00:41:10.022540 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:10.022436 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Nov 03 00:41:10.033926 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:10.033906 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a145389b-418b-4ef0-9c66-29c9d7786081-tmp\") pod \"downloads-6dd789c8b7-bzcwg\" (UID: \"a145389b-418b-4ef0-9c66-29c9d7786081\") " pod="openshift-console/downloads-6dd789c8b7-bzcwg" Nov 03 00:41:10.033987 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:10.033940 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8x2rs\" (UniqueName: \"kubernetes.io/projected/46039de5-8eaf-4029-9cb6-ca23508092f8-kube-api-access-8x2rs\") pod \"image-registry-5f895c599b-f8f2x\" (UID: \"46039de5-8eaf-4029-9cb6-ca23508092f8\") " pod="openshift-image-registry/image-registry-5f895c599b-f8f2x" Nov 03 00:41:10.034029 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:10.033997 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/46039de5-8eaf-4029-9cb6-ca23508092f8-registry-tls\") pod \"image-registry-5f895c599b-f8f2x\" (UID: \"46039de5-8eaf-4029-9cb6-ca23508092f8\") " pod="openshift-image-registry/image-registry-5f895c599b-f8f2x" Nov 03 00:41:10.034068 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:10.034023 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/46039de5-8eaf-4029-9cb6-ca23508092f8-trusted-ca\") pod \"image-registry-5f895c599b-f8f2x\" (UID: \"46039de5-8eaf-4029-9cb6-ca23508092f8\") " pod="openshift-image-registry/image-registry-5f895c599b-f8f2x" Nov 03 00:41:10.034068 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:10.034052 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w76fq\" (UniqueName: \"kubernetes.io/projected/a145389b-418b-4ef0-9c66-29c9d7786081-kube-api-access-w76fq\") pod \"downloads-6dd789c8b7-bzcwg\" (UID: \"a145389b-418b-4ef0-9c66-29c9d7786081\") " pod="openshift-console/downloads-6dd789c8b7-bzcwg" Nov 03 00:41:10.034136 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:10.034089 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/e906bfeb-04c0-4465-9ec6-6c18aad596ae-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-2496n\" (UID: \"e906bfeb-04c0-4465-9ec6-6c18aad596ae\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-2496n" Nov 03 00:41:10.034178 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:10.034137 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/46039de5-8eaf-4029-9cb6-ca23508092f8-image-registry-private-configuration\") pod \"image-registry-5f895c599b-f8f2x\" (UID: \"46039de5-8eaf-4029-9cb6-ca23508092f8\") " pod="openshift-image-registry/image-registry-5f895c599b-f8f2x" Nov 03 00:41:10.034217 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:10.034176 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/46039de5-8eaf-4029-9cb6-ca23508092f8-ca-trust-extracted\") pod \"image-registry-5f895c599b-f8f2x\" (UID: \"46039de5-8eaf-4029-9cb6-ca23508092f8\") " pod="openshift-image-registry/image-registry-5f895c599b-f8f2x" Nov 03 00:41:10.034217 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:10.034197 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/46039de5-8eaf-4029-9cb6-ca23508092f8-bound-sa-token\") pod \"image-registry-5f895c599b-f8f2x\" (UID: \"46039de5-8eaf-4029-9cb6-ca23508092f8\") " pod="openshift-image-registry/image-registry-5f895c599b-f8f2x" Nov 03 00:41:10.034314 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:10.034217 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/46039de5-8eaf-4029-9cb6-ca23508092f8-installation-pull-secrets\") pod \"image-registry-5f895c599b-f8f2x\" (UID: \"46039de5-8eaf-4029-9cb6-ca23508092f8\") " pod="openshift-image-registry/image-registry-5f895c599b-f8f2x" Nov 03 00:41:10.034350 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:10.034326 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/46039de5-8eaf-4029-9cb6-ca23508092f8-registry-certificates\") pod \"image-registry-5f895c599b-f8f2x\" (UID: \"46039de5-8eaf-4029-9cb6-ca23508092f8\") " pod="openshift-image-registry/image-registry-5f895c599b-f8f2x" Nov 03 00:41:10.034382 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:10.034364 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a145389b-418b-4ef0-9c66-29c9d7786081-tmp\") pod \"downloads-6dd789c8b7-bzcwg\" (UID: \"a145389b-418b-4ef0-9c66-29c9d7786081\") " pod="openshift-console/downloads-6dd789c8b7-bzcwg" Nov 03 00:41:10.045692 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:10.045675 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w76fq\" (UniqueName: \"kubernetes.io/projected/a145389b-418b-4ef0-9c66-29c9d7786081-kube-api-access-w76fq\") pod \"downloads-6dd789c8b7-bzcwg\" (UID: \"a145389b-418b-4ef0-9c66-29c9d7786081\") " pod="openshift-console/downloads-6dd789c8b7-bzcwg" Nov 03 00:41:10.134785 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:10.134745 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/46039de5-8eaf-4029-9cb6-ca23508092f8-registry-tls\") pod \"image-registry-5f895c599b-f8f2x\" (UID: \"46039de5-8eaf-4029-9cb6-ca23508092f8\") " pod="openshift-image-registry/image-registry-5f895c599b-f8f2x" Nov 03 00:41:10.134785 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:10.134782 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/46039de5-8eaf-4029-9cb6-ca23508092f8-trusted-ca\") pod \"image-registry-5f895c599b-f8f2x\" (UID: \"46039de5-8eaf-4029-9cb6-ca23508092f8\") " pod="openshift-image-registry/image-registry-5f895c599b-f8f2x" Nov 03 00:41:10.135046 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:10.134808 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/e906bfeb-04c0-4465-9ec6-6c18aad596ae-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-2496n\" (UID: \"e906bfeb-04c0-4465-9ec6-6c18aad596ae\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-2496n" Nov 03 00:41:10.135046 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:10.134828 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/46039de5-8eaf-4029-9cb6-ca23508092f8-image-registry-private-configuration\") pod \"image-registry-5f895c599b-f8f2x\" (UID: \"46039de5-8eaf-4029-9cb6-ca23508092f8\") " pod="openshift-image-registry/image-registry-5f895c599b-f8f2x" Nov 03 00:41:10.135046 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:10.134846 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/46039de5-8eaf-4029-9cb6-ca23508092f8-ca-trust-extracted\") pod \"image-registry-5f895c599b-f8f2x\" (UID: \"46039de5-8eaf-4029-9cb6-ca23508092f8\") " pod="openshift-image-registry/image-registry-5f895c599b-f8f2x" Nov 03 00:41:10.135046 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:10.134863 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/46039de5-8eaf-4029-9cb6-ca23508092f8-bound-sa-token\") pod \"image-registry-5f895c599b-f8f2x\" (UID: \"46039de5-8eaf-4029-9cb6-ca23508092f8\") " pod="openshift-image-registry/image-registry-5f895c599b-f8f2x" Nov 03 00:41:10.135046 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:10.134891 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/46039de5-8eaf-4029-9cb6-ca23508092f8-installation-pull-secrets\") pod \"image-registry-5f895c599b-f8f2x\" (UID: \"46039de5-8eaf-4029-9cb6-ca23508092f8\") " pod="openshift-image-registry/image-registry-5f895c599b-f8f2x" Nov 03 00:41:10.135046 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:10.134924 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/46039de5-8eaf-4029-9cb6-ca23508092f8-registry-certificates\") pod \"image-registry-5f895c599b-f8f2x\" (UID: \"46039de5-8eaf-4029-9cb6-ca23508092f8\") " pod="openshift-image-registry/image-registry-5f895c599b-f8f2x" Nov 03 00:41:10.135046 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:10.134963 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8x2rs\" (UniqueName: \"kubernetes.io/projected/46039de5-8eaf-4029-9cb6-ca23508092f8-kube-api-access-8x2rs\") pod \"image-registry-5f895c599b-f8f2x\" (UID: \"46039de5-8eaf-4029-9cb6-ca23508092f8\") " pod="openshift-image-registry/image-registry-5f895c599b-f8f2x" Nov 03 00:41:10.135466 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:10.135439 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/46039de5-8eaf-4029-9cb6-ca23508092f8-ca-trust-extracted\") pod \"image-registry-5f895c599b-f8f2x\" (UID: \"46039de5-8eaf-4029-9cb6-ca23508092f8\") " pod="openshift-image-registry/image-registry-5f895c599b-f8f2x" Nov 03 00:41:10.135932 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:10.135908 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/46039de5-8eaf-4029-9cb6-ca23508092f8-trusted-ca\") pod \"image-registry-5f895c599b-f8f2x\" (UID: \"46039de5-8eaf-4029-9cb6-ca23508092f8\") " pod="openshift-image-registry/image-registry-5f895c599b-f8f2x" Nov 03 00:41:10.136020 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:10.135949 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/46039de5-8eaf-4029-9cb6-ca23508092f8-registry-certificates\") pod \"image-registry-5f895c599b-f8f2x\" (UID: \"46039de5-8eaf-4029-9cb6-ca23508092f8\") " pod="openshift-image-registry/image-registry-5f895c599b-f8f2x" Nov 03 00:41:10.137892 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:10.137869 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/46039de5-8eaf-4029-9cb6-ca23508092f8-installation-pull-secrets\") pod \"image-registry-5f895c599b-f8f2x\" (UID: \"46039de5-8eaf-4029-9cb6-ca23508092f8\") " pod="openshift-image-registry/image-registry-5f895c599b-f8f2x" Nov 03 00:41:10.137978 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:10.137872 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/46039de5-8eaf-4029-9cb6-ca23508092f8-image-registry-private-configuration\") pod \"image-registry-5f895c599b-f8f2x\" (UID: \"46039de5-8eaf-4029-9cb6-ca23508092f8\") " pod="openshift-image-registry/image-registry-5f895c599b-f8f2x" Nov 03 00:41:10.137978 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:10.137946 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/e906bfeb-04c0-4465-9ec6-6c18aad596ae-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-2496n\" (UID: \"e906bfeb-04c0-4465-9ec6-6c18aad596ae\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-2496n" Nov 03 00:41:10.138348 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:10.138332 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/46039de5-8eaf-4029-9cb6-ca23508092f8-registry-tls\") pod \"image-registry-5f895c599b-f8f2x\" (UID: \"46039de5-8eaf-4029-9cb6-ca23508092f8\") " pod="openshift-image-registry/image-registry-5f895c599b-f8f2x" Nov 03 00:41:10.150137 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:10.150086 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8x2rs\" (UniqueName: \"kubernetes.io/projected/46039de5-8eaf-4029-9cb6-ca23508092f8-kube-api-access-8x2rs\") pod \"image-registry-5f895c599b-f8f2x\" (UID: \"46039de5-8eaf-4029-9cb6-ca23508092f8\") " pod="openshift-image-registry/image-registry-5f895c599b-f8f2x" Nov 03 00:41:10.150322 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:10.150304 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/46039de5-8eaf-4029-9cb6-ca23508092f8-bound-sa-token\") pod \"image-registry-5f895c599b-f8f2x\" (UID: \"46039de5-8eaf-4029-9cb6-ca23508092f8\") " pod="openshift-image-registry/image-registry-5f895c599b-f8f2x" Nov 03 00:41:10.226870 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:10.226835 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-bzcwg" Nov 03 00:41:10.235516 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:10.235484 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-2496n" Nov 03 00:41:10.305563 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:10.305535 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5f895c599b-f8f2x" Nov 03 00:41:10.509661 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:10.509628 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-2496n"] Nov 03 00:41:10.512565 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:41:10.512538 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode906bfeb_04c0_4465_9ec6_6c18aad596ae.slice/crio-4445a340ea28b48a310c4132f1b4d68b63d4596fb811db6d6bf005c8de62e37c WatchSource:0}: Error finding container 4445a340ea28b48a310c4132f1b4d68b63d4596fb811db6d6bf005c8de62e37c: Status 404 returned error can't find the container with id 4445a340ea28b48a310c4132f1b4d68b63d4596fb811db6d6bf005c8de62e37c Nov 03 00:41:10.532391 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:10.532366 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-bzcwg"] Nov 03 00:41:10.535202 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:41:10.535180 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda145389b_418b_4ef0_9c66_29c9d7786081.slice/crio-7c8092296bbc39cc2aa9919df6e38b9dfd394b5eb75cc7c288fa41c2fcb0261f WatchSource:0}: Error finding container 7c8092296bbc39cc2aa9919df6e38b9dfd394b5eb75cc7c288fa41c2fcb0261f: Status 404 returned error can't find the container with id 7c8092296bbc39cc2aa9919df6e38b9dfd394b5eb75cc7c288fa41c2fcb0261f Nov 03 00:41:10.598362 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:10.598327 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5f895c599b-f8f2x"] Nov 03 00:41:10.603108 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:41:10.603082 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46039de5_8eaf_4029_9cb6_ca23508092f8.slice/crio-8b7d54a2772b0cdcae0fbdf9adc1a48ed181e36e3a61f9fb8d678cd94304807d WatchSource:0}: Error finding container 8b7d54a2772b0cdcae0fbdf9adc1a48ed181e36e3a61f9fb8d678cd94304807d: Status 404 returned error can't find the container with id 8b7d54a2772b0cdcae0fbdf9adc1a48ed181e36e3a61f9fb8d678cd94304807d Nov 03 00:41:11.039320 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:11.039289 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-rx2xs" Nov 03 00:41:11.071783 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:11.071742 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-2496n" event={"ID":"e906bfeb-04c0-4465-9ec6-6c18aad596ae","Type":"ContainerStarted","Data":"4445a340ea28b48a310c4132f1b4d68b63d4596fb811db6d6bf005c8de62e37c"} Nov 03 00:41:11.073292 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:11.073234 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-bzcwg" event={"ID":"a145389b-418b-4ef0-9c66-29c9d7786081","Type":"ContainerStarted","Data":"7c8092296bbc39cc2aa9919df6e38b9dfd394b5eb75cc7c288fa41c2fcb0261f"} Nov 03 00:41:11.074810 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:11.074784 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5f895c599b-f8f2x" event={"ID":"46039de5-8eaf-4029-9cb6-ca23508092f8","Type":"ContainerStarted","Data":"9828e78c1e1baaa4a7802b84a2016003fc579f05427c2d21a2120a7c1c1823f1"} Nov 03 00:41:11.074947 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:11.074822 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5f895c599b-f8f2x" event={"ID":"46039de5-8eaf-4029-9cb6-ca23508092f8","Type":"ContainerStarted","Data":"8b7d54a2772b0cdcae0fbdf9adc1a48ed181e36e3a61f9fb8d678cd94304807d"} Nov 03 00:41:11.074947 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:11.074918 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5f895c599b-f8f2x" Nov 03 00:41:11.106626 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:11.106562 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5f895c599b-f8f2x" podStartSLOduration=2.106543134 podStartE2EDuration="2.106543134s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:41:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:41:11.105527247 +0000 UTC m=+13.276530344" watchObservedRunningTime="2026-11-03 00:41:11.106543134 +0000 UTC m=+13.277546224" Nov 03 00:41:12.080255 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:12.080211 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-2496n" event={"ID":"e906bfeb-04c0-4465-9ec6-6c18aad596ae","Type":"ContainerStarted","Data":"fef12b10335bd82e9324bce2b057252d6bca8dc92819d4d4faef29c97f99cb78"} Nov 03 00:41:12.080727 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:12.080474 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-2496n" Nov 03 00:41:12.086059 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:12.086033 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-2496n" Nov 03 00:41:12.086632 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:12.086618 4421 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-rx2xs_9c505674-e66b-4573-bbf5-7b6acf6684e6/dns/0.log" Nov 03 00:41:12.106853 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:12.106801 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-2496n" podStartSLOduration=2.010012483 podStartE2EDuration="3.106787999s" totalImagesPullingTime="1.096775516s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:41:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:41:12.10633224 +0000 UTC m=+14.277335336" watchObservedRunningTime="2026-11-03 00:41:12.106787999 +0000 UTC m=+14.277791086" Nov 03 00:41:12.252775 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:12.252741 4421 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-rx2xs_9c505674-e66b-4573-bbf5-7b6acf6684e6/kube-rbac-proxy/0.log" Nov 03 00:41:12.656088 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:12.656058 4421 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-cphks_e7f1faa5-fc0e-4f46-9208-b1e58f50fcf9/dns-node-resolver/0.log" Nov 03 00:41:12.732679 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:12.732640 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-ft4j8"] Nov 03 00:41:12.734739 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:12.734714 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-ft4j8" Nov 03 00:41:12.738486 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:12.738455 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Nov 03 00:41:12.747128 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:12.747105 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 03 00:41:12.747128 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:12.747109 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 03 00:41:12.748549 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:12.748531 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 03 00:41:12.757337 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:12.757313 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-ft4j8"] Nov 03 00:41:12.767585 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:12.766515 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Nov 03 00:41:12.767585 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:12.766755 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-q8ws5\"" Nov 03 00:41:12.857079 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:12.857044 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b694def6-001a-4d24-b091-eb4495bbb2d6-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-ft4j8\" (UID: \"b694def6-001a-4d24-b091-eb4495bbb2d6\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-ft4j8" Nov 03 00:41:12.857261 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:12.857091 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b694def6-001a-4d24-b091-eb4495bbb2d6-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-ft4j8\" (UID: \"b694def6-001a-4d24-b091-eb4495bbb2d6\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-ft4j8" Nov 03 00:41:12.857261 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:12.857230 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/b694def6-001a-4d24-b091-eb4495bbb2d6-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-ft4j8\" (UID: \"b694def6-001a-4d24-b091-eb4495bbb2d6\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-ft4j8" Nov 03 00:41:12.857372 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:12.857345 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ljxz\" (UniqueName: \"kubernetes.io/projected/b694def6-001a-4d24-b091-eb4495bbb2d6-kube-api-access-9ljxz\") pod \"prometheus-operator-9cd89b74c-ft4j8\" (UID: \"b694def6-001a-4d24-b091-eb4495bbb2d6\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-ft4j8" Nov 03 00:41:12.958019 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:12.957930 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ljxz\" (UniqueName: \"kubernetes.io/projected/b694def6-001a-4d24-b091-eb4495bbb2d6-kube-api-access-9ljxz\") pod \"prometheus-operator-9cd89b74c-ft4j8\" (UID: \"b694def6-001a-4d24-b091-eb4495bbb2d6\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-ft4j8" Nov 03 00:41:12.958019 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:12.957976 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b694def6-001a-4d24-b091-eb4495bbb2d6-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-ft4j8\" (UID: \"b694def6-001a-4d24-b091-eb4495bbb2d6\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-ft4j8" Nov 03 00:41:12.958019 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:12.957998 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b694def6-001a-4d24-b091-eb4495bbb2d6-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-ft4j8\" (UID: \"b694def6-001a-4d24-b091-eb4495bbb2d6\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-ft4j8" Nov 03 00:41:12.958424 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:12.958038 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/b694def6-001a-4d24-b091-eb4495bbb2d6-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-ft4j8\" (UID: \"b694def6-001a-4d24-b091-eb4495bbb2d6\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-ft4j8" Nov 03 00:41:12.958779 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:12.958747 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b694def6-001a-4d24-b091-eb4495bbb2d6-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-ft4j8\" (UID: \"b694def6-001a-4d24-b091-eb4495bbb2d6\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-ft4j8" Nov 03 00:41:12.960649 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:12.960627 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b694def6-001a-4d24-b091-eb4495bbb2d6-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-ft4j8\" (UID: \"b694def6-001a-4d24-b091-eb4495bbb2d6\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-ft4j8" Nov 03 00:41:12.961084 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:12.961062 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/b694def6-001a-4d24-b091-eb4495bbb2d6-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-ft4j8\" (UID: \"b694def6-001a-4d24-b091-eb4495bbb2d6\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-ft4j8" Nov 03 00:41:12.968122 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:12.968093 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ljxz\" (UniqueName: \"kubernetes.io/projected/b694def6-001a-4d24-b091-eb4495bbb2d6-kube-api-access-9ljxz\") pod \"prometheus-operator-9cd89b74c-ft4j8\" (UID: \"b694def6-001a-4d24-b091-eb4495bbb2d6\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-ft4j8" Nov 03 00:41:13.048833 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:13.048800 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-ft4j8" Nov 03 00:41:13.058669 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:13.058639 4421 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-5f895c599b-f8f2x_46039de5-8eaf-4029-9cb6-ca23508092f8/registry/0.log" Nov 03 00:41:13.326770 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:13.326726 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-ft4j8"] Nov 03 00:41:13.330445 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:41:13.330411 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb694def6_001a_4d24_b091_eb4495bbb2d6.slice/crio-42e4a72c2303cb994ad32d938590ba5a454264eac3616c5cbdb117b73d681af8 WatchSource:0}: Error finding container 42e4a72c2303cb994ad32d938590ba5a454264eac3616c5cbdb117b73d681af8: Status 404 returned error can't find the container with id 42e4a72c2303cb994ad32d938590ba5a454264eac3616c5cbdb117b73d681af8 Nov 03 00:41:14.053301 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:14.053222 4421 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-w8k5t_5844f274-ad38-42a3-9900-b051b01c9953/node-ca/0.log" Nov 03 00:41:14.088640 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:14.088593 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-ft4j8" event={"ID":"b694def6-001a-4d24-b091-eb4495bbb2d6","Type":"ContainerStarted","Data":"42e4a72c2303cb994ad32d938590ba5a454264eac3616c5cbdb117b73d681af8"} Nov 03 00:41:14.677459 ip-10-0-133-217 kubenswrapper[4421]: E1103 00:41:14.677419 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 00:41:14.856999 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:14.856922 4421 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-whbrp_76ccef27-d056-402e-a7e1-ee524d63bf35/serve-healthcheck-canary/0.log" Nov 03 00:41:15.097505 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:15.097464 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-ft4j8" event={"ID":"b694def6-001a-4d24-b091-eb4495bbb2d6","Type":"ContainerStarted","Data":"809d5fa226a0b8ed939b0b80f5a0ac66ce054f1abc5f2d4245a50c3e3fee13e1"} Nov 03 00:41:15.097673 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:15.097532 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-ft4j8" event={"ID":"b694def6-001a-4d24-b091-eb4495bbb2d6","Type":"ContainerStarted","Data":"3077c12b01f21c5c7e8cda1c0e9659bd280a499bceaa3619a608382f84c04ebb"} Nov 03 00:41:15.118126 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:15.118022 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-ft4j8" podStartSLOduration=1.8972946560000001 podStartE2EDuration="3.118001994s" totalImagesPullingTime="1.220707338s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:41:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:41:15.117472567 +0000 UTC m=+17.288475666" watchObservedRunningTime="2026-11-03 00:41:15.118001994 +0000 UTC m=+17.289005091" Nov 03 00:41:16.203982 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:16.202965 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zq57z" Nov 03 00:41:16.496185 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:16.496106 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-6f88db775c-bf765"] Nov 03 00:41:16.499188 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:16.499162 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f88db775c-bf765" Nov 03 00:41:16.510227 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:16.510201 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Nov 03 00:41:16.510350 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:16.510263 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Nov 03 00:41:16.510350 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:16.510321 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Nov 03 00:41:16.510927 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:16.510912 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-94ctt\"" Nov 03 00:41:16.511240 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:16.511225 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Nov 03 00:41:16.516936 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:16.516912 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6f88db775c-bf765"] Nov 03 00:41:16.585976 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:16.585939 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d08334f4-4b91-45b2-bca0-e523054f59a7-console-config\") pod \"console-6f88db775c-bf765\" (UID: \"d08334f4-4b91-45b2-bca0-e523054f59a7\") " pod="openshift-console/console-6f88db775c-bf765" Nov 03 00:41:16.585976 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:16.585974 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d08334f4-4b91-45b2-bca0-e523054f59a7-tmp\") pod \"console-6f88db775c-bf765\" (UID: \"d08334f4-4b91-45b2-bca0-e523054f59a7\") " pod="openshift-console/console-6f88db775c-bf765" Nov 03 00:41:16.586220 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:16.585999 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvxtr\" (UniqueName: \"kubernetes.io/projected/d08334f4-4b91-45b2-bca0-e523054f59a7-kube-api-access-qvxtr\") pod \"console-6f88db775c-bf765\" (UID: \"d08334f4-4b91-45b2-bca0-e523054f59a7\") " pod="openshift-console/console-6f88db775c-bf765" Nov 03 00:41:16.586220 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:16.586073 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d08334f4-4b91-45b2-bca0-e523054f59a7-console-oauth-config\") pod \"console-6f88db775c-bf765\" (UID: \"d08334f4-4b91-45b2-bca0-e523054f59a7\") " pod="openshift-console/console-6f88db775c-bf765" Nov 03 00:41:16.586220 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:16.586128 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d08334f4-4b91-45b2-bca0-e523054f59a7-service-ca\") pod \"console-6f88db775c-bf765\" (UID: \"d08334f4-4b91-45b2-bca0-e523054f59a7\") " pod="openshift-console/console-6f88db775c-bf765" Nov 03 00:41:16.586220 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:16.586147 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d08334f4-4b91-45b2-bca0-e523054f59a7-oauth-serving-cert\") pod \"console-6f88db775c-bf765\" (UID: \"d08334f4-4b91-45b2-bca0-e523054f59a7\") " pod="openshift-console/console-6f88db775c-bf765" Nov 03 00:41:16.586220 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:16.586183 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d08334f4-4b91-45b2-bca0-e523054f59a7-console-serving-cert\") pod \"console-6f88db775c-bf765\" (UID: \"d08334f4-4b91-45b2-bca0-e523054f59a7\") " pod="openshift-console/console-6f88db775c-bf765" Nov 03 00:41:16.612816 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:16.612786 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Nov 03 00:41:16.686883 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:16.686837 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d08334f4-4b91-45b2-bca0-e523054f59a7-console-oauth-config\") pod \"console-6f88db775c-bf765\" (UID: \"d08334f4-4b91-45b2-bca0-e523054f59a7\") " pod="openshift-console/console-6f88db775c-bf765" Nov 03 00:41:16.687096 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:16.686894 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d08334f4-4b91-45b2-bca0-e523054f59a7-service-ca\") pod \"console-6f88db775c-bf765\" (UID: \"d08334f4-4b91-45b2-bca0-e523054f59a7\") " pod="openshift-console/console-6f88db775c-bf765" Nov 03 00:41:16.687096 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:16.686925 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d08334f4-4b91-45b2-bca0-e523054f59a7-oauth-serving-cert\") pod \"console-6f88db775c-bf765\" (UID: \"d08334f4-4b91-45b2-bca0-e523054f59a7\") " pod="openshift-console/console-6f88db775c-bf765" Nov 03 00:41:16.687096 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:16.686968 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d08334f4-4b91-45b2-bca0-e523054f59a7-console-serving-cert\") pod \"console-6f88db775c-bf765\" (UID: \"d08334f4-4b91-45b2-bca0-e523054f59a7\") " pod="openshift-console/console-6f88db775c-bf765" Nov 03 00:41:16.687096 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:16.687011 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d08334f4-4b91-45b2-bca0-e523054f59a7-console-config\") pod \"console-6f88db775c-bf765\" (UID: \"d08334f4-4b91-45b2-bca0-e523054f59a7\") " pod="openshift-console/console-6f88db775c-bf765" Nov 03 00:41:16.687096 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:16.687038 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d08334f4-4b91-45b2-bca0-e523054f59a7-tmp\") pod \"console-6f88db775c-bf765\" (UID: \"d08334f4-4b91-45b2-bca0-e523054f59a7\") " pod="openshift-console/console-6f88db775c-bf765" Nov 03 00:41:16.687096 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:16.687078 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvxtr\" (UniqueName: \"kubernetes.io/projected/d08334f4-4b91-45b2-bca0-e523054f59a7-kube-api-access-qvxtr\") pod \"console-6f88db775c-bf765\" (UID: \"d08334f4-4b91-45b2-bca0-e523054f59a7\") " pod="openshift-console/console-6f88db775c-bf765" Nov 03 00:41:16.687626 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:16.687579 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d08334f4-4b91-45b2-bca0-e523054f59a7-tmp\") pod \"console-6f88db775c-bf765\" (UID: \"d08334f4-4b91-45b2-bca0-e523054f59a7\") " pod="openshift-console/console-6f88db775c-bf765" Nov 03 00:41:16.687841 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:16.687822 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d08334f4-4b91-45b2-bca0-e523054f59a7-service-ca\") pod \"console-6f88db775c-bf765\" (UID: \"d08334f4-4b91-45b2-bca0-e523054f59a7\") " pod="openshift-console/console-6f88db775c-bf765" Nov 03 00:41:16.687924 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:16.687856 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d08334f4-4b91-45b2-bca0-e523054f59a7-console-config\") pod \"console-6f88db775c-bf765\" (UID: \"d08334f4-4b91-45b2-bca0-e523054f59a7\") " pod="openshift-console/console-6f88db775c-bf765" Nov 03 00:41:16.687924 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:16.687916 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d08334f4-4b91-45b2-bca0-e523054f59a7-oauth-serving-cert\") pod \"console-6f88db775c-bf765\" (UID: \"d08334f4-4b91-45b2-bca0-e523054f59a7\") " pod="openshift-console/console-6f88db775c-bf765" Nov 03 00:41:16.689533 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:16.689503 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d08334f4-4b91-45b2-bca0-e523054f59a7-console-serving-cert\") pod \"console-6f88db775c-bf765\" (UID: \"d08334f4-4b91-45b2-bca0-e523054f59a7\") " pod="openshift-console/console-6f88db775c-bf765" Nov 03 00:41:16.689533 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:16.689505 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d08334f4-4b91-45b2-bca0-e523054f59a7-console-oauth-config\") pod \"console-6f88db775c-bf765\" (UID: \"d08334f4-4b91-45b2-bca0-e523054f59a7\") " pod="openshift-console/console-6f88db775c-bf765" Nov 03 00:41:16.698109 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:16.698088 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvxtr\" (UniqueName: \"kubernetes.io/projected/d08334f4-4b91-45b2-bca0-e523054f59a7-kube-api-access-qvxtr\") pod \"console-6f88db775c-bf765\" (UID: \"d08334f4-4b91-45b2-bca0-e523054f59a7\") " pod="openshift-console/console-6f88db775c-bf765" Nov 03 00:41:16.808831 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:16.808787 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f88db775c-bf765" Nov 03 00:41:17.106110 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:17.106031 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6f88db775c-bf765"] Nov 03 00:41:17.108742 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:41:17.108706 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd08334f4_4b91_45b2_bca0_e523054f59a7.slice/crio-8d7e7ef70302d9f496f5a3a791c519137e651621999feffa86263ce8619b3011 WatchSource:0}: Error finding container 8d7e7ef70302d9f496f5a3a791c519137e651621999feffa86263ce8619b3011: Status 404 returned error can't find the container with id 8d7e7ef70302d9f496f5a3a791c519137e651621999feffa86263ce8619b3011 Nov 03 00:41:17.222608 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:17.222565 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-vb7lq"] Nov 03 00:41:17.225194 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:17.225165 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-vb7lq" Nov 03 00:41:17.255649 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:17.255618 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-vb7lq"] Nov 03 00:41:17.274925 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:17.274898 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Nov 03 00:41:17.275152 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:17.275137 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Nov 03 00:41:17.275200 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:17.275148 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-96cwl\"" Nov 03 00:41:17.290855 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:17.290826 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6s88d\" (UniqueName: \"kubernetes.io/projected/9aba6c65-0e46-4eea-b9cf-f4cc8b0d5ef8-kube-api-access-6s88d\") pod \"openshift-state-metrics-7bd7489957-vb7lq\" (UID: \"9aba6c65-0e46-4eea-b9cf-f4cc8b0d5ef8\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-vb7lq" Nov 03 00:41:17.290982 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:17.290879 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9aba6c65-0e46-4eea-b9cf-f4cc8b0d5ef8-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-vb7lq\" (UID: \"9aba6c65-0e46-4eea-b9cf-f4cc8b0d5ef8\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-vb7lq" Nov 03 00:41:17.290982 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:17.290896 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9aba6c65-0e46-4eea-b9cf-f4cc8b0d5ef8-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-vb7lq\" (UID: \"9aba6c65-0e46-4eea-b9cf-f4cc8b0d5ef8\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-vb7lq" Nov 03 00:41:17.290982 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:17.290955 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9aba6c65-0e46-4eea-b9cf-f4cc8b0d5ef8-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-vb7lq\" (UID: \"9aba6c65-0e46-4eea-b9cf-f4cc8b0d5ef8\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-vb7lq" Nov 03 00:41:17.327857 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:17.327824 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-6ddbp"] Nov 03 00:41:17.329833 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:17.329818 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-6ddbp" Nov 03 00:41:17.360827 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:17.360761 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 03 00:41:17.361182 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:17.361165 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 03 00:41:17.364195 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:17.364173 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-dxq4b\"" Nov 03 00:41:17.391452 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:17.391415 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f5bb206e-db1f-4ed5-91ae-08adef206263-sys\") pod \"node-exporter-6ddbp\" (UID: \"f5bb206e-db1f-4ed5-91ae-08adef206263\") " pod="openshift-monitoring/node-exporter-6ddbp" Nov 03 00:41:17.391452 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:17.391452 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f5bb206e-db1f-4ed5-91ae-08adef206263-node-exporter-tls\") pod \"node-exporter-6ddbp\" (UID: \"f5bb206e-db1f-4ed5-91ae-08adef206263\") " pod="openshift-monitoring/node-exporter-6ddbp" Nov 03 00:41:17.391691 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:17.391496 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9aba6c65-0e46-4eea-b9cf-f4cc8b0d5ef8-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-vb7lq\" (UID: \"9aba6c65-0e46-4eea-b9cf-f4cc8b0d5ef8\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-vb7lq" Nov 03 00:41:17.391691 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:17.391535 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9aba6c65-0e46-4eea-b9cf-f4cc8b0d5ef8-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-vb7lq\" (UID: \"9aba6c65-0e46-4eea-b9cf-f4cc8b0d5ef8\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-vb7lq" Nov 03 00:41:17.391691 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:17.391578 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9aba6c65-0e46-4eea-b9cf-f4cc8b0d5ef8-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-vb7lq\" (UID: \"9aba6c65-0e46-4eea-b9cf-f4cc8b0d5ef8\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-vb7lq" Nov 03 00:41:17.391691 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:17.391619 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6s88d\" (UniqueName: \"kubernetes.io/projected/9aba6c65-0e46-4eea-b9cf-f4cc8b0d5ef8-kube-api-access-6s88d\") pod \"openshift-state-metrics-7bd7489957-vb7lq\" (UID: \"9aba6c65-0e46-4eea-b9cf-f4cc8b0d5ef8\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-vb7lq" Nov 03 00:41:17.391691 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:17.391657 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f5bb206e-db1f-4ed5-91ae-08adef206263-node-exporter-accelerators-collector-config\") pod \"node-exporter-6ddbp\" (UID: \"f5bb206e-db1f-4ed5-91ae-08adef206263\") " pod="openshift-monitoring/node-exporter-6ddbp" Nov 03 00:41:17.391691 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:17.391683 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lb2f8\" (UniqueName: \"kubernetes.io/projected/f5bb206e-db1f-4ed5-91ae-08adef206263-kube-api-access-lb2f8\") pod \"node-exporter-6ddbp\" (UID: \"f5bb206e-db1f-4ed5-91ae-08adef206263\") " pod="openshift-monitoring/node-exporter-6ddbp" Nov 03 00:41:17.391978 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:17.391713 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f5bb206e-db1f-4ed5-91ae-08adef206263-root\") pod \"node-exporter-6ddbp\" (UID: \"f5bb206e-db1f-4ed5-91ae-08adef206263\") " pod="openshift-monitoring/node-exporter-6ddbp" Nov 03 00:41:17.391978 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:17.391740 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f5bb206e-db1f-4ed5-91ae-08adef206263-node-exporter-textfile\") pod \"node-exporter-6ddbp\" (UID: \"f5bb206e-db1f-4ed5-91ae-08adef206263\") " pod="openshift-monitoring/node-exporter-6ddbp" Nov 03 00:41:17.391978 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:17.391784 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f5bb206e-db1f-4ed5-91ae-08adef206263-metrics-client-ca\") pod \"node-exporter-6ddbp\" (UID: \"f5bb206e-db1f-4ed5-91ae-08adef206263\") " pod="openshift-monitoring/node-exporter-6ddbp" Nov 03 00:41:17.391978 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:17.391855 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f5bb206e-db1f-4ed5-91ae-08adef206263-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6ddbp\" (UID: \"f5bb206e-db1f-4ed5-91ae-08adef206263\") " pod="openshift-monitoring/node-exporter-6ddbp" Nov 03 00:41:17.392115 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:17.391985 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f5bb206e-db1f-4ed5-91ae-08adef206263-node-exporter-wtmp\") pod \"node-exporter-6ddbp\" (UID: \"f5bb206e-db1f-4ed5-91ae-08adef206263\") " pod="openshift-monitoring/node-exporter-6ddbp" Nov 03 00:41:17.392365 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:17.392340 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9aba6c65-0e46-4eea-b9cf-f4cc8b0d5ef8-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-vb7lq\" (UID: \"9aba6c65-0e46-4eea-b9cf-f4cc8b0d5ef8\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-vb7lq" Nov 03 00:41:17.394064 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:17.394047 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9aba6c65-0e46-4eea-b9cf-f4cc8b0d5ef8-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-vb7lq\" (UID: \"9aba6c65-0e46-4eea-b9cf-f4cc8b0d5ef8\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-vb7lq" Nov 03 00:41:17.394149 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:17.394129 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9aba6c65-0e46-4eea-b9cf-f4cc8b0d5ef8-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-vb7lq\" (UID: \"9aba6c65-0e46-4eea-b9cf-f4cc8b0d5ef8\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-vb7lq" Nov 03 00:41:17.401913 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:17.401891 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 03 00:41:17.403464 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:17.403443 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6s88d\" (UniqueName: \"kubernetes.io/projected/9aba6c65-0e46-4eea-b9cf-f4cc8b0d5ef8-kube-api-access-6s88d\") pod \"openshift-state-metrics-7bd7489957-vb7lq\" (UID: \"9aba6c65-0e46-4eea-b9cf-f4cc8b0d5ef8\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-vb7lq" Nov 03 00:41:17.492777 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:17.492731 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f5bb206e-db1f-4ed5-91ae-08adef206263-node-exporter-accelerators-collector-config\") pod \"node-exporter-6ddbp\" (UID: \"f5bb206e-db1f-4ed5-91ae-08adef206263\") " pod="openshift-monitoring/node-exporter-6ddbp" Nov 03 00:41:17.492777 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:17.492779 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lb2f8\" (UniqueName: \"kubernetes.io/projected/f5bb206e-db1f-4ed5-91ae-08adef206263-kube-api-access-lb2f8\") pod \"node-exporter-6ddbp\" (UID: \"f5bb206e-db1f-4ed5-91ae-08adef206263\") " pod="openshift-monitoring/node-exporter-6ddbp" Nov 03 00:41:17.492994 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:17.492883 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f5bb206e-db1f-4ed5-91ae-08adef206263-root\") pod \"node-exporter-6ddbp\" (UID: \"f5bb206e-db1f-4ed5-91ae-08adef206263\") " pod="openshift-monitoring/node-exporter-6ddbp" Nov 03 00:41:17.492994 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:17.492906 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f5bb206e-db1f-4ed5-91ae-08adef206263-node-exporter-textfile\") pod \"node-exporter-6ddbp\" (UID: \"f5bb206e-db1f-4ed5-91ae-08adef206263\") " pod="openshift-monitoring/node-exporter-6ddbp" Nov 03 00:41:17.492994 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:17.492931 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f5bb206e-db1f-4ed5-91ae-08adef206263-metrics-client-ca\") pod \"node-exporter-6ddbp\" (UID: \"f5bb206e-db1f-4ed5-91ae-08adef206263\") " pod="openshift-monitoring/node-exporter-6ddbp" Nov 03 00:41:17.492994 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:17.492963 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f5bb206e-db1f-4ed5-91ae-08adef206263-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6ddbp\" (UID: \"f5bb206e-db1f-4ed5-91ae-08adef206263\") " pod="openshift-monitoring/node-exporter-6ddbp" Nov 03 00:41:17.493203 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:17.493000 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f5bb206e-db1f-4ed5-91ae-08adef206263-node-exporter-wtmp\") pod \"node-exporter-6ddbp\" (UID: \"f5bb206e-db1f-4ed5-91ae-08adef206263\") " pod="openshift-monitoring/node-exporter-6ddbp" Nov 03 00:41:17.493203 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:17.493041 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f5bb206e-db1f-4ed5-91ae-08adef206263-sys\") pod \"node-exporter-6ddbp\" (UID: \"f5bb206e-db1f-4ed5-91ae-08adef206263\") " pod="openshift-monitoring/node-exporter-6ddbp" Nov 03 00:41:17.493203 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:17.493074 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f5bb206e-db1f-4ed5-91ae-08adef206263-node-exporter-tls\") pod \"node-exporter-6ddbp\" (UID: \"f5bb206e-db1f-4ed5-91ae-08adef206263\") " pod="openshift-monitoring/node-exporter-6ddbp" Nov 03 00:41:17.493203 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:17.493036 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f5bb206e-db1f-4ed5-91ae-08adef206263-root\") pod \"node-exporter-6ddbp\" (UID: \"f5bb206e-db1f-4ed5-91ae-08adef206263\") " pod="openshift-monitoring/node-exporter-6ddbp" Nov 03 00:41:17.493203 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:17.493128 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f5bb206e-db1f-4ed5-91ae-08adef206263-sys\") pod \"node-exporter-6ddbp\" (UID: \"f5bb206e-db1f-4ed5-91ae-08adef206263\") " pod="openshift-monitoring/node-exporter-6ddbp" Nov 03 00:41:17.493203 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:17.493195 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f5bb206e-db1f-4ed5-91ae-08adef206263-node-exporter-wtmp\") pod \"node-exporter-6ddbp\" (UID: \"f5bb206e-db1f-4ed5-91ae-08adef206263\") " pod="openshift-monitoring/node-exporter-6ddbp" Nov 03 00:41:17.493505 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:17.493450 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f5bb206e-db1f-4ed5-91ae-08adef206263-node-exporter-textfile\") pod \"node-exporter-6ddbp\" (UID: \"f5bb206e-db1f-4ed5-91ae-08adef206263\") " pod="openshift-monitoring/node-exporter-6ddbp" Nov 03 00:41:17.493563 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:17.493533 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f5bb206e-db1f-4ed5-91ae-08adef206263-node-exporter-accelerators-collector-config\") pod \"node-exporter-6ddbp\" (UID: \"f5bb206e-db1f-4ed5-91ae-08adef206263\") " pod="openshift-monitoring/node-exporter-6ddbp" Nov 03 00:41:17.493634 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:17.493617 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f5bb206e-db1f-4ed5-91ae-08adef206263-metrics-client-ca\") pod \"node-exporter-6ddbp\" (UID: \"f5bb206e-db1f-4ed5-91ae-08adef206263\") " pod="openshift-monitoring/node-exporter-6ddbp" Nov 03 00:41:17.495628 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:17.495604 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f5bb206e-db1f-4ed5-91ae-08adef206263-node-exporter-tls\") pod \"node-exporter-6ddbp\" (UID: \"f5bb206e-db1f-4ed5-91ae-08adef206263\") " pod="openshift-monitoring/node-exporter-6ddbp" Nov 03 00:41:17.495687 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:17.495614 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f5bb206e-db1f-4ed5-91ae-08adef206263-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6ddbp\" (UID: \"f5bb206e-db1f-4ed5-91ae-08adef206263\") " pod="openshift-monitoring/node-exporter-6ddbp" Nov 03 00:41:17.502555 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:17.502531 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lb2f8\" (UniqueName: \"kubernetes.io/projected/f5bb206e-db1f-4ed5-91ae-08adef206263-kube-api-access-lb2f8\") pod \"node-exporter-6ddbp\" (UID: \"f5bb206e-db1f-4ed5-91ae-08adef206263\") " pod="openshift-monitoring/node-exporter-6ddbp" Nov 03 00:41:17.537516 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:17.537478 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-vb7lq" Nov 03 00:41:17.645413 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:17.645348 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-6ddbp" Nov 03 00:41:17.655373 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:41:17.655339 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5bb206e_db1f_4ed5_91ae_08adef206263.slice/crio-961e4f9af247dc63f8cf2f244921f219ad5911a8d02d1db65b12a41e4d1de028 WatchSource:0}: Error finding container 961e4f9af247dc63f8cf2f244921f219ad5911a8d02d1db65b12a41e4d1de028: Status 404 returned error can't find the container with id 961e4f9af247dc63f8cf2f244921f219ad5911a8d02d1db65b12a41e4d1de028 Nov 03 00:41:17.828726 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:17.828693 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-vb7lq"] Nov 03 00:41:17.832377 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:41:17.832340 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9aba6c65_0e46_4eea_b9cf_f4cc8b0d5ef8.slice/crio-b1ea0c4a69a0fbb050b0146c309800e82b9ef3a2a7cfee6ba0ec7adc654113ef WatchSource:0}: Error finding container b1ea0c4a69a0fbb050b0146c309800e82b9ef3a2a7cfee6ba0ec7adc654113ef: Status 404 returned error can't find the container with id b1ea0c4a69a0fbb050b0146c309800e82b9ef3a2a7cfee6ba0ec7adc654113ef Nov 03 00:41:18.115916 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:18.115874 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6ddbp" event={"ID":"f5bb206e-db1f-4ed5-91ae-08adef206263","Type":"ContainerStarted","Data":"961e4f9af247dc63f8cf2f244921f219ad5911a8d02d1db65b12a41e4d1de028"} Nov 03 00:41:18.118412 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:18.118378 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-vb7lq" event={"ID":"9aba6c65-0e46-4eea-b9cf-f4cc8b0d5ef8","Type":"ContainerStarted","Data":"117489a6110f9ad27ee82221238943a6eefd9f9b4a1b01febfb58fc6fa79584d"} Nov 03 00:41:18.118567 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:18.118444 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-vb7lq" event={"ID":"9aba6c65-0e46-4eea-b9cf-f4cc8b0d5ef8","Type":"ContainerStarted","Data":"07164216410cf4623eaf4ba09ce2433a9bfd62d00f449c8e34e50a57737ad844"} Nov 03 00:41:18.118567 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:18.118474 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-vb7lq" event={"ID":"9aba6c65-0e46-4eea-b9cf-f4cc8b0d5ef8","Type":"ContainerStarted","Data":"b1ea0c4a69a0fbb050b0146c309800e82b9ef3a2a7cfee6ba0ec7adc654113ef"} Nov 03 00:41:18.120246 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:18.120220 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f88db775c-bf765" event={"ID":"d08334f4-4b91-45b2-bca0-e523054f59a7","Type":"ContainerStarted","Data":"8d7e7ef70302d9f496f5a3a791c519137e651621999feffa86263ce8619b3011"} Nov 03 00:41:19.127251 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:19.127163 4421 generic.go:356] "Generic (PLEG): container finished" podID="f5bb206e-db1f-4ed5-91ae-08adef206263" containerID="828372b25a4e1f1224b8e8c1a93c67a27e8f62c809b54b6997df146081f36ffc" exitCode=0 Nov 03 00:41:19.127251 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:19.127237 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6ddbp" event={"ID":"f5bb206e-db1f-4ed5-91ae-08adef206263","Type":"ContainerDied","Data":"828372b25a4e1f1224b8e8c1a93c67a27e8f62c809b54b6997df146081f36ffc"} Nov 03 00:41:19.291393 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:19.288695 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-68bd88d8d8-jjhxr"] Nov 03 00:41:19.294623 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:19.294595 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-68bd88d8d8-jjhxr" Nov 03 00:41:19.307157 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:19.307134 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Nov 03 00:41:19.312497 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:19.312406 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-68bd88d8d8-jjhxr"] Nov 03 00:41:19.312953 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:19.312932 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Nov 03 00:41:19.407073 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:19.406994 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Nov 03 00:41:19.407073 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:19.407051 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Nov 03 00:41:19.407453 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:19.407434 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9f251a8f-5101-4fa3-a215-b1ee53bb822d-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-68bd88d8d8-jjhxr\" (UID: \"9f251a8f-5101-4fa3-a215-b1ee53bb822d\") " pod="openshift-monitoring/thanos-querier-68bd88d8d8-jjhxr" Nov 03 00:41:19.407539 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:19.407479 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/9f251a8f-5101-4fa3-a215-b1ee53bb822d-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-68bd88d8d8-jjhxr\" (UID: \"9f251a8f-5101-4fa3-a215-b1ee53bb822d\") " pod="openshift-monitoring/thanos-querier-68bd88d8d8-jjhxr" Nov 03 00:41:19.407539 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:19.407507 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9f251a8f-5101-4fa3-a215-b1ee53bb822d-secret-grpc-tls\") pod \"thanos-querier-68bd88d8d8-jjhxr\" (UID: \"9f251a8f-5101-4fa3-a215-b1ee53bb822d\") " pod="openshift-monitoring/thanos-querier-68bd88d8d8-jjhxr" Nov 03 00:41:19.407539 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:19.407534 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9f251a8f-5101-4fa3-a215-b1ee53bb822d-metrics-client-ca\") pod \"thanos-querier-68bd88d8d8-jjhxr\" (UID: \"9f251a8f-5101-4fa3-a215-b1ee53bb822d\") " pod="openshift-monitoring/thanos-querier-68bd88d8d8-jjhxr" Nov 03 00:41:19.407686 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:19.407573 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2ktp\" (UniqueName: \"kubernetes.io/projected/9f251a8f-5101-4fa3-a215-b1ee53bb822d-kube-api-access-b2ktp\") pod \"thanos-querier-68bd88d8d8-jjhxr\" (UID: \"9f251a8f-5101-4fa3-a215-b1ee53bb822d\") " pod="openshift-monitoring/thanos-querier-68bd88d8d8-jjhxr" Nov 03 00:41:19.407686 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:19.407604 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9f251a8f-5101-4fa3-a215-b1ee53bb822d-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-68bd88d8d8-jjhxr\" (UID: \"9f251a8f-5101-4fa3-a215-b1ee53bb822d\") " pod="openshift-monitoring/thanos-querier-68bd88d8d8-jjhxr" Nov 03 00:41:19.407686 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:19.407629 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9f251a8f-5101-4fa3-a215-b1ee53bb822d-tmp\") pod \"thanos-querier-68bd88d8d8-jjhxr\" (UID: \"9f251a8f-5101-4fa3-a215-b1ee53bb822d\") " pod="openshift-monitoring/thanos-querier-68bd88d8d8-jjhxr" Nov 03 00:41:19.407686 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:19.407658 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/9f251a8f-5101-4fa3-a215-b1ee53bb822d-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-68bd88d8d8-jjhxr\" (UID: \"9f251a8f-5101-4fa3-a215-b1ee53bb822d\") " pod="openshift-monitoring/thanos-querier-68bd88d8d8-jjhxr" Nov 03 00:41:19.407873 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:19.407691 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/9f251a8f-5101-4fa3-a215-b1ee53bb822d-secret-thanos-querier-tls\") pod \"thanos-querier-68bd88d8d8-jjhxr\" (UID: \"9f251a8f-5101-4fa3-a215-b1ee53bb822d\") " pod="openshift-monitoring/thanos-querier-68bd88d8d8-jjhxr" Nov 03 00:41:19.407985 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:19.407967 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-2f1f3dml0dhcc\"" Nov 03 00:41:19.408048 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:19.408021 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Nov 03 00:41:19.408108 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:19.408098 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-s867h\"" Nov 03 00:41:19.508950 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:19.508892 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9f251a8f-5101-4fa3-a215-b1ee53bb822d-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-68bd88d8d8-jjhxr\" (UID: \"9f251a8f-5101-4fa3-a215-b1ee53bb822d\") " pod="openshift-monitoring/thanos-querier-68bd88d8d8-jjhxr" Nov 03 00:41:19.509127 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:19.508957 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/9f251a8f-5101-4fa3-a215-b1ee53bb822d-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-68bd88d8d8-jjhxr\" (UID: \"9f251a8f-5101-4fa3-a215-b1ee53bb822d\") " pod="openshift-monitoring/thanos-querier-68bd88d8d8-jjhxr" Nov 03 00:41:19.509127 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:19.508989 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9f251a8f-5101-4fa3-a215-b1ee53bb822d-secret-grpc-tls\") pod \"thanos-querier-68bd88d8d8-jjhxr\" (UID: \"9f251a8f-5101-4fa3-a215-b1ee53bb822d\") " pod="openshift-monitoring/thanos-querier-68bd88d8d8-jjhxr" Nov 03 00:41:19.509127 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:19.509020 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9f251a8f-5101-4fa3-a215-b1ee53bb822d-metrics-client-ca\") pod \"thanos-querier-68bd88d8d8-jjhxr\" (UID: \"9f251a8f-5101-4fa3-a215-b1ee53bb822d\") " pod="openshift-monitoring/thanos-querier-68bd88d8d8-jjhxr" Nov 03 00:41:19.509127 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:19.509046 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2ktp\" (UniqueName: \"kubernetes.io/projected/9f251a8f-5101-4fa3-a215-b1ee53bb822d-kube-api-access-b2ktp\") pod \"thanos-querier-68bd88d8d8-jjhxr\" (UID: \"9f251a8f-5101-4fa3-a215-b1ee53bb822d\") " pod="openshift-monitoring/thanos-querier-68bd88d8d8-jjhxr" Nov 03 00:41:19.509127 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:19.509073 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9f251a8f-5101-4fa3-a215-b1ee53bb822d-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-68bd88d8d8-jjhxr\" (UID: \"9f251a8f-5101-4fa3-a215-b1ee53bb822d\") " pod="openshift-monitoring/thanos-querier-68bd88d8d8-jjhxr" Nov 03 00:41:19.509127 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:19.509097 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9f251a8f-5101-4fa3-a215-b1ee53bb822d-tmp\") pod \"thanos-querier-68bd88d8d8-jjhxr\" (UID: \"9f251a8f-5101-4fa3-a215-b1ee53bb822d\") " pod="openshift-monitoring/thanos-querier-68bd88d8d8-jjhxr" Nov 03 00:41:19.509127 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:19.509129 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/9f251a8f-5101-4fa3-a215-b1ee53bb822d-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-68bd88d8d8-jjhxr\" (UID: \"9f251a8f-5101-4fa3-a215-b1ee53bb822d\") " pod="openshift-monitoring/thanos-querier-68bd88d8d8-jjhxr" Nov 03 00:41:19.509574 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:19.509158 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/9f251a8f-5101-4fa3-a215-b1ee53bb822d-secret-thanos-querier-tls\") pod \"thanos-querier-68bd88d8d8-jjhxr\" (UID: \"9f251a8f-5101-4fa3-a215-b1ee53bb822d\") " pod="openshift-monitoring/thanos-querier-68bd88d8d8-jjhxr" Nov 03 00:41:19.509739 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:19.509707 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9f251a8f-5101-4fa3-a215-b1ee53bb822d-tmp\") pod \"thanos-querier-68bd88d8d8-jjhxr\" (UID: \"9f251a8f-5101-4fa3-a215-b1ee53bb822d\") " pod="openshift-monitoring/thanos-querier-68bd88d8d8-jjhxr" Nov 03 00:41:19.510454 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:19.510428 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9f251a8f-5101-4fa3-a215-b1ee53bb822d-metrics-client-ca\") pod \"thanos-querier-68bd88d8d8-jjhxr\" (UID: \"9f251a8f-5101-4fa3-a215-b1ee53bb822d\") " pod="openshift-monitoring/thanos-querier-68bd88d8d8-jjhxr" Nov 03 00:41:19.511948 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:19.511924 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9f251a8f-5101-4fa3-a215-b1ee53bb822d-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-68bd88d8d8-jjhxr\" (UID: \"9f251a8f-5101-4fa3-a215-b1ee53bb822d\") " pod="openshift-monitoring/thanos-querier-68bd88d8d8-jjhxr" Nov 03 00:41:19.512038 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:19.512009 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/9f251a8f-5101-4fa3-a215-b1ee53bb822d-secret-thanos-querier-tls\") pod \"thanos-querier-68bd88d8d8-jjhxr\" (UID: \"9f251a8f-5101-4fa3-a215-b1ee53bb822d\") " pod="openshift-monitoring/thanos-querier-68bd88d8d8-jjhxr" Nov 03 00:41:19.512539 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:19.512517 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9f251a8f-5101-4fa3-a215-b1ee53bb822d-secret-grpc-tls\") pod \"thanos-querier-68bd88d8d8-jjhxr\" (UID: \"9f251a8f-5101-4fa3-a215-b1ee53bb822d\") " pod="openshift-monitoring/thanos-querier-68bd88d8d8-jjhxr" Nov 03 00:41:19.512889 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:19.512839 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/9f251a8f-5101-4fa3-a215-b1ee53bb822d-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-68bd88d8d8-jjhxr\" (UID: \"9f251a8f-5101-4fa3-a215-b1ee53bb822d\") " pod="openshift-monitoring/thanos-querier-68bd88d8d8-jjhxr" Nov 03 00:41:19.514039 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:19.514013 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9f251a8f-5101-4fa3-a215-b1ee53bb822d-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-68bd88d8d8-jjhxr\" (UID: \"9f251a8f-5101-4fa3-a215-b1ee53bb822d\") " pod="openshift-monitoring/thanos-querier-68bd88d8d8-jjhxr" Nov 03 00:41:19.514399 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:19.514360 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/9f251a8f-5101-4fa3-a215-b1ee53bb822d-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-68bd88d8d8-jjhxr\" (UID: \"9f251a8f-5101-4fa3-a215-b1ee53bb822d\") " pod="openshift-monitoring/thanos-querier-68bd88d8d8-jjhxr" Nov 03 00:41:19.521972 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:19.521942 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2ktp\" (UniqueName: \"kubernetes.io/projected/9f251a8f-5101-4fa3-a215-b1ee53bb822d-kube-api-access-b2ktp\") pod \"thanos-querier-68bd88d8d8-jjhxr\" (UID: \"9f251a8f-5101-4fa3-a215-b1ee53bb822d\") " pod="openshift-monitoring/thanos-querier-68bd88d8d8-jjhxr" Nov 03 00:41:19.614004 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:19.613962 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-68bd88d8d8-jjhxr" Nov 03 00:41:19.917977 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:19.917945 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-68bd88d8d8-jjhxr"] Nov 03 00:41:19.922060 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:41:19.922033 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f251a8f_5101_4fa3_a215_b1ee53bb822d.slice/crio-fc90661a3eb2b44923a60a5dc58eb16e7d6f449c28a339fc696ede5bb4451edc WatchSource:0}: Error finding container fc90661a3eb2b44923a60a5dc58eb16e7d6f449c28a339fc696ede5bb4451edc: Status 404 returned error can't find the container with id fc90661a3eb2b44923a60a5dc58eb16e7d6f449c28a339fc696ede5bb4451edc Nov 03 00:41:20.135163 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:20.135119 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6ddbp" event={"ID":"f5bb206e-db1f-4ed5-91ae-08adef206263","Type":"ContainerStarted","Data":"d424903541dcec8bb6ae46efd8e7e487358e8f2a3c4e48d403d30c8d891f79b2"} Nov 03 00:41:20.135579 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:20.135196 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6ddbp" event={"ID":"f5bb206e-db1f-4ed5-91ae-08adef206263","Type":"ContainerStarted","Data":"79188dbf13cce17a5964c82e4645172c46ca57c773a68c0c79c1f4cc17cbfb71"} Nov 03 00:41:20.138749 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:20.138723 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-vb7lq" event={"ID":"9aba6c65-0e46-4eea-b9cf-f4cc8b0d5ef8","Type":"ContainerStarted","Data":"ea9431dec7d5017e72746650f4f1741feb0c1198fcf083c53ae55e1424b6f531"} Nov 03 00:41:20.140686 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:20.140656 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-68bd88d8d8-jjhxr" event={"ID":"9f251a8f-5101-4fa3-a215-b1ee53bb822d","Type":"ContainerStarted","Data":"fc90661a3eb2b44923a60a5dc58eb16e7d6f449c28a339fc696ede5bb4451edc"} Nov 03 00:41:20.170913 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:20.170856 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-6ddbp" podStartSLOduration=1.925304066 podStartE2EDuration="3.17084104s" totalImagesPullingTime="1.160077451s" totalInitContainerRuntime="85.459523ms" isStatefulPod=true podCreationTimestamp="2026-11-03 00:41:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:41:20.169594611 +0000 UTC m=+22.340597707" watchObservedRunningTime="2026-11-03 00:41:20.17084104 +0000 UTC m=+22.341844132" Nov 03 00:41:20.196446 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:20.196348 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-vb7lq" podStartSLOduration=1.928179353 podStartE2EDuration="3.196329341s" totalImagesPullingTime="1.268149988s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:41:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:41:20.195585977 +0000 UTC m=+22.366589073" watchObservedRunningTime="2026-11-03 00:41:20.196329341 +0000 UTC m=+22.367332439" Nov 03 00:41:21.146969 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:21.146885 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f88db775c-bf765" event={"ID":"d08334f4-4b91-45b2-bca0-e523054f59a7","Type":"ContainerStarted","Data":"620c2706ff820ef25cbd32803c7b22562a7814d151bca8f09c1ee60ed83e41be"} Nov 03 00:41:21.168256 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:21.168201 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-6f88db775c-bf765" podStartSLOduration=1.5269617640000002 podStartE2EDuration="5.16818053s" totalImagesPullingTime="3.641218766s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:41:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:41:21.167485415 +0000 UTC m=+23.338488514" watchObservedRunningTime="2026-11-03 00:41:21.16818053 +0000 UTC m=+23.339183629" Nov 03 00:41:21.939197 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:21.939162 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-4xbpw"] Nov 03 00:41:21.941013 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:21.940994 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-4xbpw" Nov 03 00:41:21.949626 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:21.949583 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Nov 03 00:41:21.949794 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:21.949761 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-xfjjf\"" Nov 03 00:41:21.953932 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:21.953908 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-4xbpw"] Nov 03 00:41:22.033771 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:22.033748 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxkvq\" (UniqueName: \"kubernetes.io/projected/ab520521-bd60-489a-b3bb-c7ee861809cf-kube-api-access-vxkvq\") pod \"monitoring-plugin-77f4f76594-4xbpw\" (UID: \"ab520521-bd60-489a-b3bb-c7ee861809cf\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-4xbpw" Nov 03 00:41:22.033960 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:22.033803 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ab520521-bd60-489a-b3bb-c7ee861809cf-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-4xbpw\" (UID: \"ab520521-bd60-489a-b3bb-c7ee861809cf\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-4xbpw" Nov 03 00:41:22.134591 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:22.134554 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxkvq\" (UniqueName: \"kubernetes.io/projected/ab520521-bd60-489a-b3bb-c7ee861809cf-kube-api-access-vxkvq\") pod \"monitoring-plugin-77f4f76594-4xbpw\" (UID: \"ab520521-bd60-489a-b3bb-c7ee861809cf\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-4xbpw" Nov 03 00:41:22.134707 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:22.134648 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ab520521-bd60-489a-b3bb-c7ee861809cf-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-4xbpw\" (UID: \"ab520521-bd60-489a-b3bb-c7ee861809cf\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-4xbpw" Nov 03 00:41:22.137761 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:22.137368 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ab520521-bd60-489a-b3bb-c7ee861809cf-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-4xbpw\" (UID: \"ab520521-bd60-489a-b3bb-c7ee861809cf\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-4xbpw" Nov 03 00:41:22.145216 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:22.145183 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxkvq\" (UniqueName: \"kubernetes.io/projected/ab520521-bd60-489a-b3bb-c7ee861809cf-kube-api-access-vxkvq\") pod \"monitoring-plugin-77f4f76594-4xbpw\" (UID: \"ab520521-bd60-489a-b3bb-c7ee861809cf\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-4xbpw" Nov 03 00:41:22.156000 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:22.155970 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-68bd88d8d8-jjhxr" event={"ID":"9f251a8f-5101-4fa3-a215-b1ee53bb822d","Type":"ContainerStarted","Data":"5151d6eeb164f3b907876bb6a85cf59fc561126970109b08bc8e47203fbb071b"} Nov 03 00:41:22.156346 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:22.156060 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-68bd88d8d8-jjhxr" event={"ID":"9f251a8f-5101-4fa3-a215-b1ee53bb822d","Type":"ContainerStarted","Data":"464573d1e3295343bc8891120be456dec00f90c01a90db2f6c3f1be76984c9c6"} Nov 03 00:41:22.253698 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:22.253661 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-4xbpw" Nov 03 00:41:22.544468 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:22.544431 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-4xbpw"] Nov 03 00:41:22.547022 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:41:22.546991 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab520521_bd60_489a_b3bb_c7ee861809cf.slice/crio-21d73e04d70063bc22b87c68c7ba26093c730ef18efc79e6fd1d3d614704f068 WatchSource:0}: Error finding container 21d73e04d70063bc22b87c68c7ba26093c730ef18efc79e6fd1d3d614704f068: Status 404 returned error can't find the container with id 21d73e04d70063bc22b87c68c7ba26093c730ef18efc79e6fd1d3d614704f068 Nov 03 00:41:23.165067 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:23.164975 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-4xbpw" event={"ID":"ab520521-bd60-489a-b3bb-c7ee861809cf","Type":"ContainerStarted","Data":"21d73e04d70063bc22b87c68c7ba26093c730ef18efc79e6fd1d3d614704f068"} Nov 03 00:41:23.169084 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:23.168893 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-68bd88d8d8-jjhxr" event={"ID":"9f251a8f-5101-4fa3-a215-b1ee53bb822d","Type":"ContainerStarted","Data":"6360ece7f351f84affc81353cd065bef8ce3f57eccc98088136675268c8c9bb9"} Nov 03 00:41:23.169084 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:23.168980 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-68bd88d8d8-jjhxr" event={"ID":"9f251a8f-5101-4fa3-a215-b1ee53bb822d","Type":"ContainerStarted","Data":"2fd1992dd3a5db248a32b158e40b6b18c950bd53d7c4592ef46f6bb4b6ff7f53"} Nov 03 00:41:23.169084 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:23.169024 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-68bd88d8d8-jjhxr" event={"ID":"9f251a8f-5101-4fa3-a215-b1ee53bb822d","Type":"ContainerStarted","Data":"47c1ef885fde4f4d9d2cb7efde95894a1e8162798e29dd0409aa27a1880ba9ed"} Nov 03 00:41:23.614892 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:23.614852 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 00:41:23.619682 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:23.619652 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:41:23.635045 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:23.635020 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Nov 03 00:41:23.635173 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:23.635139 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Nov 03 00:41:23.635306 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:23.635244 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Nov 03 00:41:23.635391 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:23.635372 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Nov 03 00:41:23.635480 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:23.635466 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Nov 03 00:41:23.635576 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:23.635561 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-x8s4k\"" Nov 03 00:41:23.635655 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:23.635619 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Nov 03 00:41:23.635710 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:23.635665 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Nov 03 00:41:23.635749 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:23.635728 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-5prsha9bc71b0\"" Nov 03 00:41:23.635790 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:23.635766 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Nov 03 00:41:23.635859 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:23.635844 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Nov 03 00:41:23.635937 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:23.635924 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Nov 03 00:41:23.635987 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:23.635977 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Nov 03 00:41:23.639626 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:23.639597 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Nov 03 00:41:23.655569 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:23.651736 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 00:41:23.738517 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:23.738378 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Nov 03 00:41:23.738517 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:23.738472 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Nov 03 00:41:23.740308 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:23.740263 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Nov 03 00:41:23.745813 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:23.745783 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0d569850-d136-4cfd-b0f4-45e014ae8782-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0d569850-d136-4cfd-b0f4-45e014ae8782\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:41:23.745933 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:23.745832 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d569850-d136-4cfd-b0f4-45e014ae8782-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0d569850-d136-4cfd-b0f4-45e014ae8782\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:41:23.745933 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:23.745860 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0d569850-d136-4cfd-b0f4-45e014ae8782-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0d569850-d136-4cfd-b0f4-45e014ae8782\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:41:23.745933 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:23.745896 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0d569850-d136-4cfd-b0f4-45e014ae8782-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0d569850-d136-4cfd-b0f4-45e014ae8782\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:41:23.745933 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:23.745930 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0d569850-d136-4cfd-b0f4-45e014ae8782-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0d569850-d136-4cfd-b0f4-45e014ae8782\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:41:23.746105 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:23.745967 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0d569850-d136-4cfd-b0f4-45e014ae8782-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0d569850-d136-4cfd-b0f4-45e014ae8782\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:41:23.746105 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:23.745992 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0d569850-d136-4cfd-b0f4-45e014ae8782-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0d569850-d136-4cfd-b0f4-45e014ae8782\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:41:23.746105 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:23.746015 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0d569850-d136-4cfd-b0f4-45e014ae8782-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0d569850-d136-4cfd-b0f4-45e014ae8782\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:41:23.746105 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:23.746040 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0d569850-d136-4cfd-b0f4-45e014ae8782-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0d569850-d136-4cfd-b0f4-45e014ae8782\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:41:23.746105 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:23.746064 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0d569850-d136-4cfd-b0f4-45e014ae8782-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0d569850-d136-4cfd-b0f4-45e014ae8782\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:41:23.746358 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:23.746111 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0d569850-d136-4cfd-b0f4-45e014ae8782-config-out\") pod \"prometheus-k8s-0\" (UID: \"0d569850-d136-4cfd-b0f4-45e014ae8782\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:41:23.746358 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:23.746138 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/0d569850-d136-4cfd-b0f4-45e014ae8782-config\") pod \"prometheus-k8s-0\" (UID: \"0d569850-d136-4cfd-b0f4-45e014ae8782\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:41:23.746358 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:23.746161 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0d569850-d136-4cfd-b0f4-45e014ae8782-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0d569850-d136-4cfd-b0f4-45e014ae8782\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:41:23.746358 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:23.746188 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0d569850-d136-4cfd-b0f4-45e014ae8782-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0d569850-d136-4cfd-b0f4-45e014ae8782\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:41:23.746358 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:23.746223 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d569850-d136-4cfd-b0f4-45e014ae8782-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0d569850-d136-4cfd-b0f4-45e014ae8782\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:41:23.746358 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:23.746253 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0d569850-d136-4cfd-b0f4-45e014ae8782-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0d569850-d136-4cfd-b0f4-45e014ae8782\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:41:23.746358 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:23.746304 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0d569850-d136-4cfd-b0f4-45e014ae8782-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0d569850-d136-4cfd-b0f4-45e014ae8782\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:41:23.746358 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:23.746339 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d569850-d136-4cfd-b0f4-45e014ae8782-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0d569850-d136-4cfd-b0f4-45e014ae8782\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:41:23.746710 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:23.746381 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c995h\" (UniqueName: \"kubernetes.io/projected/0d569850-d136-4cfd-b0f4-45e014ae8782-kube-api-access-c995h\") pod \"prometheus-k8s-0\" (UID: \"0d569850-d136-4cfd-b0f4-45e014ae8782\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:41:23.746710 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:23.746415 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0d569850-d136-4cfd-b0f4-45e014ae8782-web-config\") pod \"prometheus-k8s-0\" (UID: \"0d569850-d136-4cfd-b0f4-45e014ae8782\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:41:23.846995 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:23.846962 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0d569850-d136-4cfd-b0f4-45e014ae8782-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0d569850-d136-4cfd-b0f4-45e014ae8782\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:41:23.846995 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:23.847002 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0d569850-d136-4cfd-b0f4-45e014ae8782-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0d569850-d136-4cfd-b0f4-45e014ae8782\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:41:23.847245 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:23.847041 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0d569850-d136-4cfd-b0f4-45e014ae8782-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0d569850-d136-4cfd-b0f4-45e014ae8782\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:41:23.847245 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:23.847064 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0d569850-d136-4cfd-b0f4-45e014ae8782-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0d569850-d136-4cfd-b0f4-45e014ae8782\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:41:23.847245 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:23.847097 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0d569850-d136-4cfd-b0f4-45e014ae8782-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0d569850-d136-4cfd-b0f4-45e014ae8782\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:41:23.847245 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:23.847133 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0d569850-d136-4cfd-b0f4-45e014ae8782-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0d569850-d136-4cfd-b0f4-45e014ae8782\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:41:23.847245 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:23.847149 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0d569850-d136-4cfd-b0f4-45e014ae8782-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0d569850-d136-4cfd-b0f4-45e014ae8782\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:41:23.847245 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:23.847183 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0d569850-d136-4cfd-b0f4-45e014ae8782-config-out\") pod \"prometheus-k8s-0\" (UID: \"0d569850-d136-4cfd-b0f4-45e014ae8782\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:41:23.847245 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:23.847200 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/0d569850-d136-4cfd-b0f4-45e014ae8782-config\") pod \"prometheus-k8s-0\" (UID: \"0d569850-d136-4cfd-b0f4-45e014ae8782\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:41:23.847245 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:23.847215 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0d569850-d136-4cfd-b0f4-45e014ae8782-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0d569850-d136-4cfd-b0f4-45e014ae8782\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:41:23.847245 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:23.847233 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0d569850-d136-4cfd-b0f4-45e014ae8782-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0d569850-d136-4cfd-b0f4-45e014ae8782\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:41:23.847700 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:23.847253 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d569850-d136-4cfd-b0f4-45e014ae8782-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0d569850-d136-4cfd-b0f4-45e014ae8782\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:41:23.847700 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:23.847285 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0d569850-d136-4cfd-b0f4-45e014ae8782-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0d569850-d136-4cfd-b0f4-45e014ae8782\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:41:23.847700 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:23.847315 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0d569850-d136-4cfd-b0f4-45e014ae8782-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0d569850-d136-4cfd-b0f4-45e014ae8782\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:41:23.847700 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:23.847346 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d569850-d136-4cfd-b0f4-45e014ae8782-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0d569850-d136-4cfd-b0f4-45e014ae8782\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:41:23.847700 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:23.847375 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c995h\" (UniqueName: \"kubernetes.io/projected/0d569850-d136-4cfd-b0f4-45e014ae8782-kube-api-access-c995h\") pod \"prometheus-k8s-0\" (UID: \"0d569850-d136-4cfd-b0f4-45e014ae8782\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:41:23.847700 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:23.847402 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0d569850-d136-4cfd-b0f4-45e014ae8782-web-config\") pod \"prometheus-k8s-0\" (UID: \"0d569850-d136-4cfd-b0f4-45e014ae8782\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:41:23.847700 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:23.847430 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0d569850-d136-4cfd-b0f4-45e014ae8782-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0d569850-d136-4cfd-b0f4-45e014ae8782\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:41:23.847700 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:23.847464 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d569850-d136-4cfd-b0f4-45e014ae8782-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0d569850-d136-4cfd-b0f4-45e014ae8782\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:41:23.847700 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:23.847488 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0d569850-d136-4cfd-b0f4-45e014ae8782-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0d569850-d136-4cfd-b0f4-45e014ae8782\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:41:23.848157 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:23.848053 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0d569850-d136-4cfd-b0f4-45e014ae8782-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0d569850-d136-4cfd-b0f4-45e014ae8782\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:41:23.848415 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:23.848395 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0d569850-d136-4cfd-b0f4-45e014ae8782-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0d569850-d136-4cfd-b0f4-45e014ae8782\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:41:23.849812 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:23.849756 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d569850-d136-4cfd-b0f4-45e014ae8782-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0d569850-d136-4cfd-b0f4-45e014ae8782\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:41:23.850312 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:23.850287 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0d569850-d136-4cfd-b0f4-45e014ae8782-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0d569850-d136-4cfd-b0f4-45e014ae8782\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:41:23.850312 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:23.850306 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0d569850-d136-4cfd-b0f4-45e014ae8782-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0d569850-d136-4cfd-b0f4-45e014ae8782\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:41:23.851010 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:23.850982 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0d569850-d136-4cfd-b0f4-45e014ae8782-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0d569850-d136-4cfd-b0f4-45e014ae8782\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:41:23.851301 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:23.851260 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d569850-d136-4cfd-b0f4-45e014ae8782-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0d569850-d136-4cfd-b0f4-45e014ae8782\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:41:23.865889 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:23.865629 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0d569850-d136-4cfd-b0f4-45e014ae8782-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0d569850-d136-4cfd-b0f4-45e014ae8782\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:41:23.866020 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:23.865942 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0d569850-d136-4cfd-b0f4-45e014ae8782-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0d569850-d136-4cfd-b0f4-45e014ae8782\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:41:23.866532 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:23.866412 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0d569850-d136-4cfd-b0f4-45e014ae8782-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0d569850-d136-4cfd-b0f4-45e014ae8782\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:41:23.866532 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:23.866428 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0d569850-d136-4cfd-b0f4-45e014ae8782-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0d569850-d136-4cfd-b0f4-45e014ae8782\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:41:23.871184 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:23.866580 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0d569850-d136-4cfd-b0f4-45e014ae8782-config-out\") pod \"prometheus-k8s-0\" (UID: \"0d569850-d136-4cfd-b0f4-45e014ae8782\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:41:23.871184 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:23.866878 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0d569850-d136-4cfd-b0f4-45e014ae8782-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0d569850-d136-4cfd-b0f4-45e014ae8782\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:41:23.871184 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:23.866994 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0d569850-d136-4cfd-b0f4-45e014ae8782-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0d569850-d136-4cfd-b0f4-45e014ae8782\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:41:23.871184 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:23.866993 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0d569850-d136-4cfd-b0f4-45e014ae8782-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0d569850-d136-4cfd-b0f4-45e014ae8782\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:41:23.871184 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:23.867644 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d569850-d136-4cfd-b0f4-45e014ae8782-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0d569850-d136-4cfd-b0f4-45e014ae8782\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:41:23.871184 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:23.870442 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/0d569850-d136-4cfd-b0f4-45e014ae8782-config\") pod \"prometheus-k8s-0\" (UID: \"0d569850-d136-4cfd-b0f4-45e014ae8782\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:41:23.871802 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:23.871782 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c995h\" (UniqueName: \"kubernetes.io/projected/0d569850-d136-4cfd-b0f4-45e014ae8782-kube-api-access-c995h\") pod \"prometheus-k8s-0\" (UID: \"0d569850-d136-4cfd-b0f4-45e014ae8782\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:41:23.873581 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:23.873548 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0d569850-d136-4cfd-b0f4-45e014ae8782-web-config\") pod \"prometheus-k8s-0\" (UID: \"0d569850-d136-4cfd-b0f4-45e014ae8782\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:41:23.874969 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:23.874940 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0d569850-d136-4cfd-b0f4-45e014ae8782-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0d569850-d136-4cfd-b0f4-45e014ae8782\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:41:23.958884 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:23.958850 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:41:24.180067 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:24.180026 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-68bd88d8d8-jjhxr" event={"ID":"9f251a8f-5101-4fa3-a215-b1ee53bb822d","Type":"ContainerStarted","Data":"045569a4dd1d13710109d451c19196fbaa1f947f48b1c831e30bcc5b36149391"} Nov 03 00:41:24.180445 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:24.180147 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-68bd88d8d8-jjhxr" Nov 03 00:41:24.215411 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:24.215339 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-68bd88d8d8-jjhxr" podStartSLOduration=2.351048908 podStartE2EDuration="5.215318392s" totalImagesPullingTime="2.864269484s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:41:19 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:41:24.214532117 +0000 UTC m=+26.385535215" watchObservedRunningTime="2026-11-03 00:41:24.215318392 +0000 UTC m=+26.386321489" Nov 03 00:41:24.273516 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:24.273398 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 00:41:24.279438 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:41:24.279408 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d569850_d136_4cfd_b0f4_45e014ae8782.slice/crio-3157d67be86c50be5ec96cd25e48c41a951a2033a98bc674b0851acc3f70ab8e WatchSource:0}: Error finding container 3157d67be86c50be5ec96cd25e48c41a951a2033a98bc674b0851acc3f70ab8e: Status 404 returned error can't find the container with id 3157d67be86c50be5ec96cd25e48c41a951a2033a98bc674b0851acc3f70ab8e Nov 03 00:41:25.190420 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:25.190378 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0d569850-d136-4cfd-b0f4-45e014ae8782","Type":"ContainerStarted","Data":"3157d67be86c50be5ec96cd25e48c41a951a2033a98bc674b0851acc3f70ab8e"} Nov 03 00:41:25.192536 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:25.192450 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-4xbpw" event={"ID":"ab520521-bd60-489a-b3bb-c7ee861809cf","Type":"ContainerStarted","Data":"3110f25627b067ae480a9c208f42554b5efddeace68acad376e6f91521563289"} Nov 03 00:41:25.211936 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:25.211867 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-4xbpw" podStartSLOduration=2.637924659 podStartE2EDuration="4.211852039s" totalImagesPullingTime="1.57392738s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:41:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:41:25.211090914 +0000 UTC m=+27.382094011" watchObservedRunningTime="2026-11-03 00:41:25.211852039 +0000 UTC m=+27.382855137" Nov 03 00:41:26.198046 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:26.198007 4421 generic.go:356] "Generic (PLEG): container finished" podID="0d569850-d136-4cfd-b0f4-45e014ae8782" containerID="b7883446c04820fc06c84629eab2b4b99e7714c74a8b863660d5f231879cb14b" exitCode=0 Nov 03 00:41:26.198566 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:26.198102 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0d569850-d136-4cfd-b0f4-45e014ae8782","Type":"ContainerDied","Data":"b7883446c04820fc06c84629eab2b4b99e7714c74a8b863660d5f231879cb14b"} Nov 03 00:41:26.198566 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:26.198406 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-4xbpw" Nov 03 00:41:26.204003 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:26.203982 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-4xbpw" Nov 03 00:41:26.810360 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:26.810311 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6f88db775c-bf765" Nov 03 00:41:26.810543 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:26.810410 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6f88db775c-bf765" Nov 03 00:41:26.816510 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:26.816484 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6f88db775c-bf765" Nov 03 00:41:27.207947 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:27.207867 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6f88db775c-bf765" Nov 03 00:41:28.533200 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:28.532573 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-jltt2" Nov 03 00:41:30.200096 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:30.200065 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-68bd88d8d8-jjhxr" Nov 03 00:41:31.223512 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:31.223473 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0d569850-d136-4cfd-b0f4-45e014ae8782","Type":"ContainerStarted","Data":"b3c3949e9ad9c11e881e0801539ac41d7f799a560c07767baf41ca09bde69611"} Nov 03 00:41:31.223905 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:31.223573 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0d569850-d136-4cfd-b0f4-45e014ae8782","Type":"ContainerStarted","Data":"1f8d71c8d130bd5b79da4ea18f5069321fb7bf1c1961292af8d9817491086bfd"} Nov 03 00:41:31.223905 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:31.223600 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0d569850-d136-4cfd-b0f4-45e014ae8782","Type":"ContainerStarted","Data":"4b65fc0aad202a4508032ae2b85c27a2ec39d81992613e965a77eabac8c974ec"} Nov 03 00:41:31.223905 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:31.223626 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0d569850-d136-4cfd-b0f4-45e014ae8782","Type":"ContainerStarted","Data":"1ffdf32dbac688f7664ff3fe6eda4cdf437caf839f622b6c9b26f71b7b324f30"} Nov 03 00:41:31.223905 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:31.223648 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0d569850-d136-4cfd-b0f4-45e014ae8782","Type":"ContainerStarted","Data":"e3c8b487d6814dab847d9ff68549dcceaf2a4320bf8d16b49baf42b6fd2ad10f"} Nov 03 00:41:31.223905 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:31.223684 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0d569850-d136-4cfd-b0f4-45e014ae8782","Type":"ContainerStarted","Data":"974ed3b5941ffe78dba99eaafe74f0a3dd95a4cf48b6f7f65445938de5e94211"} Nov 03 00:41:31.265303 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:31.265184 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.20725122 podStartE2EDuration="8.265162796s" totalImagesPullingTime="5.041829724s" totalInitContainerRuntime="16.081852ms" isStatefulPod=false podCreationTimestamp="2026-11-03 00:41:23 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:41:31.263957792 +0000 UTC m=+33.434960890" watchObservedRunningTime="2026-11-03 00:41:31.265162796 +0000 UTC m=+33.436165897" Nov 03 00:41:31.549500 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:31.549387 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-6lvpt" Nov 03 00:41:32.088942 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:32.088909 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5f895c599b-f8f2x" Nov 03 00:41:33.959661 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:33.959620 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:41:34.241814 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:34.241727 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-bzcwg" event={"ID":"a145389b-418b-4ef0-9c66-29c9d7786081","Type":"ContainerStarted","Data":"0c1997decc0758619775004c151e11b417babb5cf6ee3dc225b2cb36e332ce67"} Nov 03 00:41:34.241986 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:34.241945 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-bzcwg" Nov 03 00:41:34.257264 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:34.257227 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-bzcwg" Nov 03 00:41:34.272551 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:34.272497 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-bzcwg" podStartSLOduration=2.351529621 podStartE2EDuration="25.272480566s" totalImagesPullingTime="22.920950945s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:41:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:41:34.269966595 +0000 UTC m=+36.440969693" watchObservedRunningTime="2026-11-03 00:41:34.272480566 +0000 UTC m=+36.443483659" Nov 03 00:41:39.195463 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:39.194462 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6f88db775c-bf765"] Nov 03 00:41:44.668915 ip-10-0-133-217 kubenswrapper[4421]: E1103 00:41:44.668830 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 00:41:56.648631 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:56.648587 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:41:56.698652 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:56.698621 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:41:57.345457 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:41:57.345432 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:42:04.218607 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:04.218550 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-6f88db775c-bf765" podUID="d08334f4-4b91-45b2-bca0-e523054f59a7" containerName="console" containerID="cri-o://620c2706ff820ef25cbd32803c7b22562a7814d151bca8f09c1ee60ed83e41be" gracePeriod=15 Nov 03 00:42:04.355157 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:04.355128 4421 generic.go:356] "Generic (PLEG): container finished" podID="d08334f4-4b91-45b2-bca0-e523054f59a7" containerID="620c2706ff820ef25cbd32803c7b22562a7814d151bca8f09c1ee60ed83e41be" exitCode=0 Nov 03 00:42:04.355312 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:04.355202 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f88db775c-bf765" event={"ID":"d08334f4-4b91-45b2-bca0-e523054f59a7","Type":"ContainerDied","Data":"620c2706ff820ef25cbd32803c7b22562a7814d151bca8f09c1ee60ed83e41be"} Nov 03 00:42:04.501089 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:04.501059 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f88db775c-bf765" Nov 03 00:42:04.599329 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:04.599264 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d08334f4-4b91-45b2-bca0-e523054f59a7-console-config\" (UniqueName: \"kubernetes.io/configmap/d08334f4-4b91-45b2-bca0-e523054f59a7-console-config\") pod \"d08334f4-4b91-45b2-bca0-e523054f59a7\" (UID: \"d08334f4-4b91-45b2-bca0-e523054f59a7\") " Nov 03 00:42:04.599329 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:04.599337 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d08334f4-4b91-45b2-bca0-e523054f59a7-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d08334f4-4b91-45b2-bca0-e523054f59a7-console-serving-cert\") pod \"d08334f4-4b91-45b2-bca0-e523054f59a7\" (UID: \"d08334f4-4b91-45b2-bca0-e523054f59a7\") " Nov 03 00:42:04.599581 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:04.599364 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d08334f4-4b91-45b2-bca0-e523054f59a7-tmp\" (UniqueName: \"kubernetes.io/empty-dir/d08334f4-4b91-45b2-bca0-e523054f59a7-tmp\") pod \"d08334f4-4b91-45b2-bca0-e523054f59a7\" (UID: \"d08334f4-4b91-45b2-bca0-e523054f59a7\") " Nov 03 00:42:04.599581 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:04.599381 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d08334f4-4b91-45b2-bca0-e523054f59a7-service-ca\" (UniqueName: \"kubernetes.io/configmap/d08334f4-4b91-45b2-bca0-e523054f59a7-service-ca\") pod \"d08334f4-4b91-45b2-bca0-e523054f59a7\" (UID: \"d08334f4-4b91-45b2-bca0-e523054f59a7\") " Nov 03 00:42:04.599581 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:04.599407 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d08334f4-4b91-45b2-bca0-e523054f59a7-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d08334f4-4b91-45b2-bca0-e523054f59a7-console-oauth-config\") pod \"d08334f4-4b91-45b2-bca0-e523054f59a7\" (UID: \"d08334f4-4b91-45b2-bca0-e523054f59a7\") " Nov 03 00:42:04.599729 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:04.599589 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d08334f4-4b91-45b2-bca0-e523054f59a7-kube-api-access-qvxtr\" (UniqueName: \"kubernetes.io/projected/d08334f4-4b91-45b2-bca0-e523054f59a7-kube-api-access-qvxtr\") pod \"d08334f4-4b91-45b2-bca0-e523054f59a7\" (UID: \"d08334f4-4b91-45b2-bca0-e523054f59a7\") " Nov 03 00:42:04.599729 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:04.599644 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d08334f4-4b91-45b2-bca0-e523054f59a7-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d08334f4-4b91-45b2-bca0-e523054f59a7-oauth-serving-cert\") pod \"d08334f4-4b91-45b2-bca0-e523054f59a7\" (UID: \"d08334f4-4b91-45b2-bca0-e523054f59a7\") " Nov 03 00:42:04.599729 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:04.599691 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d08334f4-4b91-45b2-bca0-e523054f59a7-tmp" pod "d08334f4-4b91-45b2-bca0-e523054f59a7" (UID: "d08334f4-4b91-45b2-bca0-e523054f59a7"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:42:04.599922 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:04.599892 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d08334f4-4b91-45b2-bca0-e523054f59a7-console-config" pod "d08334f4-4b91-45b2-bca0-e523054f59a7" (UID: "d08334f4-4b91-45b2-bca0-e523054f59a7"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:42:04.600048 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:04.600014 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d08334f4-4b91-45b2-bca0-e523054f59a7-service-ca" pod "d08334f4-4b91-45b2-bca0-e523054f59a7" (UID: "d08334f4-4b91-45b2-bca0-e523054f59a7"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:42:04.600048 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:04.599907 4421 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d08334f4-4b91-45b2-bca0-e523054f59a7-tmp\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:42:04.600236 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:04.600118 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d08334f4-4b91-45b2-bca0-e523054f59a7-oauth-serving-cert" pod "d08334f4-4b91-45b2-bca0-e523054f59a7" (UID: "d08334f4-4b91-45b2-bca0-e523054f59a7"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:42:04.601790 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:04.601770 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d08334f4-4b91-45b2-bca0-e523054f59a7-console-serving-cert" pod "d08334f4-4b91-45b2-bca0-e523054f59a7" (UID: "d08334f4-4b91-45b2-bca0-e523054f59a7"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:42:04.601883 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:04.601811 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d08334f4-4b91-45b2-bca0-e523054f59a7-kube-api-access-qvxtr" pod "d08334f4-4b91-45b2-bca0-e523054f59a7" (UID: "d08334f4-4b91-45b2-bca0-e523054f59a7"). InnerVolumeSpecName "kube-api-access-qvxtr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:42:04.601883 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:04.601864 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d08334f4-4b91-45b2-bca0-e523054f59a7-console-oauth-config" pod "d08334f4-4b91-45b2-bca0-e523054f59a7" (UID: "d08334f4-4b91-45b2-bca0-e523054f59a7"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:42:04.700438 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:04.700401 4421 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d08334f4-4b91-45b2-bca0-e523054f59a7-oauth-serving-cert\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:42:04.700438 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:04.700434 4421 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d08334f4-4b91-45b2-bca0-e523054f59a7-console-config\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:42:04.700438 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:04.700445 4421 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d08334f4-4b91-45b2-bca0-e523054f59a7-console-serving-cert\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:42:04.700673 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:04.700453 4421 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d08334f4-4b91-45b2-bca0-e523054f59a7-service-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:42:04.700673 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:04.700462 4421 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d08334f4-4b91-45b2-bca0-e523054f59a7-console-oauth-config\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:42:04.700673 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:04.700471 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvxtr\" (UniqueName: \"kubernetes.io/projected/d08334f4-4b91-45b2-bca0-e523054f59a7-kube-api-access-qvxtr\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:42:05.360083 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:05.360046 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f88db775c-bf765" event={"ID":"d08334f4-4b91-45b2-bca0-e523054f59a7","Type":"ContainerDied","Data":"8d7e7ef70302d9f496f5a3a791c519137e651621999feffa86263ce8619b3011"} Nov 03 00:42:05.360083 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:05.360073 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f88db775c-bf765" Nov 03 00:42:05.360683 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:05.360113 4421 scope.go:122] "RemoveContainer" containerID="620c2706ff820ef25cbd32803c7b22562a7814d151bca8f09c1ee60ed83e41be" Nov 03 00:42:05.382476 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:05.382416 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6f88db775c-bf765"] Nov 03 00:42:05.387459 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:05.387432 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6f88db775c-bf765"] Nov 03 00:42:05.976358 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:05.976323 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d08334f4-4b91-45b2-bca0-e523054f59a7" path="/var/lib/kubelet/pods/d08334f4-4b91-45b2-bca0-e523054f59a7/volumes" Nov 03 00:42:11.998265 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:11.998230 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 00:42:11.998842 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:11.998578 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="0d569850-d136-4cfd-b0f4-45e014ae8782" containerName="prometheus" containerID="cri-o://974ed3b5941ffe78dba99eaafe74f0a3dd95a4cf48b6f7f65445938de5e94211" gracePeriod=600 Nov 03 00:42:11.998842 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:11.998603 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="0d569850-d136-4cfd-b0f4-45e014ae8782" containerName="kube-rbac-proxy" containerID="cri-o://1f8d71c8d130bd5b79da4ea18f5069321fb7bf1c1961292af8d9817491086bfd" gracePeriod=600 Nov 03 00:42:11.998842 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:11.998643 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="0d569850-d136-4cfd-b0f4-45e014ae8782" containerName="kube-rbac-proxy-thanos" containerID="cri-o://b3c3949e9ad9c11e881e0801539ac41d7f799a560c07767baf41ca09bde69611" gracePeriod=600 Nov 03 00:42:11.998842 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:11.998670 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="0d569850-d136-4cfd-b0f4-45e014ae8782" containerName="thanos-sidecar" containerID="cri-o://1ffdf32dbac688f7664ff3fe6eda4cdf437caf839f622b6c9b26f71b7b324f30" gracePeriod=600 Nov 03 00:42:11.998842 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:11.998740 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="0d569850-d136-4cfd-b0f4-45e014ae8782" containerName="config-reloader" containerID="cri-o://e3c8b487d6814dab847d9ff68549dcceaf2a4320bf8d16b49baf42b6fd2ad10f" gracePeriod=600 Nov 03 00:42:11.998842 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:11.998787 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="0d569850-d136-4cfd-b0f4-45e014ae8782" containerName="kube-rbac-proxy-web" containerID="cri-o://4b65fc0aad202a4508032ae2b85c27a2ec39d81992613e965a77eabac8c974ec" gracePeriod=600 Nov 03 00:42:12.285600 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.285567 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:42:12.356680 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.356645 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d569850-d136-4cfd-b0f4-45e014ae8782-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0d569850-d136-4cfd-b0f4-45e014ae8782-secret-prometheus-k8s-tls\") pod \"0d569850-d136-4cfd-b0f4-45e014ae8782\" (UID: \"0d569850-d136-4cfd-b0f4-45e014ae8782\") " Nov 03 00:42:12.356869 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.356698 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d569850-d136-4cfd-b0f4-45e014ae8782-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0d569850-d136-4cfd-b0f4-45e014ae8782-secret-grpc-tls\") pod \"0d569850-d136-4cfd-b0f4-45e014ae8782\" (UID: \"0d569850-d136-4cfd-b0f4-45e014ae8782\") " Nov 03 00:42:12.356869 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.356735 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d569850-d136-4cfd-b0f4-45e014ae8782-web-config\" (UniqueName: \"kubernetes.io/secret/0d569850-d136-4cfd-b0f4-45e014ae8782-web-config\") pod \"0d569850-d136-4cfd-b0f4-45e014ae8782\" (UID: \"0d569850-d136-4cfd-b0f4-45e014ae8782\") " Nov 03 00:42:12.356869 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.356770 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d569850-d136-4cfd-b0f4-45e014ae8782-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0d569850-d136-4cfd-b0f4-45e014ae8782-prometheus-k8s-rulefiles-0\") pod \"0d569850-d136-4cfd-b0f4-45e014ae8782\" (UID: \"0d569850-d136-4cfd-b0f4-45e014ae8782\") " Nov 03 00:42:12.356869 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.356806 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d569850-d136-4cfd-b0f4-45e014ae8782-tls-assets\" (UniqueName: \"kubernetes.io/projected/0d569850-d136-4cfd-b0f4-45e014ae8782-tls-assets\") pod \"0d569850-d136-4cfd-b0f4-45e014ae8782\" (UID: \"0d569850-d136-4cfd-b0f4-45e014ae8782\") " Nov 03 00:42:12.356869 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.356837 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d569850-d136-4cfd-b0f4-45e014ae8782-kube-api-access-c995h\" (UniqueName: \"kubernetes.io/projected/0d569850-d136-4cfd-b0f4-45e014ae8782-kube-api-access-c995h\") pod \"0d569850-d136-4cfd-b0f4-45e014ae8782\" (UID: \"0d569850-d136-4cfd-b0f4-45e014ae8782\") " Nov 03 00:42:12.357171 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.356889 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d569850-d136-4cfd-b0f4-45e014ae8782-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d569850-d136-4cfd-b0f4-45e014ae8782-prometheus-trusted-ca-bundle\") pod \"0d569850-d136-4cfd-b0f4-45e014ae8782\" (UID: \"0d569850-d136-4cfd-b0f4-45e014ae8782\") " Nov 03 00:42:12.357171 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.357003 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d569850-d136-4cfd-b0f4-45e014ae8782-config-out\" (UniqueName: \"kubernetes.io/empty-dir/0d569850-d136-4cfd-b0f4-45e014ae8782-config-out\") pod \"0d569850-d136-4cfd-b0f4-45e014ae8782\" (UID: \"0d569850-d136-4cfd-b0f4-45e014ae8782\") " Nov 03 00:42:12.357171 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.357119 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d569850-d136-4cfd-b0f4-45e014ae8782-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d569850-d136-4cfd-b0f4-45e014ae8782-configmap-kubelet-serving-ca-bundle\") pod \"0d569850-d136-4cfd-b0f4-45e014ae8782\" (UID: \"0d569850-d136-4cfd-b0f4-45e014ae8782\") " Nov 03 00:42:12.357171 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.357158 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d569850-d136-4cfd-b0f4-45e014ae8782-config\" (UniqueName: \"kubernetes.io/secret/0d569850-d136-4cfd-b0f4-45e014ae8782-config\") pod \"0d569850-d136-4cfd-b0f4-45e014ae8782\" (UID: \"0d569850-d136-4cfd-b0f4-45e014ae8782\") " Nov 03 00:42:12.357392 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.357196 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d569850-d136-4cfd-b0f4-45e014ae8782-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0d569850-d136-4cfd-b0f4-45e014ae8782-secret-metrics-client-certs\") pod \"0d569850-d136-4cfd-b0f4-45e014ae8782\" (UID: \"0d569850-d136-4cfd-b0f4-45e014ae8782\") " Nov 03 00:42:12.357392 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.357251 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d569850-d136-4cfd-b0f4-45e014ae8782-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d569850-d136-4cfd-b0f4-45e014ae8782-configmap-serving-certs-ca-bundle\") pod \"0d569850-d136-4cfd-b0f4-45e014ae8782\" (UID: \"0d569850-d136-4cfd-b0f4-45e014ae8782\") " Nov 03 00:42:12.357392 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.357304 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d569850-d136-4cfd-b0f4-45e014ae8782-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0d569850-d136-4cfd-b0f4-45e014ae8782-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"0d569850-d136-4cfd-b0f4-45e014ae8782\" (UID: \"0d569850-d136-4cfd-b0f4-45e014ae8782\") " Nov 03 00:42:12.357392 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.357340 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d569850-d136-4cfd-b0f4-45e014ae8782-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0d569850-d136-4cfd-b0f4-45e014ae8782-thanos-prometheus-http-client-file\") pod \"0d569850-d136-4cfd-b0f4-45e014ae8782\" (UID: \"0d569850-d136-4cfd-b0f4-45e014ae8782\") " Nov 03 00:42:12.357597 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.357393 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d569850-d136-4cfd-b0f4-45e014ae8782-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0d569850-d136-4cfd-b0f4-45e014ae8782-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"0d569850-d136-4cfd-b0f4-45e014ae8782\" (UID: \"0d569850-d136-4cfd-b0f4-45e014ae8782\") " Nov 03 00:42:12.359918 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.358173 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d569850-d136-4cfd-b0f4-45e014ae8782-prometheus-k8s-rulefiles-0" pod "0d569850-d136-4cfd-b0f4-45e014ae8782" (UID: "0d569850-d136-4cfd-b0f4-45e014ae8782"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:42:12.359918 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.358992 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d569850-d136-4cfd-b0f4-45e014ae8782-prometheus-trusted-ca-bundle" pod "0d569850-d136-4cfd-b0f4-45e014ae8782" (UID: "0d569850-d136-4cfd-b0f4-45e014ae8782"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:42:12.360116 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.360015 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d569850-d136-4cfd-b0f4-45e014ae8782-secret-grpc-tls" pod "0d569850-d136-4cfd-b0f4-45e014ae8782" (UID: "0d569850-d136-4cfd-b0f4-45e014ae8782"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:42:12.360116 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.360040 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d569850-d136-4cfd-b0f4-45e014ae8782-configmap-kubelet-serving-ca-bundle" pod "0d569850-d136-4cfd-b0f4-45e014ae8782" (UID: "0d569850-d136-4cfd-b0f4-45e014ae8782"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:42:12.360227 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.360207 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d569850-d136-4cfd-b0f4-45e014ae8782-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0d569850-d136-4cfd-b0f4-45e014ae8782-secret-kube-rbac-proxy\") pod \"0d569850-d136-4cfd-b0f4-45e014ae8782\" (UID: \"0d569850-d136-4cfd-b0f4-45e014ae8782\") " Nov 03 00:42:12.360449 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.360307 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d569850-d136-4cfd-b0f4-45e014ae8782-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0d569850-d136-4cfd-b0f4-45e014ae8782-prometheus-k8s-db\") pod \"0d569850-d136-4cfd-b0f4-45e014ae8782\" (UID: \"0d569850-d136-4cfd-b0f4-45e014ae8782\") " Nov 03 00:42:12.360449 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.360365 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d569850-d136-4cfd-b0f4-45e014ae8782-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0d569850-d136-4cfd-b0f4-45e014ae8782-prometheus-k8s-rulefiles-1\") pod \"0d569850-d136-4cfd-b0f4-45e014ae8782\" (UID: \"0d569850-d136-4cfd-b0f4-45e014ae8782\") " Nov 03 00:42:12.360449 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.360399 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d569850-d136-4cfd-b0f4-45e014ae8782-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0d569850-d136-4cfd-b0f4-45e014ae8782-prometheus-k8s-rulefiles-2\") pod \"0d569850-d136-4cfd-b0f4-45e014ae8782\" (UID: \"0d569850-d136-4cfd-b0f4-45e014ae8782\") " Nov 03 00:42:12.360449 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.360433 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d569850-d136-4cfd-b0f4-45e014ae8782-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0d569850-d136-4cfd-b0f4-45e014ae8782-configmap-metrics-client-ca\") pod \"0d569850-d136-4cfd-b0f4-45e014ae8782\" (UID: \"0d569850-d136-4cfd-b0f4-45e014ae8782\") " Nov 03 00:42:12.360728 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.360535 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d569850-d136-4cfd-b0f4-45e014ae8782-configmap-serving-certs-ca-bundle" pod "0d569850-d136-4cfd-b0f4-45e014ae8782" (UID: "0d569850-d136-4cfd-b0f4-45e014ae8782"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:42:12.360728 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.360622 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d569850-d136-4cfd-b0f4-45e014ae8782-tls-assets" pod "0d569850-d136-4cfd-b0f4-45e014ae8782" (UID: "0d569850-d136-4cfd-b0f4-45e014ae8782"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:42:12.360820 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.360805 4421 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d569850-d136-4cfd-b0f4-45e014ae8782-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:42:12.360870 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.360831 4421 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0d569850-d136-4cfd-b0f4-45e014ae8782-secret-grpc-tls\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:42:12.360870 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.360848 4421 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0d569850-d136-4cfd-b0f4-45e014ae8782-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:42:12.360870 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.360850 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d569850-d136-4cfd-b0f4-45e014ae8782-kube-api-access-c995h" pod "0d569850-d136-4cfd-b0f4-45e014ae8782" (UID: "0d569850-d136-4cfd-b0f4-45e014ae8782"). InnerVolumeSpecName "kube-api-access-c995h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:42:12.360870 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.360863 4421 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0d569850-d136-4cfd-b0f4-45e014ae8782-tls-assets\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:42:12.361055 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.360880 4421 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d569850-d136-4cfd-b0f4-45e014ae8782-prometheus-trusted-ca-bundle\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:42:12.361055 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.360895 4421 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d569850-d136-4cfd-b0f4-45e014ae8782-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:42:12.361055 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.360918 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d569850-d136-4cfd-b0f4-45e014ae8782-secret-prometheus-k8s-tls" pod "0d569850-d136-4cfd-b0f4-45e014ae8782" (UID: "0d569850-d136-4cfd-b0f4-45e014ae8782"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:42:12.361055 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.360958 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d569850-d136-4cfd-b0f4-45e014ae8782-config-out" pod "0d569850-d136-4cfd-b0f4-45e014ae8782" (UID: "0d569850-d136-4cfd-b0f4-45e014ae8782"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:42:12.361258 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.361135 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d569850-d136-4cfd-b0f4-45e014ae8782-prometheus-k8s-rulefiles-1" pod "0d569850-d136-4cfd-b0f4-45e014ae8782" (UID: "0d569850-d136-4cfd-b0f4-45e014ae8782"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:42:12.361258 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.361224 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d569850-d136-4cfd-b0f4-45e014ae8782-prometheus-k8s-rulefiles-2" pod "0d569850-d136-4cfd-b0f4-45e014ae8782" (UID: "0d569850-d136-4cfd-b0f4-45e014ae8782"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:42:12.361574 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.361548 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d569850-d136-4cfd-b0f4-45e014ae8782-configmap-metrics-client-ca" pod "0d569850-d136-4cfd-b0f4-45e014ae8782" (UID: "0d569850-d136-4cfd-b0f4-45e014ae8782"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:42:12.362300 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.362243 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d569850-d136-4cfd-b0f4-45e014ae8782-prometheus-k8s-db" pod "0d569850-d136-4cfd-b0f4-45e014ae8782" (UID: "0d569850-d136-4cfd-b0f4-45e014ae8782"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:42:12.362825 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.362802 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d569850-d136-4cfd-b0f4-45e014ae8782-secret-prometheus-k8s-kube-rbac-proxy-web" pod "0d569850-d136-4cfd-b0f4-45e014ae8782" (UID: "0d569850-d136-4cfd-b0f4-45e014ae8782"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:42:12.363389 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.363359 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d569850-d136-4cfd-b0f4-45e014ae8782-config" pod "0d569850-d136-4cfd-b0f4-45e014ae8782" (UID: "0d569850-d136-4cfd-b0f4-45e014ae8782"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:42:12.363493 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.363469 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d569850-d136-4cfd-b0f4-45e014ae8782-thanos-prometheus-http-client-file" pod "0d569850-d136-4cfd-b0f4-45e014ae8782" (UID: "0d569850-d136-4cfd-b0f4-45e014ae8782"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:42:12.364388 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.364369 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d569850-d136-4cfd-b0f4-45e014ae8782-secret-kube-rbac-proxy" pod "0d569850-d136-4cfd-b0f4-45e014ae8782" (UID: "0d569850-d136-4cfd-b0f4-45e014ae8782"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:42:12.364471 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.364449 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d569850-d136-4cfd-b0f4-45e014ae8782-secret-prometheus-k8s-thanos-sidecar-tls" pod "0d569850-d136-4cfd-b0f4-45e014ae8782" (UID: "0d569850-d136-4cfd-b0f4-45e014ae8782"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:42:12.365527 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.365506 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d569850-d136-4cfd-b0f4-45e014ae8782-secret-metrics-client-certs" pod "0d569850-d136-4cfd-b0f4-45e014ae8782" (UID: "0d569850-d136-4cfd-b0f4-45e014ae8782"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:42:12.372432 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.372411 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d569850-d136-4cfd-b0f4-45e014ae8782-web-config" pod "0d569850-d136-4cfd-b0f4-45e014ae8782" (UID: "0d569850-d136-4cfd-b0f4-45e014ae8782"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:42:12.390117 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.390086 4421 generic.go:356] "Generic (PLEG): container finished" podID="0d569850-d136-4cfd-b0f4-45e014ae8782" containerID="b3c3949e9ad9c11e881e0801539ac41d7f799a560c07767baf41ca09bde69611" exitCode=0 Nov 03 00:42:12.390117 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.390112 4421 generic.go:356] "Generic (PLEG): container finished" podID="0d569850-d136-4cfd-b0f4-45e014ae8782" containerID="1f8d71c8d130bd5b79da4ea18f5069321fb7bf1c1961292af8d9817491086bfd" exitCode=0 Nov 03 00:42:12.390330 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.390123 4421 generic.go:356] "Generic (PLEG): container finished" podID="0d569850-d136-4cfd-b0f4-45e014ae8782" containerID="4b65fc0aad202a4508032ae2b85c27a2ec39d81992613e965a77eabac8c974ec" exitCode=0 Nov 03 00:42:12.390330 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.390131 4421 generic.go:356] "Generic (PLEG): container finished" podID="0d569850-d136-4cfd-b0f4-45e014ae8782" containerID="1ffdf32dbac688f7664ff3fe6eda4cdf437caf839f622b6c9b26f71b7b324f30" exitCode=0 Nov 03 00:42:12.390330 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.390141 4421 generic.go:356] "Generic (PLEG): container finished" podID="0d569850-d136-4cfd-b0f4-45e014ae8782" containerID="e3c8b487d6814dab847d9ff68549dcceaf2a4320bf8d16b49baf42b6fd2ad10f" exitCode=0 Nov 03 00:42:12.390330 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.390151 4421 generic.go:356] "Generic (PLEG): container finished" podID="0d569850-d136-4cfd-b0f4-45e014ae8782" containerID="974ed3b5941ffe78dba99eaafe74f0a3dd95a4cf48b6f7f65445938de5e94211" exitCode=0 Nov 03 00:42:12.390330 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.390168 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0d569850-d136-4cfd-b0f4-45e014ae8782","Type":"ContainerDied","Data":"b3c3949e9ad9c11e881e0801539ac41d7f799a560c07767baf41ca09bde69611"} Nov 03 00:42:12.390330 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.390189 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:42:12.390330 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.390263 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0d569850-d136-4cfd-b0f4-45e014ae8782","Type":"ContainerDied","Data":"1f8d71c8d130bd5b79da4ea18f5069321fb7bf1c1961292af8d9817491086bfd"} Nov 03 00:42:12.390330 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.390298 4421 scope.go:122] "RemoveContainer" containerID="b3c3949e9ad9c11e881e0801539ac41d7f799a560c07767baf41ca09bde69611" Nov 03 00:42:12.390757 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.390333 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0d569850-d136-4cfd-b0f4-45e014ae8782","Type":"ContainerDied","Data":"4b65fc0aad202a4508032ae2b85c27a2ec39d81992613e965a77eabac8c974ec"} Nov 03 00:42:12.390757 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.390376 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0d569850-d136-4cfd-b0f4-45e014ae8782","Type":"ContainerDied","Data":"1ffdf32dbac688f7664ff3fe6eda4cdf437caf839f622b6c9b26f71b7b324f30"} Nov 03 00:42:12.390757 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.390419 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0d569850-d136-4cfd-b0f4-45e014ae8782","Type":"ContainerDied","Data":"e3c8b487d6814dab847d9ff68549dcceaf2a4320bf8d16b49baf42b6fd2ad10f"} Nov 03 00:42:12.390757 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.390457 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0d569850-d136-4cfd-b0f4-45e014ae8782","Type":"ContainerDied","Data":"974ed3b5941ffe78dba99eaafe74f0a3dd95a4cf48b6f7f65445938de5e94211"} Nov 03 00:42:12.390757 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.390500 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0d569850-d136-4cfd-b0f4-45e014ae8782","Type":"ContainerDied","Data":"3157d67be86c50be5ec96cd25e48c41a951a2033a98bc674b0851acc3f70ab8e"} Nov 03 00:42:12.398401 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.398374 4421 scope.go:122] "RemoveContainer" containerID="1f8d71c8d130bd5b79da4ea18f5069321fb7bf1c1961292af8d9817491086bfd" Nov 03 00:42:12.405480 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.405458 4421 scope.go:122] "RemoveContainer" containerID="4b65fc0aad202a4508032ae2b85c27a2ec39d81992613e965a77eabac8c974ec" Nov 03 00:42:12.412381 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.412362 4421 scope.go:122] "RemoveContainer" containerID="1ffdf32dbac688f7664ff3fe6eda4cdf437caf839f622b6c9b26f71b7b324f30" Nov 03 00:42:12.418948 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.418920 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 00:42:12.420665 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.420642 4421 scope.go:122] "RemoveContainer" containerID="e3c8b487d6814dab847d9ff68549dcceaf2a4320bf8d16b49baf42b6fd2ad10f" Nov 03 00:42:12.423577 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.423556 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 00:42:12.428405 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.428388 4421 scope.go:122] "RemoveContainer" containerID="974ed3b5941ffe78dba99eaafe74f0a3dd95a4cf48b6f7f65445938de5e94211" Nov 03 00:42:12.435718 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.435697 4421 scope.go:122] "RemoveContainer" containerID="b7883446c04820fc06c84629eab2b4b99e7714c74a8b863660d5f231879cb14b" Nov 03 00:42:12.442414 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.442397 4421 scope.go:122] "RemoveContainer" containerID="b3c3949e9ad9c11e881e0801539ac41d7f799a560c07767baf41ca09bde69611" Nov 03 00:42:12.442710 ip-10-0-133-217 kubenswrapper[4421]: E1103 00:42:12.442690 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3c3949e9ad9c11e881e0801539ac41d7f799a560c07767baf41ca09bde69611\": container with ID starting with b3c3949e9ad9c11e881e0801539ac41d7f799a560c07767baf41ca09bde69611 not found: ID does not exist" containerID="b3c3949e9ad9c11e881e0801539ac41d7f799a560c07767baf41ca09bde69611" Nov 03 00:42:12.442767 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.442719 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3c3949e9ad9c11e881e0801539ac41d7f799a560c07767baf41ca09bde69611"} err="failed to get container status \"b3c3949e9ad9c11e881e0801539ac41d7f799a560c07767baf41ca09bde69611\": rpc error: code = NotFound desc = could not find container \"b3c3949e9ad9c11e881e0801539ac41d7f799a560c07767baf41ca09bde69611\": container with ID starting with b3c3949e9ad9c11e881e0801539ac41d7f799a560c07767baf41ca09bde69611 not found: ID does not exist" Nov 03 00:42:12.442767 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.442756 4421 scope.go:122] "RemoveContainer" containerID="1f8d71c8d130bd5b79da4ea18f5069321fb7bf1c1961292af8d9817491086bfd" Nov 03 00:42:12.442990 ip-10-0-133-217 kubenswrapper[4421]: E1103 00:42:12.442972 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f8d71c8d130bd5b79da4ea18f5069321fb7bf1c1961292af8d9817491086bfd\": container with ID starting with 1f8d71c8d130bd5b79da4ea18f5069321fb7bf1c1961292af8d9817491086bfd not found: ID does not exist" containerID="1f8d71c8d130bd5b79da4ea18f5069321fb7bf1c1961292af8d9817491086bfd" Nov 03 00:42:12.443039 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.442997 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f8d71c8d130bd5b79da4ea18f5069321fb7bf1c1961292af8d9817491086bfd"} err="failed to get container status \"1f8d71c8d130bd5b79da4ea18f5069321fb7bf1c1961292af8d9817491086bfd\": rpc error: code = NotFound desc = could not find container \"1f8d71c8d130bd5b79da4ea18f5069321fb7bf1c1961292af8d9817491086bfd\": container with ID starting with 1f8d71c8d130bd5b79da4ea18f5069321fb7bf1c1961292af8d9817491086bfd not found: ID does not exist" Nov 03 00:42:12.443039 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.443012 4421 scope.go:122] "RemoveContainer" containerID="4b65fc0aad202a4508032ae2b85c27a2ec39d81992613e965a77eabac8c974ec" Nov 03 00:42:12.443239 ip-10-0-133-217 kubenswrapper[4421]: E1103 00:42:12.443222 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b65fc0aad202a4508032ae2b85c27a2ec39d81992613e965a77eabac8c974ec\": container with ID starting with 4b65fc0aad202a4508032ae2b85c27a2ec39d81992613e965a77eabac8c974ec not found: ID does not exist" containerID="4b65fc0aad202a4508032ae2b85c27a2ec39d81992613e965a77eabac8c974ec" Nov 03 00:42:12.443286 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.443244 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b65fc0aad202a4508032ae2b85c27a2ec39d81992613e965a77eabac8c974ec"} err="failed to get container status \"4b65fc0aad202a4508032ae2b85c27a2ec39d81992613e965a77eabac8c974ec\": rpc error: code = NotFound desc = could not find container \"4b65fc0aad202a4508032ae2b85c27a2ec39d81992613e965a77eabac8c974ec\": container with ID starting with 4b65fc0aad202a4508032ae2b85c27a2ec39d81992613e965a77eabac8c974ec not found: ID does not exist" Nov 03 00:42:12.443286 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.443257 4421 scope.go:122] "RemoveContainer" containerID="1ffdf32dbac688f7664ff3fe6eda4cdf437caf839f622b6c9b26f71b7b324f30" Nov 03 00:42:12.443543 ip-10-0-133-217 kubenswrapper[4421]: E1103 00:42:12.443520 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ffdf32dbac688f7664ff3fe6eda4cdf437caf839f622b6c9b26f71b7b324f30\": container with ID starting with 1ffdf32dbac688f7664ff3fe6eda4cdf437caf839f622b6c9b26f71b7b324f30 not found: ID does not exist" containerID="1ffdf32dbac688f7664ff3fe6eda4cdf437caf839f622b6c9b26f71b7b324f30" Nov 03 00:42:12.443578 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.443552 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ffdf32dbac688f7664ff3fe6eda4cdf437caf839f622b6c9b26f71b7b324f30"} err="failed to get container status \"1ffdf32dbac688f7664ff3fe6eda4cdf437caf839f622b6c9b26f71b7b324f30\": rpc error: code = NotFound desc = could not find container \"1ffdf32dbac688f7664ff3fe6eda4cdf437caf839f622b6c9b26f71b7b324f30\": container with ID starting with 1ffdf32dbac688f7664ff3fe6eda4cdf437caf839f622b6c9b26f71b7b324f30 not found: ID does not exist" Nov 03 00:42:12.443578 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.443573 4421 scope.go:122] "RemoveContainer" containerID="e3c8b487d6814dab847d9ff68549dcceaf2a4320bf8d16b49baf42b6fd2ad10f" Nov 03 00:42:12.443847 ip-10-0-133-217 kubenswrapper[4421]: E1103 00:42:12.443829 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3c8b487d6814dab847d9ff68549dcceaf2a4320bf8d16b49baf42b6fd2ad10f\": container with ID starting with e3c8b487d6814dab847d9ff68549dcceaf2a4320bf8d16b49baf42b6fd2ad10f not found: ID does not exist" containerID="e3c8b487d6814dab847d9ff68549dcceaf2a4320bf8d16b49baf42b6fd2ad10f" Nov 03 00:42:12.443904 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.443856 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3c8b487d6814dab847d9ff68549dcceaf2a4320bf8d16b49baf42b6fd2ad10f"} err="failed to get container status \"e3c8b487d6814dab847d9ff68549dcceaf2a4320bf8d16b49baf42b6fd2ad10f\": rpc error: code = NotFound desc = could not find container \"e3c8b487d6814dab847d9ff68549dcceaf2a4320bf8d16b49baf42b6fd2ad10f\": container with ID starting with e3c8b487d6814dab847d9ff68549dcceaf2a4320bf8d16b49baf42b6fd2ad10f not found: ID does not exist" Nov 03 00:42:12.443904 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.443878 4421 scope.go:122] "RemoveContainer" containerID="974ed3b5941ffe78dba99eaafe74f0a3dd95a4cf48b6f7f65445938de5e94211" Nov 03 00:42:12.444110 ip-10-0-133-217 kubenswrapper[4421]: E1103 00:42:12.444094 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"974ed3b5941ffe78dba99eaafe74f0a3dd95a4cf48b6f7f65445938de5e94211\": container with ID starting with 974ed3b5941ffe78dba99eaafe74f0a3dd95a4cf48b6f7f65445938de5e94211 not found: ID does not exist" containerID="974ed3b5941ffe78dba99eaafe74f0a3dd95a4cf48b6f7f65445938de5e94211" Nov 03 00:42:12.444161 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.444112 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"974ed3b5941ffe78dba99eaafe74f0a3dd95a4cf48b6f7f65445938de5e94211"} err="failed to get container status \"974ed3b5941ffe78dba99eaafe74f0a3dd95a4cf48b6f7f65445938de5e94211\": rpc error: code = NotFound desc = could not find container \"974ed3b5941ffe78dba99eaafe74f0a3dd95a4cf48b6f7f65445938de5e94211\": container with ID starting with 974ed3b5941ffe78dba99eaafe74f0a3dd95a4cf48b6f7f65445938de5e94211 not found: ID does not exist" Nov 03 00:42:12.444161 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.444125 4421 scope.go:122] "RemoveContainer" containerID="b7883446c04820fc06c84629eab2b4b99e7714c74a8b863660d5f231879cb14b" Nov 03 00:42:12.444432 ip-10-0-133-217 kubenswrapper[4421]: E1103 00:42:12.444416 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7883446c04820fc06c84629eab2b4b99e7714c74a8b863660d5f231879cb14b\": container with ID starting with b7883446c04820fc06c84629eab2b4b99e7714c74a8b863660d5f231879cb14b not found: ID does not exist" containerID="b7883446c04820fc06c84629eab2b4b99e7714c74a8b863660d5f231879cb14b" Nov 03 00:42:12.444489 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.444434 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7883446c04820fc06c84629eab2b4b99e7714c74a8b863660d5f231879cb14b"} err="failed to get container status \"b7883446c04820fc06c84629eab2b4b99e7714c74a8b863660d5f231879cb14b\": rpc error: code = NotFound desc = could not find container \"b7883446c04820fc06c84629eab2b4b99e7714c74a8b863660d5f231879cb14b\": container with ID starting with b7883446c04820fc06c84629eab2b4b99e7714c74a8b863660d5f231879cb14b not found: ID does not exist" Nov 03 00:42:12.444489 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.444446 4421 scope.go:122] "RemoveContainer" containerID="b3c3949e9ad9c11e881e0801539ac41d7f799a560c07767baf41ca09bde69611" Nov 03 00:42:12.444703 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.444688 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3c3949e9ad9c11e881e0801539ac41d7f799a560c07767baf41ca09bde69611"} err="failed to get container status \"b3c3949e9ad9c11e881e0801539ac41d7f799a560c07767baf41ca09bde69611\": rpc error: code = NotFound desc = could not find container \"b3c3949e9ad9c11e881e0801539ac41d7f799a560c07767baf41ca09bde69611\": container with ID starting with b3c3949e9ad9c11e881e0801539ac41d7f799a560c07767baf41ca09bde69611 not found: ID does not exist" Nov 03 00:42:12.444748 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.444703 4421 scope.go:122] "RemoveContainer" containerID="1f8d71c8d130bd5b79da4ea18f5069321fb7bf1c1961292af8d9817491086bfd" Nov 03 00:42:12.444948 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.444932 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f8d71c8d130bd5b79da4ea18f5069321fb7bf1c1961292af8d9817491086bfd"} err="failed to get container status \"1f8d71c8d130bd5b79da4ea18f5069321fb7bf1c1961292af8d9817491086bfd\": rpc error: code = NotFound desc = could not find container \"1f8d71c8d130bd5b79da4ea18f5069321fb7bf1c1961292af8d9817491086bfd\": container with ID starting with 1f8d71c8d130bd5b79da4ea18f5069321fb7bf1c1961292af8d9817491086bfd not found: ID does not exist" Nov 03 00:42:12.444993 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.444947 4421 scope.go:122] "RemoveContainer" containerID="4b65fc0aad202a4508032ae2b85c27a2ec39d81992613e965a77eabac8c974ec" Nov 03 00:42:12.445176 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.445157 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b65fc0aad202a4508032ae2b85c27a2ec39d81992613e965a77eabac8c974ec"} err="failed to get container status \"4b65fc0aad202a4508032ae2b85c27a2ec39d81992613e965a77eabac8c974ec\": rpc error: code = NotFound desc = could not find container \"4b65fc0aad202a4508032ae2b85c27a2ec39d81992613e965a77eabac8c974ec\": container with ID starting with 4b65fc0aad202a4508032ae2b85c27a2ec39d81992613e965a77eabac8c974ec not found: ID does not exist" Nov 03 00:42:12.445262 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.445177 4421 scope.go:122] "RemoveContainer" containerID="1ffdf32dbac688f7664ff3fe6eda4cdf437caf839f622b6c9b26f71b7b324f30" Nov 03 00:42:12.445449 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.445434 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ffdf32dbac688f7664ff3fe6eda4cdf437caf839f622b6c9b26f71b7b324f30"} err="failed to get container status \"1ffdf32dbac688f7664ff3fe6eda4cdf437caf839f622b6c9b26f71b7b324f30\": rpc error: code = NotFound desc = could not find container \"1ffdf32dbac688f7664ff3fe6eda4cdf437caf839f622b6c9b26f71b7b324f30\": container with ID starting with 1ffdf32dbac688f7664ff3fe6eda4cdf437caf839f622b6c9b26f71b7b324f30 not found: ID does not exist" Nov 03 00:42:12.445449 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.445449 4421 scope.go:122] "RemoveContainer" containerID="e3c8b487d6814dab847d9ff68549dcceaf2a4320bf8d16b49baf42b6fd2ad10f" Nov 03 00:42:12.445699 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.445679 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3c8b487d6814dab847d9ff68549dcceaf2a4320bf8d16b49baf42b6fd2ad10f"} err="failed to get container status \"e3c8b487d6814dab847d9ff68549dcceaf2a4320bf8d16b49baf42b6fd2ad10f\": rpc error: code = NotFound desc = could not find container \"e3c8b487d6814dab847d9ff68549dcceaf2a4320bf8d16b49baf42b6fd2ad10f\": container with ID starting with e3c8b487d6814dab847d9ff68549dcceaf2a4320bf8d16b49baf42b6fd2ad10f not found: ID does not exist" Nov 03 00:42:12.445752 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.445699 4421 scope.go:122] "RemoveContainer" containerID="974ed3b5941ffe78dba99eaafe74f0a3dd95a4cf48b6f7f65445938de5e94211" Nov 03 00:42:12.445917 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.445902 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"974ed3b5941ffe78dba99eaafe74f0a3dd95a4cf48b6f7f65445938de5e94211"} err="failed to get container status \"974ed3b5941ffe78dba99eaafe74f0a3dd95a4cf48b6f7f65445938de5e94211\": rpc error: code = NotFound desc = could not find container \"974ed3b5941ffe78dba99eaafe74f0a3dd95a4cf48b6f7f65445938de5e94211\": container with ID starting with 974ed3b5941ffe78dba99eaafe74f0a3dd95a4cf48b6f7f65445938de5e94211 not found: ID does not exist" Nov 03 00:42:12.445917 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.445916 4421 scope.go:122] "RemoveContainer" containerID="b7883446c04820fc06c84629eab2b4b99e7714c74a8b863660d5f231879cb14b" Nov 03 00:42:12.446125 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.446108 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7883446c04820fc06c84629eab2b4b99e7714c74a8b863660d5f231879cb14b"} err="failed to get container status \"b7883446c04820fc06c84629eab2b4b99e7714c74a8b863660d5f231879cb14b\": rpc error: code = NotFound desc = could not find container \"b7883446c04820fc06c84629eab2b4b99e7714c74a8b863660d5f231879cb14b\": container with ID starting with b7883446c04820fc06c84629eab2b4b99e7714c74a8b863660d5f231879cb14b not found: ID does not exist" Nov 03 00:42:12.446174 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.446126 4421 scope.go:122] "RemoveContainer" containerID="b3c3949e9ad9c11e881e0801539ac41d7f799a560c07767baf41ca09bde69611" Nov 03 00:42:12.446346 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.446329 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3c3949e9ad9c11e881e0801539ac41d7f799a560c07767baf41ca09bde69611"} err="failed to get container status \"b3c3949e9ad9c11e881e0801539ac41d7f799a560c07767baf41ca09bde69611\": rpc error: code = NotFound desc = could not find container \"b3c3949e9ad9c11e881e0801539ac41d7f799a560c07767baf41ca09bde69611\": container with ID starting with b3c3949e9ad9c11e881e0801539ac41d7f799a560c07767baf41ca09bde69611 not found: ID does not exist" Nov 03 00:42:12.446346 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.446345 4421 scope.go:122] "RemoveContainer" containerID="1f8d71c8d130bd5b79da4ea18f5069321fb7bf1c1961292af8d9817491086bfd" Nov 03 00:42:12.447057 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.447027 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f8d71c8d130bd5b79da4ea18f5069321fb7bf1c1961292af8d9817491086bfd"} err="failed to get container status \"1f8d71c8d130bd5b79da4ea18f5069321fb7bf1c1961292af8d9817491086bfd\": rpc error: code = NotFound desc = could not find container \"1f8d71c8d130bd5b79da4ea18f5069321fb7bf1c1961292af8d9817491086bfd\": container with ID starting with 1f8d71c8d130bd5b79da4ea18f5069321fb7bf1c1961292af8d9817491086bfd not found: ID does not exist" Nov 03 00:42:12.447057 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.447057 4421 scope.go:122] "RemoveContainer" containerID="4b65fc0aad202a4508032ae2b85c27a2ec39d81992613e965a77eabac8c974ec" Nov 03 00:42:12.447547 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.447525 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b65fc0aad202a4508032ae2b85c27a2ec39d81992613e965a77eabac8c974ec"} err="failed to get container status \"4b65fc0aad202a4508032ae2b85c27a2ec39d81992613e965a77eabac8c974ec\": rpc error: code = NotFound desc = could not find container \"4b65fc0aad202a4508032ae2b85c27a2ec39d81992613e965a77eabac8c974ec\": container with ID starting with 4b65fc0aad202a4508032ae2b85c27a2ec39d81992613e965a77eabac8c974ec not found: ID does not exist" Nov 03 00:42:12.447675 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.447549 4421 scope.go:122] "RemoveContainer" containerID="1ffdf32dbac688f7664ff3fe6eda4cdf437caf839f622b6c9b26f71b7b324f30" Nov 03 00:42:12.447995 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.447963 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ffdf32dbac688f7664ff3fe6eda4cdf437caf839f622b6c9b26f71b7b324f30"} err="failed to get container status \"1ffdf32dbac688f7664ff3fe6eda4cdf437caf839f622b6c9b26f71b7b324f30\": rpc error: code = NotFound desc = could not find container \"1ffdf32dbac688f7664ff3fe6eda4cdf437caf839f622b6c9b26f71b7b324f30\": container with ID starting with 1ffdf32dbac688f7664ff3fe6eda4cdf437caf839f622b6c9b26f71b7b324f30 not found: ID does not exist" Nov 03 00:42:12.448091 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.447997 4421 scope.go:122] "RemoveContainer" containerID="e3c8b487d6814dab847d9ff68549dcceaf2a4320bf8d16b49baf42b6fd2ad10f" Nov 03 00:42:12.448463 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.448439 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3c8b487d6814dab847d9ff68549dcceaf2a4320bf8d16b49baf42b6fd2ad10f"} err="failed to get container status \"e3c8b487d6814dab847d9ff68549dcceaf2a4320bf8d16b49baf42b6fd2ad10f\": rpc error: code = NotFound desc = could not find container \"e3c8b487d6814dab847d9ff68549dcceaf2a4320bf8d16b49baf42b6fd2ad10f\": container with ID starting with e3c8b487d6814dab847d9ff68549dcceaf2a4320bf8d16b49baf42b6fd2ad10f not found: ID does not exist" Nov 03 00:42:12.448569 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.448470 4421 scope.go:122] "RemoveContainer" containerID="974ed3b5941ffe78dba99eaafe74f0a3dd95a4cf48b6f7f65445938de5e94211" Nov 03 00:42:12.448924 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.448883 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"974ed3b5941ffe78dba99eaafe74f0a3dd95a4cf48b6f7f65445938de5e94211"} err="failed to get container status \"974ed3b5941ffe78dba99eaafe74f0a3dd95a4cf48b6f7f65445938de5e94211\": rpc error: code = NotFound desc = could not find container \"974ed3b5941ffe78dba99eaafe74f0a3dd95a4cf48b6f7f65445938de5e94211\": container with ID starting with 974ed3b5941ffe78dba99eaafe74f0a3dd95a4cf48b6f7f65445938de5e94211 not found: ID does not exist" Nov 03 00:42:12.448924 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.448920 4421 scope.go:122] "RemoveContainer" containerID="b7883446c04820fc06c84629eab2b4b99e7714c74a8b863660d5f231879cb14b" Nov 03 00:42:12.449328 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.449302 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7883446c04820fc06c84629eab2b4b99e7714c74a8b863660d5f231879cb14b"} err="failed to get container status \"b7883446c04820fc06c84629eab2b4b99e7714c74a8b863660d5f231879cb14b\": rpc error: code = NotFound desc = could not find container \"b7883446c04820fc06c84629eab2b4b99e7714c74a8b863660d5f231879cb14b\": container with ID starting with b7883446c04820fc06c84629eab2b4b99e7714c74a8b863660d5f231879cb14b not found: ID does not exist" Nov 03 00:42:12.449429 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.449329 4421 scope.go:122] "RemoveContainer" containerID="b3c3949e9ad9c11e881e0801539ac41d7f799a560c07767baf41ca09bde69611" Nov 03 00:42:12.449745 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.449722 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3c3949e9ad9c11e881e0801539ac41d7f799a560c07767baf41ca09bde69611"} err="failed to get container status \"b3c3949e9ad9c11e881e0801539ac41d7f799a560c07767baf41ca09bde69611\": rpc error: code = NotFound desc = could not find container \"b3c3949e9ad9c11e881e0801539ac41d7f799a560c07767baf41ca09bde69611\": container with ID starting with b3c3949e9ad9c11e881e0801539ac41d7f799a560c07767baf41ca09bde69611 not found: ID does not exist" Nov 03 00:42:12.449906 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.449746 4421 scope.go:122] "RemoveContainer" containerID="1f8d71c8d130bd5b79da4ea18f5069321fb7bf1c1961292af8d9817491086bfd" Nov 03 00:42:12.450200 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.450180 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f8d71c8d130bd5b79da4ea18f5069321fb7bf1c1961292af8d9817491086bfd"} err="failed to get container status \"1f8d71c8d130bd5b79da4ea18f5069321fb7bf1c1961292af8d9817491086bfd\": rpc error: code = NotFound desc = could not find container \"1f8d71c8d130bd5b79da4ea18f5069321fb7bf1c1961292af8d9817491086bfd\": container with ID starting with 1f8d71c8d130bd5b79da4ea18f5069321fb7bf1c1961292af8d9817491086bfd not found: ID does not exist" Nov 03 00:42:12.450326 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.450203 4421 scope.go:122] "RemoveContainer" containerID="4b65fc0aad202a4508032ae2b85c27a2ec39d81992613e965a77eabac8c974ec" Nov 03 00:42:12.450687 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.450660 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b65fc0aad202a4508032ae2b85c27a2ec39d81992613e965a77eabac8c974ec"} err="failed to get container status \"4b65fc0aad202a4508032ae2b85c27a2ec39d81992613e965a77eabac8c974ec\": rpc error: code = NotFound desc = could not find container \"4b65fc0aad202a4508032ae2b85c27a2ec39d81992613e965a77eabac8c974ec\": container with ID starting with 4b65fc0aad202a4508032ae2b85c27a2ec39d81992613e965a77eabac8c974ec not found: ID does not exist" Nov 03 00:42:12.450777 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.450692 4421 scope.go:122] "RemoveContainer" containerID="1ffdf32dbac688f7664ff3fe6eda4cdf437caf839f622b6c9b26f71b7b324f30" Nov 03 00:42:12.451063 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.451042 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ffdf32dbac688f7664ff3fe6eda4cdf437caf839f622b6c9b26f71b7b324f30"} err="failed to get container status \"1ffdf32dbac688f7664ff3fe6eda4cdf437caf839f622b6c9b26f71b7b324f30\": rpc error: code = NotFound desc = could not find container \"1ffdf32dbac688f7664ff3fe6eda4cdf437caf839f622b6c9b26f71b7b324f30\": container with ID starting with 1ffdf32dbac688f7664ff3fe6eda4cdf437caf839f622b6c9b26f71b7b324f30 not found: ID does not exist" Nov 03 00:42:12.451063 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.451062 4421 scope.go:122] "RemoveContainer" containerID="e3c8b487d6814dab847d9ff68549dcceaf2a4320bf8d16b49baf42b6fd2ad10f" Nov 03 00:42:12.451510 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.451470 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3c8b487d6814dab847d9ff68549dcceaf2a4320bf8d16b49baf42b6fd2ad10f"} err="failed to get container status \"e3c8b487d6814dab847d9ff68549dcceaf2a4320bf8d16b49baf42b6fd2ad10f\": rpc error: code = NotFound desc = could not find container \"e3c8b487d6814dab847d9ff68549dcceaf2a4320bf8d16b49baf42b6fd2ad10f\": container with ID starting with e3c8b487d6814dab847d9ff68549dcceaf2a4320bf8d16b49baf42b6fd2ad10f not found: ID does not exist" Nov 03 00:42:12.451596 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.451540 4421 scope.go:122] "RemoveContainer" containerID="974ed3b5941ffe78dba99eaafe74f0a3dd95a4cf48b6f7f65445938de5e94211" Nov 03 00:42:12.452121 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.452100 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"974ed3b5941ffe78dba99eaafe74f0a3dd95a4cf48b6f7f65445938de5e94211"} err="failed to get container status \"974ed3b5941ffe78dba99eaafe74f0a3dd95a4cf48b6f7f65445938de5e94211\": rpc error: code = NotFound desc = could not find container \"974ed3b5941ffe78dba99eaafe74f0a3dd95a4cf48b6f7f65445938de5e94211\": container with ID starting with 974ed3b5941ffe78dba99eaafe74f0a3dd95a4cf48b6f7f65445938de5e94211 not found: ID does not exist" Nov 03 00:42:12.452169 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.452123 4421 scope.go:122] "RemoveContainer" containerID="b7883446c04820fc06c84629eab2b4b99e7714c74a8b863660d5f231879cb14b" Nov 03 00:42:12.452402 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.452376 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7883446c04820fc06c84629eab2b4b99e7714c74a8b863660d5f231879cb14b"} err="failed to get container status \"b7883446c04820fc06c84629eab2b4b99e7714c74a8b863660d5f231879cb14b\": rpc error: code = NotFound desc = could not find container \"b7883446c04820fc06c84629eab2b4b99e7714c74a8b863660d5f231879cb14b\": container with ID starting with b7883446c04820fc06c84629eab2b4b99e7714c74a8b863660d5f231879cb14b not found: ID does not exist" Nov 03 00:42:12.452444 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.452404 4421 scope.go:122] "RemoveContainer" containerID="b3c3949e9ad9c11e881e0801539ac41d7f799a560c07767baf41ca09bde69611" Nov 03 00:42:12.452639 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.452619 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3c3949e9ad9c11e881e0801539ac41d7f799a560c07767baf41ca09bde69611"} err="failed to get container status \"b3c3949e9ad9c11e881e0801539ac41d7f799a560c07767baf41ca09bde69611\": rpc error: code = NotFound desc = could not find container \"b3c3949e9ad9c11e881e0801539ac41d7f799a560c07767baf41ca09bde69611\": container with ID starting with b3c3949e9ad9c11e881e0801539ac41d7f799a560c07767baf41ca09bde69611 not found: ID does not exist" Nov 03 00:42:12.452639 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.452637 4421 scope.go:122] "RemoveContainer" containerID="1f8d71c8d130bd5b79da4ea18f5069321fb7bf1c1961292af8d9817491086bfd" Nov 03 00:42:12.452881 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.452861 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f8d71c8d130bd5b79da4ea18f5069321fb7bf1c1961292af8d9817491086bfd"} err="failed to get container status \"1f8d71c8d130bd5b79da4ea18f5069321fb7bf1c1961292af8d9817491086bfd\": rpc error: code = NotFound desc = could not find container \"1f8d71c8d130bd5b79da4ea18f5069321fb7bf1c1961292af8d9817491086bfd\": container with ID starting with 1f8d71c8d130bd5b79da4ea18f5069321fb7bf1c1961292af8d9817491086bfd not found: ID does not exist" Nov 03 00:42:12.452881 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.452881 4421 scope.go:122] "RemoveContainer" containerID="4b65fc0aad202a4508032ae2b85c27a2ec39d81992613e965a77eabac8c974ec" Nov 03 00:42:12.453100 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.453085 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b65fc0aad202a4508032ae2b85c27a2ec39d81992613e965a77eabac8c974ec"} err="failed to get container status \"4b65fc0aad202a4508032ae2b85c27a2ec39d81992613e965a77eabac8c974ec\": rpc error: code = NotFound desc = could not find container \"4b65fc0aad202a4508032ae2b85c27a2ec39d81992613e965a77eabac8c974ec\": container with ID starting with 4b65fc0aad202a4508032ae2b85c27a2ec39d81992613e965a77eabac8c974ec not found: ID does not exist" Nov 03 00:42:12.453142 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.453100 4421 scope.go:122] "RemoveContainer" containerID="1ffdf32dbac688f7664ff3fe6eda4cdf437caf839f622b6c9b26f71b7b324f30" Nov 03 00:42:12.453361 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.453332 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ffdf32dbac688f7664ff3fe6eda4cdf437caf839f622b6c9b26f71b7b324f30"} err="failed to get container status \"1ffdf32dbac688f7664ff3fe6eda4cdf437caf839f622b6c9b26f71b7b324f30\": rpc error: code = NotFound desc = could not find container \"1ffdf32dbac688f7664ff3fe6eda4cdf437caf839f622b6c9b26f71b7b324f30\": container with ID starting with 1ffdf32dbac688f7664ff3fe6eda4cdf437caf839f622b6c9b26f71b7b324f30 not found: ID does not exist" Nov 03 00:42:12.453448 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.453361 4421 scope.go:122] "RemoveContainer" containerID="e3c8b487d6814dab847d9ff68549dcceaf2a4320bf8d16b49baf42b6fd2ad10f" Nov 03 00:42:12.454668 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.454647 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3c8b487d6814dab847d9ff68549dcceaf2a4320bf8d16b49baf42b6fd2ad10f"} err="failed to get container status \"e3c8b487d6814dab847d9ff68549dcceaf2a4320bf8d16b49baf42b6fd2ad10f\": rpc error: code = NotFound desc = could not find container \"e3c8b487d6814dab847d9ff68549dcceaf2a4320bf8d16b49baf42b6fd2ad10f\": container with ID starting with e3c8b487d6814dab847d9ff68549dcceaf2a4320bf8d16b49baf42b6fd2ad10f not found: ID does not exist" Nov 03 00:42:12.454668 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.454667 4421 scope.go:122] "RemoveContainer" containerID="974ed3b5941ffe78dba99eaafe74f0a3dd95a4cf48b6f7f65445938de5e94211" Nov 03 00:42:12.454913 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.454897 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"974ed3b5941ffe78dba99eaafe74f0a3dd95a4cf48b6f7f65445938de5e94211"} err="failed to get container status \"974ed3b5941ffe78dba99eaafe74f0a3dd95a4cf48b6f7f65445938de5e94211\": rpc error: code = NotFound desc = could not find container \"974ed3b5941ffe78dba99eaafe74f0a3dd95a4cf48b6f7f65445938de5e94211\": container with ID starting with 974ed3b5941ffe78dba99eaafe74f0a3dd95a4cf48b6f7f65445938de5e94211 not found: ID does not exist" Nov 03 00:42:12.454913 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.454912 4421 scope.go:122] "RemoveContainer" containerID="b7883446c04820fc06c84629eab2b4b99e7714c74a8b863660d5f231879cb14b" Nov 03 00:42:12.455162 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.455142 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7883446c04820fc06c84629eab2b4b99e7714c74a8b863660d5f231879cb14b"} err="failed to get container status \"b7883446c04820fc06c84629eab2b4b99e7714c74a8b863660d5f231879cb14b\": rpc error: code = NotFound desc = could not find container \"b7883446c04820fc06c84629eab2b4b99e7714c74a8b863660d5f231879cb14b\": container with ID starting with b7883446c04820fc06c84629eab2b4b99e7714c74a8b863660d5f231879cb14b not found: ID does not exist" Nov 03 00:42:12.455241 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.455163 4421 scope.go:122] "RemoveContainer" containerID="b3c3949e9ad9c11e881e0801539ac41d7f799a560c07767baf41ca09bde69611" Nov 03 00:42:12.455415 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.455397 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3c3949e9ad9c11e881e0801539ac41d7f799a560c07767baf41ca09bde69611"} err="failed to get container status \"b3c3949e9ad9c11e881e0801539ac41d7f799a560c07767baf41ca09bde69611\": rpc error: code = NotFound desc = could not find container \"b3c3949e9ad9c11e881e0801539ac41d7f799a560c07767baf41ca09bde69611\": container with ID starting with b3c3949e9ad9c11e881e0801539ac41d7f799a560c07767baf41ca09bde69611 not found: ID does not exist" Nov 03 00:42:12.455499 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.455416 4421 scope.go:122] "RemoveContainer" containerID="1f8d71c8d130bd5b79da4ea18f5069321fb7bf1c1961292af8d9817491086bfd" Nov 03 00:42:12.455499 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.455475 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 00:42:12.455732 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.455691 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f8d71c8d130bd5b79da4ea18f5069321fb7bf1c1961292af8d9817491086bfd"} err="failed to get container status \"1f8d71c8d130bd5b79da4ea18f5069321fb7bf1c1961292af8d9817491086bfd\": rpc error: code = NotFound desc = could not find container \"1f8d71c8d130bd5b79da4ea18f5069321fb7bf1c1961292af8d9817491086bfd\": container with ID starting with 1f8d71c8d130bd5b79da4ea18f5069321fb7bf1c1961292af8d9817491086bfd not found: ID does not exist" Nov 03 00:42:12.455776 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.455737 4421 scope.go:122] "RemoveContainer" containerID="4b65fc0aad202a4508032ae2b85c27a2ec39d81992613e965a77eabac8c974ec" Nov 03 00:42:12.456022 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.456001 4421 cpu_manager.go:414] "containerMap: removing container" podUID="d08334f4-4b91-45b2-bca0-e523054f59a7" containerName="console" Nov 03 00:42:12.456086 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.456028 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d08334f4-4b91-45b2-bca0-e523054f59a7" containerName="console" Nov 03 00:42:12.456086 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.456054 4421 cpu_manager.go:414] "containerMap: removing container" podUID="0d569850-d136-4cfd-b0f4-45e014ae8782" containerName="kube-rbac-proxy" Nov 03 00:42:12.456086 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.456064 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d569850-d136-4cfd-b0f4-45e014ae8782" containerName="kube-rbac-proxy" Nov 03 00:42:12.456086 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.456076 4421 cpu_manager.go:414] "containerMap: removing container" podUID="0d569850-d136-4cfd-b0f4-45e014ae8782" containerName="init-config-reloader" Nov 03 00:42:12.456264 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.456056 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b65fc0aad202a4508032ae2b85c27a2ec39d81992613e965a77eabac8c974ec"} err="failed to get container status \"4b65fc0aad202a4508032ae2b85c27a2ec39d81992613e965a77eabac8c974ec\": rpc error: code = NotFound desc = could not find container \"4b65fc0aad202a4508032ae2b85c27a2ec39d81992613e965a77eabac8c974ec\": container with ID starting with 4b65fc0aad202a4508032ae2b85c27a2ec39d81992613e965a77eabac8c974ec not found: ID does not exist" Nov 03 00:42:12.456264 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.456123 4421 scope.go:122] "RemoveContainer" containerID="1ffdf32dbac688f7664ff3fe6eda4cdf437caf839f622b6c9b26f71b7b324f30" Nov 03 00:42:12.456264 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.456087 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d569850-d136-4cfd-b0f4-45e014ae8782" containerName="init-config-reloader" Nov 03 00:42:12.456264 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.456195 4421 cpu_manager.go:414] "containerMap: removing container" podUID="0d569850-d136-4cfd-b0f4-45e014ae8782" containerName="kube-rbac-proxy-web" Nov 03 00:42:12.456264 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.456214 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d569850-d136-4cfd-b0f4-45e014ae8782" containerName="kube-rbac-proxy-web" Nov 03 00:42:12.456264 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.456241 4421 cpu_manager.go:414] "containerMap: removing container" podUID="0d569850-d136-4cfd-b0f4-45e014ae8782" containerName="thanos-sidecar" Nov 03 00:42:12.456264 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.456251 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d569850-d136-4cfd-b0f4-45e014ae8782" containerName="thanos-sidecar" Nov 03 00:42:12.456600 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.456290 4421 cpu_manager.go:414] "containerMap: removing container" podUID="0d569850-d136-4cfd-b0f4-45e014ae8782" containerName="config-reloader" Nov 03 00:42:12.456600 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.456299 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d569850-d136-4cfd-b0f4-45e014ae8782" containerName="config-reloader" Nov 03 00:42:12.456600 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.456305 4421 cpu_manager.go:414] "containerMap: removing container" podUID="0d569850-d136-4cfd-b0f4-45e014ae8782" containerName="kube-rbac-proxy-thanos" Nov 03 00:42:12.456600 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.456309 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d569850-d136-4cfd-b0f4-45e014ae8782" containerName="kube-rbac-proxy-thanos" Nov 03 00:42:12.456600 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.456317 4421 cpu_manager.go:414] "containerMap: removing container" podUID="0d569850-d136-4cfd-b0f4-45e014ae8782" containerName="prometheus" Nov 03 00:42:12.456600 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.456328 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d569850-d136-4cfd-b0f4-45e014ae8782" containerName="prometheus" Nov 03 00:42:12.456600 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.456394 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d569850-d136-4cfd-b0f4-45e014ae8782" containerName="thanos-sidecar" Nov 03 00:42:12.456600 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.456406 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d569850-d136-4cfd-b0f4-45e014ae8782" containerName="config-reloader" Nov 03 00:42:12.456600 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.456414 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d569850-d136-4cfd-b0f4-45e014ae8782" containerName="kube-rbac-proxy-thanos" Nov 03 00:42:12.456600 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.456422 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d569850-d136-4cfd-b0f4-45e014ae8782" containerName="prometheus" Nov 03 00:42:12.456600 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.456433 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="d08334f4-4b91-45b2-bca0-e523054f59a7" containerName="console" Nov 03 00:42:12.456600 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.456441 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d569850-d136-4cfd-b0f4-45e014ae8782" containerName="kube-rbac-proxy" Nov 03 00:42:12.456600 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.456451 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d569850-d136-4cfd-b0f4-45e014ae8782" containerName="kube-rbac-proxy-web" Nov 03 00:42:12.456600 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.456455 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ffdf32dbac688f7664ff3fe6eda4cdf437caf839f622b6c9b26f71b7b324f30"} err="failed to get container status \"1ffdf32dbac688f7664ff3fe6eda4cdf437caf839f622b6c9b26f71b7b324f30\": rpc error: code = NotFound desc = could not find container \"1ffdf32dbac688f7664ff3fe6eda4cdf437caf839f622b6c9b26f71b7b324f30\": container with ID starting with 1ffdf32dbac688f7664ff3fe6eda4cdf437caf839f622b6c9b26f71b7b324f30 not found: ID does not exist" Nov 03 00:42:12.456600 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.456484 4421 scope.go:122] "RemoveContainer" containerID="e3c8b487d6814dab847d9ff68549dcceaf2a4320bf8d16b49baf42b6fd2ad10f" Nov 03 00:42:12.457120 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.456711 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3c8b487d6814dab847d9ff68549dcceaf2a4320bf8d16b49baf42b6fd2ad10f"} err="failed to get container status \"e3c8b487d6814dab847d9ff68549dcceaf2a4320bf8d16b49baf42b6fd2ad10f\": rpc error: code = NotFound desc = could not find container \"e3c8b487d6814dab847d9ff68549dcceaf2a4320bf8d16b49baf42b6fd2ad10f\": container with ID starting with e3c8b487d6814dab847d9ff68549dcceaf2a4320bf8d16b49baf42b6fd2ad10f not found: ID does not exist" Nov 03 00:42:12.457120 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.456731 4421 scope.go:122] "RemoveContainer" containerID="974ed3b5941ffe78dba99eaafe74f0a3dd95a4cf48b6f7f65445938de5e94211" Nov 03 00:42:12.457120 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.456945 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"974ed3b5941ffe78dba99eaafe74f0a3dd95a4cf48b6f7f65445938de5e94211"} err="failed to get container status \"974ed3b5941ffe78dba99eaafe74f0a3dd95a4cf48b6f7f65445938de5e94211\": rpc error: code = NotFound desc = could not find container \"974ed3b5941ffe78dba99eaafe74f0a3dd95a4cf48b6f7f65445938de5e94211\": container with ID starting with 974ed3b5941ffe78dba99eaafe74f0a3dd95a4cf48b6f7f65445938de5e94211 not found: ID does not exist" Nov 03 00:42:12.457120 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.456967 4421 scope.go:122] "RemoveContainer" containerID="b7883446c04820fc06c84629eab2b4b99e7714c74a8b863660d5f231879cb14b" Nov 03 00:42:12.457309 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.457194 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7883446c04820fc06c84629eab2b4b99e7714c74a8b863660d5f231879cb14b"} err="failed to get container status \"b7883446c04820fc06c84629eab2b4b99e7714c74a8b863660d5f231879cb14b\": rpc error: code = NotFound desc = could not find container \"b7883446c04820fc06c84629eab2b4b99e7714c74a8b863660d5f231879cb14b\": container with ID starting with b7883446c04820fc06c84629eab2b4b99e7714c74a8b863660d5f231879cb14b not found: ID does not exist" Nov 03 00:42:12.461690 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.461674 4421 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0d569850-d136-4cfd-b0f4-45e014ae8782-secret-prometheus-k8s-tls\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:42:12.461775 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.461720 4421 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0d569850-d136-4cfd-b0f4-45e014ae8782-web-config\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:42:12.461775 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.461736 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c995h\" (UniqueName: \"kubernetes.io/projected/0d569850-d136-4cfd-b0f4-45e014ae8782-kube-api-access-c995h\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:42:12.461775 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.461751 4421 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0d569850-d136-4cfd-b0f4-45e014ae8782-config-out\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:42:12.461775 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.461764 4421 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/0d569850-d136-4cfd-b0f4-45e014ae8782-config\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:42:12.461960 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.461778 4421 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0d569850-d136-4cfd-b0f4-45e014ae8782-secret-metrics-client-certs\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:42:12.461960 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.461793 4421 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0d569850-d136-4cfd-b0f4-45e014ae8782-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:42:12.461960 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.461809 4421 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0d569850-d136-4cfd-b0f4-45e014ae8782-thanos-prometheus-http-client-file\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:42:12.461960 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.461829 4421 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0d569850-d136-4cfd-b0f4-45e014ae8782-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:42:12.461960 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.461844 4421 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0d569850-d136-4cfd-b0f4-45e014ae8782-secret-kube-rbac-proxy\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:42:12.461960 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.461858 4421 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0d569850-d136-4cfd-b0f4-45e014ae8782-prometheus-k8s-db\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:42:12.461960 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.461872 4421 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0d569850-d136-4cfd-b0f4-45e014ae8782-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:42:12.461960 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.461886 4421 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0d569850-d136-4cfd-b0f4-45e014ae8782-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:42:12.461960 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.461902 4421 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0d569850-d136-4cfd-b0f4-45e014ae8782-configmap-metrics-client-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:42:12.497216 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.497187 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 00:42:12.497375 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.497366 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:42:12.503608 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.503551 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Nov 03 00:42:12.503608 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.503551 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Nov 03 00:42:12.503996 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.503622 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Nov 03 00:42:12.503996 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.503758 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Nov 03 00:42:12.503996 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.503837 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Nov 03 00:42:12.503996 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.503843 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Nov 03 00:42:12.504198 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.504182 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Nov 03 00:42:12.504707 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.504686 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Nov 03 00:42:12.504707 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.504700 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-5prsha9bc71b0\"" Nov 03 00:42:12.504949 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.504929 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Nov 03 00:42:12.505040 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.504966 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Nov 03 00:42:12.505211 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.505195 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Nov 03 00:42:12.505268 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.505256 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Nov 03 00:42:12.505326 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.505266 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-x8s4k\"" Nov 03 00:42:12.505326 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.505295 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Nov 03 00:42:12.520435 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.520381 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Nov 03 00:42:12.529154 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.529136 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Nov 03 00:42:12.562606 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.562564 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/252de3ff-2d31-4c35-961f-a93acc666093-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"252de3ff-2d31-4c35-961f-a93acc666093\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:42:12.562819 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.562624 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/252de3ff-2d31-4c35-961f-a93acc666093-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"252de3ff-2d31-4c35-961f-a93acc666093\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:42:12.562819 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.562717 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/252de3ff-2d31-4c35-961f-a93acc666093-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"252de3ff-2d31-4c35-961f-a93acc666093\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:42:12.562819 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.562766 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/252de3ff-2d31-4c35-961f-a93acc666093-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"252de3ff-2d31-4c35-961f-a93acc666093\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:42:12.562819 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.562809 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/252de3ff-2d31-4c35-961f-a93acc666093-config-out\") pod \"prometheus-k8s-0\" (UID: \"252de3ff-2d31-4c35-961f-a93acc666093\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:42:12.563035 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.562826 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/252de3ff-2d31-4c35-961f-a93acc666093-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"252de3ff-2d31-4c35-961f-a93acc666093\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:42:12.563035 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.562869 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/252de3ff-2d31-4c35-961f-a93acc666093-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"252de3ff-2d31-4c35-961f-a93acc666093\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:42:12.565347 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.565316 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/252de3ff-2d31-4c35-961f-a93acc666093-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"252de3ff-2d31-4c35-961f-a93acc666093\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:42:12.565491 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.565398 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/252de3ff-2d31-4c35-961f-a93acc666093-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"252de3ff-2d31-4c35-961f-a93acc666093\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:42:12.565491 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.565440 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/252de3ff-2d31-4c35-961f-a93acc666093-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"252de3ff-2d31-4c35-961f-a93acc666093\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:42:12.565491 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.565472 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/252de3ff-2d31-4c35-961f-a93acc666093-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"252de3ff-2d31-4c35-961f-a93acc666093\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:42:12.565674 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.565496 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/252de3ff-2d31-4c35-961f-a93acc666093-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"252de3ff-2d31-4c35-961f-a93acc666093\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:42:12.565674 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.565556 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/252de3ff-2d31-4c35-961f-a93acc666093-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"252de3ff-2d31-4c35-961f-a93acc666093\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:42:12.565674 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.565588 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/252de3ff-2d31-4c35-961f-a93acc666093-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"252de3ff-2d31-4c35-961f-a93acc666093\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:42:12.565674 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.565608 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrph6\" (UniqueName: \"kubernetes.io/projected/252de3ff-2d31-4c35-961f-a93acc666093-kube-api-access-xrph6\") pod \"prometheus-k8s-0\" (UID: \"252de3ff-2d31-4c35-961f-a93acc666093\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:42:12.565674 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.565637 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/252de3ff-2d31-4c35-961f-a93acc666093-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"252de3ff-2d31-4c35-961f-a93acc666093\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:42:12.565899 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.565792 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/252de3ff-2d31-4c35-961f-a93acc666093-web-config\") pod \"prometheus-k8s-0\" (UID: \"252de3ff-2d31-4c35-961f-a93acc666093\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:42:12.565899 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.565826 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/252de3ff-2d31-4c35-961f-a93acc666093-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"252de3ff-2d31-4c35-961f-a93acc666093\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:42:12.565899 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.565872 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/252de3ff-2d31-4c35-961f-a93acc666093-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"252de3ff-2d31-4c35-961f-a93acc666093\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:42:12.566045 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.565930 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/252de3ff-2d31-4c35-961f-a93acc666093-config\") pod \"prometheus-k8s-0\" (UID: \"252de3ff-2d31-4c35-961f-a93acc666093\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:42:12.667035 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.666994 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/252de3ff-2d31-4c35-961f-a93acc666093-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"252de3ff-2d31-4c35-961f-a93acc666093\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:42:12.667035 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.667033 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/252de3ff-2d31-4c35-961f-a93acc666093-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"252de3ff-2d31-4c35-961f-a93acc666093\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:42:12.667269 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.667056 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/252de3ff-2d31-4c35-961f-a93acc666093-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"252de3ff-2d31-4c35-961f-a93acc666093\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:42:12.667269 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.667108 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/252de3ff-2d31-4c35-961f-a93acc666093-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"252de3ff-2d31-4c35-961f-a93acc666093\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:42:12.667269 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.667145 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/252de3ff-2d31-4c35-961f-a93acc666093-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"252de3ff-2d31-4c35-961f-a93acc666093\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:42:12.667269 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.667164 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/252de3ff-2d31-4c35-961f-a93acc666093-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"252de3ff-2d31-4c35-961f-a93acc666093\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:42:12.667269 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.667179 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/252de3ff-2d31-4c35-961f-a93acc666093-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"252de3ff-2d31-4c35-961f-a93acc666093\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:42:12.667269 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.667204 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/252de3ff-2d31-4c35-961f-a93acc666093-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"252de3ff-2d31-4c35-961f-a93acc666093\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:42:12.667269 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.667233 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrph6\" (UniqueName: \"kubernetes.io/projected/252de3ff-2d31-4c35-961f-a93acc666093-kube-api-access-xrph6\") pod \"prometheus-k8s-0\" (UID: \"252de3ff-2d31-4c35-961f-a93acc666093\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:42:12.667269 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.667265 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/252de3ff-2d31-4c35-961f-a93acc666093-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"252de3ff-2d31-4c35-961f-a93acc666093\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:42:12.667680 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.667322 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/252de3ff-2d31-4c35-961f-a93acc666093-web-config\") pod \"prometheus-k8s-0\" (UID: \"252de3ff-2d31-4c35-961f-a93acc666093\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:42:12.667680 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.667346 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/252de3ff-2d31-4c35-961f-a93acc666093-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"252de3ff-2d31-4c35-961f-a93acc666093\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:42:12.667680 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.667374 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/252de3ff-2d31-4c35-961f-a93acc666093-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"252de3ff-2d31-4c35-961f-a93acc666093\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:42:12.667680 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.667410 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/252de3ff-2d31-4c35-961f-a93acc666093-config\") pod \"prometheus-k8s-0\" (UID: \"252de3ff-2d31-4c35-961f-a93acc666093\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:42:12.667891 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.667744 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/252de3ff-2d31-4c35-961f-a93acc666093-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"252de3ff-2d31-4c35-961f-a93acc666093\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:42:12.667891 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.667851 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/252de3ff-2d31-4c35-961f-a93acc666093-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"252de3ff-2d31-4c35-961f-a93acc666093\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:42:12.667999 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.667927 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/252de3ff-2d31-4c35-961f-a93acc666093-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"252de3ff-2d31-4c35-961f-a93acc666093\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:42:12.668568 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.668530 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/252de3ff-2d31-4c35-961f-a93acc666093-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"252de3ff-2d31-4c35-961f-a93acc666093\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:42:12.670687 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.670409 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/252de3ff-2d31-4c35-961f-a93acc666093-config\") pod \"prometheus-k8s-0\" (UID: \"252de3ff-2d31-4c35-961f-a93acc666093\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:42:12.670687 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.670468 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/252de3ff-2d31-4c35-961f-a93acc666093-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"252de3ff-2d31-4c35-961f-a93acc666093\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:42:12.670687 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.670505 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/252de3ff-2d31-4c35-961f-a93acc666093-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"252de3ff-2d31-4c35-961f-a93acc666093\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:42:12.670687 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.670513 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/252de3ff-2d31-4c35-961f-a93acc666093-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"252de3ff-2d31-4c35-961f-a93acc666093\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:42:12.670687 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.670539 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/252de3ff-2d31-4c35-961f-a93acc666093-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"252de3ff-2d31-4c35-961f-a93acc666093\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:42:12.670687 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.670549 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/252de3ff-2d31-4c35-961f-a93acc666093-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"252de3ff-2d31-4c35-961f-a93acc666093\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:42:12.670687 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.670591 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/252de3ff-2d31-4c35-961f-a93acc666093-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"252de3ff-2d31-4c35-961f-a93acc666093\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:42:12.670687 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.670680 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/252de3ff-2d31-4c35-961f-a93acc666093-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"252de3ff-2d31-4c35-961f-a93acc666093\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:42:12.671156 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.670731 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/252de3ff-2d31-4c35-961f-a93acc666093-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"252de3ff-2d31-4c35-961f-a93acc666093\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:42:12.671156 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.670736 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/252de3ff-2d31-4c35-961f-a93acc666093-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"252de3ff-2d31-4c35-961f-a93acc666093\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:42:12.671156 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.670768 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/252de3ff-2d31-4c35-961f-a93acc666093-config-out\") pod \"prometheus-k8s-0\" (UID: \"252de3ff-2d31-4c35-961f-a93acc666093\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:42:12.671156 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.670789 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/252de3ff-2d31-4c35-961f-a93acc666093-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"252de3ff-2d31-4c35-961f-a93acc666093\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:42:12.671156 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.671000 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/252de3ff-2d31-4c35-961f-a93acc666093-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"252de3ff-2d31-4c35-961f-a93acc666093\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:42:12.671488 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.671358 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/252de3ff-2d31-4c35-961f-a93acc666093-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"252de3ff-2d31-4c35-961f-a93acc666093\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:42:12.671488 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.671432 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/252de3ff-2d31-4c35-961f-a93acc666093-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"252de3ff-2d31-4c35-961f-a93acc666093\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:42:12.671488 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.671475 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/252de3ff-2d31-4c35-961f-a93acc666093-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"252de3ff-2d31-4c35-961f-a93acc666093\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:42:12.671654 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.671548 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/252de3ff-2d31-4c35-961f-a93acc666093-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"252de3ff-2d31-4c35-961f-a93acc666093\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:42:12.672841 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.672812 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/252de3ff-2d31-4c35-961f-a93acc666093-web-config\") pod \"prometheus-k8s-0\" (UID: \"252de3ff-2d31-4c35-961f-a93acc666093\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:42:12.673210 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.673191 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/252de3ff-2d31-4c35-961f-a93acc666093-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"252de3ff-2d31-4c35-961f-a93acc666093\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:42:12.673365 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.673347 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/252de3ff-2d31-4c35-961f-a93acc666093-config-out\") pod \"prometheus-k8s-0\" (UID: \"252de3ff-2d31-4c35-961f-a93acc666093\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:42:12.673954 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.673937 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/252de3ff-2d31-4c35-961f-a93acc666093-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"252de3ff-2d31-4c35-961f-a93acc666093\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:42:12.681656 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.681638 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrph6\" (UniqueName: \"kubernetes.io/projected/252de3ff-2d31-4c35-961f-a93acc666093-kube-api-access-xrph6\") pod \"prometheus-k8s-0\" (UID: \"252de3ff-2d31-4c35-961f-a93acc666093\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:42:12.912439 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:12.912354 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:42:13.229878 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:13.225360 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 00:42:13.396183 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:13.396152 4421 generic.go:356] "Generic (PLEG): container finished" podID="252de3ff-2d31-4c35-961f-a93acc666093" containerID="f3563b43bafd004e98f876e5d2c2403175a033b66d05e18fe05ae2d62e6a863d" exitCode=0 Nov 03 00:42:13.396351 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:13.396215 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"252de3ff-2d31-4c35-961f-a93acc666093","Type":"ContainerDied","Data":"f3563b43bafd004e98f876e5d2c2403175a033b66d05e18fe05ae2d62e6a863d"} Nov 03 00:42:13.396351 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:13.396269 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"252de3ff-2d31-4c35-961f-a93acc666093","Type":"ContainerStarted","Data":"5b369b8dc2bd401e4a90869d071b285b1f0b30272b36347422077aa0b6a6a871"} Nov 03 00:42:13.978594 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:13.978562 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d569850-d136-4cfd-b0f4-45e014ae8782" path="/var/lib/kubelet/pods/0d569850-d136-4cfd-b0f4-45e014ae8782/volumes" Nov 03 00:42:14.403356 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:14.403252 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"252de3ff-2d31-4c35-961f-a93acc666093","Type":"ContainerStarted","Data":"280076862e45cb819a3b32703a236b35caa9fd7c86a07c92600d1a16ce930bf7"} Nov 03 00:42:14.403356 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:14.403341 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"252de3ff-2d31-4c35-961f-a93acc666093","Type":"ContainerStarted","Data":"794298f0cf081f5cf1b18b25d53c4d4c499b00a34dbad575743110b7ac4080a0"} Nov 03 00:42:14.403784 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:14.403382 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"252de3ff-2d31-4c35-961f-a93acc666093","Type":"ContainerStarted","Data":"36afaf8369221c281122c7fae6d567ec49050ccd9f059107653c1fd011f1ac39"} Nov 03 00:42:14.403784 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:14.403411 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"252de3ff-2d31-4c35-961f-a93acc666093","Type":"ContainerStarted","Data":"e3b32da78714e21b49c80050ab5df5d4f2e42ae70dd85f36e5f7fbcf36cae35d"} Nov 03 00:42:14.403784 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:14.403435 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"252de3ff-2d31-4c35-961f-a93acc666093","Type":"ContainerStarted","Data":"bb8beae294f452f39df189c038fb7a530c6583c73e549c77e0ac5d11cb974380"} Nov 03 00:42:14.403784 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:14.403470 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"252de3ff-2d31-4c35-961f-a93acc666093","Type":"ContainerStarted","Data":"6db0b6bbd34cd61a0a3e83163aeae67fb660d7c296b0573de281e8332f72ceff"} Nov 03 00:42:14.437118 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:14.437069 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.415729629 podStartE2EDuration="2.437055678s" totalImagesPullingTime="0s" totalInitContainerRuntime="21.326049ms" isStatefulPod=false podCreationTimestamp="2026-11-03 00:42:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:42:14.435959215 +0000 UTC m=+76.606962350" watchObservedRunningTime="2026-11-03 00:42:14.437055678 +0000 UTC m=+76.608058807" Nov 03 00:42:14.668876 ip-10-0-133-217 kubenswrapper[4421]: E1103 00:42:14.668796 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 00:42:17.912563 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:42:17.912528 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:42:44.667437 ip-10-0-133-217 kubenswrapper[4421]: E1103 00:42:44.667397 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 00:43:12.913503 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:43:12.913401 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:43:12.929268 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:43:12.929237 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:43:13.628994 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:43:13.628965 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:43:14.667968 ip-10-0-133-217 kubenswrapper[4421]: E1103 00:43:14.667931 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 00:43:44.667550 ip-10-0-133-217 kubenswrapper[4421]: E1103 00:43:44.667512 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 00:44:14.667734 ip-10-0-133-217 kubenswrapper[4421]: E1103 00:44:14.667685 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 00:44:44.668532 ip-10-0-133-217 kubenswrapper[4421]: E1103 00:44:44.668438 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 00:44:51.084970 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:44:51.084894 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7cc9488979-bj5xx"] Nov 03 00:44:51.087152 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:44:51.087134 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7cc9488979-bj5xx" Nov 03 00:44:51.185010 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:44:51.184975 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Nov 03 00:44:51.185424 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:44:51.185392 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Nov 03 00:44:51.185568 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:44:51.185395 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Nov 03 00:44:51.185568 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:44:51.185532 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-p7qvx\"" Nov 03 00:44:51.187721 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:44:51.187697 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Nov 03 00:44:51.187990 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:44:51.187976 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Nov 03 00:44:51.230102 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:44:51.230063 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/816475cf-e48b-4d1a-a20f-16c62118bb0f-webhook-cert\") pod \"opendatahub-operator-controller-manager-7cc9488979-bj5xx\" (UID: \"816475cf-e48b-4d1a-a20f-16c62118bb0f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7cc9488979-bj5xx" Nov 03 00:44:51.230266 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:44:51.230119 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q57r7\" (UniqueName: \"kubernetes.io/projected/816475cf-e48b-4d1a-a20f-16c62118bb0f-kube-api-access-q57r7\") pod \"opendatahub-operator-controller-manager-7cc9488979-bj5xx\" (UID: \"816475cf-e48b-4d1a-a20f-16c62118bb0f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7cc9488979-bj5xx" Nov 03 00:44:51.230266 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:44:51.230197 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/816475cf-e48b-4d1a-a20f-16c62118bb0f-tmp\") pod \"opendatahub-operator-controller-manager-7cc9488979-bj5xx\" (UID: \"816475cf-e48b-4d1a-a20f-16c62118bb0f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7cc9488979-bj5xx" Nov 03 00:44:51.230266 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:44:51.230244 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/816475cf-e48b-4d1a-a20f-16c62118bb0f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7cc9488979-bj5xx\" (UID: \"816475cf-e48b-4d1a-a20f-16c62118bb0f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7cc9488979-bj5xx" Nov 03 00:44:51.230420 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:44:51.230269 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/816475cf-e48b-4d1a-a20f-16c62118bb0f-manifests\") pod \"opendatahub-operator-controller-manager-7cc9488979-bj5xx\" (UID: \"816475cf-e48b-4d1a-a20f-16c62118bb0f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7cc9488979-bj5xx" Nov 03 00:44:51.230420 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:44:51.230365 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/816475cf-e48b-4d1a-a20f-16c62118bb0f-metrics-cert\") pod \"opendatahub-operator-controller-manager-7cc9488979-bj5xx\" (UID: \"816475cf-e48b-4d1a-a20f-16c62118bb0f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7cc9488979-bj5xx" Nov 03 00:44:51.264149 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:44:51.264115 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7cc9488979-bj5xx"] Nov 03 00:44:51.331141 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:44:51.331099 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q57r7\" (UniqueName: \"kubernetes.io/projected/816475cf-e48b-4d1a-a20f-16c62118bb0f-kube-api-access-q57r7\") pod \"opendatahub-operator-controller-manager-7cc9488979-bj5xx\" (UID: \"816475cf-e48b-4d1a-a20f-16c62118bb0f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7cc9488979-bj5xx" Nov 03 00:44:51.331141 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:44:51.331145 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/816475cf-e48b-4d1a-a20f-16c62118bb0f-tmp\") pod \"opendatahub-operator-controller-manager-7cc9488979-bj5xx\" (UID: \"816475cf-e48b-4d1a-a20f-16c62118bb0f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7cc9488979-bj5xx" Nov 03 00:44:51.331426 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:44:51.331176 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/816475cf-e48b-4d1a-a20f-16c62118bb0f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7cc9488979-bj5xx\" (UID: \"816475cf-e48b-4d1a-a20f-16c62118bb0f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7cc9488979-bj5xx" Nov 03 00:44:51.331426 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:44:51.331199 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/816475cf-e48b-4d1a-a20f-16c62118bb0f-manifests\") pod \"opendatahub-operator-controller-manager-7cc9488979-bj5xx\" (UID: \"816475cf-e48b-4d1a-a20f-16c62118bb0f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7cc9488979-bj5xx" Nov 03 00:44:51.331426 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:44:51.331221 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/816475cf-e48b-4d1a-a20f-16c62118bb0f-metrics-cert\") pod \"opendatahub-operator-controller-manager-7cc9488979-bj5xx\" (UID: \"816475cf-e48b-4d1a-a20f-16c62118bb0f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7cc9488979-bj5xx" Nov 03 00:44:51.331426 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:44:51.331251 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/816475cf-e48b-4d1a-a20f-16c62118bb0f-webhook-cert\") pod \"opendatahub-operator-controller-manager-7cc9488979-bj5xx\" (UID: \"816475cf-e48b-4d1a-a20f-16c62118bb0f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7cc9488979-bj5xx" Nov 03 00:44:51.331825 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:44:51.331794 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/816475cf-e48b-4d1a-a20f-16c62118bb0f-manifests\") pod \"opendatahub-operator-controller-manager-7cc9488979-bj5xx\" (UID: \"816475cf-e48b-4d1a-a20f-16c62118bb0f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7cc9488979-bj5xx" Nov 03 00:44:51.333740 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:44:51.333709 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/816475cf-e48b-4d1a-a20f-16c62118bb0f-tmp\") pod \"opendatahub-operator-controller-manager-7cc9488979-bj5xx\" (UID: \"816475cf-e48b-4d1a-a20f-16c62118bb0f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7cc9488979-bj5xx" Nov 03 00:44:51.333890 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:44:51.333762 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/816475cf-e48b-4d1a-a20f-16c62118bb0f-metrics-cert\") pod \"opendatahub-operator-controller-manager-7cc9488979-bj5xx\" (UID: \"816475cf-e48b-4d1a-a20f-16c62118bb0f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7cc9488979-bj5xx" Nov 03 00:44:51.333953 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:44:51.333894 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/816475cf-e48b-4d1a-a20f-16c62118bb0f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7cc9488979-bj5xx\" (UID: \"816475cf-e48b-4d1a-a20f-16c62118bb0f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7cc9488979-bj5xx" Nov 03 00:44:51.333953 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:44:51.333895 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/816475cf-e48b-4d1a-a20f-16c62118bb0f-webhook-cert\") pod \"opendatahub-operator-controller-manager-7cc9488979-bj5xx\" (UID: \"816475cf-e48b-4d1a-a20f-16c62118bb0f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7cc9488979-bj5xx" Nov 03 00:44:51.341638 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:44:51.341578 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q57r7\" (UniqueName: \"kubernetes.io/projected/816475cf-e48b-4d1a-a20f-16c62118bb0f-kube-api-access-q57r7\") pod \"opendatahub-operator-controller-manager-7cc9488979-bj5xx\" (UID: \"816475cf-e48b-4d1a-a20f-16c62118bb0f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7cc9488979-bj5xx" Nov 03 00:44:51.398629 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:44:51.398585 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7cc9488979-bj5xx" Nov 03 00:44:51.691759 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:44:51.691664 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7cc9488979-bj5xx"] Nov 03 00:44:51.695106 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:44:51.695060 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod816475cf_e48b_4d1a_a20f_16c62118bb0f.slice/crio-61fa8b58b209134a9b5d452ef16a247c2ffdd815ed6601890def3dc11653be51 WatchSource:0}: Error finding container 61fa8b58b209134a9b5d452ef16a247c2ffdd815ed6601890def3dc11653be51: Status 404 returned error can't find the container with id 61fa8b58b209134a9b5d452ef16a247c2ffdd815ed6601890def3dc11653be51 Nov 03 00:44:51.964623 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:44:51.964527 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7cc9488979-bj5xx" event={"ID":"816475cf-e48b-4d1a-a20f-16c62118bb0f","Type":"ContainerStarted","Data":"61fa8b58b209134a9b5d452ef16a247c2ffdd815ed6601890def3dc11653be51"} Nov 03 00:44:54.978953 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:44:54.978856 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7cc9488979-bj5xx" event={"ID":"816475cf-e48b-4d1a-a20f-16c62118bb0f","Type":"ContainerStarted","Data":"98d4d4749a48b57295b2cb5348db9dd1eb3f8a60b56d403d3f47f8afeb50ec19"} Nov 03 00:44:55.983900 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:44:55.983860 4421 generic.go:356] "Generic (PLEG): container finished" podID="816475cf-e48b-4d1a-a20f-16c62118bb0f" containerID="98d4d4749a48b57295b2cb5348db9dd1eb3f8a60b56d403d3f47f8afeb50ec19" exitCode=0 Nov 03 00:44:55.984314 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:44:55.983942 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7cc9488979-bj5xx" event={"ID":"816475cf-e48b-4d1a-a20f-16c62118bb0f","Type":"ContainerDied","Data":"98d4d4749a48b57295b2cb5348db9dd1eb3f8a60b56d403d3f47f8afeb50ec19"} Nov 03 00:44:56.989025 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:44:56.988925 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7cc9488979-bj5xx" event={"ID":"816475cf-e48b-4d1a-a20f-16c62118bb0f","Type":"ContainerStarted","Data":"589d4909ce04b8b7900a7b3f694cc99c8eab9d5b4af93d36c112625954a9be0d"} Nov 03 00:44:56.989666 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:44:56.989618 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7cc9488979-bj5xx" Nov 03 00:44:57.145595 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:44:57.145540 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7cc9488979-bj5xx" podStartSLOduration=2.223333651 podStartE2EDuration="6.14552554s" totalImagesPullingTime="3.459240246s" totalInitContainerRuntime="462.951643ms" isStatefulPod=false podCreationTimestamp="2026-11-03 00:44:51 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:44:57.127506724 +0000 UTC m=+239.298509820" watchObservedRunningTime="2026-11-03 00:44:57.14552554 +0000 UTC m=+239.316528635" Nov 03 00:45:07.994916 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:45:07.994888 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7cc9488979-bj5xx" Nov 03 00:45:13.635833 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:45:13.635799 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-ktz44"] Nov 03 00:45:13.639181 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:45:13.639159 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-ktz44" Nov 03 00:45:13.651685 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:45:13.651657 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-7fx89\"" Nov 03 00:45:13.651803 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:45:13.651751 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Nov 03 00:45:13.651803 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:45:13.651795 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Nov 03 00:45:13.652319 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:45:13.652293 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-ktz44"] Nov 03 00:45:13.711130 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:45:13.711091 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ee2112c8-97e1-487d-aba9-66cf6ff002e7-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-ktz44\" (UID: \"ee2112c8-97e1-487d-aba9-66cf6ff002e7\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-ktz44" Nov 03 00:45:13.711338 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:45:13.711228 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hksl\" (UniqueName: \"kubernetes.io/projected/ee2112c8-97e1-487d-aba9-66cf6ff002e7-kube-api-access-4hksl\") pod \"cert-manager-webhook-5dbbbf75cc-ktz44\" (UID: \"ee2112c8-97e1-487d-aba9-66cf6ff002e7\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-ktz44" Nov 03 00:45:13.811633 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:45:13.811600 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ee2112c8-97e1-487d-aba9-66cf6ff002e7-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-ktz44\" (UID: \"ee2112c8-97e1-487d-aba9-66cf6ff002e7\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-ktz44" Nov 03 00:45:13.811840 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:45:13.811711 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hksl\" (UniqueName: \"kubernetes.io/projected/ee2112c8-97e1-487d-aba9-66cf6ff002e7-kube-api-access-4hksl\") pod \"cert-manager-webhook-5dbbbf75cc-ktz44\" (UID: \"ee2112c8-97e1-487d-aba9-66cf6ff002e7\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-ktz44" Nov 03 00:45:13.828351 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:45:13.828307 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hksl\" (UniqueName: \"kubernetes.io/projected/ee2112c8-97e1-487d-aba9-66cf6ff002e7-kube-api-access-4hksl\") pod \"cert-manager-webhook-5dbbbf75cc-ktz44\" (UID: \"ee2112c8-97e1-487d-aba9-66cf6ff002e7\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-ktz44" Nov 03 00:45:13.828556 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:45:13.828376 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ee2112c8-97e1-487d-aba9-66cf6ff002e7-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-ktz44\" (UID: \"ee2112c8-97e1-487d-aba9-66cf6ff002e7\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-ktz44" Nov 03 00:45:13.958565 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:45:13.958466 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-ktz44" Nov 03 00:45:14.235243 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:45:14.235217 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-ktz44"] Nov 03 00:45:14.237996 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:45:14.237968 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podee2112c8_97e1_487d_aba9_66cf6ff002e7.slice/crio-f0a4bf9e12f087f870c99107e65e25311b69acd142b40705d29114d1ef7d58fc WatchSource:0}: Error finding container f0a4bf9e12f087f870c99107e65e25311b69acd142b40705d29114d1ef7d58fc: Status 404 returned error can't find the container with id f0a4bf9e12f087f870c99107e65e25311b69acd142b40705d29114d1ef7d58fc Nov 03 00:45:14.669249 ip-10-0-133-217 kubenswrapper[4421]: E1103 00:45:14.669210 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 00:45:15.056213 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:45:15.056175 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-ktz44" event={"ID":"ee2112c8-97e1-487d-aba9-66cf6ff002e7","Type":"ContainerStarted","Data":"f0a4bf9e12f087f870c99107e65e25311b69acd142b40705d29114d1ef7d58fc"} Nov 03 00:45:18.113475 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:45:18.113427 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-ktz44" event={"ID":"ee2112c8-97e1-487d-aba9-66cf6ff002e7","Type":"ContainerStarted","Data":"9062485ae14b606de61ca6000377f553a9a7ece3146f30ac7975ac65e5cc2b5e"} Nov 03 00:45:18.114006 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:45:18.113612 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-ktz44" Nov 03 00:45:18.129828 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:45:18.129775 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-ktz44" podStartSLOduration=1.333252025 podStartE2EDuration="5.129761056s" totalImagesPullingTime="3.796509031s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:45:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:45:18.129722937 +0000 UTC m=+260.300726032" watchObservedRunningTime="2026-11-03 00:45:18.129761056 +0000 UTC m=+260.300764152" Nov 03 00:45:19.155844 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:45:19.155811 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-4222l"] Nov 03 00:45:19.159069 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:45:19.159048 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-4222l" Nov 03 00:45:19.168446 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:45:19.168422 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-4222l"] Nov 03 00:45:19.172702 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:45:19.172678 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Nov 03 00:45:19.173842 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:45:19.173823 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Nov 03 00:45:19.173953 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:45:19.173905 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-w9pxr\"" Nov 03 00:45:19.173953 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:45:19.173911 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Nov 03 00:45:19.173953 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:45:19.173949 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Nov 03 00:45:19.260521 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:45:19.260483 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/c2040caa-0454-46bb-bf81-a7063657d4ad-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-4222l\" (UID: \"c2040caa-0454-46bb-bf81-a7063657d4ad\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-4222l" Nov 03 00:45:19.260521 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:45:19.260531 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jknrd\" (UniqueName: \"kubernetes.io/projected/c2040caa-0454-46bb-bf81-a7063657d4ad-kube-api-access-jknrd\") pod \"kuadrant-console-plugin-66c4fb575d-4222l\" (UID: \"c2040caa-0454-46bb-bf81-a7063657d4ad\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-4222l" Nov 03 00:45:19.260751 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:45:19.260559 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c2040caa-0454-46bb-bf81-a7063657d4ad-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-4222l\" (UID: \"c2040caa-0454-46bb-bf81-a7063657d4ad\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-4222l" Nov 03 00:45:19.361241 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:45:19.361203 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/c2040caa-0454-46bb-bf81-a7063657d4ad-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-4222l\" (UID: \"c2040caa-0454-46bb-bf81-a7063657d4ad\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-4222l" Nov 03 00:45:19.361483 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:45:19.361250 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jknrd\" (UniqueName: \"kubernetes.io/projected/c2040caa-0454-46bb-bf81-a7063657d4ad-kube-api-access-jknrd\") pod \"kuadrant-console-plugin-66c4fb575d-4222l\" (UID: \"c2040caa-0454-46bb-bf81-a7063657d4ad\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-4222l" Nov 03 00:45:19.361483 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:45:19.361297 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c2040caa-0454-46bb-bf81-a7063657d4ad-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-4222l\" (UID: \"c2040caa-0454-46bb-bf81-a7063657d4ad\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-4222l" Nov 03 00:45:19.361941 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:45:19.361920 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c2040caa-0454-46bb-bf81-a7063657d4ad-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-4222l\" (UID: \"c2040caa-0454-46bb-bf81-a7063657d4ad\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-4222l" Nov 03 00:45:19.363762 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:45:19.363739 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/c2040caa-0454-46bb-bf81-a7063657d4ad-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-4222l\" (UID: \"c2040caa-0454-46bb-bf81-a7063657d4ad\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-4222l" Nov 03 00:45:19.371194 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:45:19.371174 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jknrd\" (UniqueName: \"kubernetes.io/projected/c2040caa-0454-46bb-bf81-a7063657d4ad-kube-api-access-jknrd\") pod \"kuadrant-console-plugin-66c4fb575d-4222l\" (UID: \"c2040caa-0454-46bb-bf81-a7063657d4ad\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-4222l" Nov 03 00:45:19.470422 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:45:19.470328 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-4222l" Nov 03 00:45:19.745116 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:45:19.745090 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-4222l"] Nov 03 00:45:19.747338 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:45:19.747303 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc2040caa_0454_46bb_bf81_a7063657d4ad.slice/crio-9de5982f3f823626fce4a2c85a6e10ba97fdb1a6c0e1332c38f89a29bfb13a10 WatchSource:0}: Error finding container 9de5982f3f823626fce4a2c85a6e10ba97fdb1a6c0e1332c38f89a29bfb13a10: Status 404 returned error can't find the container with id 9de5982f3f823626fce4a2c85a6e10ba97fdb1a6c0e1332c38f89a29bfb13a10 Nov 03 00:45:20.122228 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:45:20.122190 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-4222l" event={"ID":"c2040caa-0454-46bb-bf81-a7063657d4ad","Type":"ContainerStarted","Data":"9de5982f3f823626fce4a2c85a6e10ba97fdb1a6c0e1332c38f89a29bfb13a10"} Nov 03 00:45:20.724096 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:45:20.724063 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-b2m52"] Nov 03 00:45:20.728391 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:45:20.728373 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-b2m52" Nov 03 00:45:20.734706 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:45:20.734684 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-w26wb\"" Nov 03 00:45:20.738694 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:45:20.738662 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-b2m52"] Nov 03 00:45:20.771995 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:45:20.771963 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8sbrw\" (UniqueName: \"kubernetes.io/projected/ce8cb8c1-3506-45e8-8f70-83131970c3d5-kube-api-access-8sbrw\") pod \"cert-manager-cainjector-5c57bbd4fc-b2m52\" (UID: \"ce8cb8c1-3506-45e8-8f70-83131970c3d5\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-b2m52" Nov 03 00:45:20.772154 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:45:20.772021 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ce8cb8c1-3506-45e8-8f70-83131970c3d5-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-b2m52\" (UID: \"ce8cb8c1-3506-45e8-8f70-83131970c3d5\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-b2m52" Nov 03 00:45:20.873284 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:45:20.873246 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8sbrw\" (UniqueName: \"kubernetes.io/projected/ce8cb8c1-3506-45e8-8f70-83131970c3d5-kube-api-access-8sbrw\") pod \"cert-manager-cainjector-5c57bbd4fc-b2m52\" (UID: \"ce8cb8c1-3506-45e8-8f70-83131970c3d5\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-b2m52" Nov 03 00:45:20.873463 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:45:20.873314 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ce8cb8c1-3506-45e8-8f70-83131970c3d5-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-b2m52\" (UID: \"ce8cb8c1-3506-45e8-8f70-83131970c3d5\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-b2m52" Nov 03 00:45:20.884255 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:45:20.884221 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ce8cb8c1-3506-45e8-8f70-83131970c3d5-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-b2m52\" (UID: \"ce8cb8c1-3506-45e8-8f70-83131970c3d5\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-b2m52" Nov 03 00:45:20.884391 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:45:20.884359 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8sbrw\" (UniqueName: \"kubernetes.io/projected/ce8cb8c1-3506-45e8-8f70-83131970c3d5-kube-api-access-8sbrw\") pod \"cert-manager-cainjector-5c57bbd4fc-b2m52\" (UID: \"ce8cb8c1-3506-45e8-8f70-83131970c3d5\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-b2m52" Nov 03 00:45:21.040253 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:45:21.040217 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-b2m52" Nov 03 00:45:21.317655 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:45:21.317574 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-b2m52"] Nov 03 00:45:21.321240 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:45:21.321208 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podce8cb8c1_3506_45e8_8f70_83131970c3d5.slice/crio-9c19a5c4608abfcf98dd819d016540b7a0a63daca34ebea0c587810cfbdbeb09 WatchSource:0}: Error finding container 9c19a5c4608abfcf98dd819d016540b7a0a63daca34ebea0c587810cfbdbeb09: Status 404 returned error can't find the container with id 9c19a5c4608abfcf98dd819d016540b7a0a63daca34ebea0c587810cfbdbeb09 Nov 03 00:45:22.134388 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:45:22.134340 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-b2m52" event={"ID":"ce8cb8c1-3506-45e8-8f70-83131970c3d5","Type":"ContainerStarted","Data":"f20b1b1285017d110d9b9d6d33bdb9a6a234418cd0278e90c5e7202761d74d58"} Nov 03 00:45:22.135218 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:45:22.135165 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-b2m52" event={"ID":"ce8cb8c1-3506-45e8-8f70-83131970c3d5","Type":"ContainerStarted","Data":"9c19a5c4608abfcf98dd819d016540b7a0a63daca34ebea0c587810cfbdbeb09"} Nov 03 00:45:22.151984 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:45:22.151931 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-b2m52" podStartSLOduration=2.151918255 podStartE2EDuration="2.151918255s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:45:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:45:22.150641386 +0000 UTC m=+264.321644484" watchObservedRunningTime="2026-11-03 00:45:22.151918255 +0000 UTC m=+264.322921351" Nov 03 00:45:23.007988 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:45:23.007953 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-j5b6p"] Nov 03 00:45:23.023393 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:45:23.023363 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-j5b6p" Nov 03 00:45:23.024118 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:45:23.024076 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-j5b6p"] Nov 03 00:45:23.031317 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:45:23.031251 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-ckf59\"" Nov 03 00:45:23.092818 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:45:23.092782 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4sr9\" (UniqueName: \"kubernetes.io/projected/414f5a1d-d1cf-47e1-a35d-bcda8055114c-kube-api-access-b4sr9\") pod \"cert-manager-597b95cb84-j5b6p\" (UID: \"414f5a1d-d1cf-47e1-a35d-bcda8055114c\") " pod="cert-manager/cert-manager-597b95cb84-j5b6p" Nov 03 00:45:23.093014 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:45:23.092953 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/414f5a1d-d1cf-47e1-a35d-bcda8055114c-bound-sa-token\") pod \"cert-manager-597b95cb84-j5b6p\" (UID: \"414f5a1d-d1cf-47e1-a35d-bcda8055114c\") " pod="cert-manager/cert-manager-597b95cb84-j5b6p" Nov 03 00:45:23.194150 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:45:23.194114 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/414f5a1d-d1cf-47e1-a35d-bcda8055114c-bound-sa-token\") pod \"cert-manager-597b95cb84-j5b6p\" (UID: \"414f5a1d-d1cf-47e1-a35d-bcda8055114c\") " pod="cert-manager/cert-manager-597b95cb84-j5b6p" Nov 03 00:45:23.194609 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:45:23.194335 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4sr9\" (UniqueName: \"kubernetes.io/projected/414f5a1d-d1cf-47e1-a35d-bcda8055114c-kube-api-access-b4sr9\") pod \"cert-manager-597b95cb84-j5b6p\" (UID: \"414f5a1d-d1cf-47e1-a35d-bcda8055114c\") " pod="cert-manager/cert-manager-597b95cb84-j5b6p" Nov 03 00:45:23.202922 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:45:23.202893 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/414f5a1d-d1cf-47e1-a35d-bcda8055114c-bound-sa-token\") pod \"cert-manager-597b95cb84-j5b6p\" (UID: \"414f5a1d-d1cf-47e1-a35d-bcda8055114c\") " pod="cert-manager/cert-manager-597b95cb84-j5b6p" Nov 03 00:45:23.203255 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:45:23.203237 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4sr9\" (UniqueName: \"kubernetes.io/projected/414f5a1d-d1cf-47e1-a35d-bcda8055114c-kube-api-access-b4sr9\") pod \"cert-manager-597b95cb84-j5b6p\" (UID: \"414f5a1d-d1cf-47e1-a35d-bcda8055114c\") " pod="cert-manager/cert-manager-597b95cb84-j5b6p" Nov 03 00:45:23.339127 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:45:23.339054 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-j5b6p" Nov 03 00:45:23.626506 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:45:23.626414 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-j5b6p"] Nov 03 00:45:23.630159 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:45:23.630126 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod414f5a1d_d1cf_47e1_a35d_bcda8055114c.slice/crio-5cc026373b80327326a7214cf35611d0a783b1e86532b371179448b4b4f0953e WatchSource:0}: Error finding container 5cc026373b80327326a7214cf35611d0a783b1e86532b371179448b4b4f0953e: Status 404 returned error can't find the container with id 5cc026373b80327326a7214cf35611d0a783b1e86532b371179448b4b4f0953e Nov 03 00:45:24.120180 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:45:24.120149 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-ktz44" Nov 03 00:45:24.147026 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:45:24.146748 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-j5b6p" event={"ID":"414f5a1d-d1cf-47e1-a35d-bcda8055114c","Type":"ContainerStarted","Data":"da431daff58aff0cbca85fbe5728a0d00e22946917b4feeaa84ed2823fa0c025"} Nov 03 00:45:24.147026 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:45:24.146806 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-j5b6p" event={"ID":"414f5a1d-d1cf-47e1-a35d-bcda8055114c","Type":"ContainerStarted","Data":"5cc026373b80327326a7214cf35611d0a783b1e86532b371179448b4b4f0953e"} Nov 03 00:45:24.172511 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:45:24.172452 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-j5b6p" podStartSLOduration=2.172430832 podStartE2EDuration="2.172430832s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:45:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:45:24.172185858 +0000 UTC m=+266.343188956" watchObservedRunningTime="2026-11-03 00:45:24.172430832 +0000 UTC m=+266.343433930" Nov 03 00:45:25.152129 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:45:25.152094 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-4222l" event={"ID":"c2040caa-0454-46bb-bf81-a7063657d4ad","Type":"ContainerStarted","Data":"a036f33c1dacb9bd6c0ef195e5b688e334c1970dec33aa13db57c2bd5af122ad"} Nov 03 00:45:25.169344 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:45:25.169243 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-4222l" podStartSLOduration=0.893563451 podStartE2EDuration="6.169220915s" totalImagesPullingTime="5.275657464s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:45:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:45:25.167674986 +0000 UTC m=+267.338678083" watchObservedRunningTime="2026-11-03 00:45:25.169220915 +0000 UTC m=+267.340224215" Nov 03 00:45:30.733282 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:45:30.733246 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-594b68584c-jw4kg"] Nov 03 00:45:30.737807 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:45:30.737787 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-594b68584c-jw4kg" Nov 03 00:45:30.743736 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:45:30.743702 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 03 00:45:30.744965 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:45:30.744750 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Nov 03 00:45:30.744965 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:45:30.744814 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Nov 03 00:45:30.744965 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:45:30.744879 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 03 00:45:30.744965 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:45:30.744885 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-xfr5n\"" Nov 03 00:45:30.749120 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:45:30.749093 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-594b68584c-jw4kg"] Nov 03 00:45:30.769074 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:45:30.769041 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8g9n\" (UniqueName: \"kubernetes.io/projected/4fa1295e-cc54-460b-aff5-a8e91aeb537d-kube-api-access-t8g9n\") pod \"kube-auth-proxy-594b68584c-jw4kg\" (UID: \"4fa1295e-cc54-460b-aff5-a8e91aeb537d\") " pod="openshift-ingress/kube-auth-proxy-594b68584c-jw4kg" Nov 03 00:45:30.769260 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:45:30.769087 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4fa1295e-cc54-460b-aff5-a8e91aeb537d-tmp\") pod \"kube-auth-proxy-594b68584c-jw4kg\" (UID: \"4fa1295e-cc54-460b-aff5-a8e91aeb537d\") " pod="openshift-ingress/kube-auth-proxy-594b68584c-jw4kg" Nov 03 00:45:30.769260 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:45:30.769109 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4fa1295e-cc54-460b-aff5-a8e91aeb537d-tls-certs\") pod \"kube-auth-proxy-594b68584c-jw4kg\" (UID: \"4fa1295e-cc54-460b-aff5-a8e91aeb537d\") " pod="openshift-ingress/kube-auth-proxy-594b68584c-jw4kg" Nov 03 00:45:30.870456 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:45:30.870406 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4fa1295e-cc54-460b-aff5-a8e91aeb537d-tls-certs\") pod \"kube-auth-proxy-594b68584c-jw4kg\" (UID: \"4fa1295e-cc54-460b-aff5-a8e91aeb537d\") " pod="openshift-ingress/kube-auth-proxy-594b68584c-jw4kg" Nov 03 00:45:30.870674 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:45:30.870497 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8g9n\" (UniqueName: \"kubernetes.io/projected/4fa1295e-cc54-460b-aff5-a8e91aeb537d-kube-api-access-t8g9n\") pod \"kube-auth-proxy-594b68584c-jw4kg\" (UID: \"4fa1295e-cc54-460b-aff5-a8e91aeb537d\") " pod="openshift-ingress/kube-auth-proxy-594b68584c-jw4kg" Nov 03 00:45:30.870674 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:45:30.870528 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4fa1295e-cc54-460b-aff5-a8e91aeb537d-tmp\") pod \"kube-auth-proxy-594b68584c-jw4kg\" (UID: \"4fa1295e-cc54-460b-aff5-a8e91aeb537d\") " pod="openshift-ingress/kube-auth-proxy-594b68584c-jw4kg" Nov 03 00:45:30.872947 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:45:30.872914 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4fa1295e-cc54-460b-aff5-a8e91aeb537d-tmp\") pod \"kube-auth-proxy-594b68584c-jw4kg\" (UID: \"4fa1295e-cc54-460b-aff5-a8e91aeb537d\") " pod="openshift-ingress/kube-auth-proxy-594b68584c-jw4kg" Nov 03 00:45:30.873105 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:45:30.873088 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4fa1295e-cc54-460b-aff5-a8e91aeb537d-tls-certs\") pod \"kube-auth-proxy-594b68584c-jw4kg\" (UID: \"4fa1295e-cc54-460b-aff5-a8e91aeb537d\") " pod="openshift-ingress/kube-auth-proxy-594b68584c-jw4kg" Nov 03 00:45:30.889349 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:45:30.889316 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8g9n\" (UniqueName: \"kubernetes.io/projected/4fa1295e-cc54-460b-aff5-a8e91aeb537d-kube-api-access-t8g9n\") pod \"kube-auth-proxy-594b68584c-jw4kg\" (UID: \"4fa1295e-cc54-460b-aff5-a8e91aeb537d\") " pod="openshift-ingress/kube-auth-proxy-594b68584c-jw4kg" Nov 03 00:45:31.051745 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:45:31.051710 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-594b68584c-jw4kg" Nov 03 00:45:31.340929 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:45:31.340848 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-594b68584c-jw4kg"] Nov 03 00:45:31.343911 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:45:31.343879 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fa1295e_cc54_460b_aff5_a8e91aeb537d.slice/crio-2a5a3f3cdd35fa39e5a6c74cda51c941a00dac8d63720d38e332ad264a59c9c8 WatchSource:0}: Error finding container 2a5a3f3cdd35fa39e5a6c74cda51c941a00dac8d63720d38e332ad264a59c9c8: Status 404 returned error can't find the container with id 2a5a3f3cdd35fa39e5a6c74cda51c941a00dac8d63720d38e332ad264a59c9c8 Nov 03 00:45:32.182816 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:45:32.182781 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-594b68584c-jw4kg" event={"ID":"4fa1295e-cc54-460b-aff5-a8e91aeb537d","Type":"ContainerStarted","Data":"2a5a3f3cdd35fa39e5a6c74cda51c941a00dac8d63720d38e332ad264a59c9c8"} Nov 03 00:45:32.412022 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:45:32.411983 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-l2ztx"] Nov 03 00:45:32.415949 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:45:32.415922 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-l2ztx" Nov 03 00:45:32.422227 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:45:32.420585 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-mgcs2\"" Nov 03 00:45:32.425499 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:45:32.425441 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-l2ztx"] Nov 03 00:45:32.485092 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:45:32.485005 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vx8hg\" (UniqueName: \"kubernetes.io/projected/59296f87-40cf-4ec0-8205-85f76ffe08bc-kube-api-access-vx8hg\") pod \"authorino-57f7f8659f-l2ztx\" (UID: \"59296f87-40cf-4ec0-8205-85f76ffe08bc\") " pod="kuadrant-system/authorino-57f7f8659f-l2ztx" Nov 03 00:45:32.586063 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:45:32.586006 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vx8hg\" (UniqueName: \"kubernetes.io/projected/59296f87-40cf-4ec0-8205-85f76ffe08bc-kube-api-access-vx8hg\") pod \"authorino-57f7f8659f-l2ztx\" (UID: \"59296f87-40cf-4ec0-8205-85f76ffe08bc\") " pod="kuadrant-system/authorino-57f7f8659f-l2ztx" Nov 03 00:45:32.593766 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:45:32.593733 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-v7m2q"] Nov 03 00:45:32.597297 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:45:32.597243 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vx8hg\" (UniqueName: \"kubernetes.io/projected/59296f87-40cf-4ec0-8205-85f76ffe08bc-kube-api-access-vx8hg\") pod \"authorino-57f7f8659f-l2ztx\" (UID: \"59296f87-40cf-4ec0-8205-85f76ffe08bc\") " pod="kuadrant-system/authorino-57f7f8659f-l2ztx" Nov 03 00:45:32.597429 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:45:32.597257 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-v7m2q" Nov 03 00:45:32.612947 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:45:32.612913 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-v7m2q"] Nov 03 00:45:32.687442 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:45:32.687406 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9q4xh\" (UniqueName: \"kubernetes.io/projected/fb9e4eea-1763-4abe-9d2c-a16eb9c1047e-kube-api-access-9q4xh\") pod \"authorino-56c6c9ddd5-v7m2q\" (UID: \"fb9e4eea-1763-4abe-9d2c-a16eb9c1047e\") " pod="kuadrant-system/authorino-56c6c9ddd5-v7m2q" Nov 03 00:45:32.734748 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:45:32.734712 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-l2ztx" Nov 03 00:45:32.788311 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:45:32.788252 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9q4xh\" (UniqueName: \"kubernetes.io/projected/fb9e4eea-1763-4abe-9d2c-a16eb9c1047e-kube-api-access-9q4xh\") pod \"authorino-56c6c9ddd5-v7m2q\" (UID: \"fb9e4eea-1763-4abe-9d2c-a16eb9c1047e\") " pod="kuadrant-system/authorino-56c6c9ddd5-v7m2q" Nov 03 00:45:32.801064 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:45:32.801033 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9q4xh\" (UniqueName: \"kubernetes.io/projected/fb9e4eea-1763-4abe-9d2c-a16eb9c1047e-kube-api-access-9q4xh\") pod \"authorino-56c6c9ddd5-v7m2q\" (UID: \"fb9e4eea-1763-4abe-9d2c-a16eb9c1047e\") " pod="kuadrant-system/authorino-56c6c9ddd5-v7m2q" Nov 03 00:45:32.928428 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:45:32.928399 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-v7m2q" Nov 03 00:45:33.018825 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:45:33.018691 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-l2ztx"] Nov 03 00:45:33.021881 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:45:33.021848 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod59296f87_40cf_4ec0_8205_85f76ffe08bc.slice/crio-ed791bbc739c028ed8ba8cd34d7dbb60dd160405ae476e9879f704afea8fbc18 WatchSource:0}: Error finding container ed791bbc739c028ed8ba8cd34d7dbb60dd160405ae476e9879f704afea8fbc18: Status 404 returned error can't find the container with id ed791bbc739c028ed8ba8cd34d7dbb60dd160405ae476e9879f704afea8fbc18 Nov 03 00:45:33.190469 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:45:33.190429 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-l2ztx" event={"ID":"59296f87-40cf-4ec0-8205-85f76ffe08bc","Type":"ContainerStarted","Data":"ed791bbc739c028ed8ba8cd34d7dbb60dd160405ae476e9879f704afea8fbc18"} Nov 03 00:45:33.207985 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:45:33.207949 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfb9e4eea_1763_4abe_9d2c_a16eb9c1047e.slice/crio-c4ff0aa2a5f7156d205c1c0bf9787c196365fe0709743f7cd61f9f7802fa53f3 WatchSource:0}: Error finding container c4ff0aa2a5f7156d205c1c0bf9787c196365fe0709743f7cd61f9f7802fa53f3: Status 404 returned error can't find the container with id c4ff0aa2a5f7156d205c1c0bf9787c196365fe0709743f7cd61f9f7802fa53f3 Nov 03 00:45:33.211410 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:45:33.211385 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-v7m2q"] Nov 03 00:45:34.199013 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:45:34.198485 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-594b68584c-jw4kg" event={"ID":"4fa1295e-cc54-460b-aff5-a8e91aeb537d","Type":"ContainerStarted","Data":"844b58aae844f3d372aa2b16427f13d7f9296debebd03440ff74f4c8a2e0a5a7"} Nov 03 00:45:34.200513 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:45:34.200480 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-v7m2q" event={"ID":"fb9e4eea-1763-4abe-9d2c-a16eb9c1047e","Type":"ContainerStarted","Data":"c4ff0aa2a5f7156d205c1c0bf9787c196365fe0709743f7cd61f9f7802fa53f3"} Nov 03 00:45:34.220857 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:45:34.220787 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-594b68584c-jw4kg" podStartSLOduration=1.5440672709999999 podStartE2EDuration="4.220767158s" totalImagesPullingTime="2.676699887s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:45:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:45:34.218672448 +0000 UTC m=+276.389675556" watchObservedRunningTime="2026-11-03 00:45:34.220767158 +0000 UTC m=+276.391770255" Nov 03 00:45:37.217149 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:45:37.217110 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-l2ztx" event={"ID":"59296f87-40cf-4ec0-8205-85f76ffe08bc","Type":"ContainerStarted","Data":"bb6bb60f2b0015806f8e060df9b018c5b5535167b972ba10e77e0f1f596f5ab0"} Nov 03 00:45:37.218545 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:45:37.218516 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-v7m2q" event={"ID":"fb9e4eea-1763-4abe-9d2c-a16eb9c1047e","Type":"ContainerStarted","Data":"47b366969e33bdd9fce8167739430ea633c88be75ddc388f9e17107d28a6839d"} Nov 03 00:45:37.232902 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:45:37.232849 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-l2ztx" podStartSLOduration=1.573928525 podStartE2EDuration="5.232834564s" totalImagesPullingTime="3.658906039s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:45:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:45:37.232350736 +0000 UTC m=+279.403353834" watchObservedRunningTime="2026-11-03 00:45:37.232834564 +0000 UTC m=+279.403837659" Nov 03 00:45:37.248485 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:45:37.248430 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-v7m2q" podStartSLOduration=2.045923454 podStartE2EDuration="5.248414197s" totalImagesPullingTime="3.202490743s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:45:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:45:37.247502822 +0000 UTC m=+279.418505917" watchObservedRunningTime="2026-11-03 00:45:37.248414197 +0000 UTC m=+279.419417293" Nov 03 00:45:37.270316 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:45:37.270263 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-l2ztx"] Nov 03 00:45:39.226716 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:45:39.226674 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-l2ztx" podUID="59296f87-40cf-4ec0-8205-85f76ffe08bc" containerName="authorino" containerID="cri-o://bb6bb60f2b0015806f8e060df9b018c5b5535167b972ba10e77e0f1f596f5ab0" gracePeriod=30 Nov 03 00:45:39.471793 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:45:39.471760 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-l2ztx" Nov 03 00:45:39.553038 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:45:39.552994 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59296f87-40cf-4ec0-8205-85f76ffe08bc-kube-api-access-vx8hg\" (UniqueName: \"kubernetes.io/projected/59296f87-40cf-4ec0-8205-85f76ffe08bc-kube-api-access-vx8hg\") pod \"59296f87-40cf-4ec0-8205-85f76ffe08bc\" (UID: \"59296f87-40cf-4ec0-8205-85f76ffe08bc\") " Nov 03 00:45:39.555348 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:45:39.555312 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59296f87-40cf-4ec0-8205-85f76ffe08bc-kube-api-access-vx8hg" pod "59296f87-40cf-4ec0-8205-85f76ffe08bc" (UID: "59296f87-40cf-4ec0-8205-85f76ffe08bc"). InnerVolumeSpecName "kube-api-access-vx8hg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:45:39.654036 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:45:39.653975 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vx8hg\" (UniqueName: \"kubernetes.io/projected/59296f87-40cf-4ec0-8205-85f76ffe08bc-kube-api-access-vx8hg\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:45:40.232714 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:45:40.232677 4421 generic.go:356] "Generic (PLEG): container finished" podID="59296f87-40cf-4ec0-8205-85f76ffe08bc" containerID="bb6bb60f2b0015806f8e060df9b018c5b5535167b972ba10e77e0f1f596f5ab0" exitCode=0 Nov 03 00:45:40.233126 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:45:40.232723 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-l2ztx" Nov 03 00:45:40.233126 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:45:40.232755 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-l2ztx" event={"ID":"59296f87-40cf-4ec0-8205-85f76ffe08bc","Type":"ContainerDied","Data":"bb6bb60f2b0015806f8e060df9b018c5b5535167b972ba10e77e0f1f596f5ab0"} Nov 03 00:45:40.233126 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:45:40.232805 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-l2ztx" event={"ID":"59296f87-40cf-4ec0-8205-85f76ffe08bc","Type":"ContainerDied","Data":"ed791bbc739c028ed8ba8cd34d7dbb60dd160405ae476e9879f704afea8fbc18"} Nov 03 00:45:40.233126 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:45:40.232823 4421 scope.go:122] "RemoveContainer" containerID="bb6bb60f2b0015806f8e060df9b018c5b5535167b972ba10e77e0f1f596f5ab0" Nov 03 00:45:40.241733 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:45:40.241713 4421 scope.go:122] "RemoveContainer" containerID="bb6bb60f2b0015806f8e060df9b018c5b5535167b972ba10e77e0f1f596f5ab0" Nov 03 00:45:40.242093 ip-10-0-133-217 kubenswrapper[4421]: E1103 00:45:40.242073 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb6bb60f2b0015806f8e060df9b018c5b5535167b972ba10e77e0f1f596f5ab0\": container with ID starting with bb6bb60f2b0015806f8e060df9b018c5b5535167b972ba10e77e0f1f596f5ab0 not found: ID does not exist" containerID="bb6bb60f2b0015806f8e060df9b018c5b5535167b972ba10e77e0f1f596f5ab0" Nov 03 00:45:40.242164 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:45:40.242105 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb6bb60f2b0015806f8e060df9b018c5b5535167b972ba10e77e0f1f596f5ab0"} err="failed to get container status \"bb6bb60f2b0015806f8e060df9b018c5b5535167b972ba10e77e0f1f596f5ab0\": rpc error: code = NotFound desc = could not find container \"bb6bb60f2b0015806f8e060df9b018c5b5535167b972ba10e77e0f1f596f5ab0\": container with ID starting with bb6bb60f2b0015806f8e060df9b018c5b5535167b972ba10e77e0f1f596f5ab0 not found: ID does not exist" Nov 03 00:45:40.247819 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:45:40.247793 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-l2ztx"] Nov 03 00:45:40.254004 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:45:40.253981 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-l2ztx"] Nov 03 00:45:41.976265 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:45:41.976227 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59296f87-40cf-4ec0-8205-85f76ffe08bc" path="/var/lib/kubelet/pods/59296f87-40cf-4ec0-8205-85f76ffe08bc/volumes" Nov 03 00:45:44.668968 ip-10-0-133-217 kubenswrapper[4421]: E1103 00:45:44.668929 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 00:45:46.368561 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:45:46.368502 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-6f98fffc9c-l2ptp"] Nov 03 00:45:46.368987 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:45:46.368904 4421 cpu_manager.go:414] "containerMap: removing container" podUID="59296f87-40cf-4ec0-8205-85f76ffe08bc" containerName="authorino" Nov 03 00:45:46.368987 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:45:46.368918 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59296f87-40cf-4ec0-8205-85f76ffe08bc" containerName="authorino" Nov 03 00:45:46.368987 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:45:46.368971 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="59296f87-40cf-4ec0-8205-85f76ffe08bc" containerName="authorino" Nov 03 00:45:46.373144 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:45:46.373129 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-6f98fffc9c-l2ptp" Nov 03 00:45:46.383895 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:45:46.383868 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\"" Nov 03 00:45:46.385054 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:45:46.385033 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\"" Nov 03 00:45:46.385288 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:45:46.385250 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\"" Nov 03 00:45:46.385378 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:45:46.385263 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\"" Nov 03 00:45:46.386482 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:45:46.386460 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\"" Nov 03 00:45:46.386482 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:45:46.386482 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-x4v4l\"" Nov 03 00:45:46.386835 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:45:46.386817 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-6f98fffc9c-l2ptp"] Nov 03 00:45:46.517142 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:45:46.517105 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/4d579ceb-59fa-499a-b241-7f4e940976c3-manager-config\") pod \"jobset-controller-manager-6f98fffc9c-l2ptp\" (UID: \"4d579ceb-59fa-499a-b241-7f4e940976c3\") " pod="openshift-jobset-operator/jobset-controller-manager-6f98fffc9c-l2ptp" Nov 03 00:45:46.517331 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:45:46.517160 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d579ceb-59fa-499a-b241-7f4e940976c3-metrics-certs\") pod \"jobset-controller-manager-6f98fffc9c-l2ptp\" (UID: \"4d579ceb-59fa-499a-b241-7f4e940976c3\") " pod="openshift-jobset-operator/jobset-controller-manager-6f98fffc9c-l2ptp" Nov 03 00:45:46.517331 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:45:46.517259 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nhbj\" (UniqueName: \"kubernetes.io/projected/4d579ceb-59fa-499a-b241-7f4e940976c3-kube-api-access-5nhbj\") pod \"jobset-controller-manager-6f98fffc9c-l2ptp\" (UID: \"4d579ceb-59fa-499a-b241-7f4e940976c3\") " pod="openshift-jobset-operator/jobset-controller-manager-6f98fffc9c-l2ptp" Nov 03 00:45:46.517331 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:45:46.517325 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4d579ceb-59fa-499a-b241-7f4e940976c3-cert\") pod \"jobset-controller-manager-6f98fffc9c-l2ptp\" (UID: \"4d579ceb-59fa-499a-b241-7f4e940976c3\") " pod="openshift-jobset-operator/jobset-controller-manager-6f98fffc9c-l2ptp" Nov 03 00:45:46.618668 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:45:46.618583 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d579ceb-59fa-499a-b241-7f4e940976c3-metrics-certs\") pod \"jobset-controller-manager-6f98fffc9c-l2ptp\" (UID: \"4d579ceb-59fa-499a-b241-7f4e940976c3\") " pod="openshift-jobset-operator/jobset-controller-manager-6f98fffc9c-l2ptp" Nov 03 00:45:46.618668 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:45:46.618637 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nhbj\" (UniqueName: \"kubernetes.io/projected/4d579ceb-59fa-499a-b241-7f4e940976c3-kube-api-access-5nhbj\") pod \"jobset-controller-manager-6f98fffc9c-l2ptp\" (UID: \"4d579ceb-59fa-499a-b241-7f4e940976c3\") " pod="openshift-jobset-operator/jobset-controller-manager-6f98fffc9c-l2ptp" Nov 03 00:45:46.618863 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:45:46.618700 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4d579ceb-59fa-499a-b241-7f4e940976c3-cert\") pod \"jobset-controller-manager-6f98fffc9c-l2ptp\" (UID: \"4d579ceb-59fa-499a-b241-7f4e940976c3\") " pod="openshift-jobset-operator/jobset-controller-manager-6f98fffc9c-l2ptp" Nov 03 00:45:46.618863 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:45:46.618766 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/4d579ceb-59fa-499a-b241-7f4e940976c3-manager-config\") pod \"jobset-controller-manager-6f98fffc9c-l2ptp\" (UID: \"4d579ceb-59fa-499a-b241-7f4e940976c3\") " pod="openshift-jobset-operator/jobset-controller-manager-6f98fffc9c-l2ptp" Nov 03 00:45:46.619590 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:45:46.619562 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/4d579ceb-59fa-499a-b241-7f4e940976c3-manager-config\") pod \"jobset-controller-manager-6f98fffc9c-l2ptp\" (UID: \"4d579ceb-59fa-499a-b241-7f4e940976c3\") " pod="openshift-jobset-operator/jobset-controller-manager-6f98fffc9c-l2ptp" Nov 03 00:45:46.621346 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:45:46.621321 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4d579ceb-59fa-499a-b241-7f4e940976c3-cert\") pod \"jobset-controller-manager-6f98fffc9c-l2ptp\" (UID: \"4d579ceb-59fa-499a-b241-7f4e940976c3\") " pod="openshift-jobset-operator/jobset-controller-manager-6f98fffc9c-l2ptp" Nov 03 00:45:46.621457 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:45:46.621434 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d579ceb-59fa-499a-b241-7f4e940976c3-metrics-certs\") pod \"jobset-controller-manager-6f98fffc9c-l2ptp\" (UID: \"4d579ceb-59fa-499a-b241-7f4e940976c3\") " pod="openshift-jobset-operator/jobset-controller-manager-6f98fffc9c-l2ptp" Nov 03 00:45:46.627059 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:45:46.627035 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nhbj\" (UniqueName: \"kubernetes.io/projected/4d579ceb-59fa-499a-b241-7f4e940976c3-kube-api-access-5nhbj\") pod \"jobset-controller-manager-6f98fffc9c-l2ptp\" (UID: \"4d579ceb-59fa-499a-b241-7f4e940976c3\") " pod="openshift-jobset-operator/jobset-controller-manager-6f98fffc9c-l2ptp" Nov 03 00:45:46.685086 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:45:46.685046 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-6f98fffc9c-l2ptp" Nov 03 00:45:46.970358 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:45:46.970246 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-6f98fffc9c-l2ptp"] Nov 03 00:45:46.972941 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:45:46.972910 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d579ceb_59fa_499a_b241_7f4e940976c3.slice/crio-adcb52c210813acef8ef3a49e5ea915f1c3e9d857d40b783f5384015295a92ef WatchSource:0}: Error finding container adcb52c210813acef8ef3a49e5ea915f1c3e9d857d40b783f5384015295a92ef: Status 404 returned error can't find the container with id adcb52c210813acef8ef3a49e5ea915f1c3e9d857d40b783f5384015295a92ef Nov 03 00:45:47.266403 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:45:47.266167 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-6f98fffc9c-l2ptp" event={"ID":"4d579ceb-59fa-499a-b241-7f4e940976c3","Type":"ContainerStarted","Data":"adcb52c210813acef8ef3a49e5ea915f1c3e9d857d40b783f5384015295a92ef"} Nov 03 00:45:52.288356 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:45:52.288318 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-6f98fffc9c-l2ptp" event={"ID":"4d579ceb-59fa-499a-b241-7f4e940976c3","Type":"ContainerStarted","Data":"2514591708bc044c70e2e2e96c4736520edf271ef0bfec5ab01134ecd9417785"} Nov 03 00:45:52.288800 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:45:52.288543 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-6f98fffc9c-l2ptp" Nov 03 00:45:52.305737 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:45:52.305687 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-controller-manager-6f98fffc9c-l2ptp" podStartSLOduration=1.709300177 podStartE2EDuration="6.305671814s" totalImagesPullingTime="4.596371637s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:45:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:45:52.304485873 +0000 UTC m=+294.475488970" watchObservedRunningTime="2026-11-03 00:45:52.305671814 +0000 UTC m=+294.476674910" Nov 03 00:46:03.298312 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:46:03.298268 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-6f98fffc9c-l2ptp" Nov 03 00:46:14.669670 ip-10-0-133-217 kubenswrapper[4421]: E1103 00:46:14.669585 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 00:46:44.668405 ip-10-0-133-217 kubenswrapper[4421]: E1103 00:46:44.668350 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 00:47:14.667418 ip-10-0-133-217 kubenswrapper[4421]: E1103 00:47:14.667376 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 00:47:27.218147 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:47:27.218107 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-sdt5p"] Nov 03 00:47:27.221684 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:47:27.221663 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-sdt5p" Nov 03 00:47:27.244323 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:47:27.244264 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-sdt5p"] Nov 03 00:47:27.275611 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:47:27.275574 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfqh7\" (UniqueName: \"kubernetes.io/projected/aa2ac34e-3b52-425f-8651-ff510d5738ae-kube-api-access-dfqh7\") pod \"authorino-6f6944bcdd-sdt5p\" (UID: \"aa2ac34e-3b52-425f-8651-ff510d5738ae\") " pod="kuadrant-system/authorino-6f6944bcdd-sdt5p" Nov 03 00:47:27.279578 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:47:27.279549 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-sdt5p"] Nov 03 00:47:27.279776 ip-10-0-133-217 kubenswrapper[4421]: E1103 00:47:27.279754 4421 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-dfqh7], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6f6944bcdd-sdt5p" podUID="aa2ac34e-3b52-425f-8651-ff510d5738ae" Nov 03 00:47:27.309754 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:47:27.309723 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-bsh8j"] Nov 03 00:47:27.313079 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:47:27.313058 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-bsh8j" Nov 03 00:47:27.321595 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:47:27.321567 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Nov 03 00:47:27.324072 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:47:27.324044 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-bsh8j"] Nov 03 00:47:27.376905 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:47:27.376869 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8108f33a-6c70-408c-99c6-ee8a8bb0e8af-tls-cert\") pod \"authorino-6889cbd8c4-bsh8j\" (UID: \"8108f33a-6c70-408c-99c6-ee8a8bb0e8af\") " pod="kuadrant-system/authorino-6889cbd8c4-bsh8j" Nov 03 00:47:27.376905 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:47:27.376913 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfqh7\" (UniqueName: \"kubernetes.io/projected/aa2ac34e-3b52-425f-8651-ff510d5738ae-kube-api-access-dfqh7\") pod \"authorino-6f6944bcdd-sdt5p\" (UID: \"aa2ac34e-3b52-425f-8651-ff510d5738ae\") " pod="kuadrant-system/authorino-6f6944bcdd-sdt5p" Nov 03 00:47:27.377130 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:47:27.376952 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqm9b\" (UniqueName: \"kubernetes.io/projected/8108f33a-6c70-408c-99c6-ee8a8bb0e8af-kube-api-access-zqm9b\") pod \"authorino-6889cbd8c4-bsh8j\" (UID: \"8108f33a-6c70-408c-99c6-ee8a8bb0e8af\") " pod="kuadrant-system/authorino-6889cbd8c4-bsh8j" Nov 03 00:47:27.385507 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:47:27.385476 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfqh7\" (UniqueName: \"kubernetes.io/projected/aa2ac34e-3b52-425f-8651-ff510d5738ae-kube-api-access-dfqh7\") pod \"authorino-6f6944bcdd-sdt5p\" (UID: \"aa2ac34e-3b52-425f-8651-ff510d5738ae\") " pod="kuadrant-system/authorino-6f6944bcdd-sdt5p" Nov 03 00:47:27.477823 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:47:27.477733 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqm9b\" (UniqueName: \"kubernetes.io/projected/8108f33a-6c70-408c-99c6-ee8a8bb0e8af-kube-api-access-zqm9b\") pod \"authorino-6889cbd8c4-bsh8j\" (UID: \"8108f33a-6c70-408c-99c6-ee8a8bb0e8af\") " pod="kuadrant-system/authorino-6889cbd8c4-bsh8j" Nov 03 00:47:27.477972 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:47:27.477830 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8108f33a-6c70-408c-99c6-ee8a8bb0e8af-tls-cert\") pod \"authorino-6889cbd8c4-bsh8j\" (UID: \"8108f33a-6c70-408c-99c6-ee8a8bb0e8af\") " pod="kuadrant-system/authorino-6889cbd8c4-bsh8j" Nov 03 00:47:27.480299 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:47:27.480256 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8108f33a-6c70-408c-99c6-ee8a8bb0e8af-tls-cert\") pod \"authorino-6889cbd8c4-bsh8j\" (UID: \"8108f33a-6c70-408c-99c6-ee8a8bb0e8af\") " pod="kuadrant-system/authorino-6889cbd8c4-bsh8j" Nov 03 00:47:27.486783 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:47:27.486753 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqm9b\" (UniqueName: \"kubernetes.io/projected/8108f33a-6c70-408c-99c6-ee8a8bb0e8af-kube-api-access-zqm9b\") pod \"authorino-6889cbd8c4-bsh8j\" (UID: \"8108f33a-6c70-408c-99c6-ee8a8bb0e8af\") " pod="kuadrant-system/authorino-6889cbd8c4-bsh8j" Nov 03 00:47:27.533355 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:47:27.533314 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-bsh8j"] Nov 03 00:47:27.533533 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:47:27.533522 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-bsh8j" Nov 03 00:47:27.564659 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:47:27.564620 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5fcd47cd5b-rwf6t"] Nov 03 00:47:27.568211 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:47:27.568187 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5fcd47cd5b-rwf6t" Nov 03 00:47:27.578704 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:47:27.577785 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5fcd47cd5b-rwf6t"] Nov 03 00:47:27.679721 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:47:27.679683 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glw2l\" (UniqueName: \"kubernetes.io/projected/f1378599-2104-4475-a307-097042b606a6-kube-api-access-glw2l\") pod \"authorino-5fcd47cd5b-rwf6t\" (UID: \"f1378599-2104-4475-a307-097042b606a6\") " pod="kuadrant-system/authorino-5fcd47cd5b-rwf6t" Nov 03 00:47:27.679907 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:47:27.679750 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f1378599-2104-4475-a307-097042b606a6-tls-cert\") pod \"authorino-5fcd47cd5b-rwf6t\" (UID: \"f1378599-2104-4475-a307-097042b606a6\") " pod="kuadrant-system/authorino-5fcd47cd5b-rwf6t" Nov 03 00:47:27.695087 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:47:27.695057 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-sdt5p" Nov 03 00:47:27.701923 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:47:27.701902 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-sdt5p" Nov 03 00:47:27.780535 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:47:27.780501 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa2ac34e-3b52-425f-8651-ff510d5738ae-kube-api-access-dfqh7\" (UniqueName: \"kubernetes.io/projected/aa2ac34e-3b52-425f-8651-ff510d5738ae-kube-api-access-dfqh7\") pod \"aa2ac34e-3b52-425f-8651-ff510d5738ae\" (UID: \"aa2ac34e-3b52-425f-8651-ff510d5738ae\") " Nov 03 00:47:27.780714 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:47:27.780664 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f1378599-2104-4475-a307-097042b606a6-tls-cert\") pod \"authorino-5fcd47cd5b-rwf6t\" (UID: \"f1378599-2104-4475-a307-097042b606a6\") " pod="kuadrant-system/authorino-5fcd47cd5b-rwf6t" Nov 03 00:47:27.780797 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:47:27.780727 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glw2l\" (UniqueName: \"kubernetes.io/projected/f1378599-2104-4475-a307-097042b606a6-kube-api-access-glw2l\") pod \"authorino-5fcd47cd5b-rwf6t\" (UID: \"f1378599-2104-4475-a307-097042b606a6\") " pod="kuadrant-system/authorino-5fcd47cd5b-rwf6t" Nov 03 00:47:27.782697 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:47:27.782665 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa2ac34e-3b52-425f-8651-ff510d5738ae-kube-api-access-dfqh7" pod "aa2ac34e-3b52-425f-8651-ff510d5738ae" (UID: "aa2ac34e-3b52-425f-8651-ff510d5738ae"). InnerVolumeSpecName "kube-api-access-dfqh7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:47:27.783032 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:47:27.783014 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f1378599-2104-4475-a307-097042b606a6-tls-cert\") pod \"authorino-5fcd47cd5b-rwf6t\" (UID: \"f1378599-2104-4475-a307-097042b606a6\") " pod="kuadrant-system/authorino-5fcd47cd5b-rwf6t" Nov 03 00:47:27.788730 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:47:27.788703 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-glw2l\" (UniqueName: \"kubernetes.io/projected/f1378599-2104-4475-a307-097042b606a6-kube-api-access-glw2l\") pod \"authorino-5fcd47cd5b-rwf6t\" (UID: \"f1378599-2104-4475-a307-097042b606a6\") " pod="kuadrant-system/authorino-5fcd47cd5b-rwf6t" Nov 03 00:47:27.807528 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:47:27.807507 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-bsh8j"] Nov 03 00:47:27.809644 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:47:27.809620 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8108f33a_6c70_408c_99c6_ee8a8bb0e8af.slice/crio-f1f51dbdcde1181d2b20e861c2bcffb2e57ed79ec0259b314f676fb72297facd WatchSource:0}: Error finding container f1f51dbdcde1181d2b20e861c2bcffb2e57ed79ec0259b314f676fb72297facd: Status 404 returned error can't find the container with id f1f51dbdcde1181d2b20e861c2bcffb2e57ed79ec0259b314f676fb72297facd Nov 03 00:47:27.810897 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:47:27.810878 4421 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 00:47:27.882083 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:47:27.882042 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dfqh7\" (UniqueName: \"kubernetes.io/projected/aa2ac34e-3b52-425f-8651-ff510d5738ae-kube-api-access-dfqh7\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:47:27.883961 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:47:27.883938 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5fcd47cd5b-rwf6t" Nov 03 00:47:28.159382 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:47:28.159306 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5fcd47cd5b-rwf6t"] Nov 03 00:47:28.162613 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:47:28.162586 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf1378599_2104_4475_a307_097042b606a6.slice/crio-ad37e8b44a27421264585980e80e5e7eb69ccb5c9832c4b0e8cc377709c1bde2 WatchSource:0}: Error finding container ad37e8b44a27421264585980e80e5e7eb69ccb5c9832c4b0e8cc377709c1bde2: Status 404 returned error can't find the container with id ad37e8b44a27421264585980e80e5e7eb69ccb5c9832c4b0e8cc377709c1bde2 Nov 03 00:47:28.701437 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:47:28.701403 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6889cbd8c4-bsh8j" event={"ID":"8108f33a-6c70-408c-99c6-ee8a8bb0e8af","Type":"ContainerStarted","Data":"d29832335d59e8675d1d5b9e0c534743c21dfde7fb62974a2afc9e32a33153e0"} Nov 03 00:47:28.701864 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:47:28.701450 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6889cbd8c4-bsh8j" event={"ID":"8108f33a-6c70-408c-99c6-ee8a8bb0e8af","Type":"ContainerStarted","Data":"f1f51dbdcde1181d2b20e861c2bcffb2e57ed79ec0259b314f676fb72297facd"} Nov 03 00:47:28.701864 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:47:28.701492 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-6889cbd8c4-bsh8j" podUID="8108f33a-6c70-408c-99c6-ee8a8bb0e8af" containerName="authorino" containerID="cri-o://d29832335d59e8675d1d5b9e0c534743c21dfde7fb62974a2afc9e32a33153e0" gracePeriod=30 Nov 03 00:47:28.703096 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:47:28.703068 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-sdt5p" Nov 03 00:47:28.703204 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:47:28.703111 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5fcd47cd5b-rwf6t" event={"ID":"f1378599-2104-4475-a307-097042b606a6","Type":"ContainerStarted","Data":"ad37e8b44a27421264585980e80e5e7eb69ccb5c9832c4b0e8cc377709c1bde2"} Nov 03 00:47:28.719940 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:47:28.719900 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6889cbd8c4-bsh8j" podStartSLOduration=0.901571665 podStartE2EDuration="1.719886569s" totalImagesPullingTime="818.314904ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:47:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:47:28.718630874 +0000 UTC m=+390.889633971" watchObservedRunningTime="2026-11-03 00:47:28.719886569 +0000 UTC m=+390.890889665" Nov 03 00:47:28.778990 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:47:28.778961 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-sdt5p"] Nov 03 00:47:28.785134 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:47:28.785103 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-sdt5p"] Nov 03 00:47:28.944550 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:47:28.944523 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-bsh8j" Nov 03 00:47:28.992054 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:47:28.992016 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8108f33a-6c70-408c-99c6-ee8a8bb0e8af-kube-api-access-zqm9b\" (UniqueName: \"kubernetes.io/projected/8108f33a-6c70-408c-99c6-ee8a8bb0e8af-kube-api-access-zqm9b\") pod \"8108f33a-6c70-408c-99c6-ee8a8bb0e8af\" (UID: \"8108f33a-6c70-408c-99c6-ee8a8bb0e8af\") " Nov 03 00:47:28.992231 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:47:28.992108 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8108f33a-6c70-408c-99c6-ee8a8bb0e8af-tls-cert\" (UniqueName: \"kubernetes.io/secret/8108f33a-6c70-408c-99c6-ee8a8bb0e8af-tls-cert\") pod \"8108f33a-6c70-408c-99c6-ee8a8bb0e8af\" (UID: \"8108f33a-6c70-408c-99c6-ee8a8bb0e8af\") " Nov 03 00:47:28.994308 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:47:28.994264 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8108f33a-6c70-408c-99c6-ee8a8bb0e8af-kube-api-access-zqm9b" pod "8108f33a-6c70-408c-99c6-ee8a8bb0e8af" (UID: "8108f33a-6c70-408c-99c6-ee8a8bb0e8af"). InnerVolumeSpecName "kube-api-access-zqm9b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:47:29.003128 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:47:29.003083 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8108f33a-6c70-408c-99c6-ee8a8bb0e8af-tls-cert" pod "8108f33a-6c70-408c-99c6-ee8a8bb0e8af" (UID: "8108f33a-6c70-408c-99c6-ee8a8bb0e8af"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:47:29.093693 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:47:29.093646 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqm9b\" (UniqueName: \"kubernetes.io/projected/8108f33a-6c70-408c-99c6-ee8a8bb0e8af-kube-api-access-zqm9b\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:47:29.093693 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:47:29.093692 4421 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8108f33a-6c70-408c-99c6-ee8a8bb0e8af-tls-cert\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:47:29.708203 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:47:29.708121 4421 generic.go:356] "Generic (PLEG): container finished" podID="8108f33a-6c70-408c-99c6-ee8a8bb0e8af" containerID="d29832335d59e8675d1d5b9e0c534743c21dfde7fb62974a2afc9e32a33153e0" exitCode=2 Nov 03 00:47:29.708203 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:47:29.708177 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-bsh8j" Nov 03 00:47:29.708741 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:47:29.708202 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6889cbd8c4-bsh8j" event={"ID":"8108f33a-6c70-408c-99c6-ee8a8bb0e8af","Type":"ContainerDied","Data":"d29832335d59e8675d1d5b9e0c534743c21dfde7fb62974a2afc9e32a33153e0"} Nov 03 00:47:29.708741 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:47:29.708240 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6889cbd8c4-bsh8j" event={"ID":"8108f33a-6c70-408c-99c6-ee8a8bb0e8af","Type":"ContainerDied","Data":"f1f51dbdcde1181d2b20e861c2bcffb2e57ed79ec0259b314f676fb72297facd"} Nov 03 00:47:29.708741 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:47:29.708259 4421 scope.go:122] "RemoveContainer" containerID="d29832335d59e8675d1d5b9e0c534743c21dfde7fb62974a2afc9e32a33153e0" Nov 03 00:47:29.709953 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:47:29.709890 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5fcd47cd5b-rwf6t" event={"ID":"f1378599-2104-4475-a307-097042b606a6","Type":"ContainerStarted","Data":"ef9fa9ade8ef4d06214377dc413596c7a008abc070b1d0ae451a335e104095f5"} Nov 03 00:47:29.717185 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:47:29.717170 4421 scope.go:122] "RemoveContainer" containerID="d29832335d59e8675d1d5b9e0c534743c21dfde7fb62974a2afc9e32a33153e0" Nov 03 00:47:29.717440 ip-10-0-133-217 kubenswrapper[4421]: E1103 00:47:29.717424 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d29832335d59e8675d1d5b9e0c534743c21dfde7fb62974a2afc9e32a33153e0\": container with ID starting with d29832335d59e8675d1d5b9e0c534743c21dfde7fb62974a2afc9e32a33153e0 not found: ID does not exist" containerID="d29832335d59e8675d1d5b9e0c534743c21dfde7fb62974a2afc9e32a33153e0" Nov 03 00:47:29.717493 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:47:29.717449 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d29832335d59e8675d1d5b9e0c534743c21dfde7fb62974a2afc9e32a33153e0"} err="failed to get container status \"d29832335d59e8675d1d5b9e0c534743c21dfde7fb62974a2afc9e32a33153e0\": rpc error: code = NotFound desc = could not find container \"d29832335d59e8675d1d5b9e0c534743c21dfde7fb62974a2afc9e32a33153e0\": container with ID starting with d29832335d59e8675d1d5b9e0c534743c21dfde7fb62974a2afc9e32a33153e0 not found: ID does not exist" Nov 03 00:47:29.728049 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:47:29.728006 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-5fcd47cd5b-rwf6t" podStartSLOduration=2.13860976 podStartE2EDuration="2.727992624s" totalImagesPullingTime="589.382864ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:47:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:47:29.727493407 +0000 UTC m=+391.898496517" watchObservedRunningTime="2026-11-03 00:47:29.727992624 +0000 UTC m=+391.898995720" Nov 03 00:47:29.745852 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:47:29.745778 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-bsh8j"] Nov 03 00:47:29.750257 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:47:29.750225 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-bsh8j"] Nov 03 00:47:29.752853 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:47:29.752829 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-v7m2q"] Nov 03 00:47:29.753022 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:47:29.753001 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-v7m2q" podUID="fb9e4eea-1763-4abe-9d2c-a16eb9c1047e" containerName="authorino" containerID="cri-o://47b366969e33bdd9fce8167739430ea633c88be75ddc388f9e17107d28a6839d" gracePeriod=30 Nov 03 00:47:29.977246 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:47:29.977160 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8108f33a-6c70-408c-99c6-ee8a8bb0e8af" path="/var/lib/kubelet/pods/8108f33a-6c70-408c-99c6-ee8a8bb0e8af/volumes" Nov 03 00:47:29.977623 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:47:29.977600 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa2ac34e-3b52-425f-8651-ff510d5738ae" path="/var/lib/kubelet/pods/aa2ac34e-3b52-425f-8651-ff510d5738ae/volumes" Nov 03 00:47:30.005344 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:47:30.005319 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-v7m2q" Nov 03 00:47:30.102000 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:47:30.101961 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb9e4eea-1763-4abe-9d2c-a16eb9c1047e-kube-api-access-9q4xh\" (UniqueName: \"kubernetes.io/projected/fb9e4eea-1763-4abe-9d2c-a16eb9c1047e-kube-api-access-9q4xh\") pod \"fb9e4eea-1763-4abe-9d2c-a16eb9c1047e\" (UID: \"fb9e4eea-1763-4abe-9d2c-a16eb9c1047e\") " Nov 03 00:47:30.104293 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:47:30.104244 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb9e4eea-1763-4abe-9d2c-a16eb9c1047e-kube-api-access-9q4xh" pod "fb9e4eea-1763-4abe-9d2c-a16eb9c1047e" (UID: "fb9e4eea-1763-4abe-9d2c-a16eb9c1047e"). InnerVolumeSpecName "kube-api-access-9q4xh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:47:30.203368 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:47:30.203333 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9q4xh\" (UniqueName: \"kubernetes.io/projected/fb9e4eea-1763-4abe-9d2c-a16eb9c1047e-kube-api-access-9q4xh\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:47:30.717201 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:47:30.717163 4421 generic.go:356] "Generic (PLEG): container finished" podID="fb9e4eea-1763-4abe-9d2c-a16eb9c1047e" containerID="47b366969e33bdd9fce8167739430ea633c88be75ddc388f9e17107d28a6839d" exitCode=0 Nov 03 00:47:30.717709 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:47:30.717211 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-v7m2q" Nov 03 00:47:30.717709 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:47:30.717254 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-v7m2q" event={"ID":"fb9e4eea-1763-4abe-9d2c-a16eb9c1047e","Type":"ContainerDied","Data":"47b366969e33bdd9fce8167739430ea633c88be75ddc388f9e17107d28a6839d"} Nov 03 00:47:30.717709 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:47:30.717315 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-v7m2q" event={"ID":"fb9e4eea-1763-4abe-9d2c-a16eb9c1047e","Type":"ContainerDied","Data":"c4ff0aa2a5f7156d205c1c0bf9787c196365fe0709743f7cd61f9f7802fa53f3"} Nov 03 00:47:30.717709 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:47:30.717335 4421 scope.go:122] "RemoveContainer" containerID="47b366969e33bdd9fce8167739430ea633c88be75ddc388f9e17107d28a6839d" Nov 03 00:47:30.725919 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:47:30.725900 4421 scope.go:122] "RemoveContainer" containerID="47b366969e33bdd9fce8167739430ea633c88be75ddc388f9e17107d28a6839d" Nov 03 00:47:30.726195 ip-10-0-133-217 kubenswrapper[4421]: E1103 00:47:30.726177 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47b366969e33bdd9fce8167739430ea633c88be75ddc388f9e17107d28a6839d\": container with ID starting with 47b366969e33bdd9fce8167739430ea633c88be75ddc388f9e17107d28a6839d not found: ID does not exist" containerID="47b366969e33bdd9fce8167739430ea633c88be75ddc388f9e17107d28a6839d" Nov 03 00:47:30.726243 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:47:30.726205 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47b366969e33bdd9fce8167739430ea633c88be75ddc388f9e17107d28a6839d"} err="failed to get container status \"47b366969e33bdd9fce8167739430ea633c88be75ddc388f9e17107d28a6839d\": rpc error: code = NotFound desc = could not find container \"47b366969e33bdd9fce8167739430ea633c88be75ddc388f9e17107d28a6839d\": container with ID starting with 47b366969e33bdd9fce8167739430ea633c88be75ddc388f9e17107d28a6839d not found: ID does not exist" Nov 03 00:47:30.738611 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:47:30.738582 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-v7m2q"] Nov 03 00:47:30.742300 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:47:30.742260 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-v7m2q"] Nov 03 00:47:31.977563 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:47:31.977528 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb9e4eea-1763-4abe-9d2c-a16eb9c1047e" path="/var/lib/kubelet/pods/fb9e4eea-1763-4abe-9d2c-a16eb9c1047e/volumes" Nov 03 00:47:44.667102 ip-10-0-133-217 kubenswrapper[4421]: E1103 00:47:44.667015 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 00:48:14.669431 ip-10-0-133-217 kubenswrapper[4421]: E1103 00:48:14.669393 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 00:48:42.079187 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:48:42.079152 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-tmxmj"] Nov 03 00:48:42.079724 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:48:42.079552 4421 cpu_manager.go:414] "containerMap: removing container" podUID="fb9e4eea-1763-4abe-9d2c-a16eb9c1047e" containerName="authorino" Nov 03 00:48:42.079724 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:48:42.079564 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb9e4eea-1763-4abe-9d2c-a16eb9c1047e" containerName="authorino" Nov 03 00:48:42.079724 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:48:42.079577 4421 cpu_manager.go:414] "containerMap: removing container" podUID="8108f33a-6c70-408c-99c6-ee8a8bb0e8af" containerName="authorino" Nov 03 00:48:42.079724 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:48:42.079582 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8108f33a-6c70-408c-99c6-ee8a8bb0e8af" containerName="authorino" Nov 03 00:48:42.079724 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:48:42.079663 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb9e4eea-1763-4abe-9d2c-a16eb9c1047e" containerName="authorino" Nov 03 00:48:42.079724 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:48:42.079673 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="8108f33a-6c70-408c-99c6-ee8a8bb0e8af" containerName="authorino" Nov 03 00:48:42.082572 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:48:42.082549 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tmxmj" Nov 03 00:48:42.091718 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:48:42.091699 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-jhxdz\"" Nov 03 00:48:42.091897 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:48:42.091805 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Nov 03 00:48:42.093173 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:48:42.093150 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Nov 03 00:48:42.093173 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:48:42.093157 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Nov 03 00:48:42.094897 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:48:42.094873 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-tmxmj"] Nov 03 00:48:42.224106 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:48:42.224068 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b62c15b-bbbf-4926-b4c3-27d12277743b-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-tmxmj\" (UID: \"1b62c15b-bbbf-4926-b4c3-27d12277743b\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tmxmj" Nov 03 00:48:42.224323 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:48:42.224117 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b62c15b-bbbf-4926-b4c3-27d12277743b-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-tmxmj\" (UID: \"1b62c15b-bbbf-4926-b4c3-27d12277743b\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tmxmj" Nov 03 00:48:42.224323 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:48:42.224132 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b62c15b-bbbf-4926-b4c3-27d12277743b-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-tmxmj\" (UID: \"1b62c15b-bbbf-4926-b4c3-27d12277743b\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tmxmj" Nov 03 00:48:42.224323 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:48:42.224211 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b62c15b-bbbf-4926-b4c3-27d12277743b-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-tmxmj\" (UID: \"1b62c15b-bbbf-4926-b4c3-27d12277743b\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tmxmj" Nov 03 00:48:42.224323 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:48:42.224305 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b62c15b-bbbf-4926-b4c3-27d12277743b-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-tmxmj\" (UID: \"1b62c15b-bbbf-4926-b4c3-27d12277743b\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tmxmj" Nov 03 00:48:42.224478 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:48:42.224331 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zjjc\" (UniqueName: \"kubernetes.io/projected/1b62c15b-bbbf-4926-b4c3-27d12277743b-kube-api-access-9zjjc\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-tmxmj\" (UID: \"1b62c15b-bbbf-4926-b4c3-27d12277743b\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tmxmj" Nov 03 00:48:42.325301 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:48:42.325243 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b62c15b-bbbf-4926-b4c3-27d12277743b-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-tmxmj\" (UID: \"1b62c15b-bbbf-4926-b4c3-27d12277743b\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tmxmj" Nov 03 00:48:42.325494 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:48:42.325322 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zjjc\" (UniqueName: \"kubernetes.io/projected/1b62c15b-bbbf-4926-b4c3-27d12277743b-kube-api-access-9zjjc\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-tmxmj\" (UID: \"1b62c15b-bbbf-4926-b4c3-27d12277743b\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tmxmj" Nov 03 00:48:42.325494 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:48:42.325385 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b62c15b-bbbf-4926-b4c3-27d12277743b-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-tmxmj\" (UID: \"1b62c15b-bbbf-4926-b4c3-27d12277743b\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tmxmj" Nov 03 00:48:42.325494 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:48:42.325436 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b62c15b-bbbf-4926-b4c3-27d12277743b-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-tmxmj\" (UID: \"1b62c15b-bbbf-4926-b4c3-27d12277743b\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tmxmj" Nov 03 00:48:42.325494 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:48:42.325461 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b62c15b-bbbf-4926-b4c3-27d12277743b-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-tmxmj\" (UID: \"1b62c15b-bbbf-4926-b4c3-27d12277743b\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tmxmj" Nov 03 00:48:42.325494 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:48:42.325487 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b62c15b-bbbf-4926-b4c3-27d12277743b-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-tmxmj\" (UID: \"1b62c15b-bbbf-4926-b4c3-27d12277743b\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tmxmj" Nov 03 00:48:42.325819 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:48:42.325798 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b62c15b-bbbf-4926-b4c3-27d12277743b-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-tmxmj\" (UID: \"1b62c15b-bbbf-4926-b4c3-27d12277743b\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tmxmj" Nov 03 00:48:42.325887 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:48:42.325868 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b62c15b-bbbf-4926-b4c3-27d12277743b-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-tmxmj\" (UID: \"1b62c15b-bbbf-4926-b4c3-27d12277743b\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tmxmj" Nov 03 00:48:42.325933 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:48:42.325892 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b62c15b-bbbf-4926-b4c3-27d12277743b-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-tmxmj\" (UID: \"1b62c15b-bbbf-4926-b4c3-27d12277743b\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tmxmj" Nov 03 00:48:42.327669 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:48:42.327647 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b62c15b-bbbf-4926-b4c3-27d12277743b-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-tmxmj\" (UID: \"1b62c15b-bbbf-4926-b4c3-27d12277743b\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tmxmj" Nov 03 00:48:42.327807 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:48:42.327789 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b62c15b-bbbf-4926-b4c3-27d12277743b-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-tmxmj\" (UID: \"1b62c15b-bbbf-4926-b4c3-27d12277743b\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tmxmj" Nov 03 00:48:42.334392 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:48:42.334336 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zjjc\" (UniqueName: \"kubernetes.io/projected/1b62c15b-bbbf-4926-b4c3-27d12277743b-kube-api-access-9zjjc\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-tmxmj\" (UID: \"1b62c15b-bbbf-4926-b4c3-27d12277743b\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tmxmj" Nov 03 00:48:42.398620 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:48:42.398576 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tmxmj" Nov 03 00:48:42.670654 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:48:42.670577 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-tmxmj"] Nov 03 00:48:42.673999 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:48:42.673968 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b62c15b_bbbf_4926_b4c3_27d12277743b.slice/crio-de39a07dd0c10a602cda8e67f05059ccb656064894bdd4f58cc217fec5ba5ac4 WatchSource:0}: Error finding container de39a07dd0c10a602cda8e67f05059ccb656064894bdd4f58cc217fec5ba5ac4: Status 404 returned error can't find the container with id de39a07dd0c10a602cda8e67f05059ccb656064894bdd4f58cc217fec5ba5ac4 Nov 03 00:48:43.014386 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:48:43.014349 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tmxmj" event={"ID":"1b62c15b-bbbf-4926-b4c3-27d12277743b","Type":"ContainerStarted","Data":"de39a07dd0c10a602cda8e67f05059ccb656064894bdd4f58cc217fec5ba5ac4"} Nov 03 00:48:44.022790 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:48:44.022752 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tmxmj" event={"ID":"1b62c15b-bbbf-4926-b4c3-27d12277743b","Type":"ContainerStarted","Data":"306fb4f3d529340e0928a632f05bdb4fe5871948b67496dbd71b7d7e776e5e48"} Nov 03 00:48:44.043090 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:48:44.043040 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tmxmj" podStartSLOduration=0.779576135 podStartE2EDuration="2.043025208s" totalImagesPullingTime="1.263449073s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:48:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:48:44.041580436 +0000 UTC m=+466.212583542" watchObservedRunningTime="2026-11-03 00:48:44.043025208 +0000 UTC m=+466.214028304" Nov 03 00:48:44.399368 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:48:44.399330 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tmxmj" Nov 03 00:48:44.671427 ip-10-0-133-217 kubenswrapper[4421]: E1103 00:48:44.671337 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 00:48:48.390684 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:48:48.390651 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-lc7tp"] Nov 03 00:48:48.394326 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:48:48.394307 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-lc7tp" Nov 03 00:48:48.402831 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:48:48.402808 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Nov 03 00:48:48.406010 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:48:48.405988 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-lc7tp"] Nov 03 00:48:48.488984 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:48:48.488929 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42b27a10-3a97-4e19-8b8a-d5588e9936ef-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-lc7tp\" (UID: \"42b27a10-3a97-4e19-8b8a-d5588e9936ef\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-lc7tp" Nov 03 00:48:48.489184 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:48:48.489074 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42b27a10-3a97-4e19-8b8a-d5588e9936ef-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-lc7tp\" (UID: \"42b27a10-3a97-4e19-8b8a-d5588e9936ef\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-lc7tp" Nov 03 00:48:48.489184 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:48:48.489124 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h67j5\" (UniqueName: \"kubernetes.io/projected/42b27a10-3a97-4e19-8b8a-d5588e9936ef-kube-api-access-h67j5\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-lc7tp\" (UID: \"42b27a10-3a97-4e19-8b8a-d5588e9936ef\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-lc7tp" Nov 03 00:48:48.489184 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:48:48.489154 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42b27a10-3a97-4e19-8b8a-d5588e9936ef-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-lc7tp\" (UID: \"42b27a10-3a97-4e19-8b8a-d5588e9936ef\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-lc7tp" Nov 03 00:48:48.489413 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:48:48.489337 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42b27a10-3a97-4e19-8b8a-d5588e9936ef-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-lc7tp\" (UID: \"42b27a10-3a97-4e19-8b8a-d5588e9936ef\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-lc7tp" Nov 03 00:48:48.489413 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:48:48.489378 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42b27a10-3a97-4e19-8b8a-d5588e9936ef-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-lc7tp\" (UID: \"42b27a10-3a97-4e19-8b8a-d5588e9936ef\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-lc7tp" Nov 03 00:48:48.589938 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:48:48.589898 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42b27a10-3a97-4e19-8b8a-d5588e9936ef-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-lc7tp\" (UID: \"42b27a10-3a97-4e19-8b8a-d5588e9936ef\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-lc7tp" Nov 03 00:48:48.590128 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:48:48.589967 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42b27a10-3a97-4e19-8b8a-d5588e9936ef-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-lc7tp\" (UID: \"42b27a10-3a97-4e19-8b8a-d5588e9936ef\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-lc7tp" Nov 03 00:48:48.590128 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:48:48.590011 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h67j5\" (UniqueName: \"kubernetes.io/projected/42b27a10-3a97-4e19-8b8a-d5588e9936ef-kube-api-access-h67j5\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-lc7tp\" (UID: \"42b27a10-3a97-4e19-8b8a-d5588e9936ef\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-lc7tp" Nov 03 00:48:48.590128 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:48:48.590047 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42b27a10-3a97-4e19-8b8a-d5588e9936ef-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-lc7tp\" (UID: \"42b27a10-3a97-4e19-8b8a-d5588e9936ef\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-lc7tp" Nov 03 00:48:48.590128 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:48:48.590088 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42b27a10-3a97-4e19-8b8a-d5588e9936ef-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-lc7tp\" (UID: \"42b27a10-3a97-4e19-8b8a-d5588e9936ef\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-lc7tp" Nov 03 00:48:48.590128 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:48:48.590126 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42b27a10-3a97-4e19-8b8a-d5588e9936ef-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-lc7tp\" (UID: \"42b27a10-3a97-4e19-8b8a-d5588e9936ef\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-lc7tp" Nov 03 00:48:48.590463 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:48:48.590392 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42b27a10-3a97-4e19-8b8a-d5588e9936ef-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-lc7tp\" (UID: \"42b27a10-3a97-4e19-8b8a-d5588e9936ef\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-lc7tp" Nov 03 00:48:48.590519 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:48:48.590462 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42b27a10-3a97-4e19-8b8a-d5588e9936ef-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-lc7tp\" (UID: \"42b27a10-3a97-4e19-8b8a-d5588e9936ef\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-lc7tp" Nov 03 00:48:48.590559 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:48:48.590540 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42b27a10-3a97-4e19-8b8a-d5588e9936ef-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-lc7tp\" (UID: \"42b27a10-3a97-4e19-8b8a-d5588e9936ef\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-lc7tp" Nov 03 00:48:48.592307 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:48:48.592265 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42b27a10-3a97-4e19-8b8a-d5588e9936ef-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-lc7tp\" (UID: \"42b27a10-3a97-4e19-8b8a-d5588e9936ef\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-lc7tp" Nov 03 00:48:48.592652 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:48:48.592635 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42b27a10-3a97-4e19-8b8a-d5588e9936ef-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-lc7tp\" (UID: \"42b27a10-3a97-4e19-8b8a-d5588e9936ef\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-lc7tp" Nov 03 00:48:48.598653 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:48:48.598633 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h67j5\" (UniqueName: \"kubernetes.io/projected/42b27a10-3a97-4e19-8b8a-d5588e9936ef-kube-api-access-h67j5\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-lc7tp\" (UID: \"42b27a10-3a97-4e19-8b8a-d5588e9936ef\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-lc7tp" Nov 03 00:48:48.708597 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:48:48.708479 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-lc7tp" Nov 03 00:48:48.980709 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:48:48.980683 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-lc7tp"] Nov 03 00:48:48.982756 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:48:48.982725 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42b27a10_3a97_4e19_8b8a_d5588e9936ef.slice/crio-95dd0792dd3cd5b8b8b2fc79eb292baa0435df4848317938c9ca292607d75576 WatchSource:0}: Error finding container 95dd0792dd3cd5b8b8b2fc79eb292baa0435df4848317938c9ca292607d75576: Status 404 returned error can't find the container with id 95dd0792dd3cd5b8b8b2fc79eb292baa0435df4848317938c9ca292607d75576 Nov 03 00:48:49.044575 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:48:49.044536 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-lc7tp" event={"ID":"42b27a10-3a97-4e19-8b8a-d5588e9936ef","Type":"ContainerStarted","Data":"95dd0792dd3cd5b8b8b2fc79eb292baa0435df4848317938c9ca292607d75576"} Nov 03 00:48:50.049825 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:48:50.049787 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-lc7tp" event={"ID":"42b27a10-3a97-4e19-8b8a-d5588e9936ef","Type":"ContainerStarted","Data":"c7578fbc62da63bba836264141068d2c1652d9dcff96c1b1c9d7f783c753b52e"} Nov 03 00:48:50.081129 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:48:50.081069 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-lc7tp" podStartSLOduration=1.7836044869999998 podStartE2EDuration="2.08105651s" totalImagesPullingTime="297.452023ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:48:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:48:50.080920104 +0000 UTC m=+472.251923200" watchObservedRunningTime="2026-11-03 00:48:50.08105651 +0000 UTC m=+472.252059659" Nov 03 00:48:50.709170 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:48:50.709129 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-lc7tp" Nov 03 00:48:51.240369 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:48:51.240334 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-5dqn2"] Nov 03 00:48:51.243659 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:48:51.243634 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-5dqn2" Nov 03 00:48:51.250201 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:48:51.250169 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\"" Nov 03 00:48:51.252325 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:48:51.252303 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-5dqn2"] Nov 03 00:48:51.419347 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:48:51.419310 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a54c4af-a899-4f16-83a8-8718b0c7eef5-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-5dqn2\" (UID: \"0a54c4af-a899-4f16-83a8-8718b0c7eef5\") " pod="llm/echo-simulator-kserve-64b6f8f96c-5dqn2" Nov 03 00:48:51.419347 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:48:51.419351 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a54c4af-a899-4f16-83a8-8718b0c7eef5-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-5dqn2\" (UID: \"0a54c4af-a899-4f16-83a8-8718b0c7eef5\") " pod="llm/echo-simulator-kserve-64b6f8f96c-5dqn2" Nov 03 00:48:51.419598 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:48:51.419382 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a54c4af-a899-4f16-83a8-8718b0c7eef5-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-5dqn2\" (UID: \"0a54c4af-a899-4f16-83a8-8718b0c7eef5\") " pod="llm/echo-simulator-kserve-64b6f8f96c-5dqn2" Nov 03 00:48:51.419598 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:48:51.419420 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a54c4af-a899-4f16-83a8-8718b0c7eef5-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-5dqn2\" (UID: \"0a54c4af-a899-4f16-83a8-8718b0c7eef5\") " pod="llm/echo-simulator-kserve-64b6f8f96c-5dqn2" Nov 03 00:48:51.419598 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:48:51.419531 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a54c4af-a899-4f16-83a8-8718b0c7eef5-home\") pod \"echo-simulator-kserve-64b6f8f96c-5dqn2\" (UID: \"0a54c4af-a899-4f16-83a8-8718b0c7eef5\") " pod="llm/echo-simulator-kserve-64b6f8f96c-5dqn2" Nov 03 00:48:51.419755 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:48:51.419676 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29g4b\" (UniqueName: \"kubernetes.io/projected/0a54c4af-a899-4f16-83a8-8718b0c7eef5-kube-api-access-29g4b\") pod \"echo-simulator-kserve-64b6f8f96c-5dqn2\" (UID: \"0a54c4af-a899-4f16-83a8-8718b0c7eef5\") " pod="llm/echo-simulator-kserve-64b6f8f96c-5dqn2" Nov 03 00:48:51.520917 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:48:51.520876 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29g4b\" (UniqueName: \"kubernetes.io/projected/0a54c4af-a899-4f16-83a8-8718b0c7eef5-kube-api-access-29g4b\") pod \"echo-simulator-kserve-64b6f8f96c-5dqn2\" (UID: \"0a54c4af-a899-4f16-83a8-8718b0c7eef5\") " pod="llm/echo-simulator-kserve-64b6f8f96c-5dqn2" Nov 03 00:48:51.521101 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:48:51.520933 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a54c4af-a899-4f16-83a8-8718b0c7eef5-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-5dqn2\" (UID: \"0a54c4af-a899-4f16-83a8-8718b0c7eef5\") " pod="llm/echo-simulator-kserve-64b6f8f96c-5dqn2" Nov 03 00:48:51.521101 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:48:51.521071 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a54c4af-a899-4f16-83a8-8718b0c7eef5-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-5dqn2\" (UID: \"0a54c4af-a899-4f16-83a8-8718b0c7eef5\") " pod="llm/echo-simulator-kserve-64b6f8f96c-5dqn2" Nov 03 00:48:51.521247 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:48:51.521116 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a54c4af-a899-4f16-83a8-8718b0c7eef5-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-5dqn2\" (UID: \"0a54c4af-a899-4f16-83a8-8718b0c7eef5\") " pod="llm/echo-simulator-kserve-64b6f8f96c-5dqn2" Nov 03 00:48:51.521247 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:48:51.521142 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a54c4af-a899-4f16-83a8-8718b0c7eef5-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-5dqn2\" (UID: \"0a54c4af-a899-4f16-83a8-8718b0c7eef5\") " pod="llm/echo-simulator-kserve-64b6f8f96c-5dqn2" Nov 03 00:48:51.521247 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:48:51.521222 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a54c4af-a899-4f16-83a8-8718b0c7eef5-home\") pod \"echo-simulator-kserve-64b6f8f96c-5dqn2\" (UID: \"0a54c4af-a899-4f16-83a8-8718b0c7eef5\") " pod="llm/echo-simulator-kserve-64b6f8f96c-5dqn2" Nov 03 00:48:51.521481 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:48:51.521462 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a54c4af-a899-4f16-83a8-8718b0c7eef5-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-5dqn2\" (UID: \"0a54c4af-a899-4f16-83a8-8718b0c7eef5\") " pod="llm/echo-simulator-kserve-64b6f8f96c-5dqn2" Nov 03 00:48:51.521558 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:48:51.521530 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a54c4af-a899-4f16-83a8-8718b0c7eef5-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-5dqn2\" (UID: \"0a54c4af-a899-4f16-83a8-8718b0c7eef5\") " pod="llm/echo-simulator-kserve-64b6f8f96c-5dqn2" Nov 03 00:48:51.521698 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:48:51.521664 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a54c4af-a899-4f16-83a8-8718b0c7eef5-home\") pod \"echo-simulator-kserve-64b6f8f96c-5dqn2\" (UID: \"0a54c4af-a899-4f16-83a8-8718b0c7eef5\") " pod="llm/echo-simulator-kserve-64b6f8f96c-5dqn2" Nov 03 00:48:51.523522 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:48:51.523500 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a54c4af-a899-4f16-83a8-8718b0c7eef5-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-5dqn2\" (UID: \"0a54c4af-a899-4f16-83a8-8718b0c7eef5\") " pod="llm/echo-simulator-kserve-64b6f8f96c-5dqn2" Nov 03 00:48:51.523893 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:48:51.523872 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a54c4af-a899-4f16-83a8-8718b0c7eef5-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-5dqn2\" (UID: \"0a54c4af-a899-4f16-83a8-8718b0c7eef5\") " pod="llm/echo-simulator-kserve-64b6f8f96c-5dqn2" Nov 03 00:48:51.530344 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:48:51.530318 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29g4b\" (UniqueName: \"kubernetes.io/projected/0a54c4af-a899-4f16-83a8-8718b0c7eef5-kube-api-access-29g4b\") pod \"echo-simulator-kserve-64b6f8f96c-5dqn2\" (UID: \"0a54c4af-a899-4f16-83a8-8718b0c7eef5\") " pod="llm/echo-simulator-kserve-64b6f8f96c-5dqn2" Nov 03 00:48:51.561806 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:48:51.561774 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-5dqn2" Nov 03 00:48:51.837745 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:48:51.837723 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-5dqn2"] Nov 03 00:48:51.839949 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:48:51.839920 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a54c4af_a899_4f16_83a8_8718b0c7eef5.slice/crio-564f3a536894688641f90236cc7054556752f1edec65888f46be0e8ecd859b06 WatchSource:0}: Error finding container 564f3a536894688641f90236cc7054556752f1edec65888f46be0e8ecd859b06: Status 404 returned error can't find the container with id 564f3a536894688641f90236cc7054556752f1edec65888f46be0e8ecd859b06 Nov 03 00:48:52.060953 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:48:52.060905 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-5dqn2" event={"ID":"0a54c4af-a899-4f16-83a8-8718b0c7eef5","Type":"ContainerStarted","Data":"564f3a536894688641f90236cc7054556752f1edec65888f46be0e8ecd859b06"} Nov 03 00:48:52.399715 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:48:52.399678 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tmxmj" Nov 03 00:48:52.412641 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:48:52.412613 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tmxmj" Nov 03 00:48:53.082766 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:48:53.082734 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tmxmj" Nov 03 00:48:53.931718 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:48:53.931624 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8gm8n"] Nov 03 00:48:53.935525 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:48:53.935505 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8gm8n" Nov 03 00:48:53.944765 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:48:53.944733 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Nov 03 00:48:53.946658 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:48:53.946619 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8gm8n"] Nov 03 00:48:54.045746 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:48:54.045711 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d367e12-e46f-4a68-9825-9b1c934d4d12-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-8gm8n\" (UID: \"2d367e12-e46f-4a68-9825-9b1c934d4d12\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8gm8n" Nov 03 00:48:54.045918 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:48:54.045756 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d367e12-e46f-4a68-9825-9b1c934d4d12-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-8gm8n\" (UID: \"2d367e12-e46f-4a68-9825-9b1c934d4d12\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8gm8n" Nov 03 00:48:54.045918 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:48:54.045850 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d367e12-e46f-4a68-9825-9b1c934d4d12-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-8gm8n\" (UID: \"2d367e12-e46f-4a68-9825-9b1c934d4d12\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8gm8n" Nov 03 00:48:54.045918 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:48:54.045902 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d367e12-e46f-4a68-9825-9b1c934d4d12-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-8gm8n\" (UID: \"2d367e12-e46f-4a68-9825-9b1c934d4d12\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8gm8n" Nov 03 00:48:54.046027 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:48:54.045957 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d367e12-e46f-4a68-9825-9b1c934d4d12-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-8gm8n\" (UID: \"2d367e12-e46f-4a68-9825-9b1c934d4d12\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8gm8n" Nov 03 00:48:54.046027 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:48:54.045992 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnxkn\" (UniqueName: \"kubernetes.io/projected/2d367e12-e46f-4a68-9825-9b1c934d4d12-kube-api-access-pnxkn\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-8gm8n\" (UID: \"2d367e12-e46f-4a68-9825-9b1c934d4d12\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8gm8n" Nov 03 00:48:54.073306 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:48:54.073242 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-5dqn2" event={"ID":"0a54c4af-a899-4f16-83a8-8718b0c7eef5","Type":"ContainerStarted","Data":"ac1cbabe464836894fa81224ed8d84485d103372a23e2b24c0100af8ff7ad9cc"} Nov 03 00:48:54.091544 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:48:54.091497 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/echo-simulator-kserve-64b6f8f96c-5dqn2" podStartSLOduration=1.358916096 podStartE2EDuration="3.091481432s" totalImagesPullingTime="1.732565336s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:48:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:48:54.090601918 +0000 UTC m=+476.261605038" watchObservedRunningTime="2026-11-03 00:48:54.091481432 +0000 UTC m=+476.262484528" Nov 03 00:48:54.146836 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:48:54.146802 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d367e12-e46f-4a68-9825-9b1c934d4d12-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-8gm8n\" (UID: \"2d367e12-e46f-4a68-9825-9b1c934d4d12\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8gm8n" Nov 03 00:48:54.147032 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:48:54.146847 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d367e12-e46f-4a68-9825-9b1c934d4d12-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-8gm8n\" (UID: \"2d367e12-e46f-4a68-9825-9b1c934d4d12\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8gm8n" Nov 03 00:48:54.147032 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:48:54.146895 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d367e12-e46f-4a68-9825-9b1c934d4d12-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-8gm8n\" (UID: \"2d367e12-e46f-4a68-9825-9b1c934d4d12\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8gm8n" Nov 03 00:48:54.147032 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:48:54.146917 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d367e12-e46f-4a68-9825-9b1c934d4d12-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-8gm8n\" (UID: \"2d367e12-e46f-4a68-9825-9b1c934d4d12\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8gm8n" Nov 03 00:48:54.147032 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:48:54.146934 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnxkn\" (UniqueName: \"kubernetes.io/projected/2d367e12-e46f-4a68-9825-9b1c934d4d12-kube-api-access-pnxkn\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-8gm8n\" (UID: \"2d367e12-e46f-4a68-9825-9b1c934d4d12\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8gm8n" Nov 03 00:48:54.147032 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:48:54.147003 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d367e12-e46f-4a68-9825-9b1c934d4d12-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-8gm8n\" (UID: \"2d367e12-e46f-4a68-9825-9b1c934d4d12\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8gm8n" Nov 03 00:48:54.147401 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:48:54.147373 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d367e12-e46f-4a68-9825-9b1c934d4d12-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-8gm8n\" (UID: \"2d367e12-e46f-4a68-9825-9b1c934d4d12\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8gm8n" Nov 03 00:48:54.147495 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:48:54.147433 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d367e12-e46f-4a68-9825-9b1c934d4d12-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-8gm8n\" (UID: \"2d367e12-e46f-4a68-9825-9b1c934d4d12\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8gm8n" Nov 03 00:48:54.147495 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:48:54.147459 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d367e12-e46f-4a68-9825-9b1c934d4d12-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-8gm8n\" (UID: \"2d367e12-e46f-4a68-9825-9b1c934d4d12\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8gm8n" Nov 03 00:48:54.149209 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:48:54.149184 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d367e12-e46f-4a68-9825-9b1c934d4d12-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-8gm8n\" (UID: \"2d367e12-e46f-4a68-9825-9b1c934d4d12\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8gm8n" Nov 03 00:48:54.149377 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:48:54.149359 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d367e12-e46f-4a68-9825-9b1c934d4d12-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-8gm8n\" (UID: \"2d367e12-e46f-4a68-9825-9b1c934d4d12\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8gm8n" Nov 03 00:48:54.154879 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:48:54.154854 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnxkn\" (UniqueName: \"kubernetes.io/projected/2d367e12-e46f-4a68-9825-9b1c934d4d12-kube-api-access-pnxkn\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-8gm8n\" (UID: \"2d367e12-e46f-4a68-9825-9b1c934d4d12\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8gm8n" Nov 03 00:48:54.248407 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:48:54.248376 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8gm8n" Nov 03 00:48:54.525428 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:48:54.525400 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8gm8n"] Nov 03 00:48:54.527831 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:48:54.527805 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2d367e12_e46f_4a68_9825_9b1c934d4d12.slice/crio-b784abaf9d8beece81900b3019be51a492e6cc7de6f6aa3e98d838e2a7478fba WatchSource:0}: Error finding container b784abaf9d8beece81900b3019be51a492e6cc7de6f6aa3e98d838e2a7478fba: Status 404 returned error can't find the container with id b784abaf9d8beece81900b3019be51a492e6cc7de6f6aa3e98d838e2a7478fba Nov 03 00:48:54.562439 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:48:54.562410 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-5dqn2" Nov 03 00:48:55.080561 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:48:55.080527 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8gm8n" event={"ID":"2d367e12-e46f-4a68-9825-9b1c934d4d12","Type":"ContainerStarted","Data":"41833a40dc9800795adc851ebbc34823684f950b234b56626910a6f81879df93"} Nov 03 00:48:55.080973 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:48:55.080571 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8gm8n" event={"ID":"2d367e12-e46f-4a68-9825-9b1c934d4d12","Type":"ContainerStarted","Data":"b784abaf9d8beece81900b3019be51a492e6cc7de6f6aa3e98d838e2a7478fba"} Nov 03 00:48:55.249343 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:48:55.249298 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8gm8n" Nov 03 00:48:58.708853 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:48:58.708816 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-lc7tp" Nov 03 00:48:58.721400 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:48:58.721370 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-lc7tp" Nov 03 00:48:58.739798 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:48:58.739746 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8gm8n" podStartSLOduration=5.423808168 podStartE2EDuration="5.739730536s" totalImagesPullingTime="315.922368ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:48:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:48:55.101345667 +0000 UTC m=+477.272348765" watchObservedRunningTime="2026-11-03 00:48:58.739730536 +0000 UTC m=+480.910733643" Nov 03 00:48:58.951077 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:48:58.951042 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-q5n9x"] Nov 03 00:48:58.954733 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:48:58.954706 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-q5n9x" Nov 03 00:48:58.960921 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:48:58.960853 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-pjnsk\"" Nov 03 00:48:58.964908 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:48:58.964007 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 03 00:48:58.970509 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:48:58.970479 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-q5n9x"] Nov 03 00:48:58.991357 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:48:58.991328 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e11a97cb-0980-45d1-842f-ded2fb31b03f-workload-certs\") pod \"maas-default-gateway-openshift-default-596dc9545f-q5n9x\" (UID: \"e11a97cb-0980-45d1-842f-ded2fb31b03f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-q5n9x" Nov 03 00:48:58.991357 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:48:58.991360 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e11a97cb-0980-45d1-842f-ded2fb31b03f-istio-data\") pod \"maas-default-gateway-openshift-default-596dc9545f-q5n9x\" (UID: \"e11a97cb-0980-45d1-842f-ded2fb31b03f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-q5n9x" Nov 03 00:48:58.991531 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:48:58.991399 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e11a97cb-0980-45d1-842f-ded2fb31b03f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-596dc9545f-q5n9x\" (UID: \"e11a97cb-0980-45d1-842f-ded2fb31b03f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-q5n9x" Nov 03 00:48:58.991531 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:48:58.991488 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e11a97cb-0980-45d1-842f-ded2fb31b03f-istio-token\") pod \"maas-default-gateway-openshift-default-596dc9545f-q5n9x\" (UID: \"e11a97cb-0980-45d1-842f-ded2fb31b03f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-q5n9x" Nov 03 00:48:58.991531 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:48:58.991522 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gss2\" (UniqueName: \"kubernetes.io/projected/e11a97cb-0980-45d1-842f-ded2fb31b03f-kube-api-access-9gss2\") pod \"maas-default-gateway-openshift-default-596dc9545f-q5n9x\" (UID: \"e11a97cb-0980-45d1-842f-ded2fb31b03f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-q5n9x" Nov 03 00:48:58.991670 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:48:58.991546 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e11a97cb-0980-45d1-842f-ded2fb31b03f-istio-envoy\") pod \"maas-default-gateway-openshift-default-596dc9545f-q5n9x\" (UID: \"e11a97cb-0980-45d1-842f-ded2fb31b03f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-q5n9x" Nov 03 00:48:58.991670 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:48:58.991572 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e11a97cb-0980-45d1-842f-ded2fb31b03f-credential-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-q5n9x\" (UID: \"e11a97cb-0980-45d1-842f-ded2fb31b03f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-q5n9x" Nov 03 00:48:58.991670 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:48:58.991637 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e11a97cb-0980-45d1-842f-ded2fb31b03f-workload-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-q5n9x\" (UID: \"e11a97cb-0980-45d1-842f-ded2fb31b03f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-q5n9x" Nov 03 00:48:58.991763 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:48:58.991667 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e11a97cb-0980-45d1-842f-ded2fb31b03f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-596dc9545f-q5n9x\" (UID: \"e11a97cb-0980-45d1-842f-ded2fb31b03f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-q5n9x" Nov 03 00:48:59.092791 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:48:59.092752 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e11a97cb-0980-45d1-842f-ded2fb31b03f-istio-token\") pod \"maas-default-gateway-openshift-default-596dc9545f-q5n9x\" (UID: \"e11a97cb-0980-45d1-842f-ded2fb31b03f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-q5n9x" Nov 03 00:48:59.092940 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:48:59.092805 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gss2\" (UniqueName: \"kubernetes.io/projected/e11a97cb-0980-45d1-842f-ded2fb31b03f-kube-api-access-9gss2\") pod \"maas-default-gateway-openshift-default-596dc9545f-q5n9x\" (UID: \"e11a97cb-0980-45d1-842f-ded2fb31b03f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-q5n9x" Nov 03 00:48:59.092940 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:48:59.092834 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e11a97cb-0980-45d1-842f-ded2fb31b03f-istio-envoy\") pod \"maas-default-gateway-openshift-default-596dc9545f-q5n9x\" (UID: \"e11a97cb-0980-45d1-842f-ded2fb31b03f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-q5n9x" Nov 03 00:48:59.092940 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:48:59.092856 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e11a97cb-0980-45d1-842f-ded2fb31b03f-credential-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-q5n9x\" (UID: \"e11a97cb-0980-45d1-842f-ded2fb31b03f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-q5n9x" Nov 03 00:48:59.093057 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:48:59.093001 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e11a97cb-0980-45d1-842f-ded2fb31b03f-workload-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-q5n9x\" (UID: \"e11a97cb-0980-45d1-842f-ded2fb31b03f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-q5n9x" Nov 03 00:48:59.093088 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:48:59.093051 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e11a97cb-0980-45d1-842f-ded2fb31b03f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-596dc9545f-q5n9x\" (UID: \"e11a97cb-0980-45d1-842f-ded2fb31b03f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-q5n9x" Nov 03 00:48:59.093132 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:48:59.093116 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e11a97cb-0980-45d1-842f-ded2fb31b03f-workload-certs\") pod \"maas-default-gateway-openshift-default-596dc9545f-q5n9x\" (UID: \"e11a97cb-0980-45d1-842f-ded2fb31b03f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-q5n9x" Nov 03 00:48:59.093167 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:48:59.093144 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e11a97cb-0980-45d1-842f-ded2fb31b03f-istio-data\") pod \"maas-default-gateway-openshift-default-596dc9545f-q5n9x\" (UID: \"e11a97cb-0980-45d1-842f-ded2fb31b03f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-q5n9x" Nov 03 00:48:59.093230 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:48:59.093210 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e11a97cb-0980-45d1-842f-ded2fb31b03f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-596dc9545f-q5n9x\" (UID: \"e11a97cb-0980-45d1-842f-ded2fb31b03f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-q5n9x" Nov 03 00:48:59.093434 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:48:59.093409 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e11a97cb-0980-45d1-842f-ded2fb31b03f-workload-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-q5n9x\" (UID: \"e11a97cb-0980-45d1-842f-ded2fb31b03f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-q5n9x" Nov 03 00:48:59.093512 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:48:59.093370 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e11a97cb-0980-45d1-842f-ded2fb31b03f-credential-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-q5n9x\" (UID: \"e11a97cb-0980-45d1-842f-ded2fb31b03f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-q5n9x" Nov 03 00:48:59.093573 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:48:59.093521 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e11a97cb-0980-45d1-842f-ded2fb31b03f-workload-certs\") pod \"maas-default-gateway-openshift-default-596dc9545f-q5n9x\" (UID: \"e11a97cb-0980-45d1-842f-ded2fb31b03f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-q5n9x" Nov 03 00:48:59.093629 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:48:59.093609 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e11a97cb-0980-45d1-842f-ded2fb31b03f-istio-data\") pod \"maas-default-gateway-openshift-default-596dc9545f-q5n9x\" (UID: \"e11a97cb-0980-45d1-842f-ded2fb31b03f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-q5n9x" Nov 03 00:48:59.095412 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:48:59.095375 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e11a97cb-0980-45d1-842f-ded2fb31b03f-istio-envoy\") pod \"maas-default-gateway-openshift-default-596dc9545f-q5n9x\" (UID: \"e11a97cb-0980-45d1-842f-ded2fb31b03f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-q5n9x" Nov 03 00:48:59.100897 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:48:59.100864 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e11a97cb-0980-45d1-842f-ded2fb31b03f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-596dc9545f-q5n9x\" (UID: \"e11a97cb-0980-45d1-842f-ded2fb31b03f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-q5n9x" Nov 03 00:48:59.101381 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:48:59.101348 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e11a97cb-0980-45d1-842f-ded2fb31b03f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-596dc9545f-q5n9x\" (UID: \"e11a97cb-0980-45d1-842f-ded2fb31b03f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-q5n9x" Nov 03 00:48:59.102507 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:48:59.102486 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e11a97cb-0980-45d1-842f-ded2fb31b03f-istio-token\") pod \"maas-default-gateway-openshift-default-596dc9545f-q5n9x\" (UID: \"e11a97cb-0980-45d1-842f-ded2fb31b03f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-q5n9x" Nov 03 00:48:59.105269 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:48:59.105244 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gss2\" (UniqueName: \"kubernetes.io/projected/e11a97cb-0980-45d1-842f-ded2fb31b03f-kube-api-access-9gss2\") pod \"maas-default-gateway-openshift-default-596dc9545f-q5n9x\" (UID: \"e11a97cb-0980-45d1-842f-ded2fb31b03f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-q5n9x" Nov 03 00:48:59.116248 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:48:59.116229 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-lc7tp" Nov 03 00:48:59.272319 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:48:59.272257 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-q5n9x" Nov 03 00:48:59.549027 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:48:59.548997 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-q5n9x"] Nov 03 00:48:59.551006 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:48:59.550981 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode11a97cb_0980_45d1_842f_ded2fb31b03f.slice/crio-f45a5cfe8853d6eaa44edccc790090356cfd38b3ced8ed7b70af83b483a73e70 WatchSource:0}: Error finding container f45a5cfe8853d6eaa44edccc790090356cfd38b3ced8ed7b70af83b483a73e70: Status 404 returned error can't find the container with id f45a5cfe8853d6eaa44edccc790090356cfd38b3ced8ed7b70af83b483a73e70 Nov 03 00:49:00.108565 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:00.108530 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-q5n9x" event={"ID":"e11a97cb-0980-45d1-842f-ded2fb31b03f","Type":"ContainerStarted","Data":"f45a5cfe8853d6eaa44edccc790090356cfd38b3ced8ed7b70af83b483a73e70"} Nov 03 00:49:01.562652 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:01.562618 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/echo-simulator-kserve-64b6f8f96c-5dqn2" Nov 03 00:49:01.575677 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:01.575648 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/echo-simulator-kserve-64b6f8f96c-5dqn2" Nov 03 00:49:02.132354 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:02.132323 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-5dqn2" Nov 03 00:49:02.627463 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:02.627430 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:49:02.627772 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:02.627517 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:49:02.627772 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:02.627549 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:49:03.138900 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:03.138865 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-q5n9x" event={"ID":"e11a97cb-0980-45d1-842f-ded2fb31b03f","Type":"ContainerStarted","Data":"5da7171c61ec9250086c2340c2c52a2491f3515a7d637afbbf2d4787310c7460"} Nov 03 00:49:03.165509 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:03.165447 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-q5n9x" podStartSLOduration=2.092103776 podStartE2EDuration="5.165428646s" totalImagesPullingTime="3.07332487s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:48:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:49:03.163110306 +0000 UTC m=+485.334113405" watchObservedRunningTime="2026-11-03 00:49:03.165428646 +0000 UTC m=+485.336431743" Nov 03 00:49:03.272748 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:03.272713 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-q5n9x" Nov 03 00:49:03.274350 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:03.274311 4421 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-q5n9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Nov 03 00:49:03.274487 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:03.274384 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-q5n9x" podUID="e11a97cb-0980-45d1-842f-ded2fb31b03f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Nov 03 00:49:04.249247 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:04.249213 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8gm8n" Nov 03 00:49:04.262041 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:04.262012 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8gm8n" Nov 03 00:49:04.272823 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:04.272796 4421 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-q5n9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Nov 03 00:49:04.272930 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:04.272843 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-q5n9x" podUID="e11a97cb-0980-45d1-842f-ded2fb31b03f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Nov 03 00:49:05.158787 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:05.158751 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8gm8n" Nov 03 00:49:05.273350 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:05.273316 4421 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-q5n9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Nov 03 00:49:05.273806 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:05.273372 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-q5n9x" podUID="e11a97cb-0980-45d1-842f-ded2fb31b03f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Nov 03 00:49:06.273346 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:06.273267 4421 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-q5n9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Nov 03 00:49:06.273794 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:06.273373 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-q5n9x" podUID="e11a97cb-0980-45d1-842f-ded2fb31b03f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Nov 03 00:49:07.004809 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:07.004773 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-597cdd5495-wq4hg"] Nov 03 00:49:07.032489 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:07.032448 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-597cdd5495-wq4hg"] Nov 03 00:49:07.032630 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:07.032573 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-597cdd5495-wq4hg" Nov 03 00:49:07.068387 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:07.068347 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqj24\" (UniqueName: \"kubernetes.io/projected/5f9699ce-0019-4989-a2df-3043231dd93e-kube-api-access-tqj24\") pod \"authorino-597cdd5495-wq4hg\" (UID: \"5f9699ce-0019-4989-a2df-3043231dd93e\") " pod="kuadrant-system/authorino-597cdd5495-wq4hg" Nov 03 00:49:07.068519 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:07.068391 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5f9699ce-0019-4989-a2df-3043231dd93e-tls-cert\") pod \"authorino-597cdd5495-wq4hg\" (UID: \"5f9699ce-0019-4989-a2df-3043231dd93e\") " pod="kuadrant-system/authorino-597cdd5495-wq4hg" Nov 03 00:49:07.169601 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:07.169567 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqj24\" (UniqueName: \"kubernetes.io/projected/5f9699ce-0019-4989-a2df-3043231dd93e-kube-api-access-tqj24\") pod \"authorino-597cdd5495-wq4hg\" (UID: \"5f9699ce-0019-4989-a2df-3043231dd93e\") " pod="kuadrant-system/authorino-597cdd5495-wq4hg" Nov 03 00:49:07.169778 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:07.169612 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5f9699ce-0019-4989-a2df-3043231dd93e-tls-cert\") pod \"authorino-597cdd5495-wq4hg\" (UID: \"5f9699ce-0019-4989-a2df-3043231dd93e\") " pod="kuadrant-system/authorino-597cdd5495-wq4hg" Nov 03 00:49:07.172098 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:07.172077 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5f9699ce-0019-4989-a2df-3043231dd93e-tls-cert\") pod \"authorino-597cdd5495-wq4hg\" (UID: \"5f9699ce-0019-4989-a2df-3043231dd93e\") " pod="kuadrant-system/authorino-597cdd5495-wq4hg" Nov 03 00:49:07.177766 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:07.177744 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqj24\" (UniqueName: \"kubernetes.io/projected/5f9699ce-0019-4989-a2df-3043231dd93e-kube-api-access-tqj24\") pod \"authorino-597cdd5495-wq4hg\" (UID: \"5f9699ce-0019-4989-a2df-3043231dd93e\") " pod="kuadrant-system/authorino-597cdd5495-wq4hg" Nov 03 00:49:07.273311 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:07.273203 4421 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-q5n9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Nov 03 00:49:07.273311 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:07.273260 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-q5n9x" podUID="e11a97cb-0980-45d1-842f-ded2fb31b03f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Nov 03 00:49:07.344718 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:07.344674 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-597cdd5495-wq4hg" Nov 03 00:49:07.624347 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:07.615046 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-597cdd5495-wq4hg"] Nov 03 00:49:07.633472 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:49:07.633437 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5f9699ce_0019_4989_a2df_3043231dd93e.slice/crio-a1223e9ca0783ad071e9a95246d3fd0003d59eda830c8bdf6d0e64f30ec66bc2 WatchSource:0}: Error finding container a1223e9ca0783ad071e9a95246d3fd0003d59eda830c8bdf6d0e64f30ec66bc2: Status 404 returned error can't find the container with id a1223e9ca0783ad071e9a95246d3fd0003d59eda830c8bdf6d0e64f30ec66bc2 Nov 03 00:49:08.163090 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:08.163050 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-597cdd5495-wq4hg" event={"ID":"5f9699ce-0019-4989-a2df-3043231dd93e","Type":"ContainerStarted","Data":"a1223e9ca0783ad071e9a95246d3fd0003d59eda830c8bdf6d0e64f30ec66bc2"} Nov 03 00:49:08.272783 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:08.272743 4421 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-q5n9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Nov 03 00:49:08.272972 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:08.272804 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-q5n9x" podUID="e11a97cb-0980-45d1-842f-ded2fb31b03f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Nov 03 00:49:09.174295 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:09.173875 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-597cdd5495-wq4hg" event={"ID":"5f9699ce-0019-4989-a2df-3043231dd93e","Type":"ContainerStarted","Data":"87da69bb4f0895ca14616d5ba44b9b834f24cdb4651fb8710d7ea60431b6d872"} Nov 03 00:49:09.194298 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:09.192016 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-597cdd5495-wq4hg" podStartSLOduration=2.325898946 podStartE2EDuration="3.19199701s" totalImagesPullingTime="866.098064ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:49:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:49:09.190424215 +0000 UTC m=+491.361427312" watchObservedRunningTime="2026-11-03 00:49:09.19199701 +0000 UTC m=+491.363000108" Nov 03 00:49:09.223302 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:09.222299 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5fcd47cd5b-rwf6t"] Nov 03 00:49:09.223302 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:09.222623 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-5fcd47cd5b-rwf6t" podUID="f1378599-2104-4475-a307-097042b606a6" containerName="authorino" containerID="cri-o://ef9fa9ade8ef4d06214377dc413596c7a008abc070b1d0ae451a335e104095f5" gracePeriod=30 Nov 03 00:49:09.272888 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:09.272850 4421 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-q5n9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Nov 03 00:49:09.273024 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:09.272904 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-q5n9x" podUID="e11a97cb-0980-45d1-842f-ded2fb31b03f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Nov 03 00:49:09.488495 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:09.488462 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5fcd47cd5b-rwf6t" Nov 03 00:49:09.591964 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:09.591928 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1378599-2104-4475-a307-097042b606a6-kube-api-access-glw2l\" (UniqueName: \"kubernetes.io/projected/f1378599-2104-4475-a307-097042b606a6-kube-api-access-glw2l\") pod \"f1378599-2104-4475-a307-097042b606a6\" (UID: \"f1378599-2104-4475-a307-097042b606a6\") " Nov 03 00:49:09.592139 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:09.591988 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1378599-2104-4475-a307-097042b606a6-tls-cert\" (UniqueName: \"kubernetes.io/secret/f1378599-2104-4475-a307-097042b606a6-tls-cert\") pod \"f1378599-2104-4475-a307-097042b606a6\" (UID: \"f1378599-2104-4475-a307-097042b606a6\") " Nov 03 00:49:09.594119 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:09.594092 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1378599-2104-4475-a307-097042b606a6-kube-api-access-glw2l" pod "f1378599-2104-4475-a307-097042b606a6" (UID: "f1378599-2104-4475-a307-097042b606a6"). InnerVolumeSpecName "kube-api-access-glw2l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:49:09.604099 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:09.604075 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1378599-2104-4475-a307-097042b606a6-tls-cert" pod "f1378599-2104-4475-a307-097042b606a6" (UID: "f1378599-2104-4475-a307-097042b606a6"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:49:09.693495 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:09.693453 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-glw2l\" (UniqueName: \"kubernetes.io/projected/f1378599-2104-4475-a307-097042b606a6-kube-api-access-glw2l\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:49:09.693495 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:09.693488 4421 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f1378599-2104-4475-a307-097042b606a6-tls-cert\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:49:10.183157 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:10.183073 4421 generic.go:356] "Generic (PLEG): container finished" podID="f1378599-2104-4475-a307-097042b606a6" containerID="ef9fa9ade8ef4d06214377dc413596c7a008abc070b1d0ae451a335e104095f5" exitCode=0 Nov 03 00:49:10.184349 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:10.184326 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5fcd47cd5b-rwf6t" Nov 03 00:49:10.184897 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:10.184879 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5fcd47cd5b-rwf6t" event={"ID":"f1378599-2104-4475-a307-097042b606a6","Type":"ContainerDied","Data":"ef9fa9ade8ef4d06214377dc413596c7a008abc070b1d0ae451a335e104095f5"} Nov 03 00:49:10.185026 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:10.185012 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5fcd47cd5b-rwf6t" event={"ID":"f1378599-2104-4475-a307-097042b606a6","Type":"ContainerDied","Data":"ad37e8b44a27421264585980e80e5e7eb69ccb5c9832c4b0e8cc377709c1bde2"} Nov 03 00:49:10.185121 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:10.185108 4421 scope.go:122] "RemoveContainer" containerID="ef9fa9ade8ef4d06214377dc413596c7a008abc070b1d0ae451a335e104095f5" Nov 03 00:49:10.208192 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:10.208170 4421 scope.go:122] "RemoveContainer" containerID="ef9fa9ade8ef4d06214377dc413596c7a008abc070b1d0ae451a335e104095f5" Nov 03 00:49:10.208631 ip-10-0-133-217 kubenswrapper[4421]: E1103 00:49:10.208604 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef9fa9ade8ef4d06214377dc413596c7a008abc070b1d0ae451a335e104095f5\": container with ID starting with ef9fa9ade8ef4d06214377dc413596c7a008abc070b1d0ae451a335e104095f5 not found: ID does not exist" containerID="ef9fa9ade8ef4d06214377dc413596c7a008abc070b1d0ae451a335e104095f5" Nov 03 00:49:10.208737 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:10.208637 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef9fa9ade8ef4d06214377dc413596c7a008abc070b1d0ae451a335e104095f5"} err="failed to get container status \"ef9fa9ade8ef4d06214377dc413596c7a008abc070b1d0ae451a335e104095f5\": rpc error: code = NotFound desc = could not find container \"ef9fa9ade8ef4d06214377dc413596c7a008abc070b1d0ae451a335e104095f5\": container with ID starting with ef9fa9ade8ef4d06214377dc413596c7a008abc070b1d0ae451a335e104095f5 not found: ID does not exist" Nov 03 00:49:10.228043 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:10.228010 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5fcd47cd5b-rwf6t"] Nov 03 00:49:10.230650 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:10.230625 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5fcd47cd5b-rwf6t"] Nov 03 00:49:10.273262 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:10.273227 4421 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-q5n9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Nov 03 00:49:10.273436 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:10.273309 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-q5n9x" podUID="e11a97cb-0980-45d1-842f-ded2fb31b03f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Nov 03 00:49:11.277174 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:11.277141 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-q5n9x" Nov 03 00:49:11.277644 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:11.277514 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-q5n9x" Nov 03 00:49:11.278324 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:11.278303 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-q5n9x" Nov 03 00:49:11.977793 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:11.977763 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1378599-2104-4475-a307-097042b606a6" path="/var/lib/kubelet/pods/f1378599-2104-4475-a307-097042b606a6/volumes" Nov 03 00:49:13.983296 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:13.981574 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-s4dc8"] Nov 03 00:49:13.987616 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:13.987583 4421 cpu_manager.go:414] "containerMap: removing container" podUID="f1378599-2104-4475-a307-097042b606a6" containerName="authorino" Nov 03 00:49:13.987616 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:13.987614 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1378599-2104-4475-a307-097042b606a6" containerName="authorino" Nov 03 00:49:13.996110 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:13.996084 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1378599-2104-4475-a307-097042b606a6" containerName="authorino" Nov 03 00:49:14.004552 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:14.004386 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-s4dc8"] Nov 03 00:49:14.004806 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:14.004529 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-s4dc8" Nov 03 00:49:14.044971 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:14.044930 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab5414b5-9d47-4c9f-9d0e-82242b9d6db2-workload-certs\") pod \"maas-default-gateway-openshift-default-596dc9545f-s4dc8\" (UID: \"ab5414b5-9d47-4c9f-9d0e-82242b9d6db2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-s4dc8" Nov 03 00:49:14.045170 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:14.045003 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab5414b5-9d47-4c9f-9d0e-82242b9d6db2-istio-data\") pod \"maas-default-gateway-openshift-default-596dc9545f-s4dc8\" (UID: \"ab5414b5-9d47-4c9f-9d0e-82242b9d6db2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-s4dc8" Nov 03 00:49:14.045170 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:14.045057 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab5414b5-9d47-4c9f-9d0e-82242b9d6db2-istio-envoy\") pod \"maas-default-gateway-openshift-default-596dc9545f-s4dc8\" (UID: \"ab5414b5-9d47-4c9f-9d0e-82242b9d6db2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-s4dc8" Nov 03 00:49:14.045170 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:14.045126 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab5414b5-9d47-4c9f-9d0e-82242b9d6db2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-596dc9545f-s4dc8\" (UID: \"ab5414b5-9d47-4c9f-9d0e-82242b9d6db2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-s4dc8" Nov 03 00:49:14.045170 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:14.045152 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab5414b5-9d47-4c9f-9d0e-82242b9d6db2-istio-token\") pod \"maas-default-gateway-openshift-default-596dc9545f-s4dc8\" (UID: \"ab5414b5-9d47-4c9f-9d0e-82242b9d6db2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-s4dc8" Nov 03 00:49:14.045515 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:14.045182 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab5414b5-9d47-4c9f-9d0e-82242b9d6db2-credential-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-s4dc8\" (UID: \"ab5414b5-9d47-4c9f-9d0e-82242b9d6db2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-s4dc8" Nov 03 00:49:14.045515 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:14.045237 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab5414b5-9d47-4c9f-9d0e-82242b9d6db2-workload-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-s4dc8\" (UID: \"ab5414b5-9d47-4c9f-9d0e-82242b9d6db2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-s4dc8" Nov 03 00:49:14.045515 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:14.045316 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab5414b5-9d47-4c9f-9d0e-82242b9d6db2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-596dc9545f-s4dc8\" (UID: \"ab5414b5-9d47-4c9f-9d0e-82242b9d6db2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-s4dc8" Nov 03 00:49:14.045515 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:14.045357 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4khp\" (UniqueName: \"kubernetes.io/projected/ab5414b5-9d47-4c9f-9d0e-82242b9d6db2-kube-api-access-f4khp\") pod \"maas-default-gateway-openshift-default-596dc9545f-s4dc8\" (UID: \"ab5414b5-9d47-4c9f-9d0e-82242b9d6db2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-s4dc8" Nov 03 00:49:14.146128 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:14.146093 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab5414b5-9d47-4c9f-9d0e-82242b9d6db2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-596dc9545f-s4dc8\" (UID: \"ab5414b5-9d47-4c9f-9d0e-82242b9d6db2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-s4dc8" Nov 03 00:49:14.146128 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:14.146135 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab5414b5-9d47-4c9f-9d0e-82242b9d6db2-istio-token\") pod \"maas-default-gateway-openshift-default-596dc9545f-s4dc8\" (UID: \"ab5414b5-9d47-4c9f-9d0e-82242b9d6db2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-s4dc8" Nov 03 00:49:14.146348 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:14.146236 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab5414b5-9d47-4c9f-9d0e-82242b9d6db2-credential-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-s4dc8\" (UID: \"ab5414b5-9d47-4c9f-9d0e-82242b9d6db2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-s4dc8" Nov 03 00:49:14.146348 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:14.146314 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab5414b5-9d47-4c9f-9d0e-82242b9d6db2-workload-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-s4dc8\" (UID: \"ab5414b5-9d47-4c9f-9d0e-82242b9d6db2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-s4dc8" Nov 03 00:49:14.146422 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:14.146358 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab5414b5-9d47-4c9f-9d0e-82242b9d6db2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-596dc9545f-s4dc8\" (UID: \"ab5414b5-9d47-4c9f-9d0e-82242b9d6db2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-s4dc8" Nov 03 00:49:14.146422 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:14.146396 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4khp\" (UniqueName: \"kubernetes.io/projected/ab5414b5-9d47-4c9f-9d0e-82242b9d6db2-kube-api-access-f4khp\") pod \"maas-default-gateway-openshift-default-596dc9545f-s4dc8\" (UID: \"ab5414b5-9d47-4c9f-9d0e-82242b9d6db2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-s4dc8" Nov 03 00:49:14.146492 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:14.146472 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab5414b5-9d47-4c9f-9d0e-82242b9d6db2-workload-certs\") pod \"maas-default-gateway-openshift-default-596dc9545f-s4dc8\" (UID: \"ab5414b5-9d47-4c9f-9d0e-82242b9d6db2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-s4dc8" Nov 03 00:49:14.146538 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:14.146504 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab5414b5-9d47-4c9f-9d0e-82242b9d6db2-istio-data\") pod \"maas-default-gateway-openshift-default-596dc9545f-s4dc8\" (UID: \"ab5414b5-9d47-4c9f-9d0e-82242b9d6db2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-s4dc8" Nov 03 00:49:14.146574 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:14.146559 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab5414b5-9d47-4c9f-9d0e-82242b9d6db2-istio-envoy\") pod \"maas-default-gateway-openshift-default-596dc9545f-s4dc8\" (UID: \"ab5414b5-9d47-4c9f-9d0e-82242b9d6db2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-s4dc8" Nov 03 00:49:14.147061 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:14.147029 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab5414b5-9d47-4c9f-9d0e-82242b9d6db2-workload-certs\") pod \"maas-default-gateway-openshift-default-596dc9545f-s4dc8\" (UID: \"ab5414b5-9d47-4c9f-9d0e-82242b9d6db2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-s4dc8" Nov 03 00:49:14.147162 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:14.147096 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab5414b5-9d47-4c9f-9d0e-82242b9d6db2-istio-data\") pod \"maas-default-gateway-openshift-default-596dc9545f-s4dc8\" (UID: \"ab5414b5-9d47-4c9f-9d0e-82242b9d6db2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-s4dc8" Nov 03 00:49:14.147301 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:14.147255 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab5414b5-9d47-4c9f-9d0e-82242b9d6db2-credential-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-s4dc8\" (UID: \"ab5414b5-9d47-4c9f-9d0e-82242b9d6db2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-s4dc8" Nov 03 00:49:14.147422 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:14.147305 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab5414b5-9d47-4c9f-9d0e-82242b9d6db2-workload-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-s4dc8\" (UID: \"ab5414b5-9d47-4c9f-9d0e-82242b9d6db2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-s4dc8" Nov 03 00:49:14.147483 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:14.147421 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab5414b5-9d47-4c9f-9d0e-82242b9d6db2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-596dc9545f-s4dc8\" (UID: \"ab5414b5-9d47-4c9f-9d0e-82242b9d6db2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-s4dc8" Nov 03 00:49:14.149178 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:14.149156 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab5414b5-9d47-4c9f-9d0e-82242b9d6db2-istio-envoy\") pod \"maas-default-gateway-openshift-default-596dc9545f-s4dc8\" (UID: \"ab5414b5-9d47-4c9f-9d0e-82242b9d6db2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-s4dc8" Nov 03 00:49:14.149763 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:14.149737 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab5414b5-9d47-4c9f-9d0e-82242b9d6db2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-596dc9545f-s4dc8\" (UID: \"ab5414b5-9d47-4c9f-9d0e-82242b9d6db2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-s4dc8" Nov 03 00:49:14.156761 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:14.156402 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab5414b5-9d47-4c9f-9d0e-82242b9d6db2-istio-token\") pod \"maas-default-gateway-openshift-default-596dc9545f-s4dc8\" (UID: \"ab5414b5-9d47-4c9f-9d0e-82242b9d6db2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-s4dc8" Nov 03 00:49:14.156761 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:14.156700 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4khp\" (UniqueName: \"kubernetes.io/projected/ab5414b5-9d47-4c9f-9d0e-82242b9d6db2-kube-api-access-f4khp\") pod \"maas-default-gateway-openshift-default-596dc9545f-s4dc8\" (UID: \"ab5414b5-9d47-4c9f-9d0e-82242b9d6db2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-s4dc8" Nov 03 00:49:14.331438 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:14.331398 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-s4dc8" Nov 03 00:49:14.613477 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:14.613450 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-s4dc8"] Nov 03 00:49:14.615223 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:49:14.615195 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab5414b5_9d47_4c9f_9d0e_82242b9d6db2.slice/crio-88b5d0bad443e7afe8b8f0a2cbf0ff37c99f8959b85aaae1df362ea20cc68670 WatchSource:0}: Error finding container 88b5d0bad443e7afe8b8f0a2cbf0ff37c99f8959b85aaae1df362ea20cc68670: Status 404 returned error can't find the container with id 88b5d0bad443e7afe8b8f0a2cbf0ff37c99f8959b85aaae1df362ea20cc68670 Nov 03 00:49:14.618763 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:14.618728 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:49:14.618872 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:14.618800 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:49:14.618872 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:14.618830 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:49:14.675117 ip-10-0-133-217 kubenswrapper[4421]: E1103 00:49:14.675082 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 00:49:15.234520 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:15.234476 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-s4dc8" event={"ID":"ab5414b5-9d47-4c9f-9d0e-82242b9d6db2","Type":"ContainerStarted","Data":"9d5ba681878dfed341d166951809890fac745065100910e1395bcbf58a9cc9e8"} Nov 03 00:49:15.235004 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:15.234543 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-s4dc8" event={"ID":"ab5414b5-9d47-4c9f-9d0e-82242b9d6db2","Type":"ContainerStarted","Data":"88b5d0bad443e7afe8b8f0a2cbf0ff37c99f8959b85aaae1df362ea20cc68670"} Nov 03 00:49:15.275844 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:15.275784 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-s4dc8" podStartSLOduration=2.275763845 podStartE2EDuration="2.275763845s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:49:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:49:15.275378532 +0000 UTC m=+497.446381641" watchObservedRunningTime="2026-11-03 00:49:15.275763845 +0000 UTC m=+497.446766940" Nov 03 00:49:15.332607 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:15.332562 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-s4dc8" Nov 03 00:49:15.334327 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:15.334291 4421 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-s4dc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Nov 03 00:49:15.334475 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:15.334362 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-s4dc8" podUID="ab5414b5-9d47-4c9f-9d0e-82242b9d6db2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Nov 03 00:49:16.332844 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:16.332808 4421 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-s4dc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Nov 03 00:49:16.333261 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:16.332865 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-s4dc8" podUID="ab5414b5-9d47-4c9f-9d0e-82242b9d6db2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Nov 03 00:49:17.332442 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:17.332396 4421 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-s4dc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Nov 03 00:49:17.332623 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:17.332473 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-s4dc8" podUID="ab5414b5-9d47-4c9f-9d0e-82242b9d6db2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Nov 03 00:49:18.332939 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:18.332895 4421 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-s4dc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Nov 03 00:49:18.333330 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:18.332967 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-s4dc8" podUID="ab5414b5-9d47-4c9f-9d0e-82242b9d6db2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Nov 03 00:49:19.332797 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:19.332753 4421 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-s4dc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Nov 03 00:49:19.333196 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:19.333165 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-s4dc8" podUID="ab5414b5-9d47-4c9f-9d0e-82242b9d6db2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Nov 03 00:49:20.331856 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:20.331810 4421 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-s4dc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Nov 03 00:49:20.332077 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:20.331881 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-s4dc8" podUID="ab5414b5-9d47-4c9f-9d0e-82242b9d6db2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Nov 03 00:49:21.332017 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:21.331981 4421 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-s4dc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Nov 03 00:49:21.332534 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:21.332059 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-s4dc8" podUID="ab5414b5-9d47-4c9f-9d0e-82242b9d6db2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Nov 03 00:49:22.877381 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:22.877337 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-s4dc8" Nov 03 00:49:22.878006 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:22.877981 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-s4dc8" Nov 03 00:49:22.878894 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:22.878873 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-s4dc8" Nov 03 00:49:29.016822 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:29.016787 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-t5bt4"] Nov 03 00:49:29.024658 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:29.024621 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-t5bt4" Nov 03 00:49:29.035214 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:29.035175 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-t5bt4"] Nov 03 00:49:29.120773 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:29.120736 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b55c151-d235-4bcc-b602-eca0da4a7348-istio-data\") pod \"maas-default-gateway-openshift-default-596dc9545f-t5bt4\" (UID: \"2b55c151-d235-4bcc-b602-eca0da4a7348\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-t5bt4" Nov 03 00:49:29.120951 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:29.120819 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b55c151-d235-4bcc-b602-eca0da4a7348-workload-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-t5bt4\" (UID: \"2b55c151-d235-4bcc-b602-eca0da4a7348\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-t5bt4" Nov 03 00:49:29.120951 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:29.120855 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b55c151-d235-4bcc-b602-eca0da4a7348-istio-envoy\") pod \"maas-default-gateway-openshift-default-596dc9545f-t5bt4\" (UID: \"2b55c151-d235-4bcc-b602-eca0da4a7348\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-t5bt4" Nov 03 00:49:29.121023 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:29.120985 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b55c151-d235-4bcc-b602-eca0da4a7348-workload-certs\") pod \"maas-default-gateway-openshift-default-596dc9545f-t5bt4\" (UID: \"2b55c151-d235-4bcc-b602-eca0da4a7348\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-t5bt4" Nov 03 00:49:29.121066 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:29.121051 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bb9cl\" (UniqueName: \"kubernetes.io/projected/2b55c151-d235-4bcc-b602-eca0da4a7348-kube-api-access-bb9cl\") pod \"maas-default-gateway-openshift-default-596dc9545f-t5bt4\" (UID: \"2b55c151-d235-4bcc-b602-eca0da4a7348\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-t5bt4" Nov 03 00:49:29.121103 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:29.121077 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b55c151-d235-4bcc-b602-eca0da4a7348-credential-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-t5bt4\" (UID: \"2b55c151-d235-4bcc-b602-eca0da4a7348\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-t5bt4" Nov 03 00:49:29.121137 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:29.121105 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b55c151-d235-4bcc-b602-eca0da4a7348-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-596dc9545f-t5bt4\" (UID: \"2b55c151-d235-4bcc-b602-eca0da4a7348\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-t5bt4" Nov 03 00:49:29.121243 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:29.121224 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b55c151-d235-4bcc-b602-eca0da4a7348-istio-token\") pod \"maas-default-gateway-openshift-default-596dc9545f-t5bt4\" (UID: \"2b55c151-d235-4bcc-b602-eca0da4a7348\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-t5bt4" Nov 03 00:49:29.121316 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:29.121264 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b55c151-d235-4bcc-b602-eca0da4a7348-istio-podinfo\") pod \"maas-default-gateway-openshift-default-596dc9545f-t5bt4\" (UID: \"2b55c151-d235-4bcc-b602-eca0da4a7348\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-t5bt4" Nov 03 00:49:29.222191 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:29.222147 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b55c151-d235-4bcc-b602-eca0da4a7348-workload-certs\") pod \"maas-default-gateway-openshift-default-596dc9545f-t5bt4\" (UID: \"2b55c151-d235-4bcc-b602-eca0da4a7348\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-t5bt4" Nov 03 00:49:29.222428 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:29.222209 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bb9cl\" (UniqueName: \"kubernetes.io/projected/2b55c151-d235-4bcc-b602-eca0da4a7348-kube-api-access-bb9cl\") pod \"maas-default-gateway-openshift-default-596dc9545f-t5bt4\" (UID: \"2b55c151-d235-4bcc-b602-eca0da4a7348\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-t5bt4" Nov 03 00:49:29.222428 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:29.222238 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b55c151-d235-4bcc-b602-eca0da4a7348-credential-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-t5bt4\" (UID: \"2b55c151-d235-4bcc-b602-eca0da4a7348\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-t5bt4" Nov 03 00:49:29.222428 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:29.222306 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b55c151-d235-4bcc-b602-eca0da4a7348-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-596dc9545f-t5bt4\" (UID: \"2b55c151-d235-4bcc-b602-eca0da4a7348\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-t5bt4" Nov 03 00:49:29.222428 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:29.222358 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b55c151-d235-4bcc-b602-eca0da4a7348-istio-token\") pod \"maas-default-gateway-openshift-default-596dc9545f-t5bt4\" (UID: \"2b55c151-d235-4bcc-b602-eca0da4a7348\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-t5bt4" Nov 03 00:49:29.222428 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:29.222395 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b55c151-d235-4bcc-b602-eca0da4a7348-istio-podinfo\") pod \"maas-default-gateway-openshift-default-596dc9545f-t5bt4\" (UID: \"2b55c151-d235-4bcc-b602-eca0da4a7348\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-t5bt4" Nov 03 00:49:29.222428 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:29.222423 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b55c151-d235-4bcc-b602-eca0da4a7348-istio-data\") pod \"maas-default-gateway-openshift-default-596dc9545f-t5bt4\" (UID: \"2b55c151-d235-4bcc-b602-eca0da4a7348\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-t5bt4" Nov 03 00:49:29.222822 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:29.222455 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b55c151-d235-4bcc-b602-eca0da4a7348-workload-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-t5bt4\" (UID: \"2b55c151-d235-4bcc-b602-eca0da4a7348\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-t5bt4" Nov 03 00:49:29.222822 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:29.222497 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b55c151-d235-4bcc-b602-eca0da4a7348-istio-envoy\") pod \"maas-default-gateway-openshift-default-596dc9545f-t5bt4\" (UID: \"2b55c151-d235-4bcc-b602-eca0da4a7348\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-t5bt4" Nov 03 00:49:29.222822 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:29.222641 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b55c151-d235-4bcc-b602-eca0da4a7348-workload-certs\") pod \"maas-default-gateway-openshift-default-596dc9545f-t5bt4\" (UID: \"2b55c151-d235-4bcc-b602-eca0da4a7348\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-t5bt4" Nov 03 00:49:29.222822 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:29.222687 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b55c151-d235-4bcc-b602-eca0da4a7348-credential-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-t5bt4\" (UID: \"2b55c151-d235-4bcc-b602-eca0da4a7348\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-t5bt4" Nov 03 00:49:29.222993 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:29.222887 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b55c151-d235-4bcc-b602-eca0da4a7348-workload-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-t5bt4\" (UID: \"2b55c151-d235-4bcc-b602-eca0da4a7348\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-t5bt4" Nov 03 00:49:29.222993 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:29.222939 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b55c151-d235-4bcc-b602-eca0da4a7348-istio-data\") pod \"maas-default-gateway-openshift-default-596dc9545f-t5bt4\" (UID: \"2b55c151-d235-4bcc-b602-eca0da4a7348\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-t5bt4" Nov 03 00:49:29.223099 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:29.223081 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b55c151-d235-4bcc-b602-eca0da4a7348-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-596dc9545f-t5bt4\" (UID: \"2b55c151-d235-4bcc-b602-eca0da4a7348\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-t5bt4" Nov 03 00:49:29.224790 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:29.224770 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b55c151-d235-4bcc-b602-eca0da4a7348-istio-envoy\") pod \"maas-default-gateway-openshift-default-596dc9545f-t5bt4\" (UID: \"2b55c151-d235-4bcc-b602-eca0da4a7348\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-t5bt4" Nov 03 00:49:29.224907 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:29.224889 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b55c151-d235-4bcc-b602-eca0da4a7348-istio-podinfo\") pod \"maas-default-gateway-openshift-default-596dc9545f-t5bt4\" (UID: \"2b55c151-d235-4bcc-b602-eca0da4a7348\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-t5bt4" Nov 03 00:49:29.230984 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:29.230949 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b55c151-d235-4bcc-b602-eca0da4a7348-istio-token\") pod \"maas-default-gateway-openshift-default-596dc9545f-t5bt4\" (UID: \"2b55c151-d235-4bcc-b602-eca0da4a7348\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-t5bt4" Nov 03 00:49:29.231087 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:29.231019 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bb9cl\" (UniqueName: \"kubernetes.io/projected/2b55c151-d235-4bcc-b602-eca0da4a7348-kube-api-access-bb9cl\") pod \"maas-default-gateway-openshift-default-596dc9545f-t5bt4\" (UID: \"2b55c151-d235-4bcc-b602-eca0da4a7348\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-t5bt4" Nov 03 00:49:29.341982 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:29.341869 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-t5bt4" Nov 03 00:49:29.620312 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:29.620265 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-t5bt4"] Nov 03 00:49:29.623021 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:49:29.622977 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b55c151_d235_4bcc_b602_eca0da4a7348.slice/crio-63b97dfc18d3a7aaa7edfda6c2ec84004adcafecad96f2c1069981bde452f7ce WatchSource:0}: Error finding container 63b97dfc18d3a7aaa7edfda6c2ec84004adcafecad96f2c1069981bde452f7ce: Status 404 returned error can't find the container with id 63b97dfc18d3a7aaa7edfda6c2ec84004adcafecad96f2c1069981bde452f7ce Nov 03 00:49:29.625390 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:29.625347 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:49:29.625474 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:29.625443 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:49:29.625512 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:29.625493 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:49:30.331613 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:30.331574 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-t5bt4" event={"ID":"2b55c151-d235-4bcc-b602-eca0da4a7348","Type":"ContainerStarted","Data":"28c9237ffae3781146e130dd9c4cd0da86a188f0798d40ea2f29e94c02af4605"} Nov 03 00:49:30.332238 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:30.332208 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-t5bt4" event={"ID":"2b55c151-d235-4bcc-b602-eca0da4a7348","Type":"ContainerStarted","Data":"63b97dfc18d3a7aaa7edfda6c2ec84004adcafecad96f2c1069981bde452f7ce"} Nov 03 00:49:30.342643 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:30.342616 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-t5bt4" Nov 03 00:49:30.344071 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:30.344049 4421 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-t5bt4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Nov 03 00:49:30.344169 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:30.344095 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-t5bt4" podUID="2b55c151-d235-4bcc-b602-eca0da4a7348" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Nov 03 00:49:30.353900 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:30.353854 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-t5bt4" podStartSLOduration=2.35384074 podStartE2EDuration="2.35384074s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:49:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:49:30.352401479 +0000 UTC m=+512.523404594" watchObservedRunningTime="2026-11-03 00:49:30.35384074 +0000 UTC m=+512.524843835" Nov 03 00:49:31.342972 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:31.342933 4421 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-t5bt4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Nov 03 00:49:31.343395 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:31.342990 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-t5bt4" podUID="2b55c151-d235-4bcc-b602-eca0da4a7348" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Nov 03 00:49:32.342712 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:32.342680 4421 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-t5bt4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Nov 03 00:49:32.342898 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:32.342734 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-t5bt4" podUID="2b55c151-d235-4bcc-b602-eca0da4a7348" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Nov 03 00:49:33.343117 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:33.343081 4421 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-t5bt4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Nov 03 00:49:33.343530 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:33.343141 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-t5bt4" podUID="2b55c151-d235-4bcc-b602-eca0da4a7348" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Nov 03 00:49:34.342342 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:34.342298 4421 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-t5bt4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Nov 03 00:49:34.342514 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:34.342357 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-t5bt4" podUID="2b55c151-d235-4bcc-b602-eca0da4a7348" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Nov 03 00:49:35.342462 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:35.342420 4421 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-t5bt4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Nov 03 00:49:35.342903 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:35.342486 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-t5bt4" podUID="2b55c151-d235-4bcc-b602-eca0da4a7348" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Nov 03 00:49:36.342880 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:36.342831 4421 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-t5bt4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Nov 03 00:49:36.343310 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:36.342928 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-t5bt4" podUID="2b55c151-d235-4bcc-b602-eca0da4a7348" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Nov 03 00:49:37.834962 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:37.834930 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-t5bt4" Nov 03 00:49:37.835852 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:37.835828 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-t5bt4" Nov 03 00:49:37.836886 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:37.836865 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-t5bt4" Nov 03 00:49:44.669202 ip-10-0-133-217 kubenswrapper[4421]: E1103 00:49:44.669167 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 00:49:52.692850 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:52.692809 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-e9b9924e-gw-openshift-default-58b595c4df-xtlrf"] Nov 03 00:49:52.697035 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:52.697013 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-e9b9924e-gw-openshift-default-58b595c4df-xtlrf" Nov 03 00:49:52.701909 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:52.701883 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-e9b9924e-gw-openshift-default-dockercfg-5w69v\"" Nov 03 00:49:52.709687 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:52.709662 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-e9b9924e-gw-openshift-default-58b595c4df-xtlrf"] Nov 03 00:49:52.762868 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:52.762831 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/282b2fbe-dcda-488c-90df-3d087cc51a46-istio-podinfo\") pod \"e2e-migrate-e9b9924e-gw-openshift-default-58b595c4df-xtlrf\" (UID: \"282b2fbe-dcda-488c-90df-3d087cc51a46\") " pod="openshift-ingress/e2e-migrate-e9b9924e-gw-openshift-default-58b595c4df-xtlrf" Nov 03 00:49:52.763044 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:52.762871 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/282b2fbe-dcda-488c-90df-3d087cc51a46-istiod-ca-cert\") pod \"e2e-migrate-e9b9924e-gw-openshift-default-58b595c4df-xtlrf\" (UID: \"282b2fbe-dcda-488c-90df-3d087cc51a46\") " pod="openshift-ingress/e2e-migrate-e9b9924e-gw-openshift-default-58b595c4df-xtlrf" Nov 03 00:49:52.763044 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:52.762952 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/282b2fbe-dcda-488c-90df-3d087cc51a46-workload-certs\") pod \"e2e-migrate-e9b9924e-gw-openshift-default-58b595c4df-xtlrf\" (UID: \"282b2fbe-dcda-488c-90df-3d087cc51a46\") " pod="openshift-ingress/e2e-migrate-e9b9924e-gw-openshift-default-58b595c4df-xtlrf" Nov 03 00:49:52.763044 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:52.763007 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/282b2fbe-dcda-488c-90df-3d087cc51a46-workload-socket\") pod \"e2e-migrate-e9b9924e-gw-openshift-default-58b595c4df-xtlrf\" (UID: \"282b2fbe-dcda-488c-90df-3d087cc51a46\") " pod="openshift-ingress/e2e-migrate-e9b9924e-gw-openshift-default-58b595c4df-xtlrf" Nov 03 00:49:52.763218 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:52.763118 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/282b2fbe-dcda-488c-90df-3d087cc51a46-istio-data\") pod \"e2e-migrate-e9b9924e-gw-openshift-default-58b595c4df-xtlrf\" (UID: \"282b2fbe-dcda-488c-90df-3d087cc51a46\") " pod="openshift-ingress/e2e-migrate-e9b9924e-gw-openshift-default-58b595c4df-xtlrf" Nov 03 00:49:52.763218 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:52.763156 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/282b2fbe-dcda-488c-90df-3d087cc51a46-istio-envoy\") pod \"e2e-migrate-e9b9924e-gw-openshift-default-58b595c4df-xtlrf\" (UID: \"282b2fbe-dcda-488c-90df-3d087cc51a46\") " pod="openshift-ingress/e2e-migrate-e9b9924e-gw-openshift-default-58b595c4df-xtlrf" Nov 03 00:49:52.763369 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:52.763237 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqwnc\" (UniqueName: \"kubernetes.io/projected/282b2fbe-dcda-488c-90df-3d087cc51a46-kube-api-access-vqwnc\") pod \"e2e-migrate-e9b9924e-gw-openshift-default-58b595c4df-xtlrf\" (UID: \"282b2fbe-dcda-488c-90df-3d087cc51a46\") " pod="openshift-ingress/e2e-migrate-e9b9924e-gw-openshift-default-58b595c4df-xtlrf" Nov 03 00:49:52.763369 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:52.763325 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/282b2fbe-dcda-488c-90df-3d087cc51a46-istio-token\") pod \"e2e-migrate-e9b9924e-gw-openshift-default-58b595c4df-xtlrf\" (UID: \"282b2fbe-dcda-488c-90df-3d087cc51a46\") " pod="openshift-ingress/e2e-migrate-e9b9924e-gw-openshift-default-58b595c4df-xtlrf" Nov 03 00:49:52.763369 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:52.763362 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/282b2fbe-dcda-488c-90df-3d087cc51a46-credential-socket\") pod \"e2e-migrate-e9b9924e-gw-openshift-default-58b595c4df-xtlrf\" (UID: \"282b2fbe-dcda-488c-90df-3d087cc51a46\") " pod="openshift-ingress/e2e-migrate-e9b9924e-gw-openshift-default-58b595c4df-xtlrf" Nov 03 00:49:52.864213 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:52.864170 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/282b2fbe-dcda-488c-90df-3d087cc51a46-istio-podinfo\") pod \"e2e-migrate-e9b9924e-gw-openshift-default-58b595c4df-xtlrf\" (UID: \"282b2fbe-dcda-488c-90df-3d087cc51a46\") " pod="openshift-ingress/e2e-migrate-e9b9924e-gw-openshift-default-58b595c4df-xtlrf" Nov 03 00:49:52.864421 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:52.864225 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/282b2fbe-dcda-488c-90df-3d087cc51a46-istiod-ca-cert\") pod \"e2e-migrate-e9b9924e-gw-openshift-default-58b595c4df-xtlrf\" (UID: \"282b2fbe-dcda-488c-90df-3d087cc51a46\") " pod="openshift-ingress/e2e-migrate-e9b9924e-gw-openshift-default-58b595c4df-xtlrf" Nov 03 00:49:52.864421 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:52.864257 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/282b2fbe-dcda-488c-90df-3d087cc51a46-workload-certs\") pod \"e2e-migrate-e9b9924e-gw-openshift-default-58b595c4df-xtlrf\" (UID: \"282b2fbe-dcda-488c-90df-3d087cc51a46\") " pod="openshift-ingress/e2e-migrate-e9b9924e-gw-openshift-default-58b595c4df-xtlrf" Nov 03 00:49:52.864421 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:52.864322 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/282b2fbe-dcda-488c-90df-3d087cc51a46-workload-socket\") pod \"e2e-migrate-e9b9924e-gw-openshift-default-58b595c4df-xtlrf\" (UID: \"282b2fbe-dcda-488c-90df-3d087cc51a46\") " pod="openshift-ingress/e2e-migrate-e9b9924e-gw-openshift-default-58b595c4df-xtlrf" Nov 03 00:49:52.864421 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:52.864375 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/282b2fbe-dcda-488c-90df-3d087cc51a46-istio-data\") pod \"e2e-migrate-e9b9924e-gw-openshift-default-58b595c4df-xtlrf\" (UID: \"282b2fbe-dcda-488c-90df-3d087cc51a46\") " pod="openshift-ingress/e2e-migrate-e9b9924e-gw-openshift-default-58b595c4df-xtlrf" Nov 03 00:49:52.864640 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:52.864480 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/282b2fbe-dcda-488c-90df-3d087cc51a46-istio-envoy\") pod \"e2e-migrate-e9b9924e-gw-openshift-default-58b595c4df-xtlrf\" (UID: \"282b2fbe-dcda-488c-90df-3d087cc51a46\") " pod="openshift-ingress/e2e-migrate-e9b9924e-gw-openshift-default-58b595c4df-xtlrf" Nov 03 00:49:52.864640 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:52.864554 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqwnc\" (UniqueName: \"kubernetes.io/projected/282b2fbe-dcda-488c-90df-3d087cc51a46-kube-api-access-vqwnc\") pod \"e2e-migrate-e9b9924e-gw-openshift-default-58b595c4df-xtlrf\" (UID: \"282b2fbe-dcda-488c-90df-3d087cc51a46\") " pod="openshift-ingress/e2e-migrate-e9b9924e-gw-openshift-default-58b595c4df-xtlrf" Nov 03 00:49:52.864640 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:52.864587 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/282b2fbe-dcda-488c-90df-3d087cc51a46-istio-token\") pod \"e2e-migrate-e9b9924e-gw-openshift-default-58b595c4df-xtlrf\" (UID: \"282b2fbe-dcda-488c-90df-3d087cc51a46\") " pod="openshift-ingress/e2e-migrate-e9b9924e-gw-openshift-default-58b595c4df-xtlrf" Nov 03 00:49:52.864640 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:52.864617 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/282b2fbe-dcda-488c-90df-3d087cc51a46-credential-socket\") pod \"e2e-migrate-e9b9924e-gw-openshift-default-58b595c4df-xtlrf\" (UID: \"282b2fbe-dcda-488c-90df-3d087cc51a46\") " pod="openshift-ingress/e2e-migrate-e9b9924e-gw-openshift-default-58b595c4df-xtlrf" Nov 03 00:49:52.865132 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:52.865108 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/282b2fbe-dcda-488c-90df-3d087cc51a46-istiod-ca-cert\") pod \"e2e-migrate-e9b9924e-gw-openshift-default-58b595c4df-xtlrf\" (UID: \"282b2fbe-dcda-488c-90df-3d087cc51a46\") " pod="openshift-ingress/e2e-migrate-e9b9924e-gw-openshift-default-58b595c4df-xtlrf" Nov 03 00:49:52.865476 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:52.865451 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/282b2fbe-dcda-488c-90df-3d087cc51a46-workload-certs\") pod \"e2e-migrate-e9b9924e-gw-openshift-default-58b595c4df-xtlrf\" (UID: \"282b2fbe-dcda-488c-90df-3d087cc51a46\") " pod="openshift-ingress/e2e-migrate-e9b9924e-gw-openshift-default-58b595c4df-xtlrf" Nov 03 00:49:52.865563 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:52.865519 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/282b2fbe-dcda-488c-90df-3d087cc51a46-istio-data\") pod \"e2e-migrate-e9b9924e-gw-openshift-default-58b595c4df-xtlrf\" (UID: \"282b2fbe-dcda-488c-90df-3d087cc51a46\") " pod="openshift-ingress/e2e-migrate-e9b9924e-gw-openshift-default-58b595c4df-xtlrf" Nov 03 00:49:52.865616 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:52.865558 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/282b2fbe-dcda-488c-90df-3d087cc51a46-credential-socket\") pod \"e2e-migrate-e9b9924e-gw-openshift-default-58b595c4df-xtlrf\" (UID: \"282b2fbe-dcda-488c-90df-3d087cc51a46\") " pod="openshift-ingress/e2e-migrate-e9b9924e-gw-openshift-default-58b595c4df-xtlrf" Nov 03 00:49:52.865786 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:52.865763 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/282b2fbe-dcda-488c-90df-3d087cc51a46-workload-socket\") pod \"e2e-migrate-e9b9924e-gw-openshift-default-58b595c4df-xtlrf\" (UID: \"282b2fbe-dcda-488c-90df-3d087cc51a46\") " pod="openshift-ingress/e2e-migrate-e9b9924e-gw-openshift-default-58b595c4df-xtlrf" Nov 03 00:49:52.866978 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:52.866953 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/282b2fbe-dcda-488c-90df-3d087cc51a46-istio-podinfo\") pod \"e2e-migrate-e9b9924e-gw-openshift-default-58b595c4df-xtlrf\" (UID: \"282b2fbe-dcda-488c-90df-3d087cc51a46\") " pod="openshift-ingress/e2e-migrate-e9b9924e-gw-openshift-default-58b595c4df-xtlrf" Nov 03 00:49:52.867759 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:52.867732 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/282b2fbe-dcda-488c-90df-3d087cc51a46-istio-envoy\") pod \"e2e-migrate-e9b9924e-gw-openshift-default-58b595c4df-xtlrf\" (UID: \"282b2fbe-dcda-488c-90df-3d087cc51a46\") " pod="openshift-ingress/e2e-migrate-e9b9924e-gw-openshift-default-58b595c4df-xtlrf" Nov 03 00:49:52.872775 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:52.872749 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/282b2fbe-dcda-488c-90df-3d087cc51a46-istio-token\") pod \"e2e-migrate-e9b9924e-gw-openshift-default-58b595c4df-xtlrf\" (UID: \"282b2fbe-dcda-488c-90df-3d087cc51a46\") " pod="openshift-ingress/e2e-migrate-e9b9924e-gw-openshift-default-58b595c4df-xtlrf" Nov 03 00:49:52.873126 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:52.873103 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqwnc\" (UniqueName: \"kubernetes.io/projected/282b2fbe-dcda-488c-90df-3d087cc51a46-kube-api-access-vqwnc\") pod \"e2e-migrate-e9b9924e-gw-openshift-default-58b595c4df-xtlrf\" (UID: \"282b2fbe-dcda-488c-90df-3d087cc51a46\") " pod="openshift-ingress/e2e-migrate-e9b9924e-gw-openshift-default-58b595c4df-xtlrf" Nov 03 00:49:53.010481 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:53.010438 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-e9b9924e-gw-openshift-default-58b595c4df-xtlrf" Nov 03 00:49:53.289601 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:53.289571 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-e9b9924e-gw-openshift-default-58b595c4df-xtlrf"] Nov 03 00:49:53.292083 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:49:53.292052 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod282b2fbe_dcda_488c_90df_3d087cc51a46.slice/crio-e5038c98994f536848cca99b0cd431daa0b1411dbb50eaf7672d961958406c90 WatchSource:0}: Error finding container e5038c98994f536848cca99b0cd431daa0b1411dbb50eaf7672d961958406c90: Status 404 returned error can't find the container with id e5038c98994f536848cca99b0cd431daa0b1411dbb50eaf7672d961958406c90 Nov 03 00:49:53.294818 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:53.294768 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:49:53.294907 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:53.294861 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:49:53.294946 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:53.294904 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:49:53.438022 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:53.437977 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-e9b9924e-gw-openshift-default-58b595c4df-xtlrf" event={"ID":"282b2fbe-dcda-488c-90df-3d087cc51a46","Type":"ContainerStarted","Data":"88b8332b90873e266f07d615d4357cca294363854473c58117db245af97d5c26"} Nov 03 00:49:53.438221 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:53.438041 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-e9b9924e-gw-openshift-default-58b595c4df-xtlrf" event={"ID":"282b2fbe-dcda-488c-90df-3d087cc51a46","Type":"ContainerStarted","Data":"e5038c98994f536848cca99b0cd431daa0b1411dbb50eaf7672d961958406c90"} Nov 03 00:49:54.011032 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:54.010998 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-e9b9924e-gw-openshift-default-58b595c4df-xtlrf" Nov 03 00:49:54.016502 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:54.016474 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-e9b9924e-gw-openshift-default-58b595c4df-xtlrf" Nov 03 00:49:54.047743 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:54.047690 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-e9b9924e-gw-openshift-default-58b595c4df-xtlrf" podStartSLOduration=2.047675156 podStartE2EDuration="2.047675156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:49:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:49:53.464954606 +0000 UTC m=+535.635957702" watchObservedRunningTime="2026-11-03 00:49:54.047675156 +0000 UTC m=+536.218678253" Nov 03 00:49:54.068542 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:54.068506 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-c83251ee-openshift-default-5567cd685c-69vg9"] Nov 03 00:49:54.072293 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:54.072252 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c83251ee-openshift-default-5567cd685c-69vg9" Nov 03 00:49:54.081769 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:54.081745 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-c83251ee-openshift-default-dockercfg-5sm8t\"" Nov 03 00:49:54.087225 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:54.087200 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-c83251ee-openshift-default-5567cd685c-69vg9"] Nov 03 00:49:54.176818 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:54.176785 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d793f1a-5d2f-4435-aee8-38af07f46693-credential-socket\") pod \"e2e-ait-c83251ee-openshift-default-5567cd685c-69vg9\" (UID: \"4d793f1a-5d2f-4435-aee8-38af07f46693\") " pod="openshift-ingress/e2e-ait-c83251ee-openshift-default-5567cd685c-69vg9" Nov 03 00:49:54.177032 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:54.176834 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d793f1a-5d2f-4435-aee8-38af07f46693-workload-certs\") pod \"e2e-ait-c83251ee-openshift-default-5567cd685c-69vg9\" (UID: \"4d793f1a-5d2f-4435-aee8-38af07f46693\") " pod="openshift-ingress/e2e-ait-c83251ee-openshift-default-5567cd685c-69vg9" Nov 03 00:49:54.177032 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:54.176992 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d793f1a-5d2f-4435-aee8-38af07f46693-istio-podinfo\") pod \"e2e-ait-c83251ee-openshift-default-5567cd685c-69vg9\" (UID: \"4d793f1a-5d2f-4435-aee8-38af07f46693\") " pod="openshift-ingress/e2e-ait-c83251ee-openshift-default-5567cd685c-69vg9" Nov 03 00:49:54.177032 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:54.177018 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d793f1a-5d2f-4435-aee8-38af07f46693-istiod-ca-cert\") pod \"e2e-ait-c83251ee-openshift-default-5567cd685c-69vg9\" (UID: \"4d793f1a-5d2f-4435-aee8-38af07f46693\") " pod="openshift-ingress/e2e-ait-c83251ee-openshift-default-5567cd685c-69vg9" Nov 03 00:49:54.177202 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:54.177054 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9fcc\" (UniqueName: \"kubernetes.io/projected/4d793f1a-5d2f-4435-aee8-38af07f46693-kube-api-access-b9fcc\") pod \"e2e-ait-c83251ee-openshift-default-5567cd685c-69vg9\" (UID: \"4d793f1a-5d2f-4435-aee8-38af07f46693\") " pod="openshift-ingress/e2e-ait-c83251ee-openshift-default-5567cd685c-69vg9" Nov 03 00:49:54.177202 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:54.177125 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d793f1a-5d2f-4435-aee8-38af07f46693-istio-envoy\") pod \"e2e-ait-c83251ee-openshift-default-5567cd685c-69vg9\" (UID: \"4d793f1a-5d2f-4435-aee8-38af07f46693\") " pod="openshift-ingress/e2e-ait-c83251ee-openshift-default-5567cd685c-69vg9" Nov 03 00:49:54.177202 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:54.177150 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d793f1a-5d2f-4435-aee8-38af07f46693-istio-data\") pod \"e2e-ait-c83251ee-openshift-default-5567cd685c-69vg9\" (UID: \"4d793f1a-5d2f-4435-aee8-38af07f46693\") " pod="openshift-ingress/e2e-ait-c83251ee-openshift-default-5567cd685c-69vg9" Nov 03 00:49:54.177202 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:54.177184 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d793f1a-5d2f-4435-aee8-38af07f46693-workload-socket\") pod \"e2e-ait-c83251ee-openshift-default-5567cd685c-69vg9\" (UID: \"4d793f1a-5d2f-4435-aee8-38af07f46693\") " pod="openshift-ingress/e2e-ait-c83251ee-openshift-default-5567cd685c-69vg9" Nov 03 00:49:54.177452 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:54.177225 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d793f1a-5d2f-4435-aee8-38af07f46693-istio-token\") pod \"e2e-ait-c83251ee-openshift-default-5567cd685c-69vg9\" (UID: \"4d793f1a-5d2f-4435-aee8-38af07f46693\") " pod="openshift-ingress/e2e-ait-c83251ee-openshift-default-5567cd685c-69vg9" Nov 03 00:49:54.277928 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:54.277833 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d793f1a-5d2f-4435-aee8-38af07f46693-credential-socket\") pod \"e2e-ait-c83251ee-openshift-default-5567cd685c-69vg9\" (UID: \"4d793f1a-5d2f-4435-aee8-38af07f46693\") " pod="openshift-ingress/e2e-ait-c83251ee-openshift-default-5567cd685c-69vg9" Nov 03 00:49:54.277928 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:54.277876 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d793f1a-5d2f-4435-aee8-38af07f46693-workload-certs\") pod \"e2e-ait-c83251ee-openshift-default-5567cd685c-69vg9\" (UID: \"4d793f1a-5d2f-4435-aee8-38af07f46693\") " pod="openshift-ingress/e2e-ait-c83251ee-openshift-default-5567cd685c-69vg9" Nov 03 00:49:54.277928 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:54.277921 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d793f1a-5d2f-4435-aee8-38af07f46693-istio-podinfo\") pod \"e2e-ait-c83251ee-openshift-default-5567cd685c-69vg9\" (UID: \"4d793f1a-5d2f-4435-aee8-38af07f46693\") " pod="openshift-ingress/e2e-ait-c83251ee-openshift-default-5567cd685c-69vg9" Nov 03 00:49:54.278209 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:54.277944 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d793f1a-5d2f-4435-aee8-38af07f46693-istiod-ca-cert\") pod \"e2e-ait-c83251ee-openshift-default-5567cd685c-69vg9\" (UID: \"4d793f1a-5d2f-4435-aee8-38af07f46693\") " pod="openshift-ingress/e2e-ait-c83251ee-openshift-default-5567cd685c-69vg9" Nov 03 00:49:54.278209 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:54.277971 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9fcc\" (UniqueName: \"kubernetes.io/projected/4d793f1a-5d2f-4435-aee8-38af07f46693-kube-api-access-b9fcc\") pod \"e2e-ait-c83251ee-openshift-default-5567cd685c-69vg9\" (UID: \"4d793f1a-5d2f-4435-aee8-38af07f46693\") " pod="openshift-ingress/e2e-ait-c83251ee-openshift-default-5567cd685c-69vg9" Nov 03 00:49:54.278209 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:54.278006 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d793f1a-5d2f-4435-aee8-38af07f46693-istio-envoy\") pod \"e2e-ait-c83251ee-openshift-default-5567cd685c-69vg9\" (UID: \"4d793f1a-5d2f-4435-aee8-38af07f46693\") " pod="openshift-ingress/e2e-ait-c83251ee-openshift-default-5567cd685c-69vg9" Nov 03 00:49:54.278209 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:54.278039 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d793f1a-5d2f-4435-aee8-38af07f46693-istio-data\") pod \"e2e-ait-c83251ee-openshift-default-5567cd685c-69vg9\" (UID: \"4d793f1a-5d2f-4435-aee8-38af07f46693\") " pod="openshift-ingress/e2e-ait-c83251ee-openshift-default-5567cd685c-69vg9" Nov 03 00:49:54.278209 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:54.278088 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d793f1a-5d2f-4435-aee8-38af07f46693-workload-socket\") pod \"e2e-ait-c83251ee-openshift-default-5567cd685c-69vg9\" (UID: \"4d793f1a-5d2f-4435-aee8-38af07f46693\") " pod="openshift-ingress/e2e-ait-c83251ee-openshift-default-5567cd685c-69vg9" Nov 03 00:49:54.278209 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:54.278122 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d793f1a-5d2f-4435-aee8-38af07f46693-istio-token\") pod \"e2e-ait-c83251ee-openshift-default-5567cd685c-69vg9\" (UID: \"4d793f1a-5d2f-4435-aee8-38af07f46693\") " pod="openshift-ingress/e2e-ait-c83251ee-openshift-default-5567cd685c-69vg9" Nov 03 00:49:54.278540 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:54.278516 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d793f1a-5d2f-4435-aee8-38af07f46693-credential-socket\") pod \"e2e-ait-c83251ee-openshift-default-5567cd685c-69vg9\" (UID: \"4d793f1a-5d2f-4435-aee8-38af07f46693\") " pod="openshift-ingress/e2e-ait-c83251ee-openshift-default-5567cd685c-69vg9" Nov 03 00:49:54.278688 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:54.278665 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d793f1a-5d2f-4435-aee8-38af07f46693-workload-socket\") pod \"e2e-ait-c83251ee-openshift-default-5567cd685c-69vg9\" (UID: \"4d793f1a-5d2f-4435-aee8-38af07f46693\") " pod="openshift-ingress/e2e-ait-c83251ee-openshift-default-5567cd685c-69vg9" Nov 03 00:49:54.278753 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:54.278719 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d793f1a-5d2f-4435-aee8-38af07f46693-istio-data\") pod \"e2e-ait-c83251ee-openshift-default-5567cd685c-69vg9\" (UID: \"4d793f1a-5d2f-4435-aee8-38af07f46693\") " pod="openshift-ingress/e2e-ait-c83251ee-openshift-default-5567cd685c-69vg9" Nov 03 00:49:54.278806 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:54.278746 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d793f1a-5d2f-4435-aee8-38af07f46693-workload-certs\") pod \"e2e-ait-c83251ee-openshift-default-5567cd685c-69vg9\" (UID: \"4d793f1a-5d2f-4435-aee8-38af07f46693\") " pod="openshift-ingress/e2e-ait-c83251ee-openshift-default-5567cd685c-69vg9" Nov 03 00:49:54.278806 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:54.278775 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d793f1a-5d2f-4435-aee8-38af07f46693-istiod-ca-cert\") pod \"e2e-ait-c83251ee-openshift-default-5567cd685c-69vg9\" (UID: \"4d793f1a-5d2f-4435-aee8-38af07f46693\") " pod="openshift-ingress/e2e-ait-c83251ee-openshift-default-5567cd685c-69vg9" Nov 03 00:49:54.280433 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:54.280413 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d793f1a-5d2f-4435-aee8-38af07f46693-istio-envoy\") pod \"e2e-ait-c83251ee-openshift-default-5567cd685c-69vg9\" (UID: \"4d793f1a-5d2f-4435-aee8-38af07f46693\") " pod="openshift-ingress/e2e-ait-c83251ee-openshift-default-5567cd685c-69vg9" Nov 03 00:49:54.280726 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:54.280704 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d793f1a-5d2f-4435-aee8-38af07f46693-istio-podinfo\") pod \"e2e-ait-c83251ee-openshift-default-5567cd685c-69vg9\" (UID: \"4d793f1a-5d2f-4435-aee8-38af07f46693\") " pod="openshift-ingress/e2e-ait-c83251ee-openshift-default-5567cd685c-69vg9" Nov 03 00:49:54.286308 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:54.286267 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9fcc\" (UniqueName: \"kubernetes.io/projected/4d793f1a-5d2f-4435-aee8-38af07f46693-kube-api-access-b9fcc\") pod \"e2e-ait-c83251ee-openshift-default-5567cd685c-69vg9\" (UID: \"4d793f1a-5d2f-4435-aee8-38af07f46693\") " pod="openshift-ingress/e2e-ait-c83251ee-openshift-default-5567cd685c-69vg9" Nov 03 00:49:54.286420 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:54.286395 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d793f1a-5d2f-4435-aee8-38af07f46693-istio-token\") pod \"e2e-ait-c83251ee-openshift-default-5567cd685c-69vg9\" (UID: \"4d793f1a-5d2f-4435-aee8-38af07f46693\") " pod="openshift-ingress/e2e-ait-c83251ee-openshift-default-5567cd685c-69vg9" Nov 03 00:49:54.385976 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:54.385938 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c83251ee-openshift-default-5567cd685c-69vg9" Nov 03 00:49:54.445538 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:54.445509 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-e9b9924e-gw-openshift-default-58b595c4df-xtlrf" Nov 03 00:49:54.453304 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:54.453260 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-e9b9924e-gw-openshift-default-58b595c4df-xtlrf" Nov 03 00:49:54.674573 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:54.674490 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-c83251ee-openshift-default-5567cd685c-69vg9"] Nov 03 00:49:54.679981 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:49:54.679952 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d793f1a_5d2f_4435_aee8_38af07f46693.slice/crio-a2fd23a31d7ea91e15e0ca2e56563a1b0325cfe7e786807e2afa969f71c7c1ac WatchSource:0}: Error finding container a2fd23a31d7ea91e15e0ca2e56563a1b0325cfe7e786807e2afa969f71c7c1ac: Status 404 returned error can't find the container with id a2fd23a31d7ea91e15e0ca2e56563a1b0325cfe7e786807e2afa969f71c7c1ac Nov 03 00:49:54.682236 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:54.682206 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:49:54.682342 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:54.682285 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:49:54.682342 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:54.682323 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:49:55.456775 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:55.456733 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c83251ee-openshift-default-5567cd685c-69vg9" event={"ID":"4d793f1a-5d2f-4435-aee8-38af07f46693","Type":"ContainerStarted","Data":"8d67ac4ad544f7c7b7b090c212723c6d805e6eff70c4eca87ea2efcc05237760"} Nov 03 00:49:55.457259 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:55.456818 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c83251ee-openshift-default-5567cd685c-69vg9" event={"ID":"4d793f1a-5d2f-4435-aee8-38af07f46693","Type":"ContainerStarted","Data":"a2fd23a31d7ea91e15e0ca2e56563a1b0325cfe7e786807e2afa969f71c7c1ac"} Nov 03 00:49:55.494184 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:55.494102 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-c83251ee-openshift-default-5567cd685c-69vg9" podStartSLOduration=1.4940894820000001 podStartE2EDuration="1.494089482s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:49:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:49:55.492479718 +0000 UTC m=+537.663482805" watchObservedRunningTime="2026-11-03 00:49:55.494089482 +0000 UTC m=+537.665092577" Nov 03 00:49:56.386637 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:56.386600 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-c83251ee-openshift-default-5567cd685c-69vg9" Nov 03 00:49:56.391754 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:56.391727 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-c83251ee-openshift-default-5567cd685c-69vg9" Nov 03 00:49:56.462347 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:56.462309 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-c83251ee-openshift-default-5567cd685c-69vg9" Nov 03 00:49:56.463212 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:56.463192 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-c83251ee-openshift-default-5567cd685c-69vg9" Nov 03 00:49:56.961498 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:56.961461 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-331bcaa1-589958875f-gsq9q"] Nov 03 00:49:56.965221 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:56.965199 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-331bcaa1-589958875f-gsq9q" Nov 03 00:49:56.972004 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:56.971976 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-331bcaa1\"" Nov 03 00:49:56.981782 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:56.981715 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-331bcaa1-589958875f-gsq9q"] Nov 03 00:49:57.003524 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:57.003491 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-331bcaa1-846ff86f65-qqczq"] Nov 03 00:49:57.005181 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:57.004742 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-331bcaa1-dockercfg-484z6\"" Nov 03 00:49:57.007453 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:57.007434 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-331bcaa1-846ff86f65-qqczq" Nov 03 00:49:57.019431 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:57.019368 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-331bcaa1-846ff86f65-qqczq"] Nov 03 00:49:57.106771 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:57.106732 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7mxn\" (UniqueName: \"kubernetes.io/projected/69476020-213f-41b5-9bac-62655d2a4a53-kube-api-access-l7mxn\") pod \"payload-pre-processing-e2e-ait-331bcaa1-589958875f-gsq9q\" (UID: \"69476020-213f-41b5-9bac-62655d2a4a53\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-331bcaa1-589958875f-gsq9q" Nov 03 00:49:57.106954 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:57.106778 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vk4sf\" (UniqueName: \"kubernetes.io/projected/5a139387-07a4-4f16-86a3-3992d14c79b4-kube-api-access-vk4sf\") pod \"payload-processing-e2e-ait-331bcaa1-846ff86f65-qqczq\" (UID: \"5a139387-07a4-4f16-86a3-3992d14c79b4\") " pod="openshift-ingress/payload-processing-e2e-ait-331bcaa1-846ff86f65-qqczq" Nov 03 00:49:57.106954 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:57.106798 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a139387-07a4-4f16-86a3-3992d14c79b4-plugins-config-volume\") pod \"payload-processing-e2e-ait-331bcaa1-846ff86f65-qqczq\" (UID: \"5a139387-07a4-4f16-86a3-3992d14c79b4\") " pod="openshift-ingress/payload-processing-e2e-ait-331bcaa1-846ff86f65-qqczq" Nov 03 00:49:57.107053 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:57.106957 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69476020-213f-41b5-9bac-62655d2a4a53-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-331bcaa1-589958875f-gsq9q\" (UID: \"69476020-213f-41b5-9bac-62655d2a4a53\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-331bcaa1-589958875f-gsq9q" Nov 03 00:49:57.208723 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:57.208684 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69476020-213f-41b5-9bac-62655d2a4a53-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-331bcaa1-589958875f-gsq9q\" (UID: \"69476020-213f-41b5-9bac-62655d2a4a53\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-331bcaa1-589958875f-gsq9q" Nov 03 00:49:57.208949 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:57.208830 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7mxn\" (UniqueName: \"kubernetes.io/projected/69476020-213f-41b5-9bac-62655d2a4a53-kube-api-access-l7mxn\") pod \"payload-pre-processing-e2e-ait-331bcaa1-589958875f-gsq9q\" (UID: \"69476020-213f-41b5-9bac-62655d2a4a53\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-331bcaa1-589958875f-gsq9q" Nov 03 00:49:57.208949 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:57.208879 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vk4sf\" (UniqueName: \"kubernetes.io/projected/5a139387-07a4-4f16-86a3-3992d14c79b4-kube-api-access-vk4sf\") pod \"payload-processing-e2e-ait-331bcaa1-846ff86f65-qqczq\" (UID: \"5a139387-07a4-4f16-86a3-3992d14c79b4\") " pod="openshift-ingress/payload-processing-e2e-ait-331bcaa1-846ff86f65-qqczq" Nov 03 00:49:57.208949 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:57.208903 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a139387-07a4-4f16-86a3-3992d14c79b4-plugins-config-volume\") pod \"payload-processing-e2e-ait-331bcaa1-846ff86f65-qqczq\" (UID: \"5a139387-07a4-4f16-86a3-3992d14c79b4\") " pod="openshift-ingress/payload-processing-e2e-ait-331bcaa1-846ff86f65-qqczq" Nov 03 00:49:57.209596 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:57.209570 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69476020-213f-41b5-9bac-62655d2a4a53-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-331bcaa1-589958875f-gsq9q\" (UID: \"69476020-213f-41b5-9bac-62655d2a4a53\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-331bcaa1-589958875f-gsq9q" Nov 03 00:49:57.209801 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:57.209780 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a139387-07a4-4f16-86a3-3992d14c79b4-plugins-config-volume\") pod \"payload-processing-e2e-ait-331bcaa1-846ff86f65-qqczq\" (UID: \"5a139387-07a4-4f16-86a3-3992d14c79b4\") " pod="openshift-ingress/payload-processing-e2e-ait-331bcaa1-846ff86f65-qqczq" Nov 03 00:49:57.217661 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:57.217592 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vk4sf\" (UniqueName: \"kubernetes.io/projected/5a139387-07a4-4f16-86a3-3992d14c79b4-kube-api-access-vk4sf\") pod \"payload-processing-e2e-ait-331bcaa1-846ff86f65-qqczq\" (UID: \"5a139387-07a4-4f16-86a3-3992d14c79b4\") " pod="openshift-ingress/payload-processing-e2e-ait-331bcaa1-846ff86f65-qqczq" Nov 03 00:49:57.217661 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:57.217609 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7mxn\" (UniqueName: \"kubernetes.io/projected/69476020-213f-41b5-9bac-62655d2a4a53-kube-api-access-l7mxn\") pod \"payload-pre-processing-e2e-ait-331bcaa1-589958875f-gsq9q\" (UID: \"69476020-213f-41b5-9bac-62655d2a4a53\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-331bcaa1-589958875f-gsq9q" Nov 03 00:49:57.279133 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:57.279093 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-331bcaa1-589958875f-gsq9q" Nov 03 00:49:57.321262 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:57.321226 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-331bcaa1-846ff86f65-qqczq" Nov 03 00:49:57.566713 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:57.566679 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-15abbf02-6fbdf5668c-r7klb"] Nov 03 00:49:57.571986 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:57.571564 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-15abbf02-6fbdf5668c-r7klb" Nov 03 00:49:57.574411 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:57.574384 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-331bcaa1-589958875f-gsq9q"] Nov 03 00:49:57.576109 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:49:57.576072 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69476020_213f_41b5_9bac_62655d2a4a53.slice/crio-e296bfedef9bb2a59984478fdf08a1d8a7583c1316f35c3d0a52325be8fcc5b5 WatchSource:0}: Error finding container e296bfedef9bb2a59984478fdf08a1d8a7583c1316f35c3d0a52325be8fcc5b5: Status 404 returned error can't find the container with id e296bfedef9bb2a59984478fdf08a1d8a7583c1316f35c3d0a52325be8fcc5b5 Nov 03 00:49:57.579196 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:57.579070 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-15abbf02-dockercfg-tzdsv\"" Nov 03 00:49:57.590905 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:57.590870 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-15abbf02-6fbdf5668c-r7klb"] Nov 03 00:49:57.618540 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:57.618511 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-331bcaa1-846ff86f65-qqczq"] Nov 03 00:49:57.620042 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:49:57.620014 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a139387_07a4_4f16_86a3_3992d14c79b4.slice/crio-c29eb23e3964910cc95ee7e53ae3a9abae605f63835586597f1bd9a816e46101 WatchSource:0}: Error finding container c29eb23e3964910cc95ee7e53ae3a9abae605f63835586597f1bd9a816e46101: Status 404 returned error can't find the container with id c29eb23e3964910cc95ee7e53ae3a9abae605f63835586597f1bd9a816e46101 Nov 03 00:49:57.641887 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:57.641848 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-15abbf02\"" Nov 03 00:49:57.714479 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:57.714427 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46553f3d-e4a6-4ef4-a601-c7c01b9f057e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-15abbf02-6fbdf5668c-r7klb\" (UID: \"46553f3d-e4a6-4ef4-a601-c7c01b9f057e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-15abbf02-6fbdf5668c-r7klb" Nov 03 00:49:57.714479 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:57.714482 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbkmf\" (UniqueName: \"kubernetes.io/projected/46553f3d-e4a6-4ef4-a601-c7c01b9f057e-kube-api-access-zbkmf\") pod \"payload-pre-processing-e2e-ait-15abbf02-6fbdf5668c-r7klb\" (UID: \"46553f3d-e4a6-4ef4-a601-c7c01b9f057e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-15abbf02-6fbdf5668c-r7klb" Nov 03 00:49:57.815980 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:57.815943 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46553f3d-e4a6-4ef4-a601-c7c01b9f057e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-15abbf02-6fbdf5668c-r7klb\" (UID: \"46553f3d-e4a6-4ef4-a601-c7c01b9f057e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-15abbf02-6fbdf5668c-r7klb" Nov 03 00:49:57.815980 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:57.815985 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbkmf\" (UniqueName: \"kubernetes.io/projected/46553f3d-e4a6-4ef4-a601-c7c01b9f057e-kube-api-access-zbkmf\") pod \"payload-pre-processing-e2e-ait-15abbf02-6fbdf5668c-r7klb\" (UID: \"46553f3d-e4a6-4ef4-a601-c7c01b9f057e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-15abbf02-6fbdf5668c-r7klb" Nov 03 00:49:57.816777 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:57.816718 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46553f3d-e4a6-4ef4-a601-c7c01b9f057e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-15abbf02-6fbdf5668c-r7klb\" (UID: \"46553f3d-e4a6-4ef4-a601-c7c01b9f057e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-15abbf02-6fbdf5668c-r7klb" Nov 03 00:49:57.824734 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:57.824710 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbkmf\" (UniqueName: \"kubernetes.io/projected/46553f3d-e4a6-4ef4-a601-c7c01b9f057e-kube-api-access-zbkmf\") pod \"payload-pre-processing-e2e-ait-15abbf02-6fbdf5668c-r7klb\" (UID: \"46553f3d-e4a6-4ef4-a601-c7c01b9f057e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-15abbf02-6fbdf5668c-r7klb" Nov 03 00:49:57.887471 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:57.887426 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-15abbf02-6fbdf5668c-r7klb" Nov 03 00:49:58.171503 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:58.171480 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-15abbf02-6fbdf5668c-r7klb"] Nov 03 00:49:58.174394 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:49:58.174368 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46553f3d_e4a6_4ef4_a601_c7c01b9f057e.slice/crio-98dc32727b9c48121c1c5bbaf12bbf5b58d864ebb8a738faf35d5abab0dbb6a1 WatchSource:0}: Error finding container 98dc32727b9c48121c1c5bbaf12bbf5b58d864ebb8a738faf35d5abab0dbb6a1: Status 404 returned error can't find the container with id 98dc32727b9c48121c1c5bbaf12bbf5b58d864ebb8a738faf35d5abab0dbb6a1 Nov 03 00:49:58.476529 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:58.476304 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-331bcaa1-589958875f-gsq9q" event={"ID":"69476020-213f-41b5-9bac-62655d2a4a53","Type":"ContainerStarted","Data":"e296bfedef9bb2a59984478fdf08a1d8a7583c1316f35c3d0a52325be8fcc5b5"} Nov 03 00:49:58.488910 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:58.488869 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-15abbf02-6fbdf5668c-r7klb" event={"ID":"46553f3d-e4a6-4ef4-a601-c7c01b9f057e","Type":"ContainerStarted","Data":"98dc32727b9c48121c1c5bbaf12bbf5b58d864ebb8a738faf35d5abab0dbb6a1"} Nov 03 00:49:58.491529 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:58.491494 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-331bcaa1-846ff86f65-qqczq" event={"ID":"5a139387-07a4-4f16-86a3-3992d14c79b4","Type":"ContainerStarted","Data":"c29eb23e3964910cc95ee7e53ae3a9abae605f63835586597f1bd9a816e46101"} Nov 03 00:49:59.655236 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:59.655176 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c83251ee-9b676fc95-cpf7t"] Nov 03 00:49:59.666022 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:59.665798 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c83251ee-78c8c48447-h8glz"] Nov 03 00:49:59.666022 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:59.665928 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c83251ee-9b676fc95-cpf7t" Nov 03 00:49:59.671707 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:59.671678 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c83251ee-78c8c48447-h8glz" Nov 03 00:49:59.673077 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:59.673012 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c83251ee-9b676fc95-cpf7t"] Nov 03 00:49:59.684652 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:59.684623 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-c83251ee-dockercfg-6qwlf\"" Nov 03 00:49:59.684859 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:59.684842 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-c83251ee\"" Nov 03 00:49:59.689344 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:59.688925 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c83251ee-78c8c48447-h8glz"] Nov 03 00:49:59.846244 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:59.846199 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff0c52dc-11c3-476b-b4c1-d9575ca5129b-plugins-config-volume\") pod \"payload-processing-e2e-ait-c83251ee-9b676fc95-cpf7t\" (UID: \"ff0c52dc-11c3-476b-b4c1-d9575ca5129b\") " pod="openshift-ingress/payload-processing-e2e-ait-c83251ee-9b676fc95-cpf7t" Nov 03 00:49:59.846470 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:59.846354 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mmk2\" (UniqueName: \"kubernetes.io/projected/f2157bb1-e415-40a5-89da-e21b8564e471-kube-api-access-8mmk2\") pod \"payload-pre-processing-e2e-ait-c83251ee-78c8c48447-h8glz\" (UID: \"f2157bb1-e415-40a5-89da-e21b8564e471\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c83251ee-78c8c48447-h8glz" Nov 03 00:49:59.846470 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:59.846392 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cp4zs\" (UniqueName: \"kubernetes.io/projected/ff0c52dc-11c3-476b-b4c1-d9575ca5129b-kube-api-access-cp4zs\") pod \"payload-processing-e2e-ait-c83251ee-9b676fc95-cpf7t\" (UID: \"ff0c52dc-11c3-476b-b4c1-d9575ca5129b\") " pod="openshift-ingress/payload-processing-e2e-ait-c83251ee-9b676fc95-cpf7t" Nov 03 00:49:59.846583 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:59.846484 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2157bb1-e415-40a5-89da-e21b8564e471-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-c83251ee-78c8c48447-h8glz\" (UID: \"f2157bb1-e415-40a5-89da-e21b8564e471\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c83251ee-78c8c48447-h8glz" Nov 03 00:49:59.949383 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:59.948682 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mmk2\" (UniqueName: \"kubernetes.io/projected/f2157bb1-e415-40a5-89da-e21b8564e471-kube-api-access-8mmk2\") pod \"payload-pre-processing-e2e-ait-c83251ee-78c8c48447-h8glz\" (UID: \"f2157bb1-e415-40a5-89da-e21b8564e471\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c83251ee-78c8c48447-h8glz" Nov 03 00:49:59.949383 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:59.948747 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cp4zs\" (UniqueName: \"kubernetes.io/projected/ff0c52dc-11c3-476b-b4c1-d9575ca5129b-kube-api-access-cp4zs\") pod \"payload-processing-e2e-ait-c83251ee-9b676fc95-cpf7t\" (UID: \"ff0c52dc-11c3-476b-b4c1-d9575ca5129b\") " pod="openshift-ingress/payload-processing-e2e-ait-c83251ee-9b676fc95-cpf7t" Nov 03 00:49:59.949383 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:59.948842 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2157bb1-e415-40a5-89da-e21b8564e471-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-c83251ee-78c8c48447-h8glz\" (UID: \"f2157bb1-e415-40a5-89da-e21b8564e471\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c83251ee-78c8c48447-h8glz" Nov 03 00:49:59.949383 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:59.948894 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff0c52dc-11c3-476b-b4c1-d9575ca5129b-plugins-config-volume\") pod \"payload-processing-e2e-ait-c83251ee-9b676fc95-cpf7t\" (UID: \"ff0c52dc-11c3-476b-b4c1-d9575ca5129b\") " pod="openshift-ingress/payload-processing-e2e-ait-c83251ee-9b676fc95-cpf7t" Nov 03 00:49:59.951033 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:59.951001 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2157bb1-e415-40a5-89da-e21b8564e471-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-c83251ee-78c8c48447-h8glz\" (UID: \"f2157bb1-e415-40a5-89da-e21b8564e471\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c83251ee-78c8c48447-h8glz" Nov 03 00:49:59.971638 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:59.971594 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff0c52dc-11c3-476b-b4c1-d9575ca5129b-plugins-config-volume\") pod \"payload-processing-e2e-ait-c83251ee-9b676fc95-cpf7t\" (UID: \"ff0c52dc-11c3-476b-b4c1-d9575ca5129b\") " pod="openshift-ingress/payload-processing-e2e-ait-c83251ee-9b676fc95-cpf7t" Nov 03 00:49:59.987903 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:59.987861 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cp4zs\" (UniqueName: \"kubernetes.io/projected/ff0c52dc-11c3-476b-b4c1-d9575ca5129b-kube-api-access-cp4zs\") pod \"payload-processing-e2e-ait-c83251ee-9b676fc95-cpf7t\" (UID: \"ff0c52dc-11c3-476b-b4c1-d9575ca5129b\") " pod="openshift-ingress/payload-processing-e2e-ait-c83251ee-9b676fc95-cpf7t" Nov 03 00:49:59.988050 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:59.987970 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mmk2\" (UniqueName: \"kubernetes.io/projected/f2157bb1-e415-40a5-89da-e21b8564e471-kube-api-access-8mmk2\") pod \"payload-pre-processing-e2e-ait-c83251ee-78c8c48447-h8glz\" (UID: \"f2157bb1-e415-40a5-89da-e21b8564e471\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c83251ee-78c8c48447-h8glz" Nov 03 00:49:59.992301 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:49:59.992245 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c83251ee-9b676fc95-cpf7t" Nov 03 00:50:00.011512 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:00.011226 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c83251ee-78c8c48447-h8glz" Nov 03 00:50:00.293070 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:00.293027 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c83251ee-9b676fc95-cpf7t"] Nov 03 00:50:00.295724 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:50:00.295678 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff0c52dc_11c3_476b_b4c1_d9575ca5129b.slice/crio-eb6d24ece2101c4eb7b6a602d882fc5275669f15b54cc743f470987ae7f4e4c6 WatchSource:0}: Error finding container eb6d24ece2101c4eb7b6a602d882fc5275669f15b54cc743f470987ae7f4e4c6: Status 404 returned error can't find the container with id eb6d24ece2101c4eb7b6a602d882fc5275669f15b54cc743f470987ae7f4e4c6 Nov 03 00:50:00.353558 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:00.353330 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c83251ee-78c8c48447-h8glz"] Nov 03 00:50:00.357302 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:50:00.354772 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2157bb1_e415_40a5_89da_e21b8564e471.slice/crio-b682284cebc229616b8a70d0348386526bb15399f4b70a68d678499985497293 WatchSource:0}: Error finding container b682284cebc229616b8a70d0348386526bb15399f4b70a68d678499985497293: Status 404 returned error can't find the container with id b682284cebc229616b8a70d0348386526bb15399f4b70a68d678499985497293 Nov 03 00:50:00.518374 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:00.518335 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c83251ee-78c8c48447-h8glz" event={"ID":"f2157bb1-e415-40a5-89da-e21b8564e471","Type":"ContainerStarted","Data":"b682284cebc229616b8a70d0348386526bb15399f4b70a68d678499985497293"} Nov 03 00:50:00.520948 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:00.520905 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c83251ee-9b676fc95-cpf7t" event={"ID":"ff0c52dc-11c3-476b-b4c1-d9575ca5129b","Type":"ContainerStarted","Data":"eb6d24ece2101c4eb7b6a602d882fc5275669f15b54cc743f470987ae7f4e4c6"} Nov 03 00:50:01.390047 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:01.386665 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-4aec6006-openshift-default-77f675f877-fjbkq"] Nov 03 00:50:01.423800 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:01.423772 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-4aec6006-openshift-default-77f675f877-fjbkq"] Nov 03 00:50:01.423942 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:01.423869 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-4aec6006-openshift-default-77f675f877-fjbkq" Nov 03 00:50:01.442902 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:01.442867 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-4aec6006-openshift-default-dockercfg-rmbzg\"" Nov 03 00:50:01.535320 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:01.534478 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-331bcaa1-589958875f-gsq9q" event={"ID":"69476020-213f-41b5-9bac-62655d2a4a53","Type":"ContainerStarted","Data":"6663355ad2fc1da6eab826730d8b354174a335d2efda3efdbece2d4b4276802b"} Nov 03 00:50:01.535487 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:01.535387 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-331bcaa1-589958875f-gsq9q" Nov 03 00:50:01.559847 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:01.559785 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-331bcaa1-589958875f-gsq9q" podStartSLOduration=1.814604585 podStartE2EDuration="5.55976359s" totalImagesPullingTime="3.745159005s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:49:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:50:01.55585275 +0000 UTC m=+543.726855829" watchObservedRunningTime="2026-11-03 00:50:01.55976359 +0000 UTC m=+543.730766687" Nov 03 00:50:01.570507 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:01.570452 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07184f66-00b4-45e3-a60e-503b9dd4b6fa-credential-socket\") pod \"e2e-derive-4aec6006-openshift-default-77f675f877-fjbkq\" (UID: \"07184f66-00b4-45e3-a60e-503b9dd4b6fa\") " pod="openshift-ingress/e2e-derive-4aec6006-openshift-default-77f675f877-fjbkq" Nov 03 00:50:01.570507 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:01.570483 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07184f66-00b4-45e3-a60e-503b9dd4b6fa-istio-data\") pod \"e2e-derive-4aec6006-openshift-default-77f675f877-fjbkq\" (UID: \"07184f66-00b4-45e3-a60e-503b9dd4b6fa\") " pod="openshift-ingress/e2e-derive-4aec6006-openshift-default-77f675f877-fjbkq" Nov 03 00:50:01.570710 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:01.570565 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07184f66-00b4-45e3-a60e-503b9dd4b6fa-istiod-ca-cert\") pod \"e2e-derive-4aec6006-openshift-default-77f675f877-fjbkq\" (UID: \"07184f66-00b4-45e3-a60e-503b9dd4b6fa\") " pod="openshift-ingress/e2e-derive-4aec6006-openshift-default-77f675f877-fjbkq" Nov 03 00:50:01.570710 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:01.570639 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07184f66-00b4-45e3-a60e-503b9dd4b6fa-workload-certs\") pod \"e2e-derive-4aec6006-openshift-default-77f675f877-fjbkq\" (UID: \"07184f66-00b4-45e3-a60e-503b9dd4b6fa\") " pod="openshift-ingress/e2e-derive-4aec6006-openshift-default-77f675f877-fjbkq" Nov 03 00:50:01.570890 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:01.570717 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9dmn\" (UniqueName: \"kubernetes.io/projected/07184f66-00b4-45e3-a60e-503b9dd4b6fa-kube-api-access-j9dmn\") pod \"e2e-derive-4aec6006-openshift-default-77f675f877-fjbkq\" (UID: \"07184f66-00b4-45e3-a60e-503b9dd4b6fa\") " pod="openshift-ingress/e2e-derive-4aec6006-openshift-default-77f675f877-fjbkq" Nov 03 00:50:01.571711 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:01.571643 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07184f66-00b4-45e3-a60e-503b9dd4b6fa-istio-podinfo\") pod \"e2e-derive-4aec6006-openshift-default-77f675f877-fjbkq\" (UID: \"07184f66-00b4-45e3-a60e-503b9dd4b6fa\") " pod="openshift-ingress/e2e-derive-4aec6006-openshift-default-77f675f877-fjbkq" Nov 03 00:50:01.571802 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:01.571724 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07184f66-00b4-45e3-a60e-503b9dd4b6fa-workload-socket\") pod \"e2e-derive-4aec6006-openshift-default-77f675f877-fjbkq\" (UID: \"07184f66-00b4-45e3-a60e-503b9dd4b6fa\") " pod="openshift-ingress/e2e-derive-4aec6006-openshift-default-77f675f877-fjbkq" Nov 03 00:50:01.571802 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:01.571760 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07184f66-00b4-45e3-a60e-503b9dd4b6fa-istio-envoy\") pod \"e2e-derive-4aec6006-openshift-default-77f675f877-fjbkq\" (UID: \"07184f66-00b4-45e3-a60e-503b9dd4b6fa\") " pod="openshift-ingress/e2e-derive-4aec6006-openshift-default-77f675f877-fjbkq" Nov 03 00:50:01.571802 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:01.571786 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07184f66-00b4-45e3-a60e-503b9dd4b6fa-istio-token\") pod \"e2e-derive-4aec6006-openshift-default-77f675f877-fjbkq\" (UID: \"07184f66-00b4-45e3-a60e-503b9dd4b6fa\") " pod="openshift-ingress/e2e-derive-4aec6006-openshift-default-77f675f877-fjbkq" Nov 03 00:50:01.673449 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:01.673407 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9dmn\" (UniqueName: \"kubernetes.io/projected/07184f66-00b4-45e3-a60e-503b9dd4b6fa-kube-api-access-j9dmn\") pod \"e2e-derive-4aec6006-openshift-default-77f675f877-fjbkq\" (UID: \"07184f66-00b4-45e3-a60e-503b9dd4b6fa\") " pod="openshift-ingress/e2e-derive-4aec6006-openshift-default-77f675f877-fjbkq" Nov 03 00:50:01.673614 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:01.673482 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07184f66-00b4-45e3-a60e-503b9dd4b6fa-istio-podinfo\") pod \"e2e-derive-4aec6006-openshift-default-77f675f877-fjbkq\" (UID: \"07184f66-00b4-45e3-a60e-503b9dd4b6fa\") " pod="openshift-ingress/e2e-derive-4aec6006-openshift-default-77f675f877-fjbkq" Nov 03 00:50:01.673614 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:01.673562 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07184f66-00b4-45e3-a60e-503b9dd4b6fa-workload-socket\") pod \"e2e-derive-4aec6006-openshift-default-77f675f877-fjbkq\" (UID: \"07184f66-00b4-45e3-a60e-503b9dd4b6fa\") " pod="openshift-ingress/e2e-derive-4aec6006-openshift-default-77f675f877-fjbkq" Nov 03 00:50:01.673614 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:01.673597 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07184f66-00b4-45e3-a60e-503b9dd4b6fa-istio-envoy\") pod \"e2e-derive-4aec6006-openshift-default-77f675f877-fjbkq\" (UID: \"07184f66-00b4-45e3-a60e-503b9dd4b6fa\") " pod="openshift-ingress/e2e-derive-4aec6006-openshift-default-77f675f877-fjbkq" Nov 03 00:50:01.673824 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:01.673621 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07184f66-00b4-45e3-a60e-503b9dd4b6fa-istio-token\") pod \"e2e-derive-4aec6006-openshift-default-77f675f877-fjbkq\" (UID: \"07184f66-00b4-45e3-a60e-503b9dd4b6fa\") " pod="openshift-ingress/e2e-derive-4aec6006-openshift-default-77f675f877-fjbkq" Nov 03 00:50:01.673824 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:01.673682 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07184f66-00b4-45e3-a60e-503b9dd4b6fa-credential-socket\") pod \"e2e-derive-4aec6006-openshift-default-77f675f877-fjbkq\" (UID: \"07184f66-00b4-45e3-a60e-503b9dd4b6fa\") " pod="openshift-ingress/e2e-derive-4aec6006-openshift-default-77f675f877-fjbkq" Nov 03 00:50:01.673824 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:01.673708 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07184f66-00b4-45e3-a60e-503b9dd4b6fa-istio-data\") pod \"e2e-derive-4aec6006-openshift-default-77f675f877-fjbkq\" (UID: \"07184f66-00b4-45e3-a60e-503b9dd4b6fa\") " pod="openshift-ingress/e2e-derive-4aec6006-openshift-default-77f675f877-fjbkq" Nov 03 00:50:01.673824 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:01.673736 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07184f66-00b4-45e3-a60e-503b9dd4b6fa-istiod-ca-cert\") pod \"e2e-derive-4aec6006-openshift-default-77f675f877-fjbkq\" (UID: \"07184f66-00b4-45e3-a60e-503b9dd4b6fa\") " pod="openshift-ingress/e2e-derive-4aec6006-openshift-default-77f675f877-fjbkq" Nov 03 00:50:01.673824 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:01.673771 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07184f66-00b4-45e3-a60e-503b9dd4b6fa-workload-certs\") pod \"e2e-derive-4aec6006-openshift-default-77f675f877-fjbkq\" (UID: \"07184f66-00b4-45e3-a60e-503b9dd4b6fa\") " pod="openshift-ingress/e2e-derive-4aec6006-openshift-default-77f675f877-fjbkq" Nov 03 00:50:01.674115 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:01.674095 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07184f66-00b4-45e3-a60e-503b9dd4b6fa-credential-socket\") pod \"e2e-derive-4aec6006-openshift-default-77f675f877-fjbkq\" (UID: \"07184f66-00b4-45e3-a60e-503b9dd4b6fa\") " pod="openshift-ingress/e2e-derive-4aec6006-openshift-default-77f675f877-fjbkq" Nov 03 00:50:01.674203 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:01.674186 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07184f66-00b4-45e3-a60e-503b9dd4b6fa-istio-data\") pod \"e2e-derive-4aec6006-openshift-default-77f675f877-fjbkq\" (UID: \"07184f66-00b4-45e3-a60e-503b9dd4b6fa\") " pod="openshift-ingress/e2e-derive-4aec6006-openshift-default-77f675f877-fjbkq" Nov 03 00:50:01.674341 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:01.674290 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07184f66-00b4-45e3-a60e-503b9dd4b6fa-workload-certs\") pod \"e2e-derive-4aec6006-openshift-default-77f675f877-fjbkq\" (UID: \"07184f66-00b4-45e3-a60e-503b9dd4b6fa\") " pod="openshift-ingress/e2e-derive-4aec6006-openshift-default-77f675f877-fjbkq" Nov 03 00:50:01.674341 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:01.674298 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07184f66-00b4-45e3-a60e-503b9dd4b6fa-workload-socket\") pod \"e2e-derive-4aec6006-openshift-default-77f675f877-fjbkq\" (UID: \"07184f66-00b4-45e3-a60e-503b9dd4b6fa\") " pod="openshift-ingress/e2e-derive-4aec6006-openshift-default-77f675f877-fjbkq" Nov 03 00:50:01.674872 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:01.674832 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07184f66-00b4-45e3-a60e-503b9dd4b6fa-istiod-ca-cert\") pod \"e2e-derive-4aec6006-openshift-default-77f675f877-fjbkq\" (UID: \"07184f66-00b4-45e3-a60e-503b9dd4b6fa\") " pod="openshift-ingress/e2e-derive-4aec6006-openshift-default-77f675f877-fjbkq" Nov 03 00:50:01.676857 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:01.676825 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07184f66-00b4-45e3-a60e-503b9dd4b6fa-istio-envoy\") pod \"e2e-derive-4aec6006-openshift-default-77f675f877-fjbkq\" (UID: \"07184f66-00b4-45e3-a60e-503b9dd4b6fa\") " pod="openshift-ingress/e2e-derive-4aec6006-openshift-default-77f675f877-fjbkq" Nov 03 00:50:01.676940 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:01.676918 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07184f66-00b4-45e3-a60e-503b9dd4b6fa-istio-podinfo\") pod \"e2e-derive-4aec6006-openshift-default-77f675f877-fjbkq\" (UID: \"07184f66-00b4-45e3-a60e-503b9dd4b6fa\") " pod="openshift-ingress/e2e-derive-4aec6006-openshift-default-77f675f877-fjbkq" Nov 03 00:50:01.681898 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:01.681871 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07184f66-00b4-45e3-a60e-503b9dd4b6fa-istio-token\") pod \"e2e-derive-4aec6006-openshift-default-77f675f877-fjbkq\" (UID: \"07184f66-00b4-45e3-a60e-503b9dd4b6fa\") " pod="openshift-ingress/e2e-derive-4aec6006-openshift-default-77f675f877-fjbkq" Nov 03 00:50:01.682196 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:01.682171 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9dmn\" (UniqueName: \"kubernetes.io/projected/07184f66-00b4-45e3-a60e-503b9dd4b6fa-kube-api-access-j9dmn\") pod \"e2e-derive-4aec6006-openshift-default-77f675f877-fjbkq\" (UID: \"07184f66-00b4-45e3-a60e-503b9dd4b6fa\") " pod="openshift-ingress/e2e-derive-4aec6006-openshift-default-77f675f877-fjbkq" Nov 03 00:50:01.748356 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:01.748321 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-4aec6006-openshift-default-77f675f877-fjbkq" Nov 03 00:50:02.046415 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:02.045872 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-4aec6006-openshift-default-77f675f877-fjbkq"] Nov 03 00:50:02.050796 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:50:02.050760 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07184f66_00b4_45e3_a60e_503b9dd4b6fa.slice/crio-58434545a79afc71b6e3c8f5939d598dd96b96730ff4cffba6973c06c56d4a12 WatchSource:0}: Error finding container 58434545a79afc71b6e3c8f5939d598dd96b96730ff4cffba6973c06c56d4a12: Status 404 returned error can't find the container with id 58434545a79afc71b6e3c8f5939d598dd96b96730ff4cffba6973c06c56d4a12 Nov 03 00:50:02.054771 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:02.054726 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:50:02.054879 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:02.054820 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:50:02.054879 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:02.054864 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:50:02.541265 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:02.541225 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c83251ee-78c8c48447-h8glz" event={"ID":"f2157bb1-e415-40a5-89da-e21b8564e471","Type":"ContainerStarted","Data":"e64d173aab6dce39f161dc62e3f78db18c0a6908c232562e88e859ce510283a6"} Nov 03 00:50:02.541760 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:02.541434 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-c83251ee-78c8c48447-h8glz" Nov 03 00:50:02.542807 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:02.542785 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-4aec6006-openshift-default-77f675f877-fjbkq" event={"ID":"07184f66-00b4-45e3-a60e-503b9dd4b6fa","Type":"ContainerStarted","Data":"5434745fe472eed3455af9288ae82bfc6926ad3a14203de7094fe748380096d4"} Nov 03 00:50:02.542934 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:02.542826 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-4aec6006-openshift-default-77f675f877-fjbkq" event={"ID":"07184f66-00b4-45e3-a60e-503b9dd4b6fa","Type":"ContainerStarted","Data":"58434545a79afc71b6e3c8f5939d598dd96b96730ff4cffba6973c06c56d4a12"} Nov 03 00:50:02.544403 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:02.544381 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c83251ee-9b676fc95-cpf7t" event={"ID":"ff0c52dc-11c3-476b-b4c1-d9575ca5129b","Type":"ContainerStarted","Data":"47f6674e72d89afc2886199f258dc360e197b2f13f39289e4ae4273de76aff7e"} Nov 03 00:50:02.544500 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:02.544426 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-c83251ee-9b676fc95-cpf7t" Nov 03 00:50:02.545983 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:02.545961 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-15abbf02-6fbdf5668c-r7klb" event={"ID":"46553f3d-e4a6-4ef4-a601-c7c01b9f057e","Type":"ContainerStarted","Data":"d0fa0a83b26dbc7c2b2a427149aa55563afb6959b3edaaf3d01ad292a2f40024"} Nov 03 00:50:02.546122 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:02.546077 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-15abbf02-6fbdf5668c-r7klb" Nov 03 00:50:02.547410 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:02.547386 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-331bcaa1-846ff86f65-qqczq" event={"ID":"5a139387-07a4-4f16-86a3-3992d14c79b4","Type":"ContainerStarted","Data":"5326feea98dd14ba18ef22c9aa70dffa27a772ac0ebea8a5849752c87ec9e775"} Nov 03 00:50:02.547689 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:02.547671 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-331bcaa1-846ff86f65-qqczq" Nov 03 00:50:02.559910 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:02.559868 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-c83251ee-78c8c48447-h8glz" podStartSLOduration=1.6753112940000001 podStartE2EDuration="3.559856034s" totalImagesPullingTime="1.88454474s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:49:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:50:02.55853361 +0000 UTC m=+544.729536706" watchObservedRunningTime="2026-11-03 00:50:02.559856034 +0000 UTC m=+544.730859123" Nov 03 00:50:02.577093 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:02.577030 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-15abbf02-6fbdf5668c-r7klb" podStartSLOduration=2.040391233 podStartE2EDuration="5.577010785s" totalImagesPullingTime="3.536619552s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:49:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:50:02.575597716 +0000 UTC m=+544.746600814" watchObservedRunningTime="2026-11-03 00:50:02.577010785 +0000 UTC m=+544.748013885" Nov 03 00:50:02.598928 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:02.598878 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-331bcaa1-846ff86f65-qqczq" podStartSLOduration=2.714604973 podStartE2EDuration="6.598865058s" totalImagesPullingTime="3.884260085s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:49:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:50:02.597250798 +0000 UTC m=+544.768253893" watchObservedRunningTime="2026-11-03 00:50:02.598865058 +0000 UTC m=+544.769868154" Nov 03 00:50:02.616784 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:02.616727 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-c83251ee-9b676fc95-cpf7t" podStartSLOduration=1.517409136 podStartE2EDuration="3.616711813s" totalImagesPullingTime="2.099302677s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:49:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:50:02.615386861 +0000 UTC m=+544.786389957" watchObservedRunningTime="2026-11-03 00:50:02.616711813 +0000 UTC m=+544.787714909" Nov 03 00:50:02.636186 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:02.636133 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-4aec6006-openshift-default-77f675f877-fjbkq" podStartSLOduration=1.636119453 podStartE2EDuration="1.636119453s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:50:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:50:02.635033728 +0000 UTC m=+544.806036826" watchObservedRunningTime="2026-11-03 00:50:02.636119453 +0000 UTC m=+544.807122590" Nov 03 00:50:02.748852 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:02.748815 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-4aec6006-openshift-default-77f675f877-fjbkq" Nov 03 00:50:03.754912 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:03.754874 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-4aec6006-openshift-default-77f675f877-fjbkq" Nov 03 00:50:04.557592 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:04.557551 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-4aec6006-openshift-default-77f675f877-fjbkq" Nov 03 00:50:04.558749 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:04.558724 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-4aec6006-openshift-default-77f675f877-fjbkq" Nov 03 00:50:05.572358 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:05.572315 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-4aec6006-5b7d45d8d7-9jwv2"] Nov 03 00:50:05.576572 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:05.576546 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-4aec6006-5b7d45d8d7-9jwv2"] Nov 03 00:50:05.576712 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:05.576629 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-4aec6006-5b7d45d8d7-9jwv2" Nov 03 00:50:05.664670 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:05.664637 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-4aec6006\"" Nov 03 00:50:05.665062 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:05.665041 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-4aec6006-dockercfg-6895z\"" Nov 03 00:50:05.680295 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:05.676865 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tkcb\" (UniqueName: \"kubernetes.io/projected/8c0989d6-ad4a-49ad-9a0e-c01f1ef0e797-kube-api-access-5tkcb\") pod \"payload-pre-processing-e2e-derive-4aec6006-5b7d45d8d7-9jwv2\" (UID: \"8c0989d6-ad4a-49ad-9a0e-c01f1ef0e797\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-4aec6006-5b7d45d8d7-9jwv2" Nov 03 00:50:05.680295 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:05.677073 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c0989d6-ad4a-49ad-9a0e-c01f1ef0e797-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-4aec6006-5b7d45d8d7-9jwv2\" (UID: \"8c0989d6-ad4a-49ad-9a0e-c01f1ef0e797\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-4aec6006-5b7d45d8d7-9jwv2" Nov 03 00:50:05.778535 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:05.778491 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c0989d6-ad4a-49ad-9a0e-c01f1ef0e797-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-4aec6006-5b7d45d8d7-9jwv2\" (UID: \"8c0989d6-ad4a-49ad-9a0e-c01f1ef0e797\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-4aec6006-5b7d45d8d7-9jwv2" Nov 03 00:50:05.779442 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:05.779414 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tkcb\" (UniqueName: \"kubernetes.io/projected/8c0989d6-ad4a-49ad-9a0e-c01f1ef0e797-kube-api-access-5tkcb\") pod \"payload-pre-processing-e2e-derive-4aec6006-5b7d45d8d7-9jwv2\" (UID: \"8c0989d6-ad4a-49ad-9a0e-c01f1ef0e797\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-4aec6006-5b7d45d8d7-9jwv2" Nov 03 00:50:05.779621 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:05.779593 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c0989d6-ad4a-49ad-9a0e-c01f1ef0e797-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-4aec6006-5b7d45d8d7-9jwv2\" (UID: \"8c0989d6-ad4a-49ad-9a0e-c01f1ef0e797\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-4aec6006-5b7d45d8d7-9jwv2" Nov 03 00:50:05.790033 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:05.790002 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tkcb\" (UniqueName: \"kubernetes.io/projected/8c0989d6-ad4a-49ad-9a0e-c01f1ef0e797-kube-api-access-5tkcb\") pod \"payload-pre-processing-e2e-derive-4aec6006-5b7d45d8d7-9jwv2\" (UID: \"8c0989d6-ad4a-49ad-9a0e-c01f1ef0e797\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-4aec6006-5b7d45d8d7-9jwv2" Nov 03 00:50:05.889468 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:05.889390 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-4aec6006-5b7d45d8d7-9jwv2" Nov 03 00:50:06.263509 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:06.263483 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-4aec6006-5b7d45d8d7-9jwv2"] Nov 03 00:50:06.582588 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:06.582487 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-4aec6006-5b7d45d8d7-9jwv2" event={"ID":"8c0989d6-ad4a-49ad-9a0e-c01f1ef0e797","Type":"ContainerStarted","Data":"32ae8d2435bbc615574dbf5878fc8c14a0b13163eb76c8d0bc51b43c916d29d1"} Nov 03 00:50:06.582588 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:06.582552 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-4aec6006-5b7d45d8d7-9jwv2" event={"ID":"8c0989d6-ad4a-49ad-9a0e-c01f1ef0e797","Type":"ContainerStarted","Data":"86d925ff78c1989eb73e6dcf3f6fa1f87082302fbae9abc7307a315b43ebd6ea"} Nov 03 00:50:06.582588 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:06.582584 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-4aec6006-5b7d45d8d7-9jwv2" Nov 03 00:50:06.601180 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:06.601132 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-4aec6006-5b7d45d8d7-9jwv2" podStartSLOduration=1.601119693 podStartE2EDuration="1.601119693s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:50:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:50:06.599932248 +0000 UTC m=+548.770935345" watchObservedRunningTime="2026-11-03 00:50:06.601119693 +0000 UTC m=+548.772122788" Nov 03 00:50:10.747004 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:10.746969 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-331bcaa1-zmsf6"] Nov 03 00:50:10.751997 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:10.751975 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-331bcaa1-zmsf6" Nov 03 00:50:10.757367 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:10.757341 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6q4xp\"" Nov 03 00:50:10.757367 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:10.757346 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:50:10.758772 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:10.758752 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 00:50:10.758772 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:10.758754 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 00:50:10.766340 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:10.764467 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-331bcaa1-zmsf6"] Nov 03 00:50:10.885668 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:10.885567 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5n4dm\" (UniqueName: \"kubernetes.io/projected/c3b55f5e-f86b-4b23-adbf-9bccb4089326-kube-api-access-5n4dm\") pod \"maas-api-revoke-keys-e2e-ait-331bcaa1-zmsf6\" (UID: \"c3b55f5e-f86b-4b23-adbf-9bccb4089326\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-331bcaa1-zmsf6" Nov 03 00:50:10.885668 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:10.885614 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3b55f5e-f86b-4b23-adbf-9bccb4089326-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-331bcaa1-zmsf6\" (UID: \"c3b55f5e-f86b-4b23-adbf-9bccb4089326\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-331bcaa1-zmsf6" Nov 03 00:50:10.885840 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:10.885796 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3b55f5e-f86b-4b23-adbf-9bccb4089326-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-331bcaa1-zmsf6\" (UID: \"c3b55f5e-f86b-4b23-adbf-9bccb4089326\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-331bcaa1-zmsf6" Nov 03 00:50:10.987329 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:10.987285 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3b55f5e-f86b-4b23-adbf-9bccb4089326-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-331bcaa1-zmsf6\" (UID: \"c3b55f5e-f86b-4b23-adbf-9bccb4089326\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-331bcaa1-zmsf6" Nov 03 00:50:10.987500 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:10.987445 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5n4dm\" (UniqueName: \"kubernetes.io/projected/c3b55f5e-f86b-4b23-adbf-9bccb4089326-kube-api-access-5n4dm\") pod \"maas-api-revoke-keys-e2e-ait-331bcaa1-zmsf6\" (UID: \"c3b55f5e-f86b-4b23-adbf-9bccb4089326\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-331bcaa1-zmsf6" Nov 03 00:50:10.987500 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:10.987474 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3b55f5e-f86b-4b23-adbf-9bccb4089326-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-331bcaa1-zmsf6\" (UID: \"c3b55f5e-f86b-4b23-adbf-9bccb4089326\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-331bcaa1-zmsf6" Nov 03 00:50:10.988114 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:10.988086 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3b55f5e-f86b-4b23-adbf-9bccb4089326-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-331bcaa1-zmsf6\" (UID: \"c3b55f5e-f86b-4b23-adbf-9bccb4089326\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-331bcaa1-zmsf6" Nov 03 00:50:10.989770 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:10.989740 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3b55f5e-f86b-4b23-adbf-9bccb4089326-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-331bcaa1-zmsf6\" (UID: \"c3b55f5e-f86b-4b23-adbf-9bccb4089326\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-331bcaa1-zmsf6" Nov 03 00:50:10.995366 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:10.995336 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5n4dm\" (UniqueName: \"kubernetes.io/projected/c3b55f5e-f86b-4b23-adbf-9bccb4089326-kube-api-access-5n4dm\") pod \"maas-api-revoke-keys-e2e-ait-331bcaa1-zmsf6\" (UID: \"c3b55f5e-f86b-4b23-adbf-9bccb4089326\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-331bcaa1-zmsf6" Nov 03 00:50:11.078966 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:11.078917 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-331bcaa1-zmsf6" Nov 03 00:50:11.388392 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:11.388360 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-331bcaa1-zmsf6"] Nov 03 00:50:11.391525 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:50:11.391490 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3b55f5e_f86b_4b23_adbf_9bccb4089326.slice/crio-1ece6da1cd42b7416b084f0b0ae09cc90d3a055323128920151a0212bd737e13 WatchSource:0}: Error finding container 1ece6da1cd42b7416b084f0b0ae09cc90d3a055323128920151a0212bd737e13: Status 404 returned error can't find the container with id 1ece6da1cd42b7416b084f0b0ae09cc90d3a055323128920151a0212bd737e13 Nov 03 00:50:11.614661 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:11.614609 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-331bcaa1-zmsf6" event={"ID":"c3b55f5e-f86b-4b23-adbf-9bccb4089326","Type":"ContainerStarted","Data":"1ece6da1cd42b7416b084f0b0ae09cc90d3a055323128920151a0212bd737e13"} Nov 03 00:50:12.504466 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:12.504421 4421 patch_prober.go:29] interesting pod/e2e-ait-c83251ee-openshift-default-5567cd685c-69vg9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 00:50:12.504992 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:12.504487 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-c83251ee-openshift-default-5567cd685c-69vg9" podUID="4d793f1a-5d2f-4435-aee8-38af07f46693" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 00:50:13.555834 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:13.555804 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-c83251ee-9b676fc95-cpf7t" Nov 03 00:50:13.556364 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:13.556185 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-15abbf02-6fbdf5668c-r7klb" Nov 03 00:50:13.556364 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:13.556356 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-331bcaa1-589958875f-gsq9q" Nov 03 00:50:13.556728 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:13.556706 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-331bcaa1-846ff86f65-qqczq" Nov 03 00:50:13.557136 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:13.557116 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-c83251ee-78c8c48447-h8glz" Nov 03 00:50:14.634966 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:14.634929 4421 generic.go:356] "Generic (PLEG): container finished" podID="282b2fbe-dcda-488c-90df-3d087cc51a46" containerID="88b8332b90873e266f07d615d4357cca294363854473c58117db245af97d5c26" exitCode=0 Nov 03 00:50:14.635480 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:14.635006 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-e9b9924e-gw-openshift-default-58b595c4df-xtlrf" event={"ID":"282b2fbe-dcda-488c-90df-3d087cc51a46","Type":"ContainerDied","Data":"88b8332b90873e266f07d615d4357cca294363854473c58117db245af97d5c26"} Nov 03 00:50:14.635480 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:14.635463 4421 scope.go:122] "RemoveContainer" containerID="88b8332b90873e266f07d615d4357cca294363854473c58117db245af97d5c26" Nov 03 00:50:14.637126 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:14.637102 4421 generic.go:356] "Generic (PLEG): container finished" podID="4d793f1a-5d2f-4435-aee8-38af07f46693" containerID="8d67ac4ad544f7c7b7b090c212723c6d805e6eff70c4eca87ea2efcc05237760" exitCode=0 Nov 03 00:50:14.637221 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:14.637121 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:50:14.637221 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:14.637159 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c83251ee-openshift-default-5567cd685c-69vg9" event={"ID":"4d793f1a-5d2f-4435-aee8-38af07f46693","Type":"ContainerDied","Data":"8d67ac4ad544f7c7b7b090c212723c6d805e6eff70c4eca87ea2efcc05237760"} Nov 03 00:50:14.637221 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:14.637189 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:50:14.637352 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:14.637223 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:50:14.637516 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:14.637502 4421 scope.go:122] "RemoveContainer" containerID="8d67ac4ad544f7c7b7b090c212723c6d805e6eff70c4eca87ea2efcc05237760" Nov 03 00:50:14.638782 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:14.638753 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:50:14.638871 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:14.638845 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:50:14.638921 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:14.638895 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:50:14.679422 ip-10-0-133-217 kubenswrapper[4421]: E1103 00:50:14.679388 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 00:50:15.369682 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:15.369647 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e9b9924e-sqmpb"] Nov 03 00:50:15.372624 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:15.372606 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e9b9924e-sqmpb" Nov 03 00:50:15.389841 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:15.389810 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e9b9924e-sqmpb"] Nov 03 00:50:15.467411 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:15.467370 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1d2cd46-7a05-4eae-bd0e-257e7791b10b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-e9b9924e-sqmpb\" (UID: \"f1d2cd46-7a05-4eae-bd0e-257e7791b10b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e9b9924e-sqmpb" Nov 03 00:50:15.467599 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:15.467442 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1d2cd46-7a05-4eae-bd0e-257e7791b10b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-e9b9924e-sqmpb\" (UID: \"f1d2cd46-7a05-4eae-bd0e-257e7791b10b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e9b9924e-sqmpb" Nov 03 00:50:15.467599 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:15.467466 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29w4k\" (UniqueName: \"kubernetes.io/projected/f1d2cd46-7a05-4eae-bd0e-257e7791b10b-kube-api-access-29w4k\") pod \"maas-api-revoke-keys-e2e-migrate-e9b9924e-sqmpb\" (UID: \"f1d2cd46-7a05-4eae-bd0e-257e7791b10b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e9b9924e-sqmpb" Nov 03 00:50:15.568775 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:15.568740 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1d2cd46-7a05-4eae-bd0e-257e7791b10b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-e9b9924e-sqmpb\" (UID: \"f1d2cd46-7a05-4eae-bd0e-257e7791b10b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e9b9924e-sqmpb" Nov 03 00:50:15.568947 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:15.568811 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1d2cd46-7a05-4eae-bd0e-257e7791b10b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-e9b9924e-sqmpb\" (UID: \"f1d2cd46-7a05-4eae-bd0e-257e7791b10b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e9b9924e-sqmpb" Nov 03 00:50:15.568947 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:15.568833 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29w4k\" (UniqueName: \"kubernetes.io/projected/f1d2cd46-7a05-4eae-bd0e-257e7791b10b-kube-api-access-29w4k\") pod \"maas-api-revoke-keys-e2e-migrate-e9b9924e-sqmpb\" (UID: \"f1d2cd46-7a05-4eae-bd0e-257e7791b10b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e9b9924e-sqmpb" Nov 03 00:50:15.569466 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:15.569446 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1d2cd46-7a05-4eae-bd0e-257e7791b10b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-e9b9924e-sqmpb\" (UID: \"f1d2cd46-7a05-4eae-bd0e-257e7791b10b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e9b9924e-sqmpb" Nov 03 00:50:15.571483 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:15.571455 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1d2cd46-7a05-4eae-bd0e-257e7791b10b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-e9b9924e-sqmpb\" (UID: \"f1d2cd46-7a05-4eae-bd0e-257e7791b10b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e9b9924e-sqmpb" Nov 03 00:50:15.575113 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:15.575088 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-15abbf02-94mn4"] Nov 03 00:50:15.577936 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:15.577916 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29w4k\" (UniqueName: \"kubernetes.io/projected/f1d2cd46-7a05-4eae-bd0e-257e7791b10b-kube-api-access-29w4k\") pod \"maas-api-revoke-keys-e2e-migrate-e9b9924e-sqmpb\" (UID: \"f1d2cd46-7a05-4eae-bd0e-257e7791b10b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e9b9924e-sqmpb" Nov 03 00:50:15.578111 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:15.578099 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-15abbf02-94mn4" Nov 03 00:50:15.587164 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:15.587115 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-15abbf02-94mn4"] Nov 03 00:50:15.668016 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:15.667908 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-e9b9924e-gw-openshift-default-58b595c4df-xtlrf" event={"ID":"282b2fbe-dcda-488c-90df-3d087cc51a46","Type":"ContainerStarted","Data":"7c66ee9cd90d493f3c00a909c069dc858015c05ac67e8eb7358f601faf5c22e1"} Nov 03 00:50:15.669379 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:15.669351 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99554349-2804-47e8-9dd7-9c531c15dd72-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-15abbf02-94mn4\" (UID: \"99554349-2804-47e8-9dd7-9c531c15dd72\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-15abbf02-94mn4" Nov 03 00:50:15.669511 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:15.669402 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99554349-2804-47e8-9dd7-9c531c15dd72-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-15abbf02-94mn4\" (UID: \"99554349-2804-47e8-9dd7-9c531c15dd72\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-15abbf02-94mn4" Nov 03 00:50:15.669511 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:15.669472 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mcx6\" (UniqueName: \"kubernetes.io/projected/99554349-2804-47e8-9dd7-9c531c15dd72-kube-api-access-9mcx6\") pod \"maas-api-revoke-keys-e2e-ait-15abbf02-94mn4\" (UID: \"99554349-2804-47e8-9dd7-9c531c15dd72\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-15abbf02-94mn4" Nov 03 00:50:15.671100 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:15.671076 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c83251ee-openshift-default-5567cd685c-69vg9" event={"ID":"4d793f1a-5d2f-4435-aee8-38af07f46693","Type":"ContainerStarted","Data":"cd6ded1d86651bfdeb6b404d49b9400452cfbfbfcb2fe6c1187633c91e45143d"} Nov 03 00:50:15.673838 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:15.673804 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-331bcaa1-zmsf6" event={"ID":"c3b55f5e-f86b-4b23-adbf-9bccb4089326","Type":"ContainerStarted","Data":"30c6c0c59eeb4a6bd6b8d8f3c58db77542b699fe4676ea14b34e348f0d1a0f48"} Nov 03 00:50:15.685582 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:15.685555 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e9b9924e-sqmpb" Nov 03 00:50:15.770812 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:15.770777 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99554349-2804-47e8-9dd7-9c531c15dd72-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-15abbf02-94mn4\" (UID: \"99554349-2804-47e8-9dd7-9c531c15dd72\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-15abbf02-94mn4" Nov 03 00:50:15.770945 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:15.770845 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99554349-2804-47e8-9dd7-9c531c15dd72-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-15abbf02-94mn4\" (UID: \"99554349-2804-47e8-9dd7-9c531c15dd72\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-15abbf02-94mn4" Nov 03 00:50:15.771881 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:15.771148 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mcx6\" (UniqueName: \"kubernetes.io/projected/99554349-2804-47e8-9dd7-9c531c15dd72-kube-api-access-9mcx6\") pod \"maas-api-revoke-keys-e2e-ait-15abbf02-94mn4\" (UID: \"99554349-2804-47e8-9dd7-9c531c15dd72\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-15abbf02-94mn4" Nov 03 00:50:15.771881 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:15.771743 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99554349-2804-47e8-9dd7-9c531c15dd72-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-15abbf02-94mn4\" (UID: \"99554349-2804-47e8-9dd7-9c531c15dd72\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-15abbf02-94mn4" Nov 03 00:50:15.774399 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:15.774373 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99554349-2804-47e8-9dd7-9c531c15dd72-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-15abbf02-94mn4\" (UID: \"99554349-2804-47e8-9dd7-9c531c15dd72\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-15abbf02-94mn4" Nov 03 00:50:15.781065 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:15.781043 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mcx6\" (UniqueName: \"kubernetes.io/projected/99554349-2804-47e8-9dd7-9c531c15dd72-kube-api-access-9mcx6\") pod \"maas-api-revoke-keys-e2e-ait-15abbf02-94mn4\" (UID: \"99554349-2804-47e8-9dd7-9c531c15dd72\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-15abbf02-94mn4" Nov 03 00:50:15.798317 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:15.797986 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-331bcaa1-zmsf6" podStartSLOduration=2.587610231 podStartE2EDuration="5.797968066s" totalImagesPullingTime="3.210357835s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:50:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:50:15.767684898 +0000 UTC m=+557.938687995" watchObservedRunningTime="2026-11-03 00:50:15.797968066 +0000 UTC m=+557.968971162" Nov 03 00:50:15.957311 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:15.956450 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-15abbf02-94mn4" Nov 03 00:50:16.011729 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:16.011630 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-e9b9924e-gw-openshift-default-58b595c4df-xtlrf" Nov 03 00:50:16.055756 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:16.030781 4421 patch_prober.go:29] interesting pod/e2e-migrate-e9b9924e-gw-openshift-default-58b595c4df-xtlrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Nov 03 00:50:16.055756 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:16.030838 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-e9b9924e-gw-openshift-default-58b595c4df-xtlrf" podUID="282b2fbe-dcda-488c-90df-3d087cc51a46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Nov 03 00:50:16.060256 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:50:16.060197 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1d2cd46_7a05_4eae_bd0e_257e7791b10b.slice/crio-4df0758cb5ef43d0af49c2ae535d50d8c7be9d45a91f7e739437fbc0123c2f94 WatchSource:0}: Error finding container 4df0758cb5ef43d0af49c2ae535d50d8c7be9d45a91f7e739437fbc0123c2f94: Status 404 returned error can't find the container with id 4df0758cb5ef43d0af49c2ae535d50d8c7be9d45a91f7e739437fbc0123c2f94 Nov 03 00:50:16.060784 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:16.060695 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e9b9924e-sqmpb"] Nov 03 00:50:16.286589 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:16.286255 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-15abbf02-94mn4"] Nov 03 00:50:16.287721 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:50:16.287695 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99554349_2804_47e8_9dd7_9c531c15dd72.slice/crio-ceab4e20bf77dbc2052985ef053f1a19683e1f1e0eb921038e1e3f99bb5f22e5 WatchSource:0}: Error finding container ceab4e20bf77dbc2052985ef053f1a19683e1f1e0eb921038e1e3f99bb5f22e5: Status 404 returned error can't find the container with id ceab4e20bf77dbc2052985ef053f1a19683e1f1e0eb921038e1e3f99bb5f22e5 Nov 03 00:50:16.386807 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:16.386772 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-c83251ee-openshift-default-5567cd685c-69vg9" Nov 03 00:50:16.387338 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:16.387308 4421 patch_prober.go:29] interesting pod/e2e-ait-c83251ee-openshift-default-5567cd685c-69vg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Nov 03 00:50:16.387442 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:16.387364 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c83251ee-openshift-default-5567cd685c-69vg9" podUID="4d793f1a-5d2f-4435-aee8-38af07f46693" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Nov 03 00:50:16.679844 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:16.679803 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-15abbf02-94mn4" event={"ID":"99554349-2804-47e8-9dd7-9c531c15dd72","Type":"ContainerStarted","Data":"ceab4e20bf77dbc2052985ef053f1a19683e1f1e0eb921038e1e3f99bb5f22e5"} Nov 03 00:50:16.681977 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:16.681954 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e9b9924e-sqmpb" event={"ID":"f1d2cd46-7a05-4eae-bd0e-257e7791b10b","Type":"ContainerStarted","Data":"3e87362075a2eda870b3a1a565a523aa7873106ad9a7e18fa44364c7f74430db"} Nov 03 00:50:16.682117 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:16.681991 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e9b9924e-sqmpb" event={"ID":"f1d2cd46-7a05-4eae-bd0e-257e7791b10b","Type":"ContainerStarted","Data":"4df0758cb5ef43d0af49c2ae535d50d8c7be9d45a91f7e739437fbc0123c2f94"} Nov 03 00:50:16.683477 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:16.683456 4421 generic.go:356] "Generic (PLEG): container finished" podID="c3b55f5e-f86b-4b23-adbf-9bccb4089326" containerID="30c6c0c59eeb4a6bd6b8d8f3c58db77542b699fe4676ea14b34e348f0d1a0f48" exitCode=0 Nov 03 00:50:16.683686 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:16.683664 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-331bcaa1-zmsf6" event={"ID":"c3b55f5e-f86b-4b23-adbf-9bccb4089326","Type":"ContainerDied","Data":"30c6c0c59eeb4a6bd6b8d8f3c58db77542b699fe4676ea14b34e348f0d1a0f48"} Nov 03 00:50:16.715297 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:16.704486 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e9b9924e-sqmpb" podStartSLOduration=1.7044718410000002 podStartE2EDuration="1.704471841s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:50:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:50:16.702293879 +0000 UTC m=+558.873296968" watchObservedRunningTime="2026-11-03 00:50:16.704471841 +0000 UTC m=+558.875474937" Nov 03 00:50:17.025298 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:17.022737 4421 patch_prober.go:29] interesting pod/e2e-migrate-e9b9924e-gw-openshift-default-58b595c4df-xtlrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Nov 03 00:50:17.025298 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:17.022801 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-e9b9924e-gw-openshift-default-58b595c4df-xtlrf" podUID="282b2fbe-dcda-488c-90df-3d087cc51a46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Nov 03 00:50:17.387750 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:17.387662 4421 patch_prober.go:29] interesting pod/e2e-ait-c83251ee-openshift-default-5567cd685c-69vg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Nov 03 00:50:17.387958 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:17.387934 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c83251ee-openshift-default-5567cd685c-69vg9" podUID="4d793f1a-5d2f-4435-aee8-38af07f46693" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Nov 03 00:50:17.592290 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:17.592242 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-4aec6006-5b7d45d8d7-9jwv2" Nov 03 00:50:17.693385 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:17.693293 4421 generic.go:356] "Generic (PLEG): container finished" podID="99554349-2804-47e8-9dd7-9c531c15dd72" containerID="f4a3c5df3590866857371e13146eac6c0f9be2404213e824abbb091840751afd" exitCode=0 Nov 03 00:50:17.693740 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:17.693385 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-15abbf02-94mn4" event={"ID":"99554349-2804-47e8-9dd7-9c531c15dd72","Type":"ContainerDied","Data":"f4a3c5df3590866857371e13146eac6c0f9be2404213e824abbb091840751afd"} Nov 03 00:50:17.696040 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:17.696012 4421 generic.go:356] "Generic (PLEG): container finished" podID="f1d2cd46-7a05-4eae-bd0e-257e7791b10b" containerID="3e87362075a2eda870b3a1a565a523aa7873106ad9a7e18fa44364c7f74430db" exitCode=0 Nov 03 00:50:17.696189 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:17.696112 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e9b9924e-sqmpb" event={"ID":"f1d2cd46-7a05-4eae-bd0e-257e7791b10b","Type":"ContainerDied","Data":"3e87362075a2eda870b3a1a565a523aa7873106ad9a7e18fa44364c7f74430db"} Nov 03 00:50:18.034705 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:18.015730 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c83251ee-rnw88"] Nov 03 00:50:18.065311 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:18.061817 4421 patch_prober.go:29] interesting pod/e2e-migrate-e9b9924e-gw-openshift-default-58b595c4df-xtlrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Nov 03 00:50:18.065311 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:18.061876 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-e9b9924e-gw-openshift-default-58b595c4df-xtlrf" podUID="282b2fbe-dcda-488c-90df-3d087cc51a46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Nov 03 00:50:18.073874 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:18.073578 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-331bcaa1-zmsf6" Nov 03 00:50:18.074254 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:18.074237 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c83251ee-rnw88"] Nov 03 00:50:18.074508 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:18.074496 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c83251ee-rnw88" Nov 03 00:50:18.196505 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:18.196471 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c3b55f5e-f86b-4b23-adbf-9bccb4089326-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3b55f5e-f86b-4b23-adbf-9bccb4089326-maas-api-xks-ca\") pod \"c3b55f5e-f86b-4b23-adbf-9bccb4089326\" (UID: \"c3b55f5e-f86b-4b23-adbf-9bccb4089326\") " Nov 03 00:50:18.196668 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:18.196556 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3b55f5e-f86b-4b23-adbf-9bccb4089326-kube-api-access-5n4dm\" (UniqueName: \"kubernetes.io/projected/c3b55f5e-f86b-4b23-adbf-9bccb4089326-kube-api-access-5n4dm\") pod \"c3b55f5e-f86b-4b23-adbf-9bccb4089326\" (UID: \"c3b55f5e-f86b-4b23-adbf-9bccb4089326\") " Nov 03 00:50:18.196729 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:18.196666 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3b55f5e-f86b-4b23-adbf-9bccb4089326-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3b55f5e-f86b-4b23-adbf-9bccb4089326-maas-api-openshift-ca\") pod \"c3b55f5e-f86b-4b23-adbf-9bccb4089326\" (UID: \"c3b55f5e-f86b-4b23-adbf-9bccb4089326\") " Nov 03 00:50:18.196844 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:18.196822 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cd7k\" (UniqueName: \"kubernetes.io/projected/09b6a3e4-8451-46d5-b337-dcd47488291e-kube-api-access-8cd7k\") pod \"maas-api-revoke-keys-e2e-ait-c83251ee-rnw88\" (UID: \"09b6a3e4-8451-46d5-b337-dcd47488291e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c83251ee-rnw88" Nov 03 00:50:18.196930 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:18.196918 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09b6a3e4-8451-46d5-b337-dcd47488291e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-c83251ee-rnw88\" (UID: \"09b6a3e4-8451-46d5-b337-dcd47488291e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c83251ee-rnw88" Nov 03 00:50:18.197083 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:18.197059 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09b6a3e4-8451-46d5-b337-dcd47488291e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-c83251ee-rnw88\" (UID: \"09b6a3e4-8451-46d5-b337-dcd47488291e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c83251ee-rnw88" Nov 03 00:50:18.197481 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:18.197448 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3b55f5e-f86b-4b23-adbf-9bccb4089326-maas-api-openshift-ca" pod "c3b55f5e-f86b-4b23-adbf-9bccb4089326" (UID: "c3b55f5e-f86b-4b23-adbf-9bccb4089326"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:50:18.199698 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:18.199359 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3b55f5e-f86b-4b23-adbf-9bccb4089326-maas-api-xks-ca" pod "c3b55f5e-f86b-4b23-adbf-9bccb4089326" (UID: "c3b55f5e-f86b-4b23-adbf-9bccb4089326"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:50:18.199698 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:18.199631 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3b55f5e-f86b-4b23-adbf-9bccb4089326-kube-api-access-5n4dm" pod "c3b55f5e-f86b-4b23-adbf-9bccb4089326" (UID: "c3b55f5e-f86b-4b23-adbf-9bccb4089326"). InnerVolumeSpecName "kube-api-access-5n4dm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:50:18.298837 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:18.298740 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09b6a3e4-8451-46d5-b337-dcd47488291e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-c83251ee-rnw88\" (UID: \"09b6a3e4-8451-46d5-b337-dcd47488291e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c83251ee-rnw88" Nov 03 00:50:18.298837 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:18.298792 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cd7k\" (UniqueName: \"kubernetes.io/projected/09b6a3e4-8451-46d5-b337-dcd47488291e-kube-api-access-8cd7k\") pod \"maas-api-revoke-keys-e2e-ait-c83251ee-rnw88\" (UID: \"09b6a3e4-8451-46d5-b337-dcd47488291e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c83251ee-rnw88" Nov 03 00:50:18.299034 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:18.298844 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09b6a3e4-8451-46d5-b337-dcd47488291e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-c83251ee-rnw88\" (UID: \"09b6a3e4-8451-46d5-b337-dcd47488291e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c83251ee-rnw88" Nov 03 00:50:18.299034 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:18.298884 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3b55f5e-f86b-4b23-adbf-9bccb4089326-maas-api-xks-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:50:18.299034 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:18.298895 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5n4dm\" (UniqueName: \"kubernetes.io/projected/c3b55f5e-f86b-4b23-adbf-9bccb4089326-kube-api-access-5n4dm\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:50:18.299034 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:18.298905 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3b55f5e-f86b-4b23-adbf-9bccb4089326-maas-api-openshift-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:50:18.299761 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:18.299734 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09b6a3e4-8451-46d5-b337-dcd47488291e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-c83251ee-rnw88\" (UID: \"09b6a3e4-8451-46d5-b337-dcd47488291e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c83251ee-rnw88" Nov 03 00:50:18.301637 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:18.301612 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09b6a3e4-8451-46d5-b337-dcd47488291e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-c83251ee-rnw88\" (UID: \"09b6a3e4-8451-46d5-b337-dcd47488291e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c83251ee-rnw88" Nov 03 00:50:18.344311 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:18.344094 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cd7k\" (UniqueName: \"kubernetes.io/projected/09b6a3e4-8451-46d5-b337-dcd47488291e-kube-api-access-8cd7k\") pod \"maas-api-revoke-keys-e2e-ait-c83251ee-rnw88\" (UID: \"09b6a3e4-8451-46d5-b337-dcd47488291e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c83251ee-rnw88" Nov 03 00:50:18.389247 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:18.386793 4421 patch_prober.go:29] interesting pod/e2e-ait-c83251ee-openshift-default-5567cd685c-69vg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Nov 03 00:50:18.389247 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:18.386849 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c83251ee-openshift-default-5567cd685c-69vg9" podUID="4d793f1a-5d2f-4435-aee8-38af07f46693" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Nov 03 00:50:18.391012 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:18.390988 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c83251ee-rnw88" Nov 03 00:50:18.765482 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:18.764609 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c83251ee-rnw88"] Nov 03 00:50:18.770398 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:18.770372 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-331bcaa1-zmsf6" Nov 03 00:50:18.772894 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:18.772830 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-331bcaa1-zmsf6" event={"ID":"c3b55f5e-f86b-4b23-adbf-9bccb4089326","Type":"ContainerDied","Data":"1ece6da1cd42b7416b084f0b0ae09cc90d3a055323128920151a0212bd737e13"} Nov 03 00:50:18.772894 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:18.772870 4421 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1ece6da1cd42b7416b084f0b0ae09cc90d3a055323128920151a0212bd737e13" Nov 03 00:50:19.015436 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:19.014477 4421 patch_prober.go:29] interesting pod/e2e-migrate-e9b9924e-gw-openshift-default-58b595c4df-xtlrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Nov 03 00:50:19.015436 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:19.014541 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-e9b9924e-gw-openshift-default-58b595c4df-xtlrf" podUID="282b2fbe-dcda-488c-90df-3d087cc51a46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Nov 03 00:50:19.085258 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:19.085226 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e9b9924e-sqmpb" Nov 03 00:50:19.191195 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:19.191169 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-15abbf02-94mn4" Nov 03 00:50:19.252754 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:19.252713 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1d2cd46-7a05-4eae-bd0e-257e7791b10b-kube-api-access-29w4k\" (UniqueName: \"kubernetes.io/projected/f1d2cd46-7a05-4eae-bd0e-257e7791b10b-kube-api-access-29w4k\") pod \"f1d2cd46-7a05-4eae-bd0e-257e7791b10b\" (UID: \"f1d2cd46-7a05-4eae-bd0e-257e7791b10b\") " Nov 03 00:50:19.253482 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:19.253455 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1d2cd46-7a05-4eae-bd0e-257e7791b10b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1d2cd46-7a05-4eae-bd0e-257e7791b10b-maas-api-openshift-ca\") pod \"f1d2cd46-7a05-4eae-bd0e-257e7791b10b\" (UID: \"f1d2cd46-7a05-4eae-bd0e-257e7791b10b\") " Nov 03 00:50:19.253751 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:19.253731 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1d2cd46-7a05-4eae-bd0e-257e7791b10b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1d2cd46-7a05-4eae-bd0e-257e7791b10b-maas-api-xks-ca\") pod \"f1d2cd46-7a05-4eae-bd0e-257e7791b10b\" (UID: \"f1d2cd46-7a05-4eae-bd0e-257e7791b10b\") " Nov 03 00:50:19.254900 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:19.254877 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1d2cd46-7a05-4eae-bd0e-257e7791b10b-maas-api-openshift-ca" pod "f1d2cd46-7a05-4eae-bd0e-257e7791b10b" (UID: "f1d2cd46-7a05-4eae-bd0e-257e7791b10b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:50:19.267760 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:19.267589 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1d2cd46-7a05-4eae-bd0e-257e7791b10b-kube-api-access-29w4k" pod "f1d2cd46-7a05-4eae-bd0e-257e7791b10b" (UID: "f1d2cd46-7a05-4eae-bd0e-257e7791b10b"). InnerVolumeSpecName "kube-api-access-29w4k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:50:19.268021 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:19.267998 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1d2cd46-7a05-4eae-bd0e-257e7791b10b-maas-api-xks-ca" pod "f1d2cd46-7a05-4eae-bd0e-257e7791b10b" (UID: "f1d2cd46-7a05-4eae-bd0e-257e7791b10b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:50:19.356298 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:19.355616 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99554349-2804-47e8-9dd7-9c531c15dd72-kube-api-access-9mcx6\" (UniqueName: \"kubernetes.io/projected/99554349-2804-47e8-9dd7-9c531c15dd72-kube-api-access-9mcx6\") pod \"99554349-2804-47e8-9dd7-9c531c15dd72\" (UID: \"99554349-2804-47e8-9dd7-9c531c15dd72\") " Nov 03 00:50:19.356298 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:19.355765 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99554349-2804-47e8-9dd7-9c531c15dd72-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99554349-2804-47e8-9dd7-9c531c15dd72-maas-api-openshift-ca\") pod \"99554349-2804-47e8-9dd7-9c531c15dd72\" (UID: \"99554349-2804-47e8-9dd7-9c531c15dd72\") " Nov 03 00:50:19.356298 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:19.355872 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/99554349-2804-47e8-9dd7-9c531c15dd72-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99554349-2804-47e8-9dd7-9c531c15dd72-maas-api-xks-ca\") pod \"99554349-2804-47e8-9dd7-9c531c15dd72\" (UID: \"99554349-2804-47e8-9dd7-9c531c15dd72\") " Nov 03 00:50:19.356298 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:19.356239 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1d2cd46-7a05-4eae-bd0e-257e7791b10b-maas-api-xks-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:50:19.356298 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:19.356265 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29w4k\" (UniqueName: \"kubernetes.io/projected/f1d2cd46-7a05-4eae-bd0e-257e7791b10b-kube-api-access-29w4k\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:50:19.356298 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:19.356303 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1d2cd46-7a05-4eae-bd0e-257e7791b10b-maas-api-openshift-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:50:19.361062 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:19.358991 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99554349-2804-47e8-9dd7-9c531c15dd72-maas-api-openshift-ca" pod "99554349-2804-47e8-9dd7-9c531c15dd72" (UID: "99554349-2804-47e8-9dd7-9c531c15dd72"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:50:19.361427 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:19.361382 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99554349-2804-47e8-9dd7-9c531c15dd72-maas-api-xks-ca" pod "99554349-2804-47e8-9dd7-9c531c15dd72" (UID: "99554349-2804-47e8-9dd7-9c531c15dd72"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:50:19.361593 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:19.361506 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99554349-2804-47e8-9dd7-9c531c15dd72-kube-api-access-9mcx6" pod "99554349-2804-47e8-9dd7-9c531c15dd72" (UID: "99554349-2804-47e8-9dd7-9c531c15dd72"). InnerVolumeSpecName "kube-api-access-9mcx6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:50:19.386328 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:19.386294 4421 patch_prober.go:29] interesting pod/e2e-ait-c83251ee-openshift-default-5567cd685c-69vg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Nov 03 00:50:19.386547 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:19.386359 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c83251ee-openshift-default-5567cd685c-69vg9" podUID="4d793f1a-5d2f-4435-aee8-38af07f46693" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Nov 03 00:50:19.465368 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:19.465313 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mcx6\" (UniqueName: \"kubernetes.io/projected/99554349-2804-47e8-9dd7-9c531c15dd72-kube-api-access-9mcx6\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:50:19.465603 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:19.465589 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99554349-2804-47e8-9dd7-9c531c15dd72-maas-api-openshift-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:50:19.465691 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:19.465682 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99554349-2804-47e8-9dd7-9c531c15dd72-maas-api-xks-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:50:19.778548 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:19.778514 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-15abbf02-94mn4" event={"ID":"99554349-2804-47e8-9dd7-9c531c15dd72","Type":"ContainerDied","Data":"ceab4e20bf77dbc2052985ef053f1a19683e1f1e0eb921038e1e3f99bb5f22e5"} Nov 03 00:50:19.779062 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:19.778594 4421 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ceab4e20bf77dbc2052985ef053f1a19683e1f1e0eb921038e1e3f99bb5f22e5" Nov 03 00:50:19.779062 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:19.778656 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-15abbf02-94mn4" Nov 03 00:50:19.780937 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:19.780910 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e9b9924e-sqmpb" Nov 03 00:50:19.780937 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:19.780928 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e9b9924e-sqmpb" event={"ID":"f1d2cd46-7a05-4eae-bd0e-257e7791b10b","Type":"ContainerDied","Data":"4df0758cb5ef43d0af49c2ae535d50d8c7be9d45a91f7e739437fbc0123c2f94"} Nov 03 00:50:19.781161 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:19.780975 4421 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4df0758cb5ef43d0af49c2ae535d50d8c7be9d45a91f7e739437fbc0123c2f94" Nov 03 00:50:19.783436 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:19.783411 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c83251ee-rnw88" event={"ID":"09b6a3e4-8451-46d5-b337-dcd47488291e","Type":"ContainerStarted","Data":"6c333fb53e598c58e7a96a37e1b91630f499567d88d390375409d5ef9977878a"} Nov 03 00:50:19.783582 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:19.783449 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c83251ee-rnw88" event={"ID":"09b6a3e4-8451-46d5-b337-dcd47488291e","Type":"ContainerStarted","Data":"8ae4e13bfc773229c1b4451945a61f73aab16382c5d1c26d22190667e3e6c752"} Nov 03 00:50:19.804303 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:19.804242 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c83251ee-rnw88" podStartSLOduration=2.804225276 podStartE2EDuration="2.804225276s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:50:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:50:19.802468603 +0000 UTC m=+561.973471699" watchObservedRunningTime="2026-11-03 00:50:19.804225276 +0000 UTC m=+561.975228372" Nov 03 00:50:20.048367 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:20.034685 4421 patch_prober.go:29] interesting pod/e2e-migrate-e9b9924e-gw-openshift-default-58b595c4df-xtlrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Nov 03 00:50:20.048367 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:20.034768 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-e9b9924e-gw-openshift-default-58b595c4df-xtlrf" podUID="282b2fbe-dcda-488c-90df-3d087cc51a46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Nov 03 00:50:20.386965 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:20.386876 4421 patch_prober.go:29] interesting pod/e2e-ait-c83251ee-openshift-default-5567cd685c-69vg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Nov 03 00:50:20.386965 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:20.386929 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c83251ee-openshift-default-5567cd685c-69vg9" podUID="4d793f1a-5d2f-4435-aee8-38af07f46693" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Nov 03 00:50:20.807084 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:20.795610 4421 generic.go:356] "Generic (PLEG): container finished" podID="09b6a3e4-8451-46d5-b337-dcd47488291e" containerID="6c333fb53e598c58e7a96a37e1b91630f499567d88d390375409d5ef9977878a" exitCode=0 Nov 03 00:50:20.807084 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:20.795797 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c83251ee-rnw88" event={"ID":"09b6a3e4-8451-46d5-b337-dcd47488291e","Type":"ContainerDied","Data":"6c333fb53e598c58e7a96a37e1b91630f499567d88d390375409d5ef9977878a"} Nov 03 00:50:21.030986 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:21.030933 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-e9b9924e-gw-openshift-default-58b595c4df-xtlrf" Nov 03 00:50:21.031477 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:21.031451 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-e9b9924e-gw-openshift-default-58b595c4df-xtlrf" Nov 03 00:50:21.032660 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:21.032637 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-e9b9924e-gw-openshift-default-58b595c4df-xtlrf" Nov 03 00:50:21.141682 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:21.141584 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-331bcaa1-589958875f-gsq9q"] Nov 03 00:50:21.149343 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:21.149311 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-331bcaa1-846ff86f65-qqczq"] Nov 03 00:50:21.390931 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:21.390902 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-c83251ee-openshift-default-5567cd685c-69vg9" Nov 03 00:50:21.391209 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:21.391183 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-c83251ee-openshift-default-5567cd685c-69vg9" Nov 03 00:50:21.391877 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:21.391836 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-c83251ee-openshift-default-5567cd685c-69vg9" Nov 03 00:50:22.044166 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:22.044133 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c83251ee-rnw88" Nov 03 00:50:22.126366 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:22.105935 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09b6a3e4-8451-46d5-b337-dcd47488291e-kube-api-access-8cd7k\" (UniqueName: \"kubernetes.io/projected/09b6a3e4-8451-46d5-b337-dcd47488291e-kube-api-access-8cd7k\") pod \"09b6a3e4-8451-46d5-b337-dcd47488291e\" (UID: \"09b6a3e4-8451-46d5-b337-dcd47488291e\") " Nov 03 00:50:22.126366 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:22.105992 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/09b6a3e4-8451-46d5-b337-dcd47488291e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09b6a3e4-8451-46d5-b337-dcd47488291e-maas-api-xks-ca\") pod \"09b6a3e4-8451-46d5-b337-dcd47488291e\" (UID: \"09b6a3e4-8451-46d5-b337-dcd47488291e\") " Nov 03 00:50:22.126366 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:22.106037 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09b6a3e4-8451-46d5-b337-dcd47488291e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09b6a3e4-8451-46d5-b337-dcd47488291e-maas-api-openshift-ca\") pod \"09b6a3e4-8451-46d5-b337-dcd47488291e\" (UID: \"09b6a3e4-8451-46d5-b337-dcd47488291e\") " Nov 03 00:50:22.127321 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:22.126915 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09b6a3e4-8451-46d5-b337-dcd47488291e-maas-api-openshift-ca" pod "09b6a3e4-8451-46d5-b337-dcd47488291e" (UID: "09b6a3e4-8451-46d5-b337-dcd47488291e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:50:22.133884 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:22.133837 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09b6a3e4-8451-46d5-b337-dcd47488291e-kube-api-access-8cd7k" pod "09b6a3e4-8451-46d5-b337-dcd47488291e" (UID: "09b6a3e4-8451-46d5-b337-dcd47488291e"). InnerVolumeSpecName "kube-api-access-8cd7k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:50:22.134099 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:22.134070 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09b6a3e4-8451-46d5-b337-dcd47488291e-maas-api-xks-ca" pod "09b6a3e4-8451-46d5-b337-dcd47488291e" (UID: "09b6a3e4-8451-46d5-b337-dcd47488291e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:50:22.226298 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:22.225975 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8cd7k\" (UniqueName: \"kubernetes.io/projected/09b6a3e4-8451-46d5-b337-dcd47488291e-kube-api-access-8cd7k\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:50:22.226298 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:22.226023 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09b6a3e4-8451-46d5-b337-dcd47488291e-maas-api-xks-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:50:22.226298 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:22.226044 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09b6a3e4-8451-46d5-b337-dcd47488291e-maas-api-openshift-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:50:22.831721 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:22.831672 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c83251ee-rnw88" event={"ID":"09b6a3e4-8451-46d5-b337-dcd47488291e","Type":"ContainerDied","Data":"8ae4e13bfc773229c1b4451945a61f73aab16382c5d1c26d22190667e3e6c752"} Nov 03 00:50:22.831925 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:22.831752 4421 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8ae4e13bfc773229c1b4451945a61f73aab16382c5d1c26d22190667e3e6c752" Nov 03 00:50:22.831925 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:22.831691 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c83251ee-rnw88" Nov 03 00:50:23.864780 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:23.864744 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-331bcaa1-zmsf6"] Nov 03 00:50:23.869631 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:23.869604 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-331bcaa1-zmsf6"] Nov 03 00:50:23.977457 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:23.977417 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3b55f5e-f86b-4b23-adbf-9bccb4089326" path="/var/lib/kubelet/pods/c3b55f5e-f86b-4b23-adbf-9bccb4089326/volumes" Nov 03 00:50:24.031443 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:24.031400 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4aec6006-rlwf2"] Nov 03 00:50:24.034325 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:24.032177 4421 cpu_manager.go:414] "containerMap: removing container" podUID="99554349-2804-47e8-9dd7-9c531c15dd72" containerName="revoke-keys" Nov 03 00:50:24.034325 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:24.032200 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99554349-2804-47e8-9dd7-9c531c15dd72" containerName="revoke-keys" Nov 03 00:50:24.034325 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:24.032215 4421 cpu_manager.go:414] "containerMap: removing container" podUID="c3b55f5e-f86b-4b23-adbf-9bccb4089326" containerName="revoke-keys" Nov 03 00:50:24.034325 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:24.032223 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3b55f5e-f86b-4b23-adbf-9bccb4089326" containerName="revoke-keys" Nov 03 00:50:24.034325 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:24.032234 4421 cpu_manager.go:414] "containerMap: removing container" podUID="f1d2cd46-7a05-4eae-bd0e-257e7791b10b" containerName="revoke-keys" Nov 03 00:50:24.034325 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:24.032241 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1d2cd46-7a05-4eae-bd0e-257e7791b10b" containerName="revoke-keys" Nov 03 00:50:24.034325 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:24.032295 4421 cpu_manager.go:414] "containerMap: removing container" podUID="09b6a3e4-8451-46d5-b337-dcd47488291e" containerName="revoke-keys" Nov 03 00:50:24.034325 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:24.032303 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09b6a3e4-8451-46d5-b337-dcd47488291e" containerName="revoke-keys" Nov 03 00:50:24.034325 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:24.032599 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="99554349-2804-47e8-9dd7-9c531c15dd72" containerName="revoke-keys" Nov 03 00:50:24.034325 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:24.032653 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3b55f5e-f86b-4b23-adbf-9bccb4089326" containerName="revoke-keys" Nov 03 00:50:24.034325 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:24.032664 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1d2cd46-7a05-4eae-bd0e-257e7791b10b" containerName="revoke-keys" Nov 03 00:50:24.034325 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:24.032694 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="09b6a3e4-8451-46d5-b337-dcd47488291e" containerName="revoke-keys" Nov 03 00:50:24.040342 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:24.038827 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4aec6006-rlwf2" Nov 03 00:50:24.043932 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:24.040987 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4aec6006-rlwf2"] Nov 03 00:50:24.048644 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:24.048611 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 00:50:24.048873 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:24.048771 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 00:50:24.048873 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:24.048848 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:50:24.049050 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:24.048970 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6q4xp\"" Nov 03 00:50:24.159584 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:24.159491 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bc1b51f-1f52-4707-b325-8ebc9afa40ef-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-4aec6006-rlwf2\" (UID: \"9bc1b51f-1f52-4707-b325-8ebc9afa40ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4aec6006-rlwf2" Nov 03 00:50:24.159737 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:24.159649 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bc1b51f-1f52-4707-b325-8ebc9afa40ef-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-4aec6006-rlwf2\" (UID: \"9bc1b51f-1f52-4707-b325-8ebc9afa40ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4aec6006-rlwf2" Nov 03 00:50:24.159737 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:24.159715 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hh4hx\" (UniqueName: \"kubernetes.io/projected/9bc1b51f-1f52-4707-b325-8ebc9afa40ef-kube-api-access-hh4hx\") pod \"maas-api-revoke-keys-e2e-derive-4aec6006-rlwf2\" (UID: \"9bc1b51f-1f52-4707-b325-8ebc9afa40ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4aec6006-rlwf2" Nov 03 00:50:24.260255 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:24.260213 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bc1b51f-1f52-4707-b325-8ebc9afa40ef-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-4aec6006-rlwf2\" (UID: \"9bc1b51f-1f52-4707-b325-8ebc9afa40ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4aec6006-rlwf2" Nov 03 00:50:24.260484 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:24.260345 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hh4hx\" (UniqueName: \"kubernetes.io/projected/9bc1b51f-1f52-4707-b325-8ebc9afa40ef-kube-api-access-hh4hx\") pod \"maas-api-revoke-keys-e2e-derive-4aec6006-rlwf2\" (UID: \"9bc1b51f-1f52-4707-b325-8ebc9afa40ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4aec6006-rlwf2" Nov 03 00:50:24.260484 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:24.260387 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bc1b51f-1f52-4707-b325-8ebc9afa40ef-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-4aec6006-rlwf2\" (UID: \"9bc1b51f-1f52-4707-b325-8ebc9afa40ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4aec6006-rlwf2" Nov 03 00:50:24.261012 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:24.260989 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bc1b51f-1f52-4707-b325-8ebc9afa40ef-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-4aec6006-rlwf2\" (UID: \"9bc1b51f-1f52-4707-b325-8ebc9afa40ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4aec6006-rlwf2" Nov 03 00:50:24.262899 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:24.262873 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bc1b51f-1f52-4707-b325-8ebc9afa40ef-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-4aec6006-rlwf2\" (UID: \"9bc1b51f-1f52-4707-b325-8ebc9afa40ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4aec6006-rlwf2" Nov 03 00:50:24.270285 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:24.270247 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hh4hx\" (UniqueName: \"kubernetes.io/projected/9bc1b51f-1f52-4707-b325-8ebc9afa40ef-kube-api-access-hh4hx\") pod \"maas-api-revoke-keys-e2e-derive-4aec6006-rlwf2\" (UID: \"9bc1b51f-1f52-4707-b325-8ebc9afa40ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4aec6006-rlwf2" Nov 03 00:50:24.364833 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:24.364793 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4aec6006-rlwf2" Nov 03 00:50:24.691067 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:24.691029 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4aec6006-rlwf2"] Nov 03 00:50:24.691728 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:50:24.691662 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bc1b51f_1f52_4707_b325_8ebc9afa40ef.slice/crio-b17fa50935d3712d7e1fa6575e355856e47045bc20cdf951c00e78f8c9a9622d WatchSource:0}: Error finding container b17fa50935d3712d7e1fa6575e355856e47045bc20cdf951c00e78f8c9a9622d: Status 404 returned error can't find the container with id b17fa50935d3712d7e1fa6575e355856e47045bc20cdf951c00e78f8c9a9622d Nov 03 00:50:24.846344 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:24.846306 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4aec6006-rlwf2" event={"ID":"9bc1b51f-1f52-4707-b325-8ebc9afa40ef","Type":"ContainerStarted","Data":"b17fa50935d3712d7e1fa6575e355856e47045bc20cdf951c00e78f8c9a9622d"} Nov 03 00:50:25.781575 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:25.781543 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-15abbf02-6fbdf5668c-r7klb"] Nov 03 00:50:25.853793 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:25.853759 4421 generic.go:356] "Generic (PLEG): container finished" podID="9bc1b51f-1f52-4707-b325-8ebc9afa40ef" containerID="75d4f6dec2b5fe1b64a5054c87f5fdcf5b198d986b2750a14eb64fad62743b04" exitCode=0 Nov 03 00:50:25.854033 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:25.853828 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4aec6006-rlwf2" event={"ID":"9bc1b51f-1f52-4707-b325-8ebc9afa40ef","Type":"ContainerDied","Data":"75d4f6dec2b5fe1b64a5054c87f5fdcf5b198d986b2750a14eb64fad62743b04"} Nov 03 00:50:26.143184 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:26.143139 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-331bcaa1-589958875f-gsq9q" podUID="69476020-213f-41b5-9bac-62655d2a4a53" containerName="payload-pre-processing" containerID="cri-o://6663355ad2fc1da6eab826730d8b354174a335d2efda3efdbece2d4b4276802b" gracePeriod=25 Nov 03 00:50:26.150286 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:26.150083 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-331bcaa1-846ff86f65-qqczq" podUID="5a139387-07a4-4f16-86a3-3992d14c79b4" containerName="payload-processing" containerID="cri-o://5326feea98dd14ba18ef22c9aa70dffa27a772ac0ebea8a5849752c87ec9e775" gracePeriod=25 Nov 03 00:50:26.463205 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:26.463175 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-331bcaa1-589958875f-gsq9q" Nov 03 00:50:26.478456 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:26.478410 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69476020-213f-41b5-9bac-62655d2a4a53-kube-api-access-l7mxn\" (UniqueName: \"kubernetes.io/projected/69476020-213f-41b5-9bac-62655d2a4a53-kube-api-access-l7mxn\") pod \"69476020-213f-41b5-9bac-62655d2a4a53\" (UID: \"69476020-213f-41b5-9bac-62655d2a4a53\") " Nov 03 00:50:26.478639 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:26.478544 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69476020-213f-41b5-9bac-62655d2a4a53-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69476020-213f-41b5-9bac-62655d2a4a53-plugins-config-volume\") pod \"69476020-213f-41b5-9bac-62655d2a4a53\" (UID: \"69476020-213f-41b5-9bac-62655d2a4a53\") " Nov 03 00:50:26.478924 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:26.478897 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69476020-213f-41b5-9bac-62655d2a4a53-plugins-config-volume" pod "69476020-213f-41b5-9bac-62655d2a4a53" (UID: "69476020-213f-41b5-9bac-62655d2a4a53"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:50:26.481779 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:26.481746 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69476020-213f-41b5-9bac-62655d2a4a53-kube-api-access-l7mxn" pod "69476020-213f-41b5-9bac-62655d2a4a53" (UID: "69476020-213f-41b5-9bac-62655d2a4a53"). InnerVolumeSpecName "kube-api-access-l7mxn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:50:26.501569 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:26.501539 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-331bcaa1-846ff86f65-qqczq" Nov 03 00:50:26.579189 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:26.579149 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a139387-07a4-4f16-86a3-3992d14c79b4-kube-api-access-vk4sf\" (UniqueName: \"kubernetes.io/projected/5a139387-07a4-4f16-86a3-3992d14c79b4-kube-api-access-vk4sf\") pod \"5a139387-07a4-4f16-86a3-3992d14c79b4\" (UID: \"5a139387-07a4-4f16-86a3-3992d14c79b4\") " Nov 03 00:50:26.579395 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:26.579208 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a139387-07a4-4f16-86a3-3992d14c79b4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a139387-07a4-4f16-86a3-3992d14c79b4-plugins-config-volume\") pod \"5a139387-07a4-4f16-86a3-3992d14c79b4\" (UID: \"5a139387-07a4-4f16-86a3-3992d14c79b4\") " Nov 03 00:50:26.579499 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:26.579485 4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69476020-213f-41b5-9bac-62655d2a4a53-plugins-config-volume\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:50:26.579567 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:26.579500 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7mxn\" (UniqueName: \"kubernetes.io/projected/69476020-213f-41b5-9bac-62655d2a4a53-kube-api-access-l7mxn\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:50:26.579698 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:26.579669 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a139387-07a4-4f16-86a3-3992d14c79b4-plugins-config-volume" pod "5a139387-07a4-4f16-86a3-3992d14c79b4" (UID: "5a139387-07a4-4f16-86a3-3992d14c79b4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:50:26.581390 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:26.581366 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a139387-07a4-4f16-86a3-3992d14c79b4-kube-api-access-vk4sf" pod "5a139387-07a4-4f16-86a3-3992d14c79b4" (UID: "5a139387-07a4-4f16-86a3-3992d14c79b4"). InnerVolumeSpecName "kube-api-access-vk4sf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:50:26.680086 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:26.679989 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vk4sf\" (UniqueName: \"kubernetes.io/projected/5a139387-07a4-4f16-86a3-3992d14c79b4-kube-api-access-vk4sf\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:50:26.680086 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:26.680028 4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a139387-07a4-4f16-86a3-3992d14c79b4-plugins-config-volume\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:50:26.860495 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:26.860458 4421 generic.go:356] "Generic (PLEG): container finished" podID="69476020-213f-41b5-9bac-62655d2a4a53" containerID="6663355ad2fc1da6eab826730d8b354174a335d2efda3efdbece2d4b4276802b" exitCode=0 Nov 03 00:50:26.861220 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:26.860528 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-331bcaa1-589958875f-gsq9q" Nov 03 00:50:26.861976 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:26.860518 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-331bcaa1-589958875f-gsq9q" event={"ID":"69476020-213f-41b5-9bac-62655d2a4a53","Type":"ContainerDied","Data":"6663355ad2fc1da6eab826730d8b354174a335d2efda3efdbece2d4b4276802b"} Nov 03 00:50:26.862254 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:26.862228 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-331bcaa1-589958875f-gsq9q" event={"ID":"69476020-213f-41b5-9bac-62655d2a4a53","Type":"ContainerDied","Data":"e296bfedef9bb2a59984478fdf08a1d8a7583c1316f35c3d0a52325be8fcc5b5"} Nov 03 00:50:26.862459 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:26.862422 4421 scope.go:122] "RemoveContainer" containerID="6663355ad2fc1da6eab826730d8b354174a335d2efda3efdbece2d4b4276802b" Nov 03 00:50:26.863992 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:26.863916 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-331bcaa1-846ff86f65-qqczq" event={"ID":"5a139387-07a4-4f16-86a3-3992d14c79b4","Type":"ContainerDied","Data":"5326feea98dd14ba18ef22c9aa70dffa27a772ac0ebea8a5849752c87ec9e775"} Nov 03 00:50:26.864206 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:26.864174 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-331bcaa1-846ff86f65-qqczq" Nov 03 00:50:26.864876 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:26.864262 4421 generic.go:356] "Generic (PLEG): container finished" podID="5a139387-07a4-4f16-86a3-3992d14c79b4" containerID="5326feea98dd14ba18ef22c9aa70dffa27a772ac0ebea8a5849752c87ec9e775" exitCode=0 Nov 03 00:50:26.865411 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:26.865382 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-331bcaa1-846ff86f65-qqczq" event={"ID":"5a139387-07a4-4f16-86a3-3992d14c79b4","Type":"ContainerDied","Data":"c29eb23e3964910cc95ee7e53ae3a9abae605f63835586597f1bd9a816e46101"} Nov 03 00:50:26.878166 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:26.878141 4421 scope.go:122] "RemoveContainer" containerID="6663355ad2fc1da6eab826730d8b354174a335d2efda3efdbece2d4b4276802b" Nov 03 00:50:26.879321 ip-10-0-133-217 kubenswrapper[4421]: E1103 00:50:26.878675 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6663355ad2fc1da6eab826730d8b354174a335d2efda3efdbece2d4b4276802b\": container with ID starting with 6663355ad2fc1da6eab826730d8b354174a335d2efda3efdbece2d4b4276802b not found: ID does not exist" containerID="6663355ad2fc1da6eab826730d8b354174a335d2efda3efdbece2d4b4276802b" Nov 03 00:50:26.879321 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:26.878739 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6663355ad2fc1da6eab826730d8b354174a335d2efda3efdbece2d4b4276802b"} err="failed to get container status \"6663355ad2fc1da6eab826730d8b354174a335d2efda3efdbece2d4b4276802b\": rpc error: code = NotFound desc = could not find container \"6663355ad2fc1da6eab826730d8b354174a335d2efda3efdbece2d4b4276802b\": container with ID starting with 6663355ad2fc1da6eab826730d8b354174a335d2efda3efdbece2d4b4276802b not found: ID does not exist" Nov 03 00:50:26.879321 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:26.878765 4421 scope.go:122] "RemoveContainer" containerID="5326feea98dd14ba18ef22c9aa70dffa27a772ac0ebea8a5849752c87ec9e775" Nov 03 00:50:26.891648 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:26.891613 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-331bcaa1-846ff86f65-qqczq"] Nov 03 00:50:26.894039 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:26.894017 4421 scope.go:122] "RemoveContainer" containerID="5326feea98dd14ba18ef22c9aa70dffa27a772ac0ebea8a5849752c87ec9e775" Nov 03 00:50:26.894403 ip-10-0-133-217 kubenswrapper[4421]: E1103 00:50:26.894379 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5326feea98dd14ba18ef22c9aa70dffa27a772ac0ebea8a5849752c87ec9e775\": container with ID starting with 5326feea98dd14ba18ef22c9aa70dffa27a772ac0ebea8a5849752c87ec9e775 not found: ID does not exist" containerID="5326feea98dd14ba18ef22c9aa70dffa27a772ac0ebea8a5849752c87ec9e775" Nov 03 00:50:26.894499 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:26.894419 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5326feea98dd14ba18ef22c9aa70dffa27a772ac0ebea8a5849752c87ec9e775"} err="failed to get container status \"5326feea98dd14ba18ef22c9aa70dffa27a772ac0ebea8a5849752c87ec9e775\": rpc error: code = NotFound desc = could not find container \"5326feea98dd14ba18ef22c9aa70dffa27a772ac0ebea8a5849752c87ec9e775\": container with ID starting with 5326feea98dd14ba18ef22c9aa70dffa27a772ac0ebea8a5849752c87ec9e775 not found: ID does not exist" Nov 03 00:50:26.895258 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:26.895235 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-331bcaa1-846ff86f65-qqczq"] Nov 03 00:50:26.903159 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:26.903126 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-331bcaa1-589958875f-gsq9q"] Nov 03 00:50:26.907737 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:26.907674 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-331bcaa1-589958875f-gsq9q"] Nov 03 00:50:27.009831 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:27.009803 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4aec6006-rlwf2" Nov 03 00:50:27.083837 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:27.083802 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9bc1b51f-1f52-4707-b325-8ebc9afa40ef-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bc1b51f-1f52-4707-b325-8ebc9afa40ef-maas-api-openshift-ca\") pod \"9bc1b51f-1f52-4707-b325-8ebc9afa40ef\" (UID: \"9bc1b51f-1f52-4707-b325-8ebc9afa40ef\") " Nov 03 00:50:27.083991 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:27.083844 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bc1b51f-1f52-4707-b325-8ebc9afa40ef-kube-api-access-hh4hx\" (UniqueName: \"kubernetes.io/projected/9bc1b51f-1f52-4707-b325-8ebc9afa40ef-kube-api-access-hh4hx\") pod \"9bc1b51f-1f52-4707-b325-8ebc9afa40ef\" (UID: \"9bc1b51f-1f52-4707-b325-8ebc9afa40ef\") " Nov 03 00:50:27.083991 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:27.083981 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9bc1b51f-1f52-4707-b325-8ebc9afa40ef-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bc1b51f-1f52-4707-b325-8ebc9afa40ef-maas-api-xks-ca\") pod \"9bc1b51f-1f52-4707-b325-8ebc9afa40ef\" (UID: \"9bc1b51f-1f52-4707-b325-8ebc9afa40ef\") " Nov 03 00:50:27.084253 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:27.084222 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bc1b51f-1f52-4707-b325-8ebc9afa40ef-maas-api-openshift-ca" pod "9bc1b51f-1f52-4707-b325-8ebc9afa40ef" (UID: "9bc1b51f-1f52-4707-b325-8ebc9afa40ef"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:50:27.084412 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:27.084376 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bc1b51f-1f52-4707-b325-8ebc9afa40ef-maas-api-openshift-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:50:27.086059 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:27.086030 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bc1b51f-1f52-4707-b325-8ebc9afa40ef-kube-api-access-hh4hx" pod "9bc1b51f-1f52-4707-b325-8ebc9afa40ef" (UID: "9bc1b51f-1f52-4707-b325-8ebc9afa40ef"). InnerVolumeSpecName "kube-api-access-hh4hx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:50:27.086174 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:27.086078 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9bc1b51f-1f52-4707-b325-8ebc9afa40ef-maas-api-xks-ca" pod "9bc1b51f-1f52-4707-b325-8ebc9afa40ef" (UID: "9bc1b51f-1f52-4707-b325-8ebc9afa40ef"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:50:27.185682 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:27.185591 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bc1b51f-1f52-4707-b325-8ebc9afa40ef-maas-api-xks-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:50:27.185682 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:27.185622 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hh4hx\" (UniqueName: \"kubernetes.io/projected/9bc1b51f-1f52-4707-b325-8ebc9afa40ef-kube-api-access-hh4hx\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:50:27.876702 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:27.876668 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4aec6006-rlwf2" Nov 03 00:50:27.877206 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:27.876674 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4aec6006-rlwf2" event={"ID":"9bc1b51f-1f52-4707-b325-8ebc9afa40ef","Type":"ContainerDied","Data":"b17fa50935d3712d7e1fa6575e355856e47045bc20cdf951c00e78f8c9a9622d"} Nov 03 00:50:27.877206 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:27.876813 4421 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b17fa50935d3712d7e1fa6575e355856e47045bc20cdf951c00e78f8c9a9622d" Nov 03 00:50:27.980087 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:27.980049 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a139387-07a4-4f16-86a3-3992d14c79b4" path="/var/lib/kubelet/pods/5a139387-07a4-4f16-86a3-3992d14c79b4/volumes" Nov 03 00:50:27.980681 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:27.980662 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69476020-213f-41b5-9bac-62655d2a4a53" path="/var/lib/kubelet/pods/69476020-213f-41b5-9bac-62655d2a4a53/volumes" Nov 03 00:50:28.124324 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:28.124251 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c83251ee-9b676fc95-cpf7t"] Nov 03 00:50:28.137544 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:28.137459 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c83251ee-78c8c48447-h8glz"] Nov 03 00:50:28.601588 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:28.601547 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-15abbf02-94mn4"] Nov 03 00:50:28.603671 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:28.603638 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-15abbf02-94mn4"] Nov 03 00:50:28.994986 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:28.994897 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e9b9924e-sqmpb"] Nov 03 00:50:29.001512 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:29.001475 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e9b9924e-sqmpb"] Nov 03 00:50:29.163639 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:29.163599 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-e9b9924e-gw-openshift-default-58b595c4df-xtlrf"] Nov 03 00:50:29.163888 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:29.163859 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-e9b9924e-gw-openshift-default-58b595c4df-xtlrf" podUID="282b2fbe-dcda-488c-90df-3d087cc51a46" containerName="istio-proxy" containerID="cri-o://7c66ee9cd90d493f3c00a909c069dc858015c05ac67e8eb7358f601faf5c22e1" gracePeriod=30 Nov 03 00:50:29.980311 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:29.980245 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99554349-2804-47e8-9dd7-9c531c15dd72" path="/var/lib/kubelet/pods/99554349-2804-47e8-9dd7-9c531c15dd72/volumes" Nov 03 00:50:29.982434 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:29.980821 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1d2cd46-7a05-4eae-bd0e-257e7791b10b" path="/var/lib/kubelet/pods/f1d2cd46-7a05-4eae-bd0e-257e7791b10b/volumes" Nov 03 00:50:29.993813 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:29.993780 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-2fkgc"] Nov 03 00:50:29.994238 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:29.994222 4421 cpu_manager.go:414] "containerMap: removing container" podUID="5a139387-07a4-4f16-86a3-3992d14c79b4" containerName="payload-processing" Nov 03 00:50:29.994315 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:29.994240 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a139387-07a4-4f16-86a3-3992d14c79b4" containerName="payload-processing" Nov 03 00:50:29.994315 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:29.994260 4421 cpu_manager.go:414] "containerMap: removing container" podUID="69476020-213f-41b5-9bac-62655d2a4a53" containerName="payload-pre-processing" Nov 03 00:50:29.994315 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:29.994265 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69476020-213f-41b5-9bac-62655d2a4a53" containerName="payload-pre-processing" Nov 03 00:50:29.994315 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:29.994292 4421 cpu_manager.go:414] "containerMap: removing container" podUID="9bc1b51f-1f52-4707-b325-8ebc9afa40ef" containerName="revoke-keys" Nov 03 00:50:29.994315 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:29.994298 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bc1b51f-1f52-4707-b325-8ebc9afa40ef" containerName="revoke-keys" Nov 03 00:50:29.994595 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:29.994387 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="69476020-213f-41b5-9bac-62655d2a4a53" containerName="payload-pre-processing" Nov 03 00:50:29.994595 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:29.994396 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bc1b51f-1f52-4707-b325-8ebc9afa40ef" containerName="revoke-keys" Nov 03 00:50:29.994595 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:29.994405 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a139387-07a4-4f16-86a3-3992d14c79b4" containerName="payload-processing" Nov 03 00:50:29.997457 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:29.997425 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-2fkgc" Nov 03 00:50:30.001495 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:30.001466 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-f7680d-openshift-default-dockercfg-fr542\"" Nov 03 00:50:30.014472 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:30.014429 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-2fkgc"] Nov 03 00:50:30.019935 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:30.019901 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0c98b4a-9b13-4ad5-9e0b-ac50ae14fa5d-workload-certs\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-2fkgc\" (UID: \"f0c98b4a-9b13-4ad5-9e0b-ac50ae14fa5d\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-2fkgc" Nov 03 00:50:30.020115 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:30.019954 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvnjx\" (UniqueName: \"kubernetes.io/projected/f0c98b4a-9b13-4ad5-9e0b-ac50ae14fa5d-kube-api-access-zvnjx\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-2fkgc\" (UID: \"f0c98b4a-9b13-4ad5-9e0b-ac50ae14fa5d\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-2fkgc" Nov 03 00:50:30.020115 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:30.019976 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0c98b4a-9b13-4ad5-9e0b-ac50ae14fa5d-istio-data\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-2fkgc\" (UID: \"f0c98b4a-9b13-4ad5-9e0b-ac50ae14fa5d\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-2fkgc" Nov 03 00:50:30.020246 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:30.020147 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0c98b4a-9b13-4ad5-9e0b-ac50ae14fa5d-istio-envoy\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-2fkgc\" (UID: \"f0c98b4a-9b13-4ad5-9e0b-ac50ae14fa5d\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-2fkgc" Nov 03 00:50:30.020246 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:30.020176 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0c98b4a-9b13-4ad5-9e0b-ac50ae14fa5d-workload-socket\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-2fkgc\" (UID: \"f0c98b4a-9b13-4ad5-9e0b-ac50ae14fa5d\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-2fkgc" Nov 03 00:50:30.020380 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:30.020256 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0c98b4a-9b13-4ad5-9e0b-ac50ae14fa5d-credential-socket\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-2fkgc\" (UID: \"f0c98b4a-9b13-4ad5-9e0b-ac50ae14fa5d\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-2fkgc" Nov 03 00:50:30.020380 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:30.020335 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0c98b4a-9b13-4ad5-9e0b-ac50ae14fa5d-istio-token\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-2fkgc\" (UID: \"f0c98b4a-9b13-4ad5-9e0b-ac50ae14fa5d\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-2fkgc" Nov 03 00:50:30.020380 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:30.020367 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0c98b4a-9b13-4ad5-9e0b-ac50ae14fa5d-istiod-ca-cert\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-2fkgc\" (UID: \"f0c98b4a-9b13-4ad5-9e0b-ac50ae14fa5d\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-2fkgc" Nov 03 00:50:30.020544 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:30.020477 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0c98b4a-9b13-4ad5-9e0b-ac50ae14fa5d-istio-podinfo\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-2fkgc\" (UID: \"f0c98b4a-9b13-4ad5-9e0b-ac50ae14fa5d\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-2fkgc" Nov 03 00:50:30.060602 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:30.060574 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c83251ee-rnw88"] Nov 03 00:50:30.064940 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:30.064916 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c83251ee-rnw88"] Nov 03 00:50:30.121579 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:30.121535 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0c98b4a-9b13-4ad5-9e0b-ac50ae14fa5d-istiod-ca-cert\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-2fkgc\" (UID: \"f0c98b4a-9b13-4ad5-9e0b-ac50ae14fa5d\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-2fkgc" Nov 03 00:50:30.121773 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:30.121620 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0c98b4a-9b13-4ad5-9e0b-ac50ae14fa5d-istio-podinfo\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-2fkgc\" (UID: \"f0c98b4a-9b13-4ad5-9e0b-ac50ae14fa5d\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-2fkgc" Nov 03 00:50:30.121773 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:30.121671 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0c98b4a-9b13-4ad5-9e0b-ac50ae14fa5d-workload-certs\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-2fkgc\" (UID: \"f0c98b4a-9b13-4ad5-9e0b-ac50ae14fa5d\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-2fkgc" Nov 03 00:50:30.121773 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:30.121698 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvnjx\" (UniqueName: \"kubernetes.io/projected/f0c98b4a-9b13-4ad5-9e0b-ac50ae14fa5d-kube-api-access-zvnjx\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-2fkgc\" (UID: \"f0c98b4a-9b13-4ad5-9e0b-ac50ae14fa5d\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-2fkgc" Nov 03 00:50:30.121773 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:30.121728 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0c98b4a-9b13-4ad5-9e0b-ac50ae14fa5d-istio-data\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-2fkgc\" (UID: \"f0c98b4a-9b13-4ad5-9e0b-ac50ae14fa5d\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-2fkgc" Nov 03 00:50:30.121998 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:30.121791 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0c98b4a-9b13-4ad5-9e0b-ac50ae14fa5d-istio-envoy\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-2fkgc\" (UID: \"f0c98b4a-9b13-4ad5-9e0b-ac50ae14fa5d\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-2fkgc" Nov 03 00:50:30.121998 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:30.121813 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0c98b4a-9b13-4ad5-9e0b-ac50ae14fa5d-workload-socket\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-2fkgc\" (UID: \"f0c98b4a-9b13-4ad5-9e0b-ac50ae14fa5d\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-2fkgc" Nov 03 00:50:30.121998 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:30.121835 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0c98b4a-9b13-4ad5-9e0b-ac50ae14fa5d-credential-socket\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-2fkgc\" (UID: \"f0c98b4a-9b13-4ad5-9e0b-ac50ae14fa5d\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-2fkgc" Nov 03 00:50:30.121998 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:30.121868 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0c98b4a-9b13-4ad5-9e0b-ac50ae14fa5d-istio-token\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-2fkgc\" (UID: \"f0c98b4a-9b13-4ad5-9e0b-ac50ae14fa5d\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-2fkgc" Nov 03 00:50:30.122417 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:30.122379 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0c98b4a-9b13-4ad5-9e0b-ac50ae14fa5d-credential-socket\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-2fkgc\" (UID: \"f0c98b4a-9b13-4ad5-9e0b-ac50ae14fa5d\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-2fkgc" Nov 03 00:50:30.122564 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:30.122543 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0c98b4a-9b13-4ad5-9e0b-ac50ae14fa5d-istiod-ca-cert\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-2fkgc\" (UID: \"f0c98b4a-9b13-4ad5-9e0b-ac50ae14fa5d\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-2fkgc" Nov 03 00:50:30.122831 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:30.122806 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0c98b4a-9b13-4ad5-9e0b-ac50ae14fa5d-workload-certs\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-2fkgc\" (UID: \"f0c98b4a-9b13-4ad5-9e0b-ac50ae14fa5d\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-2fkgc" Nov 03 00:50:30.122932 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:30.122840 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0c98b4a-9b13-4ad5-9e0b-ac50ae14fa5d-istio-data\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-2fkgc\" (UID: \"f0c98b4a-9b13-4ad5-9e0b-ac50ae14fa5d\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-2fkgc" Nov 03 00:50:30.122932 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:30.122912 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0c98b4a-9b13-4ad5-9e0b-ac50ae14fa5d-workload-socket\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-2fkgc\" (UID: \"f0c98b4a-9b13-4ad5-9e0b-ac50ae14fa5d\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-2fkgc" Nov 03 00:50:30.124349 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:30.124328 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0c98b4a-9b13-4ad5-9e0b-ac50ae14fa5d-istio-envoy\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-2fkgc\" (UID: \"f0c98b4a-9b13-4ad5-9e0b-ac50ae14fa5d\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-2fkgc" Nov 03 00:50:30.124596 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:30.124569 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0c98b4a-9b13-4ad5-9e0b-ac50ae14fa5d-istio-podinfo\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-2fkgc\" (UID: \"f0c98b4a-9b13-4ad5-9e0b-ac50ae14fa5d\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-2fkgc" Nov 03 00:50:30.133814 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:30.133781 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0c98b4a-9b13-4ad5-9e0b-ac50ae14fa5d-istio-token\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-2fkgc\" (UID: \"f0c98b4a-9b13-4ad5-9e0b-ac50ae14fa5d\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-2fkgc" Nov 03 00:50:30.133976 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:30.133933 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvnjx\" (UniqueName: \"kubernetes.io/projected/f0c98b4a-9b13-4ad5-9e0b-ac50ae14fa5d-kube-api-access-zvnjx\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-2fkgc\" (UID: \"f0c98b4a-9b13-4ad5-9e0b-ac50ae14fa5d\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-2fkgc" Nov 03 00:50:30.316768 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:30.316729 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-2fkgc" Nov 03 00:50:30.608823 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:30.608796 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-2fkgc"] Nov 03 00:50:30.611520 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:50:30.611491 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0c98b4a_9b13_4ad5_9e0b_ac50ae14fa5d.slice/crio-f235caf7ea49d70e7fb0e1e6da53c7f90c7466ddd68d12d6604cc5fc72510e69 WatchSource:0}: Error finding container f235caf7ea49d70e7fb0e1e6da53c7f90c7466ddd68d12d6604cc5fc72510e69: Status 404 returned error can't find the container with id f235caf7ea49d70e7fb0e1e6da53c7f90c7466ddd68d12d6604cc5fc72510e69 Nov 03 00:50:30.613741 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:30.613709 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:50:30.613815 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:30.613781 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:50:30.613852 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:30.613813 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:50:30.765554 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:30.765521 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-c83251ee-openshift-default-5567cd685c-69vg9"] Nov 03 00:50:30.765963 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:30.765932 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-c83251ee-openshift-default-5567cd685c-69vg9" podUID="4d793f1a-5d2f-4435-aee8-38af07f46693" containerName="istio-proxy" containerID="cri-o://cd6ded1d86651bfdeb6b404d49b9400452cfbfbfcb2fe6c1187633c91e45143d" gracePeriod=30 Nov 03 00:50:30.782563 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:30.782513 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-15abbf02-6fbdf5668c-r7klb" podUID="46553f3d-e4a6-4ef4-a601-c7c01b9f057e" containerName="payload-pre-processing" containerID="cri-o://d0fa0a83b26dbc7c2b2a427149aa55563afb6959b3edaaf3d01ad292a2f40024" gracePeriod=25 Nov 03 00:50:30.900897 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:30.900787 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-2fkgc" event={"ID":"f0c98b4a-9b13-4ad5-9e0b-ac50ae14fa5d","Type":"ContainerStarted","Data":"9ef83ec7d0dc2496a4493b5009f4a307086c31c8e36533834180854f6ae4a2ad"} Nov 03 00:50:30.901263 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:30.901215 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-2fkgc" event={"ID":"f0c98b4a-9b13-4ad5-9e0b-ac50ae14fa5d","Type":"ContainerStarted","Data":"f235caf7ea49d70e7fb0e1e6da53c7f90c7466ddd68d12d6604cc5fc72510e69"} Nov 03 00:50:30.904468 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:30.904438 4421 generic.go:356] "Generic (PLEG): container finished" podID="07184f66-00b4-45e3-a60e-503b9dd4b6fa" containerID="5434745fe472eed3455af9288ae82bfc6926ad3a14203de7094fe748380096d4" exitCode=0 Nov 03 00:50:30.904597 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:30.904531 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-4aec6006-openshift-default-77f675f877-fjbkq" event={"ID":"07184f66-00b4-45e3-a60e-503b9dd4b6fa","Type":"ContainerDied","Data":"5434745fe472eed3455af9288ae82bfc6926ad3a14203de7094fe748380096d4"} Nov 03 00:50:30.905042 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:30.905021 4421 scope.go:122] "RemoveContainer" containerID="5434745fe472eed3455af9288ae82bfc6926ad3a14203de7094fe748380096d4" Nov 03 00:50:30.908925 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:30.908883 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:50:30.909050 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:30.908984 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:50:30.909050 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:30.909030 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:50:30.950354 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:30.948815 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-2fkgc" podStartSLOduration=1.9487943589999999 podStartE2EDuration="1.948794359s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:50:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:50:30.925690384 +0000 UTC m=+573.096693482" watchObservedRunningTime="2026-11-03 00:50:30.948794359 +0000 UTC m=+573.119797457" Nov 03 00:50:31.120449 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:31.120410 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-15abbf02-6fbdf5668c-r7klb" Nov 03 00:50:31.133739 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:31.133700 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46553f3d-e4a6-4ef4-a601-c7c01b9f057e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46553f3d-e4a6-4ef4-a601-c7c01b9f057e-plugins-config-volume\") pod \"46553f3d-e4a6-4ef4-a601-c7c01b9f057e\" (UID: \"46553f3d-e4a6-4ef4-a601-c7c01b9f057e\") " Nov 03 00:50:31.133995 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:31.133792 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46553f3d-e4a6-4ef4-a601-c7c01b9f057e-kube-api-access-zbkmf\" (UniqueName: \"kubernetes.io/projected/46553f3d-e4a6-4ef4-a601-c7c01b9f057e-kube-api-access-zbkmf\") pod \"46553f3d-e4a6-4ef4-a601-c7c01b9f057e\" (UID: \"46553f3d-e4a6-4ef4-a601-c7c01b9f057e\") " Nov 03 00:50:31.134128 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:31.134100 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46553f3d-e4a6-4ef4-a601-c7c01b9f057e-plugins-config-volume" pod "46553f3d-e4a6-4ef4-a601-c7c01b9f057e" (UID: "46553f3d-e4a6-4ef4-a601-c7c01b9f057e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:50:31.136623 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:31.136590 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46553f3d-e4a6-4ef4-a601-c7c01b9f057e-kube-api-access-zbkmf" pod "46553f3d-e4a6-4ef4-a601-c7c01b9f057e" (UID: "46553f3d-e4a6-4ef4-a601-c7c01b9f057e"). InnerVolumeSpecName "kube-api-access-zbkmf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:50:31.236963 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:31.235623 4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46553f3d-e4a6-4ef4-a601-c7c01b9f057e-plugins-config-volume\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:50:31.236963 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:31.235662 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbkmf\" (UniqueName: \"kubernetes.io/projected/46553f3d-e4a6-4ef4-a601-c7c01b9f057e-kube-api-access-zbkmf\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:50:31.318119 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:31.318071 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-2fkgc" Nov 03 00:50:31.323753 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:31.323728 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-2fkgc" Nov 03 00:50:31.934507 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:31.934467 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-4aec6006-openshift-default-77f675f877-fjbkq" event={"ID":"07184f66-00b4-45e3-a60e-503b9dd4b6fa","Type":"ContainerStarted","Data":"1f42a74cd65307be2d561bb30b71f24440123834604818be21ae4f5f41884f98"} Nov 03 00:50:31.936119 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:31.936093 4421 generic.go:356] "Generic (PLEG): container finished" podID="46553f3d-e4a6-4ef4-a601-c7c01b9f057e" containerID="d0fa0a83b26dbc7c2b2a427149aa55563afb6959b3edaaf3d01ad292a2f40024" exitCode=0 Nov 03 00:50:31.936313 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:31.936134 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-15abbf02-6fbdf5668c-r7klb" Nov 03 00:50:31.936313 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:31.936215 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-15abbf02-6fbdf5668c-r7klb" event={"ID":"46553f3d-e4a6-4ef4-a601-c7c01b9f057e","Type":"ContainerDied","Data":"d0fa0a83b26dbc7c2b2a427149aa55563afb6959b3edaaf3d01ad292a2f40024"} Nov 03 00:50:31.936313 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:31.936245 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-15abbf02-6fbdf5668c-r7klb" event={"ID":"46553f3d-e4a6-4ef4-a601-c7c01b9f057e","Type":"ContainerDied","Data":"98dc32727b9c48121c1c5bbaf12bbf5b58d864ebb8a738faf35d5abab0dbb6a1"} Nov 03 00:50:31.936313 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:31.936267 4421 scope.go:122] "RemoveContainer" containerID="d0fa0a83b26dbc7c2b2a427149aa55563afb6959b3edaaf3d01ad292a2f40024" Nov 03 00:50:31.936742 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:31.936701 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-2fkgc" Nov 03 00:50:31.937894 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:31.937875 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-2fkgc" Nov 03 00:50:31.946643 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:31.946626 4421 scope.go:122] "RemoveContainer" containerID="d0fa0a83b26dbc7c2b2a427149aa55563afb6959b3edaaf3d01ad292a2f40024" Nov 03 00:50:31.946943 ip-10-0-133-217 kubenswrapper[4421]: E1103 00:50:31.946921 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0fa0a83b26dbc7c2b2a427149aa55563afb6959b3edaaf3d01ad292a2f40024\": container with ID starting with d0fa0a83b26dbc7c2b2a427149aa55563afb6959b3edaaf3d01ad292a2f40024 not found: ID does not exist" containerID="d0fa0a83b26dbc7c2b2a427149aa55563afb6959b3edaaf3d01ad292a2f40024" Nov 03 00:50:31.946991 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:31.946952 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0fa0a83b26dbc7c2b2a427149aa55563afb6959b3edaaf3d01ad292a2f40024"} err="failed to get container status \"d0fa0a83b26dbc7c2b2a427149aa55563afb6959b3edaaf3d01ad292a2f40024\": rpc error: code = NotFound desc = could not find container \"d0fa0a83b26dbc7c2b2a427149aa55563afb6959b3edaaf3d01ad292a2f40024\": container with ID starting with d0fa0a83b26dbc7c2b2a427149aa55563afb6959b3edaaf3d01ad292a2f40024 not found: ID does not exist" Nov 03 00:50:31.978375 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:31.978334 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09b6a3e4-8451-46d5-b337-dcd47488291e" path="/var/lib/kubelet/pods/09b6a3e4-8451-46d5-b337-dcd47488291e/volumes" Nov 03 00:50:32.004510 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:32.000265 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-15abbf02-6fbdf5668c-r7klb"] Nov 03 00:50:32.051258 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:32.051227 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-15abbf02-6fbdf5668c-r7klb"] Nov 03 00:50:32.748620 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:32.748579 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-4aec6006-openshift-default-77f675f877-fjbkq" Nov 03 00:50:32.749953 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:32.749930 4421 patch_prober.go:29] interesting pod/e2e-derive-4aec6006-openshift-default-77f675f877-fjbkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Nov 03 00:50:32.750074 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:32.749979 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-4aec6006-openshift-default-77f675f877-fjbkq" podUID="07184f66-00b4-45e3-a60e-503b9dd4b6fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Nov 03 00:50:33.125865 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:33.124714 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-c83251ee-9b676fc95-cpf7t" podUID="ff0c52dc-11c3-476b-b4c1-d9575ca5129b" containerName="payload-processing" containerID="cri-o://47f6674e72d89afc2886199f258dc360e197b2f13f39289e4ae4273de76aff7e" gracePeriod=25 Nov 03 00:50:33.138571 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:33.138512 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-c83251ee-78c8c48447-h8glz" podUID="f2157bb1-e415-40a5-89da-e21b8564e471" containerName="payload-pre-processing" containerID="cri-o://e64d173aab6dce39f161dc62e3f78db18c0a6908c232562e88e859ce510283a6" gracePeriod=25 Nov 03 00:50:33.448765 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:33.448736 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c83251ee-9b676fc95-cpf7t" Nov 03 00:50:33.459313 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:33.459262 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff0c52dc-11c3-476b-b4c1-d9575ca5129b-kube-api-access-cp4zs\" (UniqueName: \"kubernetes.io/projected/ff0c52dc-11c3-476b-b4c1-d9575ca5129b-kube-api-access-cp4zs\") pod \"ff0c52dc-11c3-476b-b4c1-d9575ca5129b\" (UID: \"ff0c52dc-11c3-476b-b4c1-d9575ca5129b\") " Nov 03 00:50:33.459461 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:33.459400 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff0c52dc-11c3-476b-b4c1-d9575ca5129b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff0c52dc-11c3-476b-b4c1-d9575ca5129b-plugins-config-volume\") pod \"ff0c52dc-11c3-476b-b4c1-d9575ca5129b\" (UID: \"ff0c52dc-11c3-476b-b4c1-d9575ca5129b\") " Nov 03 00:50:33.459818 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:33.459796 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff0c52dc-11c3-476b-b4c1-d9575ca5129b-plugins-config-volume" pod "ff0c52dc-11c3-476b-b4c1-d9575ca5129b" (UID: "ff0c52dc-11c3-476b-b4c1-d9575ca5129b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:50:33.461609 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:33.461584 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff0c52dc-11c3-476b-b4c1-d9575ca5129b-kube-api-access-cp4zs" pod "ff0c52dc-11c3-476b-b4c1-d9575ca5129b" (UID: "ff0c52dc-11c3-476b-b4c1-d9575ca5129b"). InnerVolumeSpecName "kube-api-access-cp4zs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:50:33.481354 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:33.480671 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c83251ee-78c8c48447-h8glz" Nov 03 00:50:33.560469 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:33.560432 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2157bb1-e415-40a5-89da-e21b8564e471-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2157bb1-e415-40a5-89da-e21b8564e471-plugins-config-volume\") pod \"f2157bb1-e415-40a5-89da-e21b8564e471\" (UID: \"f2157bb1-e415-40a5-89da-e21b8564e471\") " Nov 03 00:50:33.560632 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:33.560499 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2157bb1-e415-40a5-89da-e21b8564e471-kube-api-access-8mmk2\" (UniqueName: \"kubernetes.io/projected/f2157bb1-e415-40a5-89da-e21b8564e471-kube-api-access-8mmk2\") pod \"f2157bb1-e415-40a5-89da-e21b8564e471\" (UID: \"f2157bb1-e415-40a5-89da-e21b8564e471\") " Nov 03 00:50:33.560729 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:33.560717 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cp4zs\" (UniqueName: \"kubernetes.io/projected/ff0c52dc-11c3-476b-b4c1-d9575ca5129b-kube-api-access-cp4zs\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:50:33.560764 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:33.560731 4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff0c52dc-11c3-476b-b4c1-d9575ca5129b-plugins-config-volume\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:50:33.560860 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:33.560836 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2157bb1-e415-40a5-89da-e21b8564e471-plugins-config-volume" pod "f2157bb1-e415-40a5-89da-e21b8564e471" (UID: "f2157bb1-e415-40a5-89da-e21b8564e471"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:50:33.562611 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:33.562590 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2157bb1-e415-40a5-89da-e21b8564e471-kube-api-access-8mmk2" pod "f2157bb1-e415-40a5-89da-e21b8564e471" (UID: "f2157bb1-e415-40a5-89da-e21b8564e471"). InnerVolumeSpecName "kube-api-access-8mmk2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:50:33.661785 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:33.661696 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mmk2\" (UniqueName: \"kubernetes.io/projected/f2157bb1-e415-40a5-89da-e21b8564e471-kube-api-access-8mmk2\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:50:33.661785 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:33.661733 4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2157bb1-e415-40a5-89da-e21b8564e471-plugins-config-volume\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:50:33.749209 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:33.748786 4421 patch_prober.go:29] interesting pod/e2e-derive-4aec6006-openshift-default-77f675f877-fjbkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Nov 03 00:50:33.749209 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:33.748850 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-4aec6006-openshift-default-77f675f877-fjbkq" podUID="07184f66-00b4-45e3-a60e-503b9dd4b6fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Nov 03 00:50:33.953393 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:33.953310 4421 generic.go:356] "Generic (PLEG): container finished" podID="ff0c52dc-11c3-476b-b4c1-d9575ca5129b" containerID="47f6674e72d89afc2886199f258dc360e197b2f13f39289e4ae4273de76aff7e" exitCode=0 Nov 03 00:50:33.953572 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:33.953406 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c83251ee-9b676fc95-cpf7t" Nov 03 00:50:33.953572 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:33.953444 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c83251ee-9b676fc95-cpf7t" event={"ID":"ff0c52dc-11c3-476b-b4c1-d9575ca5129b","Type":"ContainerDied","Data":"47f6674e72d89afc2886199f258dc360e197b2f13f39289e4ae4273de76aff7e"} Nov 03 00:50:33.953572 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:33.953533 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c83251ee-9b676fc95-cpf7t" event={"ID":"ff0c52dc-11c3-476b-b4c1-d9575ca5129b","Type":"ContainerDied","Data":"eb6d24ece2101c4eb7b6a602d882fc5275669f15b54cc743f470987ae7f4e4c6"} Nov 03 00:50:33.953572 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:33.953562 4421 scope.go:122] "RemoveContainer" containerID="47f6674e72d89afc2886199f258dc360e197b2f13f39289e4ae4273de76aff7e" Nov 03 00:50:33.955247 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:33.955218 4421 generic.go:356] "Generic (PLEG): container finished" podID="f2157bb1-e415-40a5-89da-e21b8564e471" containerID="e64d173aab6dce39f161dc62e3f78db18c0a6908c232562e88e859ce510283a6" exitCode=0 Nov 03 00:50:33.955357 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:33.955321 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c83251ee-78c8c48447-h8glz" Nov 03 00:50:33.955400 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:33.955372 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c83251ee-78c8c48447-h8glz" event={"ID":"f2157bb1-e415-40a5-89da-e21b8564e471","Type":"ContainerDied","Data":"e64d173aab6dce39f161dc62e3f78db18c0a6908c232562e88e859ce510283a6"} Nov 03 00:50:33.955436 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:33.955416 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c83251ee-78c8c48447-h8glz" event={"ID":"f2157bb1-e415-40a5-89da-e21b8564e471","Type":"ContainerDied","Data":"b682284cebc229616b8a70d0348386526bb15399f4b70a68d678499985497293"} Nov 03 00:50:33.967550 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:33.967533 4421 scope.go:122] "RemoveContainer" containerID="47f6674e72d89afc2886199f258dc360e197b2f13f39289e4ae4273de76aff7e" Nov 03 00:50:33.967882 ip-10-0-133-217 kubenswrapper[4421]: E1103 00:50:33.967863 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47f6674e72d89afc2886199f258dc360e197b2f13f39289e4ae4273de76aff7e\": container with ID starting with 47f6674e72d89afc2886199f258dc360e197b2f13f39289e4ae4273de76aff7e not found: ID does not exist" containerID="47f6674e72d89afc2886199f258dc360e197b2f13f39289e4ae4273de76aff7e" Nov 03 00:50:33.967965 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:33.967891 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47f6674e72d89afc2886199f258dc360e197b2f13f39289e4ae4273de76aff7e"} err="failed to get container status \"47f6674e72d89afc2886199f258dc360e197b2f13f39289e4ae4273de76aff7e\": rpc error: code = NotFound desc = could not find container \"47f6674e72d89afc2886199f258dc360e197b2f13f39289e4ae4273de76aff7e\": container with ID starting with 47f6674e72d89afc2886199f258dc360e197b2f13f39289e4ae4273de76aff7e not found: ID does not exist" Nov 03 00:50:33.967965 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:33.967910 4421 scope.go:122] "RemoveContainer" containerID="e64d173aab6dce39f161dc62e3f78db18c0a6908c232562e88e859ce510283a6" Nov 03 00:50:33.977236 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:33.977217 4421 scope.go:122] "RemoveContainer" containerID="e64d173aab6dce39f161dc62e3f78db18c0a6908c232562e88e859ce510283a6" Nov 03 00:50:33.977543 ip-10-0-133-217 kubenswrapper[4421]: E1103 00:50:33.977523 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e64d173aab6dce39f161dc62e3f78db18c0a6908c232562e88e859ce510283a6\": container with ID starting with e64d173aab6dce39f161dc62e3f78db18c0a6908c232562e88e859ce510283a6 not found: ID does not exist" containerID="e64d173aab6dce39f161dc62e3f78db18c0a6908c232562e88e859ce510283a6" Nov 03 00:50:33.977609 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:33.977555 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e64d173aab6dce39f161dc62e3f78db18c0a6908c232562e88e859ce510283a6"} err="failed to get container status \"e64d173aab6dce39f161dc62e3f78db18c0a6908c232562e88e859ce510283a6\": rpc error: code = NotFound desc = could not find container \"e64d173aab6dce39f161dc62e3f78db18c0a6908c232562e88e859ce510283a6\": container with ID starting with e64d173aab6dce39f161dc62e3f78db18c0a6908c232562e88e859ce510283a6 not found: ID does not exist" Nov 03 00:50:33.979163 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:33.979099 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46553f3d-e4a6-4ef4-a601-c7c01b9f057e" path="/var/lib/kubelet/pods/46553f3d-e4a6-4ef4-a601-c7c01b9f057e/volumes" Nov 03 00:50:33.980998 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:33.980977 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c83251ee-78c8c48447-h8glz"] Nov 03 00:50:33.986701 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:33.986677 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c83251ee-78c8c48447-h8glz"] Nov 03 00:50:33.998082 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:33.998035 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c83251ee-9b676fc95-cpf7t"] Nov 03 00:50:34.000981 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:34.000950 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c83251ee-9b676fc95-cpf7t"] Nov 03 00:50:34.355823 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:34.355780 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-4aec6006-5b7d45d8d7-9jwv2"] Nov 03 00:50:34.511756 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:34.511409 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-e9b9924e-gw-openshift-default-58b595c4df-xtlrf" Nov 03 00:50:34.558103 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:34.558000 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-4aec6006-openshift-default-77f675f877-fjbkq" Nov 03 00:50:34.571100 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:34.571071 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/282b2fbe-dcda-488c-90df-3d087cc51a46-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/282b2fbe-dcda-488c-90df-3d087cc51a46-workload-socket\") pod \"282b2fbe-dcda-488c-90df-3d087cc51a46\" (UID: \"282b2fbe-dcda-488c-90df-3d087cc51a46\") " Nov 03 00:50:34.571262 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:34.571133 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/282b2fbe-dcda-488c-90df-3d087cc51a46-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/282b2fbe-dcda-488c-90df-3d087cc51a46-credential-socket\") pod \"282b2fbe-dcda-488c-90df-3d087cc51a46\" (UID: \"282b2fbe-dcda-488c-90df-3d087cc51a46\") " Nov 03 00:50:34.571262 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:34.571157 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/282b2fbe-dcda-488c-90df-3d087cc51a46-kube-api-access-vqwnc\" (UniqueName: \"kubernetes.io/projected/282b2fbe-dcda-488c-90df-3d087cc51a46-kube-api-access-vqwnc\") pod \"282b2fbe-dcda-488c-90df-3d087cc51a46\" (UID: \"282b2fbe-dcda-488c-90df-3d087cc51a46\") " Nov 03 00:50:34.571262 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:34.571192 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/282b2fbe-dcda-488c-90df-3d087cc51a46-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/282b2fbe-dcda-488c-90df-3d087cc51a46-istio-envoy\") pod \"282b2fbe-dcda-488c-90df-3d087cc51a46\" (UID: \"282b2fbe-dcda-488c-90df-3d087cc51a46\") " Nov 03 00:50:34.571262 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:34.571235 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/282b2fbe-dcda-488c-90df-3d087cc51a46-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/282b2fbe-dcda-488c-90df-3d087cc51a46-istiod-ca-cert\") pod \"282b2fbe-dcda-488c-90df-3d087cc51a46\" (UID: \"282b2fbe-dcda-488c-90df-3d087cc51a46\") " Nov 03 00:50:34.571262 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:34.571251 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/282b2fbe-dcda-488c-90df-3d087cc51a46-istio-token\" (UniqueName: \"kubernetes.io/projected/282b2fbe-dcda-488c-90df-3d087cc51a46-istio-token\") pod \"282b2fbe-dcda-488c-90df-3d087cc51a46\" (UID: \"282b2fbe-dcda-488c-90df-3d087cc51a46\") " Nov 03 00:50:34.571459 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:34.571336 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/282b2fbe-dcda-488c-90df-3d087cc51a46-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/282b2fbe-dcda-488c-90df-3d087cc51a46-istio-podinfo\") pod \"282b2fbe-dcda-488c-90df-3d087cc51a46\" (UID: \"282b2fbe-dcda-488c-90df-3d087cc51a46\") " Nov 03 00:50:34.571459 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:34.571444 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/282b2fbe-dcda-488c-90df-3d087cc51a46-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/282b2fbe-dcda-488c-90df-3d087cc51a46-workload-certs\") pod \"282b2fbe-dcda-488c-90df-3d087cc51a46\" (UID: \"282b2fbe-dcda-488c-90df-3d087cc51a46\") " Nov 03 00:50:34.571520 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:34.571476 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/282b2fbe-dcda-488c-90df-3d087cc51a46-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/282b2fbe-dcda-488c-90df-3d087cc51a46-istio-data\") pod \"282b2fbe-dcda-488c-90df-3d087cc51a46\" (UID: \"282b2fbe-dcda-488c-90df-3d087cc51a46\") " Nov 03 00:50:34.572249 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:34.572079 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/282b2fbe-dcda-488c-90df-3d087cc51a46-workload-certs" pod "282b2fbe-dcda-488c-90df-3d087cc51a46" (UID: "282b2fbe-dcda-488c-90df-3d087cc51a46"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:50:34.572621 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:34.572297 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/282b2fbe-dcda-488c-90df-3d087cc51a46-istio-data" pod "282b2fbe-dcda-488c-90df-3d087cc51a46" (UID: "282b2fbe-dcda-488c-90df-3d087cc51a46"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:50:34.572621 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:34.572500 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/282b2fbe-dcda-488c-90df-3d087cc51a46-workload-socket" pod "282b2fbe-dcda-488c-90df-3d087cc51a46" (UID: "282b2fbe-dcda-488c-90df-3d087cc51a46"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:50:34.572621 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:34.572533 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/282b2fbe-dcda-488c-90df-3d087cc51a46-credential-socket" pod "282b2fbe-dcda-488c-90df-3d087cc51a46" (UID: "282b2fbe-dcda-488c-90df-3d087cc51a46"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:50:34.572813 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:34.572769 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/282b2fbe-dcda-488c-90df-3d087cc51a46-istiod-ca-cert" pod "282b2fbe-dcda-488c-90df-3d087cc51a46" (UID: "282b2fbe-dcda-488c-90df-3d087cc51a46"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:50:34.574123 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:34.574091 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/282b2fbe-dcda-488c-90df-3d087cc51a46-istio-envoy" pod "282b2fbe-dcda-488c-90df-3d087cc51a46" (UID: "282b2fbe-dcda-488c-90df-3d087cc51a46"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:50:34.574296 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:34.574257 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/282b2fbe-dcda-488c-90df-3d087cc51a46-istio-token" pod "282b2fbe-dcda-488c-90df-3d087cc51a46" (UID: "282b2fbe-dcda-488c-90df-3d087cc51a46"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:50:34.575038 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:34.574997 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/282b2fbe-dcda-488c-90df-3d087cc51a46-istio-podinfo" pod "282b2fbe-dcda-488c-90df-3d087cc51a46" (UID: "282b2fbe-dcda-488c-90df-3d087cc51a46"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:50:34.575325 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:34.575304 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/282b2fbe-dcda-488c-90df-3d087cc51a46-kube-api-access-vqwnc" pod "282b2fbe-dcda-488c-90df-3d087cc51a46" (UID: "282b2fbe-dcda-488c-90df-3d087cc51a46"). InnerVolumeSpecName "kube-api-access-vqwnc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:50:34.672798 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:34.672763 4421 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/282b2fbe-dcda-488c-90df-3d087cc51a46-workload-certs\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:50:34.672798 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:34.672792 4421 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/282b2fbe-dcda-488c-90df-3d087cc51a46-istio-data\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:50:34.672798 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:34.672802 4421 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/282b2fbe-dcda-488c-90df-3d087cc51a46-workload-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:50:34.673034 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:34.672811 4421 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/282b2fbe-dcda-488c-90df-3d087cc51a46-credential-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:50:34.673034 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:34.672821 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vqwnc\" (UniqueName: \"kubernetes.io/projected/282b2fbe-dcda-488c-90df-3d087cc51a46-kube-api-access-vqwnc\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:50:34.673034 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:34.672831 4421 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/282b2fbe-dcda-488c-90df-3d087cc51a46-istio-envoy\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:50:34.673034 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:34.672840 4421 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/282b2fbe-dcda-488c-90df-3d087cc51a46-istiod-ca-cert\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:50:34.673034 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:34.672848 4421 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/282b2fbe-dcda-488c-90df-3d087cc51a46-istio-token\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:50:34.673034 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:34.672855 4421 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/282b2fbe-dcda-488c-90df-3d087cc51a46-istio-podinfo\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:50:34.749372 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:34.749334 4421 patch_prober.go:29] interesting pod/e2e-derive-4aec6006-openshift-default-77f675f877-fjbkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Nov 03 00:50:34.749775 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:34.749393 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-4aec6006-openshift-default-77f675f877-fjbkq" podUID="07184f66-00b4-45e3-a60e-503b9dd4b6fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Nov 03 00:50:34.936072 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:34.935985 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-f7680d-59ddf48c45-8xgwl"] Nov 03 00:50:34.936731 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:34.936708 4421 cpu_manager.go:414] "containerMap: removing container" podUID="ff0c52dc-11c3-476b-b4c1-d9575ca5129b" containerName="payload-processing" Nov 03 00:50:34.936855 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:34.936845 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff0c52dc-11c3-476b-b4c1-d9575ca5129b" containerName="payload-processing" Nov 03 00:50:34.936921 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:34.936913 4421 cpu_manager.go:414] "containerMap: removing container" podUID="282b2fbe-dcda-488c-90df-3d087cc51a46" containerName="istio-proxy" Nov 03 00:50:34.937013 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:34.937003 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="282b2fbe-dcda-488c-90df-3d087cc51a46" containerName="istio-proxy" Nov 03 00:50:34.937066 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:34.937059 4421 cpu_manager.go:414] "containerMap: removing container" podUID="46553f3d-e4a6-4ef4-a601-c7c01b9f057e" containerName="payload-pre-processing" Nov 03 00:50:34.937121 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:34.937112 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46553f3d-e4a6-4ef4-a601-c7c01b9f057e" containerName="payload-pre-processing" Nov 03 00:50:34.937216 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:34.937208 4421 cpu_manager.go:414] "containerMap: removing container" podUID="282b2fbe-dcda-488c-90df-3d087cc51a46" containerName="istio-proxy" Nov 03 00:50:34.937554 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:34.937266 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="282b2fbe-dcda-488c-90df-3d087cc51a46" containerName="istio-proxy" Nov 03 00:50:34.937554 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:34.937311 4421 cpu_manager.go:414] "containerMap: removing container" podUID="f2157bb1-e415-40a5-89da-e21b8564e471" containerName="payload-pre-processing" Nov 03 00:50:34.937554 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:34.937319 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2157bb1-e415-40a5-89da-e21b8564e471" containerName="payload-pre-processing" Nov 03 00:50:34.937554 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:34.937447 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2157bb1-e415-40a5-89da-e21b8564e471" containerName="payload-pre-processing" Nov 03 00:50:34.937554 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:34.937472 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff0c52dc-11c3-476b-b4c1-d9575ca5129b" containerName="payload-processing" Nov 03 00:50:34.937554 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:34.937483 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="282b2fbe-dcda-488c-90df-3d087cc51a46" containerName="istio-proxy" Nov 03 00:50:34.937554 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:34.937490 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="46553f3d-e4a6-4ef4-a601-c7c01b9f057e" containerName="payload-pre-processing" Nov 03 00:50:34.937554 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:34.937499 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="282b2fbe-dcda-488c-90df-3d087cc51a46" containerName="istio-proxy" Nov 03 00:50:34.942659 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:34.942540 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-f7680d-59ddf48c45-8xgwl"] Nov 03 00:50:34.942812 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:34.942640 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-f7680d-59ddf48c45-8xgwl" Nov 03 00:50:34.944003 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:34.943980 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-f7680d-67845fdfd7-r56nd"] Nov 03 00:50:34.947980 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:34.947957 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-f7680d-67845fdfd7-r56nd" Nov 03 00:50:34.951193 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:34.951163 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-f7680d\"" Nov 03 00:50:34.955854 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:34.955835 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-f7680d-dockercfg-9dmls\"" Nov 03 00:50:34.966837 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:34.966803 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-f7680d-67845fdfd7-r56nd"] Nov 03 00:50:34.974689 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:34.974663 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-e9b9924e-gw-openshift-default-58b595c4df-xtlrf" Nov 03 00:50:34.975937 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:34.975911 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z96wd\" (UniqueName: \"kubernetes.io/projected/a5e0534c-5a31-4081-a049-5777d11925e1-kube-api-access-z96wd\") pod \"payload-processing-e2e-worker-w2-f7680d-59ddf48c45-8xgwl\" (UID: \"a5e0534c-5a31-4081-a049-5777d11925e1\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-f7680d-59ddf48c45-8xgwl" Nov 03 00:50:34.975995 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:34.975959 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08dc3816-9d73-4dcf-abff-cef7f40b0009-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-f7680d-67845fdfd7-r56nd\" (UID: \"08dc3816-9d73-4dcf-abff-cef7f40b0009\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-f7680d-67845fdfd7-r56nd" Nov 03 00:50:34.976069 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:34.976049 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhb8b\" (UniqueName: \"kubernetes.io/projected/08dc3816-9d73-4dcf-abff-cef7f40b0009-kube-api-access-qhb8b\") pod \"payload-pre-processing-e2e-worker-w2-f7680d-67845fdfd7-r56nd\" (UID: \"08dc3816-9d73-4dcf-abff-cef7f40b0009\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-f7680d-67845fdfd7-r56nd" Nov 03 00:50:34.976116 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:34.976097 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5e0534c-5a31-4081-a049-5777d11925e1-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-f7680d-59ddf48c45-8xgwl\" (UID: \"a5e0534c-5a31-4081-a049-5777d11925e1\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-f7680d-59ddf48c45-8xgwl" Nov 03 00:50:34.976178 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:34.976156 4421 generic.go:356] "Generic (PLEG): container finished" podID="282b2fbe-dcda-488c-90df-3d087cc51a46" containerID="7c66ee9cd90d493f3c00a909c069dc858015c05ac67e8eb7358f601faf5c22e1" exitCode=0 Nov 03 00:50:34.976241 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:34.976220 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-e9b9924e-gw-openshift-default-58b595c4df-xtlrf" event={"ID":"282b2fbe-dcda-488c-90df-3d087cc51a46","Type":"ContainerDied","Data":"7c66ee9cd90d493f3c00a909c069dc858015c05ac67e8eb7358f601faf5c22e1"} Nov 03 00:50:34.976308 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:34.976289 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-e9b9924e-gw-openshift-default-58b595c4df-xtlrf" event={"ID":"282b2fbe-dcda-488c-90df-3d087cc51a46","Type":"ContainerDied","Data":"e5038c98994f536848cca99b0cd431daa0b1411dbb50eaf7672d961958406c90"} Nov 03 00:50:34.976361 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:34.976324 4421 scope.go:122] "RemoveContainer" containerID="7c66ee9cd90d493f3c00a909c069dc858015c05ac67e8eb7358f601faf5c22e1" Nov 03 00:50:34.987443 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:34.987425 4421 scope.go:122] "RemoveContainer" containerID="88b8332b90873e266f07d615d4357cca294363854473c58117db245af97d5c26" Nov 03 00:50:35.000204 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:35.000183 4421 scope.go:122] "RemoveContainer" containerID="7c66ee9cd90d493f3c00a909c069dc858015c05ac67e8eb7358f601faf5c22e1" Nov 03 00:50:35.000600 ip-10-0-133-217 kubenswrapper[4421]: E1103 00:50:35.000578 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c66ee9cd90d493f3c00a909c069dc858015c05ac67e8eb7358f601faf5c22e1\": container with ID starting with 7c66ee9cd90d493f3c00a909c069dc858015c05ac67e8eb7358f601faf5c22e1 not found: ID does not exist" containerID="7c66ee9cd90d493f3c00a909c069dc858015c05ac67e8eb7358f601faf5c22e1" Nov 03 00:50:35.000711 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:35.000691 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c66ee9cd90d493f3c00a909c069dc858015c05ac67e8eb7358f601faf5c22e1"} err="failed to get container status \"7c66ee9cd90d493f3c00a909c069dc858015c05ac67e8eb7358f601faf5c22e1\": rpc error: code = NotFound desc = could not find container \"7c66ee9cd90d493f3c00a909c069dc858015c05ac67e8eb7358f601faf5c22e1\": container with ID starting with 7c66ee9cd90d493f3c00a909c069dc858015c05ac67e8eb7358f601faf5c22e1 not found: ID does not exist" Nov 03 00:50:35.000774 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:35.000764 4421 scope.go:122] "RemoveContainer" containerID="88b8332b90873e266f07d615d4357cca294363854473c58117db245af97d5c26" Nov 03 00:50:35.001075 ip-10-0-133-217 kubenswrapper[4421]: E1103 00:50:35.001056 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88b8332b90873e266f07d615d4357cca294363854473c58117db245af97d5c26\": container with ID starting with 88b8332b90873e266f07d615d4357cca294363854473c58117db245af97d5c26 not found: ID does not exist" containerID="88b8332b90873e266f07d615d4357cca294363854473c58117db245af97d5c26" Nov 03 00:50:35.001126 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:35.001085 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88b8332b90873e266f07d615d4357cca294363854473c58117db245af97d5c26"} err="failed to get container status \"88b8332b90873e266f07d615d4357cca294363854473c58117db245af97d5c26\": rpc error: code = NotFound desc = could not find container \"88b8332b90873e266f07d615d4357cca294363854473c58117db245af97d5c26\": container with ID starting with 88b8332b90873e266f07d615d4357cca294363854473c58117db245af97d5c26 not found: ID does not exist" Nov 03 00:50:35.033398 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:35.033346 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-e9b9924e-gw-openshift-default-58b595c4df-xtlrf"] Nov 03 00:50:35.036910 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:35.036864 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-e9b9924e-gw-openshift-default-58b595c4df-xtlrf"] Nov 03 00:50:35.078433 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:35.077914 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z96wd\" (UniqueName: \"kubernetes.io/projected/a5e0534c-5a31-4081-a049-5777d11925e1-kube-api-access-z96wd\") pod \"payload-processing-e2e-worker-w2-f7680d-59ddf48c45-8xgwl\" (UID: \"a5e0534c-5a31-4081-a049-5777d11925e1\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-f7680d-59ddf48c45-8xgwl" Nov 03 00:50:35.078625 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:35.078467 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08dc3816-9d73-4dcf-abff-cef7f40b0009-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-f7680d-67845fdfd7-r56nd\" (UID: \"08dc3816-9d73-4dcf-abff-cef7f40b0009\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-f7680d-67845fdfd7-r56nd" Nov 03 00:50:35.078625 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:35.078578 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhb8b\" (UniqueName: \"kubernetes.io/projected/08dc3816-9d73-4dcf-abff-cef7f40b0009-kube-api-access-qhb8b\") pod \"payload-pre-processing-e2e-worker-w2-f7680d-67845fdfd7-r56nd\" (UID: \"08dc3816-9d73-4dcf-abff-cef7f40b0009\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-f7680d-67845fdfd7-r56nd" Nov 03 00:50:35.078704 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:35.078628 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5e0534c-5a31-4081-a049-5777d11925e1-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-f7680d-59ddf48c45-8xgwl\" (UID: \"a5e0534c-5a31-4081-a049-5777d11925e1\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-f7680d-59ddf48c45-8xgwl" Nov 03 00:50:35.079421 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:35.079401 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08dc3816-9d73-4dcf-abff-cef7f40b0009-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-f7680d-67845fdfd7-r56nd\" (UID: \"08dc3816-9d73-4dcf-abff-cef7f40b0009\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-f7680d-67845fdfd7-r56nd" Nov 03 00:50:35.079499 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:35.079477 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5e0534c-5a31-4081-a049-5777d11925e1-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-f7680d-59ddf48c45-8xgwl\" (UID: \"a5e0534c-5a31-4081-a049-5777d11925e1\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-f7680d-59ddf48c45-8xgwl" Nov 03 00:50:35.087997 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:35.087972 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhb8b\" (UniqueName: \"kubernetes.io/projected/08dc3816-9d73-4dcf-abff-cef7f40b0009-kube-api-access-qhb8b\") pod \"payload-pre-processing-e2e-worker-w2-f7680d-67845fdfd7-r56nd\" (UID: \"08dc3816-9d73-4dcf-abff-cef7f40b0009\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-f7680d-67845fdfd7-r56nd" Nov 03 00:50:35.088470 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:35.088453 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z96wd\" (UniqueName: \"kubernetes.io/projected/a5e0534c-5a31-4081-a049-5777d11925e1-kube-api-access-z96wd\") pod \"payload-processing-e2e-worker-w2-f7680d-59ddf48c45-8xgwl\" (UID: \"a5e0534c-5a31-4081-a049-5777d11925e1\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-f7680d-59ddf48c45-8xgwl" Nov 03 00:50:35.262295 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:35.262230 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-f7680d-59ddf48c45-8xgwl" Nov 03 00:50:35.273449 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:35.273410 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-f7680d-67845fdfd7-r56nd" Nov 03 00:50:35.570596 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:35.570571 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-f7680d-59ddf48c45-8xgwl"] Nov 03 00:50:35.573636 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:50:35.573606 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5e0534c_5a31_4081_a049_5777d11925e1.slice/crio-08e453e4b63892619dfe14e1415e8c0bb83f43cbc77d2b43f33281e78e6fca57 WatchSource:0}: Error finding container 08e453e4b63892619dfe14e1415e8c0bb83f43cbc77d2b43f33281e78e6fca57: Status 404 returned error can't find the container with id 08e453e4b63892619dfe14e1415e8c0bb83f43cbc77d2b43f33281e78e6fca57 Nov 03 00:50:35.574904 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:35.574878 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-f7680d-67845fdfd7-r56nd"] Nov 03 00:50:35.577974 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:50:35.577946 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08dc3816_9d73_4dcf_abff_cef7f40b0009.slice/crio-2846f83e2ef2deb7fc8dcd8b5409f8391345a56e6dbb64918be1d5e3f06e13cd WatchSource:0}: Error finding container 2846f83e2ef2deb7fc8dcd8b5409f8391345a56e6dbb64918be1d5e3f06e13cd: Status 404 returned error can't find the container with id 2846f83e2ef2deb7fc8dcd8b5409f8391345a56e6dbb64918be1d5e3f06e13cd Nov 03 00:50:35.748953 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:35.748918 4421 patch_prober.go:29] interesting pod/e2e-derive-4aec6006-openshift-default-77f675f877-fjbkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Nov 03 00:50:35.749112 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:35.748975 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-4aec6006-openshift-default-77f675f877-fjbkq" podUID="07184f66-00b4-45e3-a60e-503b9dd4b6fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Nov 03 00:50:35.759906 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:35.759873 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-83755a-67c54f67f5-mxhfv"] Nov 03 00:50:35.763690 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:35.763614 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-83755a-67c54f67f5-mxhfv" Nov 03 00:50:35.771429 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:35.771257 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-83755a\"" Nov 03 00:50:35.771429 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:35.771372 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-83755a-dockercfg-74c4d\"" Nov 03 00:50:35.780310 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:35.778833 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-83755a-67c54f67f5-mxhfv"] Nov 03 00:50:35.785842 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:35.785811 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-83755a-7dfdb9c56c-l78nm"] Nov 03 00:50:35.787343 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:35.787318 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5b4f\" (UniqueName: \"kubernetes.io/projected/2fb9b19f-9286-4d24-813c-be4055c5e8e2-kube-api-access-m5b4f\") pod \"payload-processing-e2e-worker-w3-83755a-67c54f67f5-mxhfv\" (UID: \"2fb9b19f-9286-4d24-813c-be4055c5e8e2\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-83755a-67c54f67f5-mxhfv" Nov 03 00:50:35.787435 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:35.787373 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2fb9b19f-9286-4d24-813c-be4055c5e8e2-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-83755a-67c54f67f5-mxhfv\" (UID: \"2fb9b19f-9286-4d24-813c-be4055c5e8e2\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-83755a-67c54f67f5-mxhfv" Nov 03 00:50:35.790226 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:35.790206 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-83755a-7dfdb9c56c-l78nm" Nov 03 00:50:35.839380 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:35.837336 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-83755a-7dfdb9c56c-l78nm"] Nov 03 00:50:35.888575 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:35.888549 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5b4f\" (UniqueName: \"kubernetes.io/projected/2fb9b19f-9286-4d24-813c-be4055c5e8e2-kube-api-access-m5b4f\") pod \"payload-processing-e2e-worker-w3-83755a-67c54f67f5-mxhfv\" (UID: \"2fb9b19f-9286-4d24-813c-be4055c5e8e2\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-83755a-67c54f67f5-mxhfv" Nov 03 00:50:35.888677 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:35.888593 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2fb9b19f-9286-4d24-813c-be4055c5e8e2-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-83755a-67c54f67f5-mxhfv\" (UID: \"2fb9b19f-9286-4d24-813c-be4055c5e8e2\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-83755a-67c54f67f5-mxhfv" Nov 03 00:50:35.888677 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:35.888650 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7b858b0-7dd7-42f7-bb5f-22920b707d4c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-83755a-7dfdb9c56c-l78nm\" (UID: \"f7b858b0-7dd7-42f7-bb5f-22920b707d4c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-83755a-7dfdb9c56c-l78nm" Nov 03 00:50:35.888751 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:35.888687 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scszf\" (UniqueName: \"kubernetes.io/projected/f7b858b0-7dd7-42f7-bb5f-22920b707d4c-kube-api-access-scszf\") pod \"payload-pre-processing-e2e-worker-w3-83755a-7dfdb9c56c-l78nm\" (UID: \"f7b858b0-7dd7-42f7-bb5f-22920b707d4c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-83755a-7dfdb9c56c-l78nm" Nov 03 00:50:35.889385 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:35.889366 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2fb9b19f-9286-4d24-813c-be4055c5e8e2-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-83755a-67c54f67f5-mxhfv\" (UID: \"2fb9b19f-9286-4d24-813c-be4055c5e8e2\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-83755a-67c54f67f5-mxhfv" Nov 03 00:50:35.899032 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:35.899008 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5b4f\" (UniqueName: \"kubernetes.io/projected/2fb9b19f-9286-4d24-813c-be4055c5e8e2-kube-api-access-m5b4f\") pod \"payload-processing-e2e-worker-w3-83755a-67c54f67f5-mxhfv\" (UID: \"2fb9b19f-9286-4d24-813c-be4055c5e8e2\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-83755a-67c54f67f5-mxhfv" Nov 03 00:50:35.988922 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:35.988888 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="282b2fbe-dcda-488c-90df-3d087cc51a46" path="/var/lib/kubelet/pods/282b2fbe-dcda-488c-90df-3d087cc51a46/volumes" Nov 03 00:50:35.989613 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:35.989589 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2157bb1-e415-40a5-89da-e21b8564e471" path="/var/lib/kubelet/pods/f2157bb1-e415-40a5-89da-e21b8564e471/volumes" Nov 03 00:50:35.989811 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:35.989782 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7b858b0-7dd7-42f7-bb5f-22920b707d4c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-83755a-7dfdb9c56c-l78nm\" (UID: \"f7b858b0-7dd7-42f7-bb5f-22920b707d4c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-83755a-7dfdb9c56c-l78nm" Nov 03 00:50:35.989885 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:35.989865 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scszf\" (UniqueName: \"kubernetes.io/projected/f7b858b0-7dd7-42f7-bb5f-22920b707d4c-kube-api-access-scszf\") pod \"payload-pre-processing-e2e-worker-w3-83755a-7dfdb9c56c-l78nm\" (UID: \"f7b858b0-7dd7-42f7-bb5f-22920b707d4c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-83755a-7dfdb9c56c-l78nm" Nov 03 00:50:35.990444 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:35.990417 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff0c52dc-11c3-476b-b4c1-d9575ca5129b" path="/var/lib/kubelet/pods/ff0c52dc-11c3-476b-b4c1-d9575ca5129b/volumes" Nov 03 00:50:35.991167 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:35.991144 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7b858b0-7dd7-42f7-bb5f-22920b707d4c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-83755a-7dfdb9c56c-l78nm\" (UID: \"f7b858b0-7dd7-42f7-bb5f-22920b707d4c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-83755a-7dfdb9c56c-l78nm" Nov 03 00:50:35.994365 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:35.993636 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-f7680d-59ddf48c45-8xgwl" event={"ID":"a5e0534c-5a31-4081-a049-5777d11925e1","Type":"ContainerStarted","Data":"b31d098b3dd0a05021850986a7657443452ccc5d2d5863072ecd1b3cf0881c0d"} Nov 03 00:50:35.994365 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:35.993685 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-f7680d-59ddf48c45-8xgwl" event={"ID":"a5e0534c-5a31-4081-a049-5777d11925e1","Type":"ContainerStarted","Data":"08e453e4b63892619dfe14e1415e8c0bb83f43cbc77d2b43f33281e78e6fca57"} Nov 03 00:50:35.994365 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:35.994326 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-f7680d-59ddf48c45-8xgwl" Nov 03 00:50:35.999667 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:35.998774 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-f7680d-67845fdfd7-r56nd" event={"ID":"08dc3816-9d73-4dcf-abff-cef7f40b0009","Type":"ContainerStarted","Data":"ac6c24b89e688aaa135a985a9e843b8e06ba6aa6dc64ff236fd40f3ee8c4cdfb"} Nov 03 00:50:35.999667 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:35.998826 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-f7680d-67845fdfd7-r56nd" event={"ID":"08dc3816-9d73-4dcf-abff-cef7f40b0009","Type":"ContainerStarted","Data":"2846f83e2ef2deb7fc8dcd8b5409f8391345a56e6dbb64918be1d5e3f06e13cd"} Nov 03 00:50:35.999667 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:35.999567 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-f7680d-67845fdfd7-r56nd" Nov 03 00:50:36.001139 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:36.001090 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-scszf\" (UniqueName: \"kubernetes.io/projected/f7b858b0-7dd7-42f7-bb5f-22920b707d4c-kube-api-access-scszf\") pod \"payload-pre-processing-e2e-worker-w3-83755a-7dfdb9c56c-l78nm\" (UID: \"f7b858b0-7dd7-42f7-bb5f-22920b707d4c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-83755a-7dfdb9c56c-l78nm" Nov 03 00:50:36.003416 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:36.003394 4421 generic.go:356] "Generic (PLEG): container finished" podID="4d793f1a-5d2f-4435-aee8-38af07f46693" containerID="cd6ded1d86651bfdeb6b404d49b9400452cfbfbfcb2fe6c1187633c91e45143d" exitCode=0 Nov 03 00:50:36.003520 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:36.003435 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c83251ee-openshift-default-5567cd685c-69vg9" event={"ID":"4d793f1a-5d2f-4435-aee8-38af07f46693","Type":"ContainerDied","Data":"cd6ded1d86651bfdeb6b404d49b9400452cfbfbfcb2fe6c1187633c91e45143d"} Nov 03 00:50:36.003520 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:36.003472 4421 scope.go:122] "RemoveContainer" containerID="8d67ac4ad544f7c7b7b090c212723c6d805e6eff70c4eca87ea2efcc05237760" Nov 03 00:50:36.041099 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:36.039382 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-f7680d-59ddf48c45-8xgwl" podStartSLOduration=2.0393546 podStartE2EDuration="2.0393546s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:50:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:50:36.009994115 +0000 UTC m=+578.180997212" watchObservedRunningTime="2026-11-03 00:50:36.0393546 +0000 UTC m=+578.210357730" Nov 03 00:50:36.049101 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:36.049049 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-f7680d-67845fdfd7-r56nd" podStartSLOduration=2.049033019 podStartE2EDuration="2.049033019s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:50:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:50:36.048584359 +0000 UTC m=+578.219587456" watchObservedRunningTime="2026-11-03 00:50:36.049033019 +0000 UTC m=+578.220036135" Nov 03 00:50:36.065411 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:36.065383 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c83251ee-openshift-default-5567cd685c-69vg9" Nov 03 00:50:36.079122 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:36.079067 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-83755a-67c54f67f5-mxhfv" Nov 03 00:50:36.090814 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:36.090748 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d793f1a-5d2f-4435-aee8-38af07f46693-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d793f1a-5d2f-4435-aee8-38af07f46693-workload-certs\") pod \"4d793f1a-5d2f-4435-aee8-38af07f46693\" (UID: \"4d793f1a-5d2f-4435-aee8-38af07f46693\") " Nov 03 00:50:36.090814 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:36.090783 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d793f1a-5d2f-4435-aee8-38af07f46693-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d793f1a-5d2f-4435-aee8-38af07f46693-workload-socket\") pod \"4d793f1a-5d2f-4435-aee8-38af07f46693\" (UID: \"4d793f1a-5d2f-4435-aee8-38af07f46693\") " Nov 03 00:50:36.090965 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:36.090821 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d793f1a-5d2f-4435-aee8-38af07f46693-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d793f1a-5d2f-4435-aee8-38af07f46693-istio-envoy\") pod \"4d793f1a-5d2f-4435-aee8-38af07f46693\" (UID: \"4d793f1a-5d2f-4435-aee8-38af07f46693\") " Nov 03 00:50:36.090965 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:36.090856 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d793f1a-5d2f-4435-aee8-38af07f46693-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d793f1a-5d2f-4435-aee8-38af07f46693-credential-socket\") pod \"4d793f1a-5d2f-4435-aee8-38af07f46693\" (UID: \"4d793f1a-5d2f-4435-aee8-38af07f46693\") " Nov 03 00:50:36.090965 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:36.090883 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d793f1a-5d2f-4435-aee8-38af07f46693-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d793f1a-5d2f-4435-aee8-38af07f46693-istiod-ca-cert\") pod \"4d793f1a-5d2f-4435-aee8-38af07f46693\" (UID: \"4d793f1a-5d2f-4435-aee8-38af07f46693\") " Nov 03 00:50:36.090965 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:36.090949 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d793f1a-5d2f-4435-aee8-38af07f46693-istio-token\" (UniqueName: \"kubernetes.io/projected/4d793f1a-5d2f-4435-aee8-38af07f46693-istio-token\") pod \"4d793f1a-5d2f-4435-aee8-38af07f46693\" (UID: \"4d793f1a-5d2f-4435-aee8-38af07f46693\") " Nov 03 00:50:36.091101 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:36.090990 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d793f1a-5d2f-4435-aee8-38af07f46693-kube-api-access-b9fcc\" (UniqueName: \"kubernetes.io/projected/4d793f1a-5d2f-4435-aee8-38af07f46693-kube-api-access-b9fcc\") pod \"4d793f1a-5d2f-4435-aee8-38af07f46693\" (UID: \"4d793f1a-5d2f-4435-aee8-38af07f46693\") " Nov 03 00:50:36.091101 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:36.090992 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d793f1a-5d2f-4435-aee8-38af07f46693-workload-certs" pod "4d793f1a-5d2f-4435-aee8-38af07f46693" (UID: "4d793f1a-5d2f-4435-aee8-38af07f46693"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:50:36.091101 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:36.091028 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d793f1a-5d2f-4435-aee8-38af07f46693-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d793f1a-5d2f-4435-aee8-38af07f46693-istio-data\") pod \"4d793f1a-5d2f-4435-aee8-38af07f46693\" (UID: \"4d793f1a-5d2f-4435-aee8-38af07f46693\") " Nov 03 00:50:36.091101 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:36.091087 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4d793f1a-5d2f-4435-aee8-38af07f46693-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d793f1a-5d2f-4435-aee8-38af07f46693-istio-podinfo\") pod \"4d793f1a-5d2f-4435-aee8-38af07f46693\" (UID: \"4d793f1a-5d2f-4435-aee8-38af07f46693\") " Nov 03 00:50:36.091232 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:36.091144 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d793f1a-5d2f-4435-aee8-38af07f46693-credential-socket" pod "4d793f1a-5d2f-4435-aee8-38af07f46693" (UID: "4d793f1a-5d2f-4435-aee8-38af07f46693"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:50:36.091296 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:36.091256 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d793f1a-5d2f-4435-aee8-38af07f46693-workload-socket" pod "4d793f1a-5d2f-4435-aee8-38af07f46693" (UID: "4d793f1a-5d2f-4435-aee8-38af07f46693"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:50:36.091397 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:36.091383 4421 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d793f1a-5d2f-4435-aee8-38af07f46693-workload-certs\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:50:36.091460 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:36.091400 4421 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d793f1a-5d2f-4435-aee8-38af07f46693-workload-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:50:36.091460 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:36.091409 4421 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d793f1a-5d2f-4435-aee8-38af07f46693-credential-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:50:36.092061 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:36.092037 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d793f1a-5d2f-4435-aee8-38af07f46693-istiod-ca-cert" pod "4d793f1a-5d2f-4435-aee8-38af07f46693" (UID: "4d793f1a-5d2f-4435-aee8-38af07f46693"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:50:36.092981 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:36.092949 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d793f1a-5d2f-4435-aee8-38af07f46693-istio-data" pod "4d793f1a-5d2f-4435-aee8-38af07f46693" (UID: "4d793f1a-5d2f-4435-aee8-38af07f46693"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:50:36.094993 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:36.094972 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d793f1a-5d2f-4435-aee8-38af07f46693-istio-envoy" pod "4d793f1a-5d2f-4435-aee8-38af07f46693" (UID: "4d793f1a-5d2f-4435-aee8-38af07f46693"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:50:36.096123 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:36.096089 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d793f1a-5d2f-4435-aee8-38af07f46693-istio-token" pod "4d793f1a-5d2f-4435-aee8-38af07f46693" (UID: "4d793f1a-5d2f-4435-aee8-38af07f46693"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:50:36.096298 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:36.096285 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d793f1a-5d2f-4435-aee8-38af07f46693-kube-api-access-b9fcc" pod "4d793f1a-5d2f-4435-aee8-38af07f46693" (UID: "4d793f1a-5d2f-4435-aee8-38af07f46693"). InnerVolumeSpecName "kube-api-access-b9fcc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:50:36.096764 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:36.096742 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4d793f1a-5d2f-4435-aee8-38af07f46693-istio-podinfo" pod "4d793f1a-5d2f-4435-aee8-38af07f46693" (UID: "4d793f1a-5d2f-4435-aee8-38af07f46693"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:50:36.169573 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:36.169537 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-83755a-7dfdb9c56c-l78nm" Nov 03 00:50:36.236150 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:36.236010 4421 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d793f1a-5d2f-4435-aee8-38af07f46693-istio-data\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:50:36.236150 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:36.236054 4421 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d793f1a-5d2f-4435-aee8-38af07f46693-istio-podinfo\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:50:36.236150 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:36.236068 4421 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d793f1a-5d2f-4435-aee8-38af07f46693-istio-envoy\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:50:36.236150 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:36.236084 4421 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d793f1a-5d2f-4435-aee8-38af07f46693-istiod-ca-cert\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:50:36.236150 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:36.236098 4421 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d793f1a-5d2f-4435-aee8-38af07f46693-istio-token\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:50:36.236150 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:36.236112 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9fcc\" (UniqueName: \"kubernetes.io/projected/4d793f1a-5d2f-4435-aee8-38af07f46693-kube-api-access-b9fcc\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:50:36.433501 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:36.433398 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-83755a-67c54f67f5-mxhfv"] Nov 03 00:50:36.437872 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:50:36.437826 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fb9b19f_9286_4d24_813c_be4055c5e8e2.slice/crio-29f2ab9f42c1c95f9b34794cdaf292a86811aed821bc8448063bfb25dec2e929 WatchSource:0}: Error finding container 29f2ab9f42c1c95f9b34794cdaf292a86811aed821bc8448063bfb25dec2e929: Status 404 returned error can't find the container with id 29f2ab9f42c1c95f9b34794cdaf292a86811aed821bc8448063bfb25dec2e929 Nov 03 00:50:36.553530 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:36.553053 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-83755a-7dfdb9c56c-l78nm"] Nov 03 00:50:36.558307 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:50:36.558241 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7b858b0_7dd7_42f7_bb5f_22920b707d4c.slice/crio-aa34f1f761d93823e852c656579c3be69fc113d8ffe5f6e9caae0c0a2acf5a0f WatchSource:0}: Error finding container aa34f1f761d93823e852c656579c3be69fc113d8ffe5f6e9caae0c0a2acf5a0f: Status 404 returned error can't find the container with id aa34f1f761d93823e852c656579c3be69fc113d8ffe5f6e9caae0c0a2acf5a0f Nov 03 00:50:36.749690 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:36.749647 4421 patch_prober.go:29] interesting pod/e2e-derive-4aec6006-openshift-default-77f675f877-fjbkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Nov 03 00:50:36.749876 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:36.749702 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-4aec6006-openshift-default-77f675f877-fjbkq" podUID="07184f66-00b4-45e3-a60e-503b9dd4b6fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Nov 03 00:50:36.762617 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:36.762584 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4aec6006-rlwf2"] Nov 03 00:50:36.764477 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:36.764457 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4aec6006-rlwf2"] Nov 03 00:50:36.918525 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:36.918423 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-4aec6006-openshift-default-77f675f877-fjbkq"] Nov 03 00:50:36.918729 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:36.918703 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-4aec6006-openshift-default-77f675f877-fjbkq" podUID="07184f66-00b4-45e3-a60e-503b9dd4b6fa" containerName="istio-proxy" containerID="cri-o://1f42a74cd65307be2d561bb30b71f24440123834604818be21ae4f5f41884f98" gracePeriod=30 Nov 03 00:50:37.011395 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:37.011346 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c83251ee-openshift-default-5567cd685c-69vg9" event={"ID":"4d793f1a-5d2f-4435-aee8-38af07f46693","Type":"ContainerDied","Data":"a2fd23a31d7ea91e15e0ca2e56563a1b0325cfe7e786807e2afa969f71c7c1ac"} Nov 03 00:50:37.011580 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:37.011425 4421 scope.go:122] "RemoveContainer" containerID="cd6ded1d86651bfdeb6b404d49b9400452cfbfbfcb2fe6c1187633c91e45143d" Nov 03 00:50:37.011580 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:37.011373 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c83251ee-openshift-default-5567cd685c-69vg9" Nov 03 00:50:37.016126 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:37.016019 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-83755a-7dfdb9c56c-l78nm" event={"ID":"f7b858b0-7dd7-42f7-bb5f-22920b707d4c","Type":"ContainerStarted","Data":"5a0b873bf7ce4f774466861e5d89a026203e92953c2cee27156f614ed729ea95"} Nov 03 00:50:37.016126 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:37.016082 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-83755a-7dfdb9c56c-l78nm" event={"ID":"f7b858b0-7dd7-42f7-bb5f-22920b707d4c","Type":"ContainerStarted","Data":"aa34f1f761d93823e852c656579c3be69fc113d8ffe5f6e9caae0c0a2acf5a0f"} Nov 03 00:50:37.016347 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:37.016128 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-83755a-7dfdb9c56c-l78nm" Nov 03 00:50:37.018760 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:37.018637 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-83755a-67c54f67f5-mxhfv" event={"ID":"2fb9b19f-9286-4d24-813c-be4055c5e8e2","Type":"ContainerStarted","Data":"56488b5383deb40352cd71f0573db3fb1712d8045cc1f2d914bbcdcd990d3144"} Nov 03 00:50:37.018760 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:37.018693 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-83755a-67c54f67f5-mxhfv" event={"ID":"2fb9b19f-9286-4d24-813c-be4055c5e8e2","Type":"ContainerStarted","Data":"29f2ab9f42c1c95f9b34794cdaf292a86811aed821bc8448063bfb25dec2e929"} Nov 03 00:50:37.020196 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:37.020139 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-83755a-67c54f67f5-mxhfv" Nov 03 00:50:37.092928 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:37.092861 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-83755a-67c54f67f5-mxhfv" podStartSLOduration=2.092842476 podStartE2EDuration="2.092842476s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:50:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:50:37.053508673 +0000 UTC m=+579.224511772" watchObservedRunningTime="2026-11-03 00:50:37.092842476 +0000 UTC m=+579.263845575" Nov 03 00:50:37.094601 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:37.094554 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-83755a-7dfdb9c56c-l78nm" podStartSLOduration=2.094541703 podStartE2EDuration="2.094541703s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:50:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:50:37.09043175 +0000 UTC m=+579.261434860" watchObservedRunningTime="2026-11-03 00:50:37.094541703 +0000 UTC m=+579.265544796" Nov 03 00:50:37.119065 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:37.119024 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-c83251ee-openshift-default-5567cd685c-69vg9"] Nov 03 00:50:37.132165 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:37.132122 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-c83251ee-openshift-default-5567cd685c-69vg9"] Nov 03 00:50:37.521260 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:37.521223 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-2f9a3a-8697684787-rcsf2"] Nov 03 00:50:37.521844 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:37.521815 4421 cpu_manager.go:414] "containerMap: removing container" podUID="4d793f1a-5d2f-4435-aee8-38af07f46693" containerName="istio-proxy" Nov 03 00:50:37.521844 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:37.521836 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d793f1a-5d2f-4435-aee8-38af07f46693" containerName="istio-proxy" Nov 03 00:50:37.522023 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:37.521855 4421 cpu_manager.go:414] "containerMap: removing container" podUID="4d793f1a-5d2f-4435-aee8-38af07f46693" containerName="istio-proxy" Nov 03 00:50:37.522023 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:37.521863 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d793f1a-5d2f-4435-aee8-38af07f46693" containerName="istio-proxy" Nov 03 00:50:37.522122 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:37.522021 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d793f1a-5d2f-4435-aee8-38af07f46693" containerName="istio-proxy" Nov 03 00:50:37.522122 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:37.522036 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d793f1a-5d2f-4435-aee8-38af07f46693" containerName="istio-proxy" Nov 03 00:50:37.526510 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:37.526488 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-2f9a3a-8697684787-rcsf2" Nov 03 00:50:37.548681 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:37.548650 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-2f9a3a\"" Nov 03 00:50:37.551247 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:37.551233 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-2f9a3a-dockercfg-nfwzc\"" Nov 03 00:50:37.572661 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:37.572618 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-2f9a3a-8697684787-rcsf2"] Nov 03 00:50:37.586581 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:37.586543 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-2f9a3a-78c44fb86c-ckvcs"] Nov 03 00:50:37.590102 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:37.590082 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2f9a3a-78c44fb86c-ckvcs" Nov 03 00:50:37.607900 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:37.607863 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-2f9a3a-78c44fb86c-ckvcs"] Nov 03 00:50:37.663878 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:37.663828 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bx6zw\" (UniqueName: \"kubernetes.io/projected/d3b636b3-5a2e-467d-a364-f8c57824521f-kube-api-access-bx6zw\") pod \"payload-pre-processing-e2e-worker-w0-2f9a3a-78c44fb86c-ckvcs\" (UID: \"d3b636b3-5a2e-467d-a364-f8c57824521f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2f9a3a-78c44fb86c-ckvcs" Nov 03 00:50:37.664049 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:37.663881 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2cd8dba-bbe9-4bdf-9fca-94a4cb839ca1-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-2f9a3a-8697684787-rcsf2\" (UID: \"d2cd8dba-bbe9-4bdf-9fca-94a4cb839ca1\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-2f9a3a-8697684787-rcsf2" Nov 03 00:50:37.664049 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:37.663915 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3b636b3-5a2e-467d-a364-f8c57824521f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-2f9a3a-78c44fb86c-ckvcs\" (UID: \"d3b636b3-5a2e-467d-a364-f8c57824521f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2f9a3a-78c44fb86c-ckvcs" Nov 03 00:50:37.664049 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:37.664029 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjst9\" (UniqueName: \"kubernetes.io/projected/d2cd8dba-bbe9-4bdf-9fca-94a4cb839ca1-kube-api-access-rjst9\") pod \"payload-processing-e2e-worker-w0-2f9a3a-8697684787-rcsf2\" (UID: \"d2cd8dba-bbe9-4bdf-9fca-94a4cb839ca1\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-2f9a3a-8697684787-rcsf2" Nov 03 00:50:37.773343 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:37.769831 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjst9\" (UniqueName: \"kubernetes.io/projected/d2cd8dba-bbe9-4bdf-9fca-94a4cb839ca1-kube-api-access-rjst9\") pod \"payload-processing-e2e-worker-w0-2f9a3a-8697684787-rcsf2\" (UID: \"d2cd8dba-bbe9-4bdf-9fca-94a4cb839ca1\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-2f9a3a-8697684787-rcsf2" Nov 03 00:50:37.773343 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:37.769950 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bx6zw\" (UniqueName: \"kubernetes.io/projected/d3b636b3-5a2e-467d-a364-f8c57824521f-kube-api-access-bx6zw\") pod \"payload-pre-processing-e2e-worker-w0-2f9a3a-78c44fb86c-ckvcs\" (UID: \"d3b636b3-5a2e-467d-a364-f8c57824521f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2f9a3a-78c44fb86c-ckvcs" Nov 03 00:50:37.773343 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:37.769981 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2cd8dba-bbe9-4bdf-9fca-94a4cb839ca1-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-2f9a3a-8697684787-rcsf2\" (UID: \"d2cd8dba-bbe9-4bdf-9fca-94a4cb839ca1\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-2f9a3a-8697684787-rcsf2" Nov 03 00:50:37.773343 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:37.770011 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3b636b3-5a2e-467d-a364-f8c57824521f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-2f9a3a-78c44fb86c-ckvcs\" (UID: \"d3b636b3-5a2e-467d-a364-f8c57824521f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2f9a3a-78c44fb86c-ckvcs" Nov 03 00:50:37.773343 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:37.770977 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3b636b3-5a2e-467d-a364-f8c57824521f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-2f9a3a-78c44fb86c-ckvcs\" (UID: \"d3b636b3-5a2e-467d-a364-f8c57824521f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2f9a3a-78c44fb86c-ckvcs" Nov 03 00:50:37.773343 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:37.772238 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2cd8dba-bbe9-4bdf-9fca-94a4cb839ca1-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-2f9a3a-8697684787-rcsf2\" (UID: \"d2cd8dba-bbe9-4bdf-9fca-94a4cb839ca1\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-2f9a3a-8697684787-rcsf2" Nov 03 00:50:37.803006 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:37.802969 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjst9\" (UniqueName: \"kubernetes.io/projected/d2cd8dba-bbe9-4bdf-9fca-94a4cb839ca1-kube-api-access-rjst9\") pod \"payload-processing-e2e-worker-w0-2f9a3a-8697684787-rcsf2\" (UID: \"d2cd8dba-bbe9-4bdf-9fca-94a4cb839ca1\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-2f9a3a-8697684787-rcsf2" Nov 03 00:50:37.818021 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:37.817985 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bx6zw\" (UniqueName: \"kubernetes.io/projected/d3b636b3-5a2e-467d-a364-f8c57824521f-kube-api-access-bx6zw\") pod \"payload-pre-processing-e2e-worker-w0-2f9a3a-78c44fb86c-ckvcs\" (UID: \"d3b636b3-5a2e-467d-a364-f8c57824521f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2f9a3a-78c44fb86c-ckvcs" Nov 03 00:50:37.842519 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:37.842479 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-2f9a3a-8697684787-rcsf2" Nov 03 00:50:37.905724 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:37.905689 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2f9a3a-78c44fb86c-ckvcs" Nov 03 00:50:37.991092 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:37.991055 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d793f1a-5d2f-4435-aee8-38af07f46693" path="/var/lib/kubelet/pods/4d793f1a-5d2f-4435-aee8-38af07f46693/volumes" Nov 03 00:50:37.993760 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:37.993730 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9bc1b51f-1f52-4707-b325-8ebc9afa40ef" path="/var/lib/kubelet/pods/9bc1b51f-1f52-4707-b325-8ebc9afa40ef/volumes" Nov 03 00:50:38.165402 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:38.165366 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-2f9a3a-8697684787-rcsf2"] Nov 03 00:50:38.214360 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:38.214325 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-2f9a3a-78c44fb86c-ckvcs"] Nov 03 00:50:38.225704 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:50:38.225674 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3b636b3_5a2e_467d_a364_f8c57824521f.slice/crio-4cde18fe17352751e54fbe1c8d57e355717ed1e33825f016e943308f467b4fd2 WatchSource:0}: Error finding container 4cde18fe17352751e54fbe1c8d57e355717ed1e33825f016e943308f467b4fd2: Status 404 returned error can't find the container with id 4cde18fe17352751e54fbe1c8d57e355717ed1e33825f016e943308f467b4fd2 Nov 03 00:50:39.042987 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:39.042943 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2f9a3a-78c44fb86c-ckvcs" event={"ID":"d3b636b3-5a2e-467d-a364-f8c57824521f","Type":"ContainerStarted","Data":"12283813be518b911286b92c20e397bc64157fd0595a93b0fe1a238b297563da"} Nov 03 00:50:39.043508 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:39.043013 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2f9a3a-78c44fb86c-ckvcs" Nov 03 00:50:39.043508 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:39.043036 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2f9a3a-78c44fb86c-ckvcs" event={"ID":"d3b636b3-5a2e-467d-a364-f8c57824521f","Type":"ContainerStarted","Data":"4cde18fe17352751e54fbe1c8d57e355717ed1e33825f016e943308f467b4fd2"} Nov 03 00:50:39.044711 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:39.044684 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-2f9a3a-8697684787-rcsf2" event={"ID":"d2cd8dba-bbe9-4bdf-9fca-94a4cb839ca1","Type":"ContainerStarted","Data":"30dfeaf9971d86ed46f702780413f66434be9d5d6f3dc72379b8636c23c4ce0f"} Nov 03 00:50:39.044844 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:39.044724 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-2f9a3a-8697684787-rcsf2" event={"ID":"d2cd8dba-bbe9-4bdf-9fca-94a4cb839ca1","Type":"ContainerStarted","Data":"772e19accab7237d80f147fe70a9857ee131c89af669bb0c68939b8b831e9eb9"} Nov 03 00:50:39.044901 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:39.044855 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-2f9a3a-8697684787-rcsf2" Nov 03 00:50:39.063919 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:39.063857 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2f9a3a-78c44fb86c-ckvcs" podStartSLOduration=2.063841483 podStartE2EDuration="2.063841483s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:50:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:50:39.063212558 +0000 UTC m=+581.234215654" watchObservedRunningTime="2026-11-03 00:50:39.063841483 +0000 UTC m=+581.234844578" Nov 03 00:50:39.081546 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:39.081497 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-2f9a3a-8697684787-rcsf2" podStartSLOduration=2.081482222 podStartE2EDuration="2.081482222s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:50:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:50:39.079586664 +0000 UTC m=+581.250589762" watchObservedRunningTime="2026-11-03 00:50:39.081482222 +0000 UTC m=+581.252485318" Nov 03 00:50:39.357152 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:39.357046 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-4aec6006-5b7d45d8d7-9jwv2" podUID="8c0989d6-ad4a-49ad-9a0e-c01f1ef0e797" containerName="payload-pre-processing" containerID="cri-o://32ae8d2435bbc615574dbf5878fc8c14a0b13163eb76c8d0bc51b43c916d29d1" gracePeriod=25 Nov 03 00:50:39.676564 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:39.676523 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-4aec6006-5b7d45d8d7-9jwv2" Nov 03 00:50:39.794574 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:39.794533 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c0989d6-ad4a-49ad-9a0e-c01f1ef0e797-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c0989d6-ad4a-49ad-9a0e-c01f1ef0e797-plugins-config-volume\") pod \"8c0989d6-ad4a-49ad-9a0e-c01f1ef0e797\" (UID: \"8c0989d6-ad4a-49ad-9a0e-c01f1ef0e797\") " Nov 03 00:50:39.794732 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:39.794641 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c0989d6-ad4a-49ad-9a0e-c01f1ef0e797-kube-api-access-5tkcb\" (UniqueName: \"kubernetes.io/projected/8c0989d6-ad4a-49ad-9a0e-c01f1ef0e797-kube-api-access-5tkcb\") pod \"8c0989d6-ad4a-49ad-9a0e-c01f1ef0e797\" (UID: \"8c0989d6-ad4a-49ad-9a0e-c01f1ef0e797\") " Nov 03 00:50:39.794986 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:39.794961 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c0989d6-ad4a-49ad-9a0e-c01f1ef0e797-plugins-config-volume" pod "8c0989d6-ad4a-49ad-9a0e-c01f1ef0e797" (UID: "8c0989d6-ad4a-49ad-9a0e-c01f1ef0e797"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:50:39.796918 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:39.796890 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c0989d6-ad4a-49ad-9a0e-c01f1ef0e797-kube-api-access-5tkcb" pod "8c0989d6-ad4a-49ad-9a0e-c01f1ef0e797" (UID: "8c0989d6-ad4a-49ad-9a0e-c01f1ef0e797"). InnerVolumeSpecName "kube-api-access-5tkcb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:50:39.895774 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:39.895682 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5tkcb\" (UniqueName: \"kubernetes.io/projected/8c0989d6-ad4a-49ad-9a0e-c01f1ef0e797-kube-api-access-5tkcb\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:50:39.895774 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:39.895727 4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c0989d6-ad4a-49ad-9a0e-c01f1ef0e797-plugins-config-volume\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:50:40.052992 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:40.052954 4421 generic.go:356] "Generic (PLEG): container finished" podID="8c0989d6-ad4a-49ad-9a0e-c01f1ef0e797" containerID="32ae8d2435bbc615574dbf5878fc8c14a0b13163eb76c8d0bc51b43c916d29d1" exitCode=0 Nov 03 00:50:40.053481 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:40.053023 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-4aec6006-5b7d45d8d7-9jwv2" Nov 03 00:50:40.053481 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:40.053042 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-4aec6006-5b7d45d8d7-9jwv2" event={"ID":"8c0989d6-ad4a-49ad-9a0e-c01f1ef0e797","Type":"ContainerDied","Data":"32ae8d2435bbc615574dbf5878fc8c14a0b13163eb76c8d0bc51b43c916d29d1"} Nov 03 00:50:40.053481 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:40.053110 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-4aec6006-5b7d45d8d7-9jwv2" event={"ID":"8c0989d6-ad4a-49ad-9a0e-c01f1ef0e797","Type":"ContainerDied","Data":"86d925ff78c1989eb73e6dcf3f6fa1f87082302fbae9abc7307a315b43ebd6ea"} Nov 03 00:50:40.053481 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:40.053143 4421 scope.go:122] "RemoveContainer" containerID="32ae8d2435bbc615574dbf5878fc8c14a0b13163eb76c8d0bc51b43c916d29d1" Nov 03 00:50:40.063456 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:40.063390 4421 scope.go:122] "RemoveContainer" containerID="32ae8d2435bbc615574dbf5878fc8c14a0b13163eb76c8d0bc51b43c916d29d1" Nov 03 00:50:40.063709 ip-10-0-133-217 kubenswrapper[4421]: E1103 00:50:40.063682 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32ae8d2435bbc615574dbf5878fc8c14a0b13163eb76c8d0bc51b43c916d29d1\": container with ID starting with 32ae8d2435bbc615574dbf5878fc8c14a0b13163eb76c8d0bc51b43c916d29d1 not found: ID does not exist" containerID="32ae8d2435bbc615574dbf5878fc8c14a0b13163eb76c8d0bc51b43c916d29d1" Nov 03 00:50:40.063774 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:40.063725 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32ae8d2435bbc615574dbf5878fc8c14a0b13163eb76c8d0bc51b43c916d29d1"} err="failed to get container status \"32ae8d2435bbc615574dbf5878fc8c14a0b13163eb76c8d0bc51b43c916d29d1\": rpc error: code = NotFound desc = could not find container \"32ae8d2435bbc615574dbf5878fc8c14a0b13163eb76c8d0bc51b43c916d29d1\": container with ID starting with 32ae8d2435bbc615574dbf5878fc8c14a0b13163eb76c8d0bc51b43c916d29d1 not found: ID does not exist" Nov 03 00:50:40.074009 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:40.073985 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-4aec6006-5b7d45d8d7-9jwv2"] Nov 03 00:50:40.084269 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:40.084244 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-4aec6006-5b7d45d8d7-9jwv2"] Nov 03 00:50:41.977783 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:41.977750 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c0989d6-ad4a-49ad-9a0e-c01f1ef0e797" path="/var/lib/kubelet/pods/8c0989d6-ad4a-49ad-9a0e-c01f1ef0e797/volumes" Nov 03 00:50:42.393561 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:42.393534 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-4aec6006-openshift-default-77f675f877-fjbkq" Nov 03 00:50:42.545901 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:42.545871 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07184f66-00b4-45e3-a60e-503b9dd4b6fa-istio-token\" (UniqueName: \"kubernetes.io/projected/07184f66-00b4-45e3-a60e-503b9dd4b6fa-istio-token\") pod \"07184f66-00b4-45e3-a60e-503b9dd4b6fa\" (UID: \"07184f66-00b4-45e3-a60e-503b9dd4b6fa\") " Nov 03 00:50:42.546160 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:42.546147 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07184f66-00b4-45e3-a60e-503b9dd4b6fa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07184f66-00b4-45e3-a60e-503b9dd4b6fa-workload-certs\") pod \"07184f66-00b4-45e3-a60e-503b9dd4b6fa\" (UID: \"07184f66-00b4-45e3-a60e-503b9dd4b6fa\") " Nov 03 00:50:42.546298 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:42.546264 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07184f66-00b4-45e3-a60e-503b9dd4b6fa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07184f66-00b4-45e3-a60e-503b9dd4b6fa-workload-socket\") pod \"07184f66-00b4-45e3-a60e-503b9dd4b6fa\" (UID: \"07184f66-00b4-45e3-a60e-503b9dd4b6fa\") " Nov 03 00:50:42.546421 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:42.546402 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07184f66-00b4-45e3-a60e-503b9dd4b6fa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07184f66-00b4-45e3-a60e-503b9dd4b6fa-credential-socket\") pod \"07184f66-00b4-45e3-a60e-503b9dd4b6fa\" (UID: \"07184f66-00b4-45e3-a60e-503b9dd4b6fa\") " Nov 03 00:50:42.546541 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:42.546528 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/07184f66-00b4-45e3-a60e-503b9dd4b6fa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07184f66-00b4-45e3-a60e-503b9dd4b6fa-istio-podinfo\") pod \"07184f66-00b4-45e3-a60e-503b9dd4b6fa\" (UID: \"07184f66-00b4-45e3-a60e-503b9dd4b6fa\") " Nov 03 00:50:42.546654 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:42.546603 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07184f66-00b4-45e3-a60e-503b9dd4b6fa-workload-certs" pod "07184f66-00b4-45e3-a60e-503b9dd4b6fa" (UID: "07184f66-00b4-45e3-a60e-503b9dd4b6fa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:50:42.546654 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:42.546633 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07184f66-00b4-45e3-a60e-503b9dd4b6fa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07184f66-00b4-45e3-a60e-503b9dd4b6fa-istio-envoy\") pod \"07184f66-00b4-45e3-a60e-503b9dd4b6fa\" (UID: \"07184f66-00b4-45e3-a60e-503b9dd4b6fa\") " Nov 03 00:50:42.546837 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:42.546682 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07184f66-00b4-45e3-a60e-503b9dd4b6fa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07184f66-00b4-45e3-a60e-503b9dd4b6fa-istio-data\") pod \"07184f66-00b4-45e3-a60e-503b9dd4b6fa\" (UID: \"07184f66-00b4-45e3-a60e-503b9dd4b6fa\") " Nov 03 00:50:42.546837 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:42.546700 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07184f66-00b4-45e3-a60e-503b9dd4b6fa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07184f66-00b4-45e3-a60e-503b9dd4b6fa-istiod-ca-cert\") pod \"07184f66-00b4-45e3-a60e-503b9dd4b6fa\" (UID: \"07184f66-00b4-45e3-a60e-503b9dd4b6fa\") " Nov 03 00:50:42.546837 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:42.546721 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07184f66-00b4-45e3-a60e-503b9dd4b6fa-kube-api-access-j9dmn\" (UniqueName: \"kubernetes.io/projected/07184f66-00b4-45e3-a60e-503b9dd4b6fa-kube-api-access-j9dmn\") pod \"07184f66-00b4-45e3-a60e-503b9dd4b6fa\" (UID: \"07184f66-00b4-45e3-a60e-503b9dd4b6fa\") " Nov 03 00:50:42.547232 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:42.547059 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07184f66-00b4-45e3-a60e-503b9dd4b6fa-istio-data" pod "07184f66-00b4-45e3-a60e-503b9dd4b6fa" (UID: "07184f66-00b4-45e3-a60e-503b9dd4b6fa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:50:42.547232 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:42.547076 4421 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07184f66-00b4-45e3-a60e-503b9dd4b6fa-workload-certs\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:50:42.547560 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:42.547391 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07184f66-00b4-45e3-a60e-503b9dd4b6fa-workload-socket" pod "07184f66-00b4-45e3-a60e-503b9dd4b6fa" (UID: "07184f66-00b4-45e3-a60e-503b9dd4b6fa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:50:42.547560 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:42.547529 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07184f66-00b4-45e3-a60e-503b9dd4b6fa-istiod-ca-cert" pod "07184f66-00b4-45e3-a60e-503b9dd4b6fa" (UID: "07184f66-00b4-45e3-a60e-503b9dd4b6fa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:50:42.547801 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:42.547642 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07184f66-00b4-45e3-a60e-503b9dd4b6fa-credential-socket" pod "07184f66-00b4-45e3-a60e-503b9dd4b6fa" (UID: "07184f66-00b4-45e3-a60e-503b9dd4b6fa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:50:42.549765 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:42.549730 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07184f66-00b4-45e3-a60e-503b9dd4b6fa-istio-token" pod "07184f66-00b4-45e3-a60e-503b9dd4b6fa" (UID: "07184f66-00b4-45e3-a60e-503b9dd4b6fa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:50:42.550536 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:42.550508 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/07184f66-00b4-45e3-a60e-503b9dd4b6fa-istio-podinfo" pod "07184f66-00b4-45e3-a60e-503b9dd4b6fa" (UID: "07184f66-00b4-45e3-a60e-503b9dd4b6fa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:50:42.550621 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:42.550597 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07184f66-00b4-45e3-a60e-503b9dd4b6fa-istio-envoy" pod "07184f66-00b4-45e3-a60e-503b9dd4b6fa" (UID: "07184f66-00b4-45e3-a60e-503b9dd4b6fa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:50:42.551314 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:42.551265 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07184f66-00b4-45e3-a60e-503b9dd4b6fa-kube-api-access-j9dmn" pod "07184f66-00b4-45e3-a60e-503b9dd4b6fa" (UID: "07184f66-00b4-45e3-a60e-503b9dd4b6fa"). InnerVolumeSpecName "kube-api-access-j9dmn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:50:42.647814 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:42.647766 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9dmn\" (UniqueName: \"kubernetes.io/projected/07184f66-00b4-45e3-a60e-503b9dd4b6fa-kube-api-access-j9dmn\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:50:42.647814 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:42.647807 4421 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07184f66-00b4-45e3-a60e-503b9dd4b6fa-istio-token\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:50:42.647814 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:42.647817 4421 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07184f66-00b4-45e3-a60e-503b9dd4b6fa-workload-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:50:42.647814 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:42.647828 4421 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07184f66-00b4-45e3-a60e-503b9dd4b6fa-credential-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:50:42.648085 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:42.647836 4421 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07184f66-00b4-45e3-a60e-503b9dd4b6fa-istio-podinfo\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:50:42.648085 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:42.647845 4421 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07184f66-00b4-45e3-a60e-503b9dd4b6fa-istio-envoy\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:50:42.648085 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:42.647853 4421 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07184f66-00b4-45e3-a60e-503b9dd4b6fa-istio-data\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:50:42.648085 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:42.647862 4421 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07184f66-00b4-45e3-a60e-503b9dd4b6fa-istiod-ca-cert\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:50:43.073237 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:43.073198 4421 generic.go:356] "Generic (PLEG): container finished" podID="07184f66-00b4-45e3-a60e-503b9dd4b6fa" containerID="1f42a74cd65307be2d561bb30b71f24440123834604818be21ae4f5f41884f98" exitCode=0 Nov 03 00:50:43.073656 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:43.073250 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-4aec6006-openshift-default-77f675f877-fjbkq" Nov 03 00:50:43.073656 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:43.073289 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-4aec6006-openshift-default-77f675f877-fjbkq" event={"ID":"07184f66-00b4-45e3-a60e-503b9dd4b6fa","Type":"ContainerDied","Data":"1f42a74cd65307be2d561bb30b71f24440123834604818be21ae4f5f41884f98"} Nov 03 00:50:43.073656 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:43.073344 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-4aec6006-openshift-default-77f675f877-fjbkq" event={"ID":"07184f66-00b4-45e3-a60e-503b9dd4b6fa","Type":"ContainerDied","Data":"58434545a79afc71b6e3c8f5939d598dd96b96730ff4cffba6973c06c56d4a12"} Nov 03 00:50:43.073656 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:43.073367 4421 scope.go:122] "RemoveContainer" containerID="1f42a74cd65307be2d561bb30b71f24440123834604818be21ae4f5f41884f98" Nov 03 00:50:43.091457 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:43.091431 4421 scope.go:122] "RemoveContainer" containerID="5434745fe472eed3455af9288ae82bfc6926ad3a14203de7094fe748380096d4" Nov 03 00:50:43.100765 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:43.100740 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-4aec6006-openshift-default-77f675f877-fjbkq"] Nov 03 00:50:43.103480 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:43.102664 4421 scope.go:122] "RemoveContainer" containerID="1f42a74cd65307be2d561bb30b71f24440123834604818be21ae4f5f41884f98" Nov 03 00:50:43.103675 ip-10-0-133-217 kubenswrapper[4421]: E1103 00:50:43.103651 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f42a74cd65307be2d561bb30b71f24440123834604818be21ae4f5f41884f98\": container with ID starting with 1f42a74cd65307be2d561bb30b71f24440123834604818be21ae4f5f41884f98 not found: ID does not exist" containerID="1f42a74cd65307be2d561bb30b71f24440123834604818be21ae4f5f41884f98" Nov 03 00:50:43.103800 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:43.103781 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f42a74cd65307be2d561bb30b71f24440123834604818be21ae4f5f41884f98"} err="failed to get container status \"1f42a74cd65307be2d561bb30b71f24440123834604818be21ae4f5f41884f98\": rpc error: code = NotFound desc = could not find container \"1f42a74cd65307be2d561bb30b71f24440123834604818be21ae4f5f41884f98\": container with ID starting with 1f42a74cd65307be2d561bb30b71f24440123834604818be21ae4f5f41884f98 not found: ID does not exist" Nov 03 00:50:43.103870 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:43.103862 4421 scope.go:122] "RemoveContainer" containerID="5434745fe472eed3455af9288ae82bfc6926ad3a14203de7094fe748380096d4" Nov 03 00:50:43.104208 ip-10-0-133-217 kubenswrapper[4421]: E1103 00:50:43.104189 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5434745fe472eed3455af9288ae82bfc6926ad3a14203de7094fe748380096d4\": container with ID starting with 5434745fe472eed3455af9288ae82bfc6926ad3a14203de7094fe748380096d4 not found: ID does not exist" containerID="5434745fe472eed3455af9288ae82bfc6926ad3a14203de7094fe748380096d4" Nov 03 00:50:43.104293 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:43.104214 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5434745fe472eed3455af9288ae82bfc6926ad3a14203de7094fe748380096d4"} err="failed to get container status \"5434745fe472eed3455af9288ae82bfc6926ad3a14203de7094fe748380096d4\": rpc error: code = NotFound desc = could not find container \"5434745fe472eed3455af9288ae82bfc6926ad3a14203de7094fe748380096d4\": container with ID starting with 5434745fe472eed3455af9288ae82bfc6926ad3a14203de7094fe748380096d4 not found: ID does not exist" Nov 03 00:50:43.132617 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:43.132576 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-4aec6006-openshift-default-77f675f877-fjbkq"] Nov 03 00:50:43.471041 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:43.470960 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-88bbf7-554c5df5-dxctw"] Nov 03 00:50:43.471476 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:43.471460 4421 cpu_manager.go:414] "containerMap: removing container" podUID="8c0989d6-ad4a-49ad-9a0e-c01f1ef0e797" containerName="payload-pre-processing" Nov 03 00:50:43.471530 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:43.471477 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c0989d6-ad4a-49ad-9a0e-c01f1ef0e797" containerName="payload-pre-processing" Nov 03 00:50:43.471530 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:43.471491 4421 cpu_manager.go:414] "containerMap: removing container" podUID="07184f66-00b4-45e3-a60e-503b9dd4b6fa" containerName="istio-proxy" Nov 03 00:50:43.471530 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:43.471496 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07184f66-00b4-45e3-a60e-503b9dd4b6fa" containerName="istio-proxy" Nov 03 00:50:43.471530 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:43.471508 4421 cpu_manager.go:414] "containerMap: removing container" podUID="07184f66-00b4-45e3-a60e-503b9dd4b6fa" containerName="istio-proxy" Nov 03 00:50:43.471530 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:43.471513 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07184f66-00b4-45e3-a60e-503b9dd4b6fa" containerName="istio-proxy" Nov 03 00:50:43.471685 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:43.471571 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="07184f66-00b4-45e3-a60e-503b9dd4b6fa" containerName="istio-proxy" Nov 03 00:50:43.471685 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:43.471581 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="07184f66-00b4-45e3-a60e-503b9dd4b6fa" containerName="istio-proxy" Nov 03 00:50:43.471685 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:43.471591 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c0989d6-ad4a-49ad-9a0e-c01f1ef0e797" containerName="payload-pre-processing" Nov 03 00:50:43.476532 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:43.476511 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-88bbf7-554c5df5-dxctw" Nov 03 00:50:43.489188 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:43.489161 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-88bbf7-554c5df5-dxctw"] Nov 03 00:50:43.491031 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:43.491011 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-88bbf7\"" Nov 03 00:50:43.538535 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:43.538493 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-88bbf7-78b86696f9-x7mgb"] Nov 03 00:50:43.542085 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:43.542066 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-88bbf7-78b86696f9-x7mgb" Nov 03 00:50:43.565070 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:43.565043 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-88bbf7-dockercfg-qzlq7\"" Nov 03 00:50:43.656599 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:43.656566 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plh56\" (UniqueName: \"kubernetes.io/projected/a454ee55-ade7-4715-bf20-8cc451458d99-kube-api-access-plh56\") pod \"payload-processing-e2e-worker-w1-88bbf7-78b86696f9-x7mgb\" (UID: \"a454ee55-ade7-4715-bf20-8cc451458d99\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-88bbf7-78b86696f9-x7mgb" Nov 03 00:50:43.656844 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:43.656822 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f2cede9-84b3-4d23-8456-38016e4ff7fb-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-88bbf7-554c5df5-dxctw\" (UID: \"4f2cede9-84b3-4d23-8456-38016e4ff7fb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-88bbf7-554c5df5-dxctw" Nov 03 00:50:43.656973 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:43.656962 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knf9j\" (UniqueName: \"kubernetes.io/projected/4f2cede9-84b3-4d23-8456-38016e4ff7fb-kube-api-access-knf9j\") pod \"payload-pre-processing-e2e-worker-w1-88bbf7-554c5df5-dxctw\" (UID: \"4f2cede9-84b3-4d23-8456-38016e4ff7fb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-88bbf7-554c5df5-dxctw" Nov 03 00:50:43.657065 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:43.657055 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a454ee55-ade7-4715-bf20-8cc451458d99-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-88bbf7-78b86696f9-x7mgb\" (UID: \"a454ee55-ade7-4715-bf20-8cc451458d99\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-88bbf7-78b86696f9-x7mgb" Nov 03 00:50:43.668025 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:43.667994 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-88bbf7-78b86696f9-x7mgb"] Nov 03 00:50:43.758312 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:43.758251 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f2cede9-84b3-4d23-8456-38016e4ff7fb-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-88bbf7-554c5df5-dxctw\" (UID: \"4f2cede9-84b3-4d23-8456-38016e4ff7fb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-88bbf7-554c5df5-dxctw" Nov 03 00:50:43.758522 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:43.758348 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knf9j\" (UniqueName: \"kubernetes.io/projected/4f2cede9-84b3-4d23-8456-38016e4ff7fb-kube-api-access-knf9j\") pod \"payload-pre-processing-e2e-worker-w1-88bbf7-554c5df5-dxctw\" (UID: \"4f2cede9-84b3-4d23-8456-38016e4ff7fb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-88bbf7-554c5df5-dxctw" Nov 03 00:50:43.758522 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:43.758412 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a454ee55-ade7-4715-bf20-8cc451458d99-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-88bbf7-78b86696f9-x7mgb\" (UID: \"a454ee55-ade7-4715-bf20-8cc451458d99\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-88bbf7-78b86696f9-x7mgb" Nov 03 00:50:43.758647 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:43.758569 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plh56\" (UniqueName: \"kubernetes.io/projected/a454ee55-ade7-4715-bf20-8cc451458d99-kube-api-access-plh56\") pod \"payload-processing-e2e-worker-w1-88bbf7-78b86696f9-x7mgb\" (UID: \"a454ee55-ade7-4715-bf20-8cc451458d99\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-88bbf7-78b86696f9-x7mgb" Nov 03 00:50:43.759111 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:43.759070 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a454ee55-ade7-4715-bf20-8cc451458d99-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-88bbf7-78b86696f9-x7mgb\" (UID: \"a454ee55-ade7-4715-bf20-8cc451458d99\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-88bbf7-78b86696f9-x7mgb" Nov 03 00:50:43.759321 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:43.759300 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f2cede9-84b3-4d23-8456-38016e4ff7fb-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-88bbf7-554c5df5-dxctw\" (UID: \"4f2cede9-84b3-4d23-8456-38016e4ff7fb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-88bbf7-554c5df5-dxctw" Nov 03 00:50:43.771675 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:43.771647 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-knf9j\" (UniqueName: \"kubernetes.io/projected/4f2cede9-84b3-4d23-8456-38016e4ff7fb-kube-api-access-knf9j\") pod \"payload-pre-processing-e2e-worker-w1-88bbf7-554c5df5-dxctw\" (UID: \"4f2cede9-84b3-4d23-8456-38016e4ff7fb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-88bbf7-554c5df5-dxctw" Nov 03 00:50:43.772898 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:43.772875 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-plh56\" (UniqueName: \"kubernetes.io/projected/a454ee55-ade7-4715-bf20-8cc451458d99-kube-api-access-plh56\") pod \"payload-processing-e2e-worker-w1-88bbf7-78b86696f9-x7mgb\" (UID: \"a454ee55-ade7-4715-bf20-8cc451458d99\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-88bbf7-78b86696f9-x7mgb" Nov 03 00:50:43.793679 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:43.793618 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-88bbf7-554c5df5-dxctw" Nov 03 00:50:43.856454 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:43.856157 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-88bbf7-78b86696f9-x7mgb" Nov 03 00:50:43.980316 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:43.979952 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07184f66-00b4-45e3-a60e-503b9dd4b6fa" path="/var/lib/kubelet/pods/07184f66-00b4-45e3-a60e-503b9dd4b6fa/volumes" Nov 03 00:50:44.123634 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:44.123523 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-88bbf7-554c5df5-dxctw"] Nov 03 00:50:44.128530 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:50:44.128502 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f2cede9_84b3_4d23_8456_38016e4ff7fb.slice/crio-a2911f66a6a8f3cfb35b1e461a88733ec65462629d7c5b85c904cf1ba1531971 WatchSource:0}: Error finding container a2911f66a6a8f3cfb35b1e461a88733ec65462629d7c5b85c904cf1ba1531971: Status 404 returned error can't find the container with id a2911f66a6a8f3cfb35b1e461a88733ec65462629d7c5b85c904cf1ba1531971 Nov 03 00:50:44.178259 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:44.178233 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-88bbf7-78b86696f9-x7mgb"] Nov 03 00:50:44.180341 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:50:44.180314 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda454ee55_ade7_4715_bf20_8cc451458d99.slice/crio-ac7fdeb1c87c6eee91d403b70bf45e60abb903f9ba8f6b9b0ca8dafcf5e438c6 WatchSource:0}: Error finding container ac7fdeb1c87c6eee91d403b70bf45e60abb903f9ba8f6b9b0ca8dafcf5e438c6: Status 404 returned error can't find the container with id ac7fdeb1c87c6eee91d403b70bf45e60abb903f9ba8f6b9b0ca8dafcf5e438c6 Nov 03 00:50:44.671542 ip-10-0-133-217 kubenswrapper[4421]: E1103 00:50:44.671501 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 00:50:45.088716 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:45.088683 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-88bbf7-78b86696f9-x7mgb" event={"ID":"a454ee55-ade7-4715-bf20-8cc451458d99","Type":"ContainerStarted","Data":"1d8ffb67f7937117e6bd70d56767fe923a236863c7c970cecb5a9b92948570d3"} Nov 03 00:50:45.088934 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:45.088748 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-88bbf7-78b86696f9-x7mgb" event={"ID":"a454ee55-ade7-4715-bf20-8cc451458d99","Type":"ContainerStarted","Data":"ac7fdeb1c87c6eee91d403b70bf45e60abb903f9ba8f6b9b0ca8dafcf5e438c6"} Nov 03 00:50:45.088934 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:45.088881 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-88bbf7-78b86696f9-x7mgb" Nov 03 00:50:45.090566 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:45.090536 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-88bbf7-554c5df5-dxctw" event={"ID":"4f2cede9-84b3-4d23-8456-38016e4ff7fb","Type":"ContainerStarted","Data":"0ff9ab7becf756d32b71503c6f764e90d53a670eec8139025660a683194927a7"} Nov 03 00:50:45.090686 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:45.090578 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-88bbf7-554c5df5-dxctw" event={"ID":"4f2cede9-84b3-4d23-8456-38016e4ff7fb","Type":"ContainerStarted","Data":"a2911f66a6a8f3cfb35b1e461a88733ec65462629d7c5b85c904cf1ba1531971"} Nov 03 00:50:45.090686 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:45.090598 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-88bbf7-554c5df5-dxctw" Nov 03 00:50:45.108493 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:45.108434 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-88bbf7-78b86696f9-x7mgb" podStartSLOduration=2.108416277 podStartE2EDuration="2.108416277s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:50:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:50:45.105557488 +0000 UTC m=+587.276560584" watchObservedRunningTime="2026-11-03 00:50:45.108416277 +0000 UTC m=+587.279419373" Nov 03 00:50:45.122419 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:45.122351 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-88bbf7-554c5df5-dxctw" podStartSLOduration=2.122337051 podStartE2EDuration="2.122337051s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:50:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:50:45.121582196 +0000 UTC m=+587.292585295" watchObservedRunningTime="2026-11-03 00:50:45.122337051 +0000 UTC m=+587.293340147" Nov 03 00:50:47.512966 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:47.512381 4421 patch_prober.go:29] interesting pod/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-2fkgc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": read tcp 10.134.0.2:54500->10.134.0.53:15021: read: connection reset by peer" start-of-body= Nov 03 00:50:47.512966 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:47.512469 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-2fkgc" podUID="f0c98b4a-9b13-4ad5-9e0b-ac50ae14fa5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": read tcp 10.134.0.2:54500->10.134.0.53:15021: read: connection reset by peer" Nov 03 00:50:48.041266 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:48.041235 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-f7680d-67845fdfd7-r56nd" Nov 03 00:50:48.041475 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:48.041441 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-83755a-7dfdb9c56c-l78nm" Nov 03 00:50:48.041570 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:48.041549 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-f7680d-59ddf48c45-8xgwl" Nov 03 00:50:48.113240 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:48.113210 4421 generic.go:356] "Generic (PLEG): container finished" podID="f0c98b4a-9b13-4ad5-9e0b-ac50ae14fa5d" containerID="9ef83ec7d0dc2496a4493b5009f4a307086c31c8e36533834180854f6ae4a2ad" exitCode=0 Nov 03 00:50:48.113442 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:48.113320 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-2fkgc" event={"ID":"f0c98b4a-9b13-4ad5-9e0b-ac50ae14fa5d","Type":"ContainerDied","Data":"9ef83ec7d0dc2496a4493b5009f4a307086c31c8e36533834180854f6ae4a2ad"} Nov 03 00:50:48.113701 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:48.113679 4421 scope.go:122] "RemoveContainer" containerID="9ef83ec7d0dc2496a4493b5009f4a307086c31c8e36533834180854f6ae4a2ad" Nov 03 00:50:48.115300 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:48.115236 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:50:48.115419 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:48.115347 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:50:48.115419 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:48.115400 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:50:49.048725 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:49.048695 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-83755a-67c54f67f5-mxhfv" Nov 03 00:50:49.141331 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:49.141297 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-2fkgc" event={"ID":"f0c98b4a-9b13-4ad5-9e0b-ac50ae14fa5d","Type":"ContainerStarted","Data":"deb7249f7de228dd52e91ac11967498578fdb606c5f6b720428814bb6f8fb0da"} Nov 03 00:50:49.325783 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:49.325688 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-2fkgc" Nov 03 00:50:49.326121 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:49.326098 4421 patch_prober.go:29] interesting pod/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-2fkgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Nov 03 00:50:49.326314 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:49.326253 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-2fkgc" podUID="f0c98b4a-9b13-4ad5-9e0b-ac50ae14fa5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Nov 03 00:50:50.056315 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:50.056265 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2f9a3a-78c44fb86c-ckvcs" Nov 03 00:50:50.056719 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:50.056359 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-2f9a3a-8697684787-rcsf2" Nov 03 00:50:50.330451 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:50.326577 4421 patch_prober.go:29] interesting pod/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-2fkgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Nov 03 00:50:50.330451 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:50.326653 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-2fkgc" podUID="f0c98b4a-9b13-4ad5-9e0b-ac50ae14fa5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Nov 03 00:50:51.328297 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:51.327740 4421 patch_prober.go:29] interesting pod/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-2fkgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Nov 03 00:50:51.328297 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:51.327824 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-2fkgc" podUID="f0c98b4a-9b13-4ad5-9e0b-ac50ae14fa5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Nov 03 00:50:51.733440 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:51.733349 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-650be0/simulator-w4-650be0-kserve-7bc588774f-gvz2k"] Nov 03 00:50:51.737462 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:51.737440 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-650be0/simulator-w4-650be0-kserve-7bc588774f-gvz2k" Nov 03 00:50:51.755785 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:51.755745 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a827fa88-f078-4312-8f12-2c799c11b461-tls-certs\") pod \"simulator-w4-650be0-kserve-7bc588774f-gvz2k\" (UID: \"a827fa88-f078-4312-8f12-2c799c11b461\") " pod="e2e-models-e2e-worker-w4-650be0/simulator-w4-650be0-kserve-7bc588774f-gvz2k" Nov 03 00:50:51.755947 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:51.755857 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a827fa88-f078-4312-8f12-2c799c11b461-home\") pod \"simulator-w4-650be0-kserve-7bc588774f-gvz2k\" (UID: \"a827fa88-f078-4312-8f12-2c799c11b461\") " pod="e2e-models-e2e-worker-w4-650be0/simulator-w4-650be0-kserve-7bc588774f-gvz2k" Nov 03 00:50:51.755947 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:51.755903 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r99xx\" (UniqueName: \"kubernetes.io/projected/a827fa88-f078-4312-8f12-2c799c11b461-kube-api-access-r99xx\") pod \"simulator-w4-650be0-kserve-7bc588774f-gvz2k\" (UID: \"a827fa88-f078-4312-8f12-2c799c11b461\") " pod="e2e-models-e2e-worker-w4-650be0/simulator-w4-650be0-kserve-7bc588774f-gvz2k" Nov 03 00:50:51.755947 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:51.755935 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a827fa88-f078-4312-8f12-2c799c11b461-tmp-dir\") pod \"simulator-w4-650be0-kserve-7bc588774f-gvz2k\" (UID: \"a827fa88-f078-4312-8f12-2c799c11b461\") " pod="e2e-models-e2e-worker-w4-650be0/simulator-w4-650be0-kserve-7bc588774f-gvz2k" Nov 03 00:50:51.756113 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:51.755953 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a827fa88-f078-4312-8f12-2c799c11b461-dshm\") pod \"simulator-w4-650be0-kserve-7bc588774f-gvz2k\" (UID: \"a827fa88-f078-4312-8f12-2c799c11b461\") " pod="e2e-models-e2e-worker-w4-650be0/simulator-w4-650be0-kserve-7bc588774f-gvz2k" Nov 03 00:50:51.756172 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:51.756117 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a827fa88-f078-4312-8f12-2c799c11b461-model-cache\") pod \"simulator-w4-650be0-kserve-7bc588774f-gvz2k\" (UID: \"a827fa88-f078-4312-8f12-2c799c11b461\") " pod="e2e-models-e2e-worker-w4-650be0/simulator-w4-650be0-kserve-7bc588774f-gvz2k" Nov 03 00:50:51.762473 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:51.762439 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-650be0\"/\"default-dockercfg-ggdgp\"" Nov 03 00:50:51.762473 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:51.762464 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-650be0\"/\"simulator-w4-650be0-kserve-self-signed-certs\"" Nov 03 00:50:51.762659 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:51.762472 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-650be0\"/\"openshift-service-ca.crt\"" Nov 03 00:50:51.762659 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:51.762558 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-650be0\"/\"kube-root-ca.crt\"" Nov 03 00:50:51.775998 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:51.775968 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-650be0/simulator-w4-650be0-kserve-7bc588774f-gvz2k"] Nov 03 00:50:51.856951 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:51.856918 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a827fa88-f078-4312-8f12-2c799c11b461-model-cache\") pod \"simulator-w4-650be0-kserve-7bc588774f-gvz2k\" (UID: \"a827fa88-f078-4312-8f12-2c799c11b461\") " pod="e2e-models-e2e-worker-w4-650be0/simulator-w4-650be0-kserve-7bc588774f-gvz2k" Nov 03 00:50:51.857168 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:51.856975 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a827fa88-f078-4312-8f12-2c799c11b461-tls-certs\") pod \"simulator-w4-650be0-kserve-7bc588774f-gvz2k\" (UID: \"a827fa88-f078-4312-8f12-2c799c11b461\") " pod="e2e-models-e2e-worker-w4-650be0/simulator-w4-650be0-kserve-7bc588774f-gvz2k" Nov 03 00:50:51.857168 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:51.857036 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a827fa88-f078-4312-8f12-2c799c11b461-home\") pod \"simulator-w4-650be0-kserve-7bc588774f-gvz2k\" (UID: \"a827fa88-f078-4312-8f12-2c799c11b461\") " pod="e2e-models-e2e-worker-w4-650be0/simulator-w4-650be0-kserve-7bc588774f-gvz2k" Nov 03 00:50:51.857168 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:51.857061 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r99xx\" (UniqueName: \"kubernetes.io/projected/a827fa88-f078-4312-8f12-2c799c11b461-kube-api-access-r99xx\") pod \"simulator-w4-650be0-kserve-7bc588774f-gvz2k\" (UID: \"a827fa88-f078-4312-8f12-2c799c11b461\") " pod="e2e-models-e2e-worker-w4-650be0/simulator-w4-650be0-kserve-7bc588774f-gvz2k" Nov 03 00:50:51.857168 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:51.857103 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a827fa88-f078-4312-8f12-2c799c11b461-tmp-dir\") pod \"simulator-w4-650be0-kserve-7bc588774f-gvz2k\" (UID: \"a827fa88-f078-4312-8f12-2c799c11b461\") " pod="e2e-models-e2e-worker-w4-650be0/simulator-w4-650be0-kserve-7bc588774f-gvz2k" Nov 03 00:50:51.857168 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:51.857134 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a827fa88-f078-4312-8f12-2c799c11b461-dshm\") pod \"simulator-w4-650be0-kserve-7bc588774f-gvz2k\" (UID: \"a827fa88-f078-4312-8f12-2c799c11b461\") " pod="e2e-models-e2e-worker-w4-650be0/simulator-w4-650be0-kserve-7bc588774f-gvz2k" Nov 03 00:50:51.857486 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:51.857456 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a827fa88-f078-4312-8f12-2c799c11b461-model-cache\") pod \"simulator-w4-650be0-kserve-7bc588774f-gvz2k\" (UID: \"a827fa88-f078-4312-8f12-2c799c11b461\") " pod="e2e-models-e2e-worker-w4-650be0/simulator-w4-650be0-kserve-7bc588774f-gvz2k" Nov 03 00:50:51.857563 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:51.857505 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a827fa88-f078-4312-8f12-2c799c11b461-home\") pod \"simulator-w4-650be0-kserve-7bc588774f-gvz2k\" (UID: \"a827fa88-f078-4312-8f12-2c799c11b461\") " pod="e2e-models-e2e-worker-w4-650be0/simulator-w4-650be0-kserve-7bc588774f-gvz2k" Nov 03 00:50:51.857616 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:51.857557 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a827fa88-f078-4312-8f12-2c799c11b461-tmp-dir\") pod \"simulator-w4-650be0-kserve-7bc588774f-gvz2k\" (UID: \"a827fa88-f078-4312-8f12-2c799c11b461\") " pod="e2e-models-e2e-worker-w4-650be0/simulator-w4-650be0-kserve-7bc588774f-gvz2k" Nov 03 00:50:51.859534 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:51.859510 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a827fa88-f078-4312-8f12-2c799c11b461-dshm\") pod \"simulator-w4-650be0-kserve-7bc588774f-gvz2k\" (UID: \"a827fa88-f078-4312-8f12-2c799c11b461\") " pod="e2e-models-e2e-worker-w4-650be0/simulator-w4-650be0-kserve-7bc588774f-gvz2k" Nov 03 00:50:51.859670 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:51.859653 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a827fa88-f078-4312-8f12-2c799c11b461-tls-certs\") pod \"simulator-w4-650be0-kserve-7bc588774f-gvz2k\" (UID: \"a827fa88-f078-4312-8f12-2c799c11b461\") " pod="e2e-models-e2e-worker-w4-650be0/simulator-w4-650be0-kserve-7bc588774f-gvz2k" Nov 03 00:50:51.865459 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:51.865437 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r99xx\" (UniqueName: \"kubernetes.io/projected/a827fa88-f078-4312-8f12-2c799c11b461-kube-api-access-r99xx\") pod \"simulator-w4-650be0-kserve-7bc588774f-gvz2k\" (UID: \"a827fa88-f078-4312-8f12-2c799c11b461\") " pod="e2e-models-e2e-worker-w4-650be0/simulator-w4-650be0-kserve-7bc588774f-gvz2k" Nov 03 00:50:52.050211 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:52.050171 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-650be0/simulator-w4-650be0-kserve-7bc588774f-gvz2k" Nov 03 00:50:52.330304 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:52.329749 4421 patch_prober.go:29] interesting pod/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-2fkgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Nov 03 00:50:52.330304 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:52.329815 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-2fkgc" podUID="f0c98b4a-9b13-4ad5-9e0b-ac50ae14fa5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Nov 03 00:50:52.372055 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:52.372027 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-650be0/simulator-w4-650be0-kserve-7bc588774f-gvz2k"] Nov 03 00:50:52.374288 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:50:52.374248 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda827fa88_f078_4312_8f12_2c799c11b461.slice/crio-295ac683966dd8ccb6c1db5f9ca9e5fbc9f2506375901e3723514c614957a636 WatchSource:0}: Error finding container 295ac683966dd8ccb6c1db5f9ca9e5fbc9f2506375901e3723514c614957a636: Status 404 returned error can't find the container with id 295ac683966dd8ccb6c1db5f9ca9e5fbc9f2506375901e3723514c614957a636 Nov 03 00:50:53.202505 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:53.202468 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-650be0/simulator-w4-650be0-kserve-7bc588774f-gvz2k" event={"ID":"a827fa88-f078-4312-8f12-2c799c11b461","Type":"ContainerStarted","Data":"ad7a4e337194ba2f2ac79ba4bc0d9251212a38a57d514c25b56b24fd41f233a2"} Nov 03 00:50:53.202505 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:53.202513 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-650be0/simulator-w4-650be0-kserve-7bc588774f-gvz2k" event={"ID":"a827fa88-f078-4312-8f12-2c799c11b461","Type":"ContainerStarted","Data":"295ac683966dd8ccb6c1db5f9ca9e5fbc9f2506375901e3723514c614957a636"} Nov 03 00:50:53.247429 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:53.247375 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-650be0/simulator-w4-650be0-kserve-7bc588774f-gvz2k" podStartSLOduration=2.247358547 podStartE2EDuration="2.247358547s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:50:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:50:53.245235017 +0000 UTC m=+595.416238121" watchObservedRunningTime="2026-11-03 00:50:53.247358547 +0000 UTC m=+595.418361642" Nov 03 00:50:53.326297 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:53.324395 4421 patch_prober.go:29] interesting pod/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-2fkgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Nov 03 00:50:53.326297 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:53.324460 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-2fkgc" podUID="f0c98b4a-9b13-4ad5-9e0b-ac50ae14fa5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Nov 03 00:50:54.051127 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:54.051096 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-650be0/simulator-w4-650be0-kserve-7bc588774f-gvz2k" Nov 03 00:50:54.538944 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:54.538854 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-2fkgc" Nov 03 00:50:54.540370 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:54.540166 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-2fkgc" Nov 03 00:50:54.541537 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:54.541489 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-2fkgc" Nov 03 00:50:56.099923 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:56.099888 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-88bbf7-78b86696f9-x7mgb" Nov 03 00:50:56.101416 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:56.101390 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-88bbf7-554c5df5-dxctw" Nov 03 00:50:59.401871 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:59.401827 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-f7680d/simulator-w2-f7680d-kserve-85bdc86cd6-8gxrg"] Nov 03 00:50:59.407921 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:59.407897 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-f7680d/simulator-w2-f7680d-kserve-85bdc86cd6-8gxrg" Nov 03 00:50:59.415238 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:59.415201 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-f7680d/simulator-w2-f7680d-kserve-85bdc86cd6-8gxrg"] Nov 03 00:50:59.416262 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:59.416241 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-f7680d\"/\"kube-root-ca.crt\"" Nov 03 00:50:59.416383 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:59.416294 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-f7680d\"/\"openshift-service-ca.crt\"" Nov 03 00:50:59.441231 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:59.441201 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2bdcf521-cebd-4d6b-8554-1b60db8b0f3c-tls-certs\") pod \"simulator-w2-f7680d-kserve-85bdc86cd6-8gxrg\" (UID: \"2bdcf521-cebd-4d6b-8554-1b60db8b0f3c\") " pod="e2e-models-e2e-worker-w2-f7680d/simulator-w2-f7680d-kserve-85bdc86cd6-8gxrg" Nov 03 00:50:59.441445 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:59.441246 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2bdcf521-cebd-4d6b-8554-1b60db8b0f3c-model-cache\") pod \"simulator-w2-f7680d-kserve-85bdc86cd6-8gxrg\" (UID: \"2bdcf521-cebd-4d6b-8554-1b60db8b0f3c\") " pod="e2e-models-e2e-worker-w2-f7680d/simulator-w2-f7680d-kserve-85bdc86cd6-8gxrg" Nov 03 00:50:59.441445 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:59.441303 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2bdcf521-cebd-4d6b-8554-1b60db8b0f3c-dshm\") pod \"simulator-w2-f7680d-kserve-85bdc86cd6-8gxrg\" (UID: \"2bdcf521-cebd-4d6b-8554-1b60db8b0f3c\") " pod="e2e-models-e2e-worker-w2-f7680d/simulator-w2-f7680d-kserve-85bdc86cd6-8gxrg" Nov 03 00:50:59.441445 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:59.441402 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2bdcf521-cebd-4d6b-8554-1b60db8b0f3c-tmp-dir\") pod \"simulator-w2-f7680d-kserve-85bdc86cd6-8gxrg\" (UID: \"2bdcf521-cebd-4d6b-8554-1b60db8b0f3c\") " pod="e2e-models-e2e-worker-w2-f7680d/simulator-w2-f7680d-kserve-85bdc86cd6-8gxrg" Nov 03 00:50:59.441445 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:59.441429 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2bdcf521-cebd-4d6b-8554-1b60db8b0f3c-home\") pod \"simulator-w2-f7680d-kserve-85bdc86cd6-8gxrg\" (UID: \"2bdcf521-cebd-4d6b-8554-1b60db8b0f3c\") " pod="e2e-models-e2e-worker-w2-f7680d/simulator-w2-f7680d-kserve-85bdc86cd6-8gxrg" Nov 03 00:50:59.441666 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:59.441530 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rr5fm\" (UniqueName: \"kubernetes.io/projected/2bdcf521-cebd-4d6b-8554-1b60db8b0f3c-kube-api-access-rr5fm\") pod \"simulator-w2-f7680d-kserve-85bdc86cd6-8gxrg\" (UID: \"2bdcf521-cebd-4d6b-8554-1b60db8b0f3c\") " pod="e2e-models-e2e-worker-w2-f7680d/simulator-w2-f7680d-kserve-85bdc86cd6-8gxrg" Nov 03 00:50:59.442570 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:59.442535 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-f7680d\"/\"default-dockercfg-zmrzj\"" Nov 03 00:50:59.442752 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:59.442729 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-f7680d\"/\"simulator-w2-f7680d-kserve-self-signed-certs\"" Nov 03 00:50:59.542106 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:59.542072 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2bdcf521-cebd-4d6b-8554-1b60db8b0f3c-tmp-dir\") pod \"simulator-w2-f7680d-kserve-85bdc86cd6-8gxrg\" (UID: \"2bdcf521-cebd-4d6b-8554-1b60db8b0f3c\") " pod="e2e-models-e2e-worker-w2-f7680d/simulator-w2-f7680d-kserve-85bdc86cd6-8gxrg" Nov 03 00:50:59.542344 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:59.542116 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2bdcf521-cebd-4d6b-8554-1b60db8b0f3c-home\") pod \"simulator-w2-f7680d-kserve-85bdc86cd6-8gxrg\" (UID: \"2bdcf521-cebd-4d6b-8554-1b60db8b0f3c\") " pod="e2e-models-e2e-worker-w2-f7680d/simulator-w2-f7680d-kserve-85bdc86cd6-8gxrg" Nov 03 00:50:59.542344 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:59.542242 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rr5fm\" (UniqueName: \"kubernetes.io/projected/2bdcf521-cebd-4d6b-8554-1b60db8b0f3c-kube-api-access-rr5fm\") pod \"simulator-w2-f7680d-kserve-85bdc86cd6-8gxrg\" (UID: \"2bdcf521-cebd-4d6b-8554-1b60db8b0f3c\") " pod="e2e-models-e2e-worker-w2-f7680d/simulator-w2-f7680d-kserve-85bdc86cd6-8gxrg" Nov 03 00:50:59.542344 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:59.542326 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2bdcf521-cebd-4d6b-8554-1b60db8b0f3c-tls-certs\") pod \"simulator-w2-f7680d-kserve-85bdc86cd6-8gxrg\" (UID: \"2bdcf521-cebd-4d6b-8554-1b60db8b0f3c\") " pod="e2e-models-e2e-worker-w2-f7680d/simulator-w2-f7680d-kserve-85bdc86cd6-8gxrg" Nov 03 00:50:59.542554 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:59.542384 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2bdcf521-cebd-4d6b-8554-1b60db8b0f3c-model-cache\") pod \"simulator-w2-f7680d-kserve-85bdc86cd6-8gxrg\" (UID: \"2bdcf521-cebd-4d6b-8554-1b60db8b0f3c\") " pod="e2e-models-e2e-worker-w2-f7680d/simulator-w2-f7680d-kserve-85bdc86cd6-8gxrg" Nov 03 00:50:59.542554 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:59.542435 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2bdcf521-cebd-4d6b-8554-1b60db8b0f3c-dshm\") pod \"simulator-w2-f7680d-kserve-85bdc86cd6-8gxrg\" (UID: \"2bdcf521-cebd-4d6b-8554-1b60db8b0f3c\") " pod="e2e-models-e2e-worker-w2-f7680d/simulator-w2-f7680d-kserve-85bdc86cd6-8gxrg" Nov 03 00:50:59.542667 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:59.542578 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2bdcf521-cebd-4d6b-8554-1b60db8b0f3c-home\") pod \"simulator-w2-f7680d-kserve-85bdc86cd6-8gxrg\" (UID: \"2bdcf521-cebd-4d6b-8554-1b60db8b0f3c\") " pod="e2e-models-e2e-worker-w2-f7680d/simulator-w2-f7680d-kserve-85bdc86cd6-8gxrg" Nov 03 00:50:59.542667 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:59.542636 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2bdcf521-cebd-4d6b-8554-1b60db8b0f3c-tmp-dir\") pod \"simulator-w2-f7680d-kserve-85bdc86cd6-8gxrg\" (UID: \"2bdcf521-cebd-4d6b-8554-1b60db8b0f3c\") " pod="e2e-models-e2e-worker-w2-f7680d/simulator-w2-f7680d-kserve-85bdc86cd6-8gxrg" Nov 03 00:50:59.542876 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:59.542860 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2bdcf521-cebd-4d6b-8554-1b60db8b0f3c-model-cache\") pod \"simulator-w2-f7680d-kserve-85bdc86cd6-8gxrg\" (UID: \"2bdcf521-cebd-4d6b-8554-1b60db8b0f3c\") " pod="e2e-models-e2e-worker-w2-f7680d/simulator-w2-f7680d-kserve-85bdc86cd6-8gxrg" Nov 03 00:50:59.544948 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:59.544913 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2bdcf521-cebd-4d6b-8554-1b60db8b0f3c-dshm\") pod \"simulator-w2-f7680d-kserve-85bdc86cd6-8gxrg\" (UID: \"2bdcf521-cebd-4d6b-8554-1b60db8b0f3c\") " pod="e2e-models-e2e-worker-w2-f7680d/simulator-w2-f7680d-kserve-85bdc86cd6-8gxrg" Nov 03 00:50:59.545643 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:59.545619 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2bdcf521-cebd-4d6b-8554-1b60db8b0f3c-tls-certs\") pod \"simulator-w2-f7680d-kserve-85bdc86cd6-8gxrg\" (UID: \"2bdcf521-cebd-4d6b-8554-1b60db8b0f3c\") " pod="e2e-models-e2e-worker-w2-f7680d/simulator-w2-f7680d-kserve-85bdc86cd6-8gxrg" Nov 03 00:50:59.552676 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:59.552640 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rr5fm\" (UniqueName: \"kubernetes.io/projected/2bdcf521-cebd-4d6b-8554-1b60db8b0f3c-kube-api-access-rr5fm\") pod \"simulator-w2-f7680d-kserve-85bdc86cd6-8gxrg\" (UID: \"2bdcf521-cebd-4d6b-8554-1b60db8b0f3c\") " pod="e2e-models-e2e-worker-w2-f7680d/simulator-w2-f7680d-kserve-85bdc86cd6-8gxrg" Nov 03 00:50:59.721830 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:50:59.721737 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-f7680d/simulator-w2-f7680d-kserve-85bdc86cd6-8gxrg" Nov 03 00:51:00.000585 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:00.000559 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-f7680d/simulator-w2-f7680d-kserve-85bdc86cd6-8gxrg"] Nov 03 00:51:00.002885 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:51:00.002854 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2bdcf521_cebd_4d6b_8554_1b60db8b0f3c.slice/crio-7488f92228758641ac38f77b2498265f2ddddf5000f83ef4b196bf6d52af02cd WatchSource:0}: Error finding container 7488f92228758641ac38f77b2498265f2ddddf5000f83ef4b196bf6d52af02cd: Status 404 returned error can't find the container with id 7488f92228758641ac38f77b2498265f2ddddf5000f83ef4b196bf6d52af02cd Nov 03 00:51:00.261041 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:00.260942 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-f7680d/simulator-w2-f7680d-kserve-85bdc86cd6-8gxrg" event={"ID":"2bdcf521-cebd-4d6b-8554-1b60db8b0f3c","Type":"ContainerStarted","Data":"e1d27bc4e9bea20398f7045149f5ec0e5093a5044e55cd165135ea66390f613e"} Nov 03 00:51:00.261041 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:00.260987 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-f7680d/simulator-w2-f7680d-kserve-85bdc86cd6-8gxrg" event={"ID":"2bdcf521-cebd-4d6b-8554-1b60db8b0f3c","Type":"ContainerStarted","Data":"7488f92228758641ac38f77b2498265f2ddddf5000f83ef4b196bf6d52af02cd"} Nov 03 00:51:00.278441 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:00.278372 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-f7680d/simulator-w2-f7680d-kserve-85bdc86cd6-8gxrg" podStartSLOduration=1.27835605 podStartE2EDuration="1.27835605s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:50:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:51:00.277349028 +0000 UTC m=+602.448352128" watchObservedRunningTime="2026-11-03 00:51:00.27835605 +0000 UTC m=+602.449359145" Nov 03 00:51:00.722172 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:00.722063 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f7680d/simulator-w2-f7680d-kserve-85bdc86cd6-8gxrg" Nov 03 00:51:01.355017 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:01.354977 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-83755a/simulator-w3-83755a-kserve-77d6664559-psqtg"] Nov 03 00:51:01.358886 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:01.358859 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-83755a/simulator-w3-83755a-kserve-77d6664559-psqtg" Nov 03 00:51:01.365222 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:01.365171 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-83755a\"/\"kube-root-ca.crt\"" Nov 03 00:51:01.365222 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:01.365171 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-83755a\"/\"openshift-service-ca.crt\"" Nov 03 00:51:01.365762 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:01.365744 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-83755a\"/\"default-dockercfg-tmlfc\"" Nov 03 00:51:01.366074 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:01.365852 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-83755a\"/\"simulator-w3-83755a-kserve-self-signed-certs\"" Nov 03 00:51:01.368940 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:01.368914 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-83755a/simulator-w3-83755a-kserve-77d6664559-psqtg"] Nov 03 00:51:01.463814 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:01.463771 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0a4c234-af58-4da0-a1a4-e1f167953f6a-model-cache\") pod \"simulator-w3-83755a-kserve-77d6664559-psqtg\" (UID: \"b0a4c234-af58-4da0-a1a4-e1f167953f6a\") " pod="e2e-models-e2e-worker-w3-83755a/simulator-w3-83755a-kserve-77d6664559-psqtg" Nov 03 00:51:01.463990 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:01.463834 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0a4c234-af58-4da0-a1a4-e1f167953f6a-tmp-dir\") pod \"simulator-w3-83755a-kserve-77d6664559-psqtg\" (UID: \"b0a4c234-af58-4da0-a1a4-e1f167953f6a\") " pod="e2e-models-e2e-worker-w3-83755a/simulator-w3-83755a-kserve-77d6664559-psqtg" Nov 03 00:51:01.463990 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:01.463962 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qks2d\" (UniqueName: \"kubernetes.io/projected/b0a4c234-af58-4da0-a1a4-e1f167953f6a-kube-api-access-qks2d\") pod \"simulator-w3-83755a-kserve-77d6664559-psqtg\" (UID: \"b0a4c234-af58-4da0-a1a4-e1f167953f6a\") " pod="e2e-models-e2e-worker-w3-83755a/simulator-w3-83755a-kserve-77d6664559-psqtg" Nov 03 00:51:01.464086 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:01.464007 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0a4c234-af58-4da0-a1a4-e1f167953f6a-home\") pod \"simulator-w3-83755a-kserve-77d6664559-psqtg\" (UID: \"b0a4c234-af58-4da0-a1a4-e1f167953f6a\") " pod="e2e-models-e2e-worker-w3-83755a/simulator-w3-83755a-kserve-77d6664559-psqtg" Nov 03 00:51:01.464086 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:01.464027 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0a4c234-af58-4da0-a1a4-e1f167953f6a-dshm\") pod \"simulator-w3-83755a-kserve-77d6664559-psqtg\" (UID: \"b0a4c234-af58-4da0-a1a4-e1f167953f6a\") " pod="e2e-models-e2e-worker-w3-83755a/simulator-w3-83755a-kserve-77d6664559-psqtg" Nov 03 00:51:01.464178 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:01.464132 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0a4c234-af58-4da0-a1a4-e1f167953f6a-tls-certs\") pod \"simulator-w3-83755a-kserve-77d6664559-psqtg\" (UID: \"b0a4c234-af58-4da0-a1a4-e1f167953f6a\") " pod="e2e-models-e2e-worker-w3-83755a/simulator-w3-83755a-kserve-77d6664559-psqtg" Nov 03 00:51:01.565051 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:01.565015 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0a4c234-af58-4da0-a1a4-e1f167953f6a-model-cache\") pod \"simulator-w3-83755a-kserve-77d6664559-psqtg\" (UID: \"b0a4c234-af58-4da0-a1a4-e1f167953f6a\") " pod="e2e-models-e2e-worker-w3-83755a/simulator-w3-83755a-kserve-77d6664559-psqtg" Nov 03 00:51:01.565263 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:01.565058 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0a4c234-af58-4da0-a1a4-e1f167953f6a-tmp-dir\") pod \"simulator-w3-83755a-kserve-77d6664559-psqtg\" (UID: \"b0a4c234-af58-4da0-a1a4-e1f167953f6a\") " pod="e2e-models-e2e-worker-w3-83755a/simulator-w3-83755a-kserve-77d6664559-psqtg" Nov 03 00:51:01.565263 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:01.565123 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qks2d\" (UniqueName: \"kubernetes.io/projected/b0a4c234-af58-4da0-a1a4-e1f167953f6a-kube-api-access-qks2d\") pod \"simulator-w3-83755a-kserve-77d6664559-psqtg\" (UID: \"b0a4c234-af58-4da0-a1a4-e1f167953f6a\") " pod="e2e-models-e2e-worker-w3-83755a/simulator-w3-83755a-kserve-77d6664559-psqtg" Nov 03 00:51:01.565536 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:01.565506 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0a4c234-af58-4da0-a1a4-e1f167953f6a-model-cache\") pod \"simulator-w3-83755a-kserve-77d6664559-psqtg\" (UID: \"b0a4c234-af58-4da0-a1a4-e1f167953f6a\") " pod="e2e-models-e2e-worker-w3-83755a/simulator-w3-83755a-kserve-77d6664559-psqtg" Nov 03 00:51:01.565536 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:01.565531 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0a4c234-af58-4da0-a1a4-e1f167953f6a-home\") pod \"simulator-w3-83755a-kserve-77d6664559-psqtg\" (UID: \"b0a4c234-af58-4da0-a1a4-e1f167953f6a\") " pod="e2e-models-e2e-worker-w3-83755a/simulator-w3-83755a-kserve-77d6664559-psqtg" Nov 03 00:51:01.565741 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:01.565572 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0a4c234-af58-4da0-a1a4-e1f167953f6a-dshm\") pod \"simulator-w3-83755a-kserve-77d6664559-psqtg\" (UID: \"b0a4c234-af58-4da0-a1a4-e1f167953f6a\") " pod="e2e-models-e2e-worker-w3-83755a/simulator-w3-83755a-kserve-77d6664559-psqtg" Nov 03 00:51:01.565741 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:01.565658 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0a4c234-af58-4da0-a1a4-e1f167953f6a-tls-certs\") pod \"simulator-w3-83755a-kserve-77d6664559-psqtg\" (UID: \"b0a4c234-af58-4da0-a1a4-e1f167953f6a\") " pod="e2e-models-e2e-worker-w3-83755a/simulator-w3-83755a-kserve-77d6664559-psqtg" Nov 03 00:51:01.565852 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:01.565782 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0a4c234-af58-4da0-a1a4-e1f167953f6a-home\") pod \"simulator-w3-83755a-kserve-77d6664559-psqtg\" (UID: \"b0a4c234-af58-4da0-a1a4-e1f167953f6a\") " pod="e2e-models-e2e-worker-w3-83755a/simulator-w3-83755a-kserve-77d6664559-psqtg" Nov 03 00:51:01.565852 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:01.565577 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0a4c234-af58-4da0-a1a4-e1f167953f6a-tmp-dir\") pod \"simulator-w3-83755a-kserve-77d6664559-psqtg\" (UID: \"b0a4c234-af58-4da0-a1a4-e1f167953f6a\") " pod="e2e-models-e2e-worker-w3-83755a/simulator-w3-83755a-kserve-77d6664559-psqtg" Nov 03 00:51:01.567985 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:01.567956 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0a4c234-af58-4da0-a1a4-e1f167953f6a-dshm\") pod \"simulator-w3-83755a-kserve-77d6664559-psqtg\" (UID: \"b0a4c234-af58-4da0-a1a4-e1f167953f6a\") " pod="e2e-models-e2e-worker-w3-83755a/simulator-w3-83755a-kserve-77d6664559-psqtg" Nov 03 00:51:01.568539 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:01.568515 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0a4c234-af58-4da0-a1a4-e1f167953f6a-tls-certs\") pod \"simulator-w3-83755a-kserve-77d6664559-psqtg\" (UID: \"b0a4c234-af58-4da0-a1a4-e1f167953f6a\") " pod="e2e-models-e2e-worker-w3-83755a/simulator-w3-83755a-kserve-77d6664559-psqtg" Nov 03 00:51:01.573389 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:01.573364 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qks2d\" (UniqueName: \"kubernetes.io/projected/b0a4c234-af58-4da0-a1a4-e1f167953f6a-kube-api-access-qks2d\") pod \"simulator-w3-83755a-kserve-77d6664559-psqtg\" (UID: \"b0a4c234-af58-4da0-a1a4-e1f167953f6a\") " pod="e2e-models-e2e-worker-w3-83755a/simulator-w3-83755a-kserve-77d6664559-psqtg" Nov 03 00:51:01.673900 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:01.673814 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-83755a/simulator-w3-83755a-kserve-77d6664559-psqtg" Nov 03 00:51:01.974239 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:51:01.974193 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb0a4c234_af58_4da0_a1a4_e1f167953f6a.slice/crio-dc39bf25a53fecfe9cb1dcbbc512b0ae9c0e98d93e3a90dfb39c60eadf39f114 WatchSource:0}: Error finding container dc39bf25a53fecfe9cb1dcbbc512b0ae9c0e98d93e3a90dfb39c60eadf39f114: Status 404 returned error can't find the container with id dc39bf25a53fecfe9cb1dcbbc512b0ae9c0e98d93e3a90dfb39c60eadf39f114 Nov 03 00:51:01.978461 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:01.978426 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-83755a/simulator-w3-83755a-kserve-77d6664559-psqtg"] Nov 03 00:51:02.054491 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:02.052570 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-650be0/simulator-w4-650be0-kserve-7bc588774f-gvz2k" Nov 03 00:51:02.083315 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:02.080102 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-650be0/simulator-w4-650be0-kserve-7bc588774f-gvz2k" Nov 03 00:51:02.275203 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:02.275164 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-83755a/simulator-w3-83755a-kserve-77d6664559-psqtg" event={"ID":"b0a4c234-af58-4da0-a1a4-e1f167953f6a","Type":"ContainerStarted","Data":"c7df4530e82e735f7a56e2000452563afb60bde4657ba112acb41f449d0dbc27"} Nov 03 00:51:02.275446 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:02.275212 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-83755a/simulator-w3-83755a-kserve-77d6664559-psqtg" event={"ID":"b0a4c234-af58-4da0-a1a4-e1f167953f6a","Type":"ContainerStarted","Data":"dc39bf25a53fecfe9cb1dcbbc512b0ae9c0e98d93e3a90dfb39c60eadf39f114"} Nov 03 00:51:02.288405 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:02.288372 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-650be0/simulator-w4-650be0-kserve-7bc588774f-gvz2k" Nov 03 00:51:02.300574 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:02.300518 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-83755a/simulator-w3-83755a-kserve-77d6664559-psqtg" podStartSLOduration=1.30050305 podStartE2EDuration="1.30050305s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:51:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:51:02.298766456 +0000 UTC m=+604.469769542" watchObservedRunningTime="2026-11-03 00:51:02.30050305 +0000 UTC m=+604.471506145" Nov 03 00:51:02.674394 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:02.674302 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-83755a/simulator-w3-83755a-kserve-77d6664559-psqtg" Nov 03 00:51:09.722097 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:09.722057 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-f7680d/simulator-w2-f7680d-kserve-85bdc86cd6-8gxrg" Nov 03 00:51:09.734811 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:09.734782 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-f7680d/simulator-w2-f7680d-kserve-85bdc86cd6-8gxrg" Nov 03 00:51:10.331403 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:10.331370 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f7680d/simulator-w2-f7680d-kserve-85bdc86cd6-8gxrg" Nov 03 00:51:11.674111 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:11.674076 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-83755a/simulator-w3-83755a-kserve-77d6664559-psqtg" Nov 03 00:51:11.687096 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:11.687070 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-83755a/simulator-w3-83755a-kserve-77d6664559-psqtg" Nov 03 00:51:12.342680 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:12.342652 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-83755a/simulator-w3-83755a-kserve-77d6664559-psqtg" Nov 03 00:51:13.173695 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:13.173662 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-650be0/premium-w4-650be0-kserve-7bf8555dfb-z97n5"] Nov 03 00:51:13.177472 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:13.177448 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-650be0/premium-w4-650be0-kserve-7bf8555dfb-z97n5" Nov 03 00:51:13.184284 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:13.184242 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-650be0\"/\"premium-w4-650be0-kserve-self-signed-certs\"" Nov 03 00:51:13.188976 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:13.188942 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-650be0/premium-w4-650be0-kserve-7bf8555dfb-z97n5"] Nov 03 00:51:13.302749 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:13.302713 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40601dc5-808f-456f-826f-6e0c35966ada-tls-certs\") pod \"premium-w4-650be0-kserve-7bf8555dfb-z97n5\" (UID: \"40601dc5-808f-456f-826f-6e0c35966ada\") " pod="e2e-models-e2e-worker-w4-650be0/premium-w4-650be0-kserve-7bf8555dfb-z97n5" Nov 03 00:51:13.302918 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:13.302764 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40601dc5-808f-456f-826f-6e0c35966ada-tmp-dir\") pod \"premium-w4-650be0-kserve-7bf8555dfb-z97n5\" (UID: \"40601dc5-808f-456f-826f-6e0c35966ada\") " pod="e2e-models-e2e-worker-w4-650be0/premium-w4-650be0-kserve-7bf8555dfb-z97n5" Nov 03 00:51:13.302918 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:13.302877 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7f4hv\" (UniqueName: \"kubernetes.io/projected/40601dc5-808f-456f-826f-6e0c35966ada-kube-api-access-7f4hv\") pod \"premium-w4-650be0-kserve-7bf8555dfb-z97n5\" (UID: \"40601dc5-808f-456f-826f-6e0c35966ada\") " pod="e2e-models-e2e-worker-w4-650be0/premium-w4-650be0-kserve-7bf8555dfb-z97n5" Nov 03 00:51:13.302918 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:13.302897 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40601dc5-808f-456f-826f-6e0c35966ada-dshm\") pod \"premium-w4-650be0-kserve-7bf8555dfb-z97n5\" (UID: \"40601dc5-808f-456f-826f-6e0c35966ada\") " pod="e2e-models-e2e-worker-w4-650be0/premium-w4-650be0-kserve-7bf8555dfb-z97n5" Nov 03 00:51:13.303042 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:13.302919 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40601dc5-808f-456f-826f-6e0c35966ada-home\") pod \"premium-w4-650be0-kserve-7bf8555dfb-z97n5\" (UID: \"40601dc5-808f-456f-826f-6e0c35966ada\") " pod="e2e-models-e2e-worker-w4-650be0/premium-w4-650be0-kserve-7bf8555dfb-z97n5" Nov 03 00:51:13.303109 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:13.303080 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40601dc5-808f-456f-826f-6e0c35966ada-model-cache\") pod \"premium-w4-650be0-kserve-7bf8555dfb-z97n5\" (UID: \"40601dc5-808f-456f-826f-6e0c35966ada\") " pod="e2e-models-e2e-worker-w4-650be0/premium-w4-650be0-kserve-7bf8555dfb-z97n5" Nov 03 00:51:13.404349 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:13.404306 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40601dc5-808f-456f-826f-6e0c35966ada-tls-certs\") pod \"premium-w4-650be0-kserve-7bf8555dfb-z97n5\" (UID: \"40601dc5-808f-456f-826f-6e0c35966ada\") " pod="e2e-models-e2e-worker-w4-650be0/premium-w4-650be0-kserve-7bf8555dfb-z97n5" Nov 03 00:51:13.404539 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:13.404387 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40601dc5-808f-456f-826f-6e0c35966ada-tmp-dir\") pod \"premium-w4-650be0-kserve-7bf8555dfb-z97n5\" (UID: \"40601dc5-808f-456f-826f-6e0c35966ada\") " pod="e2e-models-e2e-worker-w4-650be0/premium-w4-650be0-kserve-7bf8555dfb-z97n5" Nov 03 00:51:13.404539 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:13.404432 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7f4hv\" (UniqueName: \"kubernetes.io/projected/40601dc5-808f-456f-826f-6e0c35966ada-kube-api-access-7f4hv\") pod \"premium-w4-650be0-kserve-7bf8555dfb-z97n5\" (UID: \"40601dc5-808f-456f-826f-6e0c35966ada\") " pod="e2e-models-e2e-worker-w4-650be0/premium-w4-650be0-kserve-7bf8555dfb-z97n5" Nov 03 00:51:13.404539 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:13.404457 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40601dc5-808f-456f-826f-6e0c35966ada-dshm\") pod \"premium-w4-650be0-kserve-7bf8555dfb-z97n5\" (UID: \"40601dc5-808f-456f-826f-6e0c35966ada\") " pod="e2e-models-e2e-worker-w4-650be0/premium-w4-650be0-kserve-7bf8555dfb-z97n5" Nov 03 00:51:13.404539 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:13.404484 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40601dc5-808f-456f-826f-6e0c35966ada-home\") pod \"premium-w4-650be0-kserve-7bf8555dfb-z97n5\" (UID: \"40601dc5-808f-456f-826f-6e0c35966ada\") " pod="e2e-models-e2e-worker-w4-650be0/premium-w4-650be0-kserve-7bf8555dfb-z97n5" Nov 03 00:51:13.404753 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:13.404553 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40601dc5-808f-456f-826f-6e0c35966ada-model-cache\") pod \"premium-w4-650be0-kserve-7bf8555dfb-z97n5\" (UID: \"40601dc5-808f-456f-826f-6e0c35966ada\") " pod="e2e-models-e2e-worker-w4-650be0/premium-w4-650be0-kserve-7bf8555dfb-z97n5" Nov 03 00:51:13.405025 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:13.404997 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40601dc5-808f-456f-826f-6e0c35966ada-model-cache\") pod \"premium-w4-650be0-kserve-7bf8555dfb-z97n5\" (UID: \"40601dc5-808f-456f-826f-6e0c35966ada\") " pod="e2e-models-e2e-worker-w4-650be0/premium-w4-650be0-kserve-7bf8555dfb-z97n5" Nov 03 00:51:13.406370 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:13.405474 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40601dc5-808f-456f-826f-6e0c35966ada-home\") pod \"premium-w4-650be0-kserve-7bf8555dfb-z97n5\" (UID: \"40601dc5-808f-456f-826f-6e0c35966ada\") " pod="e2e-models-e2e-worker-w4-650be0/premium-w4-650be0-kserve-7bf8555dfb-z97n5" Nov 03 00:51:13.406575 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:13.406161 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40601dc5-808f-456f-826f-6e0c35966ada-tmp-dir\") pod \"premium-w4-650be0-kserve-7bf8555dfb-z97n5\" (UID: \"40601dc5-808f-456f-826f-6e0c35966ada\") " pod="e2e-models-e2e-worker-w4-650be0/premium-w4-650be0-kserve-7bf8555dfb-z97n5" Nov 03 00:51:13.410317 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:13.409491 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40601dc5-808f-456f-826f-6e0c35966ada-tls-certs\") pod \"premium-w4-650be0-kserve-7bf8555dfb-z97n5\" (UID: \"40601dc5-808f-456f-826f-6e0c35966ada\") " pod="e2e-models-e2e-worker-w4-650be0/premium-w4-650be0-kserve-7bf8555dfb-z97n5" Nov 03 00:51:13.415307 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:13.413651 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40601dc5-808f-456f-826f-6e0c35966ada-dshm\") pod \"premium-w4-650be0-kserve-7bf8555dfb-z97n5\" (UID: \"40601dc5-808f-456f-826f-6e0c35966ada\") " pod="e2e-models-e2e-worker-w4-650be0/premium-w4-650be0-kserve-7bf8555dfb-z97n5" Nov 03 00:51:13.415307 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:13.415039 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7f4hv\" (UniqueName: \"kubernetes.io/projected/40601dc5-808f-456f-826f-6e0c35966ada-kube-api-access-7f4hv\") pod \"premium-w4-650be0-kserve-7bf8555dfb-z97n5\" (UID: \"40601dc5-808f-456f-826f-6e0c35966ada\") " pod="e2e-models-e2e-worker-w4-650be0/premium-w4-650be0-kserve-7bf8555dfb-z97n5" Nov 03 00:51:13.494058 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:13.493965 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-650be0/premium-w4-650be0-kserve-7bf8555dfb-z97n5" Nov 03 00:51:13.773781 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:13.773753 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-650be0/premium-w4-650be0-kserve-7bf8555dfb-z97n5"] Nov 03 00:51:13.775657 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:51:13.775631 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod40601dc5_808f_456f_826f_6e0c35966ada.slice/crio-87d84786c0e81f9cc4000674d9160fbc1492c6ec2b42588206fae48b26988092 WatchSource:0}: Error finding container 87d84786c0e81f9cc4000674d9160fbc1492c6ec2b42588206fae48b26988092: Status 404 returned error can't find the container with id 87d84786c0e81f9cc4000674d9160fbc1492c6ec2b42588206fae48b26988092 Nov 03 00:51:14.346639 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:14.346591 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-650be0/premium-w4-650be0-kserve-7bf8555dfb-z97n5" event={"ID":"40601dc5-808f-456f-826f-6e0c35966ada","Type":"ContainerStarted","Data":"0e06adc10ca35db139d1d5f7954e16ef1af7e2c4f7d1c6a891f85baaf4de03fb"} Nov 03 00:51:14.346639 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:14.346640 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-650be0/premium-w4-650be0-kserve-7bf8555dfb-z97n5" event={"ID":"40601dc5-808f-456f-826f-6e0c35966ada","Type":"ContainerStarted","Data":"87d84786c0e81f9cc4000674d9160fbc1492c6ec2b42588206fae48b26988092"} Nov 03 00:51:14.363924 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:14.363874 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-650be0/premium-w4-650be0-kserve-7bf8555dfb-z97n5" podStartSLOduration=1.36385753 podStartE2EDuration="1.36385753s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:51:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:51:14.362713917 +0000 UTC m=+616.533717014" watchObservedRunningTime="2026-11-03 00:51:14.36385753 +0000 UTC m=+616.534860625" Nov 03 00:51:14.494342 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:14.494309 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-650be0/premium-w4-650be0-kserve-7bf8555dfb-z97n5" Nov 03 00:51:14.669721 ip-10-0-133-217 kubenswrapper[4421]: E1103 00:51:14.669634 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 00:51:23.494799 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:23.494758 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-650be0/premium-w4-650be0-kserve-7bf8555dfb-z97n5" Nov 03 00:51:23.507795 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:23.507765 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-650be0/premium-w4-650be0-kserve-7bf8555dfb-z97n5" Nov 03 00:51:24.415773 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:24.415741 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-650be0/premium-w4-650be0-kserve-7bf8555dfb-z97n5" Nov 03 00:51:29.653297 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:29.635934 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-83755a/premium-w3-83755a-kserve-7c9d66988b-lt4xd"] Nov 03 00:51:29.662541 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:29.659900 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-83755a/premium-w3-83755a-kserve-7c9d66988b-lt4xd"] Nov 03 00:51:29.662541 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:29.660024 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-83755a/premium-w3-83755a-kserve-7c9d66988b-lt4xd" Nov 03 00:51:29.673610 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:29.673578 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-83755a\"/\"premium-w3-83755a-kserve-self-signed-certs\"" Nov 03 00:51:29.813669 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:29.813632 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0c6b02e-2ecb-428e-a13d-be6d7cd1ba39-tmp-dir\") pod \"premium-w3-83755a-kserve-7c9d66988b-lt4xd\" (UID: \"c0c6b02e-2ecb-428e-a13d-be6d7cd1ba39\") " pod="e2e-models-e2e-worker-w3-83755a/premium-w3-83755a-kserve-7c9d66988b-lt4xd" Nov 03 00:51:29.813887 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:29.813689 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0c6b02e-2ecb-428e-a13d-be6d7cd1ba39-dshm\") pod \"premium-w3-83755a-kserve-7c9d66988b-lt4xd\" (UID: \"c0c6b02e-2ecb-428e-a13d-be6d7cd1ba39\") " pod="e2e-models-e2e-worker-w3-83755a/premium-w3-83755a-kserve-7c9d66988b-lt4xd" Nov 03 00:51:29.813887 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:29.813716 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0c6b02e-2ecb-428e-a13d-be6d7cd1ba39-home\") pod \"premium-w3-83755a-kserve-7c9d66988b-lt4xd\" (UID: \"c0c6b02e-2ecb-428e-a13d-be6d7cd1ba39\") " pod="e2e-models-e2e-worker-w3-83755a/premium-w3-83755a-kserve-7c9d66988b-lt4xd" Nov 03 00:51:29.813887 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:29.813821 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvl9h\" (UniqueName: \"kubernetes.io/projected/c0c6b02e-2ecb-428e-a13d-be6d7cd1ba39-kube-api-access-jvl9h\") pod \"premium-w3-83755a-kserve-7c9d66988b-lt4xd\" (UID: \"c0c6b02e-2ecb-428e-a13d-be6d7cd1ba39\") " pod="e2e-models-e2e-worker-w3-83755a/premium-w3-83755a-kserve-7c9d66988b-lt4xd" Nov 03 00:51:29.814038 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:29.813889 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0c6b02e-2ecb-428e-a13d-be6d7cd1ba39-tls-certs\") pod \"premium-w3-83755a-kserve-7c9d66988b-lt4xd\" (UID: \"c0c6b02e-2ecb-428e-a13d-be6d7cd1ba39\") " pod="e2e-models-e2e-worker-w3-83755a/premium-w3-83755a-kserve-7c9d66988b-lt4xd" Nov 03 00:51:29.814038 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:29.814008 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0c6b02e-2ecb-428e-a13d-be6d7cd1ba39-model-cache\") pod \"premium-w3-83755a-kserve-7c9d66988b-lt4xd\" (UID: \"c0c6b02e-2ecb-428e-a13d-be6d7cd1ba39\") " pod="e2e-models-e2e-worker-w3-83755a/premium-w3-83755a-kserve-7c9d66988b-lt4xd" Nov 03 00:51:29.915168 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:29.915071 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0c6b02e-2ecb-428e-a13d-be6d7cd1ba39-tmp-dir\") pod \"premium-w3-83755a-kserve-7c9d66988b-lt4xd\" (UID: \"c0c6b02e-2ecb-428e-a13d-be6d7cd1ba39\") " pod="e2e-models-e2e-worker-w3-83755a/premium-w3-83755a-kserve-7c9d66988b-lt4xd" Nov 03 00:51:29.915168 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:29.915126 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0c6b02e-2ecb-428e-a13d-be6d7cd1ba39-dshm\") pod \"premium-w3-83755a-kserve-7c9d66988b-lt4xd\" (UID: \"c0c6b02e-2ecb-428e-a13d-be6d7cd1ba39\") " pod="e2e-models-e2e-worker-w3-83755a/premium-w3-83755a-kserve-7c9d66988b-lt4xd" Nov 03 00:51:29.915443 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:29.915224 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0c6b02e-2ecb-428e-a13d-be6d7cd1ba39-home\") pod \"premium-w3-83755a-kserve-7c9d66988b-lt4xd\" (UID: \"c0c6b02e-2ecb-428e-a13d-be6d7cd1ba39\") " pod="e2e-models-e2e-worker-w3-83755a/premium-w3-83755a-kserve-7c9d66988b-lt4xd" Nov 03 00:51:29.915443 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:29.915293 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvl9h\" (UniqueName: \"kubernetes.io/projected/c0c6b02e-2ecb-428e-a13d-be6d7cd1ba39-kube-api-access-jvl9h\") pod \"premium-w3-83755a-kserve-7c9d66988b-lt4xd\" (UID: \"c0c6b02e-2ecb-428e-a13d-be6d7cd1ba39\") " pod="e2e-models-e2e-worker-w3-83755a/premium-w3-83755a-kserve-7c9d66988b-lt4xd" Nov 03 00:51:29.915443 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:29.915362 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0c6b02e-2ecb-428e-a13d-be6d7cd1ba39-tls-certs\") pod \"premium-w3-83755a-kserve-7c9d66988b-lt4xd\" (UID: \"c0c6b02e-2ecb-428e-a13d-be6d7cd1ba39\") " pod="e2e-models-e2e-worker-w3-83755a/premium-w3-83755a-kserve-7c9d66988b-lt4xd" Nov 03 00:51:29.915606 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:29.915533 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0c6b02e-2ecb-428e-a13d-be6d7cd1ba39-model-cache\") pod \"premium-w3-83755a-kserve-7c9d66988b-lt4xd\" (UID: \"c0c6b02e-2ecb-428e-a13d-be6d7cd1ba39\") " pod="e2e-models-e2e-worker-w3-83755a/premium-w3-83755a-kserve-7c9d66988b-lt4xd" Nov 03 00:51:29.915665 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:29.915635 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0c6b02e-2ecb-428e-a13d-be6d7cd1ba39-tmp-dir\") pod \"premium-w3-83755a-kserve-7c9d66988b-lt4xd\" (UID: \"c0c6b02e-2ecb-428e-a13d-be6d7cd1ba39\") " pod="e2e-models-e2e-worker-w3-83755a/premium-w3-83755a-kserve-7c9d66988b-lt4xd" Nov 03 00:51:29.915724 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:29.915690 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0c6b02e-2ecb-428e-a13d-be6d7cd1ba39-home\") pod \"premium-w3-83755a-kserve-7c9d66988b-lt4xd\" (UID: \"c0c6b02e-2ecb-428e-a13d-be6d7cd1ba39\") " pod="e2e-models-e2e-worker-w3-83755a/premium-w3-83755a-kserve-7c9d66988b-lt4xd" Nov 03 00:51:29.915824 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:29.915802 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0c6b02e-2ecb-428e-a13d-be6d7cd1ba39-model-cache\") pod \"premium-w3-83755a-kserve-7c9d66988b-lt4xd\" (UID: \"c0c6b02e-2ecb-428e-a13d-be6d7cd1ba39\") " pod="e2e-models-e2e-worker-w3-83755a/premium-w3-83755a-kserve-7c9d66988b-lt4xd" Nov 03 00:51:29.917666 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:29.917638 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0c6b02e-2ecb-428e-a13d-be6d7cd1ba39-dshm\") pod \"premium-w3-83755a-kserve-7c9d66988b-lt4xd\" (UID: \"c0c6b02e-2ecb-428e-a13d-be6d7cd1ba39\") " pod="e2e-models-e2e-worker-w3-83755a/premium-w3-83755a-kserve-7c9d66988b-lt4xd" Nov 03 00:51:29.917782 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:29.917770 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0c6b02e-2ecb-428e-a13d-be6d7cd1ba39-tls-certs\") pod \"premium-w3-83755a-kserve-7c9d66988b-lt4xd\" (UID: \"c0c6b02e-2ecb-428e-a13d-be6d7cd1ba39\") " pod="e2e-models-e2e-worker-w3-83755a/premium-w3-83755a-kserve-7c9d66988b-lt4xd" Nov 03 00:51:29.923191 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:29.923169 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvl9h\" (UniqueName: \"kubernetes.io/projected/c0c6b02e-2ecb-428e-a13d-be6d7cd1ba39-kube-api-access-jvl9h\") pod \"premium-w3-83755a-kserve-7c9d66988b-lt4xd\" (UID: \"c0c6b02e-2ecb-428e-a13d-be6d7cd1ba39\") " pod="e2e-models-e2e-worker-w3-83755a/premium-w3-83755a-kserve-7c9d66988b-lt4xd" Nov 03 00:51:30.021203 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:30.021159 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-83755a/premium-w3-83755a-kserve-7c9d66988b-lt4xd" Nov 03 00:51:30.308000 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:30.307971 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-83755a/premium-w3-83755a-kserve-7c9d66988b-lt4xd"] Nov 03 00:51:30.310031 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:51:30.310003 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc0c6b02e_2ecb_428e_a13d_be6d7cd1ba39.slice/crio-a83c0396d408d40610706e5bdfd7ca4a0fdbf2fd9a7c96f054803050cd4b16d4 WatchSource:0}: Error finding container a83c0396d408d40610706e5bdfd7ca4a0fdbf2fd9a7c96f054803050cd4b16d4: Status 404 returned error can't find the container with id a83c0396d408d40610706e5bdfd7ca4a0fdbf2fd9a7c96f054803050cd4b16d4 Nov 03 00:51:30.444353 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:30.444314 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-83755a/premium-w3-83755a-kserve-7c9d66988b-lt4xd" event={"ID":"c0c6b02e-2ecb-428e-a13d-be6d7cd1ba39","Type":"ContainerStarted","Data":"7b75b22b2c90201f0aadd6d4faee52c88b19a60528330311b41922eb76153cae"} Nov 03 00:51:30.444527 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:30.444359 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-83755a/premium-w3-83755a-kserve-7c9d66988b-lt4xd" event={"ID":"c0c6b02e-2ecb-428e-a13d-be6d7cd1ba39","Type":"ContainerStarted","Data":"a83c0396d408d40610706e5bdfd7ca4a0fdbf2fd9a7c96f054803050cd4b16d4"} Nov 03 00:51:30.463245 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:30.463192 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-83755a/premium-w3-83755a-kserve-7c9d66988b-lt4xd" podStartSLOduration=1.463179528 podStartE2EDuration="1.463179528s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:51:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:51:30.461458196 +0000 UTC m=+632.632461285" watchObservedRunningTime="2026-11-03 00:51:30.463179528 +0000 UTC m=+632.634182654" Nov 03 00:51:30.553750 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:30.553712 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-h2p2p"] Nov 03 00:51:30.557823 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:30.557761 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-h2p2p" Nov 03 00:51:30.564988 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:30.564929 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-83755a-openshift-default-dockercfg-qhtrs\"" Nov 03 00:51:30.568629 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:30.568599 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-h2p2p"] Nov 03 00:51:30.724213 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:30.724168 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bd6d60e-043b-47ba-beb1-08b1eaace690-workload-socket\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-h2p2p\" (UID: \"5bd6d60e-043b-47ba-beb1-08b1eaace690\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-h2p2p" Nov 03 00:51:30.724213 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:30.724221 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bd6d60e-043b-47ba-beb1-08b1eaace690-istio-envoy\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-h2p2p\" (UID: \"5bd6d60e-043b-47ba-beb1-08b1eaace690\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-h2p2p" Nov 03 00:51:30.724689 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:30.724238 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bd6d60e-043b-47ba-beb1-08b1eaace690-istiod-ca-cert\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-h2p2p\" (UID: \"5bd6d60e-043b-47ba-beb1-08b1eaace690\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-h2p2p" Nov 03 00:51:30.724689 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:30.724413 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bd6d60e-043b-47ba-beb1-08b1eaace690-istio-data\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-h2p2p\" (UID: \"5bd6d60e-043b-47ba-beb1-08b1eaace690\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-h2p2p" Nov 03 00:51:30.724689 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:30.724532 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnlld\" (UniqueName: \"kubernetes.io/projected/5bd6d60e-043b-47ba-beb1-08b1eaace690-kube-api-access-tnlld\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-h2p2p\" (UID: \"5bd6d60e-043b-47ba-beb1-08b1eaace690\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-h2p2p" Nov 03 00:51:30.724689 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:30.724563 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bd6d60e-043b-47ba-beb1-08b1eaace690-istio-token\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-h2p2p\" (UID: \"5bd6d60e-043b-47ba-beb1-08b1eaace690\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-h2p2p" Nov 03 00:51:30.724689 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:30.724653 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bd6d60e-043b-47ba-beb1-08b1eaace690-credential-socket\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-h2p2p\" (UID: \"5bd6d60e-043b-47ba-beb1-08b1eaace690\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-h2p2p" Nov 03 00:51:30.724689 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:30.724689 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bd6d60e-043b-47ba-beb1-08b1eaace690-istio-podinfo\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-h2p2p\" (UID: \"5bd6d60e-043b-47ba-beb1-08b1eaace690\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-h2p2p" Nov 03 00:51:30.724901 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:30.724730 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bd6d60e-043b-47ba-beb1-08b1eaace690-workload-certs\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-h2p2p\" (UID: \"5bd6d60e-043b-47ba-beb1-08b1eaace690\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-h2p2p" Nov 03 00:51:30.825593 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:30.825483 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bd6d60e-043b-47ba-beb1-08b1eaace690-istio-podinfo\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-h2p2p\" (UID: \"5bd6d60e-043b-47ba-beb1-08b1eaace690\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-h2p2p" Nov 03 00:51:30.825593 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:30.825554 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bd6d60e-043b-47ba-beb1-08b1eaace690-workload-certs\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-h2p2p\" (UID: \"5bd6d60e-043b-47ba-beb1-08b1eaace690\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-h2p2p" Nov 03 00:51:30.825593 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:30.825578 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bd6d60e-043b-47ba-beb1-08b1eaace690-workload-socket\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-h2p2p\" (UID: \"5bd6d60e-043b-47ba-beb1-08b1eaace690\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-h2p2p" Nov 03 00:51:30.825887 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:30.825606 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bd6d60e-043b-47ba-beb1-08b1eaace690-istio-envoy\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-h2p2p\" (UID: \"5bd6d60e-043b-47ba-beb1-08b1eaace690\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-h2p2p" Nov 03 00:51:30.825887 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:30.825630 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bd6d60e-043b-47ba-beb1-08b1eaace690-istiod-ca-cert\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-h2p2p\" (UID: \"5bd6d60e-043b-47ba-beb1-08b1eaace690\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-h2p2p" Nov 03 00:51:30.825887 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:30.825672 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bd6d60e-043b-47ba-beb1-08b1eaace690-istio-data\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-h2p2p\" (UID: \"5bd6d60e-043b-47ba-beb1-08b1eaace690\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-h2p2p" Nov 03 00:51:30.825887 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:30.825759 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnlld\" (UniqueName: \"kubernetes.io/projected/5bd6d60e-043b-47ba-beb1-08b1eaace690-kube-api-access-tnlld\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-h2p2p\" (UID: \"5bd6d60e-043b-47ba-beb1-08b1eaace690\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-h2p2p" Nov 03 00:51:30.825887 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:30.825784 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bd6d60e-043b-47ba-beb1-08b1eaace690-istio-token\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-h2p2p\" (UID: \"5bd6d60e-043b-47ba-beb1-08b1eaace690\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-h2p2p" Nov 03 00:51:30.825887 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:30.825856 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bd6d60e-043b-47ba-beb1-08b1eaace690-credential-socket\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-h2p2p\" (UID: \"5bd6d60e-043b-47ba-beb1-08b1eaace690\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-h2p2p" Nov 03 00:51:30.826209 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:30.826166 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bd6d60e-043b-47ba-beb1-08b1eaace690-workload-socket\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-h2p2p\" (UID: \"5bd6d60e-043b-47ba-beb1-08b1eaace690\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-h2p2p" Nov 03 00:51:30.826265 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:30.826204 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bd6d60e-043b-47ba-beb1-08b1eaace690-istio-data\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-h2p2p\" (UID: \"5bd6d60e-043b-47ba-beb1-08b1eaace690\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-h2p2p" Nov 03 00:51:30.826363 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:30.826264 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bd6d60e-043b-47ba-beb1-08b1eaace690-credential-socket\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-h2p2p\" (UID: \"5bd6d60e-043b-47ba-beb1-08b1eaace690\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-h2p2p" Nov 03 00:51:30.826466 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:30.826444 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bd6d60e-043b-47ba-beb1-08b1eaace690-workload-certs\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-h2p2p\" (UID: \"5bd6d60e-043b-47ba-beb1-08b1eaace690\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-h2p2p" Nov 03 00:51:30.826562 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:30.826536 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bd6d60e-043b-47ba-beb1-08b1eaace690-istiod-ca-cert\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-h2p2p\" (UID: \"5bd6d60e-043b-47ba-beb1-08b1eaace690\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-h2p2p" Nov 03 00:51:30.828061 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:30.828042 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bd6d60e-043b-47ba-beb1-08b1eaace690-istio-envoy\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-h2p2p\" (UID: \"5bd6d60e-043b-47ba-beb1-08b1eaace690\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-h2p2p" Nov 03 00:51:30.828211 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:30.828195 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bd6d60e-043b-47ba-beb1-08b1eaace690-istio-podinfo\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-h2p2p\" (UID: \"5bd6d60e-043b-47ba-beb1-08b1eaace690\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-h2p2p" Nov 03 00:51:30.833946 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:30.833924 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bd6d60e-043b-47ba-beb1-08b1eaace690-istio-token\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-h2p2p\" (UID: \"5bd6d60e-043b-47ba-beb1-08b1eaace690\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-h2p2p" Nov 03 00:51:30.834126 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:30.834108 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnlld\" (UniqueName: \"kubernetes.io/projected/5bd6d60e-043b-47ba-beb1-08b1eaace690-kube-api-access-tnlld\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-h2p2p\" (UID: \"5bd6d60e-043b-47ba-beb1-08b1eaace690\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-h2p2p" Nov 03 00:51:30.874034 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:30.873994 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-h2p2p" Nov 03 00:51:31.021455 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:31.021426 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-83755a/premium-w3-83755a-kserve-7c9d66988b-lt4xd" Nov 03 00:51:31.156374 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:31.156347 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-h2p2p"] Nov 03 00:51:31.158442 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:51:31.158401 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bd6d60e_043b_47ba_beb1_08b1eaace690.slice/crio-0757afa1f0249ed383eb1f991017a880274a253cf6fd37a8049a09bc9a40f6d5 WatchSource:0}: Error finding container 0757afa1f0249ed383eb1f991017a880274a253cf6fd37a8049a09bc9a40f6d5: Status 404 returned error can't find the container with id 0757afa1f0249ed383eb1f991017a880274a253cf6fd37a8049a09bc9a40f6d5 Nov 03 00:51:31.160959 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:31.160922 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:51:31.161065 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:31.160992 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:51:31.161065 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:31.161020 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:51:31.452590 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:31.452483 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-h2p2p" event={"ID":"5bd6d60e-043b-47ba-beb1-08b1eaace690","Type":"ContainerStarted","Data":"bd1518b1ffc13c09f9f41b5e8c7aebdfa23b77711a7dd2f01a1878c5a1f832e5"} Nov 03 00:51:31.452590 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:31.452536 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-h2p2p" event={"ID":"5bd6d60e-043b-47ba-beb1-08b1eaace690","Type":"ContainerStarted","Data":"0757afa1f0249ed383eb1f991017a880274a253cf6fd37a8049a09bc9a40f6d5"} Nov 03 00:51:31.472763 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:31.472700 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-h2p2p" podStartSLOduration=1.4726782680000001 podStartE2EDuration="1.472678268s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:51:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:51:31.470707738 +0000 UTC m=+633.641710920" watchObservedRunningTime="2026-11-03 00:51:31.472678268 +0000 UTC m=+633.643681365" Nov 03 00:51:31.875102 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:31.875069 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-h2p2p" Nov 03 00:51:32.876497 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:32.876459 4421 patch_prober.go:29] interesting pod/e2e-worker-w3-83755a-openshift-default-c86f9565c-h2p2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Nov 03 00:51:32.876892 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:32.876514 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-h2p2p" podUID="5bd6d60e-043b-47ba-beb1-08b1eaace690" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Nov 03 00:51:33.875372 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:33.875326 4421 patch_prober.go:29] interesting pod/e2e-worker-w3-83755a-openshift-default-c86f9565c-h2p2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Nov 03 00:51:33.875597 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:33.875396 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-h2p2p" podUID="5bd6d60e-043b-47ba-beb1-08b1eaace690" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Nov 03 00:51:34.875074 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:34.875009 4421 patch_prober.go:29] interesting pod/e2e-worker-w3-83755a-openshift-default-c86f9565c-h2p2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Nov 03 00:51:34.875632 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:34.875107 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-h2p2p" podUID="5bd6d60e-043b-47ba-beb1-08b1eaace690" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Nov 03 00:51:35.875004 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:35.874967 4421 patch_prober.go:29] interesting pod/e2e-worker-w3-83755a-openshift-default-c86f9565c-h2p2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Nov 03 00:51:35.875254 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:35.875230 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-h2p2p" podUID="5bd6d60e-043b-47ba-beb1-08b1eaace690" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Nov 03 00:51:36.875206 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:36.875171 4421 patch_prober.go:29] interesting pod/e2e-worker-w3-83755a-openshift-default-c86f9565c-h2p2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Nov 03 00:51:36.875400 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:36.875220 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-h2p2p" podUID="5bd6d60e-043b-47ba-beb1-08b1eaace690" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Nov 03 00:51:37.879718 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:37.879686 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-h2p2p" Nov 03 00:51:37.880190 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:37.880065 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-h2p2p" Nov 03 00:51:37.880783 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:37.880760 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-h2p2p" Nov 03 00:51:40.022262 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:40.022218 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-83755a/premium-w3-83755a-kserve-7c9d66988b-lt4xd" Nov 03 00:51:40.035583 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:40.035546 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-83755a/premium-w3-83755a-kserve-7c9d66988b-lt4xd" Nov 03 00:51:40.526916 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:40.526880 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-83755a/premium-w3-83755a-kserve-7c9d66988b-lt4xd" Nov 03 00:51:41.719846 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:41.719809 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rgsmg"] Nov 03 00:51:41.724884 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:41.724858 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rgsmg" Nov 03 00:51:41.728546 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:41.728522 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-650be0-openshift-default-dockercfg-l27n6\"" Nov 03 00:51:41.736058 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:41.736031 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rgsmg"] Nov 03 00:51:41.845652 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:41.845613 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/145e8805-e1b4-4f1f-80d0-86c7da396aee-istiod-ca-cert\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rgsmg\" (UID: \"145e8805-e1b4-4f1f-80d0-86c7da396aee\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rgsmg" Nov 03 00:51:41.845652 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:41.845658 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2chs\" (UniqueName: \"kubernetes.io/projected/145e8805-e1b4-4f1f-80d0-86c7da396aee-kube-api-access-n2chs\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rgsmg\" (UID: \"145e8805-e1b4-4f1f-80d0-86c7da396aee\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rgsmg" Nov 03 00:51:41.845908 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:41.845800 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/145e8805-e1b4-4f1f-80d0-86c7da396aee-workload-socket\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rgsmg\" (UID: \"145e8805-e1b4-4f1f-80d0-86c7da396aee\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rgsmg" Nov 03 00:51:41.845908 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:41.845845 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/145e8805-e1b4-4f1f-80d0-86c7da396aee-istio-data\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rgsmg\" (UID: \"145e8805-e1b4-4f1f-80d0-86c7da396aee\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rgsmg" Nov 03 00:51:41.845908 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:41.845865 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/145e8805-e1b4-4f1f-80d0-86c7da396aee-istio-podinfo\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rgsmg\" (UID: \"145e8805-e1b4-4f1f-80d0-86c7da396aee\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rgsmg" Nov 03 00:51:41.845908 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:41.845899 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/145e8805-e1b4-4f1f-80d0-86c7da396aee-istio-token\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rgsmg\" (UID: \"145e8805-e1b4-4f1f-80d0-86c7da396aee\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rgsmg" Nov 03 00:51:41.846116 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:41.846020 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/145e8805-e1b4-4f1f-80d0-86c7da396aee-credential-socket\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rgsmg\" (UID: \"145e8805-e1b4-4f1f-80d0-86c7da396aee\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rgsmg" Nov 03 00:51:41.846116 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:41.846058 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/145e8805-e1b4-4f1f-80d0-86c7da396aee-istio-envoy\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rgsmg\" (UID: \"145e8805-e1b4-4f1f-80d0-86c7da396aee\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rgsmg" Nov 03 00:51:41.846190 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:41.846144 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/145e8805-e1b4-4f1f-80d0-86c7da396aee-workload-certs\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rgsmg\" (UID: \"145e8805-e1b4-4f1f-80d0-86c7da396aee\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rgsmg" Nov 03 00:51:41.947730 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:41.947689 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/145e8805-e1b4-4f1f-80d0-86c7da396aee-workload-socket\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rgsmg\" (UID: \"145e8805-e1b4-4f1f-80d0-86c7da396aee\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rgsmg" Nov 03 00:51:41.947923 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:41.947740 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/145e8805-e1b4-4f1f-80d0-86c7da396aee-istio-data\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rgsmg\" (UID: \"145e8805-e1b4-4f1f-80d0-86c7da396aee\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rgsmg" Nov 03 00:51:41.947923 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:41.947759 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/145e8805-e1b4-4f1f-80d0-86c7da396aee-istio-podinfo\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rgsmg\" (UID: \"145e8805-e1b4-4f1f-80d0-86c7da396aee\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rgsmg" Nov 03 00:51:41.947923 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:41.947790 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/145e8805-e1b4-4f1f-80d0-86c7da396aee-istio-token\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rgsmg\" (UID: \"145e8805-e1b4-4f1f-80d0-86c7da396aee\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rgsmg" Nov 03 00:51:41.947923 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:41.947815 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/145e8805-e1b4-4f1f-80d0-86c7da396aee-credential-socket\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rgsmg\" (UID: \"145e8805-e1b4-4f1f-80d0-86c7da396aee\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rgsmg" Nov 03 00:51:41.947923 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:41.947842 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/145e8805-e1b4-4f1f-80d0-86c7da396aee-istio-envoy\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rgsmg\" (UID: \"145e8805-e1b4-4f1f-80d0-86c7da396aee\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rgsmg" Nov 03 00:51:41.947923 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:41.947913 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/145e8805-e1b4-4f1f-80d0-86c7da396aee-workload-certs\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rgsmg\" (UID: \"145e8805-e1b4-4f1f-80d0-86c7da396aee\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rgsmg" Nov 03 00:51:41.948223 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:41.947978 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/145e8805-e1b4-4f1f-80d0-86c7da396aee-istiod-ca-cert\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rgsmg\" (UID: \"145e8805-e1b4-4f1f-80d0-86c7da396aee\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rgsmg" Nov 03 00:51:41.948223 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:41.948013 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2chs\" (UniqueName: \"kubernetes.io/projected/145e8805-e1b4-4f1f-80d0-86c7da396aee-kube-api-access-n2chs\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rgsmg\" (UID: \"145e8805-e1b4-4f1f-80d0-86c7da396aee\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rgsmg" Nov 03 00:51:41.948223 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:41.948192 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/145e8805-e1b4-4f1f-80d0-86c7da396aee-istio-data\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rgsmg\" (UID: \"145e8805-e1b4-4f1f-80d0-86c7da396aee\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rgsmg" Nov 03 00:51:41.948413 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:41.948253 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/145e8805-e1b4-4f1f-80d0-86c7da396aee-workload-socket\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rgsmg\" (UID: \"145e8805-e1b4-4f1f-80d0-86c7da396aee\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rgsmg" Nov 03 00:51:41.948413 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:41.948340 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/145e8805-e1b4-4f1f-80d0-86c7da396aee-credential-socket\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rgsmg\" (UID: \"145e8805-e1b4-4f1f-80d0-86c7da396aee\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rgsmg" Nov 03 00:51:41.948512 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:41.948484 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/145e8805-e1b4-4f1f-80d0-86c7da396aee-workload-certs\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rgsmg\" (UID: \"145e8805-e1b4-4f1f-80d0-86c7da396aee\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rgsmg" Nov 03 00:51:41.948789 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:41.948762 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/145e8805-e1b4-4f1f-80d0-86c7da396aee-istiod-ca-cert\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rgsmg\" (UID: \"145e8805-e1b4-4f1f-80d0-86c7da396aee\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rgsmg" Nov 03 00:51:41.950407 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:41.950383 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/145e8805-e1b4-4f1f-80d0-86c7da396aee-istio-envoy\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rgsmg\" (UID: \"145e8805-e1b4-4f1f-80d0-86c7da396aee\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rgsmg" Nov 03 00:51:41.950647 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:41.950622 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/145e8805-e1b4-4f1f-80d0-86c7da396aee-istio-podinfo\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rgsmg\" (UID: \"145e8805-e1b4-4f1f-80d0-86c7da396aee\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rgsmg" Nov 03 00:51:41.955940 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:41.955916 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/145e8805-e1b4-4f1f-80d0-86c7da396aee-istio-token\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rgsmg\" (UID: \"145e8805-e1b4-4f1f-80d0-86c7da396aee\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rgsmg" Nov 03 00:51:41.956043 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:41.956029 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2chs\" (UniqueName: \"kubernetes.io/projected/145e8805-e1b4-4f1f-80d0-86c7da396aee-kube-api-access-n2chs\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rgsmg\" (UID: \"145e8805-e1b4-4f1f-80d0-86c7da396aee\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rgsmg" Nov 03 00:51:42.040470 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:42.040427 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rgsmg" Nov 03 00:51:42.322078 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:42.322051 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rgsmg"] Nov 03 00:51:42.324427 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:51:42.324397 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod145e8805_e1b4_4f1f_80d0_86c7da396aee.slice/crio-a525f410061b551b1160cd77bd493426ae16fccb6b5e9616b99aa5a29af838e0 WatchSource:0}: Error finding container a525f410061b551b1160cd77bd493426ae16fccb6b5e9616b99aa5a29af838e0: Status 404 returned error can't find the container with id a525f410061b551b1160cd77bd493426ae16fccb6b5e9616b99aa5a29af838e0 Nov 03 00:51:42.326780 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:42.326745 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:51:42.326892 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:42.326839 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:51:42.326892 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:42.326883 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:51:42.528310 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:42.528263 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rgsmg" event={"ID":"145e8805-e1b4-4f1f-80d0-86c7da396aee","Type":"ContainerStarted","Data":"27b6da461720fd54d135f5a8e49e7dd07c9761a700f3e88ac4150daa2116a785"} Nov 03 00:51:42.528478 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:42.528337 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rgsmg" event={"ID":"145e8805-e1b4-4f1f-80d0-86c7da396aee","Type":"ContainerStarted","Data":"a525f410061b551b1160cd77bd493426ae16fccb6b5e9616b99aa5a29af838e0"} Nov 03 00:51:42.557877 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:42.555481 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rgsmg" podStartSLOduration=1.5554604950000002 podStartE2EDuration="1.555460495s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:51:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:51:42.548133041 +0000 UTC m=+644.719136137" watchObservedRunningTime="2026-11-03 00:51:42.555460495 +0000 UTC m=+644.726463593" Nov 03 00:51:43.040834 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:43.040802 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rgsmg" Nov 03 00:51:43.042565 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:43.042539 4421 patch_prober.go:29] interesting pod/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rgsmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 03 00:51:43.042692 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:43.042587 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rgsmg" podUID="145e8805-e1b4-4f1f-80d0-86c7da396aee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 03 00:51:44.040878 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:44.040844 4421 patch_prober.go:29] interesting pod/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rgsmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 03 00:51:44.041262 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:44.040899 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rgsmg" podUID="145e8805-e1b4-4f1f-80d0-86c7da396aee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 03 00:51:44.670262 ip-10-0-133-217 kubenswrapper[4421]: E1103 00:51:44.670222 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 00:51:45.040891 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:45.040851 4421 patch_prober.go:29] interesting pod/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rgsmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 03 00:51:45.041454 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:45.040911 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rgsmg" podUID="145e8805-e1b4-4f1f-80d0-86c7da396aee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 03 00:51:46.041762 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:46.041728 4421 patch_prober.go:29] interesting pod/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rgsmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 03 00:51:46.042265 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:46.041811 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rgsmg" podUID="145e8805-e1b4-4f1f-80d0-86c7da396aee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 03 00:51:47.041514 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:47.041480 4421 patch_prober.go:29] interesting pod/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rgsmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 03 00:51:47.041741 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:47.041543 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rgsmg" podUID="145e8805-e1b4-4f1f-80d0-86c7da396aee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 03 00:51:48.045293 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:48.044352 4421 patch_prober.go:29] interesting pod/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rgsmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 03 00:51:48.045293 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:48.044423 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rgsmg" podUID="145e8805-e1b4-4f1f-80d0-86c7da396aee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 03 00:51:50.043729 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:50.043695 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rgsmg" Nov 03 00:51:50.044385 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:50.044313 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rgsmg" Nov 03 00:51:50.045102 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:50.045084 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rgsmg" Nov 03 00:51:56.024587 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:56.024551 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f24b753-kgtxk"] Nov 03 00:51:56.028948 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:56.028925 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f24b753-kgtxk" Nov 03 00:51:56.033579 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:56.033554 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:51:56.035469 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:56.034792 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 00:51:56.035469 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:56.034815 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6q4xp\"" Nov 03 00:51:56.035469 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:56.034804 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 00:51:56.038089 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:56.037939 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f24b753-kgtxk"] Nov 03 00:51:56.067740 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:56.067701 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc00e6aa-299e-4208-a143-7a55bb0f8e5b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3f24b753-kgtxk\" (UID: \"dc00e6aa-299e-4208-a143-7a55bb0f8e5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f24b753-kgtxk" Nov 03 00:51:56.067966 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:56.067904 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mn8c7\" (UniqueName: \"kubernetes.io/projected/dc00e6aa-299e-4208-a143-7a55bb0f8e5b-kube-api-access-mn8c7\") pod \"maas-api-revoke-sub-3f24b753-kgtxk\" (UID: \"dc00e6aa-299e-4208-a143-7a55bb0f8e5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f24b753-kgtxk" Nov 03 00:51:56.068048 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:56.067978 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc00e6aa-299e-4208-a143-7a55bb0f8e5b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3f24b753-kgtxk\" (UID: \"dc00e6aa-299e-4208-a143-7a55bb0f8e5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f24b753-kgtxk" Nov 03 00:51:56.169368 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:56.169317 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mn8c7\" (UniqueName: \"kubernetes.io/projected/dc00e6aa-299e-4208-a143-7a55bb0f8e5b-kube-api-access-mn8c7\") pod \"maas-api-revoke-sub-3f24b753-kgtxk\" (UID: \"dc00e6aa-299e-4208-a143-7a55bb0f8e5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f24b753-kgtxk" Nov 03 00:51:56.169558 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:56.169392 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc00e6aa-299e-4208-a143-7a55bb0f8e5b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3f24b753-kgtxk\" (UID: \"dc00e6aa-299e-4208-a143-7a55bb0f8e5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f24b753-kgtxk" Nov 03 00:51:56.169558 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:56.169532 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc00e6aa-299e-4208-a143-7a55bb0f8e5b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3f24b753-kgtxk\" (UID: \"dc00e6aa-299e-4208-a143-7a55bb0f8e5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f24b753-kgtxk" Nov 03 00:51:56.170266 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:56.170242 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc00e6aa-299e-4208-a143-7a55bb0f8e5b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3f24b753-kgtxk\" (UID: \"dc00e6aa-299e-4208-a143-7a55bb0f8e5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f24b753-kgtxk" Nov 03 00:51:56.171805 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:56.171779 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc00e6aa-299e-4208-a143-7a55bb0f8e5b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3f24b753-kgtxk\" (UID: \"dc00e6aa-299e-4208-a143-7a55bb0f8e5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f24b753-kgtxk" Nov 03 00:51:56.177261 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:56.177242 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mn8c7\" (UniqueName: \"kubernetes.io/projected/dc00e6aa-299e-4208-a143-7a55bb0f8e5b-kube-api-access-mn8c7\") pod \"maas-api-revoke-sub-3f24b753-kgtxk\" (UID: \"dc00e6aa-299e-4208-a143-7a55bb0f8e5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f24b753-kgtxk" Nov 03 00:51:56.345007 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:56.344918 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f24b753-kgtxk" Nov 03 00:51:56.621537 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:56.621490 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f24b753-kgtxk"] Nov 03 00:51:56.624745 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:51:56.624709 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc00e6aa_299e_4208_a143_7a55bb0f8e5b.slice/crio-12d99d12e4dcc9954728177413549aa6ccb7715218a517d1006a1c5dda8344b8 WatchSource:0}: Error finding container 12d99d12e4dcc9954728177413549aa6ccb7715218a517d1006a1c5dda8344b8: Status 404 returned error can't find the container with id 12d99d12e4dcc9954728177413549aa6ccb7715218a517d1006a1c5dda8344b8 Nov 03 00:51:56.632651 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:56.632624 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f24b753-kgtxk" event={"ID":"dc00e6aa-299e-4208-a143-7a55bb0f8e5b","Type":"ContainerStarted","Data":"12d99d12e4dcc9954728177413549aa6ccb7715218a517d1006a1c5dda8344b8"} Nov 03 00:51:57.639806 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:57.639774 4421 generic.go:356] "Generic (PLEG): container finished" podID="dc00e6aa-299e-4208-a143-7a55bb0f8e5b" containerID="98469d4892095fecbac19cf8a60b9f8684ce73fbb03a91921e4a2b8e3a6fad4a" exitCode=0 Nov 03 00:51:57.640183 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:57.639852 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f24b753-kgtxk" event={"ID":"dc00e6aa-299e-4208-a143-7a55bb0f8e5b","Type":"ContainerDied","Data":"98469d4892095fecbac19cf8a60b9f8684ce73fbb03a91921e4a2b8e3a6fad4a"} Nov 03 00:51:58.813413 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:58.813383 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f24b753-kgtxk" Nov 03 00:51:58.898651 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:58.898553 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc00e6aa-299e-4208-a143-7a55bb0f8e5b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc00e6aa-299e-4208-a143-7a55bb0f8e5b-maas-api-openshift-ca\") pod \"dc00e6aa-299e-4208-a143-7a55bb0f8e5b\" (UID: \"dc00e6aa-299e-4208-a143-7a55bb0f8e5b\") " Nov 03 00:51:58.898806 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:58.898738 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc00e6aa-299e-4208-a143-7a55bb0f8e5b-kube-api-access-mn8c7\" (UniqueName: \"kubernetes.io/projected/dc00e6aa-299e-4208-a143-7a55bb0f8e5b-kube-api-access-mn8c7\") pod \"dc00e6aa-299e-4208-a143-7a55bb0f8e5b\" (UID: \"dc00e6aa-299e-4208-a143-7a55bb0f8e5b\") " Nov 03 00:51:58.898806 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:58.898796 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc00e6aa-299e-4208-a143-7a55bb0f8e5b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc00e6aa-299e-4208-a143-7a55bb0f8e5b-maas-api-xks-ca\") pod \"dc00e6aa-299e-4208-a143-7a55bb0f8e5b\" (UID: \"dc00e6aa-299e-4208-a143-7a55bb0f8e5b\") " Nov 03 00:51:58.899000 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:58.898971 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc00e6aa-299e-4208-a143-7a55bb0f8e5b-maas-api-openshift-ca" pod "dc00e6aa-299e-4208-a143-7a55bb0f8e5b" (UID: "dc00e6aa-299e-4208-a143-7a55bb0f8e5b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:51:58.899245 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:58.899223 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc00e6aa-299e-4208-a143-7a55bb0f8e5b-maas-api-openshift-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:51:58.901366 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:58.901341 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc00e6aa-299e-4208-a143-7a55bb0f8e5b-kube-api-access-mn8c7" pod "dc00e6aa-299e-4208-a143-7a55bb0f8e5b" (UID: "dc00e6aa-299e-4208-a143-7a55bb0f8e5b"). InnerVolumeSpecName "kube-api-access-mn8c7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:51:58.901512 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:58.901436 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc00e6aa-299e-4208-a143-7a55bb0f8e5b-maas-api-xks-ca" pod "dc00e6aa-299e-4208-a143-7a55bb0f8e5b" (UID: "dc00e6aa-299e-4208-a143-7a55bb0f8e5b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:51:59.000054 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:59.000011 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mn8c7\" (UniqueName: \"kubernetes.io/projected/dc00e6aa-299e-4208-a143-7a55bb0f8e5b-kube-api-access-mn8c7\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:51:59.000054 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:59.000045 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc00e6aa-299e-4208-a143-7a55bb0f8e5b-maas-api-xks-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:51:59.654077 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:59.654046 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f24b753-kgtxk" Nov 03 00:51:59.654302 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:59.654079 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f24b753-kgtxk" event={"ID":"dc00e6aa-299e-4208-a143-7a55bb0f8e5b","Type":"ContainerDied","Data":"12d99d12e4dcc9954728177413549aa6ccb7715218a517d1006a1c5dda8344b8"} Nov 03 00:51:59.654302 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:51:59.654130 4421 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="12d99d12e4dcc9954728177413549aa6ccb7715218a517d1006a1c5dda8344b8" Nov 03 00:52:05.562293 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:05.560858 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48014db0-5n4s6"] Nov 03 00:52:05.562293 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:05.561672 4421 cpu_manager.go:414] "containerMap: removing container" podUID="dc00e6aa-299e-4208-a143-7a55bb0f8e5b" containerName="revoke-keys" Nov 03 00:52:05.562293 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:05.561695 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc00e6aa-299e-4208-a143-7a55bb0f8e5b" containerName="revoke-keys" Nov 03 00:52:05.562293 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:05.561858 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc00e6aa-299e-4208-a143-7a55bb0f8e5b" containerName="revoke-keys" Nov 03 00:52:05.569081 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:05.569056 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48014db0-5n4s6" Nov 03 00:52:05.576297 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:05.574238 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:52:05.576297 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:05.575302 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 00:52:05.576297 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:05.575401 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6q4xp\"" Nov 03 00:52:05.576297 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:05.575467 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 00:52:05.618299 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:05.615949 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48014db0-5n4s6"] Nov 03 00:52:05.689489 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:05.689451 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/078e68c2-59ea-4d68-86c8-11b4b9dfa99c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-48014db0-5n4s6\" (UID: \"078e68c2-59ea-4d68-86c8-11b4b9dfa99c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48014db0-5n4s6" Nov 03 00:52:05.689762 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:05.689744 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/078e68c2-59ea-4d68-86c8-11b4b9dfa99c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-48014db0-5n4s6\" (UID: \"078e68c2-59ea-4d68-86c8-11b4b9dfa99c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48014db0-5n4s6" Nov 03 00:52:05.689908 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:05.689893 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grr7m\" (UniqueName: \"kubernetes.io/projected/078e68c2-59ea-4d68-86c8-11b4b9dfa99c-kube-api-access-grr7m\") pod \"maas-api-revoke-sub-48014db0-5n4s6\" (UID: \"078e68c2-59ea-4d68-86c8-11b4b9dfa99c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48014db0-5n4s6" Nov 03 00:52:05.792859 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:05.792817 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/078e68c2-59ea-4d68-86c8-11b4b9dfa99c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-48014db0-5n4s6\" (UID: \"078e68c2-59ea-4d68-86c8-11b4b9dfa99c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48014db0-5n4s6" Nov 03 00:52:05.793955 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:05.793931 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/078e68c2-59ea-4d68-86c8-11b4b9dfa99c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-48014db0-5n4s6\" (UID: \"078e68c2-59ea-4d68-86c8-11b4b9dfa99c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48014db0-5n4s6" Nov 03 00:52:05.800375 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:05.800340 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grr7m\" (UniqueName: \"kubernetes.io/projected/078e68c2-59ea-4d68-86c8-11b4b9dfa99c-kube-api-access-grr7m\") pod \"maas-api-revoke-sub-48014db0-5n4s6\" (UID: \"078e68c2-59ea-4d68-86c8-11b4b9dfa99c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48014db0-5n4s6" Nov 03 00:52:05.800536 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:05.793840 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/078e68c2-59ea-4d68-86c8-11b4b9dfa99c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-48014db0-5n4s6\" (UID: \"078e68c2-59ea-4d68-86c8-11b4b9dfa99c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48014db0-5n4s6" Nov 03 00:52:05.800642 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:05.800526 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/078e68c2-59ea-4d68-86c8-11b4b9dfa99c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-48014db0-5n4s6\" (UID: \"078e68c2-59ea-4d68-86c8-11b4b9dfa99c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48014db0-5n4s6" Nov 03 00:52:05.808770 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:05.808738 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-grr7m\" (UniqueName: \"kubernetes.io/projected/078e68c2-59ea-4d68-86c8-11b4b9dfa99c-kube-api-access-grr7m\") pod \"maas-api-revoke-sub-48014db0-5n4s6\" (UID: \"078e68c2-59ea-4d68-86c8-11b4b9dfa99c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48014db0-5n4s6" Nov 03 00:52:05.909773 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:05.909674 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48014db0-5n4s6" Nov 03 00:52:06.204153 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:06.204128 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48014db0-5n4s6"] Nov 03 00:52:06.206381 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:52:06.206348 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod078e68c2_59ea_4d68_86c8_11b4b9dfa99c.slice/crio-0b91bd3f84eeeb2272dc69affed3f5dffb088d343ebddd308b29ef620ed15c7e WatchSource:0}: Error finding container 0b91bd3f84eeeb2272dc69affed3f5dffb088d343ebddd308b29ef620ed15c7e: Status 404 returned error can't find the container with id 0b91bd3f84eeeb2272dc69affed3f5dffb088d343ebddd308b29ef620ed15c7e Nov 03 00:52:06.747753 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:06.747719 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48014db0-5n4s6" event={"ID":"078e68c2-59ea-4d68-86c8-11b4b9dfa99c","Type":"ContainerStarted","Data":"4db675d3d04c7be1ae90eaeeb7a9ca873355e2c34d2d13c6e3ec80911b9dd921"} Nov 03 00:52:06.748170 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:06.747786 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48014db0-5n4s6" event={"ID":"078e68c2-59ea-4d68-86c8-11b4b9dfa99c","Type":"ContainerStarted","Data":"0b91bd3f84eeeb2272dc69affed3f5dffb088d343ebddd308b29ef620ed15c7e"} Nov 03 00:52:06.767493 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:06.767395 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48014db0-5n4s6" podStartSLOduration=1.767380266 podStartE2EDuration="1.767380266s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:52:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:52:06.764835096 +0000 UTC m=+668.935838193" watchObservedRunningTime="2026-11-03 00:52:06.767380266 +0000 UTC m=+668.938383362" Nov 03 00:52:07.755404 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:07.755369 4421 generic.go:356] "Generic (PLEG): container finished" podID="078e68c2-59ea-4d68-86c8-11b4b9dfa99c" containerID="4db675d3d04c7be1ae90eaeeb7a9ca873355e2c34d2d13c6e3ec80911b9dd921" exitCode=0 Nov 03 00:52:07.755787 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:07.755438 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48014db0-5n4s6" event={"ID":"078e68c2-59ea-4d68-86c8-11b4b9dfa99c","Type":"ContainerDied","Data":"4db675d3d04c7be1ae90eaeeb7a9ca873355e2c34d2d13c6e3ec80911b9dd921"} Nov 03 00:52:08.311517 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:08.311482 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-h9mjc"] Nov 03 00:52:08.315503 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:08.315480 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-h9mjc" Nov 03 00:52:08.323483 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:08.323458 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-88bbf7-openshift-default-dockercfg-h6n25\"" Nov 03 00:52:08.326684 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:08.326650 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-h9mjc"] Nov 03 00:52:08.441012 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:08.440972 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd5f7acb-d880-4f98-8b64-8501641747b6-istio-envoy\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-h9mjc\" (UID: \"dd5f7acb-d880-4f98-8b64-8501641747b6\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-h9mjc" Nov 03 00:52:08.441190 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:08.441032 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd5f7acb-d880-4f98-8b64-8501641747b6-istio-token\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-h9mjc\" (UID: \"dd5f7acb-d880-4f98-8b64-8501641747b6\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-h9mjc" Nov 03 00:52:08.441190 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:08.441115 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd5f7acb-d880-4f98-8b64-8501641747b6-workload-certs\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-h9mjc\" (UID: \"dd5f7acb-d880-4f98-8b64-8501641747b6\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-h9mjc" Nov 03 00:52:08.441190 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:08.441155 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dj7rf\" (UniqueName: \"kubernetes.io/projected/dd5f7acb-d880-4f98-8b64-8501641747b6-kube-api-access-dj7rf\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-h9mjc\" (UID: \"dd5f7acb-d880-4f98-8b64-8501641747b6\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-h9mjc" Nov 03 00:52:08.441376 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:08.441324 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd5f7acb-d880-4f98-8b64-8501641747b6-istiod-ca-cert\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-h9mjc\" (UID: \"dd5f7acb-d880-4f98-8b64-8501641747b6\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-h9mjc" Nov 03 00:52:08.441376 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:08.441358 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd5f7acb-d880-4f98-8b64-8501641747b6-workload-socket\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-h9mjc\" (UID: \"dd5f7acb-d880-4f98-8b64-8501641747b6\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-h9mjc" Nov 03 00:52:08.441477 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:08.441375 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd5f7acb-d880-4f98-8b64-8501641747b6-istio-data\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-h9mjc\" (UID: \"dd5f7acb-d880-4f98-8b64-8501641747b6\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-h9mjc" Nov 03 00:52:08.441521 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:08.441499 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd5f7acb-d880-4f98-8b64-8501641747b6-istio-podinfo\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-h9mjc\" (UID: \"dd5f7acb-d880-4f98-8b64-8501641747b6\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-h9mjc" Nov 03 00:52:08.441572 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:08.441548 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd5f7acb-d880-4f98-8b64-8501641747b6-credential-socket\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-h9mjc\" (UID: \"dd5f7acb-d880-4f98-8b64-8501641747b6\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-h9mjc" Nov 03 00:52:08.542855 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:08.542821 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dj7rf\" (UniqueName: \"kubernetes.io/projected/dd5f7acb-d880-4f98-8b64-8501641747b6-kube-api-access-dj7rf\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-h9mjc\" (UID: \"dd5f7acb-d880-4f98-8b64-8501641747b6\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-h9mjc" Nov 03 00:52:08.543039 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:08.542880 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd5f7acb-d880-4f98-8b64-8501641747b6-istiod-ca-cert\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-h9mjc\" (UID: \"dd5f7acb-d880-4f98-8b64-8501641747b6\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-h9mjc" Nov 03 00:52:08.543039 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:08.542910 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd5f7acb-d880-4f98-8b64-8501641747b6-workload-socket\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-h9mjc\" (UID: \"dd5f7acb-d880-4f98-8b64-8501641747b6\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-h9mjc" Nov 03 00:52:08.543039 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:08.542925 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd5f7acb-d880-4f98-8b64-8501641747b6-istio-data\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-h9mjc\" (UID: \"dd5f7acb-d880-4f98-8b64-8501641747b6\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-h9mjc" Nov 03 00:52:08.543039 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:08.542982 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd5f7acb-d880-4f98-8b64-8501641747b6-istio-podinfo\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-h9mjc\" (UID: \"dd5f7acb-d880-4f98-8b64-8501641747b6\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-h9mjc" Nov 03 00:52:08.543265 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:08.543064 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd5f7acb-d880-4f98-8b64-8501641747b6-credential-socket\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-h9mjc\" (UID: \"dd5f7acb-d880-4f98-8b64-8501641747b6\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-h9mjc" Nov 03 00:52:08.543265 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:08.543148 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd5f7acb-d880-4f98-8b64-8501641747b6-istio-envoy\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-h9mjc\" (UID: \"dd5f7acb-d880-4f98-8b64-8501641747b6\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-h9mjc" Nov 03 00:52:08.543265 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:08.543219 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd5f7acb-d880-4f98-8b64-8501641747b6-istio-token\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-h9mjc\" (UID: \"dd5f7acb-d880-4f98-8b64-8501641747b6\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-h9mjc" Nov 03 00:52:08.543446 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:08.543254 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd5f7acb-d880-4f98-8b64-8501641747b6-workload-certs\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-h9mjc\" (UID: \"dd5f7acb-d880-4f98-8b64-8501641747b6\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-h9mjc" Nov 03 00:52:08.543502 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:08.543461 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd5f7acb-d880-4f98-8b64-8501641747b6-istio-data\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-h9mjc\" (UID: \"dd5f7acb-d880-4f98-8b64-8501641747b6\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-h9mjc" Nov 03 00:52:08.543556 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:08.543503 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd5f7acb-d880-4f98-8b64-8501641747b6-workload-socket\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-h9mjc\" (UID: \"dd5f7acb-d880-4f98-8b64-8501641747b6\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-h9mjc" Nov 03 00:52:08.543688 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:08.543666 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd5f7acb-d880-4f98-8b64-8501641747b6-credential-socket\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-h9mjc\" (UID: \"dd5f7acb-d880-4f98-8b64-8501641747b6\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-h9mjc" Nov 03 00:52:08.543751 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:08.543733 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd5f7acb-d880-4f98-8b64-8501641747b6-workload-certs\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-h9mjc\" (UID: \"dd5f7acb-d880-4f98-8b64-8501641747b6\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-h9mjc" Nov 03 00:52:08.543910 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:08.543893 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd5f7acb-d880-4f98-8b64-8501641747b6-istiod-ca-cert\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-h9mjc\" (UID: \"dd5f7acb-d880-4f98-8b64-8501641747b6\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-h9mjc" Nov 03 00:52:08.545604 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:08.545577 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd5f7acb-d880-4f98-8b64-8501641747b6-istio-envoy\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-h9mjc\" (UID: \"dd5f7acb-d880-4f98-8b64-8501641747b6\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-h9mjc" Nov 03 00:52:08.545703 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:08.545683 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd5f7acb-d880-4f98-8b64-8501641747b6-istio-podinfo\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-h9mjc\" (UID: \"dd5f7acb-d880-4f98-8b64-8501641747b6\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-h9mjc" Nov 03 00:52:08.551024 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:08.550999 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dj7rf\" (UniqueName: \"kubernetes.io/projected/dd5f7acb-d880-4f98-8b64-8501641747b6-kube-api-access-dj7rf\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-h9mjc\" (UID: \"dd5f7acb-d880-4f98-8b64-8501641747b6\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-h9mjc" Nov 03 00:52:08.551423 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:08.551407 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd5f7acb-d880-4f98-8b64-8501641747b6-istio-token\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-h9mjc\" (UID: \"dd5f7acb-d880-4f98-8b64-8501641747b6\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-h9mjc" Nov 03 00:52:08.630464 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:08.630384 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-h9mjc" Nov 03 00:52:08.916432 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:08.916405 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48014db0-5n4s6" Nov 03 00:52:08.919961 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:08.919929 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-h9mjc"] Nov 03 00:52:08.920321 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:52:08.920297 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd5f7acb_d880_4f98_8b64_8501641747b6.slice/crio-7863271c03fb1a86f972bb25580e780b8dca28d98cb0e4436ffaa60cf1dbbdce WatchSource:0}: Error finding container 7863271c03fb1a86f972bb25580e780b8dca28d98cb0e4436ffaa60cf1dbbdce: Status 404 returned error can't find the container with id 7863271c03fb1a86f972bb25580e780b8dca28d98cb0e4436ffaa60cf1dbbdce Nov 03 00:52:08.922852 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:08.922815 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:52:08.922931 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:08.922900 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:52:08.922970 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:08.922945 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:52:08.948416 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:08.948381 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/078e68c2-59ea-4d68-86c8-11b4b9dfa99c-kube-api-access-grr7m\" (UniqueName: \"kubernetes.io/projected/078e68c2-59ea-4d68-86c8-11b4b9dfa99c-kube-api-access-grr7m\") pod \"078e68c2-59ea-4d68-86c8-11b4b9dfa99c\" (UID: \"078e68c2-59ea-4d68-86c8-11b4b9dfa99c\") " Nov 03 00:52:08.948555 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:08.948456 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/078e68c2-59ea-4d68-86c8-11b4b9dfa99c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/078e68c2-59ea-4d68-86c8-11b4b9dfa99c-maas-api-openshift-ca\") pod \"078e68c2-59ea-4d68-86c8-11b4b9dfa99c\" (UID: \"078e68c2-59ea-4d68-86c8-11b4b9dfa99c\") " Nov 03 00:52:08.948555 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:08.948493 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/078e68c2-59ea-4d68-86c8-11b4b9dfa99c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/078e68c2-59ea-4d68-86c8-11b4b9dfa99c-maas-api-xks-ca\") pod \"078e68c2-59ea-4d68-86c8-11b4b9dfa99c\" (UID: \"078e68c2-59ea-4d68-86c8-11b4b9dfa99c\") " Nov 03 00:52:08.948876 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:08.948846 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/078e68c2-59ea-4d68-86c8-11b4b9dfa99c-maas-api-openshift-ca" pod "078e68c2-59ea-4d68-86c8-11b4b9dfa99c" (UID: "078e68c2-59ea-4d68-86c8-11b4b9dfa99c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:52:08.951074 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:08.951029 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/078e68c2-59ea-4d68-86c8-11b4b9dfa99c-maas-api-xks-ca" pod "078e68c2-59ea-4d68-86c8-11b4b9dfa99c" (UID: "078e68c2-59ea-4d68-86c8-11b4b9dfa99c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:52:08.951074 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:08.951053 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/078e68c2-59ea-4d68-86c8-11b4b9dfa99c-kube-api-access-grr7m" pod "078e68c2-59ea-4d68-86c8-11b4b9dfa99c" (UID: "078e68c2-59ea-4d68-86c8-11b4b9dfa99c"). InnerVolumeSpecName "kube-api-access-grr7m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:52:08.961420 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:08.961389 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09592f3f-rk29m"] Nov 03 00:52:08.962070 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:08.962039 4421 cpu_manager.go:414] "containerMap: removing container" podUID="078e68c2-59ea-4d68-86c8-11b4b9dfa99c" containerName="revoke-keys" Nov 03 00:52:08.962070 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:08.962061 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="078e68c2-59ea-4d68-86c8-11b4b9dfa99c" containerName="revoke-keys" Nov 03 00:52:08.962242 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:08.962210 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="078e68c2-59ea-4d68-86c8-11b4b9dfa99c" containerName="revoke-keys" Nov 03 00:52:08.967378 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:08.967357 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09592f3f-rk29m" Nov 03 00:52:08.975045 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:08.975022 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09592f3f-rk29m"] Nov 03 00:52:09.050009 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:09.049966 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8387faeb-4fb0-443f-a020-6fef897a16da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-09592f3f-rk29m\" (UID: \"8387faeb-4fb0-443f-a020-6fef897a16da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09592f3f-rk29m" Nov 03 00:52:09.050224 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:09.050027 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8387faeb-4fb0-443f-a020-6fef897a16da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-09592f3f-rk29m\" (UID: \"8387faeb-4fb0-443f-a020-6fef897a16da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09592f3f-rk29m" Nov 03 00:52:09.050224 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:09.050079 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9rmd\" (UniqueName: \"kubernetes.io/projected/8387faeb-4fb0-443f-a020-6fef897a16da-kube-api-access-q9rmd\") pod \"maas-api-revoke-sub-09592f3f-rk29m\" (UID: \"8387faeb-4fb0-443f-a020-6fef897a16da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09592f3f-rk29m" Nov 03 00:52:09.050505 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:09.050483 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-grr7m\" (UniqueName: \"kubernetes.io/projected/078e68c2-59ea-4d68-86c8-11b4b9dfa99c-kube-api-access-grr7m\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:52:09.050574 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:09.050506 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/078e68c2-59ea-4d68-86c8-11b4b9dfa99c-maas-api-openshift-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:52:09.050574 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:09.050516 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/078e68c2-59ea-4d68-86c8-11b4b9dfa99c-maas-api-xks-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:52:09.152214 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:09.152119 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8387faeb-4fb0-443f-a020-6fef897a16da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-09592f3f-rk29m\" (UID: \"8387faeb-4fb0-443f-a020-6fef897a16da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09592f3f-rk29m" Nov 03 00:52:09.152214 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:09.152204 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8387faeb-4fb0-443f-a020-6fef897a16da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-09592f3f-rk29m\" (UID: \"8387faeb-4fb0-443f-a020-6fef897a16da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09592f3f-rk29m" Nov 03 00:52:09.152438 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:09.152247 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9rmd\" (UniqueName: \"kubernetes.io/projected/8387faeb-4fb0-443f-a020-6fef897a16da-kube-api-access-q9rmd\") pod \"maas-api-revoke-sub-09592f3f-rk29m\" (UID: \"8387faeb-4fb0-443f-a020-6fef897a16da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09592f3f-rk29m" Nov 03 00:52:09.152956 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:09.152926 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8387faeb-4fb0-443f-a020-6fef897a16da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-09592f3f-rk29m\" (UID: \"8387faeb-4fb0-443f-a020-6fef897a16da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09592f3f-rk29m" Nov 03 00:52:09.154908 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:09.154880 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8387faeb-4fb0-443f-a020-6fef897a16da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-09592f3f-rk29m\" (UID: \"8387faeb-4fb0-443f-a020-6fef897a16da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09592f3f-rk29m" Nov 03 00:52:09.161891 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:09.161858 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9rmd\" (UniqueName: \"kubernetes.io/projected/8387faeb-4fb0-443f-a020-6fef897a16da-kube-api-access-q9rmd\") pod \"maas-api-revoke-sub-09592f3f-rk29m\" (UID: \"8387faeb-4fb0-443f-a020-6fef897a16da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09592f3f-rk29m" Nov 03 00:52:09.288891 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:09.288853 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09592f3f-rk29m" Nov 03 00:52:09.581607 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:09.581578 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09592f3f-rk29m"] Nov 03 00:52:09.582848 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:52:09.582817 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8387faeb_4fb0_443f_a020_6fef897a16da.slice/crio-c2306c6f57c08c5acc721743c610c6e9e79e14d22b5d9bec033372f90c971bb5 WatchSource:0}: Error finding container c2306c6f57c08c5acc721743c610c6e9e79e14d22b5d9bec033372f90c971bb5: Status 404 returned error can't find the container with id c2306c6f57c08c5acc721743c610c6e9e79e14d22b5d9bec033372f90c971bb5 Nov 03 00:52:09.776458 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:09.776417 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-h9mjc" event={"ID":"dd5f7acb-d880-4f98-8b64-8501641747b6","Type":"ContainerStarted","Data":"72ec5e73ad4ad6c8646fb93943166bf654cf890599ac54ec59e7d4e072ccfd7c"} Nov 03 00:52:09.776656 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:09.776484 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-h9mjc" event={"ID":"dd5f7acb-d880-4f98-8b64-8501641747b6","Type":"ContainerStarted","Data":"7863271c03fb1a86f972bb25580e780b8dca28d98cb0e4436ffaa60cf1dbbdce"} Nov 03 00:52:09.777716 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:09.777688 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09592f3f-rk29m" event={"ID":"8387faeb-4fb0-443f-a020-6fef897a16da","Type":"ContainerStarted","Data":"c2306c6f57c08c5acc721743c610c6e9e79e14d22b5d9bec033372f90c971bb5"} Nov 03 00:52:09.778952 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:09.778929 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48014db0-5n4s6" event={"ID":"078e68c2-59ea-4d68-86c8-11b4b9dfa99c","Type":"ContainerDied","Data":"0b91bd3f84eeeb2272dc69affed3f5dffb088d343ebddd308b29ef620ed15c7e"} Nov 03 00:52:09.779079 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:09.778971 4421 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0b91bd3f84eeeb2272dc69affed3f5dffb088d343ebddd308b29ef620ed15c7e" Nov 03 00:52:09.779079 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:09.778935 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48014db0-5n4s6" Nov 03 00:52:09.796254 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:09.796205 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-h9mjc" podStartSLOduration=1.7961912610000002 podStartE2EDuration="1.796191261s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:52:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:52:09.794334755 +0000 UTC m=+671.965337853" watchObservedRunningTime="2026-11-03 00:52:09.796191261 +0000 UTC m=+671.967194358" Nov 03 00:52:10.633299 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:10.633191 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-h9mjc" Nov 03 00:52:10.633835 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:10.633384 4421 patch_prober.go:29] interesting pod/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-h9mjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 03 00:52:10.633835 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:10.633430 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-h9mjc" podUID="dd5f7acb-d880-4f98-8b64-8501641747b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 03 00:52:10.683006 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:10.682963 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b61b4cee-dl5tp"] Nov 03 00:52:10.686934 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:10.686916 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b61b4cee-dl5tp" Nov 03 00:52:10.693976 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:10.693943 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b61b4cee-dl5tp"] Nov 03 00:52:10.772238 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:10.772200 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/697feaa8-37f3-4292-8fef-2006acc658ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b61b4cee-dl5tp\" (UID: \"697feaa8-37f3-4292-8fef-2006acc658ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b61b4cee-dl5tp" Nov 03 00:52:10.772435 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:10.772392 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/697feaa8-37f3-4292-8fef-2006acc658ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b61b4cee-dl5tp\" (UID: \"697feaa8-37f3-4292-8fef-2006acc658ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b61b4cee-dl5tp" Nov 03 00:52:10.772435 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:10.772426 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whxxt\" (UniqueName: \"kubernetes.io/projected/697feaa8-37f3-4292-8fef-2006acc658ff-kube-api-access-whxxt\") pod \"maas-api-revoke-sub-b61b4cee-dl5tp\" (UID: \"697feaa8-37f3-4292-8fef-2006acc658ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b61b4cee-dl5tp" Nov 03 00:52:10.797161 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:10.797125 4421 generic.go:356] "Generic (PLEG): container finished" podID="8387faeb-4fb0-443f-a020-6fef897a16da" containerID="467abd6290257c709088123048e4b3832847ee5e4f678e5cdb814ad21e2b3654" exitCode=0 Nov 03 00:52:10.797368 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:10.797211 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09592f3f-rk29m" event={"ID":"8387faeb-4fb0-443f-a020-6fef897a16da","Type":"ContainerDied","Data":"467abd6290257c709088123048e4b3832847ee5e4f678e5cdb814ad21e2b3654"} Nov 03 00:52:10.873995 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:10.873956 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/697feaa8-37f3-4292-8fef-2006acc658ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b61b4cee-dl5tp\" (UID: \"697feaa8-37f3-4292-8fef-2006acc658ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b61b4cee-dl5tp" Nov 03 00:52:10.873995 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:10.873998 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whxxt\" (UniqueName: \"kubernetes.io/projected/697feaa8-37f3-4292-8fef-2006acc658ff-kube-api-access-whxxt\") pod \"maas-api-revoke-sub-b61b4cee-dl5tp\" (UID: \"697feaa8-37f3-4292-8fef-2006acc658ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b61b4cee-dl5tp" Nov 03 00:52:10.874250 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:10.874191 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/697feaa8-37f3-4292-8fef-2006acc658ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b61b4cee-dl5tp\" (UID: \"697feaa8-37f3-4292-8fef-2006acc658ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b61b4cee-dl5tp" Nov 03 00:52:10.874848 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:10.874825 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/697feaa8-37f3-4292-8fef-2006acc658ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b61b4cee-dl5tp\" (UID: \"697feaa8-37f3-4292-8fef-2006acc658ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b61b4cee-dl5tp" Nov 03 00:52:10.876382 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:10.876363 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/697feaa8-37f3-4292-8fef-2006acc658ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b61b4cee-dl5tp\" (UID: \"697feaa8-37f3-4292-8fef-2006acc658ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b61b4cee-dl5tp" Nov 03 00:52:10.882348 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:10.882323 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whxxt\" (UniqueName: \"kubernetes.io/projected/697feaa8-37f3-4292-8fef-2006acc658ff-kube-api-access-whxxt\") pod \"maas-api-revoke-sub-b61b4cee-dl5tp\" (UID: \"697feaa8-37f3-4292-8fef-2006acc658ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b61b4cee-dl5tp" Nov 03 00:52:11.036344 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:11.036305 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b61b4cee-dl5tp" Nov 03 00:52:11.332369 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:11.332235 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b61b4cee-dl5tp"] Nov 03 00:52:11.335832 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:52:11.335801 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod697feaa8_37f3_4292_8fef_2006acc658ff.slice/crio-edb9966f522eab136b54df1c98495f0af503c2fc668d34cfc279e50887641716 WatchSource:0}: Error finding container edb9966f522eab136b54df1c98495f0af503c2fc668d34cfc279e50887641716: Status 404 returned error can't find the container with id edb9966f522eab136b54df1c98495f0af503c2fc668d34cfc279e50887641716 Nov 03 00:52:11.632417 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:11.632327 4421 patch_prober.go:29] interesting pod/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-h9mjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 03 00:52:11.632417 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:11.632389 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-h9mjc" podUID="dd5f7acb-d880-4f98-8b64-8501641747b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 03 00:52:11.804265 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:11.804222 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b61b4cee-dl5tp" event={"ID":"697feaa8-37f3-4292-8fef-2006acc658ff","Type":"ContainerStarted","Data":"043524b93ace404e06e4d3723e9cf4c3e1c601c2cdda974ebd8d0796136b2277"} Nov 03 00:52:11.804661 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:11.804323 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b61b4cee-dl5tp" event={"ID":"697feaa8-37f3-4292-8fef-2006acc658ff","Type":"ContainerStarted","Data":"edb9966f522eab136b54df1c98495f0af503c2fc668d34cfc279e50887641716"} Nov 03 00:52:11.840428 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:11.840377 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b61b4cee-dl5tp" podStartSLOduration=1.8403626339999999 podStartE2EDuration="1.840362634s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:52:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:52:11.840190222 +0000 UTC m=+674.011193322" watchObservedRunningTime="2026-11-03 00:52:11.840362634 +0000 UTC m=+674.011365726" Nov 03 00:52:11.996464 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:11.996436 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09592f3f-rk29m" Nov 03 00:52:12.088382 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:12.088350 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8387faeb-4fb0-443f-a020-6fef897a16da-kube-api-access-q9rmd\" (UniqueName: \"kubernetes.io/projected/8387faeb-4fb0-443f-a020-6fef897a16da-kube-api-access-q9rmd\") pod \"8387faeb-4fb0-443f-a020-6fef897a16da\" (UID: \"8387faeb-4fb0-443f-a020-6fef897a16da\") " Nov 03 00:52:12.088574 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:12.088405 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8387faeb-4fb0-443f-a020-6fef897a16da-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8387faeb-4fb0-443f-a020-6fef897a16da-maas-api-xks-ca\") pod \"8387faeb-4fb0-443f-a020-6fef897a16da\" (UID: \"8387faeb-4fb0-443f-a020-6fef897a16da\") " Nov 03 00:52:12.088574 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:12.088458 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8387faeb-4fb0-443f-a020-6fef897a16da-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8387faeb-4fb0-443f-a020-6fef897a16da-maas-api-openshift-ca\") pod \"8387faeb-4fb0-443f-a020-6fef897a16da\" (UID: \"8387faeb-4fb0-443f-a020-6fef897a16da\") " Nov 03 00:52:12.089183 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:12.088845 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8387faeb-4fb0-443f-a020-6fef897a16da-maas-api-openshift-ca" pod "8387faeb-4fb0-443f-a020-6fef897a16da" (UID: "8387faeb-4fb0-443f-a020-6fef897a16da"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:52:12.089183 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:12.088994 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8387faeb-4fb0-443f-a020-6fef897a16da-maas-api-openshift-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:52:12.090638 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:12.090609 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8387faeb-4fb0-443f-a020-6fef897a16da-maas-api-xks-ca" pod "8387faeb-4fb0-443f-a020-6fef897a16da" (UID: "8387faeb-4fb0-443f-a020-6fef897a16da"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:52:12.090903 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:12.090875 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8387faeb-4fb0-443f-a020-6fef897a16da-kube-api-access-q9rmd" pod "8387faeb-4fb0-443f-a020-6fef897a16da" (UID: "8387faeb-4fb0-443f-a020-6fef897a16da"). InnerVolumeSpecName "kube-api-access-q9rmd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:52:12.190046 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:12.189959 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9rmd\" (UniqueName: \"kubernetes.io/projected/8387faeb-4fb0-443f-a020-6fef897a16da-kube-api-access-q9rmd\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:52:12.190046 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:12.189993 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8387faeb-4fb0-443f-a020-6fef897a16da-maas-api-xks-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:52:12.631798 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:12.631762 4421 patch_prober.go:29] interesting pod/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-h9mjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 03 00:52:12.631967 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:12.631822 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-h9mjc" podUID="dd5f7acb-d880-4f98-8b64-8501641747b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 03 00:52:12.846344 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:12.846298 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09592f3f-rk29m" event={"ID":"8387faeb-4fb0-443f-a020-6fef897a16da","Type":"ContainerDied","Data":"c2306c6f57c08c5acc721743c610c6e9e79e14d22b5d9bec033372f90c971bb5"} Nov 03 00:52:12.846344 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:12.846326 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09592f3f-rk29m" Nov 03 00:52:12.846880 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:12.846364 4421 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c2306c6f57c08c5acc721743c610c6e9e79e14d22b5d9bec033372f90c971bb5" Nov 03 00:52:12.848365 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:12.848331 4421 generic.go:356] "Generic (PLEG): container finished" podID="697feaa8-37f3-4292-8fef-2006acc658ff" containerID="043524b93ace404e06e4d3723e9cf4c3e1c601c2cdda974ebd8d0796136b2277" exitCode=0 Nov 03 00:52:12.848504 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:12.848380 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b61b4cee-dl5tp" event={"ID":"697feaa8-37f3-4292-8fef-2006acc658ff","Type":"ContainerDied","Data":"043524b93ace404e06e4d3723e9cf4c3e1c601c2cdda974ebd8d0796136b2277"} Nov 03 00:52:13.631809 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:13.631760 4421 patch_prober.go:29] interesting pod/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-h9mjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 03 00:52:13.632013 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:13.631826 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-h9mjc" podUID="dd5f7acb-d880-4f98-8b64-8501641747b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 03 00:52:14.089697 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:14.089656 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b61b4cee-dl5tp" Nov 03 00:52:14.245048 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:14.242236 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/697feaa8-37f3-4292-8fef-2006acc658ff-kube-api-access-whxxt\" (UniqueName: \"kubernetes.io/projected/697feaa8-37f3-4292-8fef-2006acc658ff-kube-api-access-whxxt\") pod \"697feaa8-37f3-4292-8fef-2006acc658ff\" (UID: \"697feaa8-37f3-4292-8fef-2006acc658ff\") " Nov 03 00:52:14.245048 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:14.242520 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/697feaa8-37f3-4292-8fef-2006acc658ff-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/697feaa8-37f3-4292-8fef-2006acc658ff-maas-api-xks-ca\") pod \"697feaa8-37f3-4292-8fef-2006acc658ff\" (UID: \"697feaa8-37f3-4292-8fef-2006acc658ff\") " Nov 03 00:52:14.245048 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:14.242709 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/697feaa8-37f3-4292-8fef-2006acc658ff-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/697feaa8-37f3-4292-8fef-2006acc658ff-maas-api-openshift-ca\") pod \"697feaa8-37f3-4292-8fef-2006acc658ff\" (UID: \"697feaa8-37f3-4292-8fef-2006acc658ff\") " Nov 03 00:52:14.245048 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:14.243354 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/697feaa8-37f3-4292-8fef-2006acc658ff-maas-api-openshift-ca" pod "697feaa8-37f3-4292-8fef-2006acc658ff" (UID: "697feaa8-37f3-4292-8fef-2006acc658ff"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:52:14.252468 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:14.245817 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/697feaa8-37f3-4292-8fef-2006acc658ff-kube-api-access-whxxt" pod "697feaa8-37f3-4292-8fef-2006acc658ff" (UID: "697feaa8-37f3-4292-8fef-2006acc658ff"). InnerVolumeSpecName "kube-api-access-whxxt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:52:14.254435 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:14.254401 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/697feaa8-37f3-4292-8fef-2006acc658ff-maas-api-xks-ca" pod "697feaa8-37f3-4292-8fef-2006acc658ff" (UID: "697feaa8-37f3-4292-8fef-2006acc658ff"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:52:14.344721 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:14.344685 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/697feaa8-37f3-4292-8fef-2006acc658ff-maas-api-openshift-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:52:14.344721 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:14.344720 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whxxt\" (UniqueName: \"kubernetes.io/projected/697feaa8-37f3-4292-8fef-2006acc658ff-kube-api-access-whxxt\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:52:14.344721 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:14.344731 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/697feaa8-37f3-4292-8fef-2006acc658ff-maas-api-xks-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:52:14.631754 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:14.631712 4421 patch_prober.go:29] interesting pod/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-h9mjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 03 00:52:14.631964 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:14.631813 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-h9mjc" podUID="dd5f7acb-d880-4f98-8b64-8501641747b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 03 00:52:14.678700 ip-10-0-133-217 kubenswrapper[4421]: E1103 00:52:14.678643 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 00:52:14.862714 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:14.862682 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b61b4cee-dl5tp" Nov 03 00:52:14.862714 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:14.862707 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b61b4cee-dl5tp" event={"ID":"697feaa8-37f3-4292-8fef-2006acc658ff","Type":"ContainerDied","Data":"edb9966f522eab136b54df1c98495f0af503c2fc668d34cfc279e50887641716"} Nov 03 00:52:14.862929 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:14.862784 4421 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="edb9966f522eab136b54df1c98495f0af503c2fc668d34cfc279e50887641716" Nov 03 00:52:15.631192 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:15.631138 4421 patch_prober.go:29] interesting pod/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-h9mjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 03 00:52:15.631913 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:15.631862 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-h9mjc" podUID="dd5f7acb-d880-4f98-8b64-8501641747b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 03 00:52:16.635701 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:16.635670 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-h9mjc" Nov 03 00:52:16.636109 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:16.636037 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-h9mjc" Nov 03 00:52:16.636737 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:16.636718 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-h9mjc" Nov 03 00:52:17.204824 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:17.204794 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xmmdd"] Nov 03 00:52:17.205318 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:17.205302 4421 cpu_manager.go:414] "containerMap: removing container" podUID="697feaa8-37f3-4292-8fef-2006acc658ff" containerName="revoke-keys" Nov 03 00:52:17.205360 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:17.205320 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="697feaa8-37f3-4292-8fef-2006acc658ff" containerName="revoke-keys" Nov 03 00:52:17.205360 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:17.205338 4421 cpu_manager.go:414] "containerMap: removing container" podUID="8387faeb-4fb0-443f-a020-6fef897a16da" containerName="revoke-keys" Nov 03 00:52:17.205360 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:17.205343 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8387faeb-4fb0-443f-a020-6fef897a16da" containerName="revoke-keys" Nov 03 00:52:17.205455 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:17.205443 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="697feaa8-37f3-4292-8fef-2006acc658ff" containerName="revoke-keys" Nov 03 00:52:17.205455 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:17.205452 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="8387faeb-4fb0-443f-a020-6fef897a16da" containerName="revoke-keys" Nov 03 00:52:17.208874 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:17.208855 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xmmdd" Nov 03 00:52:17.222579 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:17.221225 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-2f9a3a-openshift-default-dockercfg-gdpt5\"" Nov 03 00:52:17.224701 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:17.224674 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xmmdd"] Nov 03 00:52:17.387426 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:17.387354 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c931c147-75a3-4a38-99ff-26d0aa98a38a-workload-certs\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xmmdd\" (UID: \"c931c147-75a3-4a38-99ff-26d0aa98a38a\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xmmdd" Nov 03 00:52:17.387426 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:17.387412 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c931c147-75a3-4a38-99ff-26d0aa98a38a-istiod-ca-cert\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xmmdd\" (UID: \"c931c147-75a3-4a38-99ff-26d0aa98a38a\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xmmdd" Nov 03 00:52:17.387712 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:17.387440 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x94sd\" (UniqueName: \"kubernetes.io/projected/c931c147-75a3-4a38-99ff-26d0aa98a38a-kube-api-access-x94sd\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xmmdd\" (UID: \"c931c147-75a3-4a38-99ff-26d0aa98a38a\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xmmdd" Nov 03 00:52:17.387712 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:17.387557 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c931c147-75a3-4a38-99ff-26d0aa98a38a-istio-token\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xmmdd\" (UID: \"c931c147-75a3-4a38-99ff-26d0aa98a38a\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xmmdd" Nov 03 00:52:17.387827 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:17.387753 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c931c147-75a3-4a38-99ff-26d0aa98a38a-workload-socket\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xmmdd\" (UID: \"c931c147-75a3-4a38-99ff-26d0aa98a38a\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xmmdd" Nov 03 00:52:17.387827 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:17.387784 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c931c147-75a3-4a38-99ff-26d0aa98a38a-istio-data\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xmmdd\" (UID: \"c931c147-75a3-4a38-99ff-26d0aa98a38a\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xmmdd" Nov 03 00:52:17.387928 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:17.387833 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c931c147-75a3-4a38-99ff-26d0aa98a38a-credential-socket\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xmmdd\" (UID: \"c931c147-75a3-4a38-99ff-26d0aa98a38a\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xmmdd" Nov 03 00:52:17.387928 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:17.387900 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c931c147-75a3-4a38-99ff-26d0aa98a38a-istio-podinfo\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xmmdd\" (UID: \"c931c147-75a3-4a38-99ff-26d0aa98a38a\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xmmdd" Nov 03 00:52:17.388029 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:17.387942 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c931c147-75a3-4a38-99ff-26d0aa98a38a-istio-envoy\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xmmdd\" (UID: \"c931c147-75a3-4a38-99ff-26d0aa98a38a\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xmmdd" Nov 03 00:52:17.488845 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:17.488736 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c931c147-75a3-4a38-99ff-26d0aa98a38a-workload-socket\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xmmdd\" (UID: \"c931c147-75a3-4a38-99ff-26d0aa98a38a\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xmmdd" Nov 03 00:52:17.488845 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:17.488782 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c931c147-75a3-4a38-99ff-26d0aa98a38a-istio-data\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xmmdd\" (UID: \"c931c147-75a3-4a38-99ff-26d0aa98a38a\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xmmdd" Nov 03 00:52:17.488845 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:17.488813 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c931c147-75a3-4a38-99ff-26d0aa98a38a-credential-socket\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xmmdd\" (UID: \"c931c147-75a3-4a38-99ff-26d0aa98a38a\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xmmdd" Nov 03 00:52:17.489156 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:17.488854 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c931c147-75a3-4a38-99ff-26d0aa98a38a-istio-podinfo\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xmmdd\" (UID: \"c931c147-75a3-4a38-99ff-26d0aa98a38a\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xmmdd" Nov 03 00:52:17.489156 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:17.488878 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c931c147-75a3-4a38-99ff-26d0aa98a38a-istio-envoy\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xmmdd\" (UID: \"c931c147-75a3-4a38-99ff-26d0aa98a38a\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xmmdd" Nov 03 00:52:17.489156 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:17.488934 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c931c147-75a3-4a38-99ff-26d0aa98a38a-workload-certs\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xmmdd\" (UID: \"c931c147-75a3-4a38-99ff-26d0aa98a38a\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xmmdd" Nov 03 00:52:17.489156 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:17.488959 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c931c147-75a3-4a38-99ff-26d0aa98a38a-istiod-ca-cert\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xmmdd\" (UID: \"c931c147-75a3-4a38-99ff-26d0aa98a38a\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xmmdd" Nov 03 00:52:17.489156 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:17.488987 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x94sd\" (UniqueName: \"kubernetes.io/projected/c931c147-75a3-4a38-99ff-26d0aa98a38a-kube-api-access-x94sd\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xmmdd\" (UID: \"c931c147-75a3-4a38-99ff-26d0aa98a38a\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xmmdd" Nov 03 00:52:17.489156 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:17.489022 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c931c147-75a3-4a38-99ff-26d0aa98a38a-istio-token\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xmmdd\" (UID: \"c931c147-75a3-4a38-99ff-26d0aa98a38a\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xmmdd" Nov 03 00:52:17.489457 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:17.489234 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c931c147-75a3-4a38-99ff-26d0aa98a38a-workload-socket\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xmmdd\" (UID: \"c931c147-75a3-4a38-99ff-26d0aa98a38a\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xmmdd" Nov 03 00:52:17.489457 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:17.489330 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c931c147-75a3-4a38-99ff-26d0aa98a38a-istio-data\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xmmdd\" (UID: \"c931c147-75a3-4a38-99ff-26d0aa98a38a\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xmmdd" Nov 03 00:52:17.489457 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:17.489362 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c931c147-75a3-4a38-99ff-26d0aa98a38a-credential-socket\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xmmdd\" (UID: \"c931c147-75a3-4a38-99ff-26d0aa98a38a\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xmmdd" Nov 03 00:52:17.489821 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:17.489793 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c931c147-75a3-4a38-99ff-26d0aa98a38a-istiod-ca-cert\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xmmdd\" (UID: \"c931c147-75a3-4a38-99ff-26d0aa98a38a\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xmmdd" Nov 03 00:52:17.489922 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:17.489854 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c931c147-75a3-4a38-99ff-26d0aa98a38a-workload-certs\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xmmdd\" (UID: \"c931c147-75a3-4a38-99ff-26d0aa98a38a\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xmmdd" Nov 03 00:52:17.491522 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:17.491493 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c931c147-75a3-4a38-99ff-26d0aa98a38a-istio-podinfo\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xmmdd\" (UID: \"c931c147-75a3-4a38-99ff-26d0aa98a38a\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xmmdd" Nov 03 00:52:17.491761 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:17.491735 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c931c147-75a3-4a38-99ff-26d0aa98a38a-istio-envoy\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xmmdd\" (UID: \"c931c147-75a3-4a38-99ff-26d0aa98a38a\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xmmdd" Nov 03 00:52:17.497412 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:17.497383 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x94sd\" (UniqueName: \"kubernetes.io/projected/c931c147-75a3-4a38-99ff-26d0aa98a38a-kube-api-access-x94sd\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xmmdd\" (UID: \"c931c147-75a3-4a38-99ff-26d0aa98a38a\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xmmdd" Nov 03 00:52:17.498158 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:17.498131 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c931c147-75a3-4a38-99ff-26d0aa98a38a-istio-token\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xmmdd\" (UID: \"c931c147-75a3-4a38-99ff-26d0aa98a38a\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xmmdd" Nov 03 00:52:17.523385 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:17.523343 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xmmdd" Nov 03 00:52:17.809173 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:17.809146 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xmmdd"] Nov 03 00:52:17.809975 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:52:17.809951 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc931c147_75a3_4a38_99ff_26d0aa98a38a.slice/crio-ee3770cfb7ab5a227363b25db165c4692f806108681726830aa7ba790fcd06ce WatchSource:0}: Error finding container ee3770cfb7ab5a227363b25db165c4692f806108681726830aa7ba790fcd06ce: Status 404 returned error can't find the container with id ee3770cfb7ab5a227363b25db165c4692f806108681726830aa7ba790fcd06ce Nov 03 00:52:17.812369 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:17.812332 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:52:17.812481 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:17.812437 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:52:17.812540 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:17.812484 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:52:17.889979 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:17.889940 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xmmdd" event={"ID":"c931c147-75a3-4a38-99ff-26d0aa98a38a","Type":"ContainerStarted","Data":"ee3770cfb7ab5a227363b25db165c4692f806108681726830aa7ba790fcd06ce"} Nov 03 00:52:18.897556 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:18.897521 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xmmdd" event={"ID":"c931c147-75a3-4a38-99ff-26d0aa98a38a","Type":"ContainerStarted","Data":"27c7b0cb36fd8ff37135e7f6e5af8b67d75d1f1e3376f811659e6845627e8df6"} Nov 03 00:52:18.931990 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:18.931925 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xmmdd" podStartSLOduration=1.931905337 podStartE2EDuration="1.931905337s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:52:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:52:18.91980974 +0000 UTC m=+681.090812836" watchObservedRunningTime="2026-11-03 00:52:18.931905337 +0000 UTC m=+681.102908434" Nov 03 00:52:19.530373 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:19.530327 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xmmdd" Nov 03 00:52:19.530661 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:19.530438 4421 patch_prober.go:29] interesting pod/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xmmdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 03 00:52:19.530808 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:19.530685 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xmmdd" podUID="c931c147-75a3-4a38-99ff-26d0aa98a38a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 03 00:52:20.531304 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:20.527692 4421 patch_prober.go:29] interesting pod/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xmmdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 03 00:52:20.531304 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:20.527793 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xmmdd" podUID="c931c147-75a3-4a38-99ff-26d0aa98a38a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 03 00:52:21.531300 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:21.527728 4421 patch_prober.go:29] interesting pod/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xmmdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 03 00:52:21.531300 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:21.527796 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xmmdd" podUID="c931c147-75a3-4a38-99ff-26d0aa98a38a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 03 00:52:22.527300 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:22.526795 4421 patch_prober.go:29] interesting pod/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xmmdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 03 00:52:22.527300 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:22.526892 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xmmdd" podUID="c931c147-75a3-4a38-99ff-26d0aa98a38a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 03 00:52:23.069299 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:23.068359 4421 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-650be0/simulator-w4-650be0-kserve-7bc588774f-gvz2k" podUID="a827fa88-f078-4312-8f12-2c799c11b461" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": context deadline exceeded" Nov 03 00:52:23.164300 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:23.163378 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8gm8n" podUID="2d367e12-e46f-4a68-9825-9b1c934d4d12" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:52:23.533490 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:23.533450 4421 patch_prober.go:29] interesting pod/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xmmdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 03 00:52:23.533784 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:23.533753 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xmmdd" podUID="c931c147-75a3-4a38-99ff-26d0aa98a38a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 03 00:52:24.523779 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:24.523738 4421 patch_prober.go:29] interesting pod/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xmmdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 03 00:52:24.524238 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:24.523810 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xmmdd" podUID="c931c147-75a3-4a38-99ff-26d0aa98a38a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 03 00:52:25.266933 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:25.266899 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39930c39-jcztx"] Nov 03 00:52:25.270868 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:25.270849 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39930c39-jcztx" Nov 03 00:52:25.276745 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:25.276723 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 00:52:25.276909 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:25.276894 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:52:25.277148 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:25.277133 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6q4xp\"" Nov 03 00:52:25.278050 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:25.278013 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 00:52:25.281305 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:25.281267 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39930c39-jcztx"] Nov 03 00:52:25.458703 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:25.458663 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d73525f8-b528-4522-9268-b170b3feb79d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-39930c39-jcztx\" (UID: \"d73525f8-b528-4522-9268-b170b3feb79d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39930c39-jcztx" Nov 03 00:52:25.458883 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:25.458858 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d73525f8-b528-4522-9268-b170b3feb79d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-39930c39-jcztx\" (UID: \"d73525f8-b528-4522-9268-b170b3feb79d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39930c39-jcztx" Nov 03 00:52:25.458971 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:25.458909 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gg5nb\" (UniqueName: \"kubernetes.io/projected/d73525f8-b528-4522-9268-b170b3feb79d-kube-api-access-gg5nb\") pod \"maas-api-revoke-sub-39930c39-jcztx\" (UID: \"d73525f8-b528-4522-9268-b170b3feb79d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39930c39-jcztx" Nov 03 00:52:25.529149 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:25.529066 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xmmdd" Nov 03 00:52:25.529665 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:25.529489 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xmmdd" Nov 03 00:52:25.530399 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:25.530377 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xmmdd" Nov 03 00:52:25.560568 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:25.560530 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d73525f8-b528-4522-9268-b170b3feb79d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-39930c39-jcztx\" (UID: \"d73525f8-b528-4522-9268-b170b3feb79d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39930c39-jcztx" Nov 03 00:52:25.560768 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:25.560676 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d73525f8-b528-4522-9268-b170b3feb79d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-39930c39-jcztx\" (UID: \"d73525f8-b528-4522-9268-b170b3feb79d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39930c39-jcztx" Nov 03 00:52:25.560768 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:25.560715 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gg5nb\" (UniqueName: \"kubernetes.io/projected/d73525f8-b528-4522-9268-b170b3feb79d-kube-api-access-gg5nb\") pod \"maas-api-revoke-sub-39930c39-jcztx\" (UID: \"d73525f8-b528-4522-9268-b170b3feb79d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39930c39-jcztx" Nov 03 00:52:25.561167 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:25.561140 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d73525f8-b528-4522-9268-b170b3feb79d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-39930c39-jcztx\" (UID: \"d73525f8-b528-4522-9268-b170b3feb79d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39930c39-jcztx" Nov 03 00:52:25.563005 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:25.562986 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d73525f8-b528-4522-9268-b170b3feb79d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-39930c39-jcztx\" (UID: \"d73525f8-b528-4522-9268-b170b3feb79d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39930c39-jcztx" Nov 03 00:52:25.571647 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:25.571621 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gg5nb\" (UniqueName: \"kubernetes.io/projected/d73525f8-b528-4522-9268-b170b3feb79d-kube-api-access-gg5nb\") pod \"maas-api-revoke-sub-39930c39-jcztx\" (UID: \"d73525f8-b528-4522-9268-b170b3feb79d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39930c39-jcztx" Nov 03 00:52:25.590029 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:25.589998 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39930c39-jcztx" Nov 03 00:52:25.873215 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:25.873187 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39930c39-jcztx"] Nov 03 00:52:25.874887 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:52:25.874862 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd73525f8_b528_4522_9268_b170b3feb79d.slice/crio-3dc238a4df5f6c70d6d0920ca1ab38a705e5061aae23e401e323383f537bace7 WatchSource:0}: Error finding container 3dc238a4df5f6c70d6d0920ca1ab38a705e5061aae23e401e323383f537bace7: Status 404 returned error can't find the container with id 3dc238a4df5f6c70d6d0920ca1ab38a705e5061aae23e401e323383f537bace7 Nov 03 00:52:26.017726 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:26.017679 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39930c39-jcztx" event={"ID":"d73525f8-b528-4522-9268-b170b3feb79d","Type":"ContainerStarted","Data":"3dc238a4df5f6c70d6d0920ca1ab38a705e5061aae23e401e323383f537bace7"} Nov 03 00:52:26.604873 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:26.604840 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-668b4d3c-r2qnq"] Nov 03 00:52:26.608915 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:26.608885 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-668b4d3c-r2qnq" Nov 03 00:52:26.616377 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:26.616350 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-668b4d3c-r2qnq"] Nov 03 00:52:26.776093 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:26.776056 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8228b121-a23c-4757-8617-69b13be7edd6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-668b4d3c-r2qnq\" (UID: \"8228b121-a23c-4757-8617-69b13be7edd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-668b4d3c-r2qnq" Nov 03 00:52:26.776349 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:26.776102 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8228b121-a23c-4757-8617-69b13be7edd6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-668b4d3c-r2qnq\" (UID: \"8228b121-a23c-4757-8617-69b13be7edd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-668b4d3c-r2qnq" Nov 03 00:52:26.776349 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:26.776130 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8bzv\" (UniqueName: \"kubernetes.io/projected/8228b121-a23c-4757-8617-69b13be7edd6-kube-api-access-q8bzv\") pod \"maas-api-revoke-sub-668b4d3c-r2qnq\" (UID: \"8228b121-a23c-4757-8617-69b13be7edd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-668b4d3c-r2qnq" Nov 03 00:52:26.877188 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:26.877101 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8228b121-a23c-4757-8617-69b13be7edd6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-668b4d3c-r2qnq\" (UID: \"8228b121-a23c-4757-8617-69b13be7edd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-668b4d3c-r2qnq" Nov 03 00:52:26.877188 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:26.877147 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8228b121-a23c-4757-8617-69b13be7edd6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-668b4d3c-r2qnq\" (UID: \"8228b121-a23c-4757-8617-69b13be7edd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-668b4d3c-r2qnq" Nov 03 00:52:26.877188 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:26.877180 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8bzv\" (UniqueName: \"kubernetes.io/projected/8228b121-a23c-4757-8617-69b13be7edd6-kube-api-access-q8bzv\") pod \"maas-api-revoke-sub-668b4d3c-r2qnq\" (UID: \"8228b121-a23c-4757-8617-69b13be7edd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-668b4d3c-r2qnq" Nov 03 00:52:26.877819 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:26.877791 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8228b121-a23c-4757-8617-69b13be7edd6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-668b4d3c-r2qnq\" (UID: \"8228b121-a23c-4757-8617-69b13be7edd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-668b4d3c-r2qnq" Nov 03 00:52:26.879555 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:26.879533 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8228b121-a23c-4757-8617-69b13be7edd6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-668b4d3c-r2qnq\" (UID: \"8228b121-a23c-4757-8617-69b13be7edd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-668b4d3c-r2qnq" Nov 03 00:52:26.890353 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:26.890142 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8bzv\" (UniqueName: \"kubernetes.io/projected/8228b121-a23c-4757-8617-69b13be7edd6-kube-api-access-q8bzv\") pod \"maas-api-revoke-sub-668b4d3c-r2qnq\" (UID: \"8228b121-a23c-4757-8617-69b13be7edd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-668b4d3c-r2qnq" Nov 03 00:52:26.937840 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:26.937808 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-668b4d3c-r2qnq" Nov 03 00:52:27.029729 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:27.028476 4421 generic.go:356] "Generic (PLEG): container finished" podID="d73525f8-b528-4522-9268-b170b3feb79d" containerID="ef21830034d3aef53dc42afb9884dfc7ecc4fdd546b17df82efb77572d0613a8" exitCode=0 Nov 03 00:52:27.029729 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:27.028540 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39930c39-jcztx" event={"ID":"d73525f8-b528-4522-9268-b170b3feb79d","Type":"ContainerDied","Data":"ef21830034d3aef53dc42afb9884dfc7ecc4fdd546b17df82efb77572d0613a8"} Nov 03 00:52:27.221565 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:27.221531 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-668b4d3c-r2qnq"] Nov 03 00:52:27.222241 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:52:27.222211 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8228b121_a23c_4757_8617_69b13be7edd6.slice/crio-ec2d0eb1bfa56733f46234e5d2d1ae71559a45d0d859fc453b234f71637f16cd WatchSource:0}: Error finding container ec2d0eb1bfa56733f46234e5d2d1ae71559a45d0d859fc453b234f71637f16cd: Status 404 returned error can't find the container with id ec2d0eb1bfa56733f46234e5d2d1ae71559a45d0d859fc453b234f71637f16cd Nov 03 00:52:28.098649 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:28.093552 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-668b4d3c-r2qnq" event={"ID":"8228b121-a23c-4757-8617-69b13be7edd6","Type":"ContainerStarted","Data":"96393556fd395ff04001ef9b8ac7c2d2651c9982b95bac2f1543d82e8d528f40"} Nov 03 00:52:28.098649 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:28.093612 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-668b4d3c-r2qnq" event={"ID":"8228b121-a23c-4757-8617-69b13be7edd6","Type":"ContainerStarted","Data":"ec2d0eb1bfa56733f46234e5d2d1ae71559a45d0d859fc453b234f71637f16cd"} Nov 03 00:52:28.122306 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:28.118617 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-668b4d3c-r2qnq" podStartSLOduration=2.118592967 podStartE2EDuration="2.118592967s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:52:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:52:28.11051248 +0000 UTC m=+690.281515577" watchObservedRunningTime="2026-11-03 00:52:28.118592967 +0000 UTC m=+690.289596069" Nov 03 00:52:28.400873 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:28.400835 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39930c39-jcztx" Nov 03 00:52:28.404309 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:28.404250 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d73525f8-b528-4522-9268-b170b3feb79d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d73525f8-b528-4522-9268-b170b3feb79d-maas-api-xks-ca\") pod \"d73525f8-b528-4522-9268-b170b3feb79d\" (UID: \"d73525f8-b528-4522-9268-b170b3feb79d\") " Nov 03 00:52:28.404444 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:28.404396 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d73525f8-b528-4522-9268-b170b3feb79d-kube-api-access-gg5nb\" (UniqueName: \"kubernetes.io/projected/d73525f8-b528-4522-9268-b170b3feb79d-kube-api-access-gg5nb\") pod \"d73525f8-b528-4522-9268-b170b3feb79d\" (UID: \"d73525f8-b528-4522-9268-b170b3feb79d\") " Nov 03 00:52:28.404511 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:28.404469 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d73525f8-b528-4522-9268-b170b3feb79d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d73525f8-b528-4522-9268-b170b3feb79d-maas-api-openshift-ca\") pod \"d73525f8-b528-4522-9268-b170b3feb79d\" (UID: \"d73525f8-b528-4522-9268-b170b3feb79d\") " Nov 03 00:52:28.405235 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:28.405206 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d73525f8-b528-4522-9268-b170b3feb79d-maas-api-openshift-ca" pod "d73525f8-b528-4522-9268-b170b3feb79d" (UID: "d73525f8-b528-4522-9268-b170b3feb79d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:52:28.411316 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:28.406674 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d73525f8-b528-4522-9268-b170b3feb79d-maas-api-xks-ca" pod "d73525f8-b528-4522-9268-b170b3feb79d" (UID: "d73525f8-b528-4522-9268-b170b3feb79d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:52:28.411316 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:28.407027 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d73525f8-b528-4522-9268-b170b3feb79d-kube-api-access-gg5nb" pod "d73525f8-b528-4522-9268-b170b3feb79d" (UID: "d73525f8-b528-4522-9268-b170b3feb79d"). InnerVolumeSpecName "kube-api-access-gg5nb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:52:28.505846 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:28.505796 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d73525f8-b528-4522-9268-b170b3feb79d-maas-api-xks-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:52:28.505846 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:28.505843 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gg5nb\" (UniqueName: \"kubernetes.io/projected/d73525f8-b528-4522-9268-b170b3feb79d-kube-api-access-gg5nb\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:52:28.506046 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:28.505861 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d73525f8-b528-4522-9268-b170b3feb79d-maas-api-openshift-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:52:29.100697 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:29.100660 4421 generic.go:356] "Generic (PLEG): container finished" podID="8228b121-a23c-4757-8617-69b13be7edd6" containerID="96393556fd395ff04001ef9b8ac7c2d2651c9982b95bac2f1543d82e8d528f40" exitCode=0 Nov 03 00:52:29.101200 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:29.100742 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-668b4d3c-r2qnq" event={"ID":"8228b121-a23c-4757-8617-69b13be7edd6","Type":"ContainerDied","Data":"96393556fd395ff04001ef9b8ac7c2d2651c9982b95bac2f1543d82e8d528f40"} Nov 03 00:52:29.102182 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:29.102150 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39930c39-jcztx" event={"ID":"d73525f8-b528-4522-9268-b170b3feb79d","Type":"ContainerDied","Data":"3dc238a4df5f6c70d6d0920ca1ab38a705e5061aae23e401e323383f537bace7"} Nov 03 00:52:29.102324 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:29.102189 4421 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3dc238a4df5f6c70d6d0920ca1ab38a705e5061aae23e401e323383f537bace7" Nov 03 00:52:29.102324 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:29.102155 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39930c39-jcztx" Nov 03 00:52:30.645721 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:30.645350 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-668b4d3c-r2qnq" Nov 03 00:52:30.689675 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:30.689641 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8228b121-a23c-4757-8617-69b13be7edd6-kube-api-access-q8bzv\" (UniqueName: \"kubernetes.io/projected/8228b121-a23c-4757-8617-69b13be7edd6-kube-api-access-q8bzv\") pod \"8228b121-a23c-4757-8617-69b13be7edd6\" (UID: \"8228b121-a23c-4757-8617-69b13be7edd6\") " Nov 03 00:52:30.689848 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:30.689809 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8228b121-a23c-4757-8617-69b13be7edd6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8228b121-a23c-4757-8617-69b13be7edd6-maas-api-xks-ca\") pod \"8228b121-a23c-4757-8617-69b13be7edd6\" (UID: \"8228b121-a23c-4757-8617-69b13be7edd6\") " Nov 03 00:52:30.690169 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:30.690144 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8228b121-a23c-4757-8617-69b13be7edd6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8228b121-a23c-4757-8617-69b13be7edd6-maas-api-openshift-ca\") pod \"8228b121-a23c-4757-8617-69b13be7edd6\" (UID: \"8228b121-a23c-4757-8617-69b13be7edd6\") " Nov 03 00:52:30.690641 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:30.690613 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8228b121-a23c-4757-8617-69b13be7edd6-maas-api-openshift-ca" pod "8228b121-a23c-4757-8617-69b13be7edd6" (UID: "8228b121-a23c-4757-8617-69b13be7edd6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:52:30.690753 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:30.690733 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8228b121-a23c-4757-8617-69b13be7edd6-maas-api-openshift-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:52:30.692822 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:30.692235 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8228b121-a23c-4757-8617-69b13be7edd6-kube-api-access-q8bzv" pod "8228b121-a23c-4757-8617-69b13be7edd6" (UID: "8228b121-a23c-4757-8617-69b13be7edd6"). InnerVolumeSpecName "kube-api-access-q8bzv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:52:30.692822 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:30.692373 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8228b121-a23c-4757-8617-69b13be7edd6-maas-api-xks-ca" pod "8228b121-a23c-4757-8617-69b13be7edd6" (UID: "8228b121-a23c-4757-8617-69b13be7edd6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:52:30.791754 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:30.791716 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8bzv\" (UniqueName: \"kubernetes.io/projected/8228b121-a23c-4757-8617-69b13be7edd6-kube-api-access-q8bzv\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:52:30.791754 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:30.791754 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8228b121-a23c-4757-8617-69b13be7edd6-maas-api-xks-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:52:31.211025 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:31.210989 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-668b4d3c-r2qnq" Nov 03 00:52:31.211025 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:31.211010 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-668b4d3c-r2qnq" event={"ID":"8228b121-a23c-4757-8617-69b13be7edd6","Type":"ContainerDied","Data":"ec2d0eb1bfa56733f46234e5d2d1ae71559a45d0d859fc453b234f71637f16cd"} Nov 03 00:52:31.211251 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:31.211064 4421 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ec2d0eb1bfa56733f46234e5d2d1ae71559a45d0d859fc453b234f71637f16cd" Nov 03 00:52:36.908460 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:36.908425 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b146b9f0-c5jsg"] Nov 03 00:52:36.908936 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:36.908922 4421 cpu_manager.go:414] "containerMap: removing container" podUID="d73525f8-b528-4522-9268-b170b3feb79d" containerName="revoke-keys" Nov 03 00:52:36.908986 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:36.908936 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d73525f8-b528-4522-9268-b170b3feb79d" containerName="revoke-keys" Nov 03 00:52:36.908986 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:36.908947 4421 cpu_manager.go:414] "containerMap: removing container" podUID="8228b121-a23c-4757-8617-69b13be7edd6" containerName="revoke-keys" Nov 03 00:52:36.908986 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:36.908952 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8228b121-a23c-4757-8617-69b13be7edd6" containerName="revoke-keys" Nov 03 00:52:36.909078 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:36.909031 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="d73525f8-b528-4522-9268-b170b3feb79d" containerName="revoke-keys" Nov 03 00:52:36.909078 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:36.909041 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="8228b121-a23c-4757-8617-69b13be7edd6" containerName="revoke-keys" Nov 03 00:52:36.912397 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:36.912379 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b146b9f0-c5jsg" Nov 03 00:52:36.918625 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:36.917755 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6q4xp\"" Nov 03 00:52:36.918625 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:36.917799 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:52:36.918625 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:36.917840 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 00:52:36.918625 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:36.917907 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 00:52:36.920568 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:36.920342 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b146b9f0-c5jsg"] Nov 03 00:52:36.967226 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:36.967187 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1396b2d7-b7dc-4d5a-b897-480303f02972-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b146b9f0-c5jsg\" (UID: \"1396b2d7-b7dc-4d5a-b897-480303f02972\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b146b9f0-c5jsg" Nov 03 00:52:36.967450 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:36.967316 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkbrk\" (UniqueName: \"kubernetes.io/projected/1396b2d7-b7dc-4d5a-b897-480303f02972-kube-api-access-wkbrk\") pod \"maas-api-revoke-sub-b146b9f0-c5jsg\" (UID: \"1396b2d7-b7dc-4d5a-b897-480303f02972\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b146b9f0-c5jsg" Nov 03 00:52:36.967450 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:36.967352 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1396b2d7-b7dc-4d5a-b897-480303f02972-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b146b9f0-c5jsg\" (UID: \"1396b2d7-b7dc-4d5a-b897-480303f02972\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b146b9f0-c5jsg" Nov 03 00:52:37.068226 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:37.068189 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkbrk\" (UniqueName: \"kubernetes.io/projected/1396b2d7-b7dc-4d5a-b897-480303f02972-kube-api-access-wkbrk\") pod \"maas-api-revoke-sub-b146b9f0-c5jsg\" (UID: \"1396b2d7-b7dc-4d5a-b897-480303f02972\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b146b9f0-c5jsg" Nov 03 00:52:37.068226 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:37.068239 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1396b2d7-b7dc-4d5a-b897-480303f02972-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b146b9f0-c5jsg\" (UID: \"1396b2d7-b7dc-4d5a-b897-480303f02972\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b146b9f0-c5jsg" Nov 03 00:52:37.068533 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:37.068440 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1396b2d7-b7dc-4d5a-b897-480303f02972-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b146b9f0-c5jsg\" (UID: \"1396b2d7-b7dc-4d5a-b897-480303f02972\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b146b9f0-c5jsg" Nov 03 00:52:37.069042 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:37.069018 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1396b2d7-b7dc-4d5a-b897-480303f02972-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b146b9f0-c5jsg\" (UID: \"1396b2d7-b7dc-4d5a-b897-480303f02972\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b146b9f0-c5jsg" Nov 03 00:52:37.070632 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:37.070610 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1396b2d7-b7dc-4d5a-b897-480303f02972-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b146b9f0-c5jsg\" (UID: \"1396b2d7-b7dc-4d5a-b897-480303f02972\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b146b9f0-c5jsg" Nov 03 00:52:37.076126 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:37.076098 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkbrk\" (UniqueName: \"kubernetes.io/projected/1396b2d7-b7dc-4d5a-b897-480303f02972-kube-api-access-wkbrk\") pod \"maas-api-revoke-sub-b146b9f0-c5jsg\" (UID: \"1396b2d7-b7dc-4d5a-b897-480303f02972\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b146b9f0-c5jsg" Nov 03 00:52:37.228406 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:37.228320 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b146b9f0-c5jsg" Nov 03 00:52:37.523902 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:37.523874 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b146b9f0-c5jsg"] Nov 03 00:52:37.525593 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:52:37.525556 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1396b2d7_b7dc_4d5a_b897_480303f02972.slice/crio-9cf380626dccb969b66aea283fd9d0bff965a6d2efd88be9c563070c214a981d WatchSource:0}: Error finding container 9cf380626dccb969b66aea283fd9d0bff965a6d2efd88be9c563070c214a981d: Status 404 returned error can't find the container with id 9cf380626dccb969b66aea283fd9d0bff965a6d2efd88be9c563070c214a981d Nov 03 00:52:38.262319 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:38.260050 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b146b9f0-c5jsg" event={"ID":"1396b2d7-b7dc-4d5a-b897-480303f02972","Type":"ContainerStarted","Data":"6aa2daf6c740422a5530ea8b75de57bef5049c2917b73267579371d19be92b86"} Nov 03 00:52:38.262319 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:38.260118 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b146b9f0-c5jsg" event={"ID":"1396b2d7-b7dc-4d5a-b897-480303f02972","Type":"ContainerStarted","Data":"9cf380626dccb969b66aea283fd9d0bff965a6d2efd88be9c563070c214a981d"} Nov 03 00:52:38.278419 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:38.278357 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b146b9f0-c5jsg" podStartSLOduration=2.2783379679999998 podStartE2EDuration="2.278337968s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:52:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:52:38.276930029 +0000 UTC m=+700.447933143" watchObservedRunningTime="2026-11-03 00:52:38.278337968 +0000 UTC m=+700.449341064" Nov 03 00:52:39.273829 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:39.273778 4421 generic.go:356] "Generic (PLEG): container finished" podID="1396b2d7-b7dc-4d5a-b897-480303f02972" containerID="6aa2daf6c740422a5530ea8b75de57bef5049c2917b73267579371d19be92b86" exitCode=0 Nov 03 00:52:39.274360 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:39.273898 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b146b9f0-c5jsg" event={"ID":"1396b2d7-b7dc-4d5a-b897-480303f02972","Type":"ContainerDied","Data":"6aa2daf6c740422a5530ea8b75de57bef5049c2917b73267579371d19be92b86"} Nov 03 00:52:40.783234 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:40.783205 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b146b9f0-c5jsg" Nov 03 00:52:40.809738 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:40.809689 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1396b2d7-b7dc-4d5a-b897-480303f02972-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1396b2d7-b7dc-4d5a-b897-480303f02972-maas-api-openshift-ca\") pod \"1396b2d7-b7dc-4d5a-b897-480303f02972\" (UID: \"1396b2d7-b7dc-4d5a-b897-480303f02972\") " Nov 03 00:52:40.809935 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:40.809872 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1396b2d7-b7dc-4d5a-b897-480303f02972-kube-api-access-wkbrk\" (UniqueName: \"kubernetes.io/projected/1396b2d7-b7dc-4d5a-b897-480303f02972-kube-api-access-wkbrk\") pod \"1396b2d7-b7dc-4d5a-b897-480303f02972\" (UID: \"1396b2d7-b7dc-4d5a-b897-480303f02972\") " Nov 03 00:52:40.810016 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:40.809964 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1396b2d7-b7dc-4d5a-b897-480303f02972-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1396b2d7-b7dc-4d5a-b897-480303f02972-maas-api-xks-ca\") pod \"1396b2d7-b7dc-4d5a-b897-480303f02972\" (UID: \"1396b2d7-b7dc-4d5a-b897-480303f02972\") " Nov 03 00:52:40.868336 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:40.867516 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1396b2d7-b7dc-4d5a-b897-480303f02972-maas-api-openshift-ca" pod "1396b2d7-b7dc-4d5a-b897-480303f02972" (UID: "1396b2d7-b7dc-4d5a-b897-480303f02972"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:52:40.898775 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:40.898724 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1396b2d7-b7dc-4d5a-b897-480303f02972-maas-api-xks-ca" pod "1396b2d7-b7dc-4d5a-b897-480303f02972" (UID: "1396b2d7-b7dc-4d5a-b897-480303f02972"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:52:40.900144 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:40.900112 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1396b2d7-b7dc-4d5a-b897-480303f02972-kube-api-access-wkbrk" pod "1396b2d7-b7dc-4d5a-b897-480303f02972" (UID: "1396b2d7-b7dc-4d5a-b897-480303f02972"). InnerVolumeSpecName "kube-api-access-wkbrk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:52:40.931309 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:40.930138 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wkbrk\" (UniqueName: \"kubernetes.io/projected/1396b2d7-b7dc-4d5a-b897-480303f02972-kube-api-access-wkbrk\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:52:40.931309 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:40.930185 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1396b2d7-b7dc-4d5a-b897-480303f02972-maas-api-xks-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:52:40.931309 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:40.930201 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1396b2d7-b7dc-4d5a-b897-480303f02972-maas-api-openshift-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:52:41.336885 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:41.336841 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b146b9f0-c5jsg" event={"ID":"1396b2d7-b7dc-4d5a-b897-480303f02972","Type":"ContainerDied","Data":"9cf380626dccb969b66aea283fd9d0bff965a6d2efd88be9c563070c214a981d"} Nov 03 00:52:41.337090 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:41.336913 4421 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9cf380626dccb969b66aea283fd9d0bff965a6d2efd88be9c563070c214a981d" Nov 03 00:52:41.337090 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:41.336987 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b146b9f0-c5jsg" Nov 03 00:52:42.970595 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:42.970560 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-6b5656f4df-j7686"] Nov 03 00:52:42.971070 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:42.971057 4421 cpu_manager.go:414] "containerMap: removing container" podUID="1396b2d7-b7dc-4d5a-b897-480303f02972" containerName="revoke-keys" Nov 03 00:52:42.971121 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:42.971071 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1396b2d7-b7dc-4d5a-b897-480303f02972" containerName="revoke-keys" Nov 03 00:52:42.971187 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:42.971178 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="1396b2d7-b7dc-4d5a-b897-480303f02972" containerName="revoke-keys" Nov 03 00:52:42.974452 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:42.974424 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6b5656f4df-j7686" Nov 03 00:52:42.992323 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:42.992295 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Nov 03 00:52:42.992464 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:42.992358 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Nov 03 00:52:42.992464 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:42.992416 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Nov 03 00:52:42.992464 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:42.992425 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Nov 03 00:52:42.999545 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:42.999528 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-48d9fe8f921f\"" Nov 03 00:52:42.999654 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:42.999639 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-6a613b831683\"" Nov 03 00:52:42.999791 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:42.999778 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-a60c08844a3e\"" Nov 03 00:52:42.999853 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:42.999782 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-b4ca36909d68\"" Nov 03 00:52:42.999940 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:42.999928 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-r9jtj\"" Nov 03 00:52:43.011871 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:43.011843 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6b5656f4df-j7686"] Nov 03 00:52:43.073268 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:43.073226 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/24b0475e-82e7-432d-a31c-093510606d91-provider-credentials\") pod \"payload-processing-external-model-6b5656f4df-j7686\" (UID: \"24b0475e-82e7-432d-a31c-093510606d91\") " pod="models-as-a-service/payload-processing-external-model-6b5656f4df-j7686" Nov 03 00:52:43.073466 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:43.073266 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jg6q\" (UniqueName: \"kubernetes.io/projected/24b0475e-82e7-432d-a31c-093510606d91-kube-api-access-6jg6q\") pod \"payload-processing-external-model-6b5656f4df-j7686\" (UID: \"24b0475e-82e7-432d-a31c-093510606d91\") " pod="models-as-a-service/payload-processing-external-model-6b5656f4df-j7686" Nov 03 00:52:43.073466 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:43.073427 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24b0475e-82e7-432d-a31c-093510606d91-plugins-config-volume\") pod \"payload-processing-external-model-6b5656f4df-j7686\" (UID: \"24b0475e-82e7-432d-a31c-093510606d91\") " pod="models-as-a-service/payload-processing-external-model-6b5656f4df-j7686" Nov 03 00:52:43.073466 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:43.073447 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/24b0475e-82e7-432d-a31c-093510606d91-routing-overlay-volume\") pod \"payload-processing-external-model-6b5656f4df-j7686\" (UID: \"24b0475e-82e7-432d-a31c-093510606d91\") " pod="models-as-a-service/payload-processing-external-model-6b5656f4df-j7686" Nov 03 00:52:43.174352 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:43.174314 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/24b0475e-82e7-432d-a31c-093510606d91-provider-credentials\") pod \"payload-processing-external-model-6b5656f4df-j7686\" (UID: \"24b0475e-82e7-432d-a31c-093510606d91\") " pod="models-as-a-service/payload-processing-external-model-6b5656f4df-j7686" Nov 03 00:52:43.174352 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:43.174352 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jg6q\" (UniqueName: \"kubernetes.io/projected/24b0475e-82e7-432d-a31c-093510606d91-kube-api-access-6jg6q\") pod \"payload-processing-external-model-6b5656f4df-j7686\" (UID: \"24b0475e-82e7-432d-a31c-093510606d91\") " pod="models-as-a-service/payload-processing-external-model-6b5656f4df-j7686" Nov 03 00:52:43.174594 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:43.174375 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24b0475e-82e7-432d-a31c-093510606d91-plugins-config-volume\") pod \"payload-processing-external-model-6b5656f4df-j7686\" (UID: \"24b0475e-82e7-432d-a31c-093510606d91\") " pod="models-as-a-service/payload-processing-external-model-6b5656f4df-j7686" Nov 03 00:52:43.174594 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:43.174392 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/24b0475e-82e7-432d-a31c-093510606d91-routing-overlay-volume\") pod \"payload-processing-external-model-6b5656f4df-j7686\" (UID: \"24b0475e-82e7-432d-a31c-093510606d91\") " pod="models-as-a-service/payload-processing-external-model-6b5656f4df-j7686" Nov 03 00:52:43.175148 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:43.175117 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/24b0475e-82e7-432d-a31c-093510606d91-routing-overlay-volume\") pod \"payload-processing-external-model-6b5656f4df-j7686\" (UID: \"24b0475e-82e7-432d-a31c-093510606d91\") " pod="models-as-a-service/payload-processing-external-model-6b5656f4df-j7686" Nov 03 00:52:43.175252 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:43.175144 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24b0475e-82e7-432d-a31c-093510606d91-plugins-config-volume\") pod \"payload-processing-external-model-6b5656f4df-j7686\" (UID: \"24b0475e-82e7-432d-a31c-093510606d91\") " pod="models-as-a-service/payload-processing-external-model-6b5656f4df-j7686" Nov 03 00:52:43.177138 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:43.177117 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/24b0475e-82e7-432d-a31c-093510606d91-provider-credentials\") pod \"payload-processing-external-model-6b5656f4df-j7686\" (UID: \"24b0475e-82e7-432d-a31c-093510606d91\") " pod="models-as-a-service/payload-processing-external-model-6b5656f4df-j7686" Nov 03 00:52:43.195768 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:43.195729 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jg6q\" (UniqueName: \"kubernetes.io/projected/24b0475e-82e7-432d-a31c-093510606d91-kube-api-access-6jg6q\") pod \"payload-processing-external-model-6b5656f4df-j7686\" (UID: \"24b0475e-82e7-432d-a31c-093510606d91\") " pod="models-as-a-service/payload-processing-external-model-6b5656f4df-j7686" Nov 03 00:52:43.287119 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:43.287019 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6b5656f4df-j7686" Nov 03 00:52:43.616191 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:43.616165 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6b5656f4df-j7686"] Nov 03 00:52:43.619041 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:52:43.619010 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24b0475e_82e7_432d_a31c_093510606d91.slice/crio-def72c2f6548d13bd0f002a4de38b0d6fa003c14a0ca408b4e54d603f51e57f5 WatchSource:0}: Error finding container def72c2f6548d13bd0f002a4de38b0d6fa003c14a0ca408b4e54d603f51e57f5: Status 404 returned error can't find the container with id def72c2f6548d13bd0f002a4de38b0d6fa003c14a0ca408b4e54d603f51e57f5 Nov 03 00:52:44.370812 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:44.370777 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6b5656f4df-j7686" event={"ID":"24b0475e-82e7-432d-a31c-093510606d91","Type":"ContainerStarted","Data":"3ea713c7d9a1194fb30ae78cad32baa4dd55849172181da43f8b1cd6b9266681"} Nov 03 00:52:44.371240 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:44.370841 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6b5656f4df-j7686" event={"ID":"24b0475e-82e7-432d-a31c-093510606d91","Type":"ContainerStarted","Data":"def72c2f6548d13bd0f002a4de38b0d6fa003c14a0ca408b4e54d603f51e57f5"} Nov 03 00:52:44.371240 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:44.370871 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-6b5656f4df-j7686" Nov 03 00:52:44.394241 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:44.394183 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-6b5656f4df-j7686" podStartSLOduration=2.394167785 podStartE2EDuration="2.394167785s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:52:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:52:44.391889988 +0000 UTC m=+706.562893102" watchObservedRunningTime="2026-11-03 00:52:44.394167785 +0000 UTC m=+706.565170881" Nov 03 00:52:44.669646 ip-10-0-133-217 kubenswrapper[4421]: E1103 00:52:44.669564 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 00:52:45.638525 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:45.638483 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-83755a/distinct-w3-83755a-kserve-86bdbbdd85-2cnnl"] Nov 03 00:52:45.643679 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:45.643650 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-83755a/distinct-w3-83755a-kserve-86bdbbdd85-2cnnl" Nov 03 00:52:45.657983 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:45.657939 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-83755a\"/\"distinct-w3-83755a-kserve-self-signed-certs\"" Nov 03 00:52:45.657983 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:45.657971 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-83755a/distinct-w3-83755a-kserve-86bdbbdd85-2cnnl"] Nov 03 00:52:45.817702 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:45.817663 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cwld\" (UniqueName: \"kubernetes.io/projected/f4ae9624-d127-4a55-837d-7c9ff0267b37-kube-api-access-9cwld\") pod \"distinct-w3-83755a-kserve-86bdbbdd85-2cnnl\" (UID: \"f4ae9624-d127-4a55-837d-7c9ff0267b37\") " pod="e2e-models-e2e-worker-w3-83755a/distinct-w3-83755a-kserve-86bdbbdd85-2cnnl" Nov 03 00:52:45.817858 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:45.817739 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4ae9624-d127-4a55-837d-7c9ff0267b37-tmp-dir\") pod \"distinct-w3-83755a-kserve-86bdbbdd85-2cnnl\" (UID: \"f4ae9624-d127-4a55-837d-7c9ff0267b37\") " pod="e2e-models-e2e-worker-w3-83755a/distinct-w3-83755a-kserve-86bdbbdd85-2cnnl" Nov 03 00:52:45.817918 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:45.817858 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4ae9624-d127-4a55-837d-7c9ff0267b37-tls-certs\") pod \"distinct-w3-83755a-kserve-86bdbbdd85-2cnnl\" (UID: \"f4ae9624-d127-4a55-837d-7c9ff0267b37\") " pod="e2e-models-e2e-worker-w3-83755a/distinct-w3-83755a-kserve-86bdbbdd85-2cnnl" Nov 03 00:52:45.817955 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:45.817918 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4ae9624-d127-4a55-837d-7c9ff0267b37-home\") pod \"distinct-w3-83755a-kserve-86bdbbdd85-2cnnl\" (UID: \"f4ae9624-d127-4a55-837d-7c9ff0267b37\") " pod="e2e-models-e2e-worker-w3-83755a/distinct-w3-83755a-kserve-86bdbbdd85-2cnnl" Nov 03 00:52:45.817995 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:45.817965 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4ae9624-d127-4a55-837d-7c9ff0267b37-model-cache\") pod \"distinct-w3-83755a-kserve-86bdbbdd85-2cnnl\" (UID: \"f4ae9624-d127-4a55-837d-7c9ff0267b37\") " pod="e2e-models-e2e-worker-w3-83755a/distinct-w3-83755a-kserve-86bdbbdd85-2cnnl" Nov 03 00:52:45.818047 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:45.818034 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4ae9624-d127-4a55-837d-7c9ff0267b37-dshm\") pod \"distinct-w3-83755a-kserve-86bdbbdd85-2cnnl\" (UID: \"f4ae9624-d127-4a55-837d-7c9ff0267b37\") " pod="e2e-models-e2e-worker-w3-83755a/distinct-w3-83755a-kserve-86bdbbdd85-2cnnl" Nov 03 00:52:45.919310 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:45.919178 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cwld\" (UniqueName: \"kubernetes.io/projected/f4ae9624-d127-4a55-837d-7c9ff0267b37-kube-api-access-9cwld\") pod \"distinct-w3-83755a-kserve-86bdbbdd85-2cnnl\" (UID: \"f4ae9624-d127-4a55-837d-7c9ff0267b37\") " pod="e2e-models-e2e-worker-w3-83755a/distinct-w3-83755a-kserve-86bdbbdd85-2cnnl" Nov 03 00:52:45.919310 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:45.919251 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4ae9624-d127-4a55-837d-7c9ff0267b37-tmp-dir\") pod \"distinct-w3-83755a-kserve-86bdbbdd85-2cnnl\" (UID: \"f4ae9624-d127-4a55-837d-7c9ff0267b37\") " pod="e2e-models-e2e-worker-w3-83755a/distinct-w3-83755a-kserve-86bdbbdd85-2cnnl" Nov 03 00:52:45.919535 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:45.919323 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4ae9624-d127-4a55-837d-7c9ff0267b37-tls-certs\") pod \"distinct-w3-83755a-kserve-86bdbbdd85-2cnnl\" (UID: \"f4ae9624-d127-4a55-837d-7c9ff0267b37\") " pod="e2e-models-e2e-worker-w3-83755a/distinct-w3-83755a-kserve-86bdbbdd85-2cnnl" Nov 03 00:52:45.919535 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:45.919358 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4ae9624-d127-4a55-837d-7c9ff0267b37-home\") pod \"distinct-w3-83755a-kserve-86bdbbdd85-2cnnl\" (UID: \"f4ae9624-d127-4a55-837d-7c9ff0267b37\") " pod="e2e-models-e2e-worker-w3-83755a/distinct-w3-83755a-kserve-86bdbbdd85-2cnnl" Nov 03 00:52:45.919535 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:45.919386 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4ae9624-d127-4a55-837d-7c9ff0267b37-model-cache\") pod \"distinct-w3-83755a-kserve-86bdbbdd85-2cnnl\" (UID: \"f4ae9624-d127-4a55-837d-7c9ff0267b37\") " pod="e2e-models-e2e-worker-w3-83755a/distinct-w3-83755a-kserve-86bdbbdd85-2cnnl" Nov 03 00:52:45.919688 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:45.919534 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4ae9624-d127-4a55-837d-7c9ff0267b37-dshm\") pod \"distinct-w3-83755a-kserve-86bdbbdd85-2cnnl\" (UID: \"f4ae9624-d127-4a55-837d-7c9ff0267b37\") " pod="e2e-models-e2e-worker-w3-83755a/distinct-w3-83755a-kserve-86bdbbdd85-2cnnl" Nov 03 00:52:45.919789 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:45.919766 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4ae9624-d127-4a55-837d-7c9ff0267b37-tmp-dir\") pod \"distinct-w3-83755a-kserve-86bdbbdd85-2cnnl\" (UID: \"f4ae9624-d127-4a55-837d-7c9ff0267b37\") " pod="e2e-models-e2e-worker-w3-83755a/distinct-w3-83755a-kserve-86bdbbdd85-2cnnl" Nov 03 00:52:45.919859 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:45.919816 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4ae9624-d127-4a55-837d-7c9ff0267b37-home\") pod \"distinct-w3-83755a-kserve-86bdbbdd85-2cnnl\" (UID: \"f4ae9624-d127-4a55-837d-7c9ff0267b37\") " pod="e2e-models-e2e-worker-w3-83755a/distinct-w3-83755a-kserve-86bdbbdd85-2cnnl" Nov 03 00:52:45.919859 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:45.919833 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4ae9624-d127-4a55-837d-7c9ff0267b37-model-cache\") pod \"distinct-w3-83755a-kserve-86bdbbdd85-2cnnl\" (UID: \"f4ae9624-d127-4a55-837d-7c9ff0267b37\") " pod="e2e-models-e2e-worker-w3-83755a/distinct-w3-83755a-kserve-86bdbbdd85-2cnnl" Nov 03 00:52:45.921701 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:45.921679 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4ae9624-d127-4a55-837d-7c9ff0267b37-dshm\") pod \"distinct-w3-83755a-kserve-86bdbbdd85-2cnnl\" (UID: \"f4ae9624-d127-4a55-837d-7c9ff0267b37\") " pod="e2e-models-e2e-worker-w3-83755a/distinct-w3-83755a-kserve-86bdbbdd85-2cnnl" Nov 03 00:52:45.921967 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:45.921948 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4ae9624-d127-4a55-837d-7c9ff0267b37-tls-certs\") pod \"distinct-w3-83755a-kserve-86bdbbdd85-2cnnl\" (UID: \"f4ae9624-d127-4a55-837d-7c9ff0267b37\") " pod="e2e-models-e2e-worker-w3-83755a/distinct-w3-83755a-kserve-86bdbbdd85-2cnnl" Nov 03 00:52:45.927482 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:45.927462 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cwld\" (UniqueName: \"kubernetes.io/projected/f4ae9624-d127-4a55-837d-7c9ff0267b37-kube-api-access-9cwld\") pod \"distinct-w3-83755a-kserve-86bdbbdd85-2cnnl\" (UID: \"f4ae9624-d127-4a55-837d-7c9ff0267b37\") " pod="e2e-models-e2e-worker-w3-83755a/distinct-w3-83755a-kserve-86bdbbdd85-2cnnl" Nov 03 00:52:45.984425 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:45.984395 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-83755a/distinct-w3-83755a-kserve-86bdbbdd85-2cnnl" Nov 03 00:52:46.262545 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:46.262519 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-83755a/distinct-w3-83755a-kserve-86bdbbdd85-2cnnl"] Nov 03 00:52:46.264194 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:52:46.264165 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf4ae9624_d127_4a55_837d_7c9ff0267b37.slice/crio-c77461ae653c3831bfa0320e184b2f0498d17a87b7dcd57deace9f8a06f504f4 WatchSource:0}: Error finding container c77461ae653c3831bfa0320e184b2f0498d17a87b7dcd57deace9f8a06f504f4: Status 404 returned error can't find the container with id c77461ae653c3831bfa0320e184b2f0498d17a87b7dcd57deace9f8a06f504f4 Nov 03 00:52:46.386944 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:46.386912 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-83755a/distinct-w3-83755a-kserve-86bdbbdd85-2cnnl" event={"ID":"f4ae9624-d127-4a55-837d-7c9ff0267b37","Type":"ContainerStarted","Data":"2633a787e3178961d8b1866a31c0a48176dc9947574a2796d272c4ead3e28c59"} Nov 03 00:52:46.387097 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:46.386953 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-83755a/distinct-w3-83755a-kserve-86bdbbdd85-2cnnl" event={"ID":"f4ae9624-d127-4a55-837d-7c9ff0267b37","Type":"ContainerStarted","Data":"c77461ae653c3831bfa0320e184b2f0498d17a87b7dcd57deace9f8a06f504f4"} Nov 03 00:52:46.404589 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:46.404539 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-83755a/distinct-w3-83755a-kserve-86bdbbdd85-2cnnl" podStartSLOduration=1.404524098 podStartE2EDuration="1.404524098s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:52:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:52:46.403467405 +0000 UTC m=+708.574470501" watchObservedRunningTime="2026-11-03 00:52:46.404524098 +0000 UTC m=+708.575527232" Nov 03 00:52:46.985462 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:46.985420 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-83755a/distinct-w3-83755a-kserve-86bdbbdd85-2cnnl" Nov 03 00:52:53.882841 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:53.882754 4421 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-t5bt4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 00:52:53.882841 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:53.882836 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-t5bt4" podUID="2b55c151-d235-4bcc-b602-eca0da4a7348" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 00:52:53.922771 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:53.922374 4421 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-s4dc8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 00:52:53.922771 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:53.922442 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-s4dc8" podUID="ab5414b5-9d47-4c9f-9d0e-82242b9d6db2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 00:52:55.381384 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:55.381353 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-6b5656f4df-j7686" Nov 03 00:52:55.985200 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:55.985169 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-83755a/distinct-w3-83755a-kserve-86bdbbdd85-2cnnl" Nov 03 00:52:55.997936 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:55.997910 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-83755a/distinct-w3-83755a-kserve-86bdbbdd85-2cnnl" Nov 03 00:52:56.504169 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:52:56.504141 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-83755a/distinct-w3-83755a-kserve-86bdbbdd85-2cnnl" Nov 03 00:53:00.208800 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:00.208758 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-bktsz"] Nov 03 00:53:00.214015 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:00.213991 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-bktsz" Nov 03 00:53:00.225349 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:00.225323 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-bktsz"] Nov 03 00:53:00.333824 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:00.333789 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hr8mk\" (UniqueName: \"kubernetes.io/projected/85cec606-e8b7-4ce1-8af2-4afb303268f9-kube-api-access-hr8mk\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-bktsz\" (UID: \"85cec606-e8b7-4ce1-8af2-4afb303268f9\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-bktsz" Nov 03 00:53:00.334033 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:00.333837 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85cec606-e8b7-4ce1-8af2-4afb303268f9-workload-socket\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-bktsz\" (UID: \"85cec606-e8b7-4ce1-8af2-4afb303268f9\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-bktsz" Nov 03 00:53:00.334033 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:00.333953 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85cec606-e8b7-4ce1-8af2-4afb303268f9-istio-envoy\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-bktsz\" (UID: \"85cec606-e8b7-4ce1-8af2-4afb303268f9\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-bktsz" Nov 03 00:53:00.334144 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:00.334054 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85cec606-e8b7-4ce1-8af2-4afb303268f9-istio-data\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-bktsz\" (UID: \"85cec606-e8b7-4ce1-8af2-4afb303268f9\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-bktsz" Nov 03 00:53:00.334144 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:00.334126 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85cec606-e8b7-4ce1-8af2-4afb303268f9-istio-podinfo\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-bktsz\" (UID: \"85cec606-e8b7-4ce1-8af2-4afb303268f9\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-bktsz" Nov 03 00:53:00.334250 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:00.334171 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85cec606-e8b7-4ce1-8af2-4afb303268f9-workload-certs\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-bktsz\" (UID: \"85cec606-e8b7-4ce1-8af2-4afb303268f9\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-bktsz" Nov 03 00:53:00.334340 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:00.334262 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85cec606-e8b7-4ce1-8af2-4afb303268f9-credential-socket\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-bktsz\" (UID: \"85cec606-e8b7-4ce1-8af2-4afb303268f9\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-bktsz" Nov 03 00:53:00.334340 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:00.334334 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85cec606-e8b7-4ce1-8af2-4afb303268f9-istio-token\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-bktsz\" (UID: \"85cec606-e8b7-4ce1-8af2-4afb303268f9\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-bktsz" Nov 03 00:53:00.334452 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:00.334363 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85cec606-e8b7-4ce1-8af2-4afb303268f9-istiod-ca-cert\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-bktsz\" (UID: \"85cec606-e8b7-4ce1-8af2-4afb303268f9\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-bktsz" Nov 03 00:53:00.435946 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:00.435893 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hr8mk\" (UniqueName: \"kubernetes.io/projected/85cec606-e8b7-4ce1-8af2-4afb303268f9-kube-api-access-hr8mk\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-bktsz\" (UID: \"85cec606-e8b7-4ce1-8af2-4afb303268f9\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-bktsz" Nov 03 00:53:00.435946 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:00.435956 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85cec606-e8b7-4ce1-8af2-4afb303268f9-workload-socket\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-bktsz\" (UID: \"85cec606-e8b7-4ce1-8af2-4afb303268f9\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-bktsz" Nov 03 00:53:00.436206 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:00.436001 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85cec606-e8b7-4ce1-8af2-4afb303268f9-istio-envoy\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-bktsz\" (UID: \"85cec606-e8b7-4ce1-8af2-4afb303268f9\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-bktsz" Nov 03 00:53:00.436206 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:00.436051 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85cec606-e8b7-4ce1-8af2-4afb303268f9-istio-data\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-bktsz\" (UID: \"85cec606-e8b7-4ce1-8af2-4afb303268f9\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-bktsz" Nov 03 00:53:00.436206 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:00.436077 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85cec606-e8b7-4ce1-8af2-4afb303268f9-istio-podinfo\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-bktsz\" (UID: \"85cec606-e8b7-4ce1-8af2-4afb303268f9\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-bktsz" Nov 03 00:53:00.436206 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:00.436104 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85cec606-e8b7-4ce1-8af2-4afb303268f9-workload-certs\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-bktsz\" (UID: \"85cec606-e8b7-4ce1-8af2-4afb303268f9\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-bktsz" Nov 03 00:53:00.436206 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:00.436135 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85cec606-e8b7-4ce1-8af2-4afb303268f9-credential-socket\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-bktsz\" (UID: \"85cec606-e8b7-4ce1-8af2-4afb303268f9\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-bktsz" Nov 03 00:53:00.436206 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:00.436159 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85cec606-e8b7-4ce1-8af2-4afb303268f9-istio-token\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-bktsz\" (UID: \"85cec606-e8b7-4ce1-8af2-4afb303268f9\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-bktsz" Nov 03 00:53:00.436206 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:00.436182 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85cec606-e8b7-4ce1-8af2-4afb303268f9-istiod-ca-cert\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-bktsz\" (UID: \"85cec606-e8b7-4ce1-8af2-4afb303268f9\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-bktsz" Nov 03 00:53:00.436594 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:00.436434 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85cec606-e8b7-4ce1-8af2-4afb303268f9-workload-socket\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-bktsz\" (UID: \"85cec606-e8b7-4ce1-8af2-4afb303268f9\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-bktsz" Nov 03 00:53:00.436594 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:00.436539 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85cec606-e8b7-4ce1-8af2-4afb303268f9-istio-data\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-bktsz\" (UID: \"85cec606-e8b7-4ce1-8af2-4afb303268f9\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-bktsz" Nov 03 00:53:00.436734 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:00.436709 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85cec606-e8b7-4ce1-8af2-4afb303268f9-credential-socket\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-bktsz\" (UID: \"85cec606-e8b7-4ce1-8af2-4afb303268f9\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-bktsz" Nov 03 00:53:00.436791 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:00.436733 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85cec606-e8b7-4ce1-8af2-4afb303268f9-workload-certs\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-bktsz\" (UID: \"85cec606-e8b7-4ce1-8af2-4afb303268f9\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-bktsz" Nov 03 00:53:00.436920 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:00.436901 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85cec606-e8b7-4ce1-8af2-4afb303268f9-istiod-ca-cert\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-bktsz\" (UID: \"85cec606-e8b7-4ce1-8af2-4afb303268f9\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-bktsz" Nov 03 00:53:00.438542 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:00.438521 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85cec606-e8b7-4ce1-8af2-4afb303268f9-istio-envoy\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-bktsz\" (UID: \"85cec606-e8b7-4ce1-8af2-4afb303268f9\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-bktsz" Nov 03 00:53:00.438704 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:00.438687 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85cec606-e8b7-4ce1-8af2-4afb303268f9-istio-podinfo\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-bktsz\" (UID: \"85cec606-e8b7-4ce1-8af2-4afb303268f9\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-bktsz" Nov 03 00:53:00.443658 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:00.443625 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85cec606-e8b7-4ce1-8af2-4afb303268f9-istio-token\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-bktsz\" (UID: \"85cec606-e8b7-4ce1-8af2-4afb303268f9\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-bktsz" Nov 03 00:53:00.443758 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:00.443728 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hr8mk\" (UniqueName: \"kubernetes.io/projected/85cec606-e8b7-4ce1-8af2-4afb303268f9-kube-api-access-hr8mk\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-bktsz\" (UID: \"85cec606-e8b7-4ce1-8af2-4afb303268f9\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-bktsz" Nov 03 00:53:00.532447 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:00.532408 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-bktsz" Nov 03 00:53:00.739416 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:00.739382 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-pm4c4"] Nov 03 00:53:00.745483 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:00.745450 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-pm4c4" Nov 03 00:53:00.755203 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:00.755172 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-pm4c4"] Nov 03 00:53:00.818170 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:00.818136 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-bktsz"] Nov 03 00:53:00.819093 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:53:00.819066 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85cec606_e8b7_4ce1_8af2_4afb303268f9.slice/crio-63527a301ff23e236d638e8f1ddc83318675abd5ed169032f09326d220be516c WatchSource:0}: Error finding container 63527a301ff23e236d638e8f1ddc83318675abd5ed169032f09326d220be516c: Status 404 returned error can't find the container with id 63527a301ff23e236d638e8f1ddc83318675abd5ed169032f09326d220be516c Nov 03 00:53:00.821675 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:00.821637 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:53:00.821763 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:00.821727 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:53:00.821809 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:00.821767 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:53:00.842038 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:00.842004 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af8d7fc5-bed0-4ebf-be1c-44805efec25a-istio-data\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-pm4c4\" (UID: \"af8d7fc5-bed0-4ebf-be1c-44805efec25a\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-pm4c4" Nov 03 00:53:00.842149 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:00.842061 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af8d7fc5-bed0-4ebf-be1c-44805efec25a-istio-token\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-pm4c4\" (UID: \"af8d7fc5-bed0-4ebf-be1c-44805efec25a\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-pm4c4" Nov 03 00:53:00.842218 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:00.842190 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af8d7fc5-bed0-4ebf-be1c-44805efec25a-istio-podinfo\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-pm4c4\" (UID: \"af8d7fc5-bed0-4ebf-be1c-44805efec25a\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-pm4c4" Nov 03 00:53:00.842396 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:00.842376 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af8d7fc5-bed0-4ebf-be1c-44805efec25a-istiod-ca-cert\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-pm4c4\" (UID: \"af8d7fc5-bed0-4ebf-be1c-44805efec25a\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-pm4c4" Nov 03 00:53:00.842478 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:00.842412 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af8d7fc5-bed0-4ebf-be1c-44805efec25a-credential-socket\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-pm4c4\" (UID: \"af8d7fc5-bed0-4ebf-be1c-44805efec25a\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-pm4c4" Nov 03 00:53:00.842478 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:00.842441 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af8d7fc5-bed0-4ebf-be1c-44805efec25a-istio-envoy\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-pm4c4\" (UID: \"af8d7fc5-bed0-4ebf-be1c-44805efec25a\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-pm4c4" Nov 03 00:53:00.842558 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:00.842541 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af8d7fc5-bed0-4ebf-be1c-44805efec25a-workload-socket\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-pm4c4\" (UID: \"af8d7fc5-bed0-4ebf-be1c-44805efec25a\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-pm4c4" Nov 03 00:53:00.842599 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:00.842575 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfbdl\" (UniqueName: \"kubernetes.io/projected/af8d7fc5-bed0-4ebf-be1c-44805efec25a-kube-api-access-wfbdl\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-pm4c4\" (UID: \"af8d7fc5-bed0-4ebf-be1c-44805efec25a\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-pm4c4" Nov 03 00:53:00.842677 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:00.842659 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af8d7fc5-bed0-4ebf-be1c-44805efec25a-workload-certs\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-pm4c4\" (UID: \"af8d7fc5-bed0-4ebf-be1c-44805efec25a\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-pm4c4" Nov 03 00:53:00.943717 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:00.943678 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af8d7fc5-bed0-4ebf-be1c-44805efec25a-istio-data\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-pm4c4\" (UID: \"af8d7fc5-bed0-4ebf-be1c-44805efec25a\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-pm4c4" Nov 03 00:53:00.943717 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:00.943724 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af8d7fc5-bed0-4ebf-be1c-44805efec25a-istio-token\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-pm4c4\" (UID: \"af8d7fc5-bed0-4ebf-be1c-44805efec25a\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-pm4c4" Nov 03 00:53:00.944019 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:00.943745 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af8d7fc5-bed0-4ebf-be1c-44805efec25a-istio-podinfo\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-pm4c4\" (UID: \"af8d7fc5-bed0-4ebf-be1c-44805efec25a\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-pm4c4" Nov 03 00:53:00.944019 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:00.943946 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af8d7fc5-bed0-4ebf-be1c-44805efec25a-istiod-ca-cert\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-pm4c4\" (UID: \"af8d7fc5-bed0-4ebf-be1c-44805efec25a\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-pm4c4" Nov 03 00:53:00.944019 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:00.943987 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af8d7fc5-bed0-4ebf-be1c-44805efec25a-credential-socket\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-pm4c4\" (UID: \"af8d7fc5-bed0-4ebf-be1c-44805efec25a\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-pm4c4" Nov 03 00:53:00.944019 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:00.944016 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af8d7fc5-bed0-4ebf-be1c-44805efec25a-istio-envoy\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-pm4c4\" (UID: \"af8d7fc5-bed0-4ebf-be1c-44805efec25a\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-pm4c4" Nov 03 00:53:00.944245 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:00.944057 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af8d7fc5-bed0-4ebf-be1c-44805efec25a-workload-socket\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-pm4c4\" (UID: \"af8d7fc5-bed0-4ebf-be1c-44805efec25a\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-pm4c4" Nov 03 00:53:00.944245 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:00.944090 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfbdl\" (UniqueName: \"kubernetes.io/projected/af8d7fc5-bed0-4ebf-be1c-44805efec25a-kube-api-access-wfbdl\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-pm4c4\" (UID: \"af8d7fc5-bed0-4ebf-be1c-44805efec25a\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-pm4c4" Nov 03 00:53:00.944245 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:00.944146 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af8d7fc5-bed0-4ebf-be1c-44805efec25a-istio-data\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-pm4c4\" (UID: \"af8d7fc5-bed0-4ebf-be1c-44805efec25a\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-pm4c4" Nov 03 00:53:00.944245 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:00.944156 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af8d7fc5-bed0-4ebf-be1c-44805efec25a-workload-certs\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-pm4c4\" (UID: \"af8d7fc5-bed0-4ebf-be1c-44805efec25a\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-pm4c4" Nov 03 00:53:00.944498 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:00.944339 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af8d7fc5-bed0-4ebf-be1c-44805efec25a-credential-socket\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-pm4c4\" (UID: \"af8d7fc5-bed0-4ebf-be1c-44805efec25a\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-pm4c4" Nov 03 00:53:00.944621 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:00.944598 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af8d7fc5-bed0-4ebf-be1c-44805efec25a-workload-certs\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-pm4c4\" (UID: \"af8d7fc5-bed0-4ebf-be1c-44805efec25a\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-pm4c4" Nov 03 00:53:00.944855 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:00.944828 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af8d7fc5-bed0-4ebf-be1c-44805efec25a-workload-socket\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-pm4c4\" (UID: \"af8d7fc5-bed0-4ebf-be1c-44805efec25a\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-pm4c4" Nov 03 00:53:00.944981 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:00.944954 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af8d7fc5-bed0-4ebf-be1c-44805efec25a-istiod-ca-cert\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-pm4c4\" (UID: \"af8d7fc5-bed0-4ebf-be1c-44805efec25a\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-pm4c4" Nov 03 00:53:00.946410 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:00.946383 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af8d7fc5-bed0-4ebf-be1c-44805efec25a-istio-envoy\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-pm4c4\" (UID: \"af8d7fc5-bed0-4ebf-be1c-44805efec25a\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-pm4c4" Nov 03 00:53:00.946517 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:00.946423 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af8d7fc5-bed0-4ebf-be1c-44805efec25a-istio-podinfo\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-pm4c4\" (UID: \"af8d7fc5-bed0-4ebf-be1c-44805efec25a\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-pm4c4" Nov 03 00:53:00.951511 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:00.951483 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af8d7fc5-bed0-4ebf-be1c-44805efec25a-istio-token\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-pm4c4\" (UID: \"af8d7fc5-bed0-4ebf-be1c-44805efec25a\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-pm4c4" Nov 03 00:53:00.951788 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:00.951768 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfbdl\" (UniqueName: \"kubernetes.io/projected/af8d7fc5-bed0-4ebf-be1c-44805efec25a-kube-api-access-wfbdl\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-pm4c4\" (UID: \"af8d7fc5-bed0-4ebf-be1c-44805efec25a\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-pm4c4" Nov 03 00:53:01.064985 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:01.064950 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-pm4c4" Nov 03 00:53:01.356564 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:01.356530 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-pm4c4"] Nov 03 00:53:01.357199 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:53:01.357155 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf8d7fc5_bed0_4ebf_be1c_44805efec25a.slice/crio-1bf8296d1c117d5a3bb1ea9b6ee7984822ad1c9b060cfa145e4d868167c549e7 WatchSource:0}: Error finding container 1bf8296d1c117d5a3bb1ea9b6ee7984822ad1c9b060cfa145e4d868167c549e7: Status 404 returned error can't find the container with id 1bf8296d1c117d5a3bb1ea9b6ee7984822ad1c9b060cfa145e4d868167c549e7 Nov 03 00:53:01.359901 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:01.359863 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:53:01.359968 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:01.359936 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:53:01.360014 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:01.359966 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:53:01.542166 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:01.541747 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-bktsz" event={"ID":"85cec606-e8b7-4ce1-8af2-4afb303268f9","Type":"ContainerStarted","Data":"37f978db4958aa10297ee8efda2770ebcf37c14ad3910ad9e10f356af46ea6d3"} Nov 03 00:53:01.542602 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:01.542526 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-bktsz" event={"ID":"85cec606-e8b7-4ce1-8af2-4afb303268f9","Type":"ContainerStarted","Data":"63527a301ff23e236d638e8f1ddc83318675abd5ed169032f09326d220be516c"} Nov 03 00:53:01.544000 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:01.543974 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-pm4c4" event={"ID":"af8d7fc5-bed0-4ebf-be1c-44805efec25a","Type":"ContainerStarted","Data":"fa60a24944e6d148d83da74de731ae827243e458dcb289705fab872206721c09"} Nov 03 00:53:01.544108 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:01.544052 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-pm4c4" event={"ID":"af8d7fc5-bed0-4ebf-be1c-44805efec25a","Type":"ContainerStarted","Data":"1bf8296d1c117d5a3bb1ea9b6ee7984822ad1c9b060cfa145e4d868167c549e7"} Nov 03 00:53:01.562541 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:01.562494 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-bktsz" podStartSLOduration=1.562477689 podStartE2EDuration="1.562477689s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:53:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:53:01.562039432 +0000 UTC m=+723.733042529" watchObservedRunningTime="2026-11-03 00:53:01.562477689 +0000 UTC m=+723.733480780" Nov 03 00:53:01.584865 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:01.584805 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-pm4c4" podStartSLOduration=1.5847897789999998 podStartE2EDuration="1.584789779s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:53:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:53:01.582928195 +0000 UTC m=+723.753931293" watchObservedRunningTime="2026-11-03 00:53:01.584789779 +0000 UTC m=+723.755792874" Nov 03 00:53:02.065628 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:02.065594 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-pm4c4" Nov 03 00:53:02.067347 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:02.067315 4421 patch_prober.go:29] interesting pod/e2e-worker-w3-83755a-openshift-default-c86f9565c-pm4c4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 03 00:53:02.067494 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:02.067372 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-pm4c4" podUID="af8d7fc5-bed0-4ebf-be1c-44805efec25a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 03 00:53:02.536299 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:02.533665 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-bktsz" Nov 03 00:53:02.537434 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:02.537401 4421 patch_prober.go:29] interesting pod/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-bktsz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 03 00:53:02.537649 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:02.537622 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-bktsz" podUID="85cec606-e8b7-4ce1-8af2-4afb303268f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 03 00:53:03.065806 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:03.065772 4421 patch_prober.go:29] interesting pod/e2e-worker-w3-83755a-openshift-default-c86f9565c-pm4c4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 03 00:53:03.066124 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:03.066079 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-pm4c4" podUID="af8d7fc5-bed0-4ebf-be1c-44805efec25a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 03 00:53:03.533725 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:03.533682 4421 patch_prober.go:29] interesting pod/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-bktsz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 03 00:53:03.534007 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:03.533975 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-bktsz" podUID="85cec606-e8b7-4ce1-8af2-4afb303268f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 03 00:53:04.068470 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:04.068433 4421 patch_prober.go:29] interesting pod/e2e-worker-w3-83755a-openshift-default-c86f9565c-pm4c4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 03 00:53:04.069066 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:04.069034 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-pm4c4" podUID="af8d7fc5-bed0-4ebf-be1c-44805efec25a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 03 00:53:04.652516 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:04.650861 4421 patch_prober.go:29] interesting pod/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-bktsz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 03 00:53:04.652516 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:04.650936 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-bktsz" podUID="85cec606-e8b7-4ce1-8af2-4afb303268f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 03 00:53:05.075760 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:05.075718 4421 patch_prober.go:29] interesting pod/e2e-worker-w3-83755a-openshift-default-c86f9565c-pm4c4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 03 00:53:05.076406 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:05.076375 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-pm4c4" podUID="af8d7fc5-bed0-4ebf-be1c-44805efec25a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 03 00:53:05.173518 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:05.173448 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8gm8n" podUID="2d367e12-e46f-4a68-9825-9b1c934d4d12" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:53:05.545479 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:05.545400 4421 patch_prober.go:29] interesting pod/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-bktsz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 03 00:53:05.546304 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:05.545749 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-bktsz" podUID="85cec606-e8b7-4ce1-8af2-4afb303268f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 03 00:53:05.561360 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:05.561322 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/distinct-w3-83755a-kserve-86bdbbdd85-2cnnl" podUID="f4ae9624-d127-4a55-837d-7c9ff0267b37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:53:06.068724 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:06.068687 4421 patch_prober.go:29] interesting pod/e2e-worker-w3-83755a-openshift-default-c86f9565c-pm4c4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 03 00:53:06.069065 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:06.069029 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-pm4c4" podUID="af8d7fc5-bed0-4ebf-be1c-44805efec25a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 03 00:53:06.184298 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:06.183842 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8gm8n" podUID="2d367e12-e46f-4a68-9825-9b1c934d4d12" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": context deadline exceeded" Nov 03 00:53:06.190298 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:06.185006 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8gm8n" Nov 03 00:53:06.293300 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:06.292120 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-650be0/simulator-w4-650be0-kserve-7bc588774f-gvz2k" podUID="a827fa88-f078-4312-8f12-2c799c11b461" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Nov 03 00:53:06.366361 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:06.365709 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/simulator-w3-83755a-kserve-77d6664559-psqtg" podUID="b0a4c234-af58-4da0-a1a4-e1f167953f6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:53:06.391300 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:06.386423 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/simulator-w2-f7680d-kserve-85bdc86cd6-8gxrg" podUID="2bdcf521-cebd-4d6b-8554-1b60db8b0f3c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 03 00:53:06.444296 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:06.439477 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-650be0/premium-w4-650be0-kserve-7bf8555dfb-z97n5" podUID="40601dc5-808f-456f-826f-6e0c35966ada" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Nov 03 00:53:06.544300 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:06.543717 4421 patch_prober.go:29] interesting pod/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-bktsz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 03 00:53:06.544300 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:06.543803 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-bktsz" podUID="85cec606-e8b7-4ce1-8af2-4afb303268f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 03 00:53:06.546967 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:06.544883 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/premium-w3-83755a-kserve-7c9d66988b-lt4xd" podUID="c0c6b02e-2ecb-428e-a13d-be6d7cd1ba39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Nov 03 00:53:06.596297 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:06.595675 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/distinct-w3-83755a-kserve-86bdbbdd85-2cnnl" podUID="f4ae9624-d127-4a55-837d-7c9ff0267b37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded" Nov 03 00:53:06.596297 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:06.596085 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-83755a/distinct-w3-83755a-kserve-86bdbbdd85-2cnnl" Nov 03 00:53:07.063324 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:07.001915 4421 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-83755a/distinct-w3-83755a-kserve-86bdbbdd85-2cnnl" podUID="f4ae9624-d127-4a55-837d-7c9ff0267b37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:53:07.132299 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:07.090031 4421 patch_prober.go:29] interesting pod/e2e-worker-w3-83755a-openshift-default-c86f9565c-pm4c4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 03 00:53:07.132299 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:07.090103 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-pm4c4" podUID="af8d7fc5-bed0-4ebf-be1c-44805efec25a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 03 00:53:07.184935 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:07.184892 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8gm8n" podUID="2d367e12-e46f-4a68-9825-9b1c934d4d12" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": context deadline exceeded" Nov 03 00:53:07.304472 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:07.304417 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-650be0/simulator-w4-650be0-kserve-7bc588774f-gvz2k" podUID="a827fa88-f078-4312-8f12-2c799c11b461" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Nov 03 00:53:07.305695 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:07.305671 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-650be0/simulator-w4-650be0-kserve-7bc588774f-gvz2k" Nov 03 00:53:07.433380 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:07.419144 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/simulator-w2-f7680d-kserve-85bdc86cd6-8gxrg" podUID="2bdcf521-cebd-4d6b-8554-1b60db8b0f3c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 03 00:53:07.454306 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:07.434148 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f7680d/simulator-w2-f7680d-kserve-85bdc86cd6-8gxrg" Nov 03 00:53:07.541511 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:07.541448 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/simulator-w3-83755a-kserve-77d6664559-psqtg" podUID="b0a4c234-af58-4da0-a1a4-e1f167953f6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 00:53:07.577954 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:07.577922 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-83755a/simulator-w3-83755a-kserve-77d6664559-psqtg" Nov 03 00:53:07.597450 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:07.583371 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/premium-w3-83755a-kserve-7c9d66988b-lt4xd" podUID="c0c6b02e-2ecb-428e-a13d-be6d7cd1ba39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:53:07.597799 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:07.596512 4421 patch_prober.go:29] interesting pod/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-bktsz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 03 00:53:07.605662 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:07.605598 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-bktsz" podUID="85cec606-e8b7-4ce1-8af2-4afb303268f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 03 00:53:07.605896 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:07.596990 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/distinct-w3-83755a-kserve-86bdbbdd85-2cnnl" podUID="f4ae9624-d127-4a55-837d-7c9ff0267b37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:53:07.624299 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:07.607637 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-83755a/premium-w3-83755a-kserve-7c9d66988b-lt4xd" Nov 03 00:53:08.070298 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:08.069346 4421 patch_prober.go:29] interesting pod/e2e-worker-w3-83755a-openshift-default-c86f9565c-pm4c4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 03 00:53:08.070298 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:08.069405 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-pm4c4" podUID="af8d7fc5-bed0-4ebf-be1c-44805efec25a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 03 00:53:08.189762 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:08.189719 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8gm8n" podUID="2d367e12-e46f-4a68-9825-9b1c934d4d12" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:53:08.313442 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:08.313380 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-650be0/simulator-w4-650be0-kserve-7bc588774f-gvz2k" podUID="a827fa88-f078-4312-8f12-2c799c11b461" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:53:08.398098 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:08.397995 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-83755a/premium-w3-83755a-kserve-7c9d66988b-lt4xd" Nov 03 00:53:08.398394 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:08.398375 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-83755a/distinct-w3-83755a-kserve-86bdbbdd85-2cnnl" Nov 03 00:53:08.398810 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:08.398792 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f7680d/simulator-w2-f7680d-kserve-85bdc86cd6-8gxrg" Nov 03 00:53:08.448325 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:08.446926 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-650be0/premium-w4-650be0-kserve-7bf8555dfb-z97n5" podUID="40601dc5-808f-456f-826f-6e0c35966ada" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Nov 03 00:53:08.537298 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:08.525628 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-83755a/simulator-w3-83755a-kserve-77d6664559-psqtg" Nov 03 00:53:08.537298 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:08.525755 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8gm8n" Nov 03 00:53:08.537298 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:08.530772 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-650be0/simulator-w4-650be0-kserve-7bc588774f-gvz2k" Nov 03 00:53:08.562959 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:08.562927 4421 patch_prober.go:29] interesting pod/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-bktsz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 03 00:53:08.564237 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:08.564200 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-bktsz" podUID="85cec606-e8b7-4ce1-8af2-4afb303268f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 03 00:53:09.066767 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:09.066722 4421 patch_prober.go:29] interesting pod/e2e-worker-w3-83755a-openshift-default-c86f9565c-pm4c4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 03 00:53:09.066956 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:09.066794 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-pm4c4" podUID="af8d7fc5-bed0-4ebf-be1c-44805efec25a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 03 00:53:10.385001 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:10.384966 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-pm4c4" Nov 03 00:53:10.385485 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:10.385373 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-pm4c4" Nov 03 00:53:10.386364 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:10.386341 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-pm4c4" Nov 03 00:53:10.520867 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:10.520822 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-bktsz" Nov 03 00:53:10.521287 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:10.521253 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-bktsz" Nov 03 00:53:10.522141 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:10.522113 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-bktsz" Nov 03 00:53:11.854992 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:11.854953 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9gm5g"] Nov 03 00:53:11.859522 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:11.859497 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9gm5g" Nov 03 00:53:11.877573 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:11.877536 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9gm5g"] Nov 03 00:53:11.943142 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:11.943109 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b76e085-63e9-4361-bab9-d6a88b76c35a-istio-podinfo\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9gm5g\" (UID: \"8b76e085-63e9-4361-bab9-d6a88b76c35a\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9gm5g" Nov 03 00:53:11.943316 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:11.943155 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b76e085-63e9-4361-bab9-d6a88b76c35a-credential-socket\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9gm5g\" (UID: \"8b76e085-63e9-4361-bab9-d6a88b76c35a\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9gm5g" Nov 03 00:53:11.943316 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:11.943174 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b76e085-63e9-4361-bab9-d6a88b76c35a-istio-token\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9gm5g\" (UID: \"8b76e085-63e9-4361-bab9-d6a88b76c35a\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9gm5g" Nov 03 00:53:11.943316 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:11.943301 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b76e085-63e9-4361-bab9-d6a88b76c35a-istio-envoy\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9gm5g\" (UID: \"8b76e085-63e9-4361-bab9-d6a88b76c35a\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9gm5g" Nov 03 00:53:11.943504 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:11.943471 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b76e085-63e9-4361-bab9-d6a88b76c35a-istiod-ca-cert\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9gm5g\" (UID: \"8b76e085-63e9-4361-bab9-d6a88b76c35a\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9gm5g" Nov 03 00:53:11.943561 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:11.943534 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b76e085-63e9-4361-bab9-d6a88b76c35a-istio-data\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9gm5g\" (UID: \"8b76e085-63e9-4361-bab9-d6a88b76c35a\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9gm5g" Nov 03 00:53:11.943619 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:11.943591 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b76e085-63e9-4361-bab9-d6a88b76c35a-workload-socket\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9gm5g\" (UID: \"8b76e085-63e9-4361-bab9-d6a88b76c35a\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9gm5g" Nov 03 00:53:11.943670 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:11.943645 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsw98\" (UniqueName: \"kubernetes.io/projected/8b76e085-63e9-4361-bab9-d6a88b76c35a-kube-api-access-xsw98\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9gm5g\" (UID: \"8b76e085-63e9-4361-bab9-d6a88b76c35a\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9gm5g" Nov 03 00:53:11.943777 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:11.943758 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b76e085-63e9-4361-bab9-d6a88b76c35a-workload-certs\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9gm5g\" (UID: \"8b76e085-63e9-4361-bab9-d6a88b76c35a\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9gm5g" Nov 03 00:53:12.045128 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:12.045089 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b76e085-63e9-4361-bab9-d6a88b76c35a-workload-certs\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9gm5g\" (UID: \"8b76e085-63e9-4361-bab9-d6a88b76c35a\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9gm5g" Nov 03 00:53:12.045128 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:12.045131 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b76e085-63e9-4361-bab9-d6a88b76c35a-istio-podinfo\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9gm5g\" (UID: \"8b76e085-63e9-4361-bab9-d6a88b76c35a\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9gm5g" Nov 03 00:53:12.045433 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:12.045158 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b76e085-63e9-4361-bab9-d6a88b76c35a-credential-socket\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9gm5g\" (UID: \"8b76e085-63e9-4361-bab9-d6a88b76c35a\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9gm5g" Nov 03 00:53:12.045433 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:12.045183 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b76e085-63e9-4361-bab9-d6a88b76c35a-istio-token\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9gm5g\" (UID: \"8b76e085-63e9-4361-bab9-d6a88b76c35a\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9gm5g" Nov 03 00:53:12.045433 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:12.045239 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b76e085-63e9-4361-bab9-d6a88b76c35a-istio-envoy\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9gm5g\" (UID: \"8b76e085-63e9-4361-bab9-d6a88b76c35a\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9gm5g" Nov 03 00:53:12.045433 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:12.045325 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b76e085-63e9-4361-bab9-d6a88b76c35a-istiod-ca-cert\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9gm5g\" (UID: \"8b76e085-63e9-4361-bab9-d6a88b76c35a\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9gm5g" Nov 03 00:53:12.045433 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:12.045382 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b76e085-63e9-4361-bab9-d6a88b76c35a-istio-data\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9gm5g\" (UID: \"8b76e085-63e9-4361-bab9-d6a88b76c35a\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9gm5g" Nov 03 00:53:12.045433 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:12.045432 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b76e085-63e9-4361-bab9-d6a88b76c35a-workload-socket\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9gm5g\" (UID: \"8b76e085-63e9-4361-bab9-d6a88b76c35a\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9gm5g" Nov 03 00:53:12.045745 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:12.045508 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsw98\" (UniqueName: \"kubernetes.io/projected/8b76e085-63e9-4361-bab9-d6a88b76c35a-kube-api-access-xsw98\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9gm5g\" (UID: \"8b76e085-63e9-4361-bab9-d6a88b76c35a\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9gm5g" Nov 03 00:53:12.045745 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:12.045606 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b76e085-63e9-4361-bab9-d6a88b76c35a-workload-certs\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9gm5g\" (UID: \"8b76e085-63e9-4361-bab9-d6a88b76c35a\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9gm5g" Nov 03 00:53:12.045898 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:12.045873 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b76e085-63e9-4361-bab9-d6a88b76c35a-credential-socket\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9gm5g\" (UID: \"8b76e085-63e9-4361-bab9-d6a88b76c35a\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9gm5g" Nov 03 00:53:12.045963 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:12.045897 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b76e085-63e9-4361-bab9-d6a88b76c35a-workload-socket\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9gm5g\" (UID: \"8b76e085-63e9-4361-bab9-d6a88b76c35a\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9gm5g" Nov 03 00:53:12.046035 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:12.045979 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b76e085-63e9-4361-bab9-d6a88b76c35a-istio-data\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9gm5g\" (UID: \"8b76e085-63e9-4361-bab9-d6a88b76c35a\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9gm5g" Nov 03 00:53:12.046195 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:12.046167 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b76e085-63e9-4361-bab9-d6a88b76c35a-istiod-ca-cert\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9gm5g\" (UID: \"8b76e085-63e9-4361-bab9-d6a88b76c35a\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9gm5g" Nov 03 00:53:12.047683 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:12.047661 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b76e085-63e9-4361-bab9-d6a88b76c35a-istio-envoy\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9gm5g\" (UID: \"8b76e085-63e9-4361-bab9-d6a88b76c35a\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9gm5g" Nov 03 00:53:12.047792 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:12.047775 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b76e085-63e9-4361-bab9-d6a88b76c35a-istio-podinfo\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9gm5g\" (UID: \"8b76e085-63e9-4361-bab9-d6a88b76c35a\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9gm5g" Nov 03 00:53:12.060015 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:12.059990 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsw98\" (UniqueName: \"kubernetes.io/projected/8b76e085-63e9-4361-bab9-d6a88b76c35a-kube-api-access-xsw98\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9gm5g\" (UID: \"8b76e085-63e9-4361-bab9-d6a88b76c35a\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9gm5g" Nov 03 00:53:12.060146 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:12.060130 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b76e085-63e9-4361-bab9-d6a88b76c35a-istio-token\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9gm5g\" (UID: \"8b76e085-63e9-4361-bab9-d6a88b76c35a\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9gm5g" Nov 03 00:53:12.177775 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:12.177683 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9gm5g" Nov 03 00:53:12.463198 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:12.463165 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9gm5g"] Nov 03 00:53:12.464162 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:53:12.464131 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b76e085_63e9_4361_bab9_d6a88b76c35a.slice/crio-2a242c4645f2cffbccf114b7add9870d49b48edf70fe76789b47bf675f0d2c4b WatchSource:0}: Error finding container 2a242c4645f2cffbccf114b7add9870d49b48edf70fe76789b47bf675f0d2c4b: Status 404 returned error can't find the container with id 2a242c4645f2cffbccf114b7add9870d49b48edf70fe76789b47bf675f0d2c4b Nov 03 00:53:12.466878 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:12.466843 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:53:12.466971 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:12.466918 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:53:12.466971 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:12.466950 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:53:13.338622 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:13.338583 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9gm5g" event={"ID":"8b76e085-63e9-4361-bab9-d6a88b76c35a","Type":"ContainerStarted","Data":"ed0aac922366879df461e8e0d690987aed3bd8654c2d20d1ae9b1f4c18a4bf55"} Nov 03 00:53:13.339107 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:13.338649 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9gm5g" event={"ID":"8b76e085-63e9-4361-bab9-d6a88b76c35a","Type":"ContainerStarted","Data":"2a242c4645f2cffbccf114b7add9870d49b48edf70fe76789b47bf675f0d2c4b"} Nov 03 00:53:14.178235 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:14.178202 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9gm5g" Nov 03 00:53:14.180149 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:14.180120 4421 patch_prober.go:29] interesting pod/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9gm5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 03 00:53:14.180283 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:14.180168 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9gm5g" podUID="8b76e085-63e9-4361-bab9-d6a88b76c35a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 03 00:53:14.679729 ip-10-0-133-217 kubenswrapper[4421]: E1103 00:53:14.679692 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 00:53:15.178930 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:15.178893 4421 patch_prober.go:29] interesting pod/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9gm5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 03 00:53:15.179118 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:15.178958 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9gm5g" podUID="8b76e085-63e9-4361-bab9-d6a88b76c35a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 03 00:53:15.852400 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:15.852339 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9gm5g" podStartSLOduration=4.852321158 podStartE2EDuration="4.852321158s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:53:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:53:13.367050924 +0000 UTC m=+735.538054021" watchObservedRunningTime="2026-11-03 00:53:15.852321158 +0000 UTC m=+738.023324253" Nov 03 00:53:15.854120 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:15.854097 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77e7728f-74k8r"] Nov 03 00:53:15.858256 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:15.858239 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77e7728f-74k8r" Nov 03 00:53:15.865352 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:15.865322 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 00:53:15.865776 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:15.865750 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:53:15.865925 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:15.865767 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 00:53:15.866005 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:15.865916 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77e7728f-74k8r"] Nov 03 00:53:15.866005 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:15.865776 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6q4xp\"" Nov 03 00:53:16.028298 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:16.005318 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cce43806-c404-486a-b604-8bdae6dd5986-maas-api-xks-ca\") pod \"maas-api-revoke-sub-77e7728f-74k8r\" (UID: \"cce43806-c404-486a-b604-8bdae6dd5986\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77e7728f-74k8r" Nov 03 00:53:16.028647 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:16.028613 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2lnd\" (UniqueName: \"kubernetes.io/projected/cce43806-c404-486a-b604-8bdae6dd5986-kube-api-access-t2lnd\") pod \"maas-api-revoke-sub-77e7728f-74k8r\" (UID: \"cce43806-c404-486a-b604-8bdae6dd5986\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77e7728f-74k8r" Nov 03 00:53:16.029008 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:16.028989 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cce43806-c404-486a-b604-8bdae6dd5986-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-77e7728f-74k8r\" (UID: \"cce43806-c404-486a-b604-8bdae6dd5986\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77e7728f-74k8r" Nov 03 00:53:16.132757 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:16.132638 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cce43806-c404-486a-b604-8bdae6dd5986-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-77e7728f-74k8r\" (UID: \"cce43806-c404-486a-b604-8bdae6dd5986\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77e7728f-74k8r" Nov 03 00:53:16.133064 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:16.133044 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cce43806-c404-486a-b604-8bdae6dd5986-maas-api-xks-ca\") pod \"maas-api-revoke-sub-77e7728f-74k8r\" (UID: \"cce43806-c404-486a-b604-8bdae6dd5986\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77e7728f-74k8r" Nov 03 00:53:16.133206 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:16.133192 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2lnd\" (UniqueName: \"kubernetes.io/projected/cce43806-c404-486a-b604-8bdae6dd5986-kube-api-access-t2lnd\") pod \"maas-api-revoke-sub-77e7728f-74k8r\" (UID: \"cce43806-c404-486a-b604-8bdae6dd5986\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77e7728f-74k8r" Nov 03 00:53:16.133484 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:16.133454 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cce43806-c404-486a-b604-8bdae6dd5986-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-77e7728f-74k8r\" (UID: \"cce43806-c404-486a-b604-8bdae6dd5986\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77e7728f-74k8r" Nov 03 00:53:16.136382 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:16.136347 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cce43806-c404-486a-b604-8bdae6dd5986-maas-api-xks-ca\") pod \"maas-api-revoke-sub-77e7728f-74k8r\" (UID: \"cce43806-c404-486a-b604-8bdae6dd5986\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77e7728f-74k8r" Nov 03 00:53:16.141980 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:16.141951 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2lnd\" (UniqueName: \"kubernetes.io/projected/cce43806-c404-486a-b604-8bdae6dd5986-kube-api-access-t2lnd\") pod \"maas-api-revoke-sub-77e7728f-74k8r\" (UID: \"cce43806-c404-486a-b604-8bdae6dd5986\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77e7728f-74k8r" Nov 03 00:53:16.176988 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:16.176945 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77e7728f-74k8r" Nov 03 00:53:16.178201 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:16.178177 4421 patch_prober.go:29] interesting pod/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9gm5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 03 00:53:16.178323 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:16.178233 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9gm5g" podUID="8b76e085-63e9-4361-bab9-d6a88b76c35a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 03 00:53:16.474003 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:16.473972 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77e7728f-74k8r"] Nov 03 00:53:16.475005 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:53:16.474979 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcce43806_c404_486a_b604_8bdae6dd5986.slice/crio-570c5cefc67cb567f8e275cfdf267b3115ea67e94e7494236567a3f2f99871c9 WatchSource:0}: Error finding container 570c5cefc67cb567f8e275cfdf267b3115ea67e94e7494236567a3f2f99871c9: Status 404 returned error can't find the container with id 570c5cefc67cb567f8e275cfdf267b3115ea67e94e7494236567a3f2f99871c9 Nov 03 00:53:17.179089 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:17.179005 4421 patch_prober.go:29] interesting pod/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9gm5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 03 00:53:17.179089 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:17.179058 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9gm5g" podUID="8b76e085-63e9-4361-bab9-d6a88b76c35a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 03 00:53:17.374774 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:17.374735 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77e7728f-74k8r" event={"ID":"cce43806-c404-486a-b604-8bdae6dd5986","Type":"ContainerStarted","Data":"bebc01dab7ff808c002663bc543eda3dde680929dc6661539aa1bc642cdc9a4f"} Nov 03 00:53:17.374958 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:17.374827 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77e7728f-74k8r" event={"ID":"cce43806-c404-486a-b604-8bdae6dd5986","Type":"ContainerStarted","Data":"570c5cefc67cb567f8e275cfdf267b3115ea67e94e7494236567a3f2f99871c9"} Nov 03 00:53:17.394815 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:17.394758 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77e7728f-74k8r" podStartSLOduration=2.394744535 podStartE2EDuration="2.394744535s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:53:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:53:17.39171741 +0000 UTC m=+739.562720526" watchObservedRunningTime="2026-11-03 00:53:17.394744535 +0000 UTC m=+739.565747630" Nov 03 00:53:18.178769 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:18.178731 4421 patch_prober.go:29] interesting pod/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9gm5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 03 00:53:18.178949 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:18.178797 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9gm5g" podUID="8b76e085-63e9-4361-bab9-d6a88b76c35a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 03 00:53:18.384138 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:18.384100 4421 generic.go:356] "Generic (PLEG): container finished" podID="cce43806-c404-486a-b604-8bdae6dd5986" containerID="bebc01dab7ff808c002663bc543eda3dde680929dc6661539aa1bc642cdc9a4f" exitCode=0 Nov 03 00:53:18.384661 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:18.384188 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77e7728f-74k8r" event={"ID":"cce43806-c404-486a-b604-8bdae6dd5986","Type":"ContainerDied","Data":"bebc01dab7ff808c002663bc543eda3dde680929dc6661539aa1bc642cdc9a4f"} Nov 03 00:53:19.178930 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:19.178891 4421 patch_prober.go:29] interesting pod/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9gm5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 03 00:53:19.179137 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:19.178946 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9gm5g" podUID="8b76e085-63e9-4361-bab9-d6a88b76c35a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 03 00:53:19.638550 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:19.638517 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77e7728f-74k8r" Nov 03 00:53:19.671734 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:19.670815 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cce43806-c404-486a-b604-8bdae6dd5986-kube-api-access-t2lnd\" (UniqueName: \"kubernetes.io/projected/cce43806-c404-486a-b604-8bdae6dd5986-kube-api-access-t2lnd\") pod \"cce43806-c404-486a-b604-8bdae6dd5986\" (UID: \"cce43806-c404-486a-b604-8bdae6dd5986\") " Nov 03 00:53:19.671734 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:19.670997 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cce43806-c404-486a-b604-8bdae6dd5986-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cce43806-c404-486a-b604-8bdae6dd5986-maas-api-xks-ca\") pod \"cce43806-c404-486a-b604-8bdae6dd5986\" (UID: \"cce43806-c404-486a-b604-8bdae6dd5986\") " Nov 03 00:53:19.671734 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:19.671032 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cce43806-c404-486a-b604-8bdae6dd5986-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cce43806-c404-486a-b604-8bdae6dd5986-maas-api-openshift-ca\") pod \"cce43806-c404-486a-b604-8bdae6dd5986\" (UID: \"cce43806-c404-486a-b604-8bdae6dd5986\") " Nov 03 00:53:19.672319 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:19.672194 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cce43806-c404-486a-b604-8bdae6dd5986-maas-api-openshift-ca" pod "cce43806-c404-486a-b604-8bdae6dd5986" (UID: "cce43806-c404-486a-b604-8bdae6dd5986"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:53:19.675103 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:19.675068 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cce43806-c404-486a-b604-8bdae6dd5986-kube-api-access-t2lnd" pod "cce43806-c404-486a-b604-8bdae6dd5986" (UID: "cce43806-c404-486a-b604-8bdae6dd5986"). InnerVolumeSpecName "kube-api-access-t2lnd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:53:19.675563 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:19.675531 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cce43806-c404-486a-b604-8bdae6dd5986-maas-api-xks-ca" pod "cce43806-c404-486a-b604-8bdae6dd5986" (UID: "cce43806-c404-486a-b604-8bdae6dd5986"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:53:19.772603 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:19.772502 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cce43806-c404-486a-b604-8bdae6dd5986-maas-api-xks-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:53:19.772603 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:19.772537 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cce43806-c404-486a-b604-8bdae6dd5986-maas-api-openshift-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:53:19.772603 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:19.772553 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2lnd\" (UniqueName: \"kubernetes.io/projected/cce43806-c404-486a-b604-8bdae6dd5986-kube-api-access-t2lnd\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:53:20.426526 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:20.423729 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77e7728f-74k8r" event={"ID":"cce43806-c404-486a-b604-8bdae6dd5986","Type":"ContainerDied","Data":"570c5cefc67cb567f8e275cfdf267b3115ea67e94e7494236567a3f2f99871c9"} Nov 03 00:53:20.426526 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:20.423837 4421 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="570c5cefc67cb567f8e275cfdf267b3115ea67e94e7494236567a3f2f99871c9" Nov 03 00:53:20.426526 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:20.423934 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77e7728f-74k8r" Nov 03 00:53:21.205959 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:21.205917 4421 patch_prober.go:29] interesting pod/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9gm5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 00:53:21.206513 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:21.205991 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9gm5g" podUID="8b76e085-63e9-4361-bab9-d6a88b76c35a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 00:53:21.207209 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:21.207183 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9gm5g" Nov 03 00:53:21.207729 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:21.207703 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9gm5g" Nov 03 00:53:21.208647 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:21.208628 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9gm5g" Nov 03 00:53:24.829539 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:24.829501 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gw8mh"] Nov 03 00:53:24.830163 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:24.830146 4421 cpu_manager.go:414] "containerMap: removing container" podUID="cce43806-c404-486a-b604-8bdae6dd5986" containerName="revoke-keys" Nov 03 00:53:24.830163 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:24.830164 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cce43806-c404-486a-b604-8bdae6dd5986" containerName="revoke-keys" Nov 03 00:53:24.830353 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:24.830295 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="cce43806-c404-486a-b604-8bdae6dd5986" containerName="revoke-keys" Nov 03 00:53:24.834393 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:24.834372 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gw8mh" Nov 03 00:53:24.841757 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:24.841730 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Nov 03 00:53:24.845865 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:24.845835 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Nov 03 00:53:24.846212 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:24.846186 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gw8mh"] Nov 03 00:53:24.846603 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:24.846578 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-9vdqv\"" Nov 03 00:53:24.846704 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:24.846672 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Nov 03 00:53:24.942174 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:24.942139 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e670068-38e4-4a0f-9313-4757438f9727-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gw8mh\" (UID: \"8e670068-38e4-4a0f-9313-4757438f9727\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gw8mh" Nov 03 00:53:24.942384 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:24.942194 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e670068-38e4-4a0f-9313-4757438f9727-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gw8mh\" (UID: \"8e670068-38e4-4a0f-9313-4757438f9727\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gw8mh" Nov 03 00:53:24.942384 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:24.942228 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/8e670068-38e4-4a0f-9313-4757438f9727-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gw8mh\" (UID: \"8e670068-38e4-4a0f-9313-4757438f9727\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gw8mh" Nov 03 00:53:24.942485 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:24.942395 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j97l5\" (UniqueName: \"kubernetes.io/projected/8e670068-38e4-4a0f-9313-4757438f9727-kube-api-access-j97l5\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gw8mh\" (UID: \"8e670068-38e4-4a0f-9313-4757438f9727\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gw8mh" Nov 03 00:53:24.942528 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:24.942507 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/8e670068-38e4-4a0f-9313-4757438f9727-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gw8mh\" (UID: \"8e670068-38e4-4a0f-9313-4757438f9727\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gw8mh" Nov 03 00:53:24.942594 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:24.942564 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/8e670068-38e4-4a0f-9313-4757438f9727-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gw8mh\" (UID: \"8e670068-38e4-4a0f-9313-4757438f9727\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gw8mh" Nov 03 00:53:24.942637 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:24.942611 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8e670068-38e4-4a0f-9313-4757438f9727-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gw8mh\" (UID: \"8e670068-38e4-4a0f-9313-4757438f9727\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gw8mh" Nov 03 00:53:25.044094 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:25.044060 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/8e670068-38e4-4a0f-9313-4757438f9727-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gw8mh\" (UID: \"8e670068-38e4-4a0f-9313-4757438f9727\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gw8mh" Nov 03 00:53:25.044307 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:25.044106 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j97l5\" (UniqueName: \"kubernetes.io/projected/8e670068-38e4-4a0f-9313-4757438f9727-kube-api-access-j97l5\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gw8mh\" (UID: \"8e670068-38e4-4a0f-9313-4757438f9727\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gw8mh" Nov 03 00:53:25.044307 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:25.044179 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/8e670068-38e4-4a0f-9313-4757438f9727-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gw8mh\" (UID: \"8e670068-38e4-4a0f-9313-4757438f9727\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gw8mh" Nov 03 00:53:25.044307 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:25.044239 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/8e670068-38e4-4a0f-9313-4757438f9727-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gw8mh\" (UID: \"8e670068-38e4-4a0f-9313-4757438f9727\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gw8mh" Nov 03 00:53:25.044307 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:25.044263 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8e670068-38e4-4a0f-9313-4757438f9727-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gw8mh\" (UID: \"8e670068-38e4-4a0f-9313-4757438f9727\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gw8mh" Nov 03 00:53:25.044562 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:25.044442 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e670068-38e4-4a0f-9313-4757438f9727-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gw8mh\" (UID: \"8e670068-38e4-4a0f-9313-4757438f9727\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gw8mh" Nov 03 00:53:25.044688 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:25.044663 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e670068-38e4-4a0f-9313-4757438f9727-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gw8mh\" (UID: \"8e670068-38e4-4a0f-9313-4757438f9727\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gw8mh" Nov 03 00:53:25.045089 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:25.045064 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/8e670068-38e4-4a0f-9313-4757438f9727-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gw8mh\" (UID: \"8e670068-38e4-4a0f-9313-4757438f9727\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gw8mh" Nov 03 00:53:25.046691 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:25.046669 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e670068-38e4-4a0f-9313-4757438f9727-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gw8mh\" (UID: \"8e670068-38e4-4a0f-9313-4757438f9727\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gw8mh" Nov 03 00:53:25.046844 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:25.046820 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8e670068-38e4-4a0f-9313-4757438f9727-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gw8mh\" (UID: \"8e670068-38e4-4a0f-9313-4757438f9727\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gw8mh" Nov 03 00:53:25.047088 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:25.047066 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/8e670068-38e4-4a0f-9313-4757438f9727-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gw8mh\" (UID: \"8e670068-38e4-4a0f-9313-4757438f9727\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gw8mh" Nov 03 00:53:25.047155 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:25.047090 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/8e670068-38e4-4a0f-9313-4757438f9727-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gw8mh\" (UID: \"8e670068-38e4-4a0f-9313-4757438f9727\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gw8mh" Nov 03 00:53:25.051698 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:25.051673 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j97l5\" (UniqueName: \"kubernetes.io/projected/8e670068-38e4-4a0f-9313-4757438f9727-kube-api-access-j97l5\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gw8mh\" (UID: \"8e670068-38e4-4a0f-9313-4757438f9727\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gw8mh" Nov 03 00:53:25.051698 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:25.051692 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e670068-38e4-4a0f-9313-4757438f9727-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gw8mh\" (UID: \"8e670068-38e4-4a0f-9313-4757438f9727\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gw8mh" Nov 03 00:53:25.155093 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:25.155007 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gw8mh" Nov 03 00:53:25.441269 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:25.441244 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gw8mh"] Nov 03 00:53:25.442648 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:53:25.442616 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e670068_38e4_4a0f_9313_4757438f9727.slice/crio-5776400ca28fb73de73f317b4da6d210da6a3c1295d911a09d22efd4ef6f7443 WatchSource:0}: Error finding container 5776400ca28fb73de73f317b4da6d210da6a3c1295d911a09d22efd4ef6f7443: Status 404 returned error can't find the container with id 5776400ca28fb73de73f317b4da6d210da6a3c1295d911a09d22efd4ef6f7443 Nov 03 00:53:25.444770 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:25.444754 4421 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 00:53:25.472850 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:25.472813 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gw8mh" event={"ID":"8e670068-38e4-4a0f-9313-4757438f9727","Type":"ContainerStarted","Data":"5776400ca28fb73de73f317b4da6d210da6a3c1295d911a09d22efd4ef6f7443"} Nov 03 00:53:27.785367 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:27.785333 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e110f7cb-fzlzx"] Nov 03 00:53:27.814437 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:27.813080 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e110f7cb-fzlzx"] Nov 03 00:53:27.814437 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:27.813246 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e110f7cb-fzlzx" Nov 03 00:53:27.821544 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:27.820065 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 00:53:27.821544 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:27.820218 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:53:27.821544 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:27.820436 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 00:53:27.821544 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:27.821059 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6q4xp\"" Nov 03 00:53:27.879098 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:27.879060 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff16a99b-497e-4503-ab96-d5e7063e679e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e110f7cb-fzlzx\" (UID: \"ff16a99b-497e-4503-ab96-d5e7063e679e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e110f7cb-fzlzx" Nov 03 00:53:27.879685 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:27.879656 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff16a99b-497e-4503-ab96-d5e7063e679e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e110f7cb-fzlzx\" (UID: \"ff16a99b-497e-4503-ab96-d5e7063e679e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e110f7cb-fzlzx" Nov 03 00:53:27.879978 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:27.879960 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7d7bz\" (UniqueName: \"kubernetes.io/projected/ff16a99b-497e-4503-ab96-d5e7063e679e-kube-api-access-7d7bz\") pod \"maas-api-revoke-sub-e110f7cb-fzlzx\" (UID: \"ff16a99b-497e-4503-ab96-d5e7063e679e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e110f7cb-fzlzx" Nov 03 00:53:27.981332 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:27.981297 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff16a99b-497e-4503-ab96-d5e7063e679e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e110f7cb-fzlzx\" (UID: \"ff16a99b-497e-4503-ab96-d5e7063e679e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e110f7cb-fzlzx" Nov 03 00:53:27.981532 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:27.981420 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7d7bz\" (UniqueName: \"kubernetes.io/projected/ff16a99b-497e-4503-ab96-d5e7063e679e-kube-api-access-7d7bz\") pod \"maas-api-revoke-sub-e110f7cb-fzlzx\" (UID: \"ff16a99b-497e-4503-ab96-d5e7063e679e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e110f7cb-fzlzx" Nov 03 00:53:27.981597 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:27.981568 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff16a99b-497e-4503-ab96-d5e7063e679e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e110f7cb-fzlzx\" (UID: \"ff16a99b-497e-4503-ab96-d5e7063e679e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e110f7cb-fzlzx" Nov 03 00:53:27.982135 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:27.982106 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff16a99b-497e-4503-ab96-d5e7063e679e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e110f7cb-fzlzx\" (UID: \"ff16a99b-497e-4503-ab96-d5e7063e679e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e110f7cb-fzlzx" Nov 03 00:53:27.984129 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:27.984102 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff16a99b-497e-4503-ab96-d5e7063e679e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e110f7cb-fzlzx\" (UID: \"ff16a99b-497e-4503-ab96-d5e7063e679e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e110f7cb-fzlzx" Nov 03 00:53:27.989716 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:27.989688 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7d7bz\" (UniqueName: \"kubernetes.io/projected/ff16a99b-497e-4503-ab96-d5e7063e679e-kube-api-access-7d7bz\") pod \"maas-api-revoke-sub-e110f7cb-fzlzx\" (UID: \"ff16a99b-497e-4503-ab96-d5e7063e679e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e110f7cb-fzlzx" Nov 03 00:53:28.144765 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:28.144675 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e110f7cb-fzlzx" Nov 03 00:53:28.438180 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:28.438139 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e110f7cb-fzlzx"] Nov 03 00:53:28.439601 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:53:28.439569 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff16a99b_497e_4503_ab96_d5e7063e679e.slice/crio-ecf5cac144888df8c1843ad99a283424892ed65ee80bfe1bbbd0763d5c0ee2bb WatchSource:0}: Error finding container ecf5cac144888df8c1843ad99a283424892ed65ee80bfe1bbbd0763d5c0ee2bb: Status 404 returned error can't find the container with id ecf5cac144888df8c1843ad99a283424892ed65ee80bfe1bbbd0763d5c0ee2bb Nov 03 00:53:28.498625 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:28.498581 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e110f7cb-fzlzx" event={"ID":"ff16a99b-497e-4503-ab96-d5e7063e679e","Type":"ContainerStarted","Data":"ecf5cac144888df8c1843ad99a283424892ed65ee80bfe1bbbd0763d5c0ee2bb"} Nov 03 00:53:29.019074 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:29.019038 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:53:29.019409 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:29.019109 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:53:29.509002 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:29.508958 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gw8mh" event={"ID":"8e670068-38e4-4a0f-9313-4757438f9727","Type":"ContainerStarted","Data":"be251286daa2cd0b2e68643a924b3c1cc424c165c6fdc84393671e7991f4d817"} Nov 03 00:53:29.509246 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:29.509215 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gw8mh" Nov 03 00:53:29.511257 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:29.511232 4421 generic.go:356] "Generic (PLEG): container finished" podID="ff16a99b-497e-4503-ab96-d5e7063e679e" containerID="e39af9db817d96030b05d6c25d7129eb1e8fae324f6bf23be3ee03f9f4b6172c" exitCode=0 Nov 03 00:53:29.511380 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:29.511338 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e110f7cb-fzlzx" event={"ID":"ff16a99b-497e-4503-ab96-d5e7063e679e","Type":"ContainerDied","Data":"e39af9db817d96030b05d6c25d7129eb1e8fae324f6bf23be3ee03f9f4b6172c"} Nov 03 00:53:29.534346 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:29.534295 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gw8mh" podStartSLOduration=1.9613600789999999 podStartE2EDuration="5.534260836s" totalImagesPullingTime="3.572900757s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:53:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:53:29.530523491 +0000 UTC m=+751.701526588" watchObservedRunningTime="2026-11-03 00:53:29.534260836 +0000 UTC m=+751.705263932" Nov 03 00:53:30.520162 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:30.520130 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gw8mh" Nov 03 00:53:30.699487 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:30.699457 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e110f7cb-fzlzx" Nov 03 00:53:30.824344 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:30.824245 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff16a99b-497e-4503-ab96-d5e7063e679e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff16a99b-497e-4503-ab96-d5e7063e679e-maas-api-xks-ca\") pod \"ff16a99b-497e-4503-ab96-d5e7063e679e\" (UID: \"ff16a99b-497e-4503-ab96-d5e7063e679e\") " Nov 03 00:53:30.824344 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:30.824311 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff16a99b-497e-4503-ab96-d5e7063e679e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff16a99b-497e-4503-ab96-d5e7063e679e-maas-api-openshift-ca\") pod \"ff16a99b-497e-4503-ab96-d5e7063e679e\" (UID: \"ff16a99b-497e-4503-ab96-d5e7063e679e\") " Nov 03 00:53:30.824573 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:30.824493 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff16a99b-497e-4503-ab96-d5e7063e679e-kube-api-access-7d7bz\" (UniqueName: \"kubernetes.io/projected/ff16a99b-497e-4503-ab96-d5e7063e679e-kube-api-access-7d7bz\") pod \"ff16a99b-497e-4503-ab96-d5e7063e679e\" (UID: \"ff16a99b-497e-4503-ab96-d5e7063e679e\") " Nov 03 00:53:30.824674 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:30.824656 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff16a99b-497e-4503-ab96-d5e7063e679e-maas-api-openshift-ca" pod "ff16a99b-497e-4503-ab96-d5e7063e679e" (UID: "ff16a99b-497e-4503-ab96-d5e7063e679e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:53:30.825016 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:30.824995 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff16a99b-497e-4503-ab96-d5e7063e679e-maas-api-openshift-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:53:30.826435 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:30.826408 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff16a99b-497e-4503-ab96-d5e7063e679e-maas-api-xks-ca" pod "ff16a99b-497e-4503-ab96-d5e7063e679e" (UID: "ff16a99b-497e-4503-ab96-d5e7063e679e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:53:30.826561 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:30.826542 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff16a99b-497e-4503-ab96-d5e7063e679e-kube-api-access-7d7bz" pod "ff16a99b-497e-4503-ab96-d5e7063e679e" (UID: "ff16a99b-497e-4503-ab96-d5e7063e679e"). InnerVolumeSpecName "kube-api-access-7d7bz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:53:30.933794 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:30.933758 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff16a99b-497e-4503-ab96-d5e7063e679e-maas-api-xks-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:53:30.933794 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:30.933790 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7d7bz\" (UniqueName: \"kubernetes.io/projected/ff16a99b-497e-4503-ab96-d5e7063e679e-kube-api-access-7d7bz\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:53:31.524835 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:31.524801 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e110f7cb-fzlzx" Nov 03 00:53:31.525633 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:31.524831 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e110f7cb-fzlzx" event={"ID":"ff16a99b-497e-4503-ab96-d5e7063e679e","Type":"ContainerDied","Data":"ecf5cac144888df8c1843ad99a283424892ed65ee80bfe1bbbd0763d5c0ee2bb"} Nov 03 00:53:31.525633 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:31.524884 4421 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ecf5cac144888df8c1843ad99a283424892ed65ee80bfe1bbbd0763d5c0ee2bb" Nov 03 00:53:38.682974 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:38.682933 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6b5656f4df-j7686"] Nov 03 00:53:38.827550 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:38.827516 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-513d2002-b9pm2"] Nov 03 00:53:38.828061 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:38.828049 4421 cpu_manager.go:414] "containerMap: removing container" podUID="ff16a99b-497e-4503-ab96-d5e7063e679e" containerName="revoke-keys" Nov 03 00:53:38.828109 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:38.828062 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff16a99b-497e-4503-ab96-d5e7063e679e" containerName="revoke-keys" Nov 03 00:53:38.828156 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:38.828147 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff16a99b-497e-4503-ab96-d5e7063e679e" containerName="revoke-keys" Nov 03 00:53:38.832728 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:38.832709 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-513d2002-b9pm2" Nov 03 00:53:38.838634 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:38.838551 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 00:53:38.838634 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:38.838597 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6q4xp\"" Nov 03 00:53:38.838634 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:38.838610 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:53:38.839053 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:38.839031 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 00:53:38.839558 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:38.839534 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-513d2002-b9pm2"] Nov 03 00:53:38.969765 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:38.969664 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aede371b-8f0e-41ed-baf4-09e0ac5afd50-maas-api-xks-ca\") pod \"maas-api-revoke-sub-513d2002-b9pm2\" (UID: \"aede371b-8f0e-41ed-baf4-09e0ac5afd50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-513d2002-b9pm2" Nov 03 00:53:38.969957 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:38.969846 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aede371b-8f0e-41ed-baf4-09e0ac5afd50-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-513d2002-b9pm2\" (UID: \"aede371b-8f0e-41ed-baf4-09e0ac5afd50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-513d2002-b9pm2" Nov 03 00:53:38.969957 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:38.969905 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvc9k\" (UniqueName: \"kubernetes.io/projected/aede371b-8f0e-41ed-baf4-09e0ac5afd50-kube-api-access-cvc9k\") pod \"maas-api-revoke-sub-513d2002-b9pm2\" (UID: \"aede371b-8f0e-41ed-baf4-09e0ac5afd50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-513d2002-b9pm2" Nov 03 00:53:39.071240 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:39.071200 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aede371b-8f0e-41ed-baf4-09e0ac5afd50-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-513d2002-b9pm2\" (UID: \"aede371b-8f0e-41ed-baf4-09e0ac5afd50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-513d2002-b9pm2" Nov 03 00:53:39.071240 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:39.071247 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvc9k\" (UniqueName: \"kubernetes.io/projected/aede371b-8f0e-41ed-baf4-09e0ac5afd50-kube-api-access-cvc9k\") pod \"maas-api-revoke-sub-513d2002-b9pm2\" (UID: \"aede371b-8f0e-41ed-baf4-09e0ac5afd50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-513d2002-b9pm2" Nov 03 00:53:39.071520 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:39.071380 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aede371b-8f0e-41ed-baf4-09e0ac5afd50-maas-api-xks-ca\") pod \"maas-api-revoke-sub-513d2002-b9pm2\" (UID: \"aede371b-8f0e-41ed-baf4-09e0ac5afd50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-513d2002-b9pm2" Nov 03 00:53:39.072025 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:39.072001 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aede371b-8f0e-41ed-baf4-09e0ac5afd50-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-513d2002-b9pm2\" (UID: \"aede371b-8f0e-41ed-baf4-09e0ac5afd50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-513d2002-b9pm2" Nov 03 00:53:39.073846 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:39.073820 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aede371b-8f0e-41ed-baf4-09e0ac5afd50-maas-api-xks-ca\") pod \"maas-api-revoke-sub-513d2002-b9pm2\" (UID: \"aede371b-8f0e-41ed-baf4-09e0ac5afd50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-513d2002-b9pm2" Nov 03 00:53:39.079465 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:39.079443 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvc9k\" (UniqueName: \"kubernetes.io/projected/aede371b-8f0e-41ed-baf4-09e0ac5afd50-kube-api-access-cvc9k\") pod \"maas-api-revoke-sub-513d2002-b9pm2\" (UID: \"aede371b-8f0e-41ed-baf4-09e0ac5afd50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-513d2002-b9pm2" Nov 03 00:53:39.154220 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:39.154180 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-513d2002-b9pm2" Nov 03 00:53:39.434513 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:53:39.434474 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaede371b_8f0e_41ed_baf4_09e0ac5afd50.slice/crio-7c53fa6badf0bb7a15a41b0dcb076682f58748f811155c5f5369ae54dc0be233 WatchSource:0}: Error finding container 7c53fa6badf0bb7a15a41b0dcb076682f58748f811155c5f5369ae54dc0be233: Status 404 returned error can't find the container with id 7c53fa6badf0bb7a15a41b0dcb076682f58748f811155c5f5369ae54dc0be233 Nov 03 00:53:39.434909 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:39.434881 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-513d2002-b9pm2"] Nov 03 00:53:39.587491 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:39.587452 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-513d2002-b9pm2" event={"ID":"aede371b-8f0e-41ed-baf4-09e0ac5afd50","Type":"ContainerStarted","Data":"7c53fa6badf0bb7a15a41b0dcb076682f58748f811155c5f5369ae54dc0be233"} Nov 03 00:53:40.596734 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:40.596629 4421 generic.go:356] "Generic (PLEG): container finished" podID="aede371b-8f0e-41ed-baf4-09e0ac5afd50" containerID="8903fcef0796c3ca9930ea8d9c034f25c65954b7895d92ff8d0389816970dea4" exitCode=0 Nov 03 00:53:40.596734 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:40.596707 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-513d2002-b9pm2" event={"ID":"aede371b-8f0e-41ed-baf4-09e0ac5afd50","Type":"ContainerDied","Data":"8903fcef0796c3ca9930ea8d9c034f25c65954b7895d92ff8d0389816970dea4"} Nov 03 00:53:41.759089 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:41.759063 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-513d2002-b9pm2" Nov 03 00:53:41.802096 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:41.802058 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aede371b-8f0e-41ed-baf4-09e0ac5afd50-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aede371b-8f0e-41ed-baf4-09e0ac5afd50-maas-api-xks-ca\") pod \"aede371b-8f0e-41ed-baf4-09e0ac5afd50\" (UID: \"aede371b-8f0e-41ed-baf4-09e0ac5afd50\") " Nov 03 00:53:41.802257 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:41.802211 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aede371b-8f0e-41ed-baf4-09e0ac5afd50-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aede371b-8f0e-41ed-baf4-09e0ac5afd50-maas-api-openshift-ca\") pod \"aede371b-8f0e-41ed-baf4-09e0ac5afd50\" (UID: \"aede371b-8f0e-41ed-baf4-09e0ac5afd50\") " Nov 03 00:53:41.802370 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:41.802356 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aede371b-8f0e-41ed-baf4-09e0ac5afd50-kube-api-access-cvc9k\" (UniqueName: \"kubernetes.io/projected/aede371b-8f0e-41ed-baf4-09e0ac5afd50-kube-api-access-cvc9k\") pod \"aede371b-8f0e-41ed-baf4-09e0ac5afd50\" (UID: \"aede371b-8f0e-41ed-baf4-09e0ac5afd50\") " Nov 03 00:53:41.802658 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:41.802630 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aede371b-8f0e-41ed-baf4-09e0ac5afd50-maas-api-openshift-ca" pod "aede371b-8f0e-41ed-baf4-09e0ac5afd50" (UID: "aede371b-8f0e-41ed-baf4-09e0ac5afd50"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:53:41.804445 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:41.804418 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aede371b-8f0e-41ed-baf4-09e0ac5afd50-kube-api-access-cvc9k" pod "aede371b-8f0e-41ed-baf4-09e0ac5afd50" (UID: "aede371b-8f0e-41ed-baf4-09e0ac5afd50"). InnerVolumeSpecName "kube-api-access-cvc9k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:53:41.804686 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:41.804670 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aede371b-8f0e-41ed-baf4-09e0ac5afd50-maas-api-xks-ca" pod "aede371b-8f0e-41ed-baf4-09e0ac5afd50" (UID: "aede371b-8f0e-41ed-baf4-09e0ac5afd50"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:53:41.904154 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:41.904065 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aede371b-8f0e-41ed-baf4-09e0ac5afd50-maas-api-openshift-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:53:41.904154 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:41.904098 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvc9k\" (UniqueName: \"kubernetes.io/projected/aede371b-8f0e-41ed-baf4-09e0ac5afd50-kube-api-access-cvc9k\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:53:41.904154 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:41.904109 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aede371b-8f0e-41ed-baf4-09e0ac5afd50-maas-api-xks-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:53:42.612025 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:42.611994 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-513d2002-b9pm2" Nov 03 00:53:42.612241 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:42.612031 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-513d2002-b9pm2" event={"ID":"aede371b-8f0e-41ed-baf4-09e0ac5afd50","Type":"ContainerDied","Data":"7c53fa6badf0bb7a15a41b0dcb076682f58748f811155c5f5369ae54dc0be233"} Nov 03 00:53:42.612241 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:42.612083 4421 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7c53fa6badf0bb7a15a41b0dcb076682f58748f811155c5f5369ae54dc0be233" Nov 03 00:53:43.683998 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:43.683933 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-6b5656f4df-j7686" podUID="24b0475e-82e7-432d-a31c-093510606d91" containerName="payload-processing" containerID="cri-o://3ea713c7d9a1194fb30ae78cad32baa4dd55849172181da43f8b1cd6b9266681" gracePeriod=25 Nov 03 00:53:43.982530 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:43.982499 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6b5656f4df-j7686" Nov 03 00:53:44.027818 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:44.027781 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24b0475e-82e7-432d-a31c-093510606d91-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24b0475e-82e7-432d-a31c-093510606d91-plugins-config-volume\") pod \"24b0475e-82e7-432d-a31c-093510606d91\" (UID: \"24b0475e-82e7-432d-a31c-093510606d91\") " Nov 03 00:53:44.027983 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:44.027833 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24b0475e-82e7-432d-a31c-093510606d91-kube-api-access-6jg6q\" (UniqueName: \"kubernetes.io/projected/24b0475e-82e7-432d-a31c-093510606d91-kube-api-access-6jg6q\") pod \"24b0475e-82e7-432d-a31c-093510606d91\" (UID: \"24b0475e-82e7-432d-a31c-093510606d91\") " Nov 03 00:53:44.027983 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:44.027930 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24b0475e-82e7-432d-a31c-093510606d91-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/24b0475e-82e7-432d-a31c-093510606d91-routing-overlay-volume\") pod \"24b0475e-82e7-432d-a31c-093510606d91\" (UID: \"24b0475e-82e7-432d-a31c-093510606d91\") " Nov 03 00:53:44.027983 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:44.027974 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24b0475e-82e7-432d-a31c-093510606d91-provider-credentials\" (UniqueName: \"kubernetes.io/projected/24b0475e-82e7-432d-a31c-093510606d91-provider-credentials\") pod \"24b0475e-82e7-432d-a31c-093510606d91\" (UID: \"24b0475e-82e7-432d-a31c-093510606d91\") " Nov 03 00:53:44.028459 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:44.028339 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24b0475e-82e7-432d-a31c-093510606d91-plugins-config-volume" pod "24b0475e-82e7-432d-a31c-093510606d91" (UID: "24b0475e-82e7-432d-a31c-093510606d91"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:53:44.028459 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:44.028392 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24b0475e-82e7-432d-a31c-093510606d91-routing-overlay-volume" pod "24b0475e-82e7-432d-a31c-093510606d91" (UID: "24b0475e-82e7-432d-a31c-093510606d91"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:53:44.028643 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:44.028508 4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24b0475e-82e7-432d-a31c-093510606d91-plugins-config-volume\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:53:44.028643 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:44.028528 4421 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/24b0475e-82e7-432d-a31c-093510606d91-routing-overlay-volume\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:53:44.030912 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:44.030878 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24b0475e-82e7-432d-a31c-093510606d91-kube-api-access-6jg6q" pod "24b0475e-82e7-432d-a31c-093510606d91" (UID: "24b0475e-82e7-432d-a31c-093510606d91"). InnerVolumeSpecName "kube-api-access-6jg6q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:53:44.031045 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:44.030974 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24b0475e-82e7-432d-a31c-093510606d91-provider-credentials" pod "24b0475e-82e7-432d-a31c-093510606d91" (UID: "24b0475e-82e7-432d-a31c-093510606d91"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:53:44.129926 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:44.129884 4421 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/24b0475e-82e7-432d-a31c-093510606d91-provider-credentials\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:53:44.129926 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:44.129917 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6jg6q\" (UniqueName: \"kubernetes.io/projected/24b0475e-82e7-432d-a31c-093510606d91-kube-api-access-6jg6q\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:53:44.630652 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:44.630618 4421 generic.go:356] "Generic (PLEG): container finished" podID="24b0475e-82e7-432d-a31c-093510606d91" containerID="3ea713c7d9a1194fb30ae78cad32baa4dd55849172181da43f8b1cd6b9266681" exitCode=0 Nov 03 00:53:44.630840 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:44.630667 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6b5656f4df-j7686" Nov 03 00:53:44.630840 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:44.630669 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6b5656f4df-j7686" event={"ID":"24b0475e-82e7-432d-a31c-093510606d91","Type":"ContainerDied","Data":"3ea713c7d9a1194fb30ae78cad32baa4dd55849172181da43f8b1cd6b9266681"} Nov 03 00:53:44.630840 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:44.630723 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6b5656f4df-j7686" event={"ID":"24b0475e-82e7-432d-a31c-093510606d91","Type":"ContainerDied","Data":"def72c2f6548d13bd0f002a4de38b0d6fa003c14a0ca408b4e54d603f51e57f5"} Nov 03 00:53:44.630840 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:44.630752 4421 scope.go:122] "RemoveContainer" containerID="3ea713c7d9a1194fb30ae78cad32baa4dd55849172181da43f8b1cd6b9266681" Nov 03 00:53:44.642636 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:44.642616 4421 scope.go:122] "RemoveContainer" containerID="3ea713c7d9a1194fb30ae78cad32baa4dd55849172181da43f8b1cd6b9266681" Nov 03 00:53:44.642965 ip-10-0-133-217 kubenswrapper[4421]: E1103 00:53:44.642937 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ea713c7d9a1194fb30ae78cad32baa4dd55849172181da43f8b1cd6b9266681\": container with ID starting with 3ea713c7d9a1194fb30ae78cad32baa4dd55849172181da43f8b1cd6b9266681 not found: ID does not exist" containerID="3ea713c7d9a1194fb30ae78cad32baa4dd55849172181da43f8b1cd6b9266681" Nov 03 00:53:44.643045 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:44.642978 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ea713c7d9a1194fb30ae78cad32baa4dd55849172181da43f8b1cd6b9266681"} err="failed to get container status \"3ea713c7d9a1194fb30ae78cad32baa4dd55849172181da43f8b1cd6b9266681\": rpc error: code = NotFound desc = could not find container \"3ea713c7d9a1194fb30ae78cad32baa4dd55849172181da43f8b1cd6b9266681\": container with ID starting with 3ea713c7d9a1194fb30ae78cad32baa4dd55849172181da43f8b1cd6b9266681 not found: ID does not exist" Nov 03 00:53:44.661041 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:44.661011 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6b5656f4df-j7686"] Nov 03 00:53:44.669438 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:44.669391 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-6b5656f4df-j7686"] Nov 03 00:53:44.672917 ip-10-0-133-217 kubenswrapper[4421]: E1103 00:53:44.672882 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 00:53:45.844899 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:45.844864 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-mlxvq"] Nov 03 00:53:45.854306 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:45.846227 4421 cpu_manager.go:414] "containerMap: removing container" podUID="24b0475e-82e7-432d-a31c-093510606d91" containerName="payload-processing" Nov 03 00:53:45.854519 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:45.854496 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24b0475e-82e7-432d-a31c-093510606d91" containerName="payload-processing" Nov 03 00:53:45.854678 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:45.854663 4421 cpu_manager.go:414] "containerMap: removing container" podUID="aede371b-8f0e-41ed-baf4-09e0ac5afd50" containerName="revoke-keys" Nov 03 00:53:45.854769 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:45.854757 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aede371b-8f0e-41ed-baf4-09e0ac5afd50" containerName="revoke-keys" Nov 03 00:53:45.855062 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:45.855045 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="24b0475e-82e7-432d-a31c-093510606d91" containerName="payload-processing" Nov 03 00:53:45.855178 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:45.855166 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="aede371b-8f0e-41ed-baf4-09e0ac5afd50" containerName="revoke-keys" Nov 03 00:53:45.860853 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:45.860831 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-mlxvq" Nov 03 00:53:45.871646 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:45.871611 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-mlxvq"] Nov 03 00:53:46.022301 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:46.022206 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcf063ce-88a5-4341-ae0f-605522bb91d5-istio-podinfo\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-mlxvq\" (UID: \"dcf063ce-88a5-4341-ae0f-605522bb91d5\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-mlxvq" Nov 03 00:53:46.022529 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:46.022359 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcf063ce-88a5-4341-ae0f-605522bb91d5-istio-data\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-mlxvq\" (UID: \"dcf063ce-88a5-4341-ae0f-605522bb91d5\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-mlxvq" Nov 03 00:53:46.022529 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:46.022405 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcf063ce-88a5-4341-ae0f-605522bb91d5-workload-certs\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-mlxvq\" (UID: \"dcf063ce-88a5-4341-ae0f-605522bb91d5\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-mlxvq" Nov 03 00:53:46.022529 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:46.022465 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcf063ce-88a5-4341-ae0f-605522bb91d5-istiod-ca-cert\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-mlxvq\" (UID: \"dcf063ce-88a5-4341-ae0f-605522bb91d5\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-mlxvq" Nov 03 00:53:46.022732 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:46.022535 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcf063ce-88a5-4341-ae0f-605522bb91d5-istio-envoy\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-mlxvq\" (UID: \"dcf063ce-88a5-4341-ae0f-605522bb91d5\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-mlxvq" Nov 03 00:53:46.022732 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:46.022568 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcf063ce-88a5-4341-ae0f-605522bb91d5-credential-socket\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-mlxvq\" (UID: \"dcf063ce-88a5-4341-ae0f-605522bb91d5\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-mlxvq" Nov 03 00:53:46.022732 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:46.022603 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcf063ce-88a5-4341-ae0f-605522bb91d5-istio-token\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-mlxvq\" (UID: \"dcf063ce-88a5-4341-ae0f-605522bb91d5\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-mlxvq" Nov 03 00:53:46.022732 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:46.022640 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmhp8\" (UniqueName: \"kubernetes.io/projected/dcf063ce-88a5-4341-ae0f-605522bb91d5-kube-api-access-fmhp8\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-mlxvq\" (UID: \"dcf063ce-88a5-4341-ae0f-605522bb91d5\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-mlxvq" Nov 03 00:53:46.022732 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:46.022719 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcf063ce-88a5-4341-ae0f-605522bb91d5-workload-socket\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-mlxvq\" (UID: \"dcf063ce-88a5-4341-ae0f-605522bb91d5\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-mlxvq" Nov 03 00:53:46.143437 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:46.080200 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24b0475e-82e7-432d-a31c-093510606d91" path="/var/lib/kubelet/pods/24b0475e-82e7-432d-a31c-093510606d91/volumes" Nov 03 00:53:46.170300 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:46.145574 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcf063ce-88a5-4341-ae0f-605522bb91d5-istio-data\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-mlxvq\" (UID: \"dcf063ce-88a5-4341-ae0f-605522bb91d5\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-mlxvq" Nov 03 00:53:46.170300 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:46.149348 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcf063ce-88a5-4341-ae0f-605522bb91d5-workload-certs\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-mlxvq\" (UID: \"dcf063ce-88a5-4341-ae0f-605522bb91d5\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-mlxvq" Nov 03 00:53:46.170300 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:46.149417 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcf063ce-88a5-4341-ae0f-605522bb91d5-istiod-ca-cert\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-mlxvq\" (UID: \"dcf063ce-88a5-4341-ae0f-605522bb91d5\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-mlxvq" Nov 03 00:53:46.170300 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:46.149480 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcf063ce-88a5-4341-ae0f-605522bb91d5-istio-envoy\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-mlxvq\" (UID: \"dcf063ce-88a5-4341-ae0f-605522bb91d5\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-mlxvq" Nov 03 00:53:46.170300 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:46.149513 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcf063ce-88a5-4341-ae0f-605522bb91d5-credential-socket\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-mlxvq\" (UID: \"dcf063ce-88a5-4341-ae0f-605522bb91d5\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-mlxvq" Nov 03 00:53:46.170300 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:46.149544 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcf063ce-88a5-4341-ae0f-605522bb91d5-istio-token\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-mlxvq\" (UID: \"dcf063ce-88a5-4341-ae0f-605522bb91d5\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-mlxvq" Nov 03 00:53:46.170300 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:46.149580 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmhp8\" (UniqueName: \"kubernetes.io/projected/dcf063ce-88a5-4341-ae0f-605522bb91d5-kube-api-access-fmhp8\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-mlxvq\" (UID: \"dcf063ce-88a5-4341-ae0f-605522bb91d5\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-mlxvq" Nov 03 00:53:46.170300 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:46.149691 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcf063ce-88a5-4341-ae0f-605522bb91d5-workload-socket\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-mlxvq\" (UID: \"dcf063ce-88a5-4341-ae0f-605522bb91d5\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-mlxvq" Nov 03 00:53:46.170300 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:46.149782 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcf063ce-88a5-4341-ae0f-605522bb91d5-istio-podinfo\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-mlxvq\" (UID: \"dcf063ce-88a5-4341-ae0f-605522bb91d5\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-mlxvq" Nov 03 00:53:46.170300 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:46.150598 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcf063ce-88a5-4341-ae0f-605522bb91d5-istio-data\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-mlxvq\" (UID: \"dcf063ce-88a5-4341-ae0f-605522bb91d5\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-mlxvq" Nov 03 00:53:46.170300 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:46.150873 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcf063ce-88a5-4341-ae0f-605522bb91d5-workload-certs\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-mlxvq\" (UID: \"dcf063ce-88a5-4341-ae0f-605522bb91d5\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-mlxvq" Nov 03 00:53:46.170300 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:46.151525 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcf063ce-88a5-4341-ae0f-605522bb91d5-istiod-ca-cert\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-mlxvq\" (UID: \"dcf063ce-88a5-4341-ae0f-605522bb91d5\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-mlxvq" Nov 03 00:53:46.170300 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:46.154953 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcf063ce-88a5-4341-ae0f-605522bb91d5-credential-socket\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-mlxvq\" (UID: \"dcf063ce-88a5-4341-ae0f-605522bb91d5\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-mlxvq" Nov 03 00:53:46.170300 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:46.155871 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcf063ce-88a5-4341-ae0f-605522bb91d5-workload-socket\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-mlxvq\" (UID: \"dcf063ce-88a5-4341-ae0f-605522bb91d5\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-mlxvq" Nov 03 00:53:46.307301 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:46.299165 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcf063ce-88a5-4341-ae0f-605522bb91d5-istio-token\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-mlxvq\" (UID: \"dcf063ce-88a5-4341-ae0f-605522bb91d5\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-mlxvq" Nov 03 00:53:46.362063 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:46.362027 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcf063ce-88a5-4341-ae0f-605522bb91d5-istio-podinfo\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-mlxvq\" (UID: \"dcf063ce-88a5-4341-ae0f-605522bb91d5\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-mlxvq" Nov 03 00:53:46.399081 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:46.398985 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcf063ce-88a5-4341-ae0f-605522bb91d5-istio-envoy\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-mlxvq\" (UID: \"dcf063ce-88a5-4341-ae0f-605522bb91d5\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-mlxvq" Nov 03 00:53:46.400763 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:46.399656 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmhp8\" (UniqueName: \"kubernetes.io/projected/dcf063ce-88a5-4341-ae0f-605522bb91d5-kube-api-access-fmhp8\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-mlxvq\" (UID: \"dcf063ce-88a5-4341-ae0f-605522bb91d5\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-mlxvq" Nov 03 00:53:46.686356 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:46.682633 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-mlxvq" Nov 03 00:53:47.065299 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:47.047333 4421 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-83755a/distinct-w3-83755a-kserve-86bdbbdd85-2cnnl" podUID="f4ae9624-d127-4a55-837d-7c9ff0267b37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 00:53:47.369377 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:47.322420 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/premium-w3-83755a-kserve-7c9d66988b-lt4xd" podUID="c0c6b02e-2ecb-428e-a13d-be6d7cd1ba39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:53:47.369377 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:47.348798 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-650be0/simulator-w4-650be0-kserve-7bc588774f-gvz2k" podUID="a827fa88-f078-4312-8f12-2c799c11b461" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:53:47.475299 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:47.409402 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/distinct-w3-83755a-kserve-86bdbbdd85-2cnnl" podUID="f4ae9624-d127-4a55-837d-7c9ff0267b37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:53:47.475299 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:47.450375 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/simulator-w2-f7680d-kserve-85bdc86cd6-8gxrg" podUID="2bdcf521-cebd-4d6b-8554-1b60db8b0f3c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:53:47.475299 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:47.463389 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-650be0/premium-w4-650be0-kserve-7bf8555dfb-z97n5" podUID="40601dc5-808f-456f-826f-6e0c35966ada" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:53:47.595342 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:47.594230 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/simulator-w3-83755a-kserve-77d6664559-psqtg" podUID="b0a4c234-af58-4da0-a1a4-e1f167953f6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Nov 03 00:53:47.661436 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:47.661349 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-mlxvq"] Nov 03 00:53:48.232345 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:48.215518 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8gm8n" podUID="2d367e12-e46f-4a68-9825-9b1c934d4d12" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:53:48.263925 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:53:48.263875 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcf063ce_88a5_4341_ae0f_605522bb91d5.slice/crio-4e0b940675d53c6f1e5283d220d817cb8b929b76b12d6c3e416bd7fdacd5e60d WatchSource:0}: Error finding container 4e0b940675d53c6f1e5283d220d817cb8b929b76b12d6c3e416bd7fdacd5e60d: Status 404 returned error can't find the container with id 4e0b940675d53c6f1e5283d220d817cb8b929b76b12d6c3e416bd7fdacd5e60d Nov 03 00:53:48.333367 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:48.333236 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/premium-w3-83755a-kserve-7c9d66988b-lt4xd" podUID="c0c6b02e-2ecb-428e-a13d-be6d7cd1ba39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:53:48.334685 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:48.334624 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:53:48.361016 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:48.360962 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:53:48.361444 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:48.361393 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:53:48.374792 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:48.374746 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-650be0/simulator-w4-650be0-kserve-7bc588774f-gvz2k" podUID="a827fa88-f078-4312-8f12-2c799c11b461" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:53:48.421701 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:48.421664 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-83755a/premium-w3-83755a-kserve-7c9d66988b-lt4xd" Nov 03 00:53:48.422017 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:48.421998 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-650be0/simulator-w4-650be0-kserve-7bc588774f-gvz2k" Nov 03 00:53:48.427454 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:48.427417 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/distinct-w3-83755a-kserve-86bdbbdd85-2cnnl" podUID="f4ae9624-d127-4a55-837d-7c9ff0267b37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:53:48.460301 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:48.436415 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-83755a/distinct-w3-83755a-kserve-86bdbbdd85-2cnnl" Nov 03 00:53:48.460639 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:48.453781 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/simulator-w2-f7680d-kserve-85bdc86cd6-8gxrg" podUID="2bdcf521-cebd-4d6b-8554-1b60db8b0f3c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:53:48.483209 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:48.483164 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f7680d/simulator-w2-f7680d-kserve-85bdc86cd6-8gxrg" Nov 03 00:53:48.483626 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:48.483569 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-650be0/premium-w4-650be0-kserve-7bf8555dfb-z97n5" podUID="40601dc5-808f-456f-826f-6e0c35966ada" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:53:48.564298 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:48.563850 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-650be0/premium-w4-650be0-kserve-7bf8555dfb-z97n5" Nov 03 00:53:48.631349 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:48.630238 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/simulator-w3-83755a-kserve-77d6664559-psqtg" podUID="b0a4c234-af58-4da0-a1a4-e1f167953f6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:53:48.642542 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:48.642494 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-83755a/simulator-w3-83755a-kserve-77d6664559-psqtg" Nov 03 00:53:48.870924 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:48.870893 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f7680d/simulator-w2-f7680d-kserve-85bdc86cd6-8gxrg" Nov 03 00:53:48.883835 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:48.883765 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-650be0/simulator-w4-650be0-kserve-7bc588774f-gvz2k" Nov 03 00:53:48.920466 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:48.893645 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-83755a/simulator-w3-83755a-kserve-77d6664559-psqtg" Nov 03 00:53:48.930387 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:48.930354 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-83755a/premium-w3-83755a-kserve-7c9d66988b-lt4xd" Nov 03 00:53:48.946332 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:48.941799 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-83755a/distinct-w3-83755a-kserve-86bdbbdd85-2cnnl" Nov 03 00:53:48.947834 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:48.947806 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-650be0/premium-w4-650be0-kserve-7bf8555dfb-z97n5" Nov 03 00:53:48.971319 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:48.967651 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-mlxvq" event={"ID":"dcf063ce-88a5-4341-ae0f-605522bb91d5","Type":"ContainerStarted","Data":"4e0b940675d53c6f1e5283d220d817cb8b929b76b12d6c3e416bd7fdacd5e60d"} Nov 03 00:53:49.967195 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:49.967152 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-mlxvq" event={"ID":"dcf063ce-88a5-4341-ae0f-605522bb91d5","Type":"ContainerStarted","Data":"dcf8a1ffba7dd0036d36c7d4b74904648e76bc6c8bdbd457c757c4fc064e6b68"} Nov 03 00:53:49.986814 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:49.986765 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-mlxvq" podStartSLOduration=4.986751542 podStartE2EDuration="4.986751542s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:53:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:53:49.985344966 +0000 UTC m=+772.156348063" watchObservedRunningTime="2026-11-03 00:53:49.986751542 +0000 UTC m=+772.157754638" Nov 03 00:53:50.685930 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:50.685891 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-mlxvq" Nov 03 00:53:50.687685 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:50.687651 4421 patch_prober.go:29] interesting pod/e2e-worker-w3-83755a-openshift-default-c86f9565c-mlxvq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Nov 03 00:53:50.687809 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:50.687709 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-mlxvq" podUID="dcf063ce-88a5-4341-ae0f-605522bb91d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Nov 03 00:53:51.685628 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:51.685597 4421 patch_prober.go:29] interesting pod/e2e-worker-w3-83755a-openshift-default-c86f9565c-mlxvq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Nov 03 00:53:51.686084 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:51.686058 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-mlxvq" podUID="dcf063ce-88a5-4341-ae0f-605522bb91d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Nov 03 00:53:52.685922 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:52.685886 4421 patch_prober.go:29] interesting pod/e2e-worker-w3-83755a-openshift-default-c86f9565c-mlxvq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Nov 03 00:53:52.686365 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:52.685960 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-mlxvq" podUID="dcf063ce-88a5-4341-ae0f-605522bb91d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Nov 03 00:53:53.687316 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:53.686880 4421 patch_prober.go:29] interesting pod/e2e-worker-w3-83755a-openshift-default-c86f9565c-mlxvq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Nov 03 00:53:53.687316 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:53.686948 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-mlxvq" podUID="dcf063ce-88a5-4341-ae0f-605522bb91d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Nov 03 00:53:53.738581 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:53.738547 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d4167a9-92r25"] Nov 03 00:53:53.751429 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:53.751252 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d4167a9-92r25"] Nov 03 00:53:53.751596 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:53.751436 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d4167a9-92r25" Nov 03 00:53:53.755497 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:53.755447 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:53:53.757206 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:53.756730 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6q4xp\"" Nov 03 00:53:53.793699 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:53.793671 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngjz5\" (UniqueName: \"kubernetes.io/projected/430ade2a-8ebf-4b6c-bfaf-71eaba2a6a58-kube-api-access-ngjz5\") pod \"maas-api-revoke-sub-2d4167a9-92r25\" (UID: \"430ade2a-8ebf-4b6c-bfaf-71eaba2a6a58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d4167a9-92r25" Nov 03 00:53:53.793984 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:53.793963 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/430ade2a-8ebf-4b6c-bfaf-71eaba2a6a58-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2d4167a9-92r25\" (UID: \"430ade2a-8ebf-4b6c-bfaf-71eaba2a6a58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d4167a9-92r25" Nov 03 00:53:53.794111 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:53.794093 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/430ade2a-8ebf-4b6c-bfaf-71eaba2a6a58-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2d4167a9-92r25\" (UID: \"430ade2a-8ebf-4b6c-bfaf-71eaba2a6a58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d4167a9-92r25" Nov 03 00:53:53.843390 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:53.843338 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 00:53:53.843572 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:53.843445 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 00:53:53.896026 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:53.895989 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngjz5\" (UniqueName: \"kubernetes.io/projected/430ade2a-8ebf-4b6c-bfaf-71eaba2a6a58-kube-api-access-ngjz5\") pod \"maas-api-revoke-sub-2d4167a9-92r25\" (UID: \"430ade2a-8ebf-4b6c-bfaf-71eaba2a6a58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d4167a9-92r25" Nov 03 00:53:53.896195 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:53.896054 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/430ade2a-8ebf-4b6c-bfaf-71eaba2a6a58-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2d4167a9-92r25\" (UID: \"430ade2a-8ebf-4b6c-bfaf-71eaba2a6a58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d4167a9-92r25" Nov 03 00:53:53.896195 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:53.896072 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/430ade2a-8ebf-4b6c-bfaf-71eaba2a6a58-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2d4167a9-92r25\" (UID: \"430ade2a-8ebf-4b6c-bfaf-71eaba2a6a58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d4167a9-92r25" Nov 03 00:53:53.897083 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:53.897049 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/430ade2a-8ebf-4b6c-bfaf-71eaba2a6a58-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2d4167a9-92r25\" (UID: \"430ade2a-8ebf-4b6c-bfaf-71eaba2a6a58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d4167a9-92r25" Nov 03 00:53:53.898677 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:53.898654 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/430ade2a-8ebf-4b6c-bfaf-71eaba2a6a58-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2d4167a9-92r25\" (UID: \"430ade2a-8ebf-4b6c-bfaf-71eaba2a6a58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d4167a9-92r25" Nov 03 00:53:53.904714 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:53.904686 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngjz5\" (UniqueName: \"kubernetes.io/projected/430ade2a-8ebf-4b6c-bfaf-71eaba2a6a58-kube-api-access-ngjz5\") pod \"maas-api-revoke-sub-2d4167a9-92r25\" (UID: \"430ade2a-8ebf-4b6c-bfaf-71eaba2a6a58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d4167a9-92r25" Nov 03 00:53:54.070761 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:54.070733 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d4167a9-92r25" Nov 03 00:53:54.433328 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:54.432658 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d4167a9-92r25"] Nov 03 00:53:54.686385 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:54.686300 4421 patch_prober.go:29] interesting pod/e2e-worker-w3-83755a-openshift-default-c86f9565c-mlxvq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Nov 03 00:53:54.686385 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:54.686370 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-mlxvq" podUID="dcf063ce-88a5-4341-ae0f-605522bb91d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Nov 03 00:53:55.032076 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:55.032027 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d4167a9-92r25" event={"ID":"430ade2a-8ebf-4b6c-bfaf-71eaba2a6a58","Type":"ContainerStarted","Data":"fc5cce0c7d248ef8b3db78c5df2bf13d5126acc1cb7cb8a4c3c727c03fbc2d03"} Nov 03 00:53:55.032565 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:55.032112 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d4167a9-92r25" event={"ID":"430ade2a-8ebf-4b6c-bfaf-71eaba2a6a58","Type":"ContainerStarted","Data":"cfe2f684ac3d430d195933b75dfd70d4c1526d31dca41c72c861fe93fe170716"} Nov 03 00:53:55.050684 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:55.050625 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d4167a9-92r25" podStartSLOduration=2.050611161 podStartE2EDuration="2.050611161s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:53:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:53:55.048034765 +0000 UTC m=+777.219037861" watchObservedRunningTime="2026-11-03 00:53:55.050611161 +0000 UTC m=+777.221614256" Nov 03 00:53:55.686657 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:55.686573 4421 patch_prober.go:29] interesting pod/e2e-worker-w3-83755a-openshift-default-c86f9565c-mlxvq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Nov 03 00:53:55.686657 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:55.686623 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-mlxvq" podUID="dcf063ce-88a5-4341-ae0f-605522bb91d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Nov 03 00:53:56.041891 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:56.041851 4421 generic.go:356] "Generic (PLEG): container finished" podID="430ade2a-8ebf-4b6c-bfaf-71eaba2a6a58" containerID="fc5cce0c7d248ef8b3db78c5df2bf13d5126acc1cb7cb8a4c3c727c03fbc2d03" exitCode=0 Nov 03 00:53:56.042381 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:56.041957 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d4167a9-92r25" event={"ID":"430ade2a-8ebf-4b6c-bfaf-71eaba2a6a58","Type":"ContainerDied","Data":"fc5cce0c7d248ef8b3db78c5df2bf13d5126acc1cb7cb8a4c3c727c03fbc2d03"} Nov 03 00:53:56.718767 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:56.718735 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-mlxvq" Nov 03 00:53:56.719075 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:56.719057 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-mlxvq" Nov 03 00:53:56.721126 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:56.721105 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-mlxvq" Nov 03 00:53:57.196565 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:57.196535 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d4167a9-92r25" Nov 03 00:53:57.264754 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:57.264717 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/430ade2a-8ebf-4b6c-bfaf-71eaba2a6a58-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/430ade2a-8ebf-4b6c-bfaf-71eaba2a6a58-maas-api-xks-ca\") pod \"430ade2a-8ebf-4b6c-bfaf-71eaba2a6a58\" (UID: \"430ade2a-8ebf-4b6c-bfaf-71eaba2a6a58\") " Nov 03 00:53:57.264971 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:57.264837 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/430ade2a-8ebf-4b6c-bfaf-71eaba2a6a58-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/430ade2a-8ebf-4b6c-bfaf-71eaba2a6a58-maas-api-openshift-ca\") pod \"430ade2a-8ebf-4b6c-bfaf-71eaba2a6a58\" (UID: \"430ade2a-8ebf-4b6c-bfaf-71eaba2a6a58\") " Nov 03 00:53:57.265048 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:57.264997 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/430ade2a-8ebf-4b6c-bfaf-71eaba2a6a58-kube-api-access-ngjz5\" (UniqueName: \"kubernetes.io/projected/430ade2a-8ebf-4b6c-bfaf-71eaba2a6a58-kube-api-access-ngjz5\") pod \"430ade2a-8ebf-4b6c-bfaf-71eaba2a6a58\" (UID: \"430ade2a-8ebf-4b6c-bfaf-71eaba2a6a58\") " Nov 03 00:53:57.265420 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:57.265389 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/430ade2a-8ebf-4b6c-bfaf-71eaba2a6a58-maas-api-openshift-ca" pod "430ade2a-8ebf-4b6c-bfaf-71eaba2a6a58" (UID: "430ade2a-8ebf-4b6c-bfaf-71eaba2a6a58"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:53:57.265825 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:57.265786 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/430ade2a-8ebf-4b6c-bfaf-71eaba2a6a58-maas-api-openshift-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:53:57.267571 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:57.267544 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/430ade2a-8ebf-4b6c-bfaf-71eaba2a6a58-maas-api-xks-ca" pod "430ade2a-8ebf-4b6c-bfaf-71eaba2a6a58" (UID: "430ade2a-8ebf-4b6c-bfaf-71eaba2a6a58"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:53:57.267824 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:57.267792 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/430ade2a-8ebf-4b6c-bfaf-71eaba2a6a58-kube-api-access-ngjz5" pod "430ade2a-8ebf-4b6c-bfaf-71eaba2a6a58" (UID: "430ade2a-8ebf-4b6c-bfaf-71eaba2a6a58"). InnerVolumeSpecName "kube-api-access-ngjz5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:53:57.366631 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:57.366540 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ngjz5\" (UniqueName: \"kubernetes.io/projected/430ade2a-8ebf-4b6c-bfaf-71eaba2a6a58-kube-api-access-ngjz5\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:53:57.366631 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:57.366567 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/430ade2a-8ebf-4b6c-bfaf-71eaba2a6a58-maas-api-xks-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:53:58.057719 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:58.057690 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d4167a9-92r25" Nov 03 00:53:58.057904 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:58.057717 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d4167a9-92r25" event={"ID":"430ade2a-8ebf-4b6c-bfaf-71eaba2a6a58","Type":"ContainerDied","Data":"cfe2f684ac3d430d195933b75dfd70d4c1526d31dca41c72c861fe93fe170716"} Nov 03 00:53:58.057904 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:53:58.057795 4421 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cfe2f684ac3d430d195933b75dfd70d4c1526d31dca41c72c861fe93fe170716" Nov 03 00:54:00.321825 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:00.321789 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-fgwv5"] Nov 03 00:54:00.322644 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:00.322610 4421 cpu_manager.go:414] "containerMap: removing container" podUID="430ade2a-8ebf-4b6c-bfaf-71eaba2a6a58" containerName="revoke-keys" Nov 03 00:54:00.322776 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:00.322648 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="430ade2a-8ebf-4b6c-bfaf-71eaba2a6a58" containerName="revoke-keys" Nov 03 00:54:00.322896 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:00.322872 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="430ade2a-8ebf-4b6c-bfaf-71eaba2a6a58" containerName="revoke-keys" Nov 03 00:54:00.345092 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:00.345057 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-fgwv5" Nov 03 00:54:00.356219 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:00.356190 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-fgwv5"] Nov 03 00:54:00.506830 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:00.506796 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/126b600b-80c4-4377-a1f8-fd5936d40bf4-credential-socket\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-fgwv5\" (UID: \"126b600b-80c4-4377-a1f8-fd5936d40bf4\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-fgwv5" Nov 03 00:54:00.507023 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:00.506843 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/126b600b-80c4-4377-a1f8-fd5936d40bf4-istio-token\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-fgwv5\" (UID: \"126b600b-80c4-4377-a1f8-fd5936d40bf4\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-fgwv5" Nov 03 00:54:00.507087 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:00.507054 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/126b600b-80c4-4377-a1f8-fd5936d40bf4-workload-certs\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-fgwv5\" (UID: \"126b600b-80c4-4377-a1f8-fd5936d40bf4\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-fgwv5" Nov 03 00:54:00.507146 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:00.507111 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/126b600b-80c4-4377-a1f8-fd5936d40bf4-workload-socket\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-fgwv5\" (UID: \"126b600b-80c4-4377-a1f8-fd5936d40bf4\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-fgwv5" Nov 03 00:54:00.507218 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:00.507201 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/126b600b-80c4-4377-a1f8-fd5936d40bf4-istio-envoy\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-fgwv5\" (UID: \"126b600b-80c4-4377-a1f8-fd5936d40bf4\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-fgwv5" Nov 03 00:54:00.507257 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:00.507226 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/126b600b-80c4-4377-a1f8-fd5936d40bf4-istio-data\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-fgwv5\" (UID: \"126b600b-80c4-4377-a1f8-fd5936d40bf4\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-fgwv5" Nov 03 00:54:00.507354 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:00.507264 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/126b600b-80c4-4377-a1f8-fd5936d40bf4-istiod-ca-cert\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-fgwv5\" (UID: \"126b600b-80c4-4377-a1f8-fd5936d40bf4\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-fgwv5" Nov 03 00:54:00.507407 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:00.507395 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmxn8\" (UniqueName: \"kubernetes.io/projected/126b600b-80c4-4377-a1f8-fd5936d40bf4-kube-api-access-cmxn8\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-fgwv5\" (UID: \"126b600b-80c4-4377-a1f8-fd5936d40bf4\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-fgwv5" Nov 03 00:54:00.507468 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:00.507451 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/126b600b-80c4-4377-a1f8-fd5936d40bf4-istio-podinfo\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-fgwv5\" (UID: \"126b600b-80c4-4377-a1f8-fd5936d40bf4\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-fgwv5" Nov 03 00:54:00.609448 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:00.609351 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/126b600b-80c4-4377-a1f8-fd5936d40bf4-istio-podinfo\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-fgwv5\" (UID: \"126b600b-80c4-4377-a1f8-fd5936d40bf4\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-fgwv5" Nov 03 00:54:00.609448 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:00.609410 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/126b600b-80c4-4377-a1f8-fd5936d40bf4-credential-socket\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-fgwv5\" (UID: \"126b600b-80c4-4377-a1f8-fd5936d40bf4\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-fgwv5" Nov 03 00:54:00.609448 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:00.609442 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/126b600b-80c4-4377-a1f8-fd5936d40bf4-istio-token\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-fgwv5\" (UID: \"126b600b-80c4-4377-a1f8-fd5936d40bf4\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-fgwv5" Nov 03 00:54:00.609769 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:00.609547 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/126b600b-80c4-4377-a1f8-fd5936d40bf4-workload-certs\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-fgwv5\" (UID: \"126b600b-80c4-4377-a1f8-fd5936d40bf4\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-fgwv5" Nov 03 00:54:00.609769 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:00.609578 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/126b600b-80c4-4377-a1f8-fd5936d40bf4-workload-socket\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-fgwv5\" (UID: \"126b600b-80c4-4377-a1f8-fd5936d40bf4\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-fgwv5" Nov 03 00:54:00.609769 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:00.609626 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/126b600b-80c4-4377-a1f8-fd5936d40bf4-istio-envoy\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-fgwv5\" (UID: \"126b600b-80c4-4377-a1f8-fd5936d40bf4\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-fgwv5" Nov 03 00:54:00.609769 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:00.609648 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/126b600b-80c4-4377-a1f8-fd5936d40bf4-istio-data\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-fgwv5\" (UID: \"126b600b-80c4-4377-a1f8-fd5936d40bf4\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-fgwv5" Nov 03 00:54:00.609769 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:00.609681 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/126b600b-80c4-4377-a1f8-fd5936d40bf4-istiod-ca-cert\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-fgwv5\" (UID: \"126b600b-80c4-4377-a1f8-fd5936d40bf4\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-fgwv5" Nov 03 00:54:00.609769 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:00.609720 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmxn8\" (UniqueName: \"kubernetes.io/projected/126b600b-80c4-4377-a1f8-fd5936d40bf4-kube-api-access-cmxn8\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-fgwv5\" (UID: \"126b600b-80c4-4377-a1f8-fd5936d40bf4\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-fgwv5" Nov 03 00:54:00.611896 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:00.610985 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/126b600b-80c4-4377-a1f8-fd5936d40bf4-workload-socket\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-fgwv5\" (UID: \"126b600b-80c4-4377-a1f8-fd5936d40bf4\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-fgwv5" Nov 03 00:54:00.611896 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:00.611323 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/126b600b-80c4-4377-a1f8-fd5936d40bf4-credential-socket\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-fgwv5\" (UID: \"126b600b-80c4-4377-a1f8-fd5936d40bf4\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-fgwv5" Nov 03 00:54:00.611896 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:00.611572 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/126b600b-80c4-4377-a1f8-fd5936d40bf4-istio-data\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-fgwv5\" (UID: \"126b600b-80c4-4377-a1f8-fd5936d40bf4\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-fgwv5" Nov 03 00:54:00.611896 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:00.611853 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/126b600b-80c4-4377-a1f8-fd5936d40bf4-workload-certs\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-fgwv5\" (UID: \"126b600b-80c4-4377-a1f8-fd5936d40bf4\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-fgwv5" Nov 03 00:54:00.612437 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:00.612405 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/126b600b-80c4-4377-a1f8-fd5936d40bf4-istiod-ca-cert\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-fgwv5\" (UID: \"126b600b-80c4-4377-a1f8-fd5936d40bf4\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-fgwv5" Nov 03 00:54:00.614349 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:00.614322 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/126b600b-80c4-4377-a1f8-fd5936d40bf4-istio-envoy\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-fgwv5\" (UID: \"126b600b-80c4-4377-a1f8-fd5936d40bf4\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-fgwv5" Nov 03 00:54:00.614854 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:00.614813 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/126b600b-80c4-4377-a1f8-fd5936d40bf4-istio-podinfo\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-fgwv5\" (UID: \"126b600b-80c4-4377-a1f8-fd5936d40bf4\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-fgwv5" Nov 03 00:54:00.624212 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:00.624176 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/126b600b-80c4-4377-a1f8-fd5936d40bf4-istio-token\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-fgwv5\" (UID: \"126b600b-80c4-4377-a1f8-fd5936d40bf4\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-fgwv5" Nov 03 00:54:00.626086 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:00.626060 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmxn8\" (UniqueName: \"kubernetes.io/projected/126b600b-80c4-4377-a1f8-fd5936d40bf4-kube-api-access-cmxn8\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-fgwv5\" (UID: \"126b600b-80c4-4377-a1f8-fd5936d40bf4\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-fgwv5" Nov 03 00:54:00.674769 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:00.674727 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-fgwv5" Nov 03 00:54:00.971429 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:00.971402 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-fgwv5"] Nov 03 00:54:00.973257 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:54:00.973225 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod126b600b_80c4_4377_a1f8_fd5936d40bf4.slice/crio-cc2c7a4927155f4124ca05dfb2c4c4d0afb354f924f7e11bede43efa57d1c455 WatchSource:0}: Error finding container cc2c7a4927155f4124ca05dfb2c4c4d0afb354f924f7e11bede43efa57d1c455: Status 404 returned error can't find the container with id cc2c7a4927155f4124ca05dfb2c4c4d0afb354f924f7e11bede43efa57d1c455 Nov 03 00:54:00.976072 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:00.976029 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:54:00.976193 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:00.976123 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:54:00.976193 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:00.976172 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:54:01.096151 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:01.095055 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-fgwv5" event={"ID":"126b600b-80c4-4377-a1f8-fd5936d40bf4","Type":"ContainerStarted","Data":"7b20ee6ba3354f9d00feb66eec971a36f11fdd3f2ca069652dbfe3fb1b67714c"} Nov 03 00:54:01.096151 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:01.096030 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-fgwv5" event={"ID":"126b600b-80c4-4377-a1f8-fd5936d40bf4","Type":"ContainerStarted","Data":"cc2c7a4927155f4124ca05dfb2c4c4d0afb354f924f7e11bede43efa57d1c455"} Nov 03 00:54:01.132331 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:01.126561 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-fgwv5" podStartSLOduration=1.126543635 podStartE2EDuration="1.126543635s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:54:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:54:01.120060465 +0000 UTC m=+783.291063562" watchObservedRunningTime="2026-11-03 00:54:01.126543635 +0000 UTC m=+783.297546731" Nov 03 00:54:01.684299 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:01.683408 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-fgwv5" Nov 03 00:54:02.695463 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:02.695416 4421 patch_prober.go:29] interesting pod/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-fgwv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Nov 03 00:54:02.698602 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:02.698553 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-fgwv5" podUID="126b600b-80c4-4377-a1f8-fd5936d40bf4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Nov 03 00:54:02.890304 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:02.888382 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-650be0/simulator-w4-650be0-kserve-7bc588774f-gvz2k" podUID="a827fa88-f078-4312-8f12-2c799c11b461" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:54:02.890304 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:02.889684 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/simulator-w2-f7680d-kserve-85bdc86cd6-8gxrg" podUID="2bdcf521-cebd-4d6b-8554-1b60db8b0f3c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:54:02.899405 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:02.899363 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/simulator-w3-83755a-kserve-77d6664559-psqtg" podUID="b0a4c234-af58-4da0-a1a4-e1f167953f6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:54:02.945666 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:02.944377 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/distinct-w3-83755a-kserve-86bdbbdd85-2cnnl" podUID="f4ae9624-d127-4a55-837d-7c9ff0267b37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:54:02.945933 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:02.945901 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/premium-w3-83755a-kserve-7c9d66988b-lt4xd" podUID="c0c6b02e-2ecb-428e-a13d-be6d7cd1ba39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Nov 03 00:54:02.953909 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:02.953870 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-650be0/premium-w4-650be0-kserve-7bf8555dfb-z97n5" podUID="40601dc5-808f-456f-826f-6e0c35966ada" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:54:03.054495 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:03.054428 4421 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-650be0/simulator-w4-650be0-kserve-7bc588774f-gvz2k" podUID="a827fa88-f078-4312-8f12-2c799c11b461" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:54:03.190298 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:03.189367 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8gm8n" podUID="2d367e12-e46f-4a68-9825-9b1c934d4d12" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:54:03.681255 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:03.680761 4421 patch_prober.go:29] interesting pod/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-fgwv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Nov 03 00:54:03.681255 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:03.680833 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-fgwv5" podUID="126b600b-80c4-4377-a1f8-fd5936d40bf4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Nov 03 00:54:03.901301 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:03.900374 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-650be0/simulator-w4-650be0-kserve-7bc588774f-gvz2k" podUID="a827fa88-f078-4312-8f12-2c799c11b461" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Nov 03 00:54:03.901301 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:03.901187 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-650be0/simulator-w4-650be0-kserve-7bc588774f-gvz2k" Nov 03 00:54:03.920305 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:03.902764 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/simulator-w3-83755a-kserve-77d6664559-psqtg" podUID="b0a4c234-af58-4da0-a1a4-e1f167953f6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:54:03.920305 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:03.903084 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-83755a/simulator-w3-83755a-kserve-77d6664559-psqtg" Nov 03 00:54:03.952356 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:03.951374 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/premium-w3-83755a-kserve-7c9d66988b-lt4xd" podUID="c0c6b02e-2ecb-428e-a13d-be6d7cd1ba39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Nov 03 00:54:03.952356 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:03.951779 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-83755a/premium-w3-83755a-kserve-7c9d66988b-lt4xd" Nov 03 00:54:03.961401 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:03.960882 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-650be0/premium-w4-650be0-kserve-7bf8555dfb-z97n5" podUID="40601dc5-808f-456f-826f-6e0c35966ada" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Nov 03 00:54:03.975167 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:03.967104 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/distinct-w3-83755a-kserve-86bdbbdd85-2cnnl" podUID="f4ae9624-d127-4a55-837d-7c9ff0267b37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded" Nov 03 00:54:03.975167 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:03.967541 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-650be0/premium-w4-650be0-kserve-7bf8555dfb-z97n5" Nov 03 00:54:03.975167 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:03.967604 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-83755a/distinct-w3-83755a-kserve-86bdbbdd85-2cnnl" Nov 03 00:54:04.485835 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:04.378763 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8gm8n" podUID="2d367e12-e46f-4a68-9825-9b1c934d4d12" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": context deadline exceeded" Nov 03 00:54:04.521758 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:04.497367 4421 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-650be0/premium-w4-650be0-kserve-7bf8555dfb-z97n5" podUID="40601dc5-808f-456f-826f-6e0c35966ada" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:54:04.583054 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:04.583018 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8gm8n" Nov 03 00:54:04.587423 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:04.587386 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-83755a/distinct-w3-83755a-kserve-86bdbbdd85-2cnnl" Nov 03 00:54:04.878902 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:04.876396 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/simulator-w2-f7680d-kserve-85bdc86cd6-8gxrg" podUID="2bdcf521-cebd-4d6b-8554-1b60db8b0f3c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:54:04.896554 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:04.896517 4421 patch_prober.go:29] interesting pod/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-fgwv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Nov 03 00:54:04.896892 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:04.896856 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-fgwv5" podUID="126b600b-80c4-4377-a1f8-fd5936d40bf4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Nov 03 00:54:04.908495 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:04.902259 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-650be0/simulator-w4-650be0-kserve-7bc588774f-gvz2k" podUID="a827fa88-f078-4312-8f12-2c799c11b461" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:54:04.916300 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:04.908378 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/simulator-w3-83755a-kserve-77d6664559-psqtg" podUID="b0a4c234-af58-4da0-a1a4-e1f167953f6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:54:05.014229 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:05.014168 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/premium-w3-83755a-kserve-7c9d66988b-lt4xd" podUID="c0c6b02e-2ecb-428e-a13d-be6d7cd1ba39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Nov 03 00:54:05.019312 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:05.014589 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-650be0/premium-w4-650be0-kserve-7bf8555dfb-z97n5" podUID="40601dc5-808f-456f-826f-6e0c35966ada" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Nov 03 00:54:05.122299 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:05.122023 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-650be0/premium-w4-650be0-kserve-7bf8555dfb-z97n5" Nov 03 00:54:05.230230 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:05.230142 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-83755a/simulator-w3-83755a-kserve-77d6664559-psqtg" Nov 03 00:54:05.280013 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:05.279616 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8gm8n" Nov 03 00:54:05.280013 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:05.279699 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-83755a/premium-w3-83755a-kserve-7c9d66988b-lt4xd" Nov 03 00:54:05.301493 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:05.301460 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-650be0/simulator-w4-650be0-kserve-7bc588774f-gvz2k" Nov 03 00:54:05.675622 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:05.675575 4421 patch_prober.go:29] interesting pod/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-fgwv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Nov 03 00:54:05.675800 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:05.675647 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-fgwv5" podUID="126b600b-80c4-4377-a1f8-fd5936d40bf4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Nov 03 00:54:06.676031 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:06.675996 4421 patch_prober.go:29] interesting pod/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-fgwv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Nov 03 00:54:06.676448 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:06.676056 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-fgwv5" podUID="126b600b-80c4-4377-a1f8-fd5936d40bf4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Nov 03 00:54:06.755213 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:06.755165 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1866d6d9-lr7c7"] Nov 03 00:54:06.764476 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:06.764440 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1866d6d9-lr7c7" Nov 03 00:54:06.766559 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:06.766530 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1866d6d9-lr7c7"] Nov 03 00:54:06.771941 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:06.771914 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 00:54:06.772758 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:06.772731 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 00:54:06.772888 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:06.772778 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:54:06.772888 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:06.772789 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6q4xp\"" Nov 03 00:54:06.863875 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:06.863830 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5mxs\" (UniqueName: \"kubernetes.io/projected/91b01607-71bd-4c79-805d-1437b55de97f-kube-api-access-s5mxs\") pod \"maas-api-revoke-sub-1866d6d9-lr7c7\" (UID: \"91b01607-71bd-4c79-805d-1437b55de97f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1866d6d9-lr7c7" Nov 03 00:54:06.863875 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:06.863876 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91b01607-71bd-4c79-805d-1437b55de97f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1866d6d9-lr7c7\" (UID: \"91b01607-71bd-4c79-805d-1437b55de97f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1866d6d9-lr7c7" Nov 03 00:54:06.864102 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:06.863974 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91b01607-71bd-4c79-805d-1437b55de97f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1866d6d9-lr7c7\" (UID: \"91b01607-71bd-4c79-805d-1437b55de97f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1866d6d9-lr7c7" Nov 03 00:54:06.965718 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:06.965626 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91b01607-71bd-4c79-805d-1437b55de97f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1866d6d9-lr7c7\" (UID: \"91b01607-71bd-4c79-805d-1437b55de97f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1866d6d9-lr7c7" Nov 03 00:54:06.965945 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:06.965919 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5mxs\" (UniqueName: \"kubernetes.io/projected/91b01607-71bd-4c79-805d-1437b55de97f-kube-api-access-s5mxs\") pod \"maas-api-revoke-sub-1866d6d9-lr7c7\" (UID: \"91b01607-71bd-4c79-805d-1437b55de97f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1866d6d9-lr7c7" Nov 03 00:54:06.966017 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:06.965965 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91b01607-71bd-4c79-805d-1437b55de97f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1866d6d9-lr7c7\" (UID: \"91b01607-71bd-4c79-805d-1437b55de97f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1866d6d9-lr7c7" Nov 03 00:54:06.966694 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:06.966672 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91b01607-71bd-4c79-805d-1437b55de97f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1866d6d9-lr7c7\" (UID: \"91b01607-71bd-4c79-805d-1437b55de97f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1866d6d9-lr7c7" Nov 03 00:54:06.968016 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:06.967989 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91b01607-71bd-4c79-805d-1437b55de97f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1866d6d9-lr7c7\" (UID: \"91b01607-71bd-4c79-805d-1437b55de97f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1866d6d9-lr7c7" Nov 03 00:54:06.974345 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:06.974324 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5mxs\" (UniqueName: \"kubernetes.io/projected/91b01607-71bd-4c79-805d-1437b55de97f-kube-api-access-s5mxs\") pod \"maas-api-revoke-sub-1866d6d9-lr7c7\" (UID: \"91b01607-71bd-4c79-805d-1437b55de97f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1866d6d9-lr7c7" Nov 03 00:54:07.084239 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:07.084209 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1866d6d9-lr7c7" Nov 03 00:54:07.433640 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:54:07.433606 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91b01607_71bd_4c79_805d_1437b55de97f.slice/crio-37f80d743212748daa5e095c42e9a83d4c004e9670d523e9cfa7815e71b09c75 WatchSource:0}: Error finding container 37f80d743212748daa5e095c42e9a83d4c004e9670d523e9cfa7815e71b09c75: Status 404 returned error can't find the container with id 37f80d743212748daa5e095c42e9a83d4c004e9670d523e9cfa7815e71b09c75 Nov 03 00:54:07.434398 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:07.434372 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1866d6d9-lr7c7"] Nov 03 00:54:07.675437 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:07.675404 4421 patch_prober.go:29] interesting pod/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-fgwv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Nov 03 00:54:07.675596 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:07.675458 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-fgwv5" podUID="126b600b-80c4-4377-a1f8-fd5936d40bf4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Nov 03 00:54:08.236904 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:08.236822 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f372f7f-9zxcl"] Nov 03 00:54:08.241065 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:08.241047 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f372f7f-9zxcl" Nov 03 00:54:08.251870 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:08.251838 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f372f7f-9zxcl"] Nov 03 00:54:08.288388 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:08.288351 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c855b88-d5c6-4c09-b6e7-3685dd1e57a1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7f372f7f-9zxcl\" (UID: \"6c855b88-d5c6-4c09-b6e7-3685dd1e57a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f372f7f-9zxcl" Nov 03 00:54:08.288586 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:08.288476 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c855b88-d5c6-4c09-b6e7-3685dd1e57a1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7f372f7f-9zxcl\" (UID: \"6c855b88-d5c6-4c09-b6e7-3685dd1e57a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f372f7f-9zxcl" Nov 03 00:54:08.288586 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:08.288552 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5m9g\" (UniqueName: \"kubernetes.io/projected/6c855b88-d5c6-4c09-b6e7-3685dd1e57a1-kube-api-access-h5m9g\") pod \"maas-api-revoke-sub-7f372f7f-9zxcl\" (UID: \"6c855b88-d5c6-4c09-b6e7-3685dd1e57a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f372f7f-9zxcl" Nov 03 00:54:08.373596 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:08.373559 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1866d6d9-lr7c7" event={"ID":"91b01607-71bd-4c79-805d-1437b55de97f","Type":"ContainerStarted","Data":"b4262ea71982eb2994361392030856f81abb26f3ed7a421901b4fe285eab25e2"} Nov 03 00:54:08.373784 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:08.373623 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1866d6d9-lr7c7" event={"ID":"91b01607-71bd-4c79-805d-1437b55de97f","Type":"ContainerStarted","Data":"37f80d743212748daa5e095c42e9a83d4c004e9670d523e9cfa7815e71b09c75"} Nov 03 00:54:08.389486 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:08.389333 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c855b88-d5c6-4c09-b6e7-3685dd1e57a1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7f372f7f-9zxcl\" (UID: \"6c855b88-d5c6-4c09-b6e7-3685dd1e57a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f372f7f-9zxcl" Nov 03 00:54:08.389486 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:08.389468 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5m9g\" (UniqueName: \"kubernetes.io/projected/6c855b88-d5c6-4c09-b6e7-3685dd1e57a1-kube-api-access-h5m9g\") pod \"maas-api-revoke-sub-7f372f7f-9zxcl\" (UID: \"6c855b88-d5c6-4c09-b6e7-3685dd1e57a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f372f7f-9zxcl" Nov 03 00:54:08.389739 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:08.389640 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c855b88-d5c6-4c09-b6e7-3685dd1e57a1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7f372f7f-9zxcl\" (UID: \"6c855b88-d5c6-4c09-b6e7-3685dd1e57a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f372f7f-9zxcl" Nov 03 00:54:08.390389 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:08.390352 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c855b88-d5c6-4c09-b6e7-3685dd1e57a1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7f372f7f-9zxcl\" (UID: \"6c855b88-d5c6-4c09-b6e7-3685dd1e57a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f372f7f-9zxcl" Nov 03 00:54:08.391468 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:08.391427 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1866d6d9-lr7c7" podStartSLOduration=2.391415909 podStartE2EDuration="2.391415909s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:54:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:54:08.390624193 +0000 UTC m=+790.561627290" watchObservedRunningTime="2026-11-03 00:54:08.391415909 +0000 UTC m=+790.562419005" Nov 03 00:54:08.393016 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:08.392995 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c855b88-d5c6-4c09-b6e7-3685dd1e57a1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7f372f7f-9zxcl\" (UID: \"6c855b88-d5c6-4c09-b6e7-3685dd1e57a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f372f7f-9zxcl" Nov 03 00:54:08.398548 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:08.398527 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5m9g\" (UniqueName: \"kubernetes.io/projected/6c855b88-d5c6-4c09-b6e7-3685dd1e57a1-kube-api-access-h5m9g\") pod \"maas-api-revoke-sub-7f372f7f-9zxcl\" (UID: \"6c855b88-d5c6-4c09-b6e7-3685dd1e57a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f372f7f-9zxcl" Nov 03 00:54:08.559469 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:08.559436 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f372f7f-9zxcl" Nov 03 00:54:08.677327 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:08.676660 4421 patch_prober.go:29] interesting pod/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-fgwv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Nov 03 00:54:08.677327 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:08.676716 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-fgwv5" podUID="126b600b-80c4-4377-a1f8-fd5936d40bf4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Nov 03 00:54:08.954795 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:08.951016 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f372f7f-9zxcl"] Nov 03 00:54:09.383513 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:09.383468 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f372f7f-9zxcl" event={"ID":"6c855b88-d5c6-4c09-b6e7-3685dd1e57a1","Type":"ContainerStarted","Data":"c18ca04f739c8d3eb074c3d93d861e154b33608599777a18138f396d1075b4af"} Nov 03 00:54:09.385413 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:09.385385 4421 generic.go:356] "Generic (PLEG): container finished" podID="91b01607-71bd-4c79-805d-1437b55de97f" containerID="b4262ea71982eb2994361392030856f81abb26f3ed7a421901b4fe285eab25e2" exitCode=0 Nov 03 00:54:09.385541 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:09.385456 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1866d6d9-lr7c7" event={"ID":"91b01607-71bd-4c79-805d-1437b55de97f","Type":"ContainerDied","Data":"b4262ea71982eb2994361392030856f81abb26f3ed7a421901b4fe285eab25e2"} Nov 03 00:54:09.676628 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:09.676592 4421 patch_prober.go:29] interesting pod/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-fgwv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Nov 03 00:54:09.676781 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:09.676646 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-fgwv5" podUID="126b600b-80c4-4377-a1f8-fd5936d40bf4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Nov 03 00:54:10.396506 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:10.396417 4421 generic.go:356] "Generic (PLEG): container finished" podID="6c855b88-d5c6-4c09-b6e7-3685dd1e57a1" containerID="b253436f1a8653c42495e7ebb7616b92eaa7886cb6ca1155eb942193f785e209" exitCode=0 Nov 03 00:54:10.396899 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:10.396509 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f372f7f-9zxcl" event={"ID":"6c855b88-d5c6-4c09-b6e7-3685dd1e57a1","Type":"ContainerDied","Data":"b253436f1a8653c42495e7ebb7616b92eaa7886cb6ca1155eb942193f785e209"} Nov 03 00:54:10.581956 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:10.581921 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1866d6d9-lr7c7" Nov 03 00:54:10.618184 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:10.618149 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91b01607-71bd-4c79-805d-1437b55de97f-kube-api-access-s5mxs\" (UniqueName: \"kubernetes.io/projected/91b01607-71bd-4c79-805d-1437b55de97f-kube-api-access-s5mxs\") pod \"91b01607-71bd-4c79-805d-1437b55de97f\" (UID: \"91b01607-71bd-4c79-805d-1437b55de97f\") " Nov 03 00:54:10.618392 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:10.618210 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/91b01607-71bd-4c79-805d-1437b55de97f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91b01607-71bd-4c79-805d-1437b55de97f-maas-api-xks-ca\") pod \"91b01607-71bd-4c79-805d-1437b55de97f\" (UID: \"91b01607-71bd-4c79-805d-1437b55de97f\") " Nov 03 00:54:10.618392 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:10.618258 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91b01607-71bd-4c79-805d-1437b55de97f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91b01607-71bd-4c79-805d-1437b55de97f-maas-api-openshift-ca\") pod \"91b01607-71bd-4c79-805d-1437b55de97f\" (UID: \"91b01607-71bd-4c79-805d-1437b55de97f\") " Nov 03 00:54:10.619010 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:10.618974 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91b01607-71bd-4c79-805d-1437b55de97f-maas-api-openshift-ca" pod "91b01607-71bd-4c79-805d-1437b55de97f" (UID: "91b01607-71bd-4c79-805d-1437b55de97f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:54:10.627317 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:10.620960 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91b01607-71bd-4c79-805d-1437b55de97f-kube-api-access-s5mxs" pod "91b01607-71bd-4c79-805d-1437b55de97f" (UID: "91b01607-71bd-4c79-805d-1437b55de97f"). InnerVolumeSpecName "kube-api-access-s5mxs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:54:10.627317 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:10.621194 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91b01607-71bd-4c79-805d-1437b55de97f-maas-api-xks-ca" pod "91b01607-71bd-4c79-805d-1437b55de97f" (UID: "91b01607-71bd-4c79-805d-1437b55de97f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:54:10.720230 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:10.720134 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5mxs\" (UniqueName: \"kubernetes.io/projected/91b01607-71bd-4c79-805d-1437b55de97f-kube-api-access-s5mxs\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:54:10.720230 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:10.720171 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91b01607-71bd-4c79-805d-1437b55de97f-maas-api-xks-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:54:10.720230 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:10.720182 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91b01607-71bd-4c79-805d-1437b55de97f-maas-api-openshift-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:54:10.726583 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:10.726547 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-fgwv5" Nov 03 00:54:10.726860 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:10.726831 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-fgwv5" Nov 03 00:54:10.727708 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:10.727689 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-fgwv5" Nov 03 00:54:11.404922 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:11.404885 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1866d6d9-lr7c7" Nov 03 00:54:11.404922 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:11.404916 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1866d6d9-lr7c7" event={"ID":"91b01607-71bd-4c79-805d-1437b55de97f","Type":"ContainerDied","Data":"37f80d743212748daa5e095c42e9a83d4c004e9670d523e9cfa7815e71b09c75"} Nov 03 00:54:11.405456 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:11.404968 4421 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="37f80d743212748daa5e095c42e9a83d4c004e9670d523e9cfa7815e71b09c75" Nov 03 00:54:11.564725 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:11.564690 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f372f7f-9zxcl" Nov 03 00:54:11.634962 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:11.634924 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6c855b88-d5c6-4c09-b6e7-3685dd1e57a1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c855b88-d5c6-4c09-b6e7-3685dd1e57a1-maas-api-xks-ca\") pod \"6c855b88-d5c6-4c09-b6e7-3685dd1e57a1\" (UID: \"6c855b88-d5c6-4c09-b6e7-3685dd1e57a1\") " Nov 03 00:54:11.635323 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:11.635260 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c855b88-d5c6-4c09-b6e7-3685dd1e57a1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c855b88-d5c6-4c09-b6e7-3685dd1e57a1-maas-api-openshift-ca\") pod \"6c855b88-d5c6-4c09-b6e7-3685dd1e57a1\" (UID: \"6c855b88-d5c6-4c09-b6e7-3685dd1e57a1\") " Nov 03 00:54:11.635444 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:11.635410 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c855b88-d5c6-4c09-b6e7-3685dd1e57a1-kube-api-access-h5m9g\" (UniqueName: \"kubernetes.io/projected/6c855b88-d5c6-4c09-b6e7-3685dd1e57a1-kube-api-access-h5m9g\") pod \"6c855b88-d5c6-4c09-b6e7-3685dd1e57a1\" (UID: \"6c855b88-d5c6-4c09-b6e7-3685dd1e57a1\") " Nov 03 00:54:11.635837 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:11.635810 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c855b88-d5c6-4c09-b6e7-3685dd1e57a1-maas-api-openshift-ca" pod "6c855b88-d5c6-4c09-b6e7-3685dd1e57a1" (UID: "6c855b88-d5c6-4c09-b6e7-3685dd1e57a1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:54:11.641760 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:11.641718 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c855b88-d5c6-4c09-b6e7-3685dd1e57a1-maas-api-xks-ca" pod "6c855b88-d5c6-4c09-b6e7-3685dd1e57a1" (UID: "6c855b88-d5c6-4c09-b6e7-3685dd1e57a1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:54:11.656739 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:11.656649 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c855b88-d5c6-4c09-b6e7-3685dd1e57a1-kube-api-access-h5m9g" pod "6c855b88-d5c6-4c09-b6e7-3685dd1e57a1" (UID: "6c855b88-d5c6-4c09-b6e7-3685dd1e57a1"). InnerVolumeSpecName "kube-api-access-h5m9g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:54:11.737569 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:11.737535 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c855b88-d5c6-4c09-b6e7-3685dd1e57a1-maas-api-xks-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:54:11.737569 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:11.737564 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c855b88-d5c6-4c09-b6e7-3685dd1e57a1-maas-api-openshift-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:54:11.737569 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:11.737574 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5m9g\" (UniqueName: \"kubernetes.io/projected/6c855b88-d5c6-4c09-b6e7-3685dd1e57a1-kube-api-access-h5m9g\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:54:11.944247 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:11.944158 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-r6t2p"] Nov 03 00:54:11.944761 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:11.944745 4421 cpu_manager.go:414] "containerMap: removing container" podUID="91b01607-71bd-4c79-805d-1437b55de97f" containerName="revoke-keys" Nov 03 00:54:11.944822 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:11.944762 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91b01607-71bd-4c79-805d-1437b55de97f" containerName="revoke-keys" Nov 03 00:54:11.944822 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:11.944785 4421 cpu_manager.go:414] "containerMap: removing container" podUID="6c855b88-d5c6-4c09-b6e7-3685dd1e57a1" containerName="revoke-keys" Nov 03 00:54:11.944822 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:11.944790 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c855b88-d5c6-4c09-b6e7-3685dd1e57a1" containerName="revoke-keys" Nov 03 00:54:11.944931 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:11.944874 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c855b88-d5c6-4c09-b6e7-3685dd1e57a1" containerName="revoke-keys" Nov 03 00:54:11.944931 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:11.944888 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="91b01607-71bd-4c79-805d-1437b55de97f" containerName="revoke-keys" Nov 03 00:54:11.948512 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:11.948478 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-r6t2p" Nov 03 00:54:11.966507 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:11.966482 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-r6t2p"] Nov 03 00:54:12.041034 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:12.041000 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/415fa10b-db0c-4ded-aedb-582612cbde2c-istio-envoy\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-r6t2p\" (UID: \"415fa10b-db0c-4ded-aedb-582612cbde2c\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-r6t2p" Nov 03 00:54:12.041034 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:12.041036 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzm6n\" (UniqueName: \"kubernetes.io/projected/415fa10b-db0c-4ded-aedb-582612cbde2c-kube-api-access-nzm6n\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-r6t2p\" (UID: \"415fa10b-db0c-4ded-aedb-582612cbde2c\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-r6t2p" Nov 03 00:54:12.041242 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:12.041069 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/415fa10b-db0c-4ded-aedb-582612cbde2c-workload-certs\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-r6t2p\" (UID: \"415fa10b-db0c-4ded-aedb-582612cbde2c\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-r6t2p" Nov 03 00:54:12.041242 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:12.041204 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/415fa10b-db0c-4ded-aedb-582612cbde2c-istio-data\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-r6t2p\" (UID: \"415fa10b-db0c-4ded-aedb-582612cbde2c\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-r6t2p" Nov 03 00:54:12.041348 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:12.041316 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/415fa10b-db0c-4ded-aedb-582612cbde2c-workload-socket\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-r6t2p\" (UID: \"415fa10b-db0c-4ded-aedb-582612cbde2c\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-r6t2p" Nov 03 00:54:12.041348 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:12.041339 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/415fa10b-db0c-4ded-aedb-582612cbde2c-istio-podinfo\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-r6t2p\" (UID: \"415fa10b-db0c-4ded-aedb-582612cbde2c\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-r6t2p" Nov 03 00:54:12.041423 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:12.041412 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/415fa10b-db0c-4ded-aedb-582612cbde2c-credential-socket\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-r6t2p\" (UID: \"415fa10b-db0c-4ded-aedb-582612cbde2c\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-r6t2p" Nov 03 00:54:12.041464 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:12.041431 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/415fa10b-db0c-4ded-aedb-582612cbde2c-istio-token\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-r6t2p\" (UID: \"415fa10b-db0c-4ded-aedb-582612cbde2c\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-r6t2p" Nov 03 00:54:12.041497 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:12.041482 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/415fa10b-db0c-4ded-aedb-582612cbde2c-istiod-ca-cert\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-r6t2p\" (UID: \"415fa10b-db0c-4ded-aedb-582612cbde2c\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-r6t2p" Nov 03 00:54:12.143117 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:12.143084 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/415fa10b-db0c-4ded-aedb-582612cbde2c-istio-envoy\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-r6t2p\" (UID: \"415fa10b-db0c-4ded-aedb-582612cbde2c\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-r6t2p" Nov 03 00:54:12.143117 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:12.143118 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzm6n\" (UniqueName: \"kubernetes.io/projected/415fa10b-db0c-4ded-aedb-582612cbde2c-kube-api-access-nzm6n\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-r6t2p\" (UID: \"415fa10b-db0c-4ded-aedb-582612cbde2c\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-r6t2p" Nov 03 00:54:12.143351 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:12.143140 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/415fa10b-db0c-4ded-aedb-582612cbde2c-workload-certs\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-r6t2p\" (UID: \"415fa10b-db0c-4ded-aedb-582612cbde2c\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-r6t2p" Nov 03 00:54:12.143351 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:12.143203 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/415fa10b-db0c-4ded-aedb-582612cbde2c-istio-data\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-r6t2p\" (UID: \"415fa10b-db0c-4ded-aedb-582612cbde2c\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-r6t2p" Nov 03 00:54:12.143351 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:12.143322 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/415fa10b-db0c-4ded-aedb-582612cbde2c-workload-socket\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-r6t2p\" (UID: \"415fa10b-db0c-4ded-aedb-582612cbde2c\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-r6t2p" Nov 03 00:54:12.143351 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:12.143348 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/415fa10b-db0c-4ded-aedb-582612cbde2c-istio-podinfo\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-r6t2p\" (UID: \"415fa10b-db0c-4ded-aedb-582612cbde2c\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-r6t2p" Nov 03 00:54:12.143533 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:12.143417 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/415fa10b-db0c-4ded-aedb-582612cbde2c-credential-socket\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-r6t2p\" (UID: \"415fa10b-db0c-4ded-aedb-582612cbde2c\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-r6t2p" Nov 03 00:54:12.143533 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:12.143447 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/415fa10b-db0c-4ded-aedb-582612cbde2c-istio-token\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-r6t2p\" (UID: \"415fa10b-db0c-4ded-aedb-582612cbde2c\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-r6t2p" Nov 03 00:54:12.143533 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:12.143500 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/415fa10b-db0c-4ded-aedb-582612cbde2c-istiod-ca-cert\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-r6t2p\" (UID: \"415fa10b-db0c-4ded-aedb-582612cbde2c\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-r6t2p" Nov 03 00:54:12.143689 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:12.143629 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/415fa10b-db0c-4ded-aedb-582612cbde2c-workload-certs\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-r6t2p\" (UID: \"415fa10b-db0c-4ded-aedb-582612cbde2c\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-r6t2p" Nov 03 00:54:12.143689 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:12.143658 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/415fa10b-db0c-4ded-aedb-582612cbde2c-istio-data\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-r6t2p\" (UID: \"415fa10b-db0c-4ded-aedb-582612cbde2c\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-r6t2p" Nov 03 00:54:12.143796 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:12.143764 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/415fa10b-db0c-4ded-aedb-582612cbde2c-credential-socket\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-r6t2p\" (UID: \"415fa10b-db0c-4ded-aedb-582612cbde2c\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-r6t2p" Nov 03 00:54:12.143854 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:12.143835 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/415fa10b-db0c-4ded-aedb-582612cbde2c-workload-socket\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-r6t2p\" (UID: \"415fa10b-db0c-4ded-aedb-582612cbde2c\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-r6t2p" Nov 03 00:54:12.144152 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:12.144134 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/415fa10b-db0c-4ded-aedb-582612cbde2c-istiod-ca-cert\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-r6t2p\" (UID: \"415fa10b-db0c-4ded-aedb-582612cbde2c\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-r6t2p" Nov 03 00:54:12.145564 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:12.145546 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/415fa10b-db0c-4ded-aedb-582612cbde2c-istio-envoy\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-r6t2p\" (UID: \"415fa10b-db0c-4ded-aedb-582612cbde2c\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-r6t2p" Nov 03 00:54:12.145876 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:12.145854 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/415fa10b-db0c-4ded-aedb-582612cbde2c-istio-podinfo\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-r6t2p\" (UID: \"415fa10b-db0c-4ded-aedb-582612cbde2c\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-r6t2p" Nov 03 00:54:12.151225 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:12.151195 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzm6n\" (UniqueName: \"kubernetes.io/projected/415fa10b-db0c-4ded-aedb-582612cbde2c-kube-api-access-nzm6n\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-r6t2p\" (UID: \"415fa10b-db0c-4ded-aedb-582612cbde2c\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-r6t2p" Nov 03 00:54:12.151381 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:12.151364 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/415fa10b-db0c-4ded-aedb-582612cbde2c-istio-token\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-r6t2p\" (UID: \"415fa10b-db0c-4ded-aedb-582612cbde2c\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-r6t2p" Nov 03 00:54:12.268788 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:12.268743 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-r6t2p" Nov 03 00:54:12.417548 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:12.415034 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f372f7f-9zxcl" Nov 03 00:54:12.417548 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:12.415957 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f372f7f-9zxcl" event={"ID":"6c855b88-d5c6-4c09-b6e7-3685dd1e57a1","Type":"ContainerDied","Data":"c18ca04f739c8d3eb074c3d93d861e154b33608599777a18138f396d1075b4af"} Nov 03 00:54:12.417548 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:12.416011 4421 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c18ca04f739c8d3eb074c3d93d861e154b33608599777a18138f396d1075b4af" Nov 03 00:54:12.560333 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:12.560299 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-r6t2p"] Nov 03 00:54:12.560558 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:54:12.560535 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod415fa10b_db0c_4ded_aedb_582612cbde2c.slice/crio-b44de0085e4e176ea05ecc2b81c25fb1de526b95bfe2105bb70a5306b23160d4 WatchSource:0}: Error finding container b44de0085e4e176ea05ecc2b81c25fb1de526b95bfe2105bb70a5306b23160d4: Status 404 returned error can't find the container with id b44de0085e4e176ea05ecc2b81c25fb1de526b95bfe2105bb70a5306b23160d4 Nov 03 00:54:12.563600 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:12.563556 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:54:12.563750 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:12.563645 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:54:12.563750 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:12.563694 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:54:13.422335 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:13.422298 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-r6t2p" event={"ID":"415fa10b-db0c-4ded-aedb-582612cbde2c","Type":"ContainerStarted","Data":"978bb67569194ec8835b6dbf6bd9aeb3af47eda4ceba67c6d19e125e1e57ff30"} Nov 03 00:54:13.422743 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:13.422362 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-r6t2p" event={"ID":"415fa10b-db0c-4ded-aedb-582612cbde2c","Type":"ContainerStarted","Data":"b44de0085e4e176ea05ecc2b81c25fb1de526b95bfe2105bb70a5306b23160d4"} Nov 03 00:54:13.445620 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:13.445567 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-r6t2p" podStartSLOduration=2.4455545 podStartE2EDuration="2.4455545s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:54:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:54:13.441916731 +0000 UTC m=+795.612919855" watchObservedRunningTime="2026-11-03 00:54:13.4455545 +0000 UTC m=+795.616557596" Nov 03 00:54:14.269005 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:14.268969 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-r6t2p" Nov 03 00:54:14.270524 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:14.270495 4421 patch_prober.go:29] interesting pod/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-r6t2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Nov 03 00:54:14.270639 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:14.270570 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-r6t2p" podUID="415fa10b-db0c-4ded-aedb-582612cbde2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Nov 03 00:54:14.669401 ip-10-0-133-217 kubenswrapper[4421]: E1103 00:54:14.669318 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 00:54:15.269556 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:15.269519 4421 patch_prober.go:29] interesting pod/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-r6t2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Nov 03 00:54:15.269752 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:15.269575 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-r6t2p" podUID="415fa10b-db0c-4ded-aedb-582612cbde2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Nov 03 00:54:16.269331 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:16.269266 4421 patch_prober.go:29] interesting pod/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-r6t2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Nov 03 00:54:16.269731 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:16.269345 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-r6t2p" podUID="415fa10b-db0c-4ded-aedb-582612cbde2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Nov 03 00:54:17.269360 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:17.269320 4421 patch_prober.go:29] interesting pod/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-r6t2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Nov 03 00:54:17.269785 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:17.269388 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-r6t2p" podUID="415fa10b-db0c-4ded-aedb-582612cbde2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Nov 03 00:54:18.270114 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:18.270078 4421 patch_prober.go:29] interesting pod/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-r6t2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Nov 03 00:54:18.270539 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:18.270135 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-r6t2p" podUID="415fa10b-db0c-4ded-aedb-582612cbde2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Nov 03 00:54:19.270296 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:19.269619 4421 patch_prober.go:29] interesting pod/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-r6t2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Nov 03 00:54:19.270296 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:19.269683 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-r6t2p" podUID="415fa10b-db0c-4ded-aedb-582612cbde2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Nov 03 00:54:20.270862 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:20.270821 4421 patch_prober.go:29] interesting pod/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-r6t2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Nov 03 00:54:20.271547 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:20.271513 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-r6t2p" podUID="415fa10b-db0c-4ded-aedb-582612cbde2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Nov 03 00:54:21.274062 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:21.274029 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-r6t2p" Nov 03 00:54:21.274572 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:21.274448 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-r6t2p" Nov 03 00:54:21.275300 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:21.275251 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-r6t2p" Nov 03 00:54:22.571176 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:22.571140 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2de4fbe5-gpm2r"] Nov 03 00:54:22.577333 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:22.577305 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2de4fbe5-gpm2r" Nov 03 00:54:22.583201 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:22.583172 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:54:22.584381 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:22.584102 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 00:54:22.584569 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:22.584198 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 00:54:22.584669 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:22.584334 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6q4xp\"" Nov 03 00:54:22.587006 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:22.586975 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2de4fbe5-gpm2r"] Nov 03 00:54:22.685554 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:22.685513 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc8f0e43-471d-4339-a877-719e2b5061fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2de4fbe5-gpm2r\" (UID: \"cc8f0e43-471d-4339-a877-719e2b5061fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2de4fbe5-gpm2r" Nov 03 00:54:22.685847 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:22.685814 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lqjh\" (UniqueName: \"kubernetes.io/projected/cc8f0e43-471d-4339-a877-719e2b5061fa-kube-api-access-6lqjh\") pod \"maas-api-revoke-sub-2de4fbe5-gpm2r\" (UID: \"cc8f0e43-471d-4339-a877-719e2b5061fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2de4fbe5-gpm2r" Nov 03 00:54:22.685981 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:22.685946 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc8f0e43-471d-4339-a877-719e2b5061fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2de4fbe5-gpm2r\" (UID: \"cc8f0e43-471d-4339-a877-719e2b5061fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2de4fbe5-gpm2r" Nov 03 00:54:22.787515 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:22.787481 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc8f0e43-471d-4339-a877-719e2b5061fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2de4fbe5-gpm2r\" (UID: \"cc8f0e43-471d-4339-a877-719e2b5061fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2de4fbe5-gpm2r" Nov 03 00:54:22.787696 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:22.787651 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lqjh\" (UniqueName: \"kubernetes.io/projected/cc8f0e43-471d-4339-a877-719e2b5061fa-kube-api-access-6lqjh\") pod \"maas-api-revoke-sub-2de4fbe5-gpm2r\" (UID: \"cc8f0e43-471d-4339-a877-719e2b5061fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2de4fbe5-gpm2r" Nov 03 00:54:22.787696 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:22.787686 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc8f0e43-471d-4339-a877-719e2b5061fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2de4fbe5-gpm2r\" (UID: \"cc8f0e43-471d-4339-a877-719e2b5061fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2de4fbe5-gpm2r" Nov 03 00:54:22.788302 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:22.788284 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc8f0e43-471d-4339-a877-719e2b5061fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2de4fbe5-gpm2r\" (UID: \"cc8f0e43-471d-4339-a877-719e2b5061fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2de4fbe5-gpm2r" Nov 03 00:54:22.789845 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:22.789825 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc8f0e43-471d-4339-a877-719e2b5061fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2de4fbe5-gpm2r\" (UID: \"cc8f0e43-471d-4339-a877-719e2b5061fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2de4fbe5-gpm2r" Nov 03 00:54:22.802073 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:22.802041 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lqjh\" (UniqueName: \"kubernetes.io/projected/cc8f0e43-471d-4339-a877-719e2b5061fa-kube-api-access-6lqjh\") pod \"maas-api-revoke-sub-2de4fbe5-gpm2r\" (UID: \"cc8f0e43-471d-4339-a877-719e2b5061fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2de4fbe5-gpm2r" Nov 03 00:54:22.900958 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:22.900864 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2de4fbe5-gpm2r" Nov 03 00:54:23.193682 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:23.193641 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2de4fbe5-gpm2r"] Nov 03 00:54:23.194038 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:54:23.194016 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc8f0e43_471d_4339_a877_719e2b5061fa.slice/crio-bd5786b40327f81293c31dda10d814e646117f25140d4b271a31d3dfea9286e4 WatchSource:0}: Error finding container bd5786b40327f81293c31dda10d814e646117f25140d4b271a31d3dfea9286e4: Status 404 returned error can't find the container with id bd5786b40327f81293c31dda10d814e646117f25140d4b271a31d3dfea9286e4 Nov 03 00:54:23.552970 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:23.552933 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2de4fbe5-gpm2r" event={"ID":"cc8f0e43-471d-4339-a877-719e2b5061fa","Type":"ContainerStarted","Data":"bd5786b40327f81293c31dda10d814e646117f25140d4b271a31d3dfea9286e4"} Nov 03 00:54:24.561864 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:24.561816 4421 generic.go:356] "Generic (PLEG): container finished" podID="cc8f0e43-471d-4339-a877-719e2b5061fa" containerID="0c2310530b2a0f532ad97b7881ff2aec365fbd7394fe12cbf77114f326e32d52" exitCode=0 Nov 03 00:54:24.562294 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:24.561890 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2de4fbe5-gpm2r" event={"ID":"cc8f0e43-471d-4339-a877-719e2b5061fa","Type":"ContainerDied","Data":"0c2310530b2a0f532ad97b7881ff2aec365fbd7394fe12cbf77114f326e32d52"} Nov 03 00:54:24.606168 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:24.606134 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0533525-shntr"] Nov 03 00:54:24.610300 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:24.610267 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0533525-shntr" Nov 03 00:54:24.617902 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:24.617875 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0533525-shntr"] Nov 03 00:54:24.715557 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:24.715524 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c364576-1756-471d-9f70-01e1d90ae1a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d0533525-shntr\" (UID: \"2c364576-1756-471d-9f70-01e1d90ae1a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0533525-shntr" Nov 03 00:54:24.715741 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:24.715680 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsxkx\" (UniqueName: \"kubernetes.io/projected/2c364576-1756-471d-9f70-01e1d90ae1a0-kube-api-access-fsxkx\") pod \"maas-api-revoke-sub-d0533525-shntr\" (UID: \"2c364576-1756-471d-9f70-01e1d90ae1a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0533525-shntr" Nov 03 00:54:24.715833 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:24.715819 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c364576-1756-471d-9f70-01e1d90ae1a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d0533525-shntr\" (UID: \"2c364576-1756-471d-9f70-01e1d90ae1a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0533525-shntr" Nov 03 00:54:24.819704 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:24.819610 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsxkx\" (UniqueName: \"kubernetes.io/projected/2c364576-1756-471d-9f70-01e1d90ae1a0-kube-api-access-fsxkx\") pod \"maas-api-revoke-sub-d0533525-shntr\" (UID: \"2c364576-1756-471d-9f70-01e1d90ae1a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0533525-shntr" Nov 03 00:54:24.819871 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:24.819771 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c364576-1756-471d-9f70-01e1d90ae1a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d0533525-shntr\" (UID: \"2c364576-1756-471d-9f70-01e1d90ae1a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0533525-shntr" Nov 03 00:54:24.819871 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:24.819857 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c364576-1756-471d-9f70-01e1d90ae1a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d0533525-shntr\" (UID: \"2c364576-1756-471d-9f70-01e1d90ae1a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0533525-shntr" Nov 03 00:54:24.820619 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:24.820593 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c364576-1756-471d-9f70-01e1d90ae1a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d0533525-shntr\" (UID: \"2c364576-1756-471d-9f70-01e1d90ae1a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0533525-shntr" Nov 03 00:54:24.823210 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:24.823188 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c364576-1756-471d-9f70-01e1d90ae1a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d0533525-shntr\" (UID: \"2c364576-1756-471d-9f70-01e1d90ae1a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0533525-shntr" Nov 03 00:54:24.830603 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:24.830571 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsxkx\" (UniqueName: \"kubernetes.io/projected/2c364576-1756-471d-9f70-01e1d90ae1a0-kube-api-access-fsxkx\") pod \"maas-api-revoke-sub-d0533525-shntr\" (UID: \"2c364576-1756-471d-9f70-01e1d90ae1a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0533525-shntr" Nov 03 00:54:24.928178 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:24.928146 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0533525-shntr" Nov 03 00:54:25.213926 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:25.213891 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0533525-shntr"] Nov 03 00:54:25.214816 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:54:25.214779 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c364576_1756_471d_9f70_01e1d90ae1a0.slice/crio-d2bed87296f558d66de4b2a15e5c230d7744ec8d2ec39f41c43baf18398c6ba4 WatchSource:0}: Error finding container d2bed87296f558d66de4b2a15e5c230d7744ec8d2ec39f41c43baf18398c6ba4: Status 404 returned error can't find the container with id d2bed87296f558d66de4b2a15e5c230d7744ec8d2ec39f41c43baf18398c6ba4 Nov 03 00:54:25.572705 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:25.572666 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0533525-shntr" event={"ID":"2c364576-1756-471d-9f70-01e1d90ae1a0","Type":"ContainerStarted","Data":"d2bed87296f558d66de4b2a15e5c230d7744ec8d2ec39f41c43baf18398c6ba4"} Nov 03 00:54:25.725976 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:25.725950 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2de4fbe5-gpm2r" Nov 03 00:54:25.733997 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:25.733967 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc8f0e43-471d-4339-a877-719e2b5061fa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc8f0e43-471d-4339-a877-719e2b5061fa-maas-api-xks-ca\") pod \"cc8f0e43-471d-4339-a877-719e2b5061fa\" (UID: \"cc8f0e43-471d-4339-a877-719e2b5061fa\") " Nov 03 00:54:25.734144 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:25.734025 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc8f0e43-471d-4339-a877-719e2b5061fa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc8f0e43-471d-4339-a877-719e2b5061fa-maas-api-openshift-ca\") pod \"cc8f0e43-471d-4339-a877-719e2b5061fa\" (UID: \"cc8f0e43-471d-4339-a877-719e2b5061fa\") " Nov 03 00:54:25.734144 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:25.734061 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc8f0e43-471d-4339-a877-719e2b5061fa-kube-api-access-6lqjh\" (UniqueName: \"kubernetes.io/projected/cc8f0e43-471d-4339-a877-719e2b5061fa-kube-api-access-6lqjh\") pod \"cc8f0e43-471d-4339-a877-719e2b5061fa\" (UID: \"cc8f0e43-471d-4339-a877-719e2b5061fa\") " Nov 03 00:54:25.734391 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:25.734367 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc8f0e43-471d-4339-a877-719e2b5061fa-maas-api-openshift-ca" pod "cc8f0e43-471d-4339-a877-719e2b5061fa" (UID: "cc8f0e43-471d-4339-a877-719e2b5061fa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:54:25.736054 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:25.736027 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc8f0e43-471d-4339-a877-719e2b5061fa-maas-api-xks-ca" pod "cc8f0e43-471d-4339-a877-719e2b5061fa" (UID: "cc8f0e43-471d-4339-a877-719e2b5061fa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:54:25.736159 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:25.736070 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc8f0e43-471d-4339-a877-719e2b5061fa-kube-api-access-6lqjh" pod "cc8f0e43-471d-4339-a877-719e2b5061fa" (UID: "cc8f0e43-471d-4339-a877-719e2b5061fa"). InnerVolumeSpecName "kube-api-access-6lqjh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:54:25.835733 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:25.835702 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc8f0e43-471d-4339-a877-719e2b5061fa-maas-api-xks-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:54:25.835831 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:25.835739 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc8f0e43-471d-4339-a877-719e2b5061fa-maas-api-openshift-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:54:25.835831 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:25.835752 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6lqjh\" (UniqueName: \"kubernetes.io/projected/cc8f0e43-471d-4339-a877-719e2b5061fa-kube-api-access-6lqjh\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:54:26.192409 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:26.192332 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc90ad3d-8drjr"] Nov 03 00:54:26.192931 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:26.192918 4421 cpu_manager.go:414] "containerMap: removing container" podUID="cc8f0e43-471d-4339-a877-719e2b5061fa" containerName="revoke-keys" Nov 03 00:54:26.192970 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:26.192932 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc8f0e43-471d-4339-a877-719e2b5061fa" containerName="revoke-keys" Nov 03 00:54:26.193044 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:26.193035 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc8f0e43-471d-4339-a877-719e2b5061fa" containerName="revoke-keys" Nov 03 00:54:26.196988 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:26.196966 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc90ad3d-8drjr" Nov 03 00:54:26.204911 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:26.204882 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc90ad3d-8drjr"] Nov 03 00:54:26.243103 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:26.243072 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c50f5d52-bbc4-4fd8-b894-8af10e0454bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc90ad3d-8drjr\" (UID: \"c50f5d52-bbc4-4fd8-b894-8af10e0454bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc90ad3d-8drjr" Nov 03 00:54:26.243297 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:26.243162 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brrrh\" (UniqueName: \"kubernetes.io/projected/c50f5d52-bbc4-4fd8-b894-8af10e0454bd-kube-api-access-brrrh\") pod \"maas-api-revoke-sub-bc90ad3d-8drjr\" (UID: \"c50f5d52-bbc4-4fd8-b894-8af10e0454bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc90ad3d-8drjr" Nov 03 00:54:26.243297 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:26.243240 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c50f5d52-bbc4-4fd8-b894-8af10e0454bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc90ad3d-8drjr\" (UID: \"c50f5d52-bbc4-4fd8-b894-8af10e0454bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc90ad3d-8drjr" Nov 03 00:54:26.345258 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:26.345207 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c50f5d52-bbc4-4fd8-b894-8af10e0454bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc90ad3d-8drjr\" (UID: \"c50f5d52-bbc4-4fd8-b894-8af10e0454bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc90ad3d-8drjr" Nov 03 00:54:26.345505 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:26.345424 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brrrh\" (UniqueName: \"kubernetes.io/projected/c50f5d52-bbc4-4fd8-b894-8af10e0454bd-kube-api-access-brrrh\") pod \"maas-api-revoke-sub-bc90ad3d-8drjr\" (UID: \"c50f5d52-bbc4-4fd8-b894-8af10e0454bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc90ad3d-8drjr" Nov 03 00:54:26.345577 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:26.345554 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c50f5d52-bbc4-4fd8-b894-8af10e0454bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc90ad3d-8drjr\" (UID: \"c50f5d52-bbc4-4fd8-b894-8af10e0454bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc90ad3d-8drjr" Nov 03 00:54:26.346007 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:26.345980 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c50f5d52-bbc4-4fd8-b894-8af10e0454bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc90ad3d-8drjr\" (UID: \"c50f5d52-bbc4-4fd8-b894-8af10e0454bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc90ad3d-8drjr" Nov 03 00:54:26.349338 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:26.349314 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c50f5d52-bbc4-4fd8-b894-8af10e0454bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc90ad3d-8drjr\" (UID: \"c50f5d52-bbc4-4fd8-b894-8af10e0454bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc90ad3d-8drjr" Nov 03 00:54:26.354307 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:26.354244 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brrrh\" (UniqueName: \"kubernetes.io/projected/c50f5d52-bbc4-4fd8-b894-8af10e0454bd-kube-api-access-brrrh\") pod \"maas-api-revoke-sub-bc90ad3d-8drjr\" (UID: \"c50f5d52-bbc4-4fd8-b894-8af10e0454bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc90ad3d-8drjr" Nov 03 00:54:26.521136 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:26.521092 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc90ad3d-8drjr" Nov 03 00:54:26.584562 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:26.584538 4421 generic.go:356] "Generic (PLEG): container finished" podID="2c364576-1756-471d-9f70-01e1d90ae1a0" containerID="196edae792056b59afcf0389ba8152d37b0562c1de6016b4c020e14abd490f96" exitCode=0 Nov 03 00:54:26.584927 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:26.584622 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0533525-shntr" event={"ID":"2c364576-1756-471d-9f70-01e1d90ae1a0","Type":"ContainerDied","Data":"196edae792056b59afcf0389ba8152d37b0562c1de6016b4c020e14abd490f96"} Nov 03 00:54:26.586257 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:26.586232 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2de4fbe5-gpm2r" event={"ID":"cc8f0e43-471d-4339-a877-719e2b5061fa","Type":"ContainerDied","Data":"bd5786b40327f81293c31dda10d814e646117f25140d4b271a31d3dfea9286e4"} Nov 03 00:54:26.586441 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:26.586257 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2de4fbe5-gpm2r" Nov 03 00:54:26.586441 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:26.586306 4421 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bd5786b40327f81293c31dda10d814e646117f25140d4b271a31d3dfea9286e4" Nov 03 00:54:26.803448 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:26.803409 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc90ad3d-8drjr"] Nov 03 00:54:26.804023 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:54:26.803998 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc50f5d52_bbc4_4fd8_b894_8af10e0454bd.slice/crio-b3d2b1aa0b1ed94b24bd8492b3ac4e20ab4bdb521cd27677ac3480d9dce3c8af WatchSource:0}: Error finding container b3d2b1aa0b1ed94b24bd8492b3ac4e20ab4bdb521cd27677ac3480d9dce3c8af: Status 404 returned error can't find the container with id b3d2b1aa0b1ed94b24bd8492b3ac4e20ab4bdb521cd27677ac3480d9dce3c8af Nov 03 00:54:27.595780 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:27.595737 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc90ad3d-8drjr" event={"ID":"c50f5d52-bbc4-4fd8-b894-8af10e0454bd","Type":"ContainerStarted","Data":"af0e0c4f4e3f353329d8e239340e6b61ce586b82f7e77922eac28d14931b17cd"} Nov 03 00:54:27.596207 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:27.595808 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc90ad3d-8drjr" event={"ID":"c50f5d52-bbc4-4fd8-b894-8af10e0454bd","Type":"ContainerStarted","Data":"b3d2b1aa0b1ed94b24bd8492b3ac4e20ab4bdb521cd27677ac3480d9dce3c8af"} Nov 03 00:54:27.615565 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:27.615505 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc90ad3d-8drjr" podStartSLOduration=1.615485929 podStartE2EDuration="1.615485929s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:54:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:54:27.611008387 +0000 UTC m=+809.782011487" watchObservedRunningTime="2026-11-03 00:54:27.615485929 +0000 UTC m=+809.786489026" Nov 03 00:54:27.827554 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:27.827526 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0533525-shntr" Nov 03 00:54:27.864434 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:27.864348 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c364576-1756-471d-9f70-01e1d90ae1a0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c364576-1756-471d-9f70-01e1d90ae1a0-maas-api-openshift-ca\") pod \"2c364576-1756-471d-9f70-01e1d90ae1a0\" (UID: \"2c364576-1756-471d-9f70-01e1d90ae1a0\") " Nov 03 00:54:27.864585 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:27.864447 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2c364576-1756-471d-9f70-01e1d90ae1a0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c364576-1756-471d-9f70-01e1d90ae1a0-maas-api-xks-ca\") pod \"2c364576-1756-471d-9f70-01e1d90ae1a0\" (UID: \"2c364576-1756-471d-9f70-01e1d90ae1a0\") " Nov 03 00:54:27.864700 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:27.864683 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c364576-1756-471d-9f70-01e1d90ae1a0-kube-api-access-fsxkx\" (UniqueName: \"kubernetes.io/projected/2c364576-1756-471d-9f70-01e1d90ae1a0-kube-api-access-fsxkx\") pod \"2c364576-1756-471d-9f70-01e1d90ae1a0\" (UID: \"2c364576-1756-471d-9f70-01e1d90ae1a0\") " Nov 03 00:54:27.864784 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:27.864754 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c364576-1756-471d-9f70-01e1d90ae1a0-maas-api-openshift-ca" pod "2c364576-1756-471d-9f70-01e1d90ae1a0" (UID: "2c364576-1756-471d-9f70-01e1d90ae1a0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:54:27.865246 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:27.865216 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c364576-1756-471d-9f70-01e1d90ae1a0-maas-api-openshift-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:54:27.866641 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:27.866611 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c364576-1756-471d-9f70-01e1d90ae1a0-maas-api-xks-ca" pod "2c364576-1756-471d-9f70-01e1d90ae1a0" (UID: "2c364576-1756-471d-9f70-01e1d90ae1a0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:54:27.866808 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:27.866790 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c364576-1756-471d-9f70-01e1d90ae1a0-kube-api-access-fsxkx" pod "2c364576-1756-471d-9f70-01e1d90ae1a0" (UID: "2c364576-1756-471d-9f70-01e1d90ae1a0"). InnerVolumeSpecName "kube-api-access-fsxkx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:54:27.966449 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:27.966414 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fsxkx\" (UniqueName: \"kubernetes.io/projected/2c364576-1756-471d-9f70-01e1d90ae1a0-kube-api-access-fsxkx\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:54:27.966449 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:27.966448 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c364576-1756-471d-9f70-01e1d90ae1a0-maas-api-xks-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:54:28.605906 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:28.605862 4421 generic.go:356] "Generic (PLEG): container finished" podID="c50f5d52-bbc4-4fd8-b894-8af10e0454bd" containerID="af0e0c4f4e3f353329d8e239340e6b61ce586b82f7e77922eac28d14931b17cd" exitCode=0 Nov 03 00:54:28.606366 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:28.605944 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc90ad3d-8drjr" event={"ID":"c50f5d52-bbc4-4fd8-b894-8af10e0454bd","Type":"ContainerDied","Data":"af0e0c4f4e3f353329d8e239340e6b61ce586b82f7e77922eac28d14931b17cd"} Nov 03 00:54:28.607578 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:28.607559 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0533525-shntr" Nov 03 00:54:28.607747 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:28.607574 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0533525-shntr" event={"ID":"2c364576-1756-471d-9f70-01e1d90ae1a0","Type":"ContainerDied","Data":"d2bed87296f558d66de4b2a15e5c230d7744ec8d2ec39f41c43baf18398c6ba4"} Nov 03 00:54:28.607747 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:28.607625 4421 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d2bed87296f558d66de4b2a15e5c230d7744ec8d2ec39f41c43baf18398c6ba4" Nov 03 00:54:30.742511 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:30.742442 4421 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-f7680d/simulator-w2-f7680d-kserve-85bdc86cd6-8gxrg" podUID="2bdcf521-cebd-4d6b-8554-1b60db8b0f3c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:54:30.832449 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:30.832418 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc90ad3d-8drjr" Nov 03 00:54:30.935373 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:30.935236 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/simulator-w2-f7680d-kserve-85bdc86cd6-8gxrg" podUID="2bdcf521-cebd-4d6b-8554-1b60db8b0f3c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:54:31.013789 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:30.935601 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/simulator-w3-83755a-kserve-77d6664559-psqtg" podUID="b0a4c234-af58-4da0-a1a4-e1f167953f6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:54:31.037312 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:30.961738 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c50f5d52-bbc4-4fd8-b894-8af10e0454bd-kube-api-access-brrrh\" (UniqueName: \"kubernetes.io/projected/c50f5d52-bbc4-4fd8-b894-8af10e0454bd-kube-api-access-brrrh\") pod \"c50f5d52-bbc4-4fd8-b894-8af10e0454bd\" (UID: \"c50f5d52-bbc4-4fd8-b894-8af10e0454bd\") " Nov 03 00:54:31.041300 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:31.037779 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c50f5d52-bbc4-4fd8-b894-8af10e0454bd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c50f5d52-bbc4-4fd8-b894-8af10e0454bd-maas-api-openshift-ca\") pod \"c50f5d52-bbc4-4fd8-b894-8af10e0454bd\" (UID: \"c50f5d52-bbc4-4fd8-b894-8af10e0454bd\") " Nov 03 00:54:31.066504 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:31.066451 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c50f5d52-bbc4-4fd8-b894-8af10e0454bd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c50f5d52-bbc4-4fd8-b894-8af10e0454bd-maas-api-xks-ca\") pod \"c50f5d52-bbc4-4fd8-b894-8af10e0454bd\" (UID: \"c50f5d52-bbc4-4fd8-b894-8af10e0454bd\") " Nov 03 00:54:31.070298 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:31.038852 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c50f5d52-bbc4-4fd8-b894-8af10e0454bd-maas-api-openshift-ca" pod "c50f5d52-bbc4-4fd8-b894-8af10e0454bd" (UID: "c50f5d52-bbc4-4fd8-b894-8af10e0454bd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:54:31.070727 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:31.070682 4421 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-83755a/premium-w3-83755a-kserve-7c9d66988b-lt4xd" podUID="c0c6b02e-2ecb-428e-a13d-be6d7cd1ba39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:54:31.081517 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:31.081479 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c50f5d52-bbc4-4fd8-b894-8af10e0454bd-kube-api-access-brrrh" pod "c50f5d52-bbc4-4fd8-b894-8af10e0454bd" (UID: "c50f5d52-bbc4-4fd8-b894-8af10e0454bd"). InnerVolumeSpecName "kube-api-access-brrrh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:54:31.142454 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:31.142409 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c50f5d52-bbc4-4fd8-b894-8af10e0454bd-maas-api-xks-ca" pod "c50f5d52-bbc4-4fd8-b894-8af10e0454bd" (UID: "c50f5d52-bbc4-4fd8-b894-8af10e0454bd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:54:31.160729 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:31.159616 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-650be0/premium-w4-650be0-kserve-7bf8555dfb-z97n5" podUID="40601dc5-808f-456f-826f-6e0c35966ada" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:54:31.187357 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:31.172820 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c50f5d52-bbc4-4fd8-b894-8af10e0454bd-maas-api-openshift-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:54:31.187623 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:31.187593 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c50f5d52-bbc4-4fd8-b894-8af10e0454bd-maas-api-xks-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:54:31.187759 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:31.187743 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brrrh\" (UniqueName: \"kubernetes.io/projected/c50f5d52-bbc4-4fd8-b894-8af10e0454bd-kube-api-access-brrrh\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:54:31.292409 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:31.292360 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8gm8n" podUID="2d367e12-e46f-4a68-9825-9b1c934d4d12" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:54:31.342301 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:31.335400 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-650be0/simulator-w4-650be0-kserve-7bc588774f-gvz2k" podUID="a827fa88-f078-4312-8f12-2c799c11b461" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:54:31.587505 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:31.586373 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/distinct-w3-83755a-kserve-86bdbbdd85-2cnnl" podUID="f4ae9624-d127-4a55-837d-7c9ff0267b37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:54:32.037878 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:32.035983 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/simulator-w2-f7680d-kserve-85bdc86cd6-8gxrg" podUID="2bdcf521-cebd-4d6b-8554-1b60db8b0f3c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:54:32.037878 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:32.036739 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/simulator-w3-83755a-kserve-77d6664559-psqtg" podUID="b0a4c234-af58-4da0-a1a4-e1f167953f6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Nov 03 00:54:32.127298 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:32.098977 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f7680d/simulator-w2-f7680d-kserve-85bdc86cd6-8gxrg" Nov 03 00:54:32.211694 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:32.211664 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-83755a/simulator-w3-83755a-kserve-77d6664559-psqtg" Nov 03 00:54:32.229296 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:32.160719 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-650be0/premium-w4-650be0-kserve-7bf8555dfb-z97n5" podUID="40601dc5-808f-456f-826f-6e0c35966ada" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Nov 03 00:54:32.230796 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:32.230764 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-650be0/premium-w4-650be0-kserve-7bf8555dfb-z97n5" Nov 03 00:54:32.246573 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:32.246453 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc90ad3d-8drjr" event={"ID":"c50f5d52-bbc4-4fd8-b894-8af10e0454bd","Type":"ContainerDied","Data":"b3d2b1aa0b1ed94b24bd8492b3ac4e20ab4bdb521cd27677ac3480d9dce3c8af"} Nov 03 00:54:32.273187 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:32.273156 4421 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b3d2b1aa0b1ed94b24bd8492b3ac4e20ab4bdb521cd27677ac3480d9dce3c8af" Nov 03 00:54:32.277382 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:32.277337 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc90ad3d-8drjr" Nov 03 00:54:32.312557 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:32.312466 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8gm8n" podUID="2d367e12-e46f-4a68-9825-9b1c934d4d12" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:54:32.313441 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:32.313417 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8gm8n" Nov 03 00:54:32.350851 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:32.350815 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-650be0/simulator-w4-650be0-kserve-7bc588774f-gvz2k" podUID="a827fa88-f078-4312-8f12-2c799c11b461" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:54:32.351666 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:32.351643 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-650be0/simulator-w4-650be0-kserve-7bc588774f-gvz2k" Nov 03 00:54:32.374074 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:32.374035 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/premium-w3-83755a-kserve-7c9d66988b-lt4xd" podUID="c0c6b02e-2ecb-428e-a13d-be6d7cd1ba39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:54:32.641947 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:32.615408 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/distinct-w3-83755a-kserve-86bdbbdd85-2cnnl" podUID="f4ae9624-d127-4a55-837d-7c9ff0267b37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:54:32.641947 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:32.616216 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-83755a/distinct-w3-83755a-kserve-86bdbbdd85-2cnnl" Nov 03 00:54:32.687411 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:32.687359 4421 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-83755a/simulator-w3-83755a-kserve-77d6664559-psqtg" podUID="b0a4c234-af58-4da0-a1a4-e1f167953f6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:54:32.722428 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:32.722383 4421 patch_prober.go:29] interesting pod/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-h9mjc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 00:54:32.722748 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:32.722715 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-h9mjc" podUID="dd5f7acb-d880-4f98-8b64-8501641747b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 00:54:33.042300 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:33.038370 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/simulator-w3-83755a-kserve-77d6664559-psqtg" podUID="b0a4c234-af58-4da0-a1a4-e1f167953f6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:54:33.070301 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:33.046686 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/simulator-w2-f7680d-kserve-85bdc86cd6-8gxrg" podUID="2bdcf521-cebd-4d6b-8554-1b60db8b0f3c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 03 00:54:33.096413 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:33.096361 4421 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-650be0/simulator-w4-650be0-kserve-7bc588774f-gvz2k" podUID="a827fa88-f078-4312-8f12-2c799c11b461" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:54:33.238431 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:33.238379 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-650be0/premium-w4-650be0-kserve-7bf8555dfb-z97n5" podUID="40601dc5-808f-456f-826f-6e0c35966ada" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:54:33.320484 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:33.320378 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8gm8n" podUID="2d367e12-e46f-4a68-9825-9b1c934d4d12" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": context deadline exceeded" Nov 03 00:54:33.383301 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:33.360120 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-650be0/simulator-w4-650be0-kserve-7bc588774f-gvz2k" podUID="a827fa88-f078-4312-8f12-2c799c11b461" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Nov 03 00:54:33.387298 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:33.386342 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/premium-w3-83755a-kserve-7c9d66988b-lt4xd" podUID="c0c6b02e-2ecb-428e-a13d-be6d7cd1ba39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:54:33.410234 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:33.387691 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-83755a/premium-w3-83755a-kserve-7c9d66988b-lt4xd" Nov 03 00:54:33.795299 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:33.790481 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/distinct-w3-83755a-kserve-86bdbbdd85-2cnnl" podUID="f4ae9624-d127-4a55-837d-7c9ff0267b37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded" Nov 03 00:54:33.870349 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:33.870310 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-650be0/premium-w4-650be0-kserve-7bf8555dfb-z97n5" Nov 03 00:54:34.046543 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:34.046440 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/simulator-w3-83755a-kserve-77d6664559-psqtg" podUID="b0a4c234-af58-4da0-a1a4-e1f167953f6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:54:34.105351 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:34.079974 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/simulator-w2-f7680d-kserve-85bdc86cd6-8gxrg" podUID="2bdcf521-cebd-4d6b-8554-1b60db8b0f3c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:54:34.187729 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:34.187697 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-83755a/simulator-w3-83755a-kserve-77d6664559-psqtg" Nov 03 00:54:34.188319 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:34.188265 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-83755a/premium-w3-83755a-kserve-7c9d66988b-lt4xd" Nov 03 00:54:34.209736 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:34.209708 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-650be0/simulator-w4-650be0-kserve-7bc588774f-gvz2k" Nov 03 00:54:34.213940 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:34.213916 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-83755a/distinct-w3-83755a-kserve-86bdbbdd85-2cnnl" Nov 03 00:54:34.219587 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:34.219559 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f7680d/simulator-w2-f7680d-kserve-85bdc86cd6-8gxrg" Nov 03 00:54:34.244116 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:34.244086 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8gm8n" Nov 03 00:54:34.371425 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:34.371337 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c09f2e44-xqjtm"] Nov 03 00:54:34.371999 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:34.371979 4421 cpu_manager.go:414] "containerMap: removing container" podUID="c50f5d52-bbc4-4fd8-b894-8af10e0454bd" containerName="revoke-keys" Nov 03 00:54:34.372102 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:34.372002 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c50f5d52-bbc4-4fd8-b894-8af10e0454bd" containerName="revoke-keys" Nov 03 00:54:34.372102 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:34.372047 4421 cpu_manager.go:414] "containerMap: removing container" podUID="2c364576-1756-471d-9f70-01e1d90ae1a0" containerName="revoke-keys" Nov 03 00:54:34.372102 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:34.372055 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c364576-1756-471d-9f70-01e1d90ae1a0" containerName="revoke-keys" Nov 03 00:54:34.372256 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:34.372190 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="c50f5d52-bbc4-4fd8-b894-8af10e0454bd" containerName="revoke-keys" Nov 03 00:54:34.372256 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:34.372207 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c364576-1756-471d-9f70-01e1d90ae1a0" containerName="revoke-keys" Nov 03 00:54:34.375701 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:34.375680 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c09f2e44-xqjtm" Nov 03 00:54:34.379626 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:34.379604 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:54:34.379969 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:34.379933 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 00:54:34.380088 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:34.380017 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 00:54:34.380230 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:34.380017 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6q4xp\"" Nov 03 00:54:34.382396 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:34.382376 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c09f2e44-xqjtm"] Nov 03 00:54:34.495148 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:34.495105 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7197ee03-efec-4b33-be93-2853619c6215-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c09f2e44-xqjtm\" (UID: \"7197ee03-efec-4b33-be93-2853619c6215\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c09f2e44-xqjtm" Nov 03 00:54:34.495366 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:34.495229 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcjdh\" (UniqueName: \"kubernetes.io/projected/7197ee03-efec-4b33-be93-2853619c6215-kube-api-access-lcjdh\") pod \"maas-api-revoke-sub-c09f2e44-xqjtm\" (UID: \"7197ee03-efec-4b33-be93-2853619c6215\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c09f2e44-xqjtm" Nov 03 00:54:34.495419 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:34.495405 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7197ee03-efec-4b33-be93-2853619c6215-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c09f2e44-xqjtm\" (UID: \"7197ee03-efec-4b33-be93-2853619c6215\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c09f2e44-xqjtm" Nov 03 00:54:34.596840 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:34.596800 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7197ee03-efec-4b33-be93-2853619c6215-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c09f2e44-xqjtm\" (UID: \"7197ee03-efec-4b33-be93-2853619c6215\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c09f2e44-xqjtm" Nov 03 00:54:34.597055 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:34.596865 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcjdh\" (UniqueName: \"kubernetes.io/projected/7197ee03-efec-4b33-be93-2853619c6215-kube-api-access-lcjdh\") pod \"maas-api-revoke-sub-c09f2e44-xqjtm\" (UID: \"7197ee03-efec-4b33-be93-2853619c6215\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c09f2e44-xqjtm" Nov 03 00:54:34.597055 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:34.596938 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7197ee03-efec-4b33-be93-2853619c6215-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c09f2e44-xqjtm\" (UID: \"7197ee03-efec-4b33-be93-2853619c6215\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c09f2e44-xqjtm" Nov 03 00:54:34.597628 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:34.597601 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7197ee03-efec-4b33-be93-2853619c6215-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c09f2e44-xqjtm\" (UID: \"7197ee03-efec-4b33-be93-2853619c6215\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c09f2e44-xqjtm" Nov 03 00:54:34.599319 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:34.599291 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7197ee03-efec-4b33-be93-2853619c6215-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c09f2e44-xqjtm\" (UID: \"7197ee03-efec-4b33-be93-2853619c6215\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c09f2e44-xqjtm" Nov 03 00:54:34.605208 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:34.605183 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcjdh\" (UniqueName: \"kubernetes.io/projected/7197ee03-efec-4b33-be93-2853619c6215-kube-api-access-lcjdh\") pod \"maas-api-revoke-sub-c09f2e44-xqjtm\" (UID: \"7197ee03-efec-4b33-be93-2853619c6215\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c09f2e44-xqjtm" Nov 03 00:54:34.692763 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:34.692667 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c09f2e44-xqjtm" Nov 03 00:54:34.705660 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:34.705634 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-259597d3-lcqhj"] Nov 03 00:54:34.711814 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:34.711794 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-259597d3-lcqhj" Nov 03 00:54:34.720441 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:34.720414 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-259597d3-lcqhj"] Nov 03 00:54:34.799574 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:34.799540 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hj5q4\" (UniqueName: \"kubernetes.io/projected/9dcc5fdc-d37a-47f7-8b6f-2359a8a377bd-kube-api-access-hj5q4\") pod \"maas-api-revoke-sub-259597d3-lcqhj\" (UID: \"9dcc5fdc-d37a-47f7-8b6f-2359a8a377bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-259597d3-lcqhj" Nov 03 00:54:34.799741 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:34.799615 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9dcc5fdc-d37a-47f7-8b6f-2359a8a377bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-259597d3-lcqhj\" (UID: \"9dcc5fdc-d37a-47f7-8b6f-2359a8a377bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-259597d3-lcqhj" Nov 03 00:54:34.799741 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:34.799733 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9dcc5fdc-d37a-47f7-8b6f-2359a8a377bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-259597d3-lcqhj\" (UID: \"9dcc5fdc-d37a-47f7-8b6f-2359a8a377bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-259597d3-lcqhj" Nov 03 00:54:34.901510 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:34.901472 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hj5q4\" (UniqueName: \"kubernetes.io/projected/9dcc5fdc-d37a-47f7-8b6f-2359a8a377bd-kube-api-access-hj5q4\") pod \"maas-api-revoke-sub-259597d3-lcqhj\" (UID: \"9dcc5fdc-d37a-47f7-8b6f-2359a8a377bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-259597d3-lcqhj" Nov 03 00:54:34.901681 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:34.901552 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9dcc5fdc-d37a-47f7-8b6f-2359a8a377bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-259597d3-lcqhj\" (UID: \"9dcc5fdc-d37a-47f7-8b6f-2359a8a377bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-259597d3-lcqhj" Nov 03 00:54:34.901681 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:34.901614 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9dcc5fdc-d37a-47f7-8b6f-2359a8a377bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-259597d3-lcqhj\" (UID: \"9dcc5fdc-d37a-47f7-8b6f-2359a8a377bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-259597d3-lcqhj" Nov 03 00:54:34.902262 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:34.902234 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9dcc5fdc-d37a-47f7-8b6f-2359a8a377bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-259597d3-lcqhj\" (UID: \"9dcc5fdc-d37a-47f7-8b6f-2359a8a377bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-259597d3-lcqhj" Nov 03 00:54:34.904024 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:34.904001 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9dcc5fdc-d37a-47f7-8b6f-2359a8a377bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-259597d3-lcqhj\" (UID: \"9dcc5fdc-d37a-47f7-8b6f-2359a8a377bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-259597d3-lcqhj" Nov 03 00:54:34.909438 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:34.909416 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hj5q4\" (UniqueName: \"kubernetes.io/projected/9dcc5fdc-d37a-47f7-8b6f-2359a8a377bd-kube-api-access-hj5q4\") pod \"maas-api-revoke-sub-259597d3-lcqhj\" (UID: \"9dcc5fdc-d37a-47f7-8b6f-2359a8a377bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-259597d3-lcqhj" Nov 03 00:54:34.989390 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:34.989347 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c09f2e44-xqjtm"] Nov 03 00:54:34.989609 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:54:34.989578 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7197ee03_efec_4b33_be93_2853619c6215.slice/crio-d6805ee47d60b9b928fb21042dd2c630fdc97d22374ad2d13d278cc7314a3918 WatchSource:0}: Error finding container d6805ee47d60b9b928fb21042dd2c630fdc97d22374ad2d13d278cc7314a3918: Status 404 returned error can't find the container with id d6805ee47d60b9b928fb21042dd2c630fdc97d22374ad2d13d278cc7314a3918 Nov 03 00:54:35.044653 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:35.044616 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-259597d3-lcqhj" Nov 03 00:54:35.330314 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:35.330255 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-259597d3-lcqhj"] Nov 03 00:54:35.330893 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:54:35.330866 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9dcc5fdc_d37a_47f7_8b6f_2359a8a377bd.slice/crio-9c88a52a7a69abfac76d126d944e6b56de70847d4f80650d50f9386f8d187e5c WatchSource:0}: Error finding container 9c88a52a7a69abfac76d126d944e6b56de70847d4f80650d50f9386f8d187e5c: Status 404 returned error can't find the container with id 9c88a52a7a69abfac76d126d944e6b56de70847d4f80650d50f9386f8d187e5c Nov 03 00:54:35.579021 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:35.578921 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-259597d3-lcqhj" event={"ID":"9dcc5fdc-d37a-47f7-8b6f-2359a8a377bd","Type":"ContainerStarted","Data":"9c88a52a7a69abfac76d126d944e6b56de70847d4f80650d50f9386f8d187e5c"} Nov 03 00:54:35.580670 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:35.580640 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c09f2e44-xqjtm" event={"ID":"7197ee03-efec-4b33-be93-2853619c6215","Type":"ContainerStarted","Data":"07735d553edacf36d3b8e4c14ef0d411e5c6837f6bbff59e126af149fc469d0c"} Nov 03 00:54:35.580793 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:35.580686 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c09f2e44-xqjtm" event={"ID":"7197ee03-efec-4b33-be93-2853619c6215","Type":"ContainerStarted","Data":"d6805ee47d60b9b928fb21042dd2c630fdc97d22374ad2d13d278cc7314a3918"} Nov 03 00:54:35.599755 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:35.599703 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c09f2e44-xqjtm" podStartSLOduration=1.5996896569999999 podStartE2EDuration="1.599689657s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:54:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:54:35.596021895 +0000 UTC m=+817.767024992" watchObservedRunningTime="2026-11-03 00:54:35.599689657 +0000 UTC m=+817.770692752" Nov 03 00:54:36.590045 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:36.589948 4421 generic.go:356] "Generic (PLEG): container finished" podID="9dcc5fdc-d37a-47f7-8b6f-2359a8a377bd" containerID="52a0a2f219f234870cf453e39508097a97c9367606e664061d427f88c751be1f" exitCode=0 Nov 03 00:54:36.590527 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:36.590041 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-259597d3-lcqhj" event={"ID":"9dcc5fdc-d37a-47f7-8b6f-2359a8a377bd","Type":"ContainerDied","Data":"52a0a2f219f234870cf453e39508097a97c9367606e664061d427f88c751be1f"} Nov 03 00:54:36.591810 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:36.591785 4421 generic.go:356] "Generic (PLEG): container finished" podID="7197ee03-efec-4b33-be93-2853619c6215" containerID="07735d553edacf36d3b8e4c14ef0d411e5c6837f6bbff59e126af149fc469d0c" exitCode=0 Nov 03 00:54:36.591924 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:36.591859 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c09f2e44-xqjtm" event={"ID":"7197ee03-efec-4b33-be93-2853619c6215","Type":"ContainerDied","Data":"07735d553edacf36d3b8e4c14ef0d411e5c6837f6bbff59e126af149fc469d0c"} Nov 03 00:54:37.759141 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:37.759105 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c09f2e44-xqjtm" Nov 03 00:54:37.797963 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:37.797933 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-259597d3-lcqhj" Nov 03 00:54:37.853376 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:37.853292 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7197ee03-efec-4b33-be93-2853619c6215-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7197ee03-efec-4b33-be93-2853619c6215-maas-api-xks-ca\") pod \"7197ee03-efec-4b33-be93-2853619c6215\" (UID: \"7197ee03-efec-4b33-be93-2853619c6215\") " Nov 03 00:54:37.853526 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:37.853399 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7197ee03-efec-4b33-be93-2853619c6215-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7197ee03-efec-4b33-be93-2853619c6215-maas-api-openshift-ca\") pod \"7197ee03-efec-4b33-be93-2853619c6215\" (UID: \"7197ee03-efec-4b33-be93-2853619c6215\") " Nov 03 00:54:37.853526 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:37.853439 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7197ee03-efec-4b33-be93-2853619c6215-kube-api-access-lcjdh\" (UniqueName: \"kubernetes.io/projected/7197ee03-efec-4b33-be93-2853619c6215-kube-api-access-lcjdh\") pod \"7197ee03-efec-4b33-be93-2853619c6215\" (UID: \"7197ee03-efec-4b33-be93-2853619c6215\") " Nov 03 00:54:37.853753 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:37.853723 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7197ee03-efec-4b33-be93-2853619c6215-maas-api-openshift-ca" pod "7197ee03-efec-4b33-be93-2853619c6215" (UID: "7197ee03-efec-4b33-be93-2853619c6215"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:54:37.853943 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:37.853923 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7197ee03-efec-4b33-be93-2853619c6215-maas-api-openshift-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:54:37.856182 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:37.856158 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7197ee03-efec-4b33-be93-2853619c6215-maas-api-xks-ca" pod "7197ee03-efec-4b33-be93-2853619c6215" (UID: "7197ee03-efec-4b33-be93-2853619c6215"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:54:37.856262 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:37.856194 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7197ee03-efec-4b33-be93-2853619c6215-kube-api-access-lcjdh" pod "7197ee03-efec-4b33-be93-2853619c6215" (UID: "7197ee03-efec-4b33-be93-2853619c6215"). InnerVolumeSpecName "kube-api-access-lcjdh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:54:37.954837 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:37.954797 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9dcc5fdc-d37a-47f7-8b6f-2359a8a377bd-kube-api-access-hj5q4\" (UniqueName: \"kubernetes.io/projected/9dcc5fdc-d37a-47f7-8b6f-2359a8a377bd-kube-api-access-hj5q4\") pod \"9dcc5fdc-d37a-47f7-8b6f-2359a8a377bd\" (UID: \"9dcc5fdc-d37a-47f7-8b6f-2359a8a377bd\") " Nov 03 00:54:37.955011 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:37.954962 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9dcc5fdc-d37a-47f7-8b6f-2359a8a377bd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9dcc5fdc-d37a-47f7-8b6f-2359a8a377bd-maas-api-openshift-ca\") pod \"9dcc5fdc-d37a-47f7-8b6f-2359a8a377bd\" (UID: \"9dcc5fdc-d37a-47f7-8b6f-2359a8a377bd\") " Nov 03 00:54:37.955011 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:37.955004 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9dcc5fdc-d37a-47f7-8b6f-2359a8a377bd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9dcc5fdc-d37a-47f7-8b6f-2359a8a377bd-maas-api-xks-ca\") pod \"9dcc5fdc-d37a-47f7-8b6f-2359a8a377bd\" (UID: \"9dcc5fdc-d37a-47f7-8b6f-2359a8a377bd\") " Nov 03 00:54:37.955359 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:37.955331 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9dcc5fdc-d37a-47f7-8b6f-2359a8a377bd-maas-api-openshift-ca" pod "9dcc5fdc-d37a-47f7-8b6f-2359a8a377bd" (UID: "9dcc5fdc-d37a-47f7-8b6f-2359a8a377bd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:54:37.955612 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:37.955591 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lcjdh\" (UniqueName: \"kubernetes.io/projected/7197ee03-efec-4b33-be93-2853619c6215-kube-api-access-lcjdh\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:54:37.955677 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:37.955620 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7197ee03-efec-4b33-be93-2853619c6215-maas-api-xks-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:54:37.955677 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:37.955637 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9dcc5fdc-d37a-47f7-8b6f-2359a8a377bd-maas-api-openshift-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:54:37.956927 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:37.956905 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9dcc5fdc-d37a-47f7-8b6f-2359a8a377bd-maas-api-xks-ca" pod "9dcc5fdc-d37a-47f7-8b6f-2359a8a377bd" (UID: "9dcc5fdc-d37a-47f7-8b6f-2359a8a377bd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:54:37.956976 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:37.956911 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9dcc5fdc-d37a-47f7-8b6f-2359a8a377bd-kube-api-access-hj5q4" pod "9dcc5fdc-d37a-47f7-8b6f-2359a8a377bd" (UID: "9dcc5fdc-d37a-47f7-8b6f-2359a8a377bd"). InnerVolumeSpecName "kube-api-access-hj5q4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:54:38.056681 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:38.056649 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9dcc5fdc-d37a-47f7-8b6f-2359a8a377bd-maas-api-xks-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:54:38.056681 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:38.056675 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hj5q4\" (UniqueName: \"kubernetes.io/projected/9dcc5fdc-d37a-47f7-8b6f-2359a8a377bd-kube-api-access-hj5q4\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:54:38.606777 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:38.606748 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c09f2e44-xqjtm" Nov 03 00:54:38.606950 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:38.606781 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c09f2e44-xqjtm" event={"ID":"7197ee03-efec-4b33-be93-2853619c6215","Type":"ContainerDied","Data":"d6805ee47d60b9b928fb21042dd2c630fdc97d22374ad2d13d278cc7314a3918"} Nov 03 00:54:38.606950 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:38.606826 4421 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d6805ee47d60b9b928fb21042dd2c630fdc97d22374ad2d13d278cc7314a3918" Nov 03 00:54:38.608454 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:38.608424 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-259597d3-lcqhj" event={"ID":"9dcc5fdc-d37a-47f7-8b6f-2359a8a377bd","Type":"ContainerDied","Data":"9c88a52a7a69abfac76d126d944e6b56de70847d4f80650d50f9386f8d187e5c"} Nov 03 00:54:38.608607 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:38.608466 4421 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9c88a52a7a69abfac76d126d944e6b56de70847d4f80650d50f9386f8d187e5c" Nov 03 00:54:38.608607 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:38.608432 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-259597d3-lcqhj" Nov 03 00:54:44.669913 ip-10-0-133-217 kubenswrapper[4421]: E1103 00:54:44.669873 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 00:54:45.089194 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:45.089159 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b529cf1-z7d5k"] Nov 03 00:54:45.089817 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:45.089797 4421 cpu_manager.go:414] "containerMap: removing container" podUID="9dcc5fdc-d37a-47f7-8b6f-2359a8a377bd" containerName="revoke-keys" Nov 03 00:54:45.089936 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:45.089818 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9dcc5fdc-d37a-47f7-8b6f-2359a8a377bd" containerName="revoke-keys" Nov 03 00:54:45.089936 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:45.089838 4421 cpu_manager.go:414] "containerMap: removing container" podUID="7197ee03-efec-4b33-be93-2853619c6215" containerName="revoke-keys" Nov 03 00:54:45.089936 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:45.089846 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7197ee03-efec-4b33-be93-2853619c6215" containerName="revoke-keys" Nov 03 00:54:45.090085 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:45.089987 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="7197ee03-efec-4b33-be93-2853619c6215" containerName="revoke-keys" Nov 03 00:54:45.090085 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:45.090011 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="9dcc5fdc-d37a-47f7-8b6f-2359a8a377bd" containerName="revoke-keys" Nov 03 00:54:45.093540 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:45.093520 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b529cf1-z7d5k" Nov 03 00:54:45.097252 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:45.097175 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 00:54:45.097430 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:45.097383 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:54:45.097812 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:45.097793 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 00:54:45.098338 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:45.098316 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6q4xp\"" Nov 03 00:54:45.100855 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:45.100831 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b529cf1-z7d5k"] Nov 03 00:54:45.249980 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:45.249941 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gh89\" (UniqueName: \"kubernetes.io/projected/848ee430-f2ec-41a6-aeb4-ac2bd369980f-kube-api-access-9gh89\") pod \"maas-api-revoke-sub-7b529cf1-z7d5k\" (UID: \"848ee430-f2ec-41a6-aeb4-ac2bd369980f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b529cf1-z7d5k" Nov 03 00:54:45.250135 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:45.250119 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/848ee430-f2ec-41a6-aeb4-ac2bd369980f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7b529cf1-z7d5k\" (UID: \"848ee430-f2ec-41a6-aeb4-ac2bd369980f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b529cf1-z7d5k" Nov 03 00:54:45.250196 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:45.250179 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/848ee430-f2ec-41a6-aeb4-ac2bd369980f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7b529cf1-z7d5k\" (UID: \"848ee430-f2ec-41a6-aeb4-ac2bd369980f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b529cf1-z7d5k" Nov 03 00:54:45.351406 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:45.351304 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/848ee430-f2ec-41a6-aeb4-ac2bd369980f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7b529cf1-z7d5k\" (UID: \"848ee430-f2ec-41a6-aeb4-ac2bd369980f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b529cf1-z7d5k" Nov 03 00:54:45.351406 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:45.351393 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/848ee430-f2ec-41a6-aeb4-ac2bd369980f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7b529cf1-z7d5k\" (UID: \"848ee430-f2ec-41a6-aeb4-ac2bd369980f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b529cf1-z7d5k" Nov 03 00:54:45.351614 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:45.351509 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gh89\" (UniqueName: \"kubernetes.io/projected/848ee430-f2ec-41a6-aeb4-ac2bd369980f-kube-api-access-9gh89\") pod \"maas-api-revoke-sub-7b529cf1-z7d5k\" (UID: \"848ee430-f2ec-41a6-aeb4-ac2bd369980f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b529cf1-z7d5k" Nov 03 00:54:45.352093 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:45.352066 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/848ee430-f2ec-41a6-aeb4-ac2bd369980f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7b529cf1-z7d5k\" (UID: \"848ee430-f2ec-41a6-aeb4-ac2bd369980f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b529cf1-z7d5k" Nov 03 00:54:45.353747 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:45.353723 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/848ee430-f2ec-41a6-aeb4-ac2bd369980f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7b529cf1-z7d5k\" (UID: \"848ee430-f2ec-41a6-aeb4-ac2bd369980f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b529cf1-z7d5k" Nov 03 00:54:45.361304 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:45.360152 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gh89\" (UniqueName: \"kubernetes.io/projected/848ee430-f2ec-41a6-aeb4-ac2bd369980f-kube-api-access-9gh89\") pod \"maas-api-revoke-sub-7b529cf1-z7d5k\" (UID: \"848ee430-f2ec-41a6-aeb4-ac2bd369980f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b529cf1-z7d5k" Nov 03 00:54:45.410103 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:45.410068 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b529cf1-z7d5k" Nov 03 00:54:45.695706 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:45.695665 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b529cf1-z7d5k"] Nov 03 00:54:46.723545 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:46.722386 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b529cf1-z7d5k" event={"ID":"848ee430-f2ec-41a6-aeb4-ac2bd369980f","Type":"ContainerStarted","Data":"f92827d6d35408a50593a9073cb31be4e570868cf8a5a8ddf16ed0c2fcde204b"} Nov 03 00:54:46.723545 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:46.722470 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b529cf1-z7d5k" event={"ID":"848ee430-f2ec-41a6-aeb4-ac2bd369980f","Type":"ContainerStarted","Data":"097237bbb91efb90661c8b2adfdf85db6de148d96b495490123c0dd1e0474192"} Nov 03 00:54:47.488813 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:47.488749 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b529cf1-z7d5k" podStartSLOduration=2.48872837 podStartE2EDuration="2.48872837s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:54:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:54:46.761012834 +0000 UTC m=+828.932015932" watchObservedRunningTime="2026-11-03 00:54:47.48872837 +0000 UTC m=+829.659731466" Nov 03 00:54:47.534384 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:47.534339 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-tc5dx"] Nov 03 00:54:47.546527 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:47.546490 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-tc5dx"] Nov 03 00:54:47.546867 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:47.546848 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-tc5dx" Nov 03 00:54:47.592349 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:47.569287 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c57e761-5812-43b8-a25f-f983c0dea9b0-credential-socket\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-tc5dx\" (UID: \"3c57e761-5812-43b8-a25f-f983c0dea9b0\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-tc5dx" Nov 03 00:54:47.592349 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:47.569388 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c57e761-5812-43b8-a25f-f983c0dea9b0-istiod-ca-cert\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-tc5dx\" (UID: \"3c57e761-5812-43b8-a25f-f983c0dea9b0\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-tc5dx" Nov 03 00:54:47.592349 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:47.569455 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c57e761-5812-43b8-a25f-f983c0dea9b0-workload-socket\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-tc5dx\" (UID: \"3c57e761-5812-43b8-a25f-f983c0dea9b0\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-tc5dx" Nov 03 00:54:47.592349 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:47.569510 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sr427\" (UniqueName: \"kubernetes.io/projected/3c57e761-5812-43b8-a25f-f983c0dea9b0-kube-api-access-sr427\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-tc5dx\" (UID: \"3c57e761-5812-43b8-a25f-f983c0dea9b0\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-tc5dx" Nov 03 00:54:47.742700 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:47.742597 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c57e761-5812-43b8-a25f-f983c0dea9b0-istio-envoy\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-tc5dx\" (UID: \"3c57e761-5812-43b8-a25f-f983c0dea9b0\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-tc5dx" Nov 03 00:54:47.749299 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:47.744220 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c57e761-5812-43b8-a25f-f983c0dea9b0-istio-token\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-tc5dx\" (UID: \"3c57e761-5812-43b8-a25f-f983c0dea9b0\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-tc5dx" Nov 03 00:54:47.749656 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:47.749625 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c57e761-5812-43b8-a25f-f983c0dea9b0-istio-podinfo\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-tc5dx\" (UID: \"3c57e761-5812-43b8-a25f-f983c0dea9b0\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-tc5dx" Nov 03 00:54:47.889299 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:47.876490 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c57e761-5812-43b8-a25f-f983c0dea9b0-workload-certs\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-tc5dx\" (UID: \"3c57e761-5812-43b8-a25f-f983c0dea9b0\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-tc5dx" Nov 03 00:54:47.889299 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:47.880566 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c57e761-5812-43b8-a25f-f983c0dea9b0-istio-data\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-tc5dx\" (UID: \"3c57e761-5812-43b8-a25f-f983c0dea9b0\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-tc5dx" Nov 03 00:54:47.946720 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:47.946185 4421 generic.go:356] "Generic (PLEG): container finished" podID="848ee430-f2ec-41a6-aeb4-ac2bd369980f" containerID="f92827d6d35408a50593a9073cb31be4e570868cf8a5a8ddf16ed0c2fcde204b" exitCode=0 Nov 03 00:54:47.947067 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:47.946407 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b529cf1-z7d5k" event={"ID":"848ee430-f2ec-41a6-aeb4-ac2bd369980f","Type":"ContainerDied","Data":"f92827d6d35408a50593a9073cb31be4e570868cf8a5a8ddf16ed0c2fcde204b"} Nov 03 00:54:48.017084 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:48.017043 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c57e761-5812-43b8-a25f-f983c0dea9b0-istio-token\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-tc5dx\" (UID: \"3c57e761-5812-43b8-a25f-f983c0dea9b0\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-tc5dx" Nov 03 00:54:48.017404 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:48.017380 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c57e761-5812-43b8-a25f-f983c0dea9b0-istio-podinfo\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-tc5dx\" (UID: \"3c57e761-5812-43b8-a25f-f983c0dea9b0\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-tc5dx" Nov 03 00:54:48.017570 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:48.017553 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c57e761-5812-43b8-a25f-f983c0dea9b0-workload-certs\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-tc5dx\" (UID: \"3c57e761-5812-43b8-a25f-f983c0dea9b0\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-tc5dx" Nov 03 00:54:48.017679 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:48.017664 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c57e761-5812-43b8-a25f-f983c0dea9b0-istio-data\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-tc5dx\" (UID: \"3c57e761-5812-43b8-a25f-f983c0dea9b0\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-tc5dx" Nov 03 00:54:48.017872 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:48.017855 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c57e761-5812-43b8-a25f-f983c0dea9b0-credential-socket\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-tc5dx\" (UID: \"3c57e761-5812-43b8-a25f-f983c0dea9b0\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-tc5dx" Nov 03 00:54:48.018037 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:48.018021 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c57e761-5812-43b8-a25f-f983c0dea9b0-istiod-ca-cert\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-tc5dx\" (UID: \"3c57e761-5812-43b8-a25f-f983c0dea9b0\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-tc5dx" Nov 03 00:54:48.018189 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:48.018173 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c57e761-5812-43b8-a25f-f983c0dea9b0-workload-socket\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-tc5dx\" (UID: \"3c57e761-5812-43b8-a25f-f983c0dea9b0\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-tc5dx" Nov 03 00:54:48.018338 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:48.018322 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sr427\" (UniqueName: \"kubernetes.io/projected/3c57e761-5812-43b8-a25f-f983c0dea9b0-kube-api-access-sr427\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-tc5dx\" (UID: \"3c57e761-5812-43b8-a25f-f983c0dea9b0\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-tc5dx" Nov 03 00:54:48.018488 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:48.018472 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c57e761-5812-43b8-a25f-f983c0dea9b0-istio-envoy\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-tc5dx\" (UID: \"3c57e761-5812-43b8-a25f-f983c0dea9b0\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-tc5dx" Nov 03 00:54:48.061303 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:48.026951 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c57e761-5812-43b8-a25f-f983c0dea9b0-credential-socket\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-tc5dx\" (UID: \"3c57e761-5812-43b8-a25f-f983c0dea9b0\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-tc5dx" Nov 03 00:54:48.061303 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:48.027637 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c57e761-5812-43b8-a25f-f983c0dea9b0-istiod-ca-cert\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-tc5dx\" (UID: \"3c57e761-5812-43b8-a25f-f983c0dea9b0\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-tc5dx" Nov 03 00:54:48.061303 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:48.027893 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c57e761-5812-43b8-a25f-f983c0dea9b0-workload-socket\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-tc5dx\" (UID: \"3c57e761-5812-43b8-a25f-f983c0dea9b0\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-tc5dx" Nov 03 00:54:48.088298 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:48.067838 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c57e761-5812-43b8-a25f-f983c0dea9b0-workload-certs\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-tc5dx\" (UID: \"3c57e761-5812-43b8-a25f-f983c0dea9b0\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-tc5dx" Nov 03 00:54:48.088298 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:48.068400 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c57e761-5812-43b8-a25f-f983c0dea9b0-istio-data\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-tc5dx\" (UID: \"3c57e761-5812-43b8-a25f-f983c0dea9b0\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-tc5dx" Nov 03 00:54:48.089034 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:48.089005 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c57e761-5812-43b8-a25f-f983c0dea9b0-istio-envoy\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-tc5dx\" (UID: \"3c57e761-5812-43b8-a25f-f983c0dea9b0\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-tc5dx" Nov 03 00:54:48.269361 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:48.252444 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8gm8n" podUID="2d367e12-e46f-4a68-9825-9b1c934d4d12" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": context deadline exceeded" Nov 03 00:54:48.273725 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:48.272165 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c57e761-5812-43b8-a25f-f983c0dea9b0-istio-podinfo\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-tc5dx\" (UID: \"3c57e761-5812-43b8-a25f-f983c0dea9b0\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-tc5dx" Nov 03 00:54:48.280319 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:48.274737 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sr427\" (UniqueName: \"kubernetes.io/projected/3c57e761-5812-43b8-a25f-f983c0dea9b0-kube-api-access-sr427\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-tc5dx\" (UID: \"3c57e761-5812-43b8-a25f-f983c0dea9b0\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-tc5dx" Nov 03 00:54:48.361309 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:48.358658 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c57e761-5812-43b8-a25f-f983c0dea9b0-istio-token\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-tc5dx\" (UID: \"3c57e761-5812-43b8-a25f-f983c0dea9b0\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-tc5dx" Nov 03 00:54:48.422309 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:48.420900 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/premium-w3-83755a-kserve-7c9d66988b-lt4xd" podUID="c0c6b02e-2ecb-428e-a13d-be6d7cd1ba39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:54:48.443955 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:48.441153 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/simulator-w3-83755a-kserve-77d6664559-psqtg" podUID="b0a4c234-af58-4da0-a1a4-e1f167953f6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Nov 03 00:54:48.443955 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:48.441370 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/distinct-w3-83755a-kserve-86bdbbdd85-2cnnl" podUID="f4ae9624-d127-4a55-837d-7c9ff0267b37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded" Nov 03 00:54:48.443955 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:48.443697 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-tc5dx" Nov 03 00:54:48.519319 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:48.518302 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-650be0/simulator-w4-650be0-kserve-7bc588774f-gvz2k" podUID="a827fa88-f078-4312-8f12-2c799c11b461" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Nov 03 00:54:48.785969 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:48.785918 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/simulator-w2-f7680d-kserve-85bdc86cd6-8gxrg" podUID="2bdcf521-cebd-4d6b-8554-1b60db8b0f3c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 00:54:48.887307 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:48.886394 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-650be0/premium-w4-650be0-kserve-7bf8555dfb-z97n5" podUID="40601dc5-808f-456f-826f-6e0c35966ada" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 00:54:49.384389 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:49.384363 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b529cf1-z7d5k" Nov 03 00:54:49.384814 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:49.384786 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-tc5dx"] Nov 03 00:54:49.384924 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:54:49.384891 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c57e761_5812_43b8_a25f_f983c0dea9b0.slice/crio-b5fa9a1ce2abf23e50e78d9db3809c58ae6edd7f7fc8d0424053f407d027ca62 WatchSource:0}: Error finding container b5fa9a1ce2abf23e50e78d9db3809c58ae6edd7f7fc8d0424053f407d027ca62: Status 404 returned error can't find the container with id b5fa9a1ce2abf23e50e78d9db3809c58ae6edd7f7fc8d0424053f407d027ca62 Nov 03 00:54:49.387425 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:49.387394 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:54:49.387496 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:49.387465 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:54:49.387572 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:49.387495 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:54:49.427416 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:49.427383 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/848ee430-f2ec-41a6-aeb4-ac2bd369980f-kube-api-access-9gh89\" (UniqueName: \"kubernetes.io/projected/848ee430-f2ec-41a6-aeb4-ac2bd369980f-kube-api-access-9gh89\") pod \"848ee430-f2ec-41a6-aeb4-ac2bd369980f\" (UID: \"848ee430-f2ec-41a6-aeb4-ac2bd369980f\") " Nov 03 00:54:49.427566 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:49.427451 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/848ee430-f2ec-41a6-aeb4-ac2bd369980f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/848ee430-f2ec-41a6-aeb4-ac2bd369980f-maas-api-xks-ca\") pod \"848ee430-f2ec-41a6-aeb4-ac2bd369980f\" (UID: \"848ee430-f2ec-41a6-aeb4-ac2bd369980f\") " Nov 03 00:54:49.427566 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:49.427528 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/848ee430-f2ec-41a6-aeb4-ac2bd369980f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/848ee430-f2ec-41a6-aeb4-ac2bd369980f-maas-api-openshift-ca\") pod \"848ee430-f2ec-41a6-aeb4-ac2bd369980f\" (UID: \"848ee430-f2ec-41a6-aeb4-ac2bd369980f\") " Nov 03 00:54:49.427964 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:49.427930 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/848ee430-f2ec-41a6-aeb4-ac2bd369980f-maas-api-openshift-ca" pod "848ee430-f2ec-41a6-aeb4-ac2bd369980f" (UID: "848ee430-f2ec-41a6-aeb4-ac2bd369980f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:54:49.428174 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:49.428146 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/848ee430-f2ec-41a6-aeb4-ac2bd369980f-maas-api-openshift-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:54:49.430031 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:49.430005 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/848ee430-f2ec-41a6-aeb4-ac2bd369980f-kube-api-access-9gh89" pod "848ee430-f2ec-41a6-aeb4-ac2bd369980f" (UID: "848ee430-f2ec-41a6-aeb4-ac2bd369980f"). InnerVolumeSpecName "kube-api-access-9gh89". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:54:49.430133 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:49.430048 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/848ee430-f2ec-41a6-aeb4-ac2bd369980f-maas-api-xks-ca" pod "848ee430-f2ec-41a6-aeb4-ac2bd369980f" (UID: "848ee430-f2ec-41a6-aeb4-ac2bd369980f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:54:49.529393 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:49.529351 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9gh89\" (UniqueName: \"kubernetes.io/projected/848ee430-f2ec-41a6-aeb4-ac2bd369980f-kube-api-access-9gh89\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:54:49.529393 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:49.529394 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/848ee430-f2ec-41a6-aeb4-ac2bd369980f-maas-api-xks-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:54:50.056864 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:50.056835 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b529cf1-z7d5k" Nov 03 00:54:50.057383 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:50.056915 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b529cf1-z7d5k" event={"ID":"848ee430-f2ec-41a6-aeb4-ac2bd369980f","Type":"ContainerDied","Data":"097237bbb91efb90661c8b2adfdf85db6de148d96b495490123c0dd1e0474192"} Nov 03 00:54:50.057383 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:50.056969 4421 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="097237bbb91efb90661c8b2adfdf85db6de148d96b495490123c0dd1e0474192" Nov 03 00:54:50.060254 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:50.060230 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-tc5dx" event={"ID":"3c57e761-5812-43b8-a25f-f983c0dea9b0","Type":"ContainerStarted","Data":"c229c496285e24181e14e79befb1ee74dc2c0d1ce42f2f3b077fad7442c800ad"} Nov 03 00:54:50.060395 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:50.060299 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-tc5dx" event={"ID":"3c57e761-5812-43b8-a25f-f983c0dea9b0","Type":"ContainerStarted","Data":"b5fa9a1ce2abf23e50e78d9db3809c58ae6edd7f7fc8d0424053f407d027ca62"} Nov 03 00:54:50.087089 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:50.087026 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-tc5dx" podStartSLOduration=3.087005064 podStartE2EDuration="3.087005064s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:54:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:54:50.085243124 +0000 UTC m=+832.256246223" watchObservedRunningTime="2026-11-03 00:54:50.087005064 +0000 UTC m=+832.258008161" Nov 03 00:54:50.456480 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:50.456446 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-tc5dx" Nov 03 00:54:50.458310 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:50.458253 4421 patch_prober.go:29] interesting pod/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-tc5dx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Nov 03 00:54:50.458470 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:50.458335 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-tc5dx" podUID="3c57e761-5812-43b8-a25f-f983c0dea9b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Nov 03 00:54:50.704521 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:50.704483 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-937c6484-5zwbf"] Nov 03 00:54:50.733357 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:50.712750 4421 cpu_manager.go:414] "containerMap: removing container" podUID="848ee430-f2ec-41a6-aeb4-ac2bd369980f" containerName="revoke-keys" Nov 03 00:54:50.733357 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:50.712785 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="848ee430-f2ec-41a6-aeb4-ac2bd369980f" containerName="revoke-keys" Nov 03 00:54:50.733357 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:50.713003 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="848ee430-f2ec-41a6-aeb4-ac2bd369980f" containerName="revoke-keys" Nov 03 00:54:50.743200 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:50.743167 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-937c6484-5zwbf"] Nov 03 00:54:50.743411 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:50.743299 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-937c6484-5zwbf" Nov 03 00:54:50.746686 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:50.746470 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8e3f769-362b-4f1a-993f-2cc73903fa25-maas-api-xks-ca\") pod \"maas-api-revoke-sub-937c6484-5zwbf\" (UID: \"a8e3f769-362b-4f1a-993f-2cc73903fa25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-937c6484-5zwbf" Nov 03 00:54:50.746686 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:50.746575 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h52fk\" (UniqueName: \"kubernetes.io/projected/a8e3f769-362b-4f1a-993f-2cc73903fa25-kube-api-access-h52fk\") pod \"maas-api-revoke-sub-937c6484-5zwbf\" (UID: \"a8e3f769-362b-4f1a-993f-2cc73903fa25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-937c6484-5zwbf" Nov 03 00:54:50.746686 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:50.746667 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8e3f769-362b-4f1a-993f-2cc73903fa25-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-937c6484-5zwbf\" (UID: \"a8e3f769-362b-4f1a-993f-2cc73903fa25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-937c6484-5zwbf" Nov 03 00:54:50.753173 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:50.752541 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 00:54:50.753173 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:50.752613 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:54:50.753173 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:50.752613 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 00:54:50.754216 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:50.754197 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6q4xp\"" Nov 03 00:54:50.847933 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:50.847894 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8e3f769-362b-4f1a-993f-2cc73903fa25-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-937c6484-5zwbf\" (UID: \"a8e3f769-362b-4f1a-993f-2cc73903fa25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-937c6484-5zwbf" Nov 03 00:54:50.848146 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:50.848035 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8e3f769-362b-4f1a-993f-2cc73903fa25-maas-api-xks-ca\") pod \"maas-api-revoke-sub-937c6484-5zwbf\" (UID: \"a8e3f769-362b-4f1a-993f-2cc73903fa25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-937c6484-5zwbf" Nov 03 00:54:50.848146 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:50.848062 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h52fk\" (UniqueName: \"kubernetes.io/projected/a8e3f769-362b-4f1a-993f-2cc73903fa25-kube-api-access-h52fk\") pod \"maas-api-revoke-sub-937c6484-5zwbf\" (UID: \"a8e3f769-362b-4f1a-993f-2cc73903fa25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-937c6484-5zwbf" Nov 03 00:54:50.848681 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:50.848653 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8e3f769-362b-4f1a-993f-2cc73903fa25-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-937c6484-5zwbf\" (UID: \"a8e3f769-362b-4f1a-993f-2cc73903fa25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-937c6484-5zwbf" Nov 03 00:54:50.850483 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:50.850454 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8e3f769-362b-4f1a-993f-2cc73903fa25-maas-api-xks-ca\") pod \"maas-api-revoke-sub-937c6484-5zwbf\" (UID: \"a8e3f769-362b-4f1a-993f-2cc73903fa25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-937c6484-5zwbf" Nov 03 00:54:50.859269 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:50.859238 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h52fk\" (UniqueName: \"kubernetes.io/projected/a8e3f769-362b-4f1a-993f-2cc73903fa25-kube-api-access-h52fk\") pod \"maas-api-revoke-sub-937c6484-5zwbf\" (UID: \"a8e3f769-362b-4f1a-993f-2cc73903fa25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-937c6484-5zwbf" Nov 03 00:54:51.061226 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:51.061190 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-937c6484-5zwbf" Nov 03 00:54:51.381936 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:51.381905 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-937c6484-5zwbf"] Nov 03 00:54:51.383875 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:54:51.383472 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8e3f769_362b_4f1a_993f_2cc73903fa25.slice/crio-3c5df24d75c476cf01468356e2e1fb4b524054f328f416c47023103b500cea61 WatchSource:0}: Error finding container 3c5df24d75c476cf01468356e2e1fb4b524054f328f416c47023103b500cea61: Status 404 returned error can't find the container with id 3c5df24d75c476cf01468356e2e1fb4b524054f328f416c47023103b500cea61 Nov 03 00:54:51.456553 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:51.456520 4421 patch_prober.go:29] interesting pod/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-tc5dx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Nov 03 00:54:51.456696 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:51.456578 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-tc5dx" podUID="3c57e761-5812-43b8-a25f-f983c0dea9b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Nov 03 00:54:52.077554 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:52.077429 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-937c6484-5zwbf" event={"ID":"a8e3f769-362b-4f1a-993f-2cc73903fa25","Type":"ContainerStarted","Data":"5012a85874dc216f7e8893aaea622d83d12dc3c34dc2a82db385f1b55e700cda"} Nov 03 00:54:52.078106 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:52.077586 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-937c6484-5zwbf" event={"ID":"a8e3f769-362b-4f1a-993f-2cc73903fa25","Type":"ContainerStarted","Data":"3c5df24d75c476cf01468356e2e1fb4b524054f328f416c47023103b500cea61"} Nov 03 00:54:52.456383 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:52.456346 4421 patch_prober.go:29] interesting pod/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-tc5dx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Nov 03 00:54:52.456510 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:52.456419 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-tc5dx" podUID="3c57e761-5812-43b8-a25f-f983c0dea9b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Nov 03 00:54:52.533955 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:52.533903 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-937c6484-5zwbf" podStartSLOduration=2.53388874 podStartE2EDuration="2.53388874s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:54:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:54:52.10466372 +0000 UTC m=+834.275666813" watchObservedRunningTime="2026-11-03 00:54:52.53388874 +0000 UTC m=+834.704891836" Nov 03 00:54:52.535686 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:52.535660 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1759d292-np6cp"] Nov 03 00:54:52.539852 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:52.539834 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1759d292-np6cp" Nov 03 00:54:52.548678 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:52.548044 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1759d292-np6cp"] Nov 03 00:54:52.575413 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:52.575379 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pr96c\" (UniqueName: \"kubernetes.io/projected/00ef641b-db94-41d3-8c7e-684daa5d7d67-kube-api-access-pr96c\") pod \"maas-api-revoke-sub-1759d292-np6cp\" (UID: \"00ef641b-db94-41d3-8c7e-684daa5d7d67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1759d292-np6cp" Nov 03 00:54:52.575545 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:52.575527 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00ef641b-db94-41d3-8c7e-684daa5d7d67-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1759d292-np6cp\" (UID: \"00ef641b-db94-41d3-8c7e-684daa5d7d67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1759d292-np6cp" Nov 03 00:54:52.575606 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:52.575592 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00ef641b-db94-41d3-8c7e-684daa5d7d67-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1759d292-np6cp\" (UID: \"00ef641b-db94-41d3-8c7e-684daa5d7d67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1759d292-np6cp" Nov 03 00:54:52.676689 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:52.676597 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00ef641b-db94-41d3-8c7e-684daa5d7d67-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1759d292-np6cp\" (UID: \"00ef641b-db94-41d3-8c7e-684daa5d7d67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1759d292-np6cp" Nov 03 00:54:52.676689 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:52.676655 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00ef641b-db94-41d3-8c7e-684daa5d7d67-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1759d292-np6cp\" (UID: \"00ef641b-db94-41d3-8c7e-684daa5d7d67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1759d292-np6cp" Nov 03 00:54:52.676913 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:52.676782 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pr96c\" (UniqueName: \"kubernetes.io/projected/00ef641b-db94-41d3-8c7e-684daa5d7d67-kube-api-access-pr96c\") pod \"maas-api-revoke-sub-1759d292-np6cp\" (UID: \"00ef641b-db94-41d3-8c7e-684daa5d7d67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1759d292-np6cp" Nov 03 00:54:52.677435 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:52.677410 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00ef641b-db94-41d3-8c7e-684daa5d7d67-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1759d292-np6cp\" (UID: \"00ef641b-db94-41d3-8c7e-684daa5d7d67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1759d292-np6cp" Nov 03 00:54:52.679028 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:52.679006 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00ef641b-db94-41d3-8c7e-684daa5d7d67-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1759d292-np6cp\" (UID: \"00ef641b-db94-41d3-8c7e-684daa5d7d67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1759d292-np6cp" Nov 03 00:54:52.685102 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:52.685078 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pr96c\" (UniqueName: \"kubernetes.io/projected/00ef641b-db94-41d3-8c7e-684daa5d7d67-kube-api-access-pr96c\") pod \"maas-api-revoke-sub-1759d292-np6cp\" (UID: \"00ef641b-db94-41d3-8c7e-684daa5d7d67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1759d292-np6cp" Nov 03 00:54:52.859118 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:52.859079 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1759d292-np6cp" Nov 03 00:54:53.091842 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:53.091809 4421 generic.go:356] "Generic (PLEG): container finished" podID="a8e3f769-362b-4f1a-993f-2cc73903fa25" containerID="5012a85874dc216f7e8893aaea622d83d12dc3c34dc2a82db385f1b55e700cda" exitCode=0 Nov 03 00:54:53.092301 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:53.091973 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-937c6484-5zwbf" event={"ID":"a8e3f769-362b-4f1a-993f-2cc73903fa25","Type":"ContainerDied","Data":"5012a85874dc216f7e8893aaea622d83d12dc3c34dc2a82db385f1b55e700cda"} Nov 03 00:54:53.167039 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:53.167002 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1759d292-np6cp"] Nov 03 00:54:53.167837 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:54:53.167807 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00ef641b_db94_41d3_8c7e_684daa5d7d67.slice/crio-ba4dc3353db5fb350c8c58608659da00277d5c2ef15ad92ba5fc54cb21c91728 WatchSource:0}: Error finding container ba4dc3353db5fb350c8c58608659da00277d5c2ef15ad92ba5fc54cb21c91728: Status 404 returned error can't find the container with id ba4dc3353db5fb350c8c58608659da00277d5c2ef15ad92ba5fc54cb21c91728 Nov 03 00:54:53.457495 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:53.457411 4421 patch_prober.go:29] interesting pod/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-tc5dx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Nov 03 00:54:53.457495 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:53.457465 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-tc5dx" podUID="3c57e761-5812-43b8-a25f-f983c0dea9b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Nov 03 00:54:54.104139 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:54.104014 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1759d292-np6cp" event={"ID":"00ef641b-db94-41d3-8c7e-684daa5d7d67","Type":"ContainerStarted","Data":"6e95a8f5d7f8b6ce32701138996de2565427fafa98af69932d0a557f423ed790"} Nov 03 00:54:54.104139 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:54.104083 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1759d292-np6cp" event={"ID":"00ef641b-db94-41d3-8c7e-684daa5d7d67","Type":"ContainerStarted","Data":"ba4dc3353db5fb350c8c58608659da00277d5c2ef15ad92ba5fc54cb21c91728"} Nov 03 00:54:54.147038 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:54.144414 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1759d292-np6cp" podStartSLOduration=2.144393791 podStartE2EDuration="2.144393791s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:54:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:54:54.141203394 +0000 UTC m=+836.312206507" watchObservedRunningTime="2026-11-03 00:54:54.144393791 +0000 UTC m=+836.315396891" Nov 03 00:54:54.418398 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:54.405381 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-937c6484-5zwbf" Nov 03 00:54:54.456851 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:54.456809 4421 patch_prober.go:29] interesting pod/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-tc5dx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Nov 03 00:54:54.457092 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:54.457069 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-tc5dx" podUID="3c57e761-5812-43b8-a25f-f983c0dea9b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Nov 03 00:54:54.502027 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:54.501994 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8e3f769-362b-4f1a-993f-2cc73903fa25-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8e3f769-362b-4f1a-993f-2cc73903fa25-maas-api-openshift-ca\") pod \"a8e3f769-362b-4f1a-993f-2cc73903fa25\" (UID: \"a8e3f769-362b-4f1a-993f-2cc73903fa25\") " Nov 03 00:54:54.502393 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:54.502342 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8e3f769-362b-4f1a-993f-2cc73903fa25-kube-api-access-h52fk\" (UniqueName: \"kubernetes.io/projected/a8e3f769-362b-4f1a-993f-2cc73903fa25-kube-api-access-h52fk\") pod \"a8e3f769-362b-4f1a-993f-2cc73903fa25\" (UID: \"a8e3f769-362b-4f1a-993f-2cc73903fa25\") " Nov 03 00:54:54.502514 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:54.502501 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a8e3f769-362b-4f1a-993f-2cc73903fa25-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8e3f769-362b-4f1a-993f-2cc73903fa25-maas-api-xks-ca\") pod \"a8e3f769-362b-4f1a-993f-2cc73903fa25\" (UID: \"a8e3f769-362b-4f1a-993f-2cc73903fa25\") " Nov 03 00:54:54.502661 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:54.502637 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8e3f769-362b-4f1a-993f-2cc73903fa25-maas-api-openshift-ca" pod "a8e3f769-362b-4f1a-993f-2cc73903fa25" (UID: "a8e3f769-362b-4f1a-993f-2cc73903fa25"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:54:54.502977 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:54.502957 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8e3f769-362b-4f1a-993f-2cc73903fa25-maas-api-openshift-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:54:54.504496 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:54.504474 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8e3f769-362b-4f1a-993f-2cc73903fa25-maas-api-xks-ca" pod "a8e3f769-362b-4f1a-993f-2cc73903fa25" (UID: "a8e3f769-362b-4f1a-993f-2cc73903fa25"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:54:54.531785 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:54.531749 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8e3f769-362b-4f1a-993f-2cc73903fa25-kube-api-access-h52fk" pod "a8e3f769-362b-4f1a-993f-2cc73903fa25" (UID: "a8e3f769-362b-4f1a-993f-2cc73903fa25"). InnerVolumeSpecName "kube-api-access-h52fk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:54:54.614455 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:54.614353 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h52fk\" (UniqueName: \"kubernetes.io/projected/a8e3f769-362b-4f1a-993f-2cc73903fa25-kube-api-access-h52fk\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:54:54.614455 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:54.614401 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8e3f769-362b-4f1a-993f-2cc73903fa25-maas-api-xks-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:54:55.144716 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:55.144680 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-937c6484-5zwbf" event={"ID":"a8e3f769-362b-4f1a-993f-2cc73903fa25","Type":"ContainerDied","Data":"3c5df24d75c476cf01468356e2e1fb4b524054f328f416c47023103b500cea61"} Nov 03 00:54:55.145211 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:55.144734 4421 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3c5df24d75c476cf01468356e2e1fb4b524054f328f416c47023103b500cea61" Nov 03 00:54:55.145211 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:55.144801 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-937c6484-5zwbf" Nov 03 00:54:55.147532 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:55.147504 4421 generic.go:356] "Generic (PLEG): container finished" podID="00ef641b-db94-41d3-8c7e-684daa5d7d67" containerID="6e95a8f5d7f8b6ce32701138996de2565427fafa98af69932d0a557f423ed790" exitCode=0 Nov 03 00:54:55.147676 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:55.147582 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1759d292-np6cp" event={"ID":"00ef641b-db94-41d3-8c7e-684daa5d7d67","Type":"ContainerDied","Data":"6e95a8f5d7f8b6ce32701138996de2565427fafa98af69932d0a557f423ed790"} Nov 03 00:54:55.456877 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:55.456777 4421 patch_prober.go:29] interesting pod/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-tc5dx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Nov 03 00:54:55.456877 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:55.456837 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-tc5dx" podUID="3c57e761-5812-43b8-a25f-f983c0dea9b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Nov 03 00:54:56.363526 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:56.363481 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1759d292-np6cp" Nov 03 00:54:56.456603 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:56.456520 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00ef641b-db94-41d3-8c7e-684daa5d7d67-kube-api-access-pr96c\" (UniqueName: \"kubernetes.io/projected/00ef641b-db94-41d3-8c7e-684daa5d7d67-kube-api-access-pr96c\") pod \"00ef641b-db94-41d3-8c7e-684daa5d7d67\" (UID: \"00ef641b-db94-41d3-8c7e-684daa5d7d67\") " Nov 03 00:54:56.456766 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:56.456724 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00ef641b-db94-41d3-8c7e-684daa5d7d67-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00ef641b-db94-41d3-8c7e-684daa5d7d67-maas-api-openshift-ca\") pod \"00ef641b-db94-41d3-8c7e-684daa5d7d67\" (UID: \"00ef641b-db94-41d3-8c7e-684daa5d7d67\") " Nov 03 00:54:56.456833 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:56.456813 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/00ef641b-db94-41d3-8c7e-684daa5d7d67-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00ef641b-db94-41d3-8c7e-684daa5d7d67-maas-api-xks-ca\") pod \"00ef641b-db94-41d3-8c7e-684daa5d7d67\" (UID: \"00ef641b-db94-41d3-8c7e-684daa5d7d67\") " Nov 03 00:54:56.457143 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:56.457109 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00ef641b-db94-41d3-8c7e-684daa5d7d67-maas-api-openshift-ca" pod "00ef641b-db94-41d3-8c7e-684daa5d7d67" (UID: "00ef641b-db94-41d3-8c7e-684daa5d7d67"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:54:56.457497 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:56.457472 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00ef641b-db94-41d3-8c7e-684daa5d7d67-maas-api-openshift-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:54:56.458834 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:56.458800 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00ef641b-db94-41d3-8c7e-684daa5d7d67-kube-api-access-pr96c" pod "00ef641b-db94-41d3-8c7e-684daa5d7d67" (UID: "00ef641b-db94-41d3-8c7e-684daa5d7d67"). InnerVolumeSpecName "kube-api-access-pr96c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:54:56.458962 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:56.458885 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00ef641b-db94-41d3-8c7e-684daa5d7d67-maas-api-xks-ca" pod "00ef641b-db94-41d3-8c7e-684daa5d7d67" (UID: "00ef641b-db94-41d3-8c7e-684daa5d7d67"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:54:56.559002 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:56.558959 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pr96c\" (UniqueName: \"kubernetes.io/projected/00ef641b-db94-41d3-8c7e-684daa5d7d67-kube-api-access-pr96c\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:54:56.559002 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:56.558996 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00ef641b-db94-41d3-8c7e-684daa5d7d67-maas-api-xks-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:54:56.760401 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:56.760365 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-tc5dx" Nov 03 00:54:56.761592 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:56.761566 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-tc5dx" Nov 03 00:54:56.762230 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:56.762208 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-tc5dx" Nov 03 00:54:57.166819 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:57.163937 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1759d292-np6cp" Nov 03 00:54:57.166819 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:57.163971 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1759d292-np6cp" event={"ID":"00ef641b-db94-41d3-8c7e-684daa5d7d67","Type":"ContainerDied","Data":"ba4dc3353db5fb350c8c58608659da00277d5c2ef15ad92ba5fc54cb21c91728"} Nov 03 00:54:57.166819 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:57.164073 4421 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ba4dc3353db5fb350c8c58608659da00277d5c2ef15ad92ba5fc54cb21c91728" Nov 03 00:54:58.014695 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:58.014663 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5fb83c19-bnnjc"] Nov 03 00:54:58.015441 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:58.015422 4421 cpu_manager.go:414] "containerMap: removing container" podUID="00ef641b-db94-41d3-8c7e-684daa5d7d67" containerName="revoke-keys" Nov 03 00:54:58.015555 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:58.015442 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00ef641b-db94-41d3-8c7e-684daa5d7d67" containerName="revoke-keys" Nov 03 00:54:58.015555 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:58.015466 4421 cpu_manager.go:414] "containerMap: removing container" podUID="a8e3f769-362b-4f1a-993f-2cc73903fa25" containerName="revoke-keys" Nov 03 00:54:58.015555 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:58.015474 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8e3f769-362b-4f1a-993f-2cc73903fa25" containerName="revoke-keys" Nov 03 00:54:58.015704 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:58.015649 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8e3f769-362b-4f1a-993f-2cc73903fa25" containerName="revoke-keys" Nov 03 00:54:58.015704 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:58.015673 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="00ef641b-db94-41d3-8c7e-684daa5d7d67" containerName="revoke-keys" Nov 03 00:54:58.019424 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:58.019404 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb83c19-bnnjc" Nov 03 00:54:58.026091 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:58.026048 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5fb83c19-bnnjc"] Nov 03 00:54:58.026332 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:58.026237 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:54:58.026332 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:58.026257 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 00:54:58.026485 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:58.026330 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6q4xp\"" Nov 03 00:54:58.026485 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:58.026412 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 00:54:58.081487 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:58.081453 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4tzm\" (UniqueName: \"kubernetes.io/projected/3ca48327-61e7-4ea8-828e-57ce1e72845e-kube-api-access-s4tzm\") pod \"maas-api-revoke-sub-5fb83c19-bnnjc\" (UID: \"3ca48327-61e7-4ea8-828e-57ce1e72845e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb83c19-bnnjc" Nov 03 00:54:58.081678 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:58.081643 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ca48327-61e7-4ea8-828e-57ce1e72845e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5fb83c19-bnnjc\" (UID: \"3ca48327-61e7-4ea8-828e-57ce1e72845e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb83c19-bnnjc" Nov 03 00:54:58.081733 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:58.081696 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ca48327-61e7-4ea8-828e-57ce1e72845e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5fb83c19-bnnjc\" (UID: \"3ca48327-61e7-4ea8-828e-57ce1e72845e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb83c19-bnnjc" Nov 03 00:54:58.182959 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:58.182924 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ca48327-61e7-4ea8-828e-57ce1e72845e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5fb83c19-bnnjc\" (UID: \"3ca48327-61e7-4ea8-828e-57ce1e72845e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb83c19-bnnjc" Nov 03 00:54:58.183148 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:58.183066 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4tzm\" (UniqueName: \"kubernetes.io/projected/3ca48327-61e7-4ea8-828e-57ce1e72845e-kube-api-access-s4tzm\") pod \"maas-api-revoke-sub-5fb83c19-bnnjc\" (UID: \"3ca48327-61e7-4ea8-828e-57ce1e72845e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb83c19-bnnjc" Nov 03 00:54:58.183243 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:58.183227 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ca48327-61e7-4ea8-828e-57ce1e72845e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5fb83c19-bnnjc\" (UID: \"3ca48327-61e7-4ea8-828e-57ce1e72845e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb83c19-bnnjc" Nov 03 00:54:58.183672 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:58.183649 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ca48327-61e7-4ea8-828e-57ce1e72845e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5fb83c19-bnnjc\" (UID: \"3ca48327-61e7-4ea8-828e-57ce1e72845e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb83c19-bnnjc" Nov 03 00:54:58.185587 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:58.185566 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ca48327-61e7-4ea8-828e-57ce1e72845e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5fb83c19-bnnjc\" (UID: \"3ca48327-61e7-4ea8-828e-57ce1e72845e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb83c19-bnnjc" Nov 03 00:54:58.191815 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:58.191789 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4tzm\" (UniqueName: \"kubernetes.io/projected/3ca48327-61e7-4ea8-828e-57ce1e72845e-kube-api-access-s4tzm\") pod \"maas-api-revoke-sub-5fb83c19-bnnjc\" (UID: \"3ca48327-61e7-4ea8-828e-57ce1e72845e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb83c19-bnnjc" Nov 03 00:54:58.339470 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:58.339385 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb83c19-bnnjc" Nov 03 00:54:58.617038 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:58.617007 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5fb83c19-bnnjc"] Nov 03 00:54:58.617177 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:54:58.617013 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ca48327_61e7_4ea8_828e_57ce1e72845e.slice/crio-0a20036a34b474a7a6d2fde5bbb0881caed42350d7475434a4f8703c649fd21b WatchSource:0}: Error finding container 0a20036a34b474a7a6d2fde5bbb0881caed42350d7475434a4f8703c649fd21b: Status 404 returned error can't find the container with id 0a20036a34b474a7a6d2fde5bbb0881caed42350d7475434a4f8703c649fd21b Nov 03 00:54:59.182418 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:59.182307 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb83c19-bnnjc" event={"ID":"3ca48327-61e7-4ea8-828e-57ce1e72845e","Type":"ContainerStarted","Data":"4fcdbd7ece1ecaf43d8c78c867cf761abecdd48fefbbb4524cda5413fff3bc03"} Nov 03 00:54:59.182418 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:59.182368 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb83c19-bnnjc" event={"ID":"3ca48327-61e7-4ea8-828e-57ce1e72845e","Type":"ContainerStarted","Data":"0a20036a34b474a7a6d2fde5bbb0881caed42350d7475434a4f8703c649fd21b"} Nov 03 00:54:59.208158 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:54:59.208103 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb83c19-bnnjc" podStartSLOduration=2.208087091 podStartE2EDuration="2.208087091s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:54:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:54:59.204501172 +0000 UTC m=+841.375504280" watchObservedRunningTime="2026-11-03 00:54:59.208087091 +0000 UTC m=+841.379090187" Nov 03 00:55:00.191098 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:00.191064 4421 generic.go:356] "Generic (PLEG): container finished" podID="3ca48327-61e7-4ea8-828e-57ce1e72845e" containerID="4fcdbd7ece1ecaf43d8c78c867cf761abecdd48fefbbb4524cda5413fff3bc03" exitCode=0 Nov 03 00:55:00.191578 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:00.191149 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb83c19-bnnjc" event={"ID":"3ca48327-61e7-4ea8-828e-57ce1e72845e","Type":"ContainerDied","Data":"4fcdbd7ece1ecaf43d8c78c867cf761abecdd48fefbbb4524cda5413fff3bc03"} Nov 03 00:55:01.258539 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:01.258504 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64f6987b-jwc7c"] Nov 03 00:55:01.327655 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:01.327621 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64f6987b-jwc7c" Nov 03 00:55:01.409302 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:01.408045 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64f6987b-jwc7c"] Nov 03 00:55:01.521357 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:01.482743 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cef01a5-6822-48dd-93b7-77f8d58f6c1b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-64f6987b-jwc7c\" (UID: \"8cef01a5-6822-48dd-93b7-77f8d58f6c1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64f6987b-jwc7c" Nov 03 00:55:01.706316 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:01.692710 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whm7z\" (UniqueName: \"kubernetes.io/projected/8cef01a5-6822-48dd-93b7-77f8d58f6c1b-kube-api-access-whm7z\") pod \"maas-api-revoke-sub-64f6987b-jwc7c\" (UID: \"8cef01a5-6822-48dd-93b7-77f8d58f6c1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64f6987b-jwc7c" Nov 03 00:55:01.706316 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:01.692940 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cef01a5-6822-48dd-93b7-77f8d58f6c1b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-64f6987b-jwc7c\" (UID: \"8cef01a5-6822-48dd-93b7-77f8d58f6c1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64f6987b-jwc7c" Nov 03 00:55:01.829867 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:01.824374 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cef01a5-6822-48dd-93b7-77f8d58f6c1b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-64f6987b-jwc7c\" (UID: \"8cef01a5-6822-48dd-93b7-77f8d58f6c1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64f6987b-jwc7c" Nov 03 00:55:01.829867 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:01.824535 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cef01a5-6822-48dd-93b7-77f8d58f6c1b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-64f6987b-jwc7c\" (UID: \"8cef01a5-6822-48dd-93b7-77f8d58f6c1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64f6987b-jwc7c" Nov 03 00:55:01.829867 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:01.824693 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whm7z\" (UniqueName: \"kubernetes.io/projected/8cef01a5-6822-48dd-93b7-77f8d58f6c1b-kube-api-access-whm7z\") pod \"maas-api-revoke-sub-64f6987b-jwc7c\" (UID: \"8cef01a5-6822-48dd-93b7-77f8d58f6c1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64f6987b-jwc7c" Nov 03 00:55:01.829867 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:01.829381 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cef01a5-6822-48dd-93b7-77f8d58f6c1b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-64f6987b-jwc7c\" (UID: \"8cef01a5-6822-48dd-93b7-77f8d58f6c1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64f6987b-jwc7c" Nov 03 00:55:01.910562 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:01.910528 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whm7z\" (UniqueName: \"kubernetes.io/projected/8cef01a5-6822-48dd-93b7-77f8d58f6c1b-kube-api-access-whm7z\") pod \"maas-api-revoke-sub-64f6987b-jwc7c\" (UID: \"8cef01a5-6822-48dd-93b7-77f8d58f6c1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64f6987b-jwc7c" Nov 03 00:55:01.911232 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:01.911208 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cef01a5-6822-48dd-93b7-77f8d58f6c1b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-64f6987b-jwc7c\" (UID: \"8cef01a5-6822-48dd-93b7-77f8d58f6c1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64f6987b-jwc7c" Nov 03 00:55:01.999927 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:01.999891 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64f6987b-jwc7c" Nov 03 00:55:02.186379 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:02.186335 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/simulator-w3-83755a-kserve-77d6664559-psqtg" podUID="b0a4c234-af58-4da0-a1a4-e1f167953f6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Nov 03 00:55:02.232451 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:02.232401 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/premium-w3-83755a-kserve-7c9d66988b-lt4xd" podUID="c0c6b02e-2ecb-428e-a13d-be6d7cd1ba39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 00:55:02.249515 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:02.248915 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-650be0/simulator-w4-650be0-kserve-7bc588774f-gvz2k" podUID="a827fa88-f078-4312-8f12-2c799c11b461" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:55:02.291527 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:02.291470 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/distinct-w3-83755a-kserve-86bdbbdd85-2cnnl" podUID="f4ae9624-d127-4a55-837d-7c9ff0267b37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 00:55:02.295082 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:02.295047 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8gm8n" podUID="2d367e12-e46f-4a68-9825-9b1c934d4d12" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 00:55:02.718210 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:02.718162 4421 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-83755a/simulator-w3-83755a-kserve-77d6664559-psqtg" podUID="b0a4c234-af58-4da0-a1a4-e1f167953f6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:55:02.960303 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:02.941129 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-650be0/premium-w4-650be0-kserve-7bf8555dfb-z97n5" podUID="40601dc5-808f-456f-826f-6e0c35966ada" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Nov 03 00:55:03.073311 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:03.072375 4421 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-650be0/simulator-w4-650be0-kserve-7bc588774f-gvz2k" podUID="a827fa88-f078-4312-8f12-2c799c11b461" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": context deadline exceeded" Nov 03 00:55:03.112307 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:03.112214 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/simulator-w2-f7680d-kserve-85bdc86cd6-8gxrg" podUID="2bdcf521-cebd-4d6b-8554-1b60db8b0f3c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:55:03.215363 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:03.213710 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/simulator-w3-83755a-kserve-77d6664559-psqtg" podUID="b0a4c234-af58-4da0-a1a4-e1f167953f6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:55:03.215363 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:03.214688 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-83755a/simulator-w3-83755a-kserve-77d6664559-psqtg" Nov 03 00:55:03.233825 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:03.233777 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/premium-w3-83755a-kserve-7c9d66988b-lt4xd" podUID="c0c6b02e-2ecb-428e-a13d-be6d7cd1ba39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:55:03.252308 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:03.243161 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-83755a/premium-w3-83755a-kserve-7c9d66988b-lt4xd" Nov 03 00:55:03.266310 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:03.253213 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-650be0/simulator-w4-650be0-kserve-7bc588774f-gvz2k" podUID="a827fa88-f078-4312-8f12-2c799c11b461" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Nov 03 00:55:03.279307 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:03.267246 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-650be0/simulator-w4-650be0-kserve-7bc588774f-gvz2k" Nov 03 00:55:03.374624 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:03.271784 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb83c19-bnnjc" Nov 03 00:55:03.390495 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:03.302132 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8gm8n" podUID="2d367e12-e46f-4a68-9825-9b1c934d4d12" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:55:03.391063 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:03.302167 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/distinct-w3-83755a-kserve-86bdbbdd85-2cnnl" podUID="f4ae9624-d127-4a55-837d-7c9ff0267b37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded" Nov 03 00:55:03.391632 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:03.391608 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8gm8n" Nov 03 00:55:03.395296 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:03.391823 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-83755a/distinct-w3-83755a-kserve-86bdbbdd85-2cnnl" Nov 03 00:55:03.532305 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:03.510224 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ca48327-61e7-4ea8-828e-57ce1e72845e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ca48327-61e7-4ea8-828e-57ce1e72845e-maas-api-openshift-ca\") pod \"3ca48327-61e7-4ea8-828e-57ce1e72845e\" (UID: \"3ca48327-61e7-4ea8-828e-57ce1e72845e\") " Nov 03 00:55:03.532305 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:03.510367 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ca48327-61e7-4ea8-828e-57ce1e72845e-kube-api-access-s4tzm\" (UniqueName: \"kubernetes.io/projected/3ca48327-61e7-4ea8-828e-57ce1e72845e-kube-api-access-s4tzm\") pod \"3ca48327-61e7-4ea8-828e-57ce1e72845e\" (UID: \"3ca48327-61e7-4ea8-828e-57ce1e72845e\") " Nov 03 00:55:03.532305 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:03.510505 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ca48327-61e7-4ea8-828e-57ce1e72845e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ca48327-61e7-4ea8-828e-57ce1e72845e-maas-api-xks-ca\") pod \"3ca48327-61e7-4ea8-828e-57ce1e72845e\" (UID: \"3ca48327-61e7-4ea8-828e-57ce1e72845e\") " Nov 03 00:55:03.541064 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:03.537538 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ca48327-61e7-4ea8-828e-57ce1e72845e-maas-api-openshift-ca" pod "3ca48327-61e7-4ea8-828e-57ce1e72845e" (UID: "3ca48327-61e7-4ea8-828e-57ce1e72845e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:55:03.946328 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:03.925555 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ca48327-61e7-4ea8-828e-57ce1e72845e-kube-api-access-s4tzm" pod "3ca48327-61e7-4ea8-828e-57ce1e72845e" (UID: "3ca48327-61e7-4ea8-828e-57ce1e72845e"). InnerVolumeSpecName "kube-api-access-s4tzm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:55:04.057651 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:03.967871 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ca48327-61e7-4ea8-828e-57ce1e72845e-maas-api-xks-ca" pod "3ca48327-61e7-4ea8-828e-57ce1e72845e" (UID: "3ca48327-61e7-4ea8-828e-57ce1e72845e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:55:04.057651 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:03.961475 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ca48327-61e7-4ea8-828e-57ce1e72845e-maas-api-openshift-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:55:04.118812 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:04.118759 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-650be0/premium-w4-650be0-kserve-7bf8555dfb-z97n5" podUID="40601dc5-808f-456f-826f-6e0c35966ada" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:55:04.232731 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:04.232633 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4tzm\" (UniqueName: \"kubernetes.io/projected/3ca48327-61e7-4ea8-828e-57ce1e72845e-kube-api-access-s4tzm\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:55:04.233011 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:04.232989 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ca48327-61e7-4ea8-828e-57ce1e72845e-maas-api-xks-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:55:04.234913 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:04.234870 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-5dqn2" podUID="0a54c4af-a899-4f16-83a8-8718b0c7eef5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 00:55:04.262295 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:04.247711 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/premium-w3-83755a-kserve-7c9d66988b-lt4xd" podUID="c0c6b02e-2ecb-428e-a13d-be6d7cd1ba39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Nov 03 00:55:04.378307 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:04.324593 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/simulator-w3-83755a-kserve-77d6664559-psqtg" podUID="b0a4c234-af58-4da0-a1a4-e1f167953f6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Nov 03 00:55:04.393935 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:04.388834 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/simulator-w2-f7680d-kserve-85bdc86cd6-8gxrg" podUID="2bdcf521-cebd-4d6b-8554-1b60db8b0f3c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 03 00:55:04.446452 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:04.446398 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8gm8n" podUID="2d367e12-e46f-4a68-9825-9b1c934d4d12" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 00:55:04.536007 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:04.535961 4421 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-650be0/premium-w4-650be0-kserve-7bf8555dfb-z97n5" podUID="40601dc5-808f-456f-826f-6e0c35966ada" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": context deadline exceeded" Nov 03 00:55:04.544264 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:04.544237 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb83c19-bnnjc" Nov 03 00:55:04.607954 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:04.591236 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb83c19-bnnjc" event={"ID":"3ca48327-61e7-4ea8-828e-57ce1e72845e","Type":"ContainerDied","Data":"0a20036a34b474a7a6d2fde5bbb0881caed42350d7475434a4f8703c649fd21b"} Nov 03 00:55:04.607954 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:04.591311 4421 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0a20036a34b474a7a6d2fde5bbb0881caed42350d7475434a4f8703c649fd21b" Nov 03 00:55:04.607954 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:04.591389 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-650be0/premium-w4-650be0-kserve-7bf8555dfb-z97n5" Nov 03 00:55:04.607954 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:04.591435 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-650be0/simulator-w4-650be0-kserve-7bc588774f-gvz2k" Nov 03 00:55:04.607954 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:04.591476 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-83755a/distinct-w3-83755a-kserve-86bdbbdd85-2cnnl" Nov 03 00:55:04.607954 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:04.591505 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-650be0/premium-w4-650be0-kserve-7bf8555dfb-z97n5" Nov 03 00:55:04.607954 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:04.591538 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f7680d/simulator-w2-f7680d-kserve-85bdc86cd6-8gxrg" Nov 03 00:55:04.607954 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:04.591816 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8gm8n" Nov 03 00:55:04.607954 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:04.591845 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64f6987b-jwc7c"] Nov 03 00:55:04.607954 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:04.591886 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-45019300-xjtwx"] Nov 03 00:55:04.607954 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:04.592784 4421 cpu_manager.go:414] "containerMap: removing container" podUID="3ca48327-61e7-4ea8-828e-57ce1e72845e" containerName="revoke-keys" Nov 03 00:55:04.607954 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:04.592804 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ca48327-61e7-4ea8-828e-57ce1e72845e" containerName="revoke-keys" Nov 03 00:55:04.607954 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:04.592958 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ca48327-61e7-4ea8-828e-57ce1e72845e" containerName="revoke-keys" Nov 03 00:55:04.609107 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:04.608954 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f7680d/simulator-w2-f7680d-kserve-85bdc86cd6-8gxrg" Nov 03 00:55:04.609107 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:04.608992 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-45019300-xjtwx"] Nov 03 00:55:04.609107 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:04.609057 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-83755a/simulator-w3-83755a-kserve-77d6664559-psqtg" Nov 03 00:55:04.609107 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:04.609094 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-83755a/premium-w3-83755a-kserve-7c9d66988b-lt4xd" Nov 03 00:55:04.609354 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:04.609340 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45019300-xjtwx" Nov 03 00:55:04.671692 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:04.671643 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9af830e-c054-4944-91d1-0b96e9a90f7c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-45019300-xjtwx\" (UID: \"d9af830e-c054-4944-91d1-0b96e9a90f7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45019300-xjtwx" Nov 03 00:55:04.671874 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:04.671766 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jljr\" (UniqueName: \"kubernetes.io/projected/d9af830e-c054-4944-91d1-0b96e9a90f7c-kube-api-access-9jljr\") pod \"maas-api-revoke-sub-45019300-xjtwx\" (UID: \"d9af830e-c054-4944-91d1-0b96e9a90f7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45019300-xjtwx" Nov 03 00:55:04.674518 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:04.671954 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9af830e-c054-4944-91d1-0b96e9a90f7c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-45019300-xjtwx\" (UID: \"d9af830e-c054-4944-91d1-0b96e9a90f7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45019300-xjtwx" Nov 03 00:55:04.773138 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:04.773103 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jljr\" (UniqueName: \"kubernetes.io/projected/d9af830e-c054-4944-91d1-0b96e9a90f7c-kube-api-access-9jljr\") pod \"maas-api-revoke-sub-45019300-xjtwx\" (UID: \"d9af830e-c054-4944-91d1-0b96e9a90f7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45019300-xjtwx" Nov 03 00:55:04.773436 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:04.773228 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9af830e-c054-4944-91d1-0b96e9a90f7c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-45019300-xjtwx\" (UID: \"d9af830e-c054-4944-91d1-0b96e9a90f7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45019300-xjtwx" Nov 03 00:55:04.773522 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:04.773453 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9af830e-c054-4944-91d1-0b96e9a90f7c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-45019300-xjtwx\" (UID: \"d9af830e-c054-4944-91d1-0b96e9a90f7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45019300-xjtwx" Nov 03 00:55:04.774399 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:04.774370 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9af830e-c054-4944-91d1-0b96e9a90f7c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-45019300-xjtwx\" (UID: \"d9af830e-c054-4944-91d1-0b96e9a90f7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45019300-xjtwx" Nov 03 00:55:04.775687 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:04.775659 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9af830e-c054-4944-91d1-0b96e9a90f7c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-45019300-xjtwx\" (UID: \"d9af830e-c054-4944-91d1-0b96e9a90f7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45019300-xjtwx" Nov 03 00:55:04.784600 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:04.784556 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jljr\" (UniqueName: \"kubernetes.io/projected/d9af830e-c054-4944-91d1-0b96e9a90f7c-kube-api-access-9jljr\") pod \"maas-api-revoke-sub-45019300-xjtwx\" (UID: \"d9af830e-c054-4944-91d1-0b96e9a90f7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45019300-xjtwx" Nov 03 00:55:04.960738 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:04.960695 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45019300-xjtwx" Nov 03 00:55:05.246657 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:55:05.246625 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9af830e_c054_4944_91d1_0b96e9a90f7c.slice/crio-8e593622bb4aa47e99a6320c7ae7074b31563f03216b576d750a6b1643e6217b WatchSource:0}: Error finding container 8e593622bb4aa47e99a6320c7ae7074b31563f03216b576d750a6b1643e6217b: Status 404 returned error can't find the container with id 8e593622bb4aa47e99a6320c7ae7074b31563f03216b576d750a6b1643e6217b Nov 03 00:55:05.249033 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:05.249000 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-45019300-xjtwx"] Nov 03 00:55:05.278230 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:05.278193 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3dc652a0-mzw8v"] Nov 03 00:55:05.284003 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:05.283983 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dc652a0-mzw8v" Nov 03 00:55:05.291604 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:05.291583 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3dc652a0-mzw8v"] Nov 03 00:55:05.382341 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:05.382308 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e4f1a8e-7ab7-45e3-8057-29a447b79a28-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3dc652a0-mzw8v\" (UID: \"8e4f1a8e-7ab7-45e3-8057-29a447b79a28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dc652a0-mzw8v" Nov 03 00:55:05.382740 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:05.382450 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e4f1a8e-7ab7-45e3-8057-29a447b79a28-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3dc652a0-mzw8v\" (UID: \"8e4f1a8e-7ab7-45e3-8057-29a447b79a28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dc652a0-mzw8v" Nov 03 00:55:05.382740 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:05.382582 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwjrm\" (UniqueName: \"kubernetes.io/projected/8e4f1a8e-7ab7-45e3-8057-29a447b79a28-kube-api-access-kwjrm\") pod \"maas-api-revoke-sub-3dc652a0-mzw8v\" (UID: \"8e4f1a8e-7ab7-45e3-8057-29a447b79a28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dc652a0-mzw8v" Nov 03 00:55:05.493555 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:05.493467 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwjrm\" (UniqueName: \"kubernetes.io/projected/8e4f1a8e-7ab7-45e3-8057-29a447b79a28-kube-api-access-kwjrm\") pod \"maas-api-revoke-sub-3dc652a0-mzw8v\" (UID: \"8e4f1a8e-7ab7-45e3-8057-29a447b79a28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dc652a0-mzw8v" Nov 03 00:55:05.493724 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:05.493600 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e4f1a8e-7ab7-45e3-8057-29a447b79a28-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3dc652a0-mzw8v\" (UID: \"8e4f1a8e-7ab7-45e3-8057-29a447b79a28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dc652a0-mzw8v" Nov 03 00:55:05.493724 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:05.493709 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e4f1a8e-7ab7-45e3-8057-29a447b79a28-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3dc652a0-mzw8v\" (UID: \"8e4f1a8e-7ab7-45e3-8057-29a447b79a28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dc652a0-mzw8v" Nov 03 00:55:05.494615 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:05.494580 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e4f1a8e-7ab7-45e3-8057-29a447b79a28-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3dc652a0-mzw8v\" (UID: \"8e4f1a8e-7ab7-45e3-8057-29a447b79a28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dc652a0-mzw8v" Nov 03 00:55:05.497435 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:05.497410 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e4f1a8e-7ab7-45e3-8057-29a447b79a28-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3dc652a0-mzw8v\" (UID: \"8e4f1a8e-7ab7-45e3-8057-29a447b79a28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dc652a0-mzw8v" Nov 03 00:55:05.503624 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:05.503600 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwjrm\" (UniqueName: \"kubernetes.io/projected/8e4f1a8e-7ab7-45e3-8057-29a447b79a28-kube-api-access-kwjrm\") pod \"maas-api-revoke-sub-3dc652a0-mzw8v\" (UID: \"8e4f1a8e-7ab7-45e3-8057-29a447b79a28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dc652a0-mzw8v" Nov 03 00:55:05.555593 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:05.554985 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45019300-xjtwx" event={"ID":"d9af830e-c054-4944-91d1-0b96e9a90f7c","Type":"ContainerStarted","Data":"29083d52d0a8d8c8cf5d78ba7f9d91d237512c5a746199b20f8868990efced29"} Nov 03 00:55:05.555593 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:05.555061 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45019300-xjtwx" event={"ID":"d9af830e-c054-4944-91d1-0b96e9a90f7c","Type":"ContainerStarted","Data":"8e593622bb4aa47e99a6320c7ae7074b31563f03216b576d750a6b1643e6217b"} Nov 03 00:55:05.558858 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:05.558819 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64f6987b-jwc7c" event={"ID":"8cef01a5-6822-48dd-93b7-77f8d58f6c1b","Type":"ContainerStarted","Data":"0fe4d66becec984da80267f73baf1cf8a7b8d9be8b15963dd7c694ed47cf0c45"} Nov 03 00:55:05.559120 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:05.559101 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64f6987b-jwc7c" event={"ID":"8cef01a5-6822-48dd-93b7-77f8d58f6c1b","Type":"ContainerStarted","Data":"26e7e75f1c6f2e4874e8438472b89f7a2ba54e6de46b538ccc1302f0123fdd8b"} Nov 03 00:55:05.577343 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:05.575040 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45019300-xjtwx" podStartSLOduration=2.575023384 podStartE2EDuration="2.575023384s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:55:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:55:05.571925741 +0000 UTC m=+847.742928836" watchObservedRunningTime="2026-11-03 00:55:05.575023384 +0000 UTC m=+847.746026471" Nov 03 00:55:05.592548 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:05.592018 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64f6987b-jwc7c" podStartSLOduration=4.591999286 podStartE2EDuration="4.591999286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:55:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:55:05.589468957 +0000 UTC m=+847.760472055" watchObservedRunningTime="2026-11-03 00:55:05.591999286 +0000 UTC m=+847.763002383" Nov 03 00:55:05.607265 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:05.607240 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dc652a0-mzw8v" Nov 03 00:55:05.894926 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:05.894890 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3dc652a0-mzw8v"] Nov 03 00:55:05.895080 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:55:05.895054 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e4f1a8e_7ab7_45e3_8057_29a447b79a28.slice/crio-78874e14de83680b9d511b1be83d47e16dede93b363d440b42176b7ce58d977c WatchSource:0}: Error finding container 78874e14de83680b9d511b1be83d47e16dede93b363d440b42176b7ce58d977c: Status 404 returned error can't find the container with id 78874e14de83680b9d511b1be83d47e16dede93b363d440b42176b7ce58d977c Nov 03 00:55:06.567620 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:06.567576 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dc652a0-mzw8v" event={"ID":"8e4f1a8e-7ab7-45e3-8057-29a447b79a28","Type":"ContainerStarted","Data":"45b723ad0c55830daa39637d038e3a4b886001a5af48a72c2e24c5737452e8ed"} Nov 03 00:55:06.568087 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:06.567679 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dc652a0-mzw8v" event={"ID":"8e4f1a8e-7ab7-45e3-8057-29a447b79a28","Type":"ContainerStarted","Data":"78874e14de83680b9d511b1be83d47e16dede93b363d440b42176b7ce58d977c"} Nov 03 00:55:06.569560 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:06.569533 4421 generic.go:356] "Generic (PLEG): container finished" podID="d9af830e-c054-4944-91d1-0b96e9a90f7c" containerID="29083d52d0a8d8c8cf5d78ba7f9d91d237512c5a746199b20f8868990efced29" exitCode=0 Nov 03 00:55:06.569715 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:06.569625 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45019300-xjtwx" event={"ID":"d9af830e-c054-4944-91d1-0b96e9a90f7c","Type":"ContainerDied","Data":"29083d52d0a8d8c8cf5d78ba7f9d91d237512c5a746199b20f8868990efced29"} Nov 03 00:55:06.571630 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:06.571604 4421 generic.go:356] "Generic (PLEG): container finished" podID="8cef01a5-6822-48dd-93b7-77f8d58f6c1b" containerID="0fe4d66becec984da80267f73baf1cf8a7b8d9be8b15963dd7c694ed47cf0c45" exitCode=0 Nov 03 00:55:06.571749 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:06.571643 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64f6987b-jwc7c" event={"ID":"8cef01a5-6822-48dd-93b7-77f8d58f6c1b","Type":"ContainerDied","Data":"0fe4d66becec984da80267f73baf1cf8a7b8d9be8b15963dd7c694ed47cf0c45"} Nov 03 00:55:06.587984 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:06.587934 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dc652a0-mzw8v" podStartSLOduration=1.587920189 podStartE2EDuration="1.587920189s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:55:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:55:06.584346608 +0000 UTC m=+848.755349706" watchObservedRunningTime="2026-11-03 00:55:06.587920189 +0000 UTC m=+848.758923285" Nov 03 00:55:07.583173 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:07.582422 4421 generic.go:356] "Generic (PLEG): container finished" podID="8e4f1a8e-7ab7-45e3-8057-29a447b79a28" containerID="45b723ad0c55830daa39637d038e3a4b886001a5af48a72c2e24c5737452e8ed" exitCode=0 Nov 03 00:55:07.583173 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:07.582608 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dc652a0-mzw8v" event={"ID":"8e4f1a8e-7ab7-45e3-8057-29a447b79a28","Type":"ContainerDied","Data":"45b723ad0c55830daa39637d038e3a4b886001a5af48a72c2e24c5737452e8ed"} Nov 03 00:55:07.775510 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:07.775482 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45019300-xjtwx" Nov 03 00:55:07.778900 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:07.778872 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64f6987b-jwc7c" Nov 03 00:55:07.943863 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:07.943777 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8cef01a5-6822-48dd-93b7-77f8d58f6c1b-kube-api-access-whm7z\" (UniqueName: \"kubernetes.io/projected/8cef01a5-6822-48dd-93b7-77f8d58f6c1b-kube-api-access-whm7z\") pod \"8cef01a5-6822-48dd-93b7-77f8d58f6c1b\" (UID: \"8cef01a5-6822-48dd-93b7-77f8d58f6c1b\") " Nov 03 00:55:07.943863 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:07.943827 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9af830e-c054-4944-91d1-0b96e9a90f7c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9af830e-c054-4944-91d1-0b96e9a90f7c-maas-api-openshift-ca\") pod \"d9af830e-c054-4944-91d1-0b96e9a90f7c\" (UID: \"d9af830e-c054-4944-91d1-0b96e9a90f7c\") " Nov 03 00:55:07.944078 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:07.943919 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9af830e-c054-4944-91d1-0b96e9a90f7c-kube-api-access-9jljr\" (UniqueName: \"kubernetes.io/projected/d9af830e-c054-4944-91d1-0b96e9a90f7c-kube-api-access-9jljr\") pod \"d9af830e-c054-4944-91d1-0b96e9a90f7c\" (UID: \"d9af830e-c054-4944-91d1-0b96e9a90f7c\") " Nov 03 00:55:07.944143 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:07.944075 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8cef01a5-6822-48dd-93b7-77f8d58f6c1b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cef01a5-6822-48dd-93b7-77f8d58f6c1b-maas-api-xks-ca\") pod \"8cef01a5-6822-48dd-93b7-77f8d58f6c1b\" (UID: \"8cef01a5-6822-48dd-93b7-77f8d58f6c1b\") " Nov 03 00:55:07.944200 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:07.944158 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8cef01a5-6822-48dd-93b7-77f8d58f6c1b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cef01a5-6822-48dd-93b7-77f8d58f6c1b-maas-api-openshift-ca\") pod \"8cef01a5-6822-48dd-93b7-77f8d58f6c1b\" (UID: \"8cef01a5-6822-48dd-93b7-77f8d58f6c1b\") " Nov 03 00:55:07.944200 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:07.944191 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d9af830e-c054-4944-91d1-0b96e9a90f7c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9af830e-c054-4944-91d1-0b96e9a90f7c-maas-api-xks-ca\") pod \"d9af830e-c054-4944-91d1-0b96e9a90f7c\" (UID: \"d9af830e-c054-4944-91d1-0b96e9a90f7c\") " Nov 03 00:55:07.944344 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:07.944254 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9af830e-c054-4944-91d1-0b96e9a90f7c-maas-api-openshift-ca" pod "d9af830e-c054-4944-91d1-0b96e9a90f7c" (UID: "d9af830e-c054-4944-91d1-0b96e9a90f7c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:55:07.944638 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:07.944608 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cef01a5-6822-48dd-93b7-77f8d58f6c1b-maas-api-openshift-ca" pod "8cef01a5-6822-48dd-93b7-77f8d58f6c1b" (UID: "8cef01a5-6822-48dd-93b7-77f8d58f6c1b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:55:07.945159 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:07.945131 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cef01a5-6822-48dd-93b7-77f8d58f6c1b-maas-api-openshift-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:55:07.945257 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:07.945165 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9af830e-c054-4944-91d1-0b96e9a90f7c-maas-api-openshift-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:55:07.946359 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:07.946257 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cef01a5-6822-48dd-93b7-77f8d58f6c1b-kube-api-access-whm7z" pod "8cef01a5-6822-48dd-93b7-77f8d58f6c1b" (UID: "8cef01a5-6822-48dd-93b7-77f8d58f6c1b"). InnerVolumeSpecName "kube-api-access-whm7z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:55:07.946359 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:07.946264 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8cef01a5-6822-48dd-93b7-77f8d58f6c1b-maas-api-xks-ca" pod "8cef01a5-6822-48dd-93b7-77f8d58f6c1b" (UID: "8cef01a5-6822-48dd-93b7-77f8d58f6c1b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:55:07.946359 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:07.946309 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9af830e-c054-4944-91d1-0b96e9a90f7c-kube-api-access-9jljr" pod "d9af830e-c054-4944-91d1-0b96e9a90f7c" (UID: "d9af830e-c054-4944-91d1-0b96e9a90f7c"). InnerVolumeSpecName "kube-api-access-9jljr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:55:07.946359 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:07.946339 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9af830e-c054-4944-91d1-0b96e9a90f7c-maas-api-xks-ca" pod "d9af830e-c054-4944-91d1-0b96e9a90f7c" (UID: "d9af830e-c054-4944-91d1-0b96e9a90f7c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:55:08.045942 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:08.045912 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cef01a5-6822-48dd-93b7-77f8d58f6c1b-maas-api-xks-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:55:08.045942 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:08.045939 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9af830e-c054-4944-91d1-0b96e9a90f7c-maas-api-xks-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:55:08.045942 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:08.045949 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whm7z\" (UniqueName: \"kubernetes.io/projected/8cef01a5-6822-48dd-93b7-77f8d58f6c1b-kube-api-access-whm7z\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:55:08.046169 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:08.045960 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jljr\" (UniqueName: \"kubernetes.io/projected/d9af830e-c054-4944-91d1-0b96e9a90f7c-kube-api-access-9jljr\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:55:08.593030 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:08.592990 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45019300-xjtwx" event={"ID":"d9af830e-c054-4944-91d1-0b96e9a90f7c","Type":"ContainerDied","Data":"8e593622bb4aa47e99a6320c7ae7074b31563f03216b576d750a6b1643e6217b"} Nov 03 00:55:08.593503 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:08.593062 4421 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8e593622bb4aa47e99a6320c7ae7074b31563f03216b576d750a6b1643e6217b" Nov 03 00:55:08.593503 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:08.593002 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45019300-xjtwx" Nov 03 00:55:08.594642 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:08.594618 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64f6987b-jwc7c" Nov 03 00:55:08.594750 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:08.594636 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64f6987b-jwc7c" event={"ID":"8cef01a5-6822-48dd-93b7-77f8d58f6c1b","Type":"ContainerDied","Data":"26e7e75f1c6f2e4874e8438472b89f7a2ba54e6de46b538ccc1302f0123fdd8b"} Nov 03 00:55:08.594750 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:08.594678 4421 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="26e7e75f1c6f2e4874e8438472b89f7a2ba54e6de46b538ccc1302f0123fdd8b" Nov 03 00:55:08.722633 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:08.722608 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dc652a0-mzw8v" Nov 03 00:55:08.858808 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:08.858723 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e4f1a8e-7ab7-45e3-8057-29a447b79a28-kube-api-access-kwjrm\" (UniqueName: \"kubernetes.io/projected/8e4f1a8e-7ab7-45e3-8057-29a447b79a28-kube-api-access-kwjrm\") pod \"8e4f1a8e-7ab7-45e3-8057-29a447b79a28\" (UID: \"8e4f1a8e-7ab7-45e3-8057-29a447b79a28\") " Nov 03 00:55:08.858974 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:08.858884 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e4f1a8e-7ab7-45e3-8057-29a447b79a28-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e4f1a8e-7ab7-45e3-8057-29a447b79a28-maas-api-xks-ca\") pod \"8e4f1a8e-7ab7-45e3-8057-29a447b79a28\" (UID: \"8e4f1a8e-7ab7-45e3-8057-29a447b79a28\") " Nov 03 00:55:08.859067 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:08.859053 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e4f1a8e-7ab7-45e3-8057-29a447b79a28-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e4f1a8e-7ab7-45e3-8057-29a447b79a28-maas-api-openshift-ca\") pod \"8e4f1a8e-7ab7-45e3-8057-29a447b79a28\" (UID: \"8e4f1a8e-7ab7-45e3-8057-29a447b79a28\") " Nov 03 00:55:08.859489 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:08.859462 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e4f1a8e-7ab7-45e3-8057-29a447b79a28-maas-api-openshift-ca" pod "8e4f1a8e-7ab7-45e3-8057-29a447b79a28" (UID: "8e4f1a8e-7ab7-45e3-8057-29a447b79a28"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:55:08.859656 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:08.859625 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e4f1a8e-7ab7-45e3-8057-29a447b79a28-maas-api-openshift-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:55:08.860941 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:08.860920 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e4f1a8e-7ab7-45e3-8057-29a447b79a28-kube-api-access-kwjrm" pod "8e4f1a8e-7ab7-45e3-8057-29a447b79a28" (UID: "8e4f1a8e-7ab7-45e3-8057-29a447b79a28"). InnerVolumeSpecName "kube-api-access-kwjrm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:55:08.861071 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:08.861056 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e4f1a8e-7ab7-45e3-8057-29a447b79a28-maas-api-xks-ca" pod "8e4f1a8e-7ab7-45e3-8057-29a447b79a28" (UID: "8e4f1a8e-7ab7-45e3-8057-29a447b79a28"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:55:08.962169 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:08.962134 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwjrm\" (UniqueName: \"kubernetes.io/projected/8e4f1a8e-7ab7-45e3-8057-29a447b79a28-kube-api-access-kwjrm\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:55:08.962422 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:08.962318 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e4f1a8e-7ab7-45e3-8057-29a447b79a28-maas-api-xks-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:55:09.602797 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:09.602771 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dc652a0-mzw8v" Nov 03 00:55:09.603222 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:09.602809 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dc652a0-mzw8v" event={"ID":"8e4f1a8e-7ab7-45e3-8057-29a447b79a28","Type":"ContainerDied","Data":"78874e14de83680b9d511b1be83d47e16dede93b363d440b42176b7ce58d977c"} Nov 03 00:55:09.603222 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:09.602863 4421 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="78874e14de83680b9d511b1be83d47e16dede93b363d440b42176b7ce58d977c" Nov 03 00:55:11.711210 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:11.711123 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-dd03e395-openshift-default-75799f6fc-kx54z"] Nov 03 00:55:11.712097 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:11.712072 4421 cpu_manager.go:414] "containerMap: removing container" podUID="d9af830e-c054-4944-91d1-0b96e9a90f7c" containerName="revoke-keys" Nov 03 00:55:11.712097 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:11.712097 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9af830e-c054-4944-91d1-0b96e9a90f7c" containerName="revoke-keys" Nov 03 00:55:11.712259 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:11.712138 4421 cpu_manager.go:414] "containerMap: removing container" podUID="8cef01a5-6822-48dd-93b7-77f8d58f6c1b" containerName="revoke-keys" Nov 03 00:55:11.712259 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:11.712146 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8cef01a5-6822-48dd-93b7-77f8d58f6c1b" containerName="revoke-keys" Nov 03 00:55:11.712259 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:11.712156 4421 cpu_manager.go:414] "containerMap: removing container" podUID="8e4f1a8e-7ab7-45e3-8057-29a447b79a28" containerName="revoke-keys" Nov 03 00:55:11.712259 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:11.712165 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e4f1a8e-7ab7-45e3-8057-29a447b79a28" containerName="revoke-keys" Nov 03 00:55:11.712422 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:11.712359 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9af830e-c054-4944-91d1-0b96e9a90f7c" containerName="revoke-keys" Nov 03 00:55:11.712422 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:11.712379 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="8cef01a5-6822-48dd-93b7-77f8d58f6c1b" containerName="revoke-keys" Nov 03 00:55:11.712422 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:11.712386 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e4f1a8e-7ab7-45e3-8057-29a447b79a28" containerName="revoke-keys" Nov 03 00:55:11.717252 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:11.717224 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-dd03e395-openshift-default-75799f6fc-kx54z" Nov 03 00:55:11.721780 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:11.721754 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-dd03e395-openshift-default-dockercfg-9rst4\"" Nov 03 00:55:11.726699 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:11.726672 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-dd03e395-openshift-default-75799f6fc-kx54z"] Nov 03 00:55:11.905654 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:11.905612 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01442ff5-1dab-4ed3-991b-ec0732c71305-workload-certs\") pod \"e2e-mt-dd03e395-openshift-default-75799f6fc-kx54z\" (UID: \"01442ff5-1dab-4ed3-991b-ec0732c71305\") " pod="openshift-ingress/e2e-mt-dd03e395-openshift-default-75799f6fc-kx54z" Nov 03 00:55:11.906000 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:11.905977 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01442ff5-1dab-4ed3-991b-ec0732c71305-istio-data\") pod \"e2e-mt-dd03e395-openshift-default-75799f6fc-kx54z\" (UID: \"01442ff5-1dab-4ed3-991b-ec0732c71305\") " pod="openshift-ingress/e2e-mt-dd03e395-openshift-default-75799f6fc-kx54z" Nov 03 00:55:11.906320 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:11.906287 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01442ff5-1dab-4ed3-991b-ec0732c71305-istiod-ca-cert\") pod \"e2e-mt-dd03e395-openshift-default-75799f6fc-kx54z\" (UID: \"01442ff5-1dab-4ed3-991b-ec0732c71305\") " pod="openshift-ingress/e2e-mt-dd03e395-openshift-default-75799f6fc-kx54z" Nov 03 00:55:11.906566 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:11.906547 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01442ff5-1dab-4ed3-991b-ec0732c71305-credential-socket\") pod \"e2e-mt-dd03e395-openshift-default-75799f6fc-kx54z\" (UID: \"01442ff5-1dab-4ed3-991b-ec0732c71305\") " pod="openshift-ingress/e2e-mt-dd03e395-openshift-default-75799f6fc-kx54z" Nov 03 00:55:11.906782 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:11.906761 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01442ff5-1dab-4ed3-991b-ec0732c71305-istio-token\") pod \"e2e-mt-dd03e395-openshift-default-75799f6fc-kx54z\" (UID: \"01442ff5-1dab-4ed3-991b-ec0732c71305\") " pod="openshift-ingress/e2e-mt-dd03e395-openshift-default-75799f6fc-kx54z" Nov 03 00:55:11.906936 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:11.906914 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01442ff5-1dab-4ed3-991b-ec0732c71305-workload-socket\") pod \"e2e-mt-dd03e395-openshift-default-75799f6fc-kx54z\" (UID: \"01442ff5-1dab-4ed3-991b-ec0732c71305\") " pod="openshift-ingress/e2e-mt-dd03e395-openshift-default-75799f6fc-kx54z" Nov 03 00:55:11.907081 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:11.907067 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01442ff5-1dab-4ed3-991b-ec0732c71305-istio-envoy\") pod \"e2e-mt-dd03e395-openshift-default-75799f6fc-kx54z\" (UID: \"01442ff5-1dab-4ed3-991b-ec0732c71305\") " pod="openshift-ingress/e2e-mt-dd03e395-openshift-default-75799f6fc-kx54z" Nov 03 00:55:11.907218 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:11.907203 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgxqr\" (UniqueName: \"kubernetes.io/projected/01442ff5-1dab-4ed3-991b-ec0732c71305-kube-api-access-wgxqr\") pod \"e2e-mt-dd03e395-openshift-default-75799f6fc-kx54z\" (UID: \"01442ff5-1dab-4ed3-991b-ec0732c71305\") " pod="openshift-ingress/e2e-mt-dd03e395-openshift-default-75799f6fc-kx54z" Nov 03 00:55:11.907403 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:11.907387 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01442ff5-1dab-4ed3-991b-ec0732c71305-istio-podinfo\") pod \"e2e-mt-dd03e395-openshift-default-75799f6fc-kx54z\" (UID: \"01442ff5-1dab-4ed3-991b-ec0732c71305\") " pod="openshift-ingress/e2e-mt-dd03e395-openshift-default-75799f6fc-kx54z" Nov 03 00:55:12.008680 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:12.008646 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01442ff5-1dab-4ed3-991b-ec0732c71305-istio-podinfo\") pod \"e2e-mt-dd03e395-openshift-default-75799f6fc-kx54z\" (UID: \"01442ff5-1dab-4ed3-991b-ec0732c71305\") " pod="openshift-ingress/e2e-mt-dd03e395-openshift-default-75799f6fc-kx54z" Nov 03 00:55:12.008856 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:12.008688 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01442ff5-1dab-4ed3-991b-ec0732c71305-workload-certs\") pod \"e2e-mt-dd03e395-openshift-default-75799f6fc-kx54z\" (UID: \"01442ff5-1dab-4ed3-991b-ec0732c71305\") " pod="openshift-ingress/e2e-mt-dd03e395-openshift-default-75799f6fc-kx54z" Nov 03 00:55:12.008856 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:12.008706 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01442ff5-1dab-4ed3-991b-ec0732c71305-istio-data\") pod \"e2e-mt-dd03e395-openshift-default-75799f6fc-kx54z\" (UID: \"01442ff5-1dab-4ed3-991b-ec0732c71305\") " pod="openshift-ingress/e2e-mt-dd03e395-openshift-default-75799f6fc-kx54z" Nov 03 00:55:12.008856 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:12.008775 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01442ff5-1dab-4ed3-991b-ec0732c71305-istiod-ca-cert\") pod \"e2e-mt-dd03e395-openshift-default-75799f6fc-kx54z\" (UID: \"01442ff5-1dab-4ed3-991b-ec0732c71305\") " pod="openshift-ingress/e2e-mt-dd03e395-openshift-default-75799f6fc-kx54z" Nov 03 00:55:12.008856 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:12.008839 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01442ff5-1dab-4ed3-991b-ec0732c71305-credential-socket\") pod \"e2e-mt-dd03e395-openshift-default-75799f6fc-kx54z\" (UID: \"01442ff5-1dab-4ed3-991b-ec0732c71305\") " pod="openshift-ingress/e2e-mt-dd03e395-openshift-default-75799f6fc-kx54z" Nov 03 00:55:12.009103 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:12.008903 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01442ff5-1dab-4ed3-991b-ec0732c71305-istio-token\") pod \"e2e-mt-dd03e395-openshift-default-75799f6fc-kx54z\" (UID: \"01442ff5-1dab-4ed3-991b-ec0732c71305\") " pod="openshift-ingress/e2e-mt-dd03e395-openshift-default-75799f6fc-kx54z" Nov 03 00:55:12.009103 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:12.008923 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01442ff5-1dab-4ed3-991b-ec0732c71305-workload-socket\") pod \"e2e-mt-dd03e395-openshift-default-75799f6fc-kx54z\" (UID: \"01442ff5-1dab-4ed3-991b-ec0732c71305\") " pod="openshift-ingress/e2e-mt-dd03e395-openshift-default-75799f6fc-kx54z" Nov 03 00:55:12.009103 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:12.008958 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01442ff5-1dab-4ed3-991b-ec0732c71305-istio-envoy\") pod \"e2e-mt-dd03e395-openshift-default-75799f6fc-kx54z\" (UID: \"01442ff5-1dab-4ed3-991b-ec0732c71305\") " pod="openshift-ingress/e2e-mt-dd03e395-openshift-default-75799f6fc-kx54z" Nov 03 00:55:12.009103 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:12.009013 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgxqr\" (UniqueName: \"kubernetes.io/projected/01442ff5-1dab-4ed3-991b-ec0732c71305-kube-api-access-wgxqr\") pod \"e2e-mt-dd03e395-openshift-default-75799f6fc-kx54z\" (UID: \"01442ff5-1dab-4ed3-991b-ec0732c71305\") " pod="openshift-ingress/e2e-mt-dd03e395-openshift-default-75799f6fc-kx54z" Nov 03 00:55:12.009319 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:12.009113 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01442ff5-1dab-4ed3-991b-ec0732c71305-workload-certs\") pod \"e2e-mt-dd03e395-openshift-default-75799f6fc-kx54z\" (UID: \"01442ff5-1dab-4ed3-991b-ec0732c71305\") " pod="openshift-ingress/e2e-mt-dd03e395-openshift-default-75799f6fc-kx54z" Nov 03 00:55:12.009319 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:12.009186 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01442ff5-1dab-4ed3-991b-ec0732c71305-istio-data\") pod \"e2e-mt-dd03e395-openshift-default-75799f6fc-kx54z\" (UID: \"01442ff5-1dab-4ed3-991b-ec0732c71305\") " pod="openshift-ingress/e2e-mt-dd03e395-openshift-default-75799f6fc-kx54z" Nov 03 00:55:12.009319 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:12.009212 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01442ff5-1dab-4ed3-991b-ec0732c71305-credential-socket\") pod \"e2e-mt-dd03e395-openshift-default-75799f6fc-kx54z\" (UID: \"01442ff5-1dab-4ed3-991b-ec0732c71305\") " pod="openshift-ingress/e2e-mt-dd03e395-openshift-default-75799f6fc-kx54z" Nov 03 00:55:12.009435 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:12.009328 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01442ff5-1dab-4ed3-991b-ec0732c71305-workload-socket\") pod \"e2e-mt-dd03e395-openshift-default-75799f6fc-kx54z\" (UID: \"01442ff5-1dab-4ed3-991b-ec0732c71305\") " pod="openshift-ingress/e2e-mt-dd03e395-openshift-default-75799f6fc-kx54z" Nov 03 00:55:12.009794 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:12.009776 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01442ff5-1dab-4ed3-991b-ec0732c71305-istiod-ca-cert\") pod \"e2e-mt-dd03e395-openshift-default-75799f6fc-kx54z\" (UID: \"01442ff5-1dab-4ed3-991b-ec0732c71305\") " pod="openshift-ingress/e2e-mt-dd03e395-openshift-default-75799f6fc-kx54z" Nov 03 00:55:12.011187 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:12.011167 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01442ff5-1dab-4ed3-991b-ec0732c71305-istio-envoy\") pod \"e2e-mt-dd03e395-openshift-default-75799f6fc-kx54z\" (UID: \"01442ff5-1dab-4ed3-991b-ec0732c71305\") " pod="openshift-ingress/e2e-mt-dd03e395-openshift-default-75799f6fc-kx54z" Nov 03 00:55:12.011464 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:12.011445 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01442ff5-1dab-4ed3-991b-ec0732c71305-istio-podinfo\") pod \"e2e-mt-dd03e395-openshift-default-75799f6fc-kx54z\" (UID: \"01442ff5-1dab-4ed3-991b-ec0732c71305\") " pod="openshift-ingress/e2e-mt-dd03e395-openshift-default-75799f6fc-kx54z" Nov 03 00:55:12.018107 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:12.018080 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01442ff5-1dab-4ed3-991b-ec0732c71305-istio-token\") pod \"e2e-mt-dd03e395-openshift-default-75799f6fc-kx54z\" (UID: \"01442ff5-1dab-4ed3-991b-ec0732c71305\") " pod="openshift-ingress/e2e-mt-dd03e395-openshift-default-75799f6fc-kx54z" Nov 03 00:55:12.018230 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:12.018206 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgxqr\" (UniqueName: \"kubernetes.io/projected/01442ff5-1dab-4ed3-991b-ec0732c71305-kube-api-access-wgxqr\") pod \"e2e-mt-dd03e395-openshift-default-75799f6fc-kx54z\" (UID: \"01442ff5-1dab-4ed3-991b-ec0732c71305\") " pod="openshift-ingress/e2e-mt-dd03e395-openshift-default-75799f6fc-kx54z" Nov 03 00:55:12.039855 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:12.039556 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-dd03e395-openshift-default-75799f6fc-kx54z" Nov 03 00:55:12.223324 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:12.223289 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd1c415e-62vfr"] Nov 03 00:55:12.228864 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:12.228846 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd1c415e-62vfr" Nov 03 00:55:12.232590 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:12.232438 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:55:12.232590 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:12.232539 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 00:55:12.232590 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:12.232549 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6q4xp\"" Nov 03 00:55:12.232862 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:12.232610 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 00:55:12.235132 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:12.235108 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd1c415e-62vfr"] Nov 03 00:55:12.312644 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:12.312557 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45eb170b-568c-4bf2-a54b-2132272783ea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bd1c415e-62vfr\" (UID: \"45eb170b-568c-4bf2-a54b-2132272783ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd1c415e-62vfr" Nov 03 00:55:12.312825 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:12.312802 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phz9q\" (UniqueName: \"kubernetes.io/projected/45eb170b-568c-4bf2-a54b-2132272783ea-kube-api-access-phz9q\") pod \"maas-api-revoke-sub-bd1c415e-62vfr\" (UID: \"45eb170b-568c-4bf2-a54b-2132272783ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd1c415e-62vfr" Nov 03 00:55:12.312885 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:12.312867 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45eb170b-568c-4bf2-a54b-2132272783ea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bd1c415e-62vfr\" (UID: \"45eb170b-568c-4bf2-a54b-2132272783ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd1c415e-62vfr" Nov 03 00:55:12.364458 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:55:12.364417 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01442ff5_1dab_4ed3_991b_ec0732c71305.slice/crio-70d1cea667022cdb8212eb6278d1ef71a0d1a5286d2b8ab4786b76cbee3ed3e1 WatchSource:0}: Error finding container 70d1cea667022cdb8212eb6278d1ef71a0d1a5286d2b8ab4786b76cbee3ed3e1: Status 404 returned error can't find the container with id 70d1cea667022cdb8212eb6278d1ef71a0d1a5286d2b8ab4786b76cbee3ed3e1 Nov 03 00:55:12.365460 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:12.365419 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-dd03e395-openshift-default-75799f6fc-kx54z"] Nov 03 00:55:12.366940 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:12.366907 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:55:12.367029 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:12.366974 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:55:12.367029 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:12.367010 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:55:12.414536 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:12.414509 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45eb170b-568c-4bf2-a54b-2132272783ea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bd1c415e-62vfr\" (UID: \"45eb170b-568c-4bf2-a54b-2132272783ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd1c415e-62vfr" Nov 03 00:55:12.414671 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:12.414565 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45eb170b-568c-4bf2-a54b-2132272783ea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bd1c415e-62vfr\" (UID: \"45eb170b-568c-4bf2-a54b-2132272783ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd1c415e-62vfr" Nov 03 00:55:12.414849 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:12.414714 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phz9q\" (UniqueName: \"kubernetes.io/projected/45eb170b-568c-4bf2-a54b-2132272783ea-kube-api-access-phz9q\") pod \"maas-api-revoke-sub-bd1c415e-62vfr\" (UID: \"45eb170b-568c-4bf2-a54b-2132272783ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd1c415e-62vfr" Nov 03 00:55:12.415377 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:12.415351 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45eb170b-568c-4bf2-a54b-2132272783ea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bd1c415e-62vfr\" (UID: \"45eb170b-568c-4bf2-a54b-2132272783ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd1c415e-62vfr" Nov 03 00:55:12.416994 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:12.416974 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45eb170b-568c-4bf2-a54b-2132272783ea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bd1c415e-62vfr\" (UID: \"45eb170b-568c-4bf2-a54b-2132272783ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd1c415e-62vfr" Nov 03 00:55:12.424020 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:12.423990 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phz9q\" (UniqueName: \"kubernetes.io/projected/45eb170b-568c-4bf2-a54b-2132272783ea-kube-api-access-phz9q\") pod \"maas-api-revoke-sub-bd1c415e-62vfr\" (UID: \"45eb170b-568c-4bf2-a54b-2132272783ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd1c415e-62vfr" Nov 03 00:55:12.550681 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:12.550634 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd1c415e-62vfr" Nov 03 00:55:12.652405 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:12.652201 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-dd03e395-openshift-default-75799f6fc-kx54z" event={"ID":"01442ff5-1dab-4ed3-991b-ec0732c71305","Type":"ContainerStarted","Data":"5f145538a80b3358ab75e1c1021050da283eed8d6d1f9782ad088f6ef764e4a4"} Nov 03 00:55:12.652405 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:12.652323 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-dd03e395-openshift-default-75799f6fc-kx54z" event={"ID":"01442ff5-1dab-4ed3-991b-ec0732c71305","Type":"ContainerStarted","Data":"70d1cea667022cdb8212eb6278d1ef71a0d1a5286d2b8ab4786b76cbee3ed3e1"} Nov 03 00:55:12.677330 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:12.677262 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-dd03e395-openshift-default-75799f6fc-kx54z" podStartSLOduration=1.6772464999999999 podStartE2EDuration="1.6772465s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:55:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:55:12.669987431 +0000 UTC m=+854.840990528" watchObservedRunningTime="2026-11-03 00:55:12.6772465 +0000 UTC m=+854.848249595" Nov 03 00:55:12.856462 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:55:12.856416 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45eb170b_568c_4bf2_a54b_2132272783ea.slice/crio-932f0b17543c63b2f1c853917e6c36805f88946c4339bdb100af67bf0c15aa0a WatchSource:0}: Error finding container 932f0b17543c63b2f1c853917e6c36805f88946c4339bdb100af67bf0c15aa0a: Status 404 returned error can't find the container with id 932f0b17543c63b2f1c853917e6c36805f88946c4339bdb100af67bf0c15aa0a Nov 03 00:55:12.861745 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:12.861716 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd1c415e-62vfr"] Nov 03 00:55:13.044838 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:13.044807 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-dd03e395-openshift-default-75799f6fc-kx54z" Nov 03 00:55:13.050906 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:13.050883 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-dd03e395-openshift-default-75799f6fc-kx54z" Nov 03 00:55:13.661617 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:13.661588 4421 generic.go:356] "Generic (PLEG): container finished" podID="45eb170b-568c-4bf2-a54b-2132272783ea" containerID="3a5fcbbc9a4d597b850c0031e022ad946d1532f1b4a4773021d3be6b63e84c05" exitCode=0 Nov 03 00:55:13.661773 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:13.661636 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd1c415e-62vfr" event={"ID":"45eb170b-568c-4bf2-a54b-2132272783ea","Type":"ContainerDied","Data":"3a5fcbbc9a4d597b850c0031e022ad946d1532f1b4a4773021d3be6b63e84c05"} Nov 03 00:55:13.661773 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:13.661702 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd1c415e-62vfr" event={"ID":"45eb170b-568c-4bf2-a54b-2132272783ea","Type":"ContainerStarted","Data":"932f0b17543c63b2f1c853917e6c36805f88946c4339bdb100af67bf0c15aa0a"} Nov 03 00:55:13.661948 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:13.661935 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-dd03e395-openshift-default-75799f6fc-kx54z" Nov 03 00:55:13.663062 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:13.663043 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-dd03e395-openshift-default-75799f6fc-kx54z" Nov 03 00:55:14.691570 ip-10-0-133-217 kubenswrapper[4421]: E1103 00:55:14.691151 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 00:55:14.846834 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:14.846808 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd1c415e-62vfr" Nov 03 00:55:14.869810 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:14.869775 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/45eb170b-568c-4bf2-a54b-2132272783ea-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45eb170b-568c-4bf2-a54b-2132272783ea-maas-api-xks-ca\") pod \"45eb170b-568c-4bf2-a54b-2132272783ea\" (UID: \"45eb170b-568c-4bf2-a54b-2132272783ea\") " Nov 03 00:55:14.869961 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:14.869867 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45eb170b-568c-4bf2-a54b-2132272783ea-kube-api-access-phz9q\" (UniqueName: \"kubernetes.io/projected/45eb170b-568c-4bf2-a54b-2132272783ea-kube-api-access-phz9q\") pod \"45eb170b-568c-4bf2-a54b-2132272783ea\" (UID: \"45eb170b-568c-4bf2-a54b-2132272783ea\") " Nov 03 00:55:14.870000 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:14.869975 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45eb170b-568c-4bf2-a54b-2132272783ea-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45eb170b-568c-4bf2-a54b-2132272783ea-maas-api-openshift-ca\") pod \"45eb170b-568c-4bf2-a54b-2132272783ea\" (UID: \"45eb170b-568c-4bf2-a54b-2132272783ea\") " Nov 03 00:55:14.870375 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:14.870348 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45eb170b-568c-4bf2-a54b-2132272783ea-maas-api-openshift-ca" pod "45eb170b-568c-4bf2-a54b-2132272783ea" (UID: "45eb170b-568c-4bf2-a54b-2132272783ea"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:55:14.870589 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:14.870571 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45eb170b-568c-4bf2-a54b-2132272783ea-maas-api-openshift-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:55:14.872516 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:14.872490 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45eb170b-568c-4bf2-a54b-2132272783ea-maas-api-xks-ca" pod "45eb170b-568c-4bf2-a54b-2132272783ea" (UID: "45eb170b-568c-4bf2-a54b-2132272783ea"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:55:14.872606 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:14.872531 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45eb170b-568c-4bf2-a54b-2132272783ea-kube-api-access-phz9q" pod "45eb170b-568c-4bf2-a54b-2132272783ea" (UID: "45eb170b-568c-4bf2-a54b-2132272783ea"). InnerVolumeSpecName "kube-api-access-phz9q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:55:14.971946 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:14.971865 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45eb170b-568c-4bf2-a54b-2132272783ea-maas-api-xks-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:55:14.971946 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:14.971895 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phz9q\" (UniqueName: \"kubernetes.io/projected/45eb170b-568c-4bf2-a54b-2132272783ea-kube-api-access-phz9q\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:55:15.693519 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:15.693490 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd1c415e-62vfr" Nov 03 00:55:15.694021 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:15.693527 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd1c415e-62vfr" event={"ID":"45eb170b-568c-4bf2-a54b-2132272783ea","Type":"ContainerDied","Data":"932f0b17543c63b2f1c853917e6c36805f88946c4339bdb100af67bf0c15aa0a"} Nov 03 00:55:15.694021 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:15.693582 4421 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="932f0b17543c63b2f1c853917e6c36805f88946c4339bdb100af67bf0c15aa0a" Nov 03 00:55:21.030301 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:21.029368 4421 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-83755a/premium-w3-83755a-kserve-7c9d66988b-lt4xd" podUID="c0c6b02e-2ecb-428e-a13d-be6d7cd1ba39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:55:21.112329 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:21.111384 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/simulator-w2-f7680d-kserve-85bdc86cd6-8gxrg" podUID="2bdcf521-cebd-4d6b-8554-1b60db8b0f3c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:55:21.442357 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:21.441388 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-650be0/simulator-w4-650be0-kserve-7bc588774f-gvz2k" podUID="a827fa88-f078-4312-8f12-2c799c11b461" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:55:21.552321 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:21.551341 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8gm8n" podUID="2d367e12-e46f-4a68-9825-9b1c934d4d12" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": context deadline exceeded" Nov 03 00:55:21.575298 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:21.572386 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/simulator-w3-83755a-kserve-77d6664559-psqtg" podUID="b0a4c234-af58-4da0-a1a4-e1f167953f6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:55:21.575298 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:21.573402 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/premium-w3-83755a-kserve-7c9d66988b-lt4xd" podUID="c0c6b02e-2ecb-428e-a13d-be6d7cd1ba39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Nov 03 00:55:21.813299 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:21.812593 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-650be0/premium-w4-650be0-kserve-7bf8555dfb-z97n5" podUID="40601dc5-808f-456f-826f-6e0c35966ada" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:55:22.127753 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:22.125670 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/simulator-w2-f7680d-kserve-85bdc86cd6-8gxrg" podUID="2bdcf521-cebd-4d6b-8554-1b60db8b0f3c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:55:22.127753 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:22.126412 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f7680d/simulator-w2-f7680d-kserve-85bdc86cd6-8gxrg" Nov 03 00:55:22.379358 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:22.376016 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/distinct-w3-83755a-kserve-86bdbbdd85-2cnnl" podUID="f4ae9624-d127-4a55-837d-7c9ff0267b37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded" Nov 03 00:55:22.545309 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:22.544405 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-650be0/simulator-w4-650be0-kserve-7bc588774f-gvz2k" podUID="a827fa88-f078-4312-8f12-2c799c11b461" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Nov 03 00:55:22.545309 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:22.545239 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-650be0/simulator-w4-650be0-kserve-7bc588774f-gvz2k" Nov 03 00:55:22.571298 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:22.570534 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8gm8n" podUID="2d367e12-e46f-4a68-9825-9b1c934d4d12" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:55:22.571298 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:22.571049 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8gm8n" Nov 03 00:55:22.600521 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:22.572587 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/simulator-w3-83755a-kserve-77d6664559-psqtg" podUID="b0a4c234-af58-4da0-a1a4-e1f167953f6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:55:22.600521 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:22.574003 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-83755a/simulator-w3-83755a-kserve-77d6664559-psqtg" Nov 03 00:55:22.600521 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:22.574778 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/premium-w3-83755a-kserve-7c9d66988b-lt4xd" podUID="c0c6b02e-2ecb-428e-a13d-be6d7cd1ba39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:55:22.600521 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:22.575071 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-83755a/premium-w3-83755a-kserve-7c9d66988b-lt4xd" Nov 03 00:55:22.777362 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:22.733931 4421 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-83755a/simulator-w3-83755a-kserve-77d6664559-psqtg" podUID="b0a4c234-af58-4da0-a1a4-e1f167953f6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:55:22.885108 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:22.817583 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-650be0/premium-w4-650be0-kserve-7bf8555dfb-z97n5" podUID="40601dc5-808f-456f-826f-6e0c35966ada" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:55:22.885729 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:22.885458 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-650be0/premium-w4-650be0-kserve-7bf8555dfb-z97n5" Nov 03 00:55:23.056358 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:23.052507 4421 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-650be0/simulator-w4-650be0-kserve-7bc588774f-gvz2k" podUID="a827fa88-f078-4312-8f12-2c799c11b461" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:55:23.182455 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:23.182403 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/simulator-w2-f7680d-kserve-85bdc86cd6-8gxrg" podUID="2bdcf521-cebd-4d6b-8554-1b60db8b0f3c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:55:23.379673 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:23.379550 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/distinct-w3-83755a-kserve-86bdbbdd85-2cnnl" podUID="f4ae9624-d127-4a55-837d-7c9ff0267b37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:55:23.381327 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:23.381298 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-83755a/distinct-w3-83755a-kserve-86bdbbdd85-2cnnl" Nov 03 00:55:23.633368 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:23.623833 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/premium-w3-83755a-kserve-7c9d66988b-lt4xd" podUID="c0c6b02e-2ecb-428e-a13d-be6d7cd1ba39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Nov 03 00:55:23.668301 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:23.643175 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8gm8n" podUID="2d367e12-e46f-4a68-9825-9b1c934d4d12" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": context deadline exceeded" Nov 03 00:55:23.720806 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:23.651039 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-650be0/simulator-w4-650be0-kserve-7bc588774f-gvz2k" podUID="a827fa88-f078-4312-8f12-2c799c11b461" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:55:23.721703 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:23.651468 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/simulator-w3-83755a-kserve-77d6664559-psqtg" podUID="b0a4c234-af58-4da0-a1a4-e1f167953f6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:55:23.923821 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:23.921445 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-650be0/premium-w4-650be0-kserve-7bf8555dfb-z97n5" podUID="40601dc5-808f-456f-826f-6e0c35966ada" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:55:23.980329 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:23.975229 4421 patch_prober.go:29] interesting pod/e2e-worker-w3-83755a-openshift-default-c86f9565c-h2p2p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 00:55:23.980329 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:23.975310 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-h2p2p" podUID="5bd6d60e-043b-47ba-beb1-08b1eaace690" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 00:55:24.013306 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:24.001168 4421 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-s4dc8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 00:55:24.013306 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:24.001239 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-s4dc8" podUID="ab5414b5-9d47-4c9f-9d0e-82242b9d6db2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 00:55:24.028483 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:24.028425 4421 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-t5bt4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 00:55:24.113778 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:24.113734 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-t5bt4" podUID="2b55c151-d235-4bcc-b602-eca0da4a7348" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 00:55:24.168236 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:24.166687 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f7680d/simulator-w2-f7680d-kserve-85bdc86cd6-8gxrg" Nov 03 00:55:24.180688 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:24.180605 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-nccrq"] Nov 03 00:55:24.182496 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:24.182465 4421 cpu_manager.go:414] "containerMap: removing container" podUID="45eb170b-568c-4bf2-a54b-2132272783ea" containerName="revoke-keys" Nov 03 00:55:24.183005 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:24.182985 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45eb170b-568c-4bf2-a54b-2132272783ea" containerName="revoke-keys" Nov 03 00:55:24.183583 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:24.183561 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="45eb170b-568c-4bf2-a54b-2132272783ea" containerName="revoke-keys" Nov 03 00:55:24.212299 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:24.201904 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-nccrq" Nov 03 00:55:24.427371 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:24.391597 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0d8360b-b4c3-483e-8b00-412893aee9a6-workload-certs\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-nccrq\" (UID: \"a0d8360b-b4c3-483e-8b00-412893aee9a6\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-nccrq" Nov 03 00:55:24.427371 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:24.391787 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0d8360b-b4c3-483e-8b00-412893aee9a6-istio-envoy\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-nccrq\" (UID: \"a0d8360b-b4c3-483e-8b00-412893aee9a6\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-nccrq" Nov 03 00:55:24.427371 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:24.391823 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0d8360b-b4c3-483e-8b00-412893aee9a6-istiod-ca-cert\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-nccrq\" (UID: \"a0d8360b-b4c3-483e-8b00-412893aee9a6\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-nccrq" Nov 03 00:55:24.427371 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:24.391879 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0d8360b-b4c3-483e-8b00-412893aee9a6-workload-socket\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-nccrq\" (UID: \"a0d8360b-b4c3-483e-8b00-412893aee9a6\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-nccrq" Nov 03 00:55:24.427371 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:24.391953 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0d8360b-b4c3-483e-8b00-412893aee9a6-istio-token\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-nccrq\" (UID: \"a0d8360b-b4c3-483e-8b00-412893aee9a6\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-nccrq" Nov 03 00:55:24.427371 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:24.392012 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0d8360b-b4c3-483e-8b00-412893aee9a6-istio-podinfo\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-nccrq\" (UID: \"a0d8360b-b4c3-483e-8b00-412893aee9a6\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-nccrq" Nov 03 00:55:24.427371 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:24.392086 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k447z\" (UniqueName: \"kubernetes.io/projected/a0d8360b-b4c3-483e-8b00-412893aee9a6-kube-api-access-k447z\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-nccrq\" (UID: \"a0d8360b-b4c3-483e-8b00-412893aee9a6\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-nccrq" Nov 03 00:55:24.427371 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:24.392136 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0d8360b-b4c3-483e-8b00-412893aee9a6-credential-socket\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-nccrq\" (UID: \"a0d8360b-b4c3-483e-8b00-412893aee9a6\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-nccrq" Nov 03 00:55:24.427371 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:24.392159 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0d8360b-b4c3-483e-8b00-412893aee9a6-istio-data\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-nccrq\" (UID: \"a0d8360b-b4c3-483e-8b00-412893aee9a6\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-nccrq" Nov 03 00:55:24.537973 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:24.508488 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-83755a/distinct-w3-83755a-kserve-86bdbbdd85-2cnnl" Nov 03 00:55:24.559804 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:24.542133 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0d8360b-b4c3-483e-8b00-412893aee9a6-workload-certs\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-nccrq\" (UID: \"a0d8360b-b4c3-483e-8b00-412893aee9a6\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-nccrq" Nov 03 00:55:24.559804 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:24.542344 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0d8360b-b4c3-483e-8b00-412893aee9a6-istio-envoy\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-nccrq\" (UID: \"a0d8360b-b4c3-483e-8b00-412893aee9a6\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-nccrq" Nov 03 00:55:24.559804 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:24.542389 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0d8360b-b4c3-483e-8b00-412893aee9a6-istiod-ca-cert\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-nccrq\" (UID: \"a0d8360b-b4c3-483e-8b00-412893aee9a6\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-nccrq" Nov 03 00:55:24.559804 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:24.542444 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0d8360b-b4c3-483e-8b00-412893aee9a6-workload-socket\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-nccrq\" (UID: \"a0d8360b-b4c3-483e-8b00-412893aee9a6\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-nccrq" Nov 03 00:55:24.559804 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:24.542525 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0d8360b-b4c3-483e-8b00-412893aee9a6-istio-token\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-nccrq\" (UID: \"a0d8360b-b4c3-483e-8b00-412893aee9a6\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-nccrq" Nov 03 00:55:24.559804 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:24.542584 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0d8360b-b4c3-483e-8b00-412893aee9a6-istio-podinfo\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-nccrq\" (UID: \"a0d8360b-b4c3-483e-8b00-412893aee9a6\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-nccrq" Nov 03 00:55:24.559804 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:24.542658 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k447z\" (UniqueName: \"kubernetes.io/projected/a0d8360b-b4c3-483e-8b00-412893aee9a6-kube-api-access-k447z\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-nccrq\" (UID: \"a0d8360b-b4c3-483e-8b00-412893aee9a6\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-nccrq" Nov 03 00:55:24.559804 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:24.542710 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0d8360b-b4c3-483e-8b00-412893aee9a6-credential-socket\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-nccrq\" (UID: \"a0d8360b-b4c3-483e-8b00-412893aee9a6\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-nccrq" Nov 03 00:55:24.559804 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:24.542736 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0d8360b-b4c3-483e-8b00-412893aee9a6-istio-data\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-nccrq\" (UID: \"a0d8360b-b4c3-483e-8b00-412893aee9a6\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-nccrq" Nov 03 00:55:24.559804 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:24.543369 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0d8360b-b4c3-483e-8b00-412893aee9a6-istio-data\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-nccrq\" (UID: \"a0d8360b-b4c3-483e-8b00-412893aee9a6\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-nccrq" Nov 03 00:55:24.559804 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:24.543650 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0d8360b-b4c3-483e-8b00-412893aee9a6-workload-certs\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-nccrq\" (UID: \"a0d8360b-b4c3-483e-8b00-412893aee9a6\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-nccrq" Nov 03 00:55:24.562332 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:24.562306 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0d8360b-b4c3-483e-8b00-412893aee9a6-workload-socket\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-nccrq\" (UID: \"a0d8360b-b4c3-483e-8b00-412893aee9a6\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-nccrq" Nov 03 00:55:24.601331 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:24.601294 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0d8360b-b4c3-483e-8b00-412893aee9a6-istiod-ca-cert\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-nccrq\" (UID: \"a0d8360b-b4c3-483e-8b00-412893aee9a6\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-nccrq" Nov 03 00:55:24.609324 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:24.609262 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0d8360b-b4c3-483e-8b00-412893aee9a6-credential-socket\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-nccrq\" (UID: \"a0d8360b-b4c3-483e-8b00-412893aee9a6\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-nccrq" Nov 03 00:55:24.655824 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:24.655772 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/premium-w3-83755a-kserve-7c9d66988b-lt4xd" podUID="c0c6b02e-2ecb-428e-a13d-be6d7cd1ba39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Nov 03 00:55:24.697829 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:24.697765 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k447z\" (UniqueName: \"kubernetes.io/projected/a0d8360b-b4c3-483e-8b00-412893aee9a6-kube-api-access-k447z\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-nccrq\" (UID: \"a0d8360b-b4c3-483e-8b00-412893aee9a6\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-nccrq" Nov 03 00:55:24.698586 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:24.698548 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0d8360b-b4c3-483e-8b00-412893aee9a6-istio-podinfo\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-nccrq\" (UID: \"a0d8360b-b4c3-483e-8b00-412893aee9a6\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-nccrq" Nov 03 00:55:24.702676 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:24.702640 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-650be0/premium-w4-650be0-kserve-7bf8555dfb-z97n5" Nov 03 00:55:24.703204 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:24.703173 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8gm8n" podUID="2d367e12-e46f-4a68-9825-9b1c934d4d12" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:55:24.726529 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:24.726487 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/simulator-w3-83755a-kserve-77d6664559-psqtg" podUID="b0a4c234-af58-4da0-a1a4-e1f167953f6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Nov 03 00:55:24.727107 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:24.727074 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-650be0/simulator-w4-650be0-kserve-7bc588774f-gvz2k" podUID="a827fa88-f078-4312-8f12-2c799c11b461" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Nov 03 00:55:24.749301 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:24.735416 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0d8360b-b4c3-483e-8b00-412893aee9a6-istio-token\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-nccrq\" (UID: \"a0d8360b-b4c3-483e-8b00-412893aee9a6\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-nccrq" Nov 03 00:55:24.749301 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:24.736095 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0d8360b-b4c3-483e-8b00-412893aee9a6-istio-envoy\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-nccrq\" (UID: \"a0d8360b-b4c3-483e-8b00-412893aee9a6\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-nccrq" Nov 03 00:55:24.958485 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:24.958357 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-nccrq"] Nov 03 00:55:25.021440 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:25.021388 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-nccrq" Nov 03 00:55:25.032367 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:25.032334 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8gm8n" Nov 03 00:55:25.186554 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:25.185877 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-83755a/simulator-w3-83755a-kserve-77d6664559-psqtg" Nov 03 00:55:25.190810 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:25.188254 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-650be0/simulator-w4-650be0-kserve-7bc588774f-gvz2k" Nov 03 00:55:25.491847 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:25.490943 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-83755a/premium-w3-83755a-kserve-7c9d66988b-lt4xd" Nov 03 00:55:25.730740 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:25.730689 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:55:25.730906 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:25.730799 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:55:25.730906 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:25.730850 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:55:25.734832 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:25.734800 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-nccrq"] Nov 03 00:55:25.774790 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:25.774750 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-nccrq" event={"ID":"a0d8360b-b4c3-483e-8b00-412893aee9a6","Type":"ContainerStarted","Data":"58d0683ab935e771010a40747d5bac5841b2305e02e74f90be840adeba1d03f5"} Nov 03 00:55:26.784347 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:26.784308 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-nccrq" event={"ID":"a0d8360b-b4c3-483e-8b00-412893aee9a6","Type":"ContainerStarted","Data":"3bf31bc22bf58fa760a72288611e2065dad3745f00cb10885401226f9b1f2a44"} Nov 03 00:55:26.838120 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:26.838042 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-nccrq" podStartSLOduration=3.838018574 podStartE2EDuration="3.838018574s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:55:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:55:26.807672181 +0000 UTC m=+868.978675277" watchObservedRunningTime="2026-11-03 00:55:26.838018574 +0000 UTC m=+869.009021673" Nov 03 00:55:27.032303 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:27.031630 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-nccrq" Nov 03 00:55:27.035502 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:27.034469 4421 patch_prober.go:29] interesting pod/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-nccrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Nov 03 00:55:27.035502 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:27.034529 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-nccrq" podUID="a0d8360b-b4c3-483e-8b00-412893aee9a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Nov 03 00:55:28.024765 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:28.024719 4421 patch_prober.go:29] interesting pod/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-nccrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Nov 03 00:55:28.025748 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:28.025449 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-nccrq" podUID="a0d8360b-b4c3-483e-8b00-412893aee9a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Nov 03 00:55:29.036298 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:29.024666 4421 patch_prober.go:29] interesting pod/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-nccrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Nov 03 00:55:29.036298 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:29.024726 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-nccrq" podUID="a0d8360b-b4c3-483e-8b00-412893aee9a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Nov 03 00:55:30.030437 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:30.030390 4421 patch_prober.go:29] interesting pod/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-nccrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Nov 03 00:55:30.030650 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:30.030488 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-nccrq" podUID="a0d8360b-b4c3-483e-8b00-412893aee9a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Nov 03 00:55:31.030475 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:31.030420 4421 patch_prober.go:29] interesting pod/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-nccrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Nov 03 00:55:31.030991 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:31.030529 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-nccrq" podUID="a0d8360b-b4c3-483e-8b00-412893aee9a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Nov 03 00:55:32.030864 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:32.030822 4421 patch_prober.go:29] interesting pod/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-nccrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Nov 03 00:55:32.031553 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:32.031503 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-nccrq" podUID="a0d8360b-b4c3-483e-8b00-412893aee9a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Nov 03 00:55:32.637846 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:32.637793 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-9jh88"] Nov 03 00:55:32.642372 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:32.642343 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-9jh88" Nov 03 00:55:32.655668 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:32.655638 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-9jh88"] Nov 03 00:55:32.753598 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:32.753553 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72f72bc8-0d67-4200-ab31-c167fe8b877e-credential-socket\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-9jh88\" (UID: \"72f72bc8-0d67-4200-ab31-c167fe8b877e\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-9jh88" Nov 03 00:55:32.753796 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:32.753609 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72f72bc8-0d67-4200-ab31-c167fe8b877e-istio-data\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-9jh88\" (UID: \"72f72bc8-0d67-4200-ab31-c167fe8b877e\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-9jh88" Nov 03 00:55:32.753796 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:32.753679 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72f72bc8-0d67-4200-ab31-c167fe8b877e-istio-token\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-9jh88\" (UID: \"72f72bc8-0d67-4200-ab31-c167fe8b877e\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-9jh88" Nov 03 00:55:32.753796 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:32.753737 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72f72bc8-0d67-4200-ab31-c167fe8b877e-istio-podinfo\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-9jh88\" (UID: \"72f72bc8-0d67-4200-ab31-c167fe8b877e\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-9jh88" Nov 03 00:55:32.753907 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:32.753894 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72f72bc8-0d67-4200-ab31-c167fe8b877e-istiod-ca-cert\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-9jh88\" (UID: \"72f72bc8-0d67-4200-ab31-c167fe8b877e\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-9jh88" Nov 03 00:55:32.753942 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:32.753923 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72f72bc8-0d67-4200-ab31-c167fe8b877e-istio-envoy\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-9jh88\" (UID: \"72f72bc8-0d67-4200-ab31-c167fe8b877e\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-9jh88" Nov 03 00:55:32.754007 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:32.753990 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72f72bc8-0d67-4200-ab31-c167fe8b877e-workload-certs\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-9jh88\" (UID: \"72f72bc8-0d67-4200-ab31-c167fe8b877e\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-9jh88" Nov 03 00:55:32.754057 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:32.754024 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgzdg\" (UniqueName: \"kubernetes.io/projected/72f72bc8-0d67-4200-ab31-c167fe8b877e-kube-api-access-rgzdg\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-9jh88\" (UID: \"72f72bc8-0d67-4200-ab31-c167fe8b877e\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-9jh88" Nov 03 00:55:32.754194 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:32.754179 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72f72bc8-0d67-4200-ab31-c167fe8b877e-workload-socket\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-9jh88\" (UID: \"72f72bc8-0d67-4200-ab31-c167fe8b877e\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-9jh88" Nov 03 00:55:32.855559 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:32.855521 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72f72bc8-0d67-4200-ab31-c167fe8b877e-workload-certs\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-9jh88\" (UID: \"72f72bc8-0d67-4200-ab31-c167fe8b877e\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-9jh88" Nov 03 00:55:32.855768 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:32.855567 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgzdg\" (UniqueName: \"kubernetes.io/projected/72f72bc8-0d67-4200-ab31-c167fe8b877e-kube-api-access-rgzdg\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-9jh88\" (UID: \"72f72bc8-0d67-4200-ab31-c167fe8b877e\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-9jh88" Nov 03 00:55:32.855768 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:32.855643 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72f72bc8-0d67-4200-ab31-c167fe8b877e-workload-socket\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-9jh88\" (UID: \"72f72bc8-0d67-4200-ab31-c167fe8b877e\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-9jh88" Nov 03 00:55:32.855768 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:32.855752 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72f72bc8-0d67-4200-ab31-c167fe8b877e-credential-socket\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-9jh88\" (UID: \"72f72bc8-0d67-4200-ab31-c167fe8b877e\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-9jh88" Nov 03 00:55:32.855954 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:32.855775 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72f72bc8-0d67-4200-ab31-c167fe8b877e-istio-data\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-9jh88\" (UID: \"72f72bc8-0d67-4200-ab31-c167fe8b877e\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-9jh88" Nov 03 00:55:32.855954 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:32.855798 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72f72bc8-0d67-4200-ab31-c167fe8b877e-istio-token\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-9jh88\" (UID: \"72f72bc8-0d67-4200-ab31-c167fe8b877e\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-9jh88" Nov 03 00:55:32.855954 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:32.855847 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72f72bc8-0d67-4200-ab31-c167fe8b877e-istio-podinfo\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-9jh88\" (UID: \"72f72bc8-0d67-4200-ab31-c167fe8b877e\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-9jh88" Nov 03 00:55:32.856113 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:32.855957 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72f72bc8-0d67-4200-ab31-c167fe8b877e-istiod-ca-cert\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-9jh88\" (UID: \"72f72bc8-0d67-4200-ab31-c167fe8b877e\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-9jh88" Nov 03 00:55:32.856113 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:32.855984 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72f72bc8-0d67-4200-ab31-c167fe8b877e-istio-envoy\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-9jh88\" (UID: \"72f72bc8-0d67-4200-ab31-c167fe8b877e\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-9jh88" Nov 03 00:55:32.856113 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:32.855993 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72f72bc8-0d67-4200-ab31-c167fe8b877e-workload-certs\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-9jh88\" (UID: \"72f72bc8-0d67-4200-ab31-c167fe8b877e\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-9jh88" Nov 03 00:55:32.856291 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:32.856114 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72f72bc8-0d67-4200-ab31-c167fe8b877e-workload-socket\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-9jh88\" (UID: \"72f72bc8-0d67-4200-ab31-c167fe8b877e\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-9jh88" Nov 03 00:55:32.856291 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:32.856137 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72f72bc8-0d67-4200-ab31-c167fe8b877e-credential-socket\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-9jh88\" (UID: \"72f72bc8-0d67-4200-ab31-c167fe8b877e\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-9jh88" Nov 03 00:55:32.856404 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:32.856308 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72f72bc8-0d67-4200-ab31-c167fe8b877e-istio-data\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-9jh88\" (UID: \"72f72bc8-0d67-4200-ab31-c167fe8b877e\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-9jh88" Nov 03 00:55:32.856715 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:32.856692 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72f72bc8-0d67-4200-ab31-c167fe8b877e-istiod-ca-cert\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-9jh88\" (UID: \"72f72bc8-0d67-4200-ab31-c167fe8b877e\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-9jh88" Nov 03 00:55:32.858305 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:32.858251 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72f72bc8-0d67-4200-ab31-c167fe8b877e-istio-envoy\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-9jh88\" (UID: \"72f72bc8-0d67-4200-ab31-c167fe8b877e\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-9jh88" Nov 03 00:55:32.858590 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:32.858571 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72f72bc8-0d67-4200-ab31-c167fe8b877e-istio-podinfo\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-9jh88\" (UID: \"72f72bc8-0d67-4200-ab31-c167fe8b877e\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-9jh88" Nov 03 00:55:32.869353 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:32.869322 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgzdg\" (UniqueName: \"kubernetes.io/projected/72f72bc8-0d67-4200-ab31-c167fe8b877e-kube-api-access-rgzdg\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-9jh88\" (UID: \"72f72bc8-0d67-4200-ab31-c167fe8b877e\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-9jh88" Nov 03 00:55:32.869462 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:32.869388 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72f72bc8-0d67-4200-ab31-c167fe8b877e-istio-token\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-9jh88\" (UID: \"72f72bc8-0d67-4200-ab31-c167fe8b877e\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-9jh88" Nov 03 00:55:32.958823 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:32.958743 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-9jh88" Nov 03 00:55:33.022613 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:33.022059 4421 patch_prober.go:29] interesting pod/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-nccrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Nov 03 00:55:33.022613 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:33.022129 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-nccrq" podUID="a0d8360b-b4c3-483e-8b00-412893aee9a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Nov 03 00:55:33.250617 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:55:33.250576 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72f72bc8_0d67_4200_ab31_c167fe8b877e.slice/crio-66cadfb8389ee7be6f698bebe30d5d456745f43c5deaed3e5854d6e25e1e32c2 WatchSource:0}: Error finding container 66cadfb8389ee7be6f698bebe30d5d456745f43c5deaed3e5854d6e25e1e32c2: Status 404 returned error can't find the container with id 66cadfb8389ee7be6f698bebe30d5d456745f43c5deaed3e5854d6e25e1e32c2 Nov 03 00:55:33.253485 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:33.253439 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:55:33.253620 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:33.253530 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:55:33.253620 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:33.253574 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:55:33.254186 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:33.254059 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-9jh88"] Nov 03 00:55:33.930439 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:33.929386 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-9jh88" event={"ID":"72f72bc8-0d67-4200-ab31-c167fe8b877e","Type":"ContainerStarted","Data":"aff03fc28fb084070ff8360f656037f46775f47711662b3bf3b6b035a848967e"} Nov 03 00:55:33.930439 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:33.929463 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-9jh88" event={"ID":"72f72bc8-0d67-4200-ab31-c167fe8b877e","Type":"ContainerStarted","Data":"66cadfb8389ee7be6f698bebe30d5d456745f43c5deaed3e5854d6e25e1e32c2"} Nov 03 00:55:33.952961 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:33.952913 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-9jh88" podStartSLOduration=1.952898942 podStartE2EDuration="1.952898942s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:55:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:55:33.949698267 +0000 UTC m=+876.120701362" watchObservedRunningTime="2026-11-03 00:55:33.952898942 +0000 UTC m=+876.123902113" Nov 03 00:55:33.958829 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:33.958800 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-9jh88" Nov 03 00:55:34.035320 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:34.035286 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-nccrq" Nov 03 00:55:34.036030 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:34.036001 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-nccrq" Nov 03 00:55:34.036790 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:34.036767 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-nccrq" Nov 03 00:55:34.960847 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:34.960812 4421 patch_prober.go:29] interesting pod/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-9jh88 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Nov 03 00:55:34.961225 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:34.960865 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-9jh88" podUID="72f72bc8-0d67-4200-ab31-c167fe8b877e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Nov 03 00:55:35.056299 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:35.056244 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ce7eb93-bpnr2"] Nov 03 00:55:35.064383 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:35.064352 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ce7eb93-bpnr2" Nov 03 00:55:35.072326 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:35.066633 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ce7eb93-bpnr2"] Nov 03 00:55:35.080486 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:35.077718 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6q4xp\"" Nov 03 00:55:35.080486 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:35.077799 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:55:35.080486 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:35.077885 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 00:55:35.080486 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:35.077924 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 00:55:35.197677 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:35.197642 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/318c5891-d03d-462f-91f1-de6c5a4a20f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2ce7eb93-bpnr2\" (UID: \"318c5891-d03d-462f-91f1-de6c5a4a20f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ce7eb93-bpnr2" Nov 03 00:55:35.197883 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:35.197714 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/318c5891-d03d-462f-91f1-de6c5a4a20f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2ce7eb93-bpnr2\" (UID: \"318c5891-d03d-462f-91f1-de6c5a4a20f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ce7eb93-bpnr2" Nov 03 00:55:35.197956 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:35.197898 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bhw4\" (UniqueName: \"kubernetes.io/projected/318c5891-d03d-462f-91f1-de6c5a4a20f6-kube-api-access-9bhw4\") pod \"maas-api-revoke-sub-2ce7eb93-bpnr2\" (UID: \"318c5891-d03d-462f-91f1-de6c5a4a20f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ce7eb93-bpnr2" Nov 03 00:55:35.299740 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:35.299695 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/318c5891-d03d-462f-91f1-de6c5a4a20f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2ce7eb93-bpnr2\" (UID: \"318c5891-d03d-462f-91f1-de6c5a4a20f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ce7eb93-bpnr2" Nov 03 00:55:35.299961 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:35.299826 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bhw4\" (UniqueName: \"kubernetes.io/projected/318c5891-d03d-462f-91f1-de6c5a4a20f6-kube-api-access-9bhw4\") pod \"maas-api-revoke-sub-2ce7eb93-bpnr2\" (UID: \"318c5891-d03d-462f-91f1-de6c5a4a20f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ce7eb93-bpnr2" Nov 03 00:55:35.300035 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:35.300013 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/318c5891-d03d-462f-91f1-de6c5a4a20f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2ce7eb93-bpnr2\" (UID: \"318c5891-d03d-462f-91f1-de6c5a4a20f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ce7eb93-bpnr2" Nov 03 00:55:35.300473 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:35.300446 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/318c5891-d03d-462f-91f1-de6c5a4a20f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2ce7eb93-bpnr2\" (UID: \"318c5891-d03d-462f-91f1-de6c5a4a20f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ce7eb93-bpnr2" Nov 03 00:55:35.302536 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:35.302513 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/318c5891-d03d-462f-91f1-de6c5a4a20f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2ce7eb93-bpnr2\" (UID: \"318c5891-d03d-462f-91f1-de6c5a4a20f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ce7eb93-bpnr2" Nov 03 00:55:35.331190 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:35.331154 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bhw4\" (UniqueName: \"kubernetes.io/projected/318c5891-d03d-462f-91f1-de6c5a4a20f6-kube-api-access-9bhw4\") pod \"maas-api-revoke-sub-2ce7eb93-bpnr2\" (UID: \"318c5891-d03d-462f-91f1-de6c5a4a20f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ce7eb93-bpnr2" Nov 03 00:55:35.400472 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:35.400426 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ce7eb93-bpnr2" Nov 03 00:55:35.759295 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:35.758833 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ce7eb93-bpnr2"] Nov 03 00:55:35.956347 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:35.956301 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ce7eb93-bpnr2" event={"ID":"318c5891-d03d-462f-91f1-de6c5a4a20f6","Type":"ContainerStarted","Data":"6519ccb0448815174b5d7163d3be656697d6dea1006ac1bb159b43f658ee7e3b"} Nov 03 00:55:35.960014 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:35.959989 4421 patch_prober.go:29] interesting pod/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-9jh88 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Nov 03 00:55:35.960158 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:35.960033 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-9jh88" podUID="72f72bc8-0d67-4200-ab31-c167fe8b877e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Nov 03 00:55:36.959137 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:36.959100 4421 patch_prober.go:29] interesting pod/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-9jh88 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Nov 03 00:55:36.959558 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:36.959168 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-9jh88" podUID="72f72bc8-0d67-4200-ab31-c167fe8b877e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Nov 03 00:55:36.966690 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:36.966662 4421 generic.go:356] "Generic (PLEG): container finished" podID="318c5891-d03d-462f-91f1-de6c5a4a20f6" containerID="551d3fdadc50e76d83b20c94806a4c04c5b24280969cf7fd3aa8a6223e7d634e" exitCode=0 Nov 03 00:55:36.966846 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:36.966741 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ce7eb93-bpnr2" event={"ID":"318c5891-d03d-462f-91f1-de6c5a4a20f6","Type":"ContainerDied","Data":"551d3fdadc50e76d83b20c94806a4c04c5b24280969cf7fd3aa8a6223e7d634e"} Nov 03 00:55:37.959409 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:37.959374 4421 patch_prober.go:29] interesting pod/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-9jh88 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Nov 03 00:55:37.959796 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:37.959424 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-9jh88" podUID="72f72bc8-0d67-4200-ab31-c167fe8b877e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Nov 03 00:55:38.179025 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:38.178997 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ce7eb93-bpnr2" Nov 03 00:55:38.276804 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:38.276771 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/318c5891-d03d-462f-91f1-de6c5a4a20f6-kube-api-access-9bhw4\" (UniqueName: \"kubernetes.io/projected/318c5891-d03d-462f-91f1-de6c5a4a20f6-kube-api-access-9bhw4\") pod \"318c5891-d03d-462f-91f1-de6c5a4a20f6\" (UID: \"318c5891-d03d-462f-91f1-de6c5a4a20f6\") " Nov 03 00:55:38.277014 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:38.276935 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/318c5891-d03d-462f-91f1-de6c5a4a20f6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/318c5891-d03d-462f-91f1-de6c5a4a20f6-maas-api-xks-ca\") pod \"318c5891-d03d-462f-91f1-de6c5a4a20f6\" (UID: \"318c5891-d03d-462f-91f1-de6c5a4a20f6\") " Nov 03 00:55:38.277014 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:38.277007 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/318c5891-d03d-462f-91f1-de6c5a4a20f6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/318c5891-d03d-462f-91f1-de6c5a4a20f6-maas-api-openshift-ca\") pod \"318c5891-d03d-462f-91f1-de6c5a4a20f6\" (UID: \"318c5891-d03d-462f-91f1-de6c5a4a20f6\") " Nov 03 00:55:38.277492 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:38.277412 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/318c5891-d03d-462f-91f1-de6c5a4a20f6-maas-api-openshift-ca" pod "318c5891-d03d-462f-91f1-de6c5a4a20f6" (UID: "318c5891-d03d-462f-91f1-de6c5a4a20f6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:55:38.277656 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:38.277639 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/318c5891-d03d-462f-91f1-de6c5a4a20f6-maas-api-openshift-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:55:38.279164 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:38.279136 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/318c5891-d03d-462f-91f1-de6c5a4a20f6-kube-api-access-9bhw4" pod "318c5891-d03d-462f-91f1-de6c5a4a20f6" (UID: "318c5891-d03d-462f-91f1-de6c5a4a20f6"). InnerVolumeSpecName "kube-api-access-9bhw4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:55:38.279251 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:38.279137 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/318c5891-d03d-462f-91f1-de6c5a4a20f6-maas-api-xks-ca" pod "318c5891-d03d-462f-91f1-de6c5a4a20f6" (UID: "318c5891-d03d-462f-91f1-de6c5a4a20f6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:55:38.379113 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:38.379077 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/318c5891-d03d-462f-91f1-de6c5a4a20f6-maas-api-xks-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:55:38.379113 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:38.379108 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9bhw4\" (UniqueName: \"kubernetes.io/projected/318c5891-d03d-462f-91f1-de6c5a4a20f6-kube-api-access-9bhw4\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:55:38.959512 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:38.959461 4421 patch_prober.go:29] interesting pod/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-9jh88 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Nov 03 00:55:38.960034 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:38.959600 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-9jh88" podUID="72f72bc8-0d67-4200-ab31-c167fe8b877e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Nov 03 00:55:38.984268 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:38.984237 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ce7eb93-bpnr2" Nov 03 00:55:38.984268 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:38.984255 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ce7eb93-bpnr2" event={"ID":"318c5891-d03d-462f-91f1-de6c5a4a20f6","Type":"ContainerDied","Data":"6519ccb0448815174b5d7163d3be656697d6dea1006ac1bb159b43f658ee7e3b"} Nov 03 00:55:38.984545 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:38.984317 4421 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6519ccb0448815174b5d7163d3be656697d6dea1006ac1bb159b43f658ee7e3b" Nov 03 00:55:39.959159 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:39.959121 4421 patch_prober.go:29] interesting pod/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-9jh88 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Nov 03 00:55:39.959362 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:39.959200 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-9jh88" podUID="72f72bc8-0d67-4200-ab31-c167fe8b877e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Nov 03 00:55:40.963862 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:40.963831 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-9jh88" Nov 03 00:55:40.964311 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:40.964237 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-9jh88" Nov 03 00:55:40.964901 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:40.964880 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-9jh88" Nov 03 00:55:44.907299 ip-10-0-133-217 kubenswrapper[4421]: E1103 00:55:44.906567 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 00:55:45.249380 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:45.246369 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/simulator-w2-f7680d-kserve-85bdc86cd6-8gxrg" podUID="2bdcf521-cebd-4d6b-8554-1b60db8b0f3c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:55:45.502492 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:45.502380 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/premium-w3-83755a-kserve-7c9d66988b-lt4xd" podUID="c0c6b02e-2ecb-428e-a13d-be6d7cd1ba39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Nov 03 00:55:45.596298 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:45.595514 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f13c8836-ns6nf"] Nov 03 00:55:45.611297 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:45.596543 4421 cpu_manager.go:414] "containerMap: removing container" podUID="318c5891-d03d-462f-91f1-de6c5a4a20f6" containerName="revoke-keys" Nov 03 00:55:45.611297 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:45.596567 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="318c5891-d03d-462f-91f1-de6c5a4a20f6" containerName="revoke-keys" Nov 03 00:55:45.611297 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:45.596716 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="318c5891-d03d-462f-91f1-de6c5a4a20f6" containerName="revoke-keys" Nov 03 00:55:45.689302 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:45.675610 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f13c8836-ns6nf" Nov 03 00:55:45.720208 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:45.720158 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8gm8n" podUID="2d367e12-e46f-4a68-9825-9b1c934d4d12" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:55:45.770393 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:45.764806 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a00a109-67c5-46a5-94ec-3bc37e97bf18-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f13c8836-ns6nf\" (UID: \"0a00a109-67c5-46a5-94ec-3bc37e97bf18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f13c8836-ns6nf" Nov 03 00:55:45.770393 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:45.764888 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a00a109-67c5-46a5-94ec-3bc37e97bf18-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f13c8836-ns6nf\" (UID: \"0a00a109-67c5-46a5-94ec-3bc37e97bf18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f13c8836-ns6nf" Nov 03 00:55:45.770393 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:45.765003 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtxjq\" (UniqueName: \"kubernetes.io/projected/0a00a109-67c5-46a5-94ec-3bc37e97bf18-kube-api-access-mtxjq\") pod \"maas-api-revoke-sub-f13c8836-ns6nf\" (UID: \"0a00a109-67c5-46a5-94ec-3bc37e97bf18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f13c8836-ns6nf" Nov 03 00:55:45.814331 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:45.813586 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:55:45.814331 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:45.813746 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6q4xp\"" Nov 03 00:55:45.814331 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:45.813834 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 00:55:45.814331 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:45.813897 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 00:55:45.885112 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:45.885062 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtxjq\" (UniqueName: \"kubernetes.io/projected/0a00a109-67c5-46a5-94ec-3bc37e97bf18-kube-api-access-mtxjq\") pod \"maas-api-revoke-sub-f13c8836-ns6nf\" (UID: \"0a00a109-67c5-46a5-94ec-3bc37e97bf18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f13c8836-ns6nf" Nov 03 00:55:45.885686 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:45.885656 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a00a109-67c5-46a5-94ec-3bc37e97bf18-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f13c8836-ns6nf\" (UID: \"0a00a109-67c5-46a5-94ec-3bc37e97bf18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f13c8836-ns6nf" Nov 03 00:55:45.885857 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:45.885837 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a00a109-67c5-46a5-94ec-3bc37e97bf18-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f13c8836-ns6nf\" (UID: \"0a00a109-67c5-46a5-94ec-3bc37e97bf18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f13c8836-ns6nf" Nov 03 00:55:45.900355 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:45.895776 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a00a109-67c5-46a5-94ec-3bc37e97bf18-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f13c8836-ns6nf\" (UID: \"0a00a109-67c5-46a5-94ec-3bc37e97bf18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f13c8836-ns6nf" Nov 03 00:55:45.904295 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:45.901689 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a00a109-67c5-46a5-94ec-3bc37e97bf18-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f13c8836-ns6nf\" (UID: \"0a00a109-67c5-46a5-94ec-3bc37e97bf18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f13c8836-ns6nf" Nov 03 00:55:45.925856 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:45.925811 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-650be0/premium-w4-650be0-kserve-7bf8555dfb-z97n5" podUID="40601dc5-808f-456f-826f-6e0c35966ada" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:55:46.026753 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:46.026354 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/distinct-w3-83755a-kserve-86bdbbdd85-2cnnl" podUID="f4ae9624-d127-4a55-837d-7c9ff0267b37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:55:46.186775 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:46.186650 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtxjq\" (UniqueName: \"kubernetes.io/projected/0a00a109-67c5-46a5-94ec-3bc37e97bf18-kube-api-access-mtxjq\") pod \"maas-api-revoke-sub-f13c8836-ns6nf\" (UID: \"0a00a109-67c5-46a5-94ec-3bc37e97bf18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f13c8836-ns6nf" Nov 03 00:55:46.205048 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:46.202997 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/simulator-w3-83755a-kserve-77d6664559-psqtg" podUID="b0a4c234-af58-4da0-a1a4-e1f167953f6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Nov 03 00:55:46.247355 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:46.247311 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/simulator-w2-f7680d-kserve-85bdc86cd6-8gxrg" podUID="2bdcf521-cebd-4d6b-8554-1b60db8b0f3c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:55:46.264344 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:46.264308 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f13c8836-ns6nf"] Nov 03 00:55:46.265024 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:46.265002 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f7680d/simulator-w2-f7680d-kserve-85bdc86cd6-8gxrg" Nov 03 00:55:46.407963 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:46.407874 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f7680d/simulator-w2-f7680d-kserve-85bdc86cd6-8gxrg" Nov 03 00:55:46.456519 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:46.456487 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f13c8836-ns6nf" Nov 03 00:55:46.782300 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:46.774350 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8gm8n" podUID="2d367e12-e46f-4a68-9825-9b1c934d4d12" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:55:46.782300 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:46.774483 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-650be0/simulator-w4-650be0-kserve-7bc588774f-gvz2k" podUID="a827fa88-f078-4312-8f12-2c799c11b461" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Nov 03 00:55:46.782300 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:46.776546 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8gm8n" Nov 03 00:55:46.832119 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:46.828425 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/premium-w3-83755a-kserve-7c9d66988b-lt4xd" podUID="c0c6b02e-2ecb-428e-a13d-be6d7cd1ba39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 00:55:46.832119 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:46.829161 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-83755a/premium-w3-83755a-kserve-7c9d66988b-lt4xd" Nov 03 00:55:46.948304 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:46.937514 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-650be0/premium-w4-650be0-kserve-7bf8555dfb-z97n5" podUID="40601dc5-808f-456f-826f-6e0c35966ada" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:55:46.948304 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:46.938335 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-650be0/premium-w4-650be0-kserve-7bf8555dfb-z97n5" Nov 03 00:55:47.093392 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:47.032444 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/distinct-w3-83755a-kserve-86bdbbdd85-2cnnl" podUID="f4ae9624-d127-4a55-837d-7c9ff0267b37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:55:47.093392 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:47.032874 4421 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-83755a/distinct-w3-83755a-kserve-86bdbbdd85-2cnnl" podUID="f4ae9624-d127-4a55-837d-7c9ff0267b37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/health\": context deadline exceeded" Nov 03 00:55:47.093392 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:47.033014 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-83755a/distinct-w3-83755a-kserve-86bdbbdd85-2cnnl" Nov 03 00:55:47.235595 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:47.235544 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/simulator-w3-83755a-kserve-77d6664559-psqtg" podUID="b0a4c234-af58-4da0-a1a4-e1f167953f6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Nov 03 00:55:47.240300 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:47.236475 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-83755a/simulator-w3-83755a-kserve-77d6664559-psqtg" Nov 03 00:55:47.449355 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:47.447367 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/simulator-w2-f7680d-kserve-85bdc86cd6-8gxrg" podUID="2bdcf521-cebd-4d6b-8554-1b60db8b0f3c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:55:47.814298 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:47.805533 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-650be0/simulator-w4-650be0-kserve-7bc588774f-gvz2k" podUID="a827fa88-f078-4312-8f12-2c799c11b461" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Nov 03 00:55:47.859917 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:47.824403 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-650be0/simulator-w4-650be0-kserve-7bc588774f-gvz2k" Nov 03 00:55:47.861064 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:47.855155 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/premium-w3-83755a-kserve-7c9d66988b-lt4xd" podUID="c0c6b02e-2ecb-428e-a13d-be6d7cd1ba39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Nov 03 00:55:47.861622 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:47.855350 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8gm8n" podUID="2d367e12-e46f-4a68-9825-9b1c934d4d12" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": context deadline exceeded" Nov 03 00:55:48.032431 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:48.032382 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-650be0/premium-w4-650be0-kserve-7bf8555dfb-z97n5" podUID="40601dc5-808f-456f-826f-6e0c35966ada" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 00:55:48.033118 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:48.033084 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/distinct-w3-83755a-kserve-86bdbbdd85-2cnnl" podUID="f4ae9624-d127-4a55-837d-7c9ff0267b37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:55:48.367035 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:48.299867 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/simulator-w3-83755a-kserve-77d6664559-psqtg" podUID="b0a4c234-af58-4da0-a1a4-e1f167953f6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:55:48.438039 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:48.435799 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-83755a/distinct-w3-83755a-kserve-86bdbbdd85-2cnnl" Nov 03 00:55:48.476432 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:48.476388 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/simulator-w2-f7680d-kserve-85bdc86cd6-8gxrg" podUID="2bdcf521-cebd-4d6b-8554-1b60db8b0f3c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 03 00:55:48.505362 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:48.479079 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f7680d/simulator-w2-f7680d-kserve-85bdc86cd6-8gxrg" Nov 03 00:55:48.820726 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:48.820667 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-650be0/simulator-w4-650be0-kserve-7bc588774f-gvz2k" podUID="a827fa88-f078-4312-8f12-2c799c11b461" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:55:48.877448 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:48.877398 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/premium-w3-83755a-kserve-7c9d66988b-lt4xd" podUID="c0c6b02e-2ecb-428e-a13d-be6d7cd1ba39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:55:49.105905 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:49.105227 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8gm8n" podUID="2d367e12-e46f-4a68-9825-9b1c934d4d12" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": context deadline exceeded" Nov 03 00:55:49.110636 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:49.106466 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-650be0/premium-w4-650be0-kserve-7bf8555dfb-z97n5" podUID="40601dc5-808f-456f-826f-6e0c35966ada" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Nov 03 00:55:49.237296 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:49.202245 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f7680d/simulator-w2-f7680d-kserve-85bdc86cd6-8gxrg" Nov 03 00:55:49.237296 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:49.220380 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8gm8n" Nov 03 00:55:49.273223 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:49.273197 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-650be0/premium-w4-650be0-kserve-7bf8555dfb-z97n5" Nov 03 00:55:49.302326 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:49.301301 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/simulator-w3-83755a-kserve-77d6664559-psqtg" podUID="b0a4c234-af58-4da0-a1a4-e1f167953f6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:55:49.359394 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:49.355842 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-650be0/simulator-w4-650be0-kserve-7bc588774f-gvz2k" Nov 03 00:55:49.360078 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:49.360051 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-83755a/simulator-w3-83755a-kserve-77d6664559-psqtg" Nov 03 00:55:49.360546 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:49.360528 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-83755a/premium-w3-83755a-kserve-7c9d66988b-lt4xd" Nov 03 00:55:49.373668 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:49.373592 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f13c8836-ns6nf"] Nov 03 00:55:50.371540 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:50.371502 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f13c8836-ns6nf" event={"ID":"0a00a109-67c5-46a5-94ec-3bc37e97bf18","Type":"ContainerStarted","Data":"9ee0e0d2bff70cd6872ba72d3c2cd2bd853bab6f4b8402e732862e525f1d89f5"} Nov 03 00:55:50.371999 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:50.371580 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f13c8836-ns6nf" event={"ID":"0a00a109-67c5-46a5-94ec-3bc37e97bf18","Type":"ContainerStarted","Data":"4847c55d1f2f2cf3b387cfbeb370164b373e071a314ff0d065b5b0a656d01be3"} Nov 03 00:55:50.392564 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:50.392489 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f13c8836-ns6nf" podStartSLOduration=5.392466641 podStartE2EDuration="5.392466641s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:55:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:55:50.388989226 +0000 UTC m=+892.559992347" watchObservedRunningTime="2026-11-03 00:55:50.392466641 +0000 UTC m=+892.563469737" Nov 03 00:55:51.379778 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:51.379744 4421 generic.go:356] "Generic (PLEG): container finished" podID="0a00a109-67c5-46a5-94ec-3bc37e97bf18" containerID="9ee0e0d2bff70cd6872ba72d3c2cd2bd853bab6f4b8402e732862e525f1d89f5" exitCode=0 Nov 03 00:55:51.380260 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:51.379836 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f13c8836-ns6nf" event={"ID":"0a00a109-67c5-46a5-94ec-3bc37e97bf18","Type":"ContainerDied","Data":"9ee0e0d2bff70cd6872ba72d3c2cd2bd853bab6f4b8402e732862e525f1d89f5"} Nov 03 00:55:52.585493 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:52.585464 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f13c8836-ns6nf" Nov 03 00:55:52.674396 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:52.674315 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a00a109-67c5-46a5-94ec-3bc37e97bf18-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a00a109-67c5-46a5-94ec-3bc37e97bf18-maas-api-openshift-ca\") pod \"0a00a109-67c5-46a5-94ec-3bc37e97bf18\" (UID: \"0a00a109-67c5-46a5-94ec-3bc37e97bf18\") " Nov 03 00:55:52.674396 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:52.674380 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a00a109-67c5-46a5-94ec-3bc37e97bf18-kube-api-access-mtxjq\" (UniqueName: \"kubernetes.io/projected/0a00a109-67c5-46a5-94ec-3bc37e97bf18-kube-api-access-mtxjq\") pod \"0a00a109-67c5-46a5-94ec-3bc37e97bf18\" (UID: \"0a00a109-67c5-46a5-94ec-3bc37e97bf18\") " Nov 03 00:55:52.674608 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:52.674486 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a00a109-67c5-46a5-94ec-3bc37e97bf18-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a00a109-67c5-46a5-94ec-3bc37e97bf18-maas-api-xks-ca\") pod \"0a00a109-67c5-46a5-94ec-3bc37e97bf18\" (UID: \"0a00a109-67c5-46a5-94ec-3bc37e97bf18\") " Nov 03 00:55:52.674670 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:52.674647 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a00a109-67c5-46a5-94ec-3bc37e97bf18-maas-api-openshift-ca" pod "0a00a109-67c5-46a5-94ec-3bc37e97bf18" (UID: "0a00a109-67c5-46a5-94ec-3bc37e97bf18"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:55:52.675110 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:52.675090 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a00a109-67c5-46a5-94ec-3bc37e97bf18-maas-api-openshift-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:55:52.676425 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:52.676393 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a00a109-67c5-46a5-94ec-3bc37e97bf18-kube-api-access-mtxjq" pod "0a00a109-67c5-46a5-94ec-3bc37e97bf18" (UID: "0a00a109-67c5-46a5-94ec-3bc37e97bf18"). InnerVolumeSpecName "kube-api-access-mtxjq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:55:52.676528 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:52.676402 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a00a109-67c5-46a5-94ec-3bc37e97bf18-maas-api-xks-ca" pod "0a00a109-67c5-46a5-94ec-3bc37e97bf18" (UID: "0a00a109-67c5-46a5-94ec-3bc37e97bf18"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:55:52.777025 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:52.776975 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mtxjq\" (UniqueName: \"kubernetes.io/projected/0a00a109-67c5-46a5-94ec-3bc37e97bf18-kube-api-access-mtxjq\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:55:52.777025 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:52.777025 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a00a109-67c5-46a5-94ec-3bc37e97bf18-maas-api-xks-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:55:53.398216 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:53.398187 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f13c8836-ns6nf" Nov 03 00:55:53.398430 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:53.398213 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f13c8836-ns6nf" event={"ID":"0a00a109-67c5-46a5-94ec-3bc37e97bf18","Type":"ContainerDied","Data":"4847c55d1f2f2cf3b387cfbeb370164b373e071a314ff0d065b5b0a656d01be3"} Nov 03 00:55:53.398430 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:53.398288 4421 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4847c55d1f2f2cf3b387cfbeb370164b373e071a314ff0d065b5b0a656d01be3" Nov 03 00:55:53.737462 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:53.737375 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-cmdm8"] Nov 03 00:55:53.738206 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:53.738182 4421 cpu_manager.go:414] "containerMap: removing container" podUID="0a00a109-67c5-46a5-94ec-3bc37e97bf18" containerName="revoke-keys" Nov 03 00:55:53.738354 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:53.738208 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a00a109-67c5-46a5-94ec-3bc37e97bf18" containerName="revoke-keys" Nov 03 00:55:53.738423 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:53.738389 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a00a109-67c5-46a5-94ec-3bc37e97bf18" containerName="revoke-keys" Nov 03 00:55:53.742350 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:53.742327 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-cmdm8" Nov 03 00:55:53.750220 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:53.750190 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-cmdm8"] Nov 03 00:55:53.791428 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:53.791395 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9916b060-ab92-4836-afab-cd531100dd69-istio-data\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-cmdm8\" (UID: \"9916b060-ab92-4836-afab-cd531100dd69\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-cmdm8" Nov 03 00:55:53.791606 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:53.791462 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9916b060-ab92-4836-afab-cd531100dd69-istio-envoy\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-cmdm8\" (UID: \"9916b060-ab92-4836-afab-cd531100dd69\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-cmdm8" Nov 03 00:55:53.791606 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:53.791544 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9916b060-ab92-4836-afab-cd531100dd69-istio-podinfo\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-cmdm8\" (UID: \"9916b060-ab92-4836-afab-cd531100dd69\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-cmdm8" Nov 03 00:55:53.791698 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:53.791606 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9916b060-ab92-4836-afab-cd531100dd69-workload-certs\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-cmdm8\" (UID: \"9916b060-ab92-4836-afab-cd531100dd69\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-cmdm8" Nov 03 00:55:53.791698 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:53.791625 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9916b060-ab92-4836-afab-cd531100dd69-credential-socket\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-cmdm8\" (UID: \"9916b060-ab92-4836-afab-cd531100dd69\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-cmdm8" Nov 03 00:55:53.791698 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:53.791641 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9916b060-ab92-4836-afab-cd531100dd69-istio-token\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-cmdm8\" (UID: \"9916b060-ab92-4836-afab-cd531100dd69\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-cmdm8" Nov 03 00:55:53.791698 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:53.791658 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9916b060-ab92-4836-afab-cd531100dd69-istiod-ca-cert\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-cmdm8\" (UID: \"9916b060-ab92-4836-afab-cd531100dd69\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-cmdm8" Nov 03 00:55:53.791847 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:53.791721 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9916b060-ab92-4836-afab-cd531100dd69-workload-socket\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-cmdm8\" (UID: \"9916b060-ab92-4836-afab-cd531100dd69\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-cmdm8" Nov 03 00:55:53.791847 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:53.791755 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wczfd\" (UniqueName: \"kubernetes.io/projected/9916b060-ab92-4836-afab-cd531100dd69-kube-api-access-wczfd\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-cmdm8\" (UID: \"9916b060-ab92-4836-afab-cd531100dd69\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-cmdm8" Nov 03 00:55:53.893076 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:53.893037 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9916b060-ab92-4836-afab-cd531100dd69-workload-certs\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-cmdm8\" (UID: \"9916b060-ab92-4836-afab-cd531100dd69\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-cmdm8" Nov 03 00:55:53.893076 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:53.893082 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9916b060-ab92-4836-afab-cd531100dd69-credential-socket\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-cmdm8\" (UID: \"9916b060-ab92-4836-afab-cd531100dd69\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-cmdm8" Nov 03 00:55:53.893346 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:53.893102 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9916b060-ab92-4836-afab-cd531100dd69-istio-token\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-cmdm8\" (UID: \"9916b060-ab92-4836-afab-cd531100dd69\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-cmdm8" Nov 03 00:55:53.893346 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:53.893201 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9916b060-ab92-4836-afab-cd531100dd69-istiod-ca-cert\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-cmdm8\" (UID: \"9916b060-ab92-4836-afab-cd531100dd69\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-cmdm8" Nov 03 00:55:53.893346 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:53.893319 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9916b060-ab92-4836-afab-cd531100dd69-workload-socket\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-cmdm8\" (UID: \"9916b060-ab92-4836-afab-cd531100dd69\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-cmdm8" Nov 03 00:55:53.893595 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:53.893373 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wczfd\" (UniqueName: \"kubernetes.io/projected/9916b060-ab92-4836-afab-cd531100dd69-kube-api-access-wczfd\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-cmdm8\" (UID: \"9916b060-ab92-4836-afab-cd531100dd69\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-cmdm8" Nov 03 00:55:53.893595 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:53.893491 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9916b060-ab92-4836-afab-cd531100dd69-istio-data\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-cmdm8\" (UID: \"9916b060-ab92-4836-afab-cd531100dd69\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-cmdm8" Nov 03 00:55:53.893595 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:53.893558 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9916b060-ab92-4836-afab-cd531100dd69-istio-envoy\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-cmdm8\" (UID: \"9916b060-ab92-4836-afab-cd531100dd69\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-cmdm8" Nov 03 00:55:53.893754 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:53.893598 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9916b060-ab92-4836-afab-cd531100dd69-credential-socket\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-cmdm8\" (UID: \"9916b060-ab92-4836-afab-cd531100dd69\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-cmdm8" Nov 03 00:55:53.893754 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:53.893615 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9916b060-ab92-4836-afab-cd531100dd69-istio-podinfo\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-cmdm8\" (UID: \"9916b060-ab92-4836-afab-cd531100dd69\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-cmdm8" Nov 03 00:55:53.893754 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:53.893606 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9916b060-ab92-4836-afab-cd531100dd69-workload-certs\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-cmdm8\" (UID: \"9916b060-ab92-4836-afab-cd531100dd69\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-cmdm8" Nov 03 00:55:53.893983 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:53.893961 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9916b060-ab92-4836-afab-cd531100dd69-workload-socket\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-cmdm8\" (UID: \"9916b060-ab92-4836-afab-cd531100dd69\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-cmdm8" Nov 03 00:55:53.894201 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:53.894146 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9916b060-ab92-4836-afab-cd531100dd69-istiod-ca-cert\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-cmdm8\" (UID: \"9916b060-ab92-4836-afab-cd531100dd69\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-cmdm8" Nov 03 00:55:53.894201 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:53.894178 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9916b060-ab92-4836-afab-cd531100dd69-istio-data\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-cmdm8\" (UID: \"9916b060-ab92-4836-afab-cd531100dd69\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-cmdm8" Nov 03 00:55:53.896513 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:53.896489 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9916b060-ab92-4836-afab-cd531100dd69-istio-envoy\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-cmdm8\" (UID: \"9916b060-ab92-4836-afab-cd531100dd69\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-cmdm8" Nov 03 00:55:53.896625 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:53.896609 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9916b060-ab92-4836-afab-cd531100dd69-istio-podinfo\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-cmdm8\" (UID: \"9916b060-ab92-4836-afab-cd531100dd69\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-cmdm8" Nov 03 00:55:53.900892 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:53.900860 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9916b060-ab92-4836-afab-cd531100dd69-istio-token\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-cmdm8\" (UID: \"9916b060-ab92-4836-afab-cd531100dd69\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-cmdm8" Nov 03 00:55:53.901492 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:53.901472 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wczfd\" (UniqueName: \"kubernetes.io/projected/9916b060-ab92-4836-afab-cd531100dd69-kube-api-access-wczfd\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-cmdm8\" (UID: \"9916b060-ab92-4836-afab-cd531100dd69\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-cmdm8" Nov 03 00:55:54.065916 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:54.065877 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-cmdm8" Nov 03 00:55:54.392801 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:54.390268 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-cmdm8"] Nov 03 00:55:54.395555 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:54.395522 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:55:54.395640 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:54.395586 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:55:54.395640 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:54.395618 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:55:54.407961 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:54.407932 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-cmdm8" event={"ID":"9916b060-ab92-4836-afab-cd531100dd69","Type":"ContainerStarted","Data":"79dd8b4e3a1dfa127919acbbae7bb94d210ca157d3facc91a2d0ad9188983ac6"} Nov 03 00:55:55.446478 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:55.446431 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-cmdm8" event={"ID":"9916b060-ab92-4836-afab-cd531100dd69","Type":"ContainerStarted","Data":"b03ef5608292522d186a3c306168b9ba29d7732b01214764dd988da139dcef5c"} Nov 03 00:55:55.482248 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:55.482187 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-cmdm8" podStartSLOduration=2.482171602 podStartE2EDuration="2.482171602s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:55:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:55:55.478457749 +0000 UTC m=+897.649460835" watchObservedRunningTime="2026-11-03 00:55:55.482171602 +0000 UTC m=+897.653174698" Nov 03 00:55:56.072115 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:56.072078 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-cmdm8" Nov 03 00:55:56.073933 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:56.073901 4421 patch_prober.go:29] interesting pod/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-cmdm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Nov 03 00:55:56.074215 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:56.074184 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-cmdm8" podUID="9916b060-ab92-4836-afab-cd531100dd69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Nov 03 00:55:57.072880 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:57.069355 4421 patch_prober.go:29] interesting pod/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-cmdm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Nov 03 00:55:57.072880 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:57.069420 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-cmdm8" podUID="9916b060-ab92-4836-afab-cd531100dd69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Nov 03 00:55:57.957514 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:57.957473 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-51119c-openshift-default-56b69d85d9-ds9fq"] Nov 03 00:55:57.963177 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:57.963150 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-51119c-openshift-default-56b69d85d9-ds9fq" Nov 03 00:55:57.977792 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:57.976855 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/934ddd77-9c41-49d5-bf51-fb5d8e765efd-istio-podinfo\") pod \"e2e-api-a-51119c-openshift-default-56b69d85d9-ds9fq\" (UID: \"934ddd77-9c41-49d5-bf51-fb5d8e765efd\") " pod="openshift-ingress/e2e-api-a-51119c-openshift-default-56b69d85d9-ds9fq" Nov 03 00:55:57.977792 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:57.976899 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/934ddd77-9c41-49d5-bf51-fb5d8e765efd-credential-socket\") pod \"e2e-api-a-51119c-openshift-default-56b69d85d9-ds9fq\" (UID: \"934ddd77-9c41-49d5-bf51-fb5d8e765efd\") " pod="openshift-ingress/e2e-api-a-51119c-openshift-default-56b69d85d9-ds9fq" Nov 03 00:55:57.979525 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:57.978128 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/934ddd77-9c41-49d5-bf51-fb5d8e765efd-workload-certs\") pod \"e2e-api-a-51119c-openshift-default-56b69d85d9-ds9fq\" (UID: \"934ddd77-9c41-49d5-bf51-fb5d8e765efd\") " pod="openshift-ingress/e2e-api-a-51119c-openshift-default-56b69d85d9-ds9fq" Nov 03 00:55:57.979525 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:57.978208 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/934ddd77-9c41-49d5-bf51-fb5d8e765efd-istiod-ca-cert\") pod \"e2e-api-a-51119c-openshift-default-56b69d85d9-ds9fq\" (UID: \"934ddd77-9c41-49d5-bf51-fb5d8e765efd\") " pod="openshift-ingress/e2e-api-a-51119c-openshift-default-56b69d85d9-ds9fq" Nov 03 00:55:57.979525 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:57.978315 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/934ddd77-9c41-49d5-bf51-fb5d8e765efd-istio-data\") pod \"e2e-api-a-51119c-openshift-default-56b69d85d9-ds9fq\" (UID: \"934ddd77-9c41-49d5-bf51-fb5d8e765efd\") " pod="openshift-ingress/e2e-api-a-51119c-openshift-default-56b69d85d9-ds9fq" Nov 03 00:55:57.979525 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:57.978355 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/934ddd77-9c41-49d5-bf51-fb5d8e765efd-istio-token\") pod \"e2e-api-a-51119c-openshift-default-56b69d85d9-ds9fq\" (UID: \"934ddd77-9c41-49d5-bf51-fb5d8e765efd\") " pod="openshift-ingress/e2e-api-a-51119c-openshift-default-56b69d85d9-ds9fq" Nov 03 00:55:57.979525 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:57.978435 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2bgq\" (UniqueName: \"kubernetes.io/projected/934ddd77-9c41-49d5-bf51-fb5d8e765efd-kube-api-access-d2bgq\") pod \"e2e-api-a-51119c-openshift-default-56b69d85d9-ds9fq\" (UID: \"934ddd77-9c41-49d5-bf51-fb5d8e765efd\") " pod="openshift-ingress/e2e-api-a-51119c-openshift-default-56b69d85d9-ds9fq" Nov 03 00:55:57.979525 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:57.978471 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/934ddd77-9c41-49d5-bf51-fb5d8e765efd-workload-socket\") pod \"e2e-api-a-51119c-openshift-default-56b69d85d9-ds9fq\" (UID: \"934ddd77-9c41-49d5-bf51-fb5d8e765efd\") " pod="openshift-ingress/e2e-api-a-51119c-openshift-default-56b69d85d9-ds9fq" Nov 03 00:55:57.979525 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:57.978555 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/934ddd77-9c41-49d5-bf51-fb5d8e765efd-istio-envoy\") pod \"e2e-api-a-51119c-openshift-default-56b69d85d9-ds9fq\" (UID: \"934ddd77-9c41-49d5-bf51-fb5d8e765efd\") " pod="openshift-ingress/e2e-api-a-51119c-openshift-default-56b69d85d9-ds9fq" Nov 03 00:55:57.980901 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:57.980857 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-51119c-openshift-default-dockercfg-jhc66\"" Nov 03 00:55:58.050438 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:58.049638 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-51119c-openshift-default-56b69d85d9-ds9fq"] Nov 03 00:55:58.067244 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:58.066681 4421 patch_prober.go:29] interesting pod/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-cmdm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Nov 03 00:55:58.067244 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:58.066734 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-cmdm8" podUID="9916b060-ab92-4836-afab-cd531100dd69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Nov 03 00:55:58.082548 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:58.081167 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/934ddd77-9c41-49d5-bf51-fb5d8e765efd-istio-envoy\") pod \"e2e-api-a-51119c-openshift-default-56b69d85d9-ds9fq\" (UID: \"934ddd77-9c41-49d5-bf51-fb5d8e765efd\") " pod="openshift-ingress/e2e-api-a-51119c-openshift-default-56b69d85d9-ds9fq" Nov 03 00:55:58.082548 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:58.081234 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/934ddd77-9c41-49d5-bf51-fb5d8e765efd-istio-podinfo\") pod \"e2e-api-a-51119c-openshift-default-56b69d85d9-ds9fq\" (UID: \"934ddd77-9c41-49d5-bf51-fb5d8e765efd\") " pod="openshift-ingress/e2e-api-a-51119c-openshift-default-56b69d85d9-ds9fq" Nov 03 00:55:58.082548 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:58.081298 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/934ddd77-9c41-49d5-bf51-fb5d8e765efd-credential-socket\") pod \"e2e-api-a-51119c-openshift-default-56b69d85d9-ds9fq\" (UID: \"934ddd77-9c41-49d5-bf51-fb5d8e765efd\") " pod="openshift-ingress/e2e-api-a-51119c-openshift-default-56b69d85d9-ds9fq" Nov 03 00:55:58.082548 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:58.081420 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/934ddd77-9c41-49d5-bf51-fb5d8e765efd-workload-certs\") pod \"e2e-api-a-51119c-openshift-default-56b69d85d9-ds9fq\" (UID: \"934ddd77-9c41-49d5-bf51-fb5d8e765efd\") " pod="openshift-ingress/e2e-api-a-51119c-openshift-default-56b69d85d9-ds9fq" Nov 03 00:55:58.082548 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:58.081498 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/934ddd77-9c41-49d5-bf51-fb5d8e765efd-istiod-ca-cert\") pod \"e2e-api-a-51119c-openshift-default-56b69d85d9-ds9fq\" (UID: \"934ddd77-9c41-49d5-bf51-fb5d8e765efd\") " pod="openshift-ingress/e2e-api-a-51119c-openshift-default-56b69d85d9-ds9fq" Nov 03 00:55:58.082548 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:58.081602 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/934ddd77-9c41-49d5-bf51-fb5d8e765efd-istio-data\") pod \"e2e-api-a-51119c-openshift-default-56b69d85d9-ds9fq\" (UID: \"934ddd77-9c41-49d5-bf51-fb5d8e765efd\") " pod="openshift-ingress/e2e-api-a-51119c-openshift-default-56b69d85d9-ds9fq" Nov 03 00:55:58.082548 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:58.081647 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/934ddd77-9c41-49d5-bf51-fb5d8e765efd-istio-token\") pod \"e2e-api-a-51119c-openshift-default-56b69d85d9-ds9fq\" (UID: \"934ddd77-9c41-49d5-bf51-fb5d8e765efd\") " pod="openshift-ingress/e2e-api-a-51119c-openshift-default-56b69d85d9-ds9fq" Nov 03 00:55:58.082548 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:58.081750 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2bgq\" (UniqueName: \"kubernetes.io/projected/934ddd77-9c41-49d5-bf51-fb5d8e765efd-kube-api-access-d2bgq\") pod \"e2e-api-a-51119c-openshift-default-56b69d85d9-ds9fq\" (UID: \"934ddd77-9c41-49d5-bf51-fb5d8e765efd\") " pod="openshift-ingress/e2e-api-a-51119c-openshift-default-56b69d85d9-ds9fq" Nov 03 00:55:58.082548 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:58.081784 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/934ddd77-9c41-49d5-bf51-fb5d8e765efd-workload-socket\") pod \"e2e-api-a-51119c-openshift-default-56b69d85d9-ds9fq\" (UID: \"934ddd77-9c41-49d5-bf51-fb5d8e765efd\") " pod="openshift-ingress/e2e-api-a-51119c-openshift-default-56b69d85d9-ds9fq" Nov 03 00:55:58.082548 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:58.082234 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/934ddd77-9c41-49d5-bf51-fb5d8e765efd-workload-socket\") pod \"e2e-api-a-51119c-openshift-default-56b69d85d9-ds9fq\" (UID: \"934ddd77-9c41-49d5-bf51-fb5d8e765efd\") " pod="openshift-ingress/e2e-api-a-51119c-openshift-default-56b69d85d9-ds9fq" Nov 03 00:55:58.083553 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:58.082906 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/934ddd77-9c41-49d5-bf51-fb5d8e765efd-credential-socket\") pod \"e2e-api-a-51119c-openshift-default-56b69d85d9-ds9fq\" (UID: \"934ddd77-9c41-49d5-bf51-fb5d8e765efd\") " pod="openshift-ingress/e2e-api-a-51119c-openshift-default-56b69d85d9-ds9fq" Nov 03 00:55:58.083553 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:58.083157 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/934ddd77-9c41-49d5-bf51-fb5d8e765efd-workload-certs\") pod \"e2e-api-a-51119c-openshift-default-56b69d85d9-ds9fq\" (UID: \"934ddd77-9c41-49d5-bf51-fb5d8e765efd\") " pod="openshift-ingress/e2e-api-a-51119c-openshift-default-56b69d85d9-ds9fq" Nov 03 00:55:58.085405 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:58.083901 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/934ddd77-9c41-49d5-bf51-fb5d8e765efd-istiod-ca-cert\") pod \"e2e-api-a-51119c-openshift-default-56b69d85d9-ds9fq\" (UID: \"934ddd77-9c41-49d5-bf51-fb5d8e765efd\") " pod="openshift-ingress/e2e-api-a-51119c-openshift-default-56b69d85d9-ds9fq" Nov 03 00:55:58.085405 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:58.084152 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/934ddd77-9c41-49d5-bf51-fb5d8e765efd-istio-data\") pod \"e2e-api-a-51119c-openshift-default-56b69d85d9-ds9fq\" (UID: \"934ddd77-9c41-49d5-bf51-fb5d8e765efd\") " pod="openshift-ingress/e2e-api-a-51119c-openshift-default-56b69d85d9-ds9fq" Nov 03 00:55:58.089768 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:58.089222 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/934ddd77-9c41-49d5-bf51-fb5d8e765efd-istio-envoy\") pod \"e2e-api-a-51119c-openshift-default-56b69d85d9-ds9fq\" (UID: \"934ddd77-9c41-49d5-bf51-fb5d8e765efd\") " pod="openshift-ingress/e2e-api-a-51119c-openshift-default-56b69d85d9-ds9fq" Nov 03 00:55:58.089768 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:58.089720 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/934ddd77-9c41-49d5-bf51-fb5d8e765efd-istio-podinfo\") pod \"e2e-api-a-51119c-openshift-default-56b69d85d9-ds9fq\" (UID: \"934ddd77-9c41-49d5-bf51-fb5d8e765efd\") " pod="openshift-ingress/e2e-api-a-51119c-openshift-default-56b69d85d9-ds9fq" Nov 03 00:55:58.098922 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:58.098890 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/934ddd77-9c41-49d5-bf51-fb5d8e765efd-istio-token\") pod \"e2e-api-a-51119c-openshift-default-56b69d85d9-ds9fq\" (UID: \"934ddd77-9c41-49d5-bf51-fb5d8e765efd\") " pod="openshift-ingress/e2e-api-a-51119c-openshift-default-56b69d85d9-ds9fq" Nov 03 00:55:58.099516 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:58.099487 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2bgq\" (UniqueName: \"kubernetes.io/projected/934ddd77-9c41-49d5-bf51-fb5d8e765efd-kube-api-access-d2bgq\") pod \"e2e-api-a-51119c-openshift-default-56b69d85d9-ds9fq\" (UID: \"934ddd77-9c41-49d5-bf51-fb5d8e765efd\") " pod="openshift-ingress/e2e-api-a-51119c-openshift-default-56b69d85d9-ds9fq" Nov 03 00:55:58.392866 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:58.391563 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-51119c-openshift-default-56b69d85d9-ds9fq" Nov 03 00:55:58.833287 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:58.833234 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-51119c-openshift-default-56b69d85d9-ds9fq"] Nov 03 00:55:58.843567 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:58.843521 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:55:58.843662 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:58.843618 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:55:58.843699 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:58.843668 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:55:59.072295 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:59.071618 4421 patch_prober.go:29] interesting pod/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-cmdm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Nov 03 00:55:59.072295 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:59.071676 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-cmdm8" podUID="9916b060-ab92-4836-afab-cd531100dd69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Nov 03 00:55:59.540299 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:59.539670 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-51119c-openshift-default-56b69d85d9-ds9fq" event={"ID":"934ddd77-9c41-49d5-bf51-fb5d8e765efd","Type":"ContainerStarted","Data":"c5363a1ef8d1f7d5e5414208fac3058904333d527167c17f09e4035434ddc459"} Nov 03 00:55:59.540299 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:59.539723 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-51119c-openshift-default-56b69d85d9-ds9fq" event={"ID":"934ddd77-9c41-49d5-bf51-fb5d8e765efd","Type":"ContainerStarted","Data":"596402284c283ae1b74bdd855f63cc3102d44907f06067752e23fc018c0af46f"} Nov 03 00:55:59.565956 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:59.565894 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-51119c-openshift-default-56b69d85d9-ds9fq" podStartSLOduration=2.565880092 podStartE2EDuration="2.565880092s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:55:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:55:59.560671816 +0000 UTC m=+901.731674913" watchObservedRunningTime="2026-11-03 00:55:59.565880092 +0000 UTC m=+901.736883187" Nov 03 00:55:59.737315 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:59.718391 4421 patch_prober.go:29] interesting pod/e2e-mt-dd03e395-openshift-default-75799f6fc-kx54z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 00:55:59.737315 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:55:59.718468 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-dd03e395-openshift-default-75799f6fc-kx54z" podUID="01442ff5-1dab-4ed3-991b-ec0732c71305" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 00:56:00.066389 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:00.066343 4421 patch_prober.go:29] interesting pod/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-cmdm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Nov 03 00:56:00.066595 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:00.066405 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-cmdm8" podUID="9916b060-ab92-4836-afab-cd531100dd69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Nov 03 00:56:00.392920 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:00.392885 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-51119c-openshift-default-56b69d85d9-ds9fq" Nov 03 00:56:00.399019 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:00.398995 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-51119c-openshift-default-56b69d85d9-ds9fq" Nov 03 00:56:00.549404 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:00.549370 4421 generic.go:356] "Generic (PLEG): container finished" podID="01442ff5-1dab-4ed3-991b-ec0732c71305" containerID="5f145538a80b3358ab75e1c1021050da283eed8d6d1f9782ad088f6ef764e4a4" exitCode=0 Nov 03 00:56:00.549910 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:00.549434 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-dd03e395-openshift-default-75799f6fc-kx54z" event={"ID":"01442ff5-1dab-4ed3-991b-ec0732c71305","Type":"ContainerDied","Data":"5f145538a80b3358ab75e1c1021050da283eed8d6d1f9782ad088f6ef764e4a4"} Nov 03 00:56:00.549910 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:00.549790 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-51119c-openshift-default-56b69d85d9-ds9fq" Nov 03 00:56:00.550040 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:00.550018 4421 scope.go:122] "RemoveContainer" containerID="5f145538a80b3358ab75e1c1021050da283eed8d6d1f9782ad088f6ef764e4a4" Nov 03 00:56:00.551098 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:00.551079 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-51119c-openshift-default-56b69d85d9-ds9fq" Nov 03 00:56:00.551588 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:00.551560 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:56:00.551643 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:00.551629 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:56:00.551679 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:00.551665 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:56:01.066679 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:01.066635 4421 patch_prober.go:29] interesting pod/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-cmdm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Nov 03 00:56:01.066895 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:01.066700 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-cmdm8" podUID="9916b060-ab92-4836-afab-cd531100dd69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Nov 03 00:56:01.560517 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:01.560475 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-dd03e395-openshift-default-75799f6fc-kx54z" event={"ID":"01442ff5-1dab-4ed3-991b-ec0732c71305","Type":"ContainerStarted","Data":"07257ec70b22e16c24e895bbf7ddfea16c3a4228ef7975bb263a6ecb2c2b5086"} Nov 03 00:56:02.045028 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:02.044986 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-dd03e395-openshift-default-75799f6fc-kx54z" Nov 03 00:56:02.045248 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:02.045227 4421 patch_prober.go:29] interesting pod/e2e-mt-dd03e395-openshift-default-75799f6fc-kx54z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body= Nov 03 00:56:02.045355 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:02.045299 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-dd03e395-openshift-default-75799f6fc-kx54z" podUID="01442ff5-1dab-4ed3-991b-ec0732c71305" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" Nov 03 00:56:03.058199 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:03.049577 4421 patch_prober.go:29] interesting pod/e2e-mt-dd03e395-openshift-default-75799f6fc-kx54z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body= Nov 03 00:56:03.058199 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:03.049639 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-dd03e395-openshift-default-75799f6fc-kx54z" podUID="01442ff5-1dab-4ed3-991b-ec0732c71305" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" Nov 03 00:56:03.068306 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:03.065547 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-51119c-7988b955f5-kr9rl"] Nov 03 00:56:03.091388 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:03.070966 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-51119c-7988b955f5-kr9rl" Nov 03 00:56:03.091388 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:03.076139 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-51119c\"" Nov 03 00:56:03.091388 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:03.076942 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-51119c-dockercfg-xvg6z\"" Nov 03 00:56:03.097918 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:03.097887 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-51119c-546967fcc9-mtcnh"] Nov 03 00:56:03.102898 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:03.102872 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-51119c-546967fcc9-mtcnh" Nov 03 00:56:03.136300 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:03.113118 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-51119c-7988b955f5-kr9rl"] Nov 03 00:56:03.136300 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:03.113738 4421 patch_prober.go:29] interesting pod/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-cmdm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 00:56:03.136300 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:03.113792 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-cmdm8" podUID="9916b060-ab92-4836-afab-cd531100dd69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 00:56:03.190299 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:03.186612 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-51119c-546967fcc9-mtcnh"] Nov 03 00:56:03.203348 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:03.191549 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4357adf-7a95-4137-a489-ba7c3c7855f3-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-51119c-7988b955f5-kr9rl\" (UID: \"a4357adf-7a95-4137-a489-ba7c3c7855f3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-51119c-7988b955f5-kr9rl" Nov 03 00:56:03.203348 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:03.191616 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pckr\" (UniqueName: \"kubernetes.io/projected/a4357adf-7a95-4137-a489-ba7c3c7855f3-kube-api-access-9pckr\") pod \"payload-pre-processing-e2e-api-a-51119c-7988b955f5-kr9rl\" (UID: \"a4357adf-7a95-4137-a489-ba7c3c7855f3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-51119c-7988b955f5-kr9rl" Nov 03 00:56:03.323354 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:03.306727 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4357adf-7a95-4137-a489-ba7c3c7855f3-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-51119c-7988b955f5-kr9rl\" (UID: \"a4357adf-7a95-4137-a489-ba7c3c7855f3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-51119c-7988b955f5-kr9rl" Nov 03 00:56:03.366301 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:03.338306 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pckr\" (UniqueName: \"kubernetes.io/projected/a4357adf-7a95-4137-a489-ba7c3c7855f3-kube-api-access-9pckr\") pod \"payload-pre-processing-e2e-api-a-51119c-7988b955f5-kr9rl\" (UID: \"a4357adf-7a95-4137-a489-ba7c3c7855f3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-51119c-7988b955f5-kr9rl" Nov 03 00:56:03.366301 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:03.338690 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28bda42f-8493-4e36-923e-8f60a48f4ed7-plugins-config-volume\") pod \"payload-processing-e2e-api-a-51119c-546967fcc9-mtcnh\" (UID: \"28bda42f-8493-4e36-923e-8f60a48f4ed7\") " pod="openshift-ingress/payload-processing-e2e-api-a-51119c-546967fcc9-mtcnh" Nov 03 00:56:03.366301 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:03.338729 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkxs2\" (UniqueName: \"kubernetes.io/projected/28bda42f-8493-4e36-923e-8f60a48f4ed7-kube-api-access-pkxs2\") pod \"payload-processing-e2e-api-a-51119c-546967fcc9-mtcnh\" (UID: \"28bda42f-8493-4e36-923e-8f60a48f4ed7\") " pod="openshift-ingress/payload-processing-e2e-api-a-51119c-546967fcc9-mtcnh" Nov 03 00:56:03.366301 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:03.341599 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4357adf-7a95-4137-a489-ba7c3c7855f3-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-51119c-7988b955f5-kr9rl\" (UID: \"a4357adf-7a95-4137-a489-ba7c3c7855f3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-51119c-7988b955f5-kr9rl" Nov 03 00:56:03.471305 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:03.440637 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28bda42f-8493-4e36-923e-8f60a48f4ed7-plugins-config-volume\") pod \"payload-processing-e2e-api-a-51119c-546967fcc9-mtcnh\" (UID: \"28bda42f-8493-4e36-923e-8f60a48f4ed7\") " pod="openshift-ingress/payload-processing-e2e-api-a-51119c-546967fcc9-mtcnh" Nov 03 00:56:03.471305 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:03.440684 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkxs2\" (UniqueName: \"kubernetes.io/projected/28bda42f-8493-4e36-923e-8f60a48f4ed7-kube-api-access-pkxs2\") pod \"payload-processing-e2e-api-a-51119c-546967fcc9-mtcnh\" (UID: \"28bda42f-8493-4e36-923e-8f60a48f4ed7\") " pod="openshift-ingress/payload-processing-e2e-api-a-51119c-546967fcc9-mtcnh" Nov 03 00:56:03.471305 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:03.446232 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28bda42f-8493-4e36-923e-8f60a48f4ed7-plugins-config-volume\") pod \"payload-processing-e2e-api-a-51119c-546967fcc9-mtcnh\" (UID: \"28bda42f-8493-4e36-923e-8f60a48f4ed7\") " pod="openshift-ingress/payload-processing-e2e-api-a-51119c-546967fcc9-mtcnh" Nov 03 00:56:03.588375 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:03.585482 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkxs2\" (UniqueName: \"kubernetes.io/projected/28bda42f-8493-4e36-923e-8f60a48f4ed7-kube-api-access-pkxs2\") pod \"payload-processing-e2e-api-a-51119c-546967fcc9-mtcnh\" (UID: \"28bda42f-8493-4e36-923e-8f60a48f4ed7\") " pod="openshift-ingress/payload-processing-e2e-api-a-51119c-546967fcc9-mtcnh" Nov 03 00:56:03.596219 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:03.594949 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pckr\" (UniqueName: \"kubernetes.io/projected/a4357adf-7a95-4137-a489-ba7c3c7855f3-kube-api-access-9pckr\") pod \"payload-pre-processing-e2e-api-a-51119c-7988b955f5-kr9rl\" (UID: \"a4357adf-7a95-4137-a489-ba7c3c7855f3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-51119c-7988b955f5-kr9rl" Nov 03 00:56:03.694645 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:03.694606 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-51119c-7988b955f5-kr9rl" Nov 03 00:56:03.841857 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:03.835173 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-cmdm8" Nov 03 00:56:03.841857 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:03.836639 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-cmdm8" Nov 03 00:56:03.883223 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:03.882807 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-51119c-546967fcc9-mtcnh" Nov 03 00:56:03.958221 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:03.958176 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-cmdm8" Nov 03 00:56:04.045360 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:04.044601 4421 patch_prober.go:29] interesting pod/e2e-mt-dd03e395-openshift-default-75799f6fc-kx54z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body= Nov 03 00:56:04.045360 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:04.045159 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-dd03e395-openshift-default-75799f6fc-kx54z" podUID="01442ff5-1dab-4ed3-991b-ec0732c71305" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" Nov 03 00:56:04.051618 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:56:04.051572 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4357adf_7a95_4137_a489_ba7c3c7855f3.slice/crio-82272e430f9d4656e6060d64643ff60dc73860e8b61c68615e289e07d93ba62a WatchSource:0}: Error finding container 82272e430f9d4656e6060d64643ff60dc73860e8b61c68615e289e07d93ba62a: Status 404 returned error can't find the container with id 82272e430f9d4656e6060d64643ff60dc73860e8b61c68615e289e07d93ba62a Nov 03 00:56:04.064905 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:04.064496 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-51119c-7988b955f5-kr9rl"] Nov 03 00:56:04.279950 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:04.279909 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-51119c-546967fcc9-mtcnh"] Nov 03 00:56:04.639504 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:04.639397 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-51119c-7988b955f5-kr9rl" event={"ID":"a4357adf-7a95-4137-a489-ba7c3c7855f3","Type":"ContainerStarted","Data":"a047a9ed47ae77394fc05180ed12cf3eafe7fb67fc8999d4fd2015eacf107ab5"} Nov 03 00:56:04.639504 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:04.639470 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-51119c-7988b955f5-kr9rl" event={"ID":"a4357adf-7a95-4137-a489-ba7c3c7855f3","Type":"ContainerStarted","Data":"82272e430f9d4656e6060d64643ff60dc73860e8b61c68615e289e07d93ba62a"} Nov 03 00:56:04.639504 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:04.639504 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-51119c-7988b955f5-kr9rl" Nov 03 00:56:04.641526 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:04.641498 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-51119c-546967fcc9-mtcnh" event={"ID":"28bda42f-8493-4e36-923e-8f60a48f4ed7","Type":"ContainerStarted","Data":"55ee7644158d41c3b365c14b1cfe2ed91390faa34345af2b237f767550cc0616"} Nov 03 00:56:04.641672 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:04.641555 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-51119c-546967fcc9-mtcnh" event={"ID":"28bda42f-8493-4e36-923e-8f60a48f4ed7","Type":"ContainerStarted","Data":"437e056fd2a62e2357681d45ab03289178efd382f0b98baf77b52c15c7428952"} Nov 03 00:56:04.641799 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:04.641779 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-51119c-546967fcc9-mtcnh" Nov 03 00:56:04.657108 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:04.657059 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-51119c-7988b955f5-kr9rl" podStartSLOduration=1.6570460040000001 podStartE2EDuration="1.657046004s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:56:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:56:04.656348315 +0000 UTC m=+906.827351412" watchObservedRunningTime="2026-11-03 00:56:04.657046004 +0000 UTC m=+906.828049130" Nov 03 00:56:04.676030 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:04.675987 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-51119c-546967fcc9-mtcnh" podStartSLOduration=1.675975211 podStartE2EDuration="1.675975211s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:56:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:56:04.671987116 +0000 UTC m=+906.842990210" watchObservedRunningTime="2026-11-03 00:56:04.675975211 +0000 UTC m=+906.846978314" Nov 03 00:56:05.051234 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:05.051179 4421 patch_prober.go:29] interesting pod/e2e-mt-dd03e395-openshift-default-75799f6fc-kx54z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body= Nov 03 00:56:05.051590 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:05.051548 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-dd03e395-openshift-default-75799f6fc-kx54z" podUID="01442ff5-1dab-4ed3-991b-ec0732c71305" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" Nov 03 00:56:06.044741 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:06.044698 4421 patch_prober.go:29] interesting pod/e2e-mt-dd03e395-openshift-default-75799f6fc-kx54z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body= Nov 03 00:56:06.045182 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:06.044768 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-dd03e395-openshift-default-75799f6fc-kx54z" podUID="01442ff5-1dab-4ed3-991b-ec0732c71305" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" Nov 03 00:56:07.044894 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:07.044859 4421 patch_prober.go:29] interesting pod/e2e-mt-dd03e395-openshift-default-75799f6fc-kx54z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body= Nov 03 00:56:07.045402 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:07.044914 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-dd03e395-openshift-default-75799f6fc-kx54z" podUID="01442ff5-1dab-4ed3-991b-ec0732c71305" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" Nov 03 00:56:08.049573 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:08.049536 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-dd03e395-openshift-default-75799f6fc-kx54z" Nov 03 00:56:08.050041 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:08.050018 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-dd03e395-openshift-default-75799f6fc-kx54z" Nov 03 00:56:08.050820 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:08.050795 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-dd03e395-openshift-default-75799f6fc-kx54z" Nov 03 00:56:08.769114 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:08.769073 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-zrdp5"] Nov 03 00:56:08.788247 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:08.778975 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-zrdp5" Nov 03 00:56:08.788247 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:08.786649 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-zrdp5"] Nov 03 00:56:08.900176 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:08.900126 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b6779b2-2b64-40e9-821c-5444681cea09-workload-certs\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-zrdp5\" (UID: \"7b6779b2-2b64-40e9-821c-5444681cea09\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-zrdp5" Nov 03 00:56:08.900176 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:08.900182 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b6779b2-2b64-40e9-821c-5444681cea09-istiod-ca-cert\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-zrdp5\" (UID: \"7b6779b2-2b64-40e9-821c-5444681cea09\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-zrdp5" Nov 03 00:56:08.900440 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:08.900351 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b6779b2-2b64-40e9-821c-5444681cea09-istio-podinfo\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-zrdp5\" (UID: \"7b6779b2-2b64-40e9-821c-5444681cea09\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-zrdp5" Nov 03 00:56:08.900481 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:08.900431 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b6779b2-2b64-40e9-821c-5444681cea09-credential-socket\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-zrdp5\" (UID: \"7b6779b2-2b64-40e9-821c-5444681cea09\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-zrdp5" Nov 03 00:56:08.900481 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:08.900474 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b6779b2-2b64-40e9-821c-5444681cea09-istio-data\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-zrdp5\" (UID: \"7b6779b2-2b64-40e9-821c-5444681cea09\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-zrdp5" Nov 03 00:56:08.900552 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:08.900516 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b6779b2-2b64-40e9-821c-5444681cea09-workload-socket\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-zrdp5\" (UID: \"7b6779b2-2b64-40e9-821c-5444681cea09\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-zrdp5" Nov 03 00:56:08.900609 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:08.900593 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qtns\" (UniqueName: \"kubernetes.io/projected/7b6779b2-2b64-40e9-821c-5444681cea09-kube-api-access-5qtns\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-zrdp5\" (UID: \"7b6779b2-2b64-40e9-821c-5444681cea09\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-zrdp5" Nov 03 00:56:08.900740 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:08.900727 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b6779b2-2b64-40e9-821c-5444681cea09-istio-token\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-zrdp5\" (UID: \"7b6779b2-2b64-40e9-821c-5444681cea09\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-zrdp5" Nov 03 00:56:08.900824 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:08.900813 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b6779b2-2b64-40e9-821c-5444681cea09-istio-envoy\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-zrdp5\" (UID: \"7b6779b2-2b64-40e9-821c-5444681cea09\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-zrdp5" Nov 03 00:56:09.001979 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:09.001931 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b6779b2-2b64-40e9-821c-5444681cea09-istio-token\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-zrdp5\" (UID: \"7b6779b2-2b64-40e9-821c-5444681cea09\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-zrdp5" Nov 03 00:56:09.002182 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:09.002032 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b6779b2-2b64-40e9-821c-5444681cea09-istio-envoy\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-zrdp5\" (UID: \"7b6779b2-2b64-40e9-821c-5444681cea09\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-zrdp5" Nov 03 00:56:09.002182 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:09.002072 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b6779b2-2b64-40e9-821c-5444681cea09-workload-certs\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-zrdp5\" (UID: \"7b6779b2-2b64-40e9-821c-5444681cea09\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-zrdp5" Nov 03 00:56:09.002182 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:09.002092 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b6779b2-2b64-40e9-821c-5444681cea09-istiod-ca-cert\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-zrdp5\" (UID: \"7b6779b2-2b64-40e9-821c-5444681cea09\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-zrdp5" Nov 03 00:56:09.002182 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:09.002120 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b6779b2-2b64-40e9-821c-5444681cea09-istio-podinfo\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-zrdp5\" (UID: \"7b6779b2-2b64-40e9-821c-5444681cea09\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-zrdp5" Nov 03 00:56:09.002182 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:09.002148 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b6779b2-2b64-40e9-821c-5444681cea09-credential-socket\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-zrdp5\" (UID: \"7b6779b2-2b64-40e9-821c-5444681cea09\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-zrdp5" Nov 03 00:56:09.002182 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:09.002177 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b6779b2-2b64-40e9-821c-5444681cea09-istio-data\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-zrdp5\" (UID: \"7b6779b2-2b64-40e9-821c-5444681cea09\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-zrdp5" Nov 03 00:56:09.002545 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:09.002217 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b6779b2-2b64-40e9-821c-5444681cea09-workload-socket\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-zrdp5\" (UID: \"7b6779b2-2b64-40e9-821c-5444681cea09\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-zrdp5" Nov 03 00:56:09.002545 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:09.002263 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qtns\" (UniqueName: \"kubernetes.io/projected/7b6779b2-2b64-40e9-821c-5444681cea09-kube-api-access-5qtns\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-zrdp5\" (UID: \"7b6779b2-2b64-40e9-821c-5444681cea09\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-zrdp5" Nov 03 00:56:09.002691 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:09.002662 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b6779b2-2b64-40e9-821c-5444681cea09-credential-socket\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-zrdp5\" (UID: \"7b6779b2-2b64-40e9-821c-5444681cea09\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-zrdp5" Nov 03 00:56:09.002751 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:09.002723 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b6779b2-2b64-40e9-821c-5444681cea09-workload-socket\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-zrdp5\" (UID: \"7b6779b2-2b64-40e9-821c-5444681cea09\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-zrdp5" Nov 03 00:56:09.002978 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:09.002950 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b6779b2-2b64-40e9-821c-5444681cea09-workload-certs\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-zrdp5\" (UID: \"7b6779b2-2b64-40e9-821c-5444681cea09\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-zrdp5" Nov 03 00:56:09.003138 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:09.002974 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b6779b2-2b64-40e9-821c-5444681cea09-istiod-ca-cert\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-zrdp5\" (UID: \"7b6779b2-2b64-40e9-821c-5444681cea09\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-zrdp5" Nov 03 00:56:09.003138 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:09.002990 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b6779b2-2b64-40e9-821c-5444681cea09-istio-data\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-zrdp5\" (UID: \"7b6779b2-2b64-40e9-821c-5444681cea09\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-zrdp5" Nov 03 00:56:09.004774 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:09.004740 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b6779b2-2b64-40e9-821c-5444681cea09-istio-envoy\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-zrdp5\" (UID: \"7b6779b2-2b64-40e9-821c-5444681cea09\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-zrdp5" Nov 03 00:56:09.005085 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:09.005058 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b6779b2-2b64-40e9-821c-5444681cea09-istio-podinfo\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-zrdp5\" (UID: \"7b6779b2-2b64-40e9-821c-5444681cea09\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-zrdp5" Nov 03 00:56:09.009645 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:09.009626 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qtns\" (UniqueName: \"kubernetes.io/projected/7b6779b2-2b64-40e9-821c-5444681cea09-kube-api-access-5qtns\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-zrdp5\" (UID: \"7b6779b2-2b64-40e9-821c-5444681cea09\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-zrdp5" Nov 03 00:56:09.009735 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:09.009714 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b6779b2-2b64-40e9-821c-5444681cea09-istio-token\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-zrdp5\" (UID: \"7b6779b2-2b64-40e9-821c-5444681cea09\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-zrdp5" Nov 03 00:56:09.114654 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:09.114569 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-zrdp5" Nov 03 00:56:09.424076 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:56:09.424041 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b6779b2_2b64_40e9_821c_5444681cea09.slice/crio-23e5d69af785381e97a8ee5bfffd6b53f000f908bc7858cb3ea4cffb3c605c4f WatchSource:0}: Error finding container 23e5d69af785381e97a8ee5bfffd6b53f000f908bc7858cb3ea4cffb3c605c4f: Status 404 returned error can't find the container with id 23e5d69af785381e97a8ee5bfffd6b53f000f908bc7858cb3ea4cffb3c605c4f Nov 03 00:56:09.424722 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:09.424630 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-zrdp5"] Nov 03 00:56:09.426711 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:09.426676 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:56:09.426819 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:09.426757 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:56:09.426819 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:09.426803 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:56:09.684172 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:09.684073 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-zrdp5" event={"ID":"7b6779b2-2b64-40e9-821c-5444681cea09","Type":"ContainerStarted","Data":"817dc1736b03a03d7b7981117dcc9a84c8fbe8722105c1052ea00962ca3ee5e1"} Nov 03 00:56:09.684172 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:09.684146 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-zrdp5" event={"ID":"7b6779b2-2b64-40e9-821c-5444681cea09","Type":"ContainerStarted","Data":"23e5d69af785381e97a8ee5bfffd6b53f000f908bc7858cb3ea4cffb3c605c4f"} Nov 03 00:56:09.711085 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:09.711016 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-zrdp5" podStartSLOduration=1.7110010469999999 podStartE2EDuration="1.711001047s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:56:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:56:09.707639106 +0000 UTC m=+911.878642236" watchObservedRunningTime="2026-11-03 00:56:09.711001047 +0000 UTC m=+911.882004143" Nov 03 00:56:10.125303 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:10.121449 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-zrdp5" Nov 03 00:56:10.125303 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:10.122080 4421 patch_prober.go:29] interesting pod/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-zrdp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Nov 03 00:56:10.125303 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:10.122132 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-zrdp5" podUID="7b6779b2-2b64-40e9-821c-5444681cea09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Nov 03 00:56:11.125309 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:11.124851 4421 patch_prober.go:29] interesting pod/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-zrdp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Nov 03 00:56:11.125309 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:11.124928 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-zrdp5" podUID="7b6779b2-2b64-40e9-821c-5444681cea09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Nov 03 00:56:12.120301 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:12.114978 4421 patch_prober.go:29] interesting pod/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-zrdp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Nov 03 00:56:12.120301 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:12.115058 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-zrdp5" podUID="7b6779b2-2b64-40e9-821c-5444681cea09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Nov 03 00:56:13.122132 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:13.121702 4421 patch_prober.go:29] interesting pod/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-zrdp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Nov 03 00:56:13.122132 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:13.121764 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-zrdp5" podUID="7b6779b2-2b64-40e9-821c-5444681cea09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Nov 03 00:56:14.131527 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:14.131481 4421 patch_prober.go:29] interesting pod/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-zrdp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Nov 03 00:56:14.132038 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:14.131560 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-zrdp5" podUID="7b6779b2-2b64-40e9-821c-5444681cea09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Nov 03 00:56:14.695076 ip-10-0-133-217 kubenswrapper[4421]: E1103 00:56:14.695027 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 00:56:15.124301 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:15.115685 4421 patch_prober.go:29] interesting pod/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-zrdp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Nov 03 00:56:15.124301 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:15.115789 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-zrdp5" podUID="7b6779b2-2b64-40e9-821c-5444681cea09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Nov 03 00:56:15.652036 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:15.652007 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-51119c-546967fcc9-mtcnh" Nov 03 00:56:15.652583 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:15.652080 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-51119c-7988b955f5-kr9rl" Nov 03 00:56:16.119766 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:16.119694 4421 patch_prober.go:29] interesting pod/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-zrdp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Nov 03 00:56:16.119766 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:16.119761 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-zrdp5" podUID="7b6779b2-2b64-40e9-821c-5444681cea09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Nov 03 00:56:17.392350 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:17.392320 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-zrdp5" Nov 03 00:56:17.392863 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:17.392841 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-zrdp5" Nov 03 00:56:17.393839 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:17.393815 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-zrdp5" Nov 03 00:56:22.805215 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:22.805169 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-65e5ec-openshift-default-84ff8668f-kjxr2"] Nov 03 00:56:22.810357 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:22.810333 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-65e5ec-openshift-default-84ff8668f-kjxr2" Nov 03 00:56:22.816197 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:22.816167 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-65e5ec-openshift-default-dockercfg-8nv8r\"" Nov 03 00:56:22.822200 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:22.822168 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-65e5ec-openshift-default-84ff8668f-kjxr2"] Nov 03 00:56:22.903952 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:22.901346 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97f665ff-bb78-4685-b51b-5c791e529d17-credential-socket\") pod \"e2e-api-b-65e5ec-openshift-default-84ff8668f-kjxr2\" (UID: \"97f665ff-bb78-4685-b51b-5c791e529d17\") " pod="openshift-ingress/e2e-api-b-65e5ec-openshift-default-84ff8668f-kjxr2" Nov 03 00:56:22.903952 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:22.901494 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97f665ff-bb78-4685-b51b-5c791e529d17-istio-envoy\") pod \"e2e-api-b-65e5ec-openshift-default-84ff8668f-kjxr2\" (UID: \"97f665ff-bb78-4685-b51b-5c791e529d17\") " pod="openshift-ingress/e2e-api-b-65e5ec-openshift-default-84ff8668f-kjxr2" Nov 03 00:56:22.903952 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:22.901609 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sd6dk\" (UniqueName: \"kubernetes.io/projected/97f665ff-bb78-4685-b51b-5c791e529d17-kube-api-access-sd6dk\") pod \"e2e-api-b-65e5ec-openshift-default-84ff8668f-kjxr2\" (UID: \"97f665ff-bb78-4685-b51b-5c791e529d17\") " pod="openshift-ingress/e2e-api-b-65e5ec-openshift-default-84ff8668f-kjxr2" Nov 03 00:56:22.903952 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:22.901869 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97f665ff-bb78-4685-b51b-5c791e529d17-workload-certs\") pod \"e2e-api-b-65e5ec-openshift-default-84ff8668f-kjxr2\" (UID: \"97f665ff-bb78-4685-b51b-5c791e529d17\") " pod="openshift-ingress/e2e-api-b-65e5ec-openshift-default-84ff8668f-kjxr2" Nov 03 00:56:22.903952 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:22.901969 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97f665ff-bb78-4685-b51b-5c791e529d17-istio-podinfo\") pod \"e2e-api-b-65e5ec-openshift-default-84ff8668f-kjxr2\" (UID: \"97f665ff-bb78-4685-b51b-5c791e529d17\") " pod="openshift-ingress/e2e-api-b-65e5ec-openshift-default-84ff8668f-kjxr2" Nov 03 00:56:22.903952 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:22.902047 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97f665ff-bb78-4685-b51b-5c791e529d17-istio-token\") pod \"e2e-api-b-65e5ec-openshift-default-84ff8668f-kjxr2\" (UID: \"97f665ff-bb78-4685-b51b-5c791e529d17\") " pod="openshift-ingress/e2e-api-b-65e5ec-openshift-default-84ff8668f-kjxr2" Nov 03 00:56:22.903952 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:22.902133 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97f665ff-bb78-4685-b51b-5c791e529d17-workload-socket\") pod \"e2e-api-b-65e5ec-openshift-default-84ff8668f-kjxr2\" (UID: \"97f665ff-bb78-4685-b51b-5c791e529d17\") " pod="openshift-ingress/e2e-api-b-65e5ec-openshift-default-84ff8668f-kjxr2" Nov 03 00:56:22.903952 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:22.902164 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97f665ff-bb78-4685-b51b-5c791e529d17-istiod-ca-cert\") pod \"e2e-api-b-65e5ec-openshift-default-84ff8668f-kjxr2\" (UID: \"97f665ff-bb78-4685-b51b-5c791e529d17\") " pod="openshift-ingress/e2e-api-b-65e5ec-openshift-default-84ff8668f-kjxr2" Nov 03 00:56:22.903952 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:22.902236 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97f665ff-bb78-4685-b51b-5c791e529d17-istio-data\") pod \"e2e-api-b-65e5ec-openshift-default-84ff8668f-kjxr2\" (UID: \"97f665ff-bb78-4685-b51b-5c791e529d17\") " pod="openshift-ingress/e2e-api-b-65e5ec-openshift-default-84ff8668f-kjxr2" Nov 03 00:56:23.012293 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:23.007718 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97f665ff-bb78-4685-b51b-5c791e529d17-istio-data\") pod \"e2e-api-b-65e5ec-openshift-default-84ff8668f-kjxr2\" (UID: \"97f665ff-bb78-4685-b51b-5c791e529d17\") " pod="openshift-ingress/e2e-api-b-65e5ec-openshift-default-84ff8668f-kjxr2" Nov 03 00:56:23.012293 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:23.007797 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97f665ff-bb78-4685-b51b-5c791e529d17-credential-socket\") pod \"e2e-api-b-65e5ec-openshift-default-84ff8668f-kjxr2\" (UID: \"97f665ff-bb78-4685-b51b-5c791e529d17\") " pod="openshift-ingress/e2e-api-b-65e5ec-openshift-default-84ff8668f-kjxr2" Nov 03 00:56:23.012293 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:23.007858 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97f665ff-bb78-4685-b51b-5c791e529d17-istio-envoy\") pod \"e2e-api-b-65e5ec-openshift-default-84ff8668f-kjxr2\" (UID: \"97f665ff-bb78-4685-b51b-5c791e529d17\") " pod="openshift-ingress/e2e-api-b-65e5ec-openshift-default-84ff8668f-kjxr2" Nov 03 00:56:23.012293 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:23.007920 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sd6dk\" (UniqueName: \"kubernetes.io/projected/97f665ff-bb78-4685-b51b-5c791e529d17-kube-api-access-sd6dk\") pod \"e2e-api-b-65e5ec-openshift-default-84ff8668f-kjxr2\" (UID: \"97f665ff-bb78-4685-b51b-5c791e529d17\") " pod="openshift-ingress/e2e-api-b-65e5ec-openshift-default-84ff8668f-kjxr2" Nov 03 00:56:23.012293 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:23.008217 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97f665ff-bb78-4685-b51b-5c791e529d17-workload-certs\") pod \"e2e-api-b-65e5ec-openshift-default-84ff8668f-kjxr2\" (UID: \"97f665ff-bb78-4685-b51b-5c791e529d17\") " pod="openshift-ingress/e2e-api-b-65e5ec-openshift-default-84ff8668f-kjxr2" Nov 03 00:56:23.012293 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:23.008297 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97f665ff-bb78-4685-b51b-5c791e529d17-istio-podinfo\") pod \"e2e-api-b-65e5ec-openshift-default-84ff8668f-kjxr2\" (UID: \"97f665ff-bb78-4685-b51b-5c791e529d17\") " pod="openshift-ingress/e2e-api-b-65e5ec-openshift-default-84ff8668f-kjxr2" Nov 03 00:56:23.012293 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:23.008341 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97f665ff-bb78-4685-b51b-5c791e529d17-istio-token\") pod \"e2e-api-b-65e5ec-openshift-default-84ff8668f-kjxr2\" (UID: \"97f665ff-bb78-4685-b51b-5c791e529d17\") " pod="openshift-ingress/e2e-api-b-65e5ec-openshift-default-84ff8668f-kjxr2" Nov 03 00:56:23.012293 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:23.008395 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97f665ff-bb78-4685-b51b-5c791e529d17-workload-socket\") pod \"e2e-api-b-65e5ec-openshift-default-84ff8668f-kjxr2\" (UID: \"97f665ff-bb78-4685-b51b-5c791e529d17\") " pod="openshift-ingress/e2e-api-b-65e5ec-openshift-default-84ff8668f-kjxr2" Nov 03 00:56:23.012293 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:23.008417 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97f665ff-bb78-4685-b51b-5c791e529d17-istiod-ca-cert\") pod \"e2e-api-b-65e5ec-openshift-default-84ff8668f-kjxr2\" (UID: \"97f665ff-bb78-4685-b51b-5c791e529d17\") " pod="openshift-ingress/e2e-api-b-65e5ec-openshift-default-84ff8668f-kjxr2" Nov 03 00:56:23.012293 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:23.009339 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97f665ff-bb78-4685-b51b-5c791e529d17-istiod-ca-cert\") pod \"e2e-api-b-65e5ec-openshift-default-84ff8668f-kjxr2\" (UID: \"97f665ff-bb78-4685-b51b-5c791e529d17\") " pod="openshift-ingress/e2e-api-b-65e5ec-openshift-default-84ff8668f-kjxr2" Nov 03 00:56:23.012293 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:23.009686 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97f665ff-bb78-4685-b51b-5c791e529d17-istio-data\") pod \"e2e-api-b-65e5ec-openshift-default-84ff8668f-kjxr2\" (UID: \"97f665ff-bb78-4685-b51b-5c791e529d17\") " pod="openshift-ingress/e2e-api-b-65e5ec-openshift-default-84ff8668f-kjxr2" Nov 03 00:56:23.012293 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:23.009921 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97f665ff-bb78-4685-b51b-5c791e529d17-credential-socket\") pod \"e2e-api-b-65e5ec-openshift-default-84ff8668f-kjxr2\" (UID: \"97f665ff-bb78-4685-b51b-5c791e529d17\") " pod="openshift-ingress/e2e-api-b-65e5ec-openshift-default-84ff8668f-kjxr2" Nov 03 00:56:23.020173 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:23.016243 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97f665ff-bb78-4685-b51b-5c791e529d17-istio-envoy\") pod \"e2e-api-b-65e5ec-openshift-default-84ff8668f-kjxr2\" (UID: \"97f665ff-bb78-4685-b51b-5c791e529d17\") " pod="openshift-ingress/e2e-api-b-65e5ec-openshift-default-84ff8668f-kjxr2" Nov 03 00:56:23.020173 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:23.016889 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97f665ff-bb78-4685-b51b-5c791e529d17-workload-certs\") pod \"e2e-api-b-65e5ec-openshift-default-84ff8668f-kjxr2\" (UID: \"97f665ff-bb78-4685-b51b-5c791e529d17\") " pod="openshift-ingress/e2e-api-b-65e5ec-openshift-default-84ff8668f-kjxr2" Nov 03 00:56:23.062004 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:23.061686 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sd6dk\" (UniqueName: \"kubernetes.io/projected/97f665ff-bb78-4685-b51b-5c791e529d17-kube-api-access-sd6dk\") pod \"e2e-api-b-65e5ec-openshift-default-84ff8668f-kjxr2\" (UID: \"97f665ff-bb78-4685-b51b-5c791e529d17\") " pod="openshift-ingress/e2e-api-b-65e5ec-openshift-default-84ff8668f-kjxr2" Nov 03 00:56:23.092305 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:23.091253 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97f665ff-bb78-4685-b51b-5c791e529d17-workload-socket\") pod \"e2e-api-b-65e5ec-openshift-default-84ff8668f-kjxr2\" (UID: \"97f665ff-bb78-4685-b51b-5c791e529d17\") " pod="openshift-ingress/e2e-api-b-65e5ec-openshift-default-84ff8668f-kjxr2" Nov 03 00:56:23.121136 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:23.119111 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97f665ff-bb78-4685-b51b-5c791e529d17-istio-podinfo\") pod \"e2e-api-b-65e5ec-openshift-default-84ff8668f-kjxr2\" (UID: \"97f665ff-bb78-4685-b51b-5c791e529d17\") " pod="openshift-ingress/e2e-api-b-65e5ec-openshift-default-84ff8668f-kjxr2" Nov 03 00:56:23.127784 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:23.127755 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97f665ff-bb78-4685-b51b-5c791e529d17-istio-token\") pod \"e2e-api-b-65e5ec-openshift-default-84ff8668f-kjxr2\" (UID: \"97f665ff-bb78-4685-b51b-5c791e529d17\") " pod="openshift-ingress/e2e-api-b-65e5ec-openshift-default-84ff8668f-kjxr2" Nov 03 00:56:23.132213 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:23.132187 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-65e5ec-openshift-default-84ff8668f-kjxr2" Nov 03 00:56:23.427415 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:23.427389 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-65e5ec-openshift-default-84ff8668f-kjxr2"] Nov 03 00:56:23.427500 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:56:23.427425 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97f665ff_bb78_4685_b51b_5c791e529d17.slice/crio-6391493df0de83724c7865a44bc9205a91d7fcafe85adda38ba7273eab6f554d WatchSource:0}: Error finding container 6391493df0de83724c7865a44bc9205a91d7fcafe85adda38ba7273eab6f554d: Status 404 returned error can't find the container with id 6391493df0de83724c7865a44bc9205a91d7fcafe85adda38ba7273eab6f554d Nov 03 00:56:23.429468 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:23.429432 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:56:23.429588 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:23.429500 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:56:23.429588 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:23.429529 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:56:23.909244 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:23.909207 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-65e5ec-openshift-default-84ff8668f-kjxr2" event={"ID":"97f665ff-bb78-4685-b51b-5c791e529d17","Type":"ContainerStarted","Data":"3efe8d913c3741324b7859cc51bca738bf17789bc1505fc9b1b4973b7fdb1c3d"} Nov 03 00:56:23.909686 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:23.909258 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-65e5ec-openshift-default-84ff8668f-kjxr2" event={"ID":"97f665ff-bb78-4685-b51b-5c791e529d17","Type":"ContainerStarted","Data":"6391493df0de83724c7865a44bc9205a91d7fcafe85adda38ba7273eab6f554d"} Nov 03 00:56:23.931592 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:23.931534 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-65e5ec-openshift-default-84ff8668f-kjxr2" podStartSLOduration=1.931517516 podStartE2EDuration="1.931517516s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:56:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:56:23.929040861 +0000 UTC m=+926.100043960" watchObservedRunningTime="2026-11-03 00:56:23.931517516 +0000 UTC m=+926.102520613" Nov 03 00:56:24.135922 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:24.135664 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-65e5ec-openshift-default-84ff8668f-kjxr2" Nov 03 00:56:24.158412 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:24.158376 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-65e5ec-openshift-default-84ff8668f-kjxr2" Nov 03 00:56:24.917167 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:24.917137 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-65e5ec-openshift-default-84ff8668f-kjxr2" Nov 03 00:56:24.918250 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:24.918226 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-65e5ec-openshift-default-84ff8668f-kjxr2" Nov 03 00:56:44.126867 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:44.126836 4421 generic.go:356] "Generic (PLEG): container finished" podID="934ddd77-9c41-49d5-bf51-fb5d8e765efd" containerID="c5363a1ef8d1f7d5e5414208fac3058904333d527167c17f09e4035434ddc459" exitCode=0 Nov 03 00:56:44.127364 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:44.126918 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-51119c-openshift-default-56b69d85d9-ds9fq" event={"ID":"934ddd77-9c41-49d5-bf51-fb5d8e765efd","Type":"ContainerDied","Data":"c5363a1ef8d1f7d5e5414208fac3058904333d527167c17f09e4035434ddc459"} Nov 03 00:56:44.127429 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:44.127407 4421 scope.go:122] "RemoveContainer" containerID="c5363a1ef8d1f7d5e5414208fac3058904333d527167c17f09e4035434ddc459" Nov 03 00:56:44.129038 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:44.129010 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:56:44.129097 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:44.129080 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:56:44.129140 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:44.129115 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:56:44.675153 ip-10-0-133-217 kubenswrapper[4421]: E1103 00:56:44.675111 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 00:56:45.199296 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:45.199240 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-51119c-openshift-default-56b69d85d9-ds9fq" event={"ID":"934ddd77-9c41-49d5-bf51-fb5d8e765efd","Type":"ContainerStarted","Data":"34a281ee2bb427fd92e190df6eaff85b9c78d9ffe446167d534e9d841f3fa295"} Nov 03 00:56:45.397456 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:45.395000 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-51119c-openshift-default-56b69d85d9-ds9fq" Nov 03 00:56:45.397456 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:45.395245 4421 patch_prober.go:29] interesting pod/e2e-api-a-51119c-openshift-default-56b69d85d9-ds9fq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Nov 03 00:56:45.397456 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:45.395345 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-51119c-openshift-default-56b69d85d9-ds9fq" podUID="934ddd77-9c41-49d5-bf51-fb5d8e765efd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Nov 03 00:56:45.550539 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:45.550502 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-51119c-openshift-default-56b69d85d9-ds9fq" Nov 03 00:56:46.393464 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:46.393424 4421 patch_prober.go:29] interesting pod/e2e-api-a-51119c-openshift-default-56b69d85d9-ds9fq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Nov 03 00:56:46.393899 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:46.393497 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-51119c-openshift-default-56b69d85d9-ds9fq" podUID="934ddd77-9c41-49d5-bf51-fb5d8e765efd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Nov 03 00:56:47.393031 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:47.392988 4421 patch_prober.go:29] interesting pod/e2e-api-a-51119c-openshift-default-56b69d85d9-ds9fq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Nov 03 00:56:47.393253 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:47.393064 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-51119c-openshift-default-56b69d85d9-ds9fq" podUID="934ddd77-9c41-49d5-bf51-fb5d8e765efd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Nov 03 00:56:48.395341 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:48.393079 4421 patch_prober.go:29] interesting pod/e2e-api-a-51119c-openshift-default-56b69d85d9-ds9fq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Nov 03 00:56:48.395341 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:48.393138 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-51119c-openshift-default-56b69d85d9-ds9fq" podUID="934ddd77-9c41-49d5-bf51-fb5d8e765efd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Nov 03 00:56:49.392517 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:49.392480 4421 patch_prober.go:29] interesting pod/e2e-api-a-51119c-openshift-default-56b69d85d9-ds9fq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Nov 03 00:56:49.392707 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:49.392538 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-51119c-openshift-default-56b69d85d9-ds9fq" podUID="934ddd77-9c41-49d5-bf51-fb5d8e765efd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Nov 03 00:56:50.392678 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:50.392634 4421 patch_prober.go:29] interesting pod/e2e-api-a-51119c-openshift-default-56b69d85d9-ds9fq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Nov 03 00:56:50.393226 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:50.392696 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-51119c-openshift-default-56b69d85d9-ds9fq" podUID="934ddd77-9c41-49d5-bf51-fb5d8e765efd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Nov 03 00:56:51.398291 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:51.398248 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-51119c-openshift-default-56b69d85d9-ds9fq" Nov 03 00:56:51.399449 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:51.399424 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-51119c-openshift-default-56b69d85d9-ds9fq" Nov 03 00:56:54.025336 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:54.025302 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-51119c-546967fcc9-mtcnh"] Nov 03 00:56:54.037201 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:54.037169 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-51119c-7988b955f5-kr9rl"] Nov 03 00:56:55.347228 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:55.347192 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21e770b3-gcrlb"] Nov 03 00:56:55.351645 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:55.351625 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21e770b3-gcrlb" Nov 03 00:56:55.361143 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:55.361111 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21e770b3-gcrlb"] Nov 03 00:56:55.361414 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:55.361394 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6q4xp\"" Nov 03 00:56:55.361493 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:55.361442 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:56:55.361493 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:55.361460 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 00:56:55.361583 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:55.361527 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 00:56:55.368251 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:55.368226 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a091a850-4bf7-4272-b9ea-3fbd5b7b3df4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-21e770b3-gcrlb\" (UID: \"a091a850-4bf7-4272-b9ea-3fbd5b7b3df4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21e770b3-gcrlb" Nov 03 00:56:55.368406 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:55.368260 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a091a850-4bf7-4272-b9ea-3fbd5b7b3df4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-21e770b3-gcrlb\" (UID: \"a091a850-4bf7-4272-b9ea-3fbd5b7b3df4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21e770b3-gcrlb" Nov 03 00:56:55.368475 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:55.368428 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zb9gf\" (UniqueName: \"kubernetes.io/projected/a091a850-4bf7-4272-b9ea-3fbd5b7b3df4-kube-api-access-zb9gf\") pod \"maas-api-revoke-sub-21e770b3-gcrlb\" (UID: \"a091a850-4bf7-4272-b9ea-3fbd5b7b3df4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21e770b3-gcrlb" Nov 03 00:56:55.469991 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:55.469950 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zb9gf\" (UniqueName: \"kubernetes.io/projected/a091a850-4bf7-4272-b9ea-3fbd5b7b3df4-kube-api-access-zb9gf\") pod \"maas-api-revoke-sub-21e770b3-gcrlb\" (UID: \"a091a850-4bf7-4272-b9ea-3fbd5b7b3df4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21e770b3-gcrlb" Nov 03 00:56:55.470195 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:55.470180 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a091a850-4bf7-4272-b9ea-3fbd5b7b3df4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-21e770b3-gcrlb\" (UID: \"a091a850-4bf7-4272-b9ea-3fbd5b7b3df4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21e770b3-gcrlb" Nov 03 00:56:55.470292 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:55.470215 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a091a850-4bf7-4272-b9ea-3fbd5b7b3df4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-21e770b3-gcrlb\" (UID: \"a091a850-4bf7-4272-b9ea-3fbd5b7b3df4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21e770b3-gcrlb" Nov 03 00:56:55.470859 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:55.470835 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a091a850-4bf7-4272-b9ea-3fbd5b7b3df4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-21e770b3-gcrlb\" (UID: \"a091a850-4bf7-4272-b9ea-3fbd5b7b3df4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21e770b3-gcrlb" Nov 03 00:56:55.472679 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:55.472654 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a091a850-4bf7-4272-b9ea-3fbd5b7b3df4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-21e770b3-gcrlb\" (UID: \"a091a850-4bf7-4272-b9ea-3fbd5b7b3df4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21e770b3-gcrlb" Nov 03 00:56:55.478090 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:55.478068 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zb9gf\" (UniqueName: \"kubernetes.io/projected/a091a850-4bf7-4272-b9ea-3fbd5b7b3df4-kube-api-access-zb9gf\") pod \"maas-api-revoke-sub-21e770b3-gcrlb\" (UID: \"a091a850-4bf7-4272-b9ea-3fbd5b7b3df4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21e770b3-gcrlb" Nov 03 00:56:55.668914 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:55.668820 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21e770b3-gcrlb" Nov 03 00:56:55.984502 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:56:55.984463 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda091a850_4bf7_4272_b9ea_3fbd5b7b3df4.slice/crio-861b3d23aa019471d85de917c4db53ea803561652483c942f0d8de1a50d29ebc WatchSource:0}: Error finding container 861b3d23aa019471d85de917c4db53ea803561652483c942f0d8de1a50d29ebc: Status 404 returned error can't find the container with id 861b3d23aa019471d85de917c4db53ea803561652483c942f0d8de1a50d29ebc Nov 03 00:56:55.985144 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:55.985068 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21e770b3-gcrlb"] Nov 03 00:56:56.466125 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:56.466087 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21e770b3-gcrlb" event={"ID":"a091a850-4bf7-4272-b9ea-3fbd5b7b3df4","Type":"ContainerStarted","Data":"560b1cdc0b3ba6d352f3d7e95f5a67721eb23e89536263fb28b4b8ab5e58ebcc"} Nov 03 00:56:56.466567 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:56.466156 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21e770b3-gcrlb" event={"ID":"a091a850-4bf7-4272-b9ea-3fbd5b7b3df4","Type":"ContainerStarted","Data":"861b3d23aa019471d85de917c4db53ea803561652483c942f0d8de1a50d29ebc"} Nov 03 00:56:56.485410 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:56.485361 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21e770b3-gcrlb" podStartSLOduration=1.485346598 podStartE2EDuration="1.485346598s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:56:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:56:56.483674909 +0000 UTC m=+958.654678006" watchObservedRunningTime="2026-11-03 00:56:56.485346598 +0000 UTC m=+958.656349694" Nov 03 00:56:57.476857 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:57.476821 4421 generic.go:356] "Generic (PLEG): container finished" podID="a091a850-4bf7-4272-b9ea-3fbd5b7b3df4" containerID="560b1cdc0b3ba6d352f3d7e95f5a67721eb23e89536263fb28b4b8ab5e58ebcc" exitCode=0 Nov 03 00:56:57.477284 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:57.476901 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21e770b3-gcrlb" event={"ID":"a091a850-4bf7-4272-b9ea-3fbd5b7b3df4","Type":"ContainerDied","Data":"560b1cdc0b3ba6d352f3d7e95f5a67721eb23e89536263fb28b4b8ab5e58ebcc"} Nov 03 00:56:58.341461 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:58.341427 4421 scope.go:122] "RemoveContainer" containerID="3e87362075a2eda870b3a1a565a523aa7873106ad9a7e18fa44364c7f74430db" Nov 03 00:56:58.360453 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:58.360426 4421 scope.go:122] "RemoveContainer" containerID="30c6c0c59eeb4a6bd6b8d8f3c58db77542b699fe4676ea14b34e348f0d1a0f48" Nov 03 00:56:58.380782 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:58.380750 4421 scope.go:122] "RemoveContainer" containerID="75d4f6dec2b5fe1b64a5054c87f5fdcf5b198d986b2750a14eb64fad62743b04" Nov 03 00:56:58.394830 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:58.394806 4421 scope.go:122] "RemoveContainer" containerID="6c333fb53e598c58e7a96a37e1b91630f499567d88d390375409d5ef9977878a" Nov 03 00:56:58.405521 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:58.405503 4421 scope.go:122] "RemoveContainer" containerID="f4a3c5df3590866857371e13146eac6c0f9be2404213e824abbb091840751afd" Nov 03 00:56:58.830165 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:58.830138 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21e770b3-gcrlb" Nov 03 00:56:58.943162 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:58.943079 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a091a850-4bf7-4272-b9ea-3fbd5b7b3df4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a091a850-4bf7-4272-b9ea-3fbd5b7b3df4-maas-api-xks-ca\") pod \"a091a850-4bf7-4272-b9ea-3fbd5b7b3df4\" (UID: \"a091a850-4bf7-4272-b9ea-3fbd5b7b3df4\") " Nov 03 00:56:58.943417 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:58.943242 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a091a850-4bf7-4272-b9ea-3fbd5b7b3df4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a091a850-4bf7-4272-b9ea-3fbd5b7b3df4-maas-api-openshift-ca\") pod \"a091a850-4bf7-4272-b9ea-3fbd5b7b3df4\" (UID: \"a091a850-4bf7-4272-b9ea-3fbd5b7b3df4\") " Nov 03 00:56:58.943417 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:58.943346 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a091a850-4bf7-4272-b9ea-3fbd5b7b3df4-kube-api-access-zb9gf\" (UniqueName: \"kubernetes.io/projected/a091a850-4bf7-4272-b9ea-3fbd5b7b3df4-kube-api-access-zb9gf\") pod \"a091a850-4bf7-4272-b9ea-3fbd5b7b3df4\" (UID: \"a091a850-4bf7-4272-b9ea-3fbd5b7b3df4\") " Nov 03 00:56:58.943668 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:58.943628 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a091a850-4bf7-4272-b9ea-3fbd5b7b3df4-maas-api-openshift-ca" pod "a091a850-4bf7-4272-b9ea-3fbd5b7b3df4" (UID: "a091a850-4bf7-4272-b9ea-3fbd5b7b3df4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:56:58.943876 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:58.943858 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a091a850-4bf7-4272-b9ea-3fbd5b7b3df4-maas-api-openshift-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:56:58.945147 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:58.945124 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a091a850-4bf7-4272-b9ea-3fbd5b7b3df4-maas-api-xks-ca" pod "a091a850-4bf7-4272-b9ea-3fbd5b7b3df4" (UID: "a091a850-4bf7-4272-b9ea-3fbd5b7b3df4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:56:58.945493 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:58.945476 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a091a850-4bf7-4272-b9ea-3fbd5b7b3df4-kube-api-access-zb9gf" pod "a091a850-4bf7-4272-b9ea-3fbd5b7b3df4" (UID: "a091a850-4bf7-4272-b9ea-3fbd5b7b3df4"). InnerVolumeSpecName "kube-api-access-zb9gf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:56:59.033188 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:59.032999 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-51119c-546967fcc9-mtcnh" podUID="28bda42f-8493-4e36-923e-8f60a48f4ed7" containerName="payload-processing" containerID="cri-o://55ee7644158d41c3b365c14b1cfe2ed91390faa34345af2b237f767550cc0616" gracePeriod=25 Nov 03 00:56:59.042336 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:59.039754 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-51119c-7988b955f5-kr9rl" podUID="a4357adf-7a95-4137-a489-ba7c3c7855f3" containerName="payload-pre-processing" containerID="cri-o://a047a9ed47ae77394fc05180ed12cf3eafe7fb67fc8999d4fd2015eacf107ab5" gracePeriod=25 Nov 03 00:56:59.042336 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:59.042293 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f24b753-kgtxk"] Nov 03 00:56:59.050790 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:59.046434 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zb9gf\" (UniqueName: \"kubernetes.io/projected/a091a850-4bf7-4272-b9ea-3fbd5b7b3df4-kube-api-access-zb9gf\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:56:59.050790 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:59.046465 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a091a850-4bf7-4272-b9ea-3fbd5b7b3df4-maas-api-xks-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:56:59.050790 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:59.049350 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f24b753-kgtxk"] Nov 03 00:56:59.328598 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:59.328570 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-51119c-7988b955f5-kr9rl" Nov 03 00:56:59.354640 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:59.354609 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-51119c-546967fcc9-mtcnh" Nov 03 00:56:59.451598 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:59.451569 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4357adf-7a95-4137-a489-ba7c3c7855f3-kube-api-access-9pckr\" (UniqueName: \"kubernetes.io/projected/a4357adf-7a95-4137-a489-ba7c3c7855f3-kube-api-access-9pckr\") pod \"a4357adf-7a95-4137-a489-ba7c3c7855f3\" (UID: \"a4357adf-7a95-4137-a489-ba7c3c7855f3\") " Nov 03 00:56:59.451765 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:59.451633 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4357adf-7a95-4137-a489-ba7c3c7855f3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4357adf-7a95-4137-a489-ba7c3c7855f3-plugins-config-volume\") pod \"a4357adf-7a95-4137-a489-ba7c3c7855f3\" (UID: \"a4357adf-7a95-4137-a489-ba7c3c7855f3\") " Nov 03 00:56:59.451765 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:59.451707 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28bda42f-8493-4e36-923e-8f60a48f4ed7-kube-api-access-pkxs2\" (UniqueName: \"kubernetes.io/projected/28bda42f-8493-4e36-923e-8f60a48f4ed7-kube-api-access-pkxs2\") pod \"28bda42f-8493-4e36-923e-8f60a48f4ed7\" (UID: \"28bda42f-8493-4e36-923e-8f60a48f4ed7\") " Nov 03 00:56:59.451886 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:59.451811 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28bda42f-8493-4e36-923e-8f60a48f4ed7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28bda42f-8493-4e36-923e-8f60a48f4ed7-plugins-config-volume\") pod \"28bda42f-8493-4e36-923e-8f60a48f4ed7\" (UID: \"28bda42f-8493-4e36-923e-8f60a48f4ed7\") " Nov 03 00:56:59.452143 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:59.452104 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4357adf-7a95-4137-a489-ba7c3c7855f3-plugins-config-volume" pod "a4357adf-7a95-4137-a489-ba7c3c7855f3" (UID: "a4357adf-7a95-4137-a489-ba7c3c7855f3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:56:59.452333 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:59.452249 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28bda42f-8493-4e36-923e-8f60a48f4ed7-plugins-config-volume" pod "28bda42f-8493-4e36-923e-8f60a48f4ed7" (UID: "28bda42f-8493-4e36-923e-8f60a48f4ed7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:56:59.452568 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:59.452548 4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4357adf-7a95-4137-a489-ba7c3c7855f3-plugins-config-volume\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:56:59.452668 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:59.452575 4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28bda42f-8493-4e36-923e-8f60a48f4ed7-plugins-config-volume\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:56:59.453983 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:59.453963 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28bda42f-8493-4e36-923e-8f60a48f4ed7-kube-api-access-pkxs2" pod "28bda42f-8493-4e36-923e-8f60a48f4ed7" (UID: "28bda42f-8493-4e36-923e-8f60a48f4ed7"). InnerVolumeSpecName "kube-api-access-pkxs2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:56:59.454058 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:59.454039 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4357adf-7a95-4137-a489-ba7c3c7855f3-kube-api-access-9pckr" pod "a4357adf-7a95-4137-a489-ba7c3c7855f3" (UID: "a4357adf-7a95-4137-a489-ba7c3c7855f3"). InnerVolumeSpecName "kube-api-access-9pckr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:56:59.494319 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:59.494184 4421 generic.go:356] "Generic (PLEG): container finished" podID="a4357adf-7a95-4137-a489-ba7c3c7855f3" containerID="a047a9ed47ae77394fc05180ed12cf3eafe7fb67fc8999d4fd2015eacf107ab5" exitCode=0 Nov 03 00:56:59.494319 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:59.494238 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-51119c-7988b955f5-kr9rl" Nov 03 00:56:59.494319 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:59.494269 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-51119c-7988b955f5-kr9rl" event={"ID":"a4357adf-7a95-4137-a489-ba7c3c7855f3","Type":"ContainerDied","Data":"a047a9ed47ae77394fc05180ed12cf3eafe7fb67fc8999d4fd2015eacf107ab5"} Nov 03 00:56:59.494617 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:59.494377 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-51119c-7988b955f5-kr9rl" event={"ID":"a4357adf-7a95-4137-a489-ba7c3c7855f3","Type":"ContainerDied","Data":"82272e430f9d4656e6060d64643ff60dc73860e8b61c68615e289e07d93ba62a"} Nov 03 00:56:59.494617 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:59.494405 4421 scope.go:122] "RemoveContainer" containerID="a047a9ed47ae77394fc05180ed12cf3eafe7fb67fc8999d4fd2015eacf107ab5" Nov 03 00:56:59.495983 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:59.495959 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21e770b3-gcrlb" event={"ID":"a091a850-4bf7-4272-b9ea-3fbd5b7b3df4","Type":"ContainerDied","Data":"861b3d23aa019471d85de917c4db53ea803561652483c942f0d8de1a50d29ebc"} Nov 03 00:56:59.496077 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:59.496013 4421 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="861b3d23aa019471d85de917c4db53ea803561652483c942f0d8de1a50d29ebc" Nov 03 00:56:59.496077 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:59.495971 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21e770b3-gcrlb" Nov 03 00:56:59.498363 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:59.498342 4421 generic.go:356] "Generic (PLEG): container finished" podID="28bda42f-8493-4e36-923e-8f60a48f4ed7" containerID="55ee7644158d41c3b365c14b1cfe2ed91390faa34345af2b237f767550cc0616" exitCode=0 Nov 03 00:56:59.498479 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:59.498393 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-51119c-546967fcc9-mtcnh" Nov 03 00:56:59.498541 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:59.498393 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-51119c-546967fcc9-mtcnh" event={"ID":"28bda42f-8493-4e36-923e-8f60a48f4ed7","Type":"ContainerDied","Data":"55ee7644158d41c3b365c14b1cfe2ed91390faa34345af2b237f767550cc0616"} Nov 03 00:56:59.498583 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:59.498536 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-51119c-546967fcc9-mtcnh" event={"ID":"28bda42f-8493-4e36-923e-8f60a48f4ed7","Type":"ContainerDied","Data":"437e056fd2a62e2357681d45ab03289178efd382f0b98baf77b52c15c7428952"} Nov 03 00:56:59.511605 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:59.511579 4421 scope.go:122] "RemoveContainer" containerID="a047a9ed47ae77394fc05180ed12cf3eafe7fb67fc8999d4fd2015eacf107ab5" Nov 03 00:56:59.511912 ip-10-0-133-217 kubenswrapper[4421]: E1103 00:56:59.511893 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a047a9ed47ae77394fc05180ed12cf3eafe7fb67fc8999d4fd2015eacf107ab5\": container with ID starting with a047a9ed47ae77394fc05180ed12cf3eafe7fb67fc8999d4fd2015eacf107ab5 not found: ID does not exist" containerID="a047a9ed47ae77394fc05180ed12cf3eafe7fb67fc8999d4fd2015eacf107ab5" Nov 03 00:56:59.511971 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:59.511920 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a047a9ed47ae77394fc05180ed12cf3eafe7fb67fc8999d4fd2015eacf107ab5"} err="failed to get container status \"a047a9ed47ae77394fc05180ed12cf3eafe7fb67fc8999d4fd2015eacf107ab5\": rpc error: code = NotFound desc = could not find container \"a047a9ed47ae77394fc05180ed12cf3eafe7fb67fc8999d4fd2015eacf107ab5\": container with ID starting with a047a9ed47ae77394fc05180ed12cf3eafe7fb67fc8999d4fd2015eacf107ab5 not found: ID does not exist" Nov 03 00:56:59.511971 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:59.511937 4421 scope.go:122] "RemoveContainer" containerID="55ee7644158d41c3b365c14b1cfe2ed91390faa34345af2b237f767550cc0616" Nov 03 00:56:59.523050 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:59.523030 4421 scope.go:122] "RemoveContainer" containerID="55ee7644158d41c3b365c14b1cfe2ed91390faa34345af2b237f767550cc0616" Nov 03 00:56:59.523436 ip-10-0-133-217 kubenswrapper[4421]: E1103 00:56:59.523415 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55ee7644158d41c3b365c14b1cfe2ed91390faa34345af2b237f767550cc0616\": container with ID starting with 55ee7644158d41c3b365c14b1cfe2ed91390faa34345af2b237f767550cc0616 not found: ID does not exist" containerID="55ee7644158d41c3b365c14b1cfe2ed91390faa34345af2b237f767550cc0616" Nov 03 00:56:59.523518 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:59.523445 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55ee7644158d41c3b365c14b1cfe2ed91390faa34345af2b237f767550cc0616"} err="failed to get container status \"55ee7644158d41c3b365c14b1cfe2ed91390faa34345af2b237f767550cc0616\": rpc error: code = NotFound desc = could not find container \"55ee7644158d41c3b365c14b1cfe2ed91390faa34345af2b237f767550cc0616\": container with ID starting with 55ee7644158d41c3b365c14b1cfe2ed91390faa34345af2b237f767550cc0616 not found: ID does not exist" Nov 03 00:56:59.523573 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:59.523556 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-51119c-7988b955f5-kr9rl"] Nov 03 00:56:59.532681 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:59.532659 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-51119c-7988b955f5-kr9rl"] Nov 03 00:56:59.542616 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:59.542580 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-51119c-546967fcc9-mtcnh"] Nov 03 00:56:59.547194 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:59.547169 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-51119c-546967fcc9-mtcnh"] Nov 03 00:56:59.553501 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:59.553471 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9pckr\" (UniqueName: \"kubernetes.io/projected/a4357adf-7a95-4137-a489-ba7c3c7855f3-kube-api-access-9pckr\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:56:59.553501 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:59.553497 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pkxs2\" (UniqueName: \"kubernetes.io/projected/28bda42f-8493-4e36-923e-8f60a48f4ed7-kube-api-access-pkxs2\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:56:59.982393 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:59.982355 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28bda42f-8493-4e36-923e-8f60a48f4ed7" path="/var/lib/kubelet/pods/28bda42f-8493-4e36-923e-8f60a48f4ed7/volumes" Nov 03 00:56:59.982934 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:59.982913 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4357adf-7a95-4137-a489-ba7c3c7855f3" path="/var/lib/kubelet/pods/a4357adf-7a95-4137-a489-ba7c3c7855f3/volumes" Nov 03 00:56:59.983384 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:56:59.983361 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc00e6aa-299e-4208-a143-7a55bb0f8e5b" path="/var/lib/kubelet/pods/dc00e6aa-299e-4208-a143-7a55bb0f8e5b/volumes" Nov 03 00:57:00.952328 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:00.952292 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-51119c-openshift-default-56b69d85d9-ds9fq"] Nov 03 00:57:00.952620 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:00.952584 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-51119c-openshift-default-56b69d85d9-ds9fq" podUID="934ddd77-9c41-49d5-bf51-fb5d8e765efd" containerName="istio-proxy" containerID="cri-o://34a281ee2bb427fd92e190df6eaff85b9c78d9ffe446167d534e9d841f3fa295" gracePeriod=30 Nov 03 00:57:06.303507 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:06.303477 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-51119c-openshift-default-56b69d85d9-ds9fq" Nov 03 00:57:06.473668 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:06.473629 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/934ddd77-9c41-49d5-bf51-fb5d8e765efd-istio-token\" (UniqueName: \"kubernetes.io/projected/934ddd77-9c41-49d5-bf51-fb5d8e765efd-istio-token\") pod \"934ddd77-9c41-49d5-bf51-fb5d8e765efd\" (UID: \"934ddd77-9c41-49d5-bf51-fb5d8e765efd\") " Nov 03 00:57:06.473668 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:06.473678 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/934ddd77-9c41-49d5-bf51-fb5d8e765efd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/934ddd77-9c41-49d5-bf51-fb5d8e765efd-istio-data\") pod \"934ddd77-9c41-49d5-bf51-fb5d8e765efd\" (UID: \"934ddd77-9c41-49d5-bf51-fb5d8e765efd\") " Nov 03 00:57:06.473945 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:06.473717 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/934ddd77-9c41-49d5-bf51-fb5d8e765efd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/934ddd77-9c41-49d5-bf51-fb5d8e765efd-istiod-ca-cert\") pod \"934ddd77-9c41-49d5-bf51-fb5d8e765efd\" (UID: \"934ddd77-9c41-49d5-bf51-fb5d8e765efd\") " Nov 03 00:57:06.473945 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:06.473825 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/934ddd77-9c41-49d5-bf51-fb5d8e765efd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/934ddd77-9c41-49d5-bf51-fb5d8e765efd-workload-socket\") pod \"934ddd77-9c41-49d5-bf51-fb5d8e765efd\" (UID: \"934ddd77-9c41-49d5-bf51-fb5d8e765efd\") " Nov 03 00:57:06.473945 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:06.473915 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/934ddd77-9c41-49d5-bf51-fb5d8e765efd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/934ddd77-9c41-49d5-bf51-fb5d8e765efd-credential-socket\") pod \"934ddd77-9c41-49d5-bf51-fb5d8e765efd\" (UID: \"934ddd77-9c41-49d5-bf51-fb5d8e765efd\") " Nov 03 00:57:06.474100 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:06.474015 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/934ddd77-9c41-49d5-bf51-fb5d8e765efd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/934ddd77-9c41-49d5-bf51-fb5d8e765efd-istio-podinfo\") pod \"934ddd77-9c41-49d5-bf51-fb5d8e765efd\" (UID: \"934ddd77-9c41-49d5-bf51-fb5d8e765efd\") " Nov 03 00:57:06.474100 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:06.474048 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/934ddd77-9c41-49d5-bf51-fb5d8e765efd-kube-api-access-d2bgq\" (UniqueName: \"kubernetes.io/projected/934ddd77-9c41-49d5-bf51-fb5d8e765efd-kube-api-access-d2bgq\") pod \"934ddd77-9c41-49d5-bf51-fb5d8e765efd\" (UID: \"934ddd77-9c41-49d5-bf51-fb5d8e765efd\") " Nov 03 00:57:06.474224 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:06.474107 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/934ddd77-9c41-49d5-bf51-fb5d8e765efd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/934ddd77-9c41-49d5-bf51-fb5d8e765efd-workload-certs\") pod \"934ddd77-9c41-49d5-bf51-fb5d8e765efd\" (UID: \"934ddd77-9c41-49d5-bf51-fb5d8e765efd\") " Nov 03 00:57:06.474224 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:06.474156 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/934ddd77-9c41-49d5-bf51-fb5d8e765efd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/934ddd77-9c41-49d5-bf51-fb5d8e765efd-istio-envoy\") pod \"934ddd77-9c41-49d5-bf51-fb5d8e765efd\" (UID: \"934ddd77-9c41-49d5-bf51-fb5d8e765efd\") " Nov 03 00:57:06.474224 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:06.474077 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/934ddd77-9c41-49d5-bf51-fb5d8e765efd-istio-data" pod "934ddd77-9c41-49d5-bf51-fb5d8e765efd" (UID: "934ddd77-9c41-49d5-bf51-fb5d8e765efd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:57:06.474438 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:06.474149 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/934ddd77-9c41-49d5-bf51-fb5d8e765efd-workload-socket" pod "934ddd77-9c41-49d5-bf51-fb5d8e765efd" (UID: "934ddd77-9c41-49d5-bf51-fb5d8e765efd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:57:06.474438 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:06.474228 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/934ddd77-9c41-49d5-bf51-fb5d8e765efd-credential-socket" pod "934ddd77-9c41-49d5-bf51-fb5d8e765efd" (UID: "934ddd77-9c41-49d5-bf51-fb5d8e765efd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:57:06.474438 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:06.474307 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/934ddd77-9c41-49d5-bf51-fb5d8e765efd-istiod-ca-cert" pod "934ddd77-9c41-49d5-bf51-fb5d8e765efd" (UID: "934ddd77-9c41-49d5-bf51-fb5d8e765efd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:57:06.474438 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:06.474418 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/934ddd77-9c41-49d5-bf51-fb5d8e765efd-workload-certs" pod "934ddd77-9c41-49d5-bf51-fb5d8e765efd" (UID: "934ddd77-9c41-49d5-bf51-fb5d8e765efd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:57:06.474954 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:06.474934 4421 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/934ddd77-9c41-49d5-bf51-fb5d8e765efd-istio-data\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:57:06.474954 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:06.474956 4421 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/934ddd77-9c41-49d5-bf51-fb5d8e765efd-istiod-ca-cert\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:57:06.475091 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:06.474966 4421 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/934ddd77-9c41-49d5-bf51-fb5d8e765efd-workload-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:57:06.475091 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:06.474975 4421 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/934ddd77-9c41-49d5-bf51-fb5d8e765efd-credential-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:57:06.475091 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:06.474983 4421 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/934ddd77-9c41-49d5-bf51-fb5d8e765efd-workload-certs\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:57:06.476261 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:06.476230 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/934ddd77-9c41-49d5-bf51-fb5d8e765efd-istio-token" pod "934ddd77-9c41-49d5-bf51-fb5d8e765efd" (UID: "934ddd77-9c41-49d5-bf51-fb5d8e765efd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:57:06.476531 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:06.476508 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/934ddd77-9c41-49d5-bf51-fb5d8e765efd-istio-envoy" pod "934ddd77-9c41-49d5-bf51-fb5d8e765efd" (UID: "934ddd77-9c41-49d5-bf51-fb5d8e765efd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:57:06.476746 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:06.476719 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/934ddd77-9c41-49d5-bf51-fb5d8e765efd-istio-podinfo" pod "934ddd77-9c41-49d5-bf51-fb5d8e765efd" (UID: "934ddd77-9c41-49d5-bf51-fb5d8e765efd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:57:06.476848 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:06.476743 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/934ddd77-9c41-49d5-bf51-fb5d8e765efd-kube-api-access-d2bgq" pod "934ddd77-9c41-49d5-bf51-fb5d8e765efd" (UID: "934ddd77-9c41-49d5-bf51-fb5d8e765efd"). InnerVolumeSpecName "kube-api-access-d2bgq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:57:06.576544 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:06.576454 4421 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/934ddd77-9c41-49d5-bf51-fb5d8e765efd-istio-podinfo\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:57:06.576544 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:06.576489 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2bgq\" (UniqueName: \"kubernetes.io/projected/934ddd77-9c41-49d5-bf51-fb5d8e765efd-kube-api-access-d2bgq\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:57:06.576544 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:06.576502 4421 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/934ddd77-9c41-49d5-bf51-fb5d8e765efd-istio-envoy\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:57:06.576544 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:06.576511 4421 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/934ddd77-9c41-49d5-bf51-fb5d8e765efd-istio-token\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:57:06.614553 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:06.614519 4421 generic.go:356] "Generic (PLEG): container finished" podID="934ddd77-9c41-49d5-bf51-fb5d8e765efd" containerID="34a281ee2bb427fd92e190df6eaff85b9c78d9ffe446167d534e9d841f3fa295" exitCode=0 Nov 03 00:57:06.614755 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:06.614580 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-51119c-openshift-default-56b69d85d9-ds9fq" Nov 03 00:57:06.614755 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:06.614604 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-51119c-openshift-default-56b69d85d9-ds9fq" event={"ID":"934ddd77-9c41-49d5-bf51-fb5d8e765efd","Type":"ContainerDied","Data":"34a281ee2bb427fd92e190df6eaff85b9c78d9ffe446167d534e9d841f3fa295"} Nov 03 00:57:06.614755 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:06.614683 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-51119c-openshift-default-56b69d85d9-ds9fq" event={"ID":"934ddd77-9c41-49d5-bf51-fb5d8e765efd","Type":"ContainerDied","Data":"596402284c283ae1b74bdd855f63cc3102d44907f06067752e23fc018c0af46f"} Nov 03 00:57:06.614755 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:06.614714 4421 scope.go:122] "RemoveContainer" containerID="34a281ee2bb427fd92e190df6eaff85b9c78d9ffe446167d534e9d841f3fa295" Nov 03 00:57:06.626083 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:06.626062 4421 scope.go:122] "RemoveContainer" containerID="c5363a1ef8d1f7d5e5414208fac3058904333d527167c17f09e4035434ddc459" Nov 03 00:57:06.636466 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:06.636445 4421 scope.go:122] "RemoveContainer" containerID="34a281ee2bb427fd92e190df6eaff85b9c78d9ffe446167d534e9d841f3fa295" Nov 03 00:57:06.636805 ip-10-0-133-217 kubenswrapper[4421]: E1103 00:57:06.636783 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34a281ee2bb427fd92e190df6eaff85b9c78d9ffe446167d534e9d841f3fa295\": container with ID starting with 34a281ee2bb427fd92e190df6eaff85b9c78d9ffe446167d534e9d841f3fa295 not found: ID does not exist" containerID="34a281ee2bb427fd92e190df6eaff85b9c78d9ffe446167d534e9d841f3fa295" Nov 03 00:57:06.636893 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:06.636812 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34a281ee2bb427fd92e190df6eaff85b9c78d9ffe446167d534e9d841f3fa295"} err="failed to get container status \"34a281ee2bb427fd92e190df6eaff85b9c78d9ffe446167d534e9d841f3fa295\": rpc error: code = NotFound desc = could not find container \"34a281ee2bb427fd92e190df6eaff85b9c78d9ffe446167d534e9d841f3fa295\": container with ID starting with 34a281ee2bb427fd92e190df6eaff85b9c78d9ffe446167d534e9d841f3fa295 not found: ID does not exist" Nov 03 00:57:06.636893 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:06.636830 4421 scope.go:122] "RemoveContainer" containerID="c5363a1ef8d1f7d5e5414208fac3058904333d527167c17f09e4035434ddc459" Nov 03 00:57:06.637134 ip-10-0-133-217 kubenswrapper[4421]: E1103 00:57:06.637116 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5363a1ef8d1f7d5e5414208fac3058904333d527167c17f09e4035434ddc459\": container with ID starting with c5363a1ef8d1f7d5e5414208fac3058904333d527167c17f09e4035434ddc459 not found: ID does not exist" containerID="c5363a1ef8d1f7d5e5414208fac3058904333d527167c17f09e4035434ddc459" Nov 03 00:57:06.637188 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:06.637144 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5363a1ef8d1f7d5e5414208fac3058904333d527167c17f09e4035434ddc459"} err="failed to get container status \"c5363a1ef8d1f7d5e5414208fac3058904333d527167c17f09e4035434ddc459\": rpc error: code = NotFound desc = could not find container \"c5363a1ef8d1f7d5e5414208fac3058904333d527167c17f09e4035434ddc459\": container with ID starting with c5363a1ef8d1f7d5e5414208fac3058904333d527167c17f09e4035434ddc459 not found: ID does not exist" Nov 03 00:57:06.640965 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:06.640943 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-51119c-openshift-default-56b69d85d9-ds9fq"] Nov 03 00:57:06.645627 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:06.645607 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-51119c-openshift-default-56b69d85d9-ds9fq"] Nov 03 00:57:08.023300 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:08.022499 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="934ddd77-9c41-49d5-bf51-fb5d8e765efd" path="/var/lib/kubelet/pods/934ddd77-9c41-49d5-bf51-fb5d8e765efd/volumes" Nov 03 00:57:08.319379 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:08.319294 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dd03e395-pbnvh"] Nov 03 00:57:08.319905 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:08.319892 4421 cpu_manager.go:414] "containerMap: removing container" podUID="934ddd77-9c41-49d5-bf51-fb5d8e765efd" containerName="istio-proxy" Nov 03 00:57:08.319944 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:08.319906 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="934ddd77-9c41-49d5-bf51-fb5d8e765efd" containerName="istio-proxy" Nov 03 00:57:08.319944 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:08.319917 4421 cpu_manager.go:414] "containerMap: removing container" podUID="a4357adf-7a95-4137-a489-ba7c3c7855f3" containerName="payload-pre-processing" Nov 03 00:57:08.319944 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:08.319922 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4357adf-7a95-4137-a489-ba7c3c7855f3" containerName="payload-pre-processing" Nov 03 00:57:08.319944 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:08.319932 4421 cpu_manager.go:414] "containerMap: removing container" podUID="934ddd77-9c41-49d5-bf51-fb5d8e765efd" containerName="istio-proxy" Nov 03 00:57:08.319944 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:08.319939 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="934ddd77-9c41-49d5-bf51-fb5d8e765efd" containerName="istio-proxy" Nov 03 00:57:08.319944 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:08.319945 4421 cpu_manager.go:414] "containerMap: removing container" podUID="a091a850-4bf7-4272-b9ea-3fbd5b7b3df4" containerName="revoke-keys" Nov 03 00:57:08.320119 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:08.319949 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a091a850-4bf7-4272-b9ea-3fbd5b7b3df4" containerName="revoke-keys" Nov 03 00:57:08.320119 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:08.319972 4421 cpu_manager.go:414] "containerMap: removing container" podUID="28bda42f-8493-4e36-923e-8f60a48f4ed7" containerName="payload-processing" Nov 03 00:57:08.320119 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:08.319976 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28bda42f-8493-4e36-923e-8f60a48f4ed7" containerName="payload-processing" Nov 03 00:57:08.320119 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:08.320066 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="28bda42f-8493-4e36-923e-8f60a48f4ed7" containerName="payload-processing" Nov 03 00:57:08.320119 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:08.320081 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="934ddd77-9c41-49d5-bf51-fb5d8e765efd" containerName="istio-proxy" Nov 03 00:57:08.320119 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:08.320088 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4357adf-7a95-4137-a489-ba7c3c7855f3" containerName="payload-pre-processing" Nov 03 00:57:08.320119 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:08.320095 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="934ddd77-9c41-49d5-bf51-fb5d8e765efd" containerName="istio-proxy" Nov 03 00:57:08.320119 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:08.320100 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="a091a850-4bf7-4272-b9ea-3fbd5b7b3df4" containerName="revoke-keys" Nov 03 00:57:08.324720 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:08.324702 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dd03e395-pbnvh" Nov 03 00:57:08.332568 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:08.332509 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:57:08.333317 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:08.333143 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 00:57:08.334144 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:08.333237 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6q4xp\"" Nov 03 00:57:08.334250 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:08.333320 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 00:57:08.335910 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:08.335883 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dd03e395-pbnvh"] Nov 03 00:57:08.385915 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:08.383831 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-54df68-openshift-default-74b6dbc6cf-4kmpk"] Nov 03 00:57:08.426080 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:08.425417 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bd24j\" (UniqueName: \"kubernetes.io/projected/0fc241d1-cb3f-4b97-af77-e850e30abcc7-kube-api-access-bd24j\") pod \"maas-api-revoke-keys-e2e-mt-dd03e395-pbnvh\" (UID: \"0fc241d1-cb3f-4b97-af77-e850e30abcc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dd03e395-pbnvh" Nov 03 00:57:08.426080 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:08.425489 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fc241d1-cb3f-4b97-af77-e850e30abcc7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-dd03e395-pbnvh\" (UID: \"0fc241d1-cb3f-4b97-af77-e850e30abcc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dd03e395-pbnvh" Nov 03 00:57:08.426080 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:08.425686 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fc241d1-cb3f-4b97-af77-e850e30abcc7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-dd03e395-pbnvh\" (UID: \"0fc241d1-cb3f-4b97-af77-e850e30abcc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dd03e395-pbnvh" Nov 03 00:57:08.437016 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:08.436000 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-54df68-openshift-default-74b6dbc6cf-4kmpk" Nov 03 00:57:08.442573 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:08.441361 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-54df68-openshift-default-dockercfg-5lxwr\"" Nov 03 00:57:08.442957 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:08.442908 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-54df68-openshift-default-74b6dbc6cf-4kmpk"] Nov 03 00:57:08.527706 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:08.527666 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bd24j\" (UniqueName: \"kubernetes.io/projected/0fc241d1-cb3f-4b97-af77-e850e30abcc7-kube-api-access-bd24j\") pod \"maas-api-revoke-keys-e2e-mt-dd03e395-pbnvh\" (UID: \"0fc241d1-cb3f-4b97-af77-e850e30abcc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dd03e395-pbnvh" Nov 03 00:57:08.527949 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:08.527927 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fc241d1-cb3f-4b97-af77-e850e30abcc7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-dd03e395-pbnvh\" (UID: \"0fc241d1-cb3f-4b97-af77-e850e30abcc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dd03e395-pbnvh" Nov 03 00:57:08.528128 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:08.528111 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6b01747-59d0-4ee6-9a96-4e3d1cad82df-istio-envoy\") pod \"e2e-api-a-54df68-openshift-default-74b6dbc6cf-4kmpk\" (UID: \"a6b01747-59d0-4ee6-9a96-4e3d1cad82df\") " pod="openshift-ingress/e2e-api-a-54df68-openshift-default-74b6dbc6cf-4kmpk" Nov 03 00:57:08.528265 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:08.528246 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6b01747-59d0-4ee6-9a96-4e3d1cad82df-credential-socket\") pod \"e2e-api-a-54df68-openshift-default-74b6dbc6cf-4kmpk\" (UID: \"a6b01747-59d0-4ee6-9a96-4e3d1cad82df\") " pod="openshift-ingress/e2e-api-a-54df68-openshift-default-74b6dbc6cf-4kmpk" Nov 03 00:57:08.528445 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:08.528427 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6b01747-59d0-4ee6-9a96-4e3d1cad82df-workload-socket\") pod \"e2e-api-a-54df68-openshift-default-74b6dbc6cf-4kmpk\" (UID: \"a6b01747-59d0-4ee6-9a96-4e3d1cad82df\") " pod="openshift-ingress/e2e-api-a-54df68-openshift-default-74b6dbc6cf-4kmpk" Nov 03 00:57:08.528583 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:08.528567 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6b01747-59d0-4ee6-9a96-4e3d1cad82df-istiod-ca-cert\") pod \"e2e-api-a-54df68-openshift-default-74b6dbc6cf-4kmpk\" (UID: \"a6b01747-59d0-4ee6-9a96-4e3d1cad82df\") " pod="openshift-ingress/e2e-api-a-54df68-openshift-default-74b6dbc6cf-4kmpk" Nov 03 00:57:08.528728 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:08.528713 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6b01747-59d0-4ee6-9a96-4e3d1cad82df-istio-podinfo\") pod \"e2e-api-a-54df68-openshift-default-74b6dbc6cf-4kmpk\" (UID: \"a6b01747-59d0-4ee6-9a96-4e3d1cad82df\") " pod="openshift-ingress/e2e-api-a-54df68-openshift-default-74b6dbc6cf-4kmpk" Nov 03 00:57:08.528987 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:08.528914 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6b01747-59d0-4ee6-9a96-4e3d1cad82df-istio-data\") pod \"e2e-api-a-54df68-openshift-default-74b6dbc6cf-4kmpk\" (UID: \"a6b01747-59d0-4ee6-9a96-4e3d1cad82df\") " pod="openshift-ingress/e2e-api-a-54df68-openshift-default-74b6dbc6cf-4kmpk" Nov 03 00:57:08.529162 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:08.529147 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fc241d1-cb3f-4b97-af77-e850e30abcc7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-dd03e395-pbnvh\" (UID: \"0fc241d1-cb3f-4b97-af77-e850e30abcc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dd03e395-pbnvh" Nov 03 00:57:08.529346 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:08.529330 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4j4lb\" (UniqueName: \"kubernetes.io/projected/a6b01747-59d0-4ee6-9a96-4e3d1cad82df-kube-api-access-4j4lb\") pod \"e2e-api-a-54df68-openshift-default-74b6dbc6cf-4kmpk\" (UID: \"a6b01747-59d0-4ee6-9a96-4e3d1cad82df\") " pod="openshift-ingress/e2e-api-a-54df68-openshift-default-74b6dbc6cf-4kmpk" Nov 03 00:57:08.529564 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:08.529542 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6b01747-59d0-4ee6-9a96-4e3d1cad82df-istio-token\") pod \"e2e-api-a-54df68-openshift-default-74b6dbc6cf-4kmpk\" (UID: \"a6b01747-59d0-4ee6-9a96-4e3d1cad82df\") " pod="openshift-ingress/e2e-api-a-54df68-openshift-default-74b6dbc6cf-4kmpk" Nov 03 00:57:08.529817 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:08.529796 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6b01747-59d0-4ee6-9a96-4e3d1cad82df-workload-certs\") pod \"e2e-api-a-54df68-openshift-default-74b6dbc6cf-4kmpk\" (UID: \"a6b01747-59d0-4ee6-9a96-4e3d1cad82df\") " pod="openshift-ingress/e2e-api-a-54df68-openshift-default-74b6dbc6cf-4kmpk" Nov 03 00:57:08.538685 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:08.538649 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fc241d1-cb3f-4b97-af77-e850e30abcc7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-dd03e395-pbnvh\" (UID: \"0fc241d1-cb3f-4b97-af77-e850e30abcc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dd03e395-pbnvh" Nov 03 00:57:08.555303 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:08.550159 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bd24j\" (UniqueName: \"kubernetes.io/projected/0fc241d1-cb3f-4b97-af77-e850e30abcc7-kube-api-access-bd24j\") pod \"maas-api-revoke-keys-e2e-mt-dd03e395-pbnvh\" (UID: \"0fc241d1-cb3f-4b97-af77-e850e30abcc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dd03e395-pbnvh" Nov 03 00:57:08.555303 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:08.554727 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fc241d1-cb3f-4b97-af77-e850e30abcc7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-dd03e395-pbnvh\" (UID: \"0fc241d1-cb3f-4b97-af77-e850e30abcc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dd03e395-pbnvh" Nov 03 00:57:08.636316 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:08.636189 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4j4lb\" (UniqueName: \"kubernetes.io/projected/a6b01747-59d0-4ee6-9a96-4e3d1cad82df-kube-api-access-4j4lb\") pod \"e2e-api-a-54df68-openshift-default-74b6dbc6cf-4kmpk\" (UID: \"a6b01747-59d0-4ee6-9a96-4e3d1cad82df\") " pod="openshift-ingress/e2e-api-a-54df68-openshift-default-74b6dbc6cf-4kmpk" Nov 03 00:57:08.665995 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:08.665964 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6b01747-59d0-4ee6-9a96-4e3d1cad82df-istio-token\") pod \"e2e-api-a-54df68-openshift-default-74b6dbc6cf-4kmpk\" (UID: \"a6b01747-59d0-4ee6-9a96-4e3d1cad82df\") " pod="openshift-ingress/e2e-api-a-54df68-openshift-default-74b6dbc6cf-4kmpk" Nov 03 00:57:08.666330 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:08.666308 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6b01747-59d0-4ee6-9a96-4e3d1cad82df-workload-certs\") pod \"e2e-api-a-54df68-openshift-default-74b6dbc6cf-4kmpk\" (UID: \"a6b01747-59d0-4ee6-9a96-4e3d1cad82df\") " pod="openshift-ingress/e2e-api-a-54df68-openshift-default-74b6dbc6cf-4kmpk" Nov 03 00:57:08.666638 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:08.666619 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6b01747-59d0-4ee6-9a96-4e3d1cad82df-istio-envoy\") pod \"e2e-api-a-54df68-openshift-default-74b6dbc6cf-4kmpk\" (UID: \"a6b01747-59d0-4ee6-9a96-4e3d1cad82df\") " pod="openshift-ingress/e2e-api-a-54df68-openshift-default-74b6dbc6cf-4kmpk" Nov 03 00:57:08.666791 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:08.666774 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6b01747-59d0-4ee6-9a96-4e3d1cad82df-credential-socket\") pod \"e2e-api-a-54df68-openshift-default-74b6dbc6cf-4kmpk\" (UID: \"a6b01747-59d0-4ee6-9a96-4e3d1cad82df\") " pod="openshift-ingress/e2e-api-a-54df68-openshift-default-74b6dbc6cf-4kmpk" Nov 03 00:57:08.666947 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:08.666932 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6b01747-59d0-4ee6-9a96-4e3d1cad82df-workload-socket\") pod \"e2e-api-a-54df68-openshift-default-74b6dbc6cf-4kmpk\" (UID: \"a6b01747-59d0-4ee6-9a96-4e3d1cad82df\") " pod="openshift-ingress/e2e-api-a-54df68-openshift-default-74b6dbc6cf-4kmpk" Nov 03 00:57:08.667062 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:08.667049 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6b01747-59d0-4ee6-9a96-4e3d1cad82df-istiod-ca-cert\") pod \"e2e-api-a-54df68-openshift-default-74b6dbc6cf-4kmpk\" (UID: \"a6b01747-59d0-4ee6-9a96-4e3d1cad82df\") " pod="openshift-ingress/e2e-api-a-54df68-openshift-default-74b6dbc6cf-4kmpk" Nov 03 00:57:08.667188 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:08.667173 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6b01747-59d0-4ee6-9a96-4e3d1cad82df-istio-podinfo\") pod \"e2e-api-a-54df68-openshift-default-74b6dbc6cf-4kmpk\" (UID: \"a6b01747-59d0-4ee6-9a96-4e3d1cad82df\") " pod="openshift-ingress/e2e-api-a-54df68-openshift-default-74b6dbc6cf-4kmpk" Nov 03 00:57:08.667330 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:08.667314 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6b01747-59d0-4ee6-9a96-4e3d1cad82df-istio-data\") pod \"e2e-api-a-54df68-openshift-default-74b6dbc6cf-4kmpk\" (UID: \"a6b01747-59d0-4ee6-9a96-4e3d1cad82df\") " pod="openshift-ingress/e2e-api-a-54df68-openshift-default-74b6dbc6cf-4kmpk" Nov 03 00:57:08.667948 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:08.667924 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6b01747-59d0-4ee6-9a96-4e3d1cad82df-istio-data\") pod \"e2e-api-a-54df68-openshift-default-74b6dbc6cf-4kmpk\" (UID: \"a6b01747-59d0-4ee6-9a96-4e3d1cad82df\") " pod="openshift-ingress/e2e-api-a-54df68-openshift-default-74b6dbc6cf-4kmpk" Nov 03 00:57:08.679352 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:08.664744 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dd03e395-pbnvh" Nov 03 00:57:08.702339 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:08.675868 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6b01747-59d0-4ee6-9a96-4e3d1cad82df-workload-certs\") pod \"e2e-api-a-54df68-openshift-default-74b6dbc6cf-4kmpk\" (UID: \"a6b01747-59d0-4ee6-9a96-4e3d1cad82df\") " pod="openshift-ingress/e2e-api-a-54df68-openshift-default-74b6dbc6cf-4kmpk" Nov 03 00:57:08.702339 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:08.680621 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6b01747-59d0-4ee6-9a96-4e3d1cad82df-workload-socket\") pod \"e2e-api-a-54df68-openshift-default-74b6dbc6cf-4kmpk\" (UID: \"a6b01747-59d0-4ee6-9a96-4e3d1cad82df\") " pod="openshift-ingress/e2e-api-a-54df68-openshift-default-74b6dbc6cf-4kmpk" Nov 03 00:57:08.702339 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:08.680863 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6b01747-59d0-4ee6-9a96-4e3d1cad82df-credential-socket\") pod \"e2e-api-a-54df68-openshift-default-74b6dbc6cf-4kmpk\" (UID: \"a6b01747-59d0-4ee6-9a96-4e3d1cad82df\") " pod="openshift-ingress/e2e-api-a-54df68-openshift-default-74b6dbc6cf-4kmpk" Nov 03 00:57:08.702339 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:08.689104 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6b01747-59d0-4ee6-9a96-4e3d1cad82df-istiod-ca-cert\") pod \"e2e-api-a-54df68-openshift-default-74b6dbc6cf-4kmpk\" (UID: \"a6b01747-59d0-4ee6-9a96-4e3d1cad82df\") " pod="openshift-ingress/e2e-api-a-54df68-openshift-default-74b6dbc6cf-4kmpk" Nov 03 00:57:08.708302 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:08.704576 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6b01747-59d0-4ee6-9a96-4e3d1cad82df-istio-envoy\") pod \"e2e-api-a-54df68-openshift-default-74b6dbc6cf-4kmpk\" (UID: \"a6b01747-59d0-4ee6-9a96-4e3d1cad82df\") " pod="openshift-ingress/e2e-api-a-54df68-openshift-default-74b6dbc6cf-4kmpk" Nov 03 00:57:08.734317 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:08.724050 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4j4lb\" (UniqueName: \"kubernetes.io/projected/a6b01747-59d0-4ee6-9a96-4e3d1cad82df-kube-api-access-4j4lb\") pod \"e2e-api-a-54df68-openshift-default-74b6dbc6cf-4kmpk\" (UID: \"a6b01747-59d0-4ee6-9a96-4e3d1cad82df\") " pod="openshift-ingress/e2e-api-a-54df68-openshift-default-74b6dbc6cf-4kmpk" Nov 03 00:57:08.760297 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:08.758965 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6b01747-59d0-4ee6-9a96-4e3d1cad82df-istio-podinfo\") pod \"e2e-api-a-54df68-openshift-default-74b6dbc6cf-4kmpk\" (UID: \"a6b01747-59d0-4ee6-9a96-4e3d1cad82df\") " pod="openshift-ingress/e2e-api-a-54df68-openshift-default-74b6dbc6cf-4kmpk" Nov 03 00:57:08.778310 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:08.765825 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6b01747-59d0-4ee6-9a96-4e3d1cad82df-istio-token\") pod \"e2e-api-a-54df68-openshift-default-74b6dbc6cf-4kmpk\" (UID: \"a6b01747-59d0-4ee6-9a96-4e3d1cad82df\") " pod="openshift-ingress/e2e-api-a-54df68-openshift-default-74b6dbc6cf-4kmpk" Nov 03 00:57:09.038790 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:09.038757 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48014db0-5n4s6"] Nov 03 00:57:09.043642 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:09.043612 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48014db0-5n4s6"] Nov 03 00:57:09.062171 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:09.062135 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-54df68-openshift-default-74b6dbc6cf-4kmpk" Nov 03 00:57:09.195781 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:09.195729 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dd03e395-pbnvh"] Nov 03 00:57:09.486437 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:57:09.486398 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6b01747_59d0_4ee6_9a96_4e3d1cad82df.slice/crio-c2daa186c29d0ad3c3d45918d9190b6e659428eb51410838703e92a9321fa9dd WatchSource:0}: Error finding container c2daa186c29d0ad3c3d45918d9190b6e659428eb51410838703e92a9321fa9dd: Status 404 returned error can't find the container with id c2daa186c29d0ad3c3d45918d9190b6e659428eb51410838703e92a9321fa9dd Nov 03 00:57:09.486866 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:09.486842 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-54df68-openshift-default-74b6dbc6cf-4kmpk"] Nov 03 00:57:09.489109 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:09.489078 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:57:09.489227 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:09.489142 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:57:09.489227 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:09.489173 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:57:09.849189 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:09.849154 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-54df68-openshift-default-74b6dbc6cf-4kmpk" event={"ID":"a6b01747-59d0-4ee6-9a96-4e3d1cad82df","Type":"ContainerStarted","Data":"c47e742c228c773354e827269a6ee024a997b9cd9dabf084161398b1ef7e0f75"} Nov 03 00:57:09.849412 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:09.849212 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-54df68-openshift-default-74b6dbc6cf-4kmpk" event={"ID":"a6b01747-59d0-4ee6-9a96-4e3d1cad82df","Type":"ContainerStarted","Data":"c2daa186c29d0ad3c3d45918d9190b6e659428eb51410838703e92a9321fa9dd"} Nov 03 00:57:09.851237 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:09.851204 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dd03e395-pbnvh" event={"ID":"0fc241d1-cb3f-4b97-af77-e850e30abcc7","Type":"ContainerStarted","Data":"f50ecb708219daf73c5693972dc90431e6192f29d538724d08f5b21b041748df"} Nov 03 00:57:09.851396 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:09.851260 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dd03e395-pbnvh" event={"ID":"0fc241d1-cb3f-4b97-af77-e850e30abcc7","Type":"ContainerStarted","Data":"0925601aad86f9bbf4f9e0e10476ec8477dc6fb91f2eb34f05df7a8375367569"} Nov 03 00:57:09.870769 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:09.870709 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-54df68-openshift-default-74b6dbc6cf-4kmpk" podStartSLOduration=1.870692512 podStartE2EDuration="1.870692512s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:57:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:57:09.870470528 +0000 UTC m=+972.041473626" watchObservedRunningTime="2026-11-03 00:57:09.870692512 +0000 UTC m=+972.041695648" Nov 03 00:57:09.893250 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:09.893197 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dd03e395-pbnvh" podStartSLOduration=1.893181299 podStartE2EDuration="1.893181299s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:57:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:57:09.891744382 +0000 UTC m=+972.062747478" watchObservedRunningTime="2026-11-03 00:57:09.893181299 +0000 UTC m=+972.064184394" Nov 03 00:57:10.000316 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:09.999669 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="078e68c2-59ea-4d68-86c8-11b4b9dfa99c" path="/var/lib/kubelet/pods/078e68c2-59ea-4d68-86c8-11b4b9dfa99c/volumes" Nov 03 00:57:10.066199 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:10.066158 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-54df68-openshift-default-74b6dbc6cf-4kmpk" Nov 03 00:57:10.077685 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:10.077652 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-54df68-openshift-default-74b6dbc6cf-4kmpk" Nov 03 00:57:10.860805 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:10.860774 4421 generic.go:356] "Generic (PLEG): container finished" podID="0fc241d1-cb3f-4b97-af77-e850e30abcc7" containerID="f50ecb708219daf73c5693972dc90431e6192f29d538724d08f5b21b041748df" exitCode=0 Nov 03 00:57:10.860981 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:10.860858 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dd03e395-pbnvh" event={"ID":"0fc241d1-cb3f-4b97-af77-e850e30abcc7","Type":"ContainerDied","Data":"f50ecb708219daf73c5693972dc90431e6192f29d538724d08f5b21b041748df"} Nov 03 00:57:10.861295 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:10.861263 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-54df68-openshift-default-74b6dbc6cf-4kmpk" Nov 03 00:57:10.862261 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:10.862239 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-54df68-openshift-default-74b6dbc6cf-4kmpk" Nov 03 00:57:12.203429 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:12.203398 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dd03e395-pbnvh" Nov 03 00:57:12.256298 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:12.256229 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0fc241d1-cb3f-4b97-af77-e850e30abcc7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fc241d1-cb3f-4b97-af77-e850e30abcc7-maas-api-xks-ca\") pod \"0fc241d1-cb3f-4b97-af77-e850e30abcc7\" (UID: \"0fc241d1-cb3f-4b97-af77-e850e30abcc7\") " Nov 03 00:57:12.256487 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:12.256387 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0fc241d1-cb3f-4b97-af77-e850e30abcc7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fc241d1-cb3f-4b97-af77-e850e30abcc7-maas-api-openshift-ca\") pod \"0fc241d1-cb3f-4b97-af77-e850e30abcc7\" (UID: \"0fc241d1-cb3f-4b97-af77-e850e30abcc7\") " Nov 03 00:57:12.256680 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:12.256656 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fc241d1-cb3f-4b97-af77-e850e30abcc7-kube-api-access-bd24j\" (UniqueName: \"kubernetes.io/projected/0fc241d1-cb3f-4b97-af77-e850e30abcc7-kube-api-access-bd24j\") pod \"0fc241d1-cb3f-4b97-af77-e850e30abcc7\" (UID: \"0fc241d1-cb3f-4b97-af77-e850e30abcc7\") " Nov 03 00:57:12.260809 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:12.260775 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fc241d1-cb3f-4b97-af77-e850e30abcc7-maas-api-openshift-ca" pod "0fc241d1-cb3f-4b97-af77-e850e30abcc7" (UID: "0fc241d1-cb3f-4b97-af77-e850e30abcc7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:57:12.263267 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:12.263230 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fc241d1-cb3f-4b97-af77-e850e30abcc7-kube-api-access-bd24j" pod "0fc241d1-cb3f-4b97-af77-e850e30abcc7" (UID: "0fc241d1-cb3f-4b97-af77-e850e30abcc7"). InnerVolumeSpecName "kube-api-access-bd24j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:57:12.275314 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:12.273496 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0fc241d1-cb3f-4b97-af77-e850e30abcc7-maas-api-xks-ca" pod "0fc241d1-cb3f-4b97-af77-e850e30abcc7" (UID: "0fc241d1-cb3f-4b97-af77-e850e30abcc7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:57:12.467374 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:12.467078 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fc241d1-cb3f-4b97-af77-e850e30abcc7-maas-api-openshift-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:57:12.467374 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:12.467112 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bd24j\" (UniqueName: \"kubernetes.io/projected/0fc241d1-cb3f-4b97-af77-e850e30abcc7-kube-api-access-bd24j\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:57:12.467374 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:12.467129 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fc241d1-cb3f-4b97-af77-e850e30abcc7-maas-api-xks-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:57:12.880089 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:12.880045 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dd03e395-pbnvh" event={"ID":"0fc241d1-cb3f-4b97-af77-e850e30abcc7","Type":"ContainerDied","Data":"0925601aad86f9bbf4f9e0e10476ec8477dc6fb91f2eb34f05df7a8375367569"} Nov 03 00:57:12.880089 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:12.880078 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dd03e395-pbnvh" Nov 03 00:57:12.880339 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:12.880103 4421 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0925601aad86f9bbf4f9e0e10476ec8477dc6fb91f2eb34f05df7a8375367569" Nov 03 00:57:13.046879 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:13.046842 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09592f3f-rk29m"] Nov 03 00:57:13.051835 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:13.051807 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09592f3f-rk29m"] Nov 03 00:57:13.644634 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:13.644597 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-54df68-59cb67b865-9d886"] Nov 03 00:57:13.646122 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:13.646092 4421 cpu_manager.go:414] "containerMap: removing container" podUID="0fc241d1-cb3f-4b97-af77-e850e30abcc7" containerName="revoke-keys" Nov 03 00:57:13.646309 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:13.646293 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fc241d1-cb3f-4b97-af77-e850e30abcc7" containerName="revoke-keys" Nov 03 00:57:13.646659 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:13.646644 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fc241d1-cb3f-4b97-af77-e850e30abcc7" containerName="revoke-keys" Nov 03 00:57:13.651338 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:13.651316 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-54df68-59cb67b865-9d886" Nov 03 00:57:13.667034 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:13.667002 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-54df68-59cb67b865-9d886"] Nov 03 00:57:13.743297 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:13.719624 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-54df68-dockercfg-2xhd6\"" Nov 03 00:57:13.743550 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:13.726461 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-54df68\"" Nov 03 00:57:13.823868 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:13.823830 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc0615b0-2580-4c26-a028-616bc6042cc3-plugins-config-volume\") pod \"payload-processing-e2e-api-a-54df68-59cb67b865-9d886\" (UID: \"fc0615b0-2580-4c26-a028-616bc6042cc3\") " pod="openshift-ingress/payload-processing-e2e-api-a-54df68-59cb67b865-9d886" Nov 03 00:57:13.845921 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:13.845880 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzcx2\" (UniqueName: \"kubernetes.io/projected/fc0615b0-2580-4c26-a028-616bc6042cc3-kube-api-access-qzcx2\") pod \"payload-processing-e2e-api-a-54df68-59cb67b865-9d886\" (UID: \"fc0615b0-2580-4c26-a028-616bc6042cc3\") " pod="openshift-ingress/payload-processing-e2e-api-a-54df68-59cb67b865-9d886" Nov 03 00:57:13.976527 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:13.974867 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzcx2\" (UniqueName: \"kubernetes.io/projected/fc0615b0-2580-4c26-a028-616bc6042cc3-kube-api-access-qzcx2\") pod \"payload-processing-e2e-api-a-54df68-59cb67b865-9d886\" (UID: \"fc0615b0-2580-4c26-a028-616bc6042cc3\") " pod="openshift-ingress/payload-processing-e2e-api-a-54df68-59cb67b865-9d886" Nov 03 00:57:13.976527 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:13.975084 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc0615b0-2580-4c26-a028-616bc6042cc3-plugins-config-volume\") pod \"payload-processing-e2e-api-a-54df68-59cb67b865-9d886\" (UID: \"fc0615b0-2580-4c26-a028-616bc6042cc3\") " pod="openshift-ingress/payload-processing-e2e-api-a-54df68-59cb67b865-9d886" Nov 03 00:57:13.976527 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:13.976040 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc0615b0-2580-4c26-a028-616bc6042cc3-plugins-config-volume\") pod \"payload-processing-e2e-api-a-54df68-59cb67b865-9d886\" (UID: \"fc0615b0-2580-4c26-a028-616bc6042cc3\") " pod="openshift-ingress/payload-processing-e2e-api-a-54df68-59cb67b865-9d886" Nov 03 00:57:13.993163 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:13.993121 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzcx2\" (UniqueName: \"kubernetes.io/projected/fc0615b0-2580-4c26-a028-616bc6042cc3-kube-api-access-qzcx2\") pod \"payload-processing-e2e-api-a-54df68-59cb67b865-9d886\" (UID: \"fc0615b0-2580-4c26-a028-616bc6042cc3\") " pod="openshift-ingress/payload-processing-e2e-api-a-54df68-59cb67b865-9d886" Nov 03 00:57:14.065688 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:14.065651 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8387faeb-4fb0-443f-a020-6fef897a16da" path="/var/lib/kubelet/pods/8387faeb-4fb0-443f-a020-6fef897a16da/volumes" Nov 03 00:57:14.066002 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:14.065986 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b61b4cee-dl5tp"] Nov 03 00:57:14.066071 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:14.066017 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b61b4cee-dl5tp"] Nov 03 00:57:14.288715 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:14.288679 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-54df68-59cb67b865-9d886" Nov 03 00:57:14.600040 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:14.599953 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-54df68-59cb67b865-9d886"] Nov 03 00:57:14.702982 ip-10-0-133-217 kubenswrapper[4421]: E1103 00:57:14.702950 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 00:57:14.990888 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:14.990790 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-54df68-59cb67b865-9d886" event={"ID":"fc0615b0-2580-4c26-a028-616bc6042cc3","Type":"ContainerStarted","Data":"7167c61602a6c290b365a586ea4e2a66b43d376ebc172fa8b7fc25e2e5a39657"} Nov 03 00:57:14.990888 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:14.990858 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-54df68-59cb67b865-9d886" event={"ID":"fc0615b0-2580-4c26-a028-616bc6042cc3","Type":"ContainerStarted","Data":"9a666883f95735ace5efff34d9804a6071ac12b40f737a35a6e16161e3d9ef77"} Nov 03 00:57:14.990888 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:14.990887 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-54df68-59cb67b865-9d886" Nov 03 00:57:15.010162 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:15.010101 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-54df68-59cb67b865-9d886" podStartSLOduration=2.010083235 podStartE2EDuration="2.010083235s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:57:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:57:15.006483947 +0000 UTC m=+977.177487045" watchObservedRunningTime="2026-11-03 00:57:15.010083235 +0000 UTC m=+977.181086330" Nov 03 00:57:15.999298 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:15.993608 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="697feaa8-37f3-4292-8fef-2006acc658ff" path="/var/lib/kubelet/pods/697feaa8-37f3-4292-8fef-2006acc658ff/volumes" Nov 03 00:57:16.197297 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:16.175884 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-65e5ec-openshift-default-84ff8668f-kjxr2"] Nov 03 00:57:16.197297 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:16.176143 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-65e5ec-openshift-default-84ff8668f-kjxr2" podUID="97f665ff-bb78-4685-b51b-5c791e529d17" containerName="istio-proxy" containerID="cri-o://3efe8d913c3741324b7859cc51bca738bf17789bc1505fc9b1b4973b7fdb1c3d" gracePeriod=30 Nov 03 00:57:19.256327 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:19.252868 4421 generic.go:356] "Generic (PLEG): container finished" podID="97f665ff-bb78-4685-b51b-5c791e529d17" containerID="3efe8d913c3741324b7859cc51bca738bf17789bc1505fc9b1b4973b7fdb1c3d" exitCode=0 Nov 03 00:57:19.256327 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:19.253174 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-65e5ec-openshift-default-84ff8668f-kjxr2" event={"ID":"97f665ff-bb78-4685-b51b-5c791e529d17","Type":"ContainerDied","Data":"3efe8d913c3741324b7859cc51bca738bf17789bc1505fc9b1b4973b7fdb1c3d"} Nov 03 00:57:19.302371 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:19.302336 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-65e5ec-openshift-default-84ff8668f-kjxr2" Nov 03 00:57:19.400803 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:19.400772 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97f665ff-bb78-4685-b51b-5c791e529d17-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97f665ff-bb78-4685-b51b-5c791e529d17-istio-envoy\") pod \"97f665ff-bb78-4685-b51b-5c791e529d17\" (UID: \"97f665ff-bb78-4685-b51b-5c791e529d17\") " Nov 03 00:57:19.400982 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:19.400815 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97f665ff-bb78-4685-b51b-5c791e529d17-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97f665ff-bb78-4685-b51b-5c791e529d17-workload-socket\") pod \"97f665ff-bb78-4685-b51b-5c791e529d17\" (UID: \"97f665ff-bb78-4685-b51b-5c791e529d17\") " Nov 03 00:57:19.400982 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:19.400871 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97f665ff-bb78-4685-b51b-5c791e529d17-kube-api-access-sd6dk\" (UniqueName: \"kubernetes.io/projected/97f665ff-bb78-4685-b51b-5c791e529d17-kube-api-access-sd6dk\") pod \"97f665ff-bb78-4685-b51b-5c791e529d17\" (UID: \"97f665ff-bb78-4685-b51b-5c791e529d17\") " Nov 03 00:57:19.400982 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:19.400931 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97f665ff-bb78-4685-b51b-5c791e529d17-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97f665ff-bb78-4685-b51b-5c791e529d17-workload-certs\") pod \"97f665ff-bb78-4685-b51b-5c791e529d17\" (UID: \"97f665ff-bb78-4685-b51b-5c791e529d17\") " Nov 03 00:57:19.400982 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:19.400958 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97f665ff-bb78-4685-b51b-5c791e529d17-istio-token\" (UniqueName: \"kubernetes.io/projected/97f665ff-bb78-4685-b51b-5c791e529d17-istio-token\") pod \"97f665ff-bb78-4685-b51b-5c791e529d17\" (UID: \"97f665ff-bb78-4685-b51b-5c791e529d17\") " Nov 03 00:57:19.401350 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:19.401036 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97f665ff-bb78-4685-b51b-5c791e529d17-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97f665ff-bb78-4685-b51b-5c791e529d17-istiod-ca-cert\") pod \"97f665ff-bb78-4685-b51b-5c791e529d17\" (UID: \"97f665ff-bb78-4685-b51b-5c791e529d17\") " Nov 03 00:57:19.401350 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:19.401135 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/97f665ff-bb78-4685-b51b-5c791e529d17-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97f665ff-bb78-4685-b51b-5c791e529d17-istio-podinfo\") pod \"97f665ff-bb78-4685-b51b-5c791e529d17\" (UID: \"97f665ff-bb78-4685-b51b-5c791e529d17\") " Nov 03 00:57:19.401350 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:19.401177 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97f665ff-bb78-4685-b51b-5c791e529d17-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97f665ff-bb78-4685-b51b-5c791e529d17-credential-socket\") pod \"97f665ff-bb78-4685-b51b-5c791e529d17\" (UID: \"97f665ff-bb78-4685-b51b-5c791e529d17\") " Nov 03 00:57:19.401350 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:19.401233 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97f665ff-bb78-4685-b51b-5c791e529d17-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97f665ff-bb78-4685-b51b-5c791e529d17-istio-data\") pod \"97f665ff-bb78-4685-b51b-5c791e529d17\" (UID: \"97f665ff-bb78-4685-b51b-5c791e529d17\") " Nov 03 00:57:19.401350 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:19.401247 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97f665ff-bb78-4685-b51b-5c791e529d17-workload-certs" pod "97f665ff-bb78-4685-b51b-5c791e529d17" (UID: "97f665ff-bb78-4685-b51b-5c791e529d17"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:57:19.401732 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:19.401700 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97f665ff-bb78-4685-b51b-5c791e529d17-credential-socket" pod "97f665ff-bb78-4685-b51b-5c791e529d17" (UID: "97f665ff-bb78-4685-b51b-5c791e529d17"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:57:19.401968 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:19.401947 4421 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97f665ff-bb78-4685-b51b-5c791e529d17-workload-certs\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:57:19.402086 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:19.402069 4421 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97f665ff-bb78-4685-b51b-5c791e529d17-credential-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:57:19.402086 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:19.401998 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97f665ff-bb78-4685-b51b-5c791e529d17-istio-data" pod "97f665ff-bb78-4685-b51b-5c791e529d17" (UID: "97f665ff-bb78-4685-b51b-5c791e529d17"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:57:19.402467 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:19.402445 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97f665ff-bb78-4685-b51b-5c791e529d17-istiod-ca-cert" pod "97f665ff-bb78-4685-b51b-5c791e529d17" (UID: "97f665ff-bb78-4685-b51b-5c791e529d17"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:57:19.402666 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:19.402642 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97f665ff-bb78-4685-b51b-5c791e529d17-workload-socket" pod "97f665ff-bb78-4685-b51b-5c791e529d17" (UID: "97f665ff-bb78-4685-b51b-5c791e529d17"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:57:19.403402 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:19.403375 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97f665ff-bb78-4685-b51b-5c791e529d17-kube-api-access-sd6dk" pod "97f665ff-bb78-4685-b51b-5c791e529d17" (UID: "97f665ff-bb78-4685-b51b-5c791e529d17"). InnerVolumeSpecName "kube-api-access-sd6dk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:57:19.403568 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:19.403543 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97f665ff-bb78-4685-b51b-5c791e529d17-istio-envoy" pod "97f665ff-bb78-4685-b51b-5c791e529d17" (UID: "97f665ff-bb78-4685-b51b-5c791e529d17"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:57:19.403685 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:19.403566 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/97f665ff-bb78-4685-b51b-5c791e529d17-istio-podinfo" pod "97f665ff-bb78-4685-b51b-5c791e529d17" (UID: "97f665ff-bb78-4685-b51b-5c791e529d17"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:57:19.404512 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:19.404491 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97f665ff-bb78-4685-b51b-5c791e529d17-istio-token" pod "97f665ff-bb78-4685-b51b-5c791e529d17" (UID: "97f665ff-bb78-4685-b51b-5c791e529d17"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:57:19.512654 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:19.506772 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sd6dk\" (UniqueName: \"kubernetes.io/projected/97f665ff-bb78-4685-b51b-5c791e529d17-kube-api-access-sd6dk\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:57:19.512654 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:19.506815 4421 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97f665ff-bb78-4685-b51b-5c791e529d17-istio-token\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:57:19.512654 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:19.506830 4421 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97f665ff-bb78-4685-b51b-5c791e529d17-istiod-ca-cert\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:57:19.512654 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:19.506843 4421 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97f665ff-bb78-4685-b51b-5c791e529d17-istio-podinfo\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:57:19.512654 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:19.506856 4421 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97f665ff-bb78-4685-b51b-5c791e529d17-istio-data\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:57:19.512654 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:19.506870 4421 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97f665ff-bb78-4685-b51b-5c791e529d17-istio-envoy\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:57:19.512654 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:19.506883 4421 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97f665ff-bb78-4685-b51b-5c791e529d17-workload-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:57:20.262361 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:20.262334 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-65e5ec-openshift-default-84ff8668f-kjxr2" Nov 03 00:57:20.262769 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:20.262337 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-65e5ec-openshift-default-84ff8668f-kjxr2" event={"ID":"97f665ff-bb78-4685-b51b-5c791e529d17","Type":"ContainerDied","Data":"6391493df0de83724c7865a44bc9205a91d7fcafe85adda38ba7273eab6f554d"} Nov 03 00:57:20.262769 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:20.262469 4421 scope.go:122] "RemoveContainer" containerID="3efe8d913c3741324b7859cc51bca738bf17789bc1505fc9b1b4973b7fdb1c3d" Nov 03 00:57:20.283842 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:20.283811 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-65e5ec-openshift-default-84ff8668f-kjxr2"] Nov 03 00:57:20.294458 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:20.294427 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-65e5ec-openshift-default-84ff8668f-kjxr2"] Nov 03 00:57:21.858064 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:21.858029 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1ea8b2e-k2nfc"] Nov 03 00:57:21.858906 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:21.858884 4421 cpu_manager.go:414] "containerMap: removing container" podUID="97f665ff-bb78-4685-b51b-5c791e529d17" containerName="istio-proxy" Nov 03 00:57:21.859047 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:21.858907 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97f665ff-bb78-4685-b51b-5c791e529d17" containerName="istio-proxy" Nov 03 00:57:21.859111 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:21.859047 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="97f665ff-bb78-4685-b51b-5c791e529d17" containerName="istio-proxy" Nov 03 00:57:21.862683 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:21.862662 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1ea8b2e-k2nfc" Nov 03 00:57:21.869746 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:21.869720 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1ea8b2e-k2nfc"] Nov 03 00:57:21.869855 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:21.869752 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:57:21.869855 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:21.869816 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 00:57:21.869855 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:21.869822 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6q4xp\"" Nov 03 00:57:21.869855 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:21.869833 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 00:57:21.963972 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:21.963938 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db84ab91-9005-4838-9c29-baad4bb6739b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d1ea8b2e-k2nfc\" (UID: \"db84ab91-9005-4838-9c29-baad4bb6739b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1ea8b2e-k2nfc" Nov 03 00:57:21.964146 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:21.964002 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db84ab91-9005-4838-9c29-baad4bb6739b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d1ea8b2e-k2nfc\" (UID: \"db84ab91-9005-4838-9c29-baad4bb6739b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1ea8b2e-k2nfc" Nov 03 00:57:21.964146 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:21.964137 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9898c\" (UniqueName: \"kubernetes.io/projected/db84ab91-9005-4838-9c29-baad4bb6739b-kube-api-access-9898c\") pod \"maas-api-revoke-sub-d1ea8b2e-k2nfc\" (UID: \"db84ab91-9005-4838-9c29-baad4bb6739b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1ea8b2e-k2nfc" Nov 03 00:57:21.979583 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:21.979549 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97f665ff-bb78-4685-b51b-5c791e529d17" path="/var/lib/kubelet/pods/97f665ff-bb78-4685-b51b-5c791e529d17/volumes" Nov 03 00:57:22.065012 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:22.064973 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db84ab91-9005-4838-9c29-baad4bb6739b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d1ea8b2e-k2nfc\" (UID: \"db84ab91-9005-4838-9c29-baad4bb6739b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1ea8b2e-k2nfc" Nov 03 00:57:22.065182 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:22.065054 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db84ab91-9005-4838-9c29-baad4bb6739b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d1ea8b2e-k2nfc\" (UID: \"db84ab91-9005-4838-9c29-baad4bb6739b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1ea8b2e-k2nfc" Nov 03 00:57:22.065182 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:22.065134 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9898c\" (UniqueName: \"kubernetes.io/projected/db84ab91-9005-4838-9c29-baad4bb6739b-kube-api-access-9898c\") pod \"maas-api-revoke-sub-d1ea8b2e-k2nfc\" (UID: \"db84ab91-9005-4838-9c29-baad4bb6739b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1ea8b2e-k2nfc" Nov 03 00:57:22.065817 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:22.065787 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db84ab91-9005-4838-9c29-baad4bb6739b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d1ea8b2e-k2nfc\" (UID: \"db84ab91-9005-4838-9c29-baad4bb6739b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1ea8b2e-k2nfc" Nov 03 00:57:22.067538 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:22.067512 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db84ab91-9005-4838-9c29-baad4bb6739b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d1ea8b2e-k2nfc\" (UID: \"db84ab91-9005-4838-9c29-baad4bb6739b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1ea8b2e-k2nfc" Nov 03 00:57:22.074227 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:22.074203 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9898c\" (UniqueName: \"kubernetes.io/projected/db84ab91-9005-4838-9c29-baad4bb6739b-kube-api-access-9898c\") pod \"maas-api-revoke-sub-d1ea8b2e-k2nfc\" (UID: \"db84ab91-9005-4838-9c29-baad4bb6739b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1ea8b2e-k2nfc" Nov 03 00:57:22.182491 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:22.182381 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1ea8b2e-k2nfc" Nov 03 00:57:22.469408 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:57:22.469377 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb84ab91_9005_4838_9c29_baad4bb6739b.slice/crio-db2bd34e5b244d7872b2e5734d276b697e949e1bca762a8c68aed47d1d9484ac WatchSource:0}: Error finding container db2bd34e5b244d7872b2e5734d276b697e949e1bca762a8c68aed47d1d9484ac: Status 404 returned error can't find the container with id db2bd34e5b244d7872b2e5734d276b697e949e1bca762a8c68aed47d1d9484ac Nov 03 00:57:22.470328 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:22.470301 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1ea8b2e-k2nfc"] Nov 03 00:57:23.289839 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:23.289801 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1ea8b2e-k2nfc" event={"ID":"db84ab91-9005-4838-9c29-baad4bb6739b","Type":"ContainerStarted","Data":"bee8bf0fd1c08956484a5c09269610a4279268366e4a1ee82cfd9e93747c6531"} Nov 03 00:57:23.290237 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:23.289876 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1ea8b2e-k2nfc" event={"ID":"db84ab91-9005-4838-9c29-baad4bb6739b","Type":"ContainerStarted","Data":"db2bd34e5b244d7872b2e5734d276b697e949e1bca762a8c68aed47d1d9484ac"} Nov 03 00:57:23.305721 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:23.305671 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1ea8b2e-k2nfc" podStartSLOduration=2.305658605 podStartE2EDuration="2.305658605s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:57:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:57:23.304349701 +0000 UTC m=+985.475352796" watchObservedRunningTime="2026-11-03 00:57:23.305658605 +0000 UTC m=+985.476661943" Nov 03 00:57:24.299152 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:24.299107 4421 generic.go:356] "Generic (PLEG): container finished" podID="db84ab91-9005-4838-9c29-baad4bb6739b" containerID="bee8bf0fd1c08956484a5c09269610a4279268366e4a1ee82cfd9e93747c6531" exitCode=0 Nov 03 00:57:24.299587 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:24.299195 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1ea8b2e-k2nfc" event={"ID":"db84ab91-9005-4838-9c29-baad4bb6739b","Type":"ContainerDied","Data":"bee8bf0fd1c08956484a5c09269610a4279268366e4a1ee82cfd9e93747c6531"} Nov 03 00:57:25.474138 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:25.474114 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1ea8b2e-k2nfc" Nov 03 00:57:25.529095 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:25.529050 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db84ab91-9005-4838-9c29-baad4bb6739b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db84ab91-9005-4838-9c29-baad4bb6739b-maas-api-xks-ca\") pod \"db84ab91-9005-4838-9c29-baad4bb6739b\" (UID: \"db84ab91-9005-4838-9c29-baad4bb6739b\") " Nov 03 00:57:25.529322 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:25.529177 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db84ab91-9005-4838-9c29-baad4bb6739b-kube-api-access-9898c\" (UniqueName: \"kubernetes.io/projected/db84ab91-9005-4838-9c29-baad4bb6739b-kube-api-access-9898c\") pod \"db84ab91-9005-4838-9c29-baad4bb6739b\" (UID: \"db84ab91-9005-4838-9c29-baad4bb6739b\") " Nov 03 00:57:25.529322 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:25.529234 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db84ab91-9005-4838-9c29-baad4bb6739b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db84ab91-9005-4838-9c29-baad4bb6739b-maas-api-openshift-ca\") pod \"db84ab91-9005-4838-9c29-baad4bb6739b\" (UID: \"db84ab91-9005-4838-9c29-baad4bb6739b\") " Nov 03 00:57:25.530366 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:25.530328 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db84ab91-9005-4838-9c29-baad4bb6739b-maas-api-openshift-ca" pod "db84ab91-9005-4838-9c29-baad4bb6739b" (UID: "db84ab91-9005-4838-9c29-baad4bb6739b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:57:25.638329 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:25.635563 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db84ab91-9005-4838-9c29-baad4bb6739b-maas-api-openshift-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:57:25.639541 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:25.639486 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db84ab91-9005-4838-9c29-baad4bb6739b-kube-api-access-9898c" pod "db84ab91-9005-4838-9c29-baad4bb6739b" (UID: "db84ab91-9005-4838-9c29-baad4bb6739b"). InnerVolumeSpecName "kube-api-access-9898c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:57:25.650439 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:25.650385 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db84ab91-9005-4838-9c29-baad4bb6739b-maas-api-xks-ca" pod "db84ab91-9005-4838-9c29-baad4bb6739b" (UID: "db84ab91-9005-4838-9c29-baad4bb6739b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:57:25.736924 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:25.736883 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db84ab91-9005-4838-9c29-baad4bb6739b-maas-api-xks-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:57:25.737193 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:25.737165 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9898c\" (UniqueName: \"kubernetes.io/projected/db84ab91-9005-4838-9c29-baad4bb6739b-kube-api-access-9898c\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:57:26.009522 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:26.009480 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-54df68-59cb67b865-9d886" Nov 03 00:57:26.361808 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:26.361769 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1ea8b2e-k2nfc" event={"ID":"db84ab91-9005-4838-9c29-baad4bb6739b","Type":"ContainerDied","Data":"db2bd34e5b244d7872b2e5734d276b697e949e1bca762a8c68aed47d1d9484ac"} Nov 03 00:57:26.362105 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:26.362084 4421 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="db2bd34e5b244d7872b2e5734d276b697e949e1bca762a8c68aed47d1d9484ac" Nov 03 00:57:26.362319 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:26.362299 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1ea8b2e-k2nfc" Nov 03 00:57:29.024389 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:29.024342 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39930c39-jcztx"] Nov 03 00:57:29.029546 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:29.029513 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39930c39-jcztx"] Nov 03 00:57:29.980230 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:29.980196 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d73525f8-b528-4522-9268-b170b3feb79d" path="/var/lib/kubelet/pods/d73525f8-b528-4522-9268-b170b3feb79d/volumes" Nov 03 00:57:30.477445 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:30.477408 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fac988dd-jkdwx"] Nov 03 00:57:30.478138 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:30.478121 4421 cpu_manager.go:414] "containerMap: removing container" podUID="db84ab91-9005-4838-9c29-baad4bb6739b" containerName="revoke-keys" Nov 03 00:57:30.478138 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:30.478139 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db84ab91-9005-4838-9c29-baad4bb6739b" containerName="revoke-keys" Nov 03 00:57:30.478338 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:30.478320 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="db84ab91-9005-4838-9c29-baad4bb6739b" containerName="revoke-keys" Nov 03 00:57:30.481346 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:30.481327 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fac988dd-jkdwx" Nov 03 00:57:30.485552 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:30.485518 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 00:57:30.486952 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:30.486931 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:57:30.487096 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:30.486982 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 00:57:30.487285 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:30.487255 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6q4xp\"" Nov 03 00:57:30.490249 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:30.490227 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fac988dd-jkdwx"] Nov 03 00:57:30.586509 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:30.586469 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd1b22e9-50b0-4359-ae9b-7302bbc4cff1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fac988dd-jkdwx\" (UID: \"fd1b22e9-50b0-4359-ae9b-7302bbc4cff1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fac988dd-jkdwx" Nov 03 00:57:30.586680 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:30.586532 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gz9j6\" (UniqueName: \"kubernetes.io/projected/fd1b22e9-50b0-4359-ae9b-7302bbc4cff1-kube-api-access-gz9j6\") pod \"maas-api-revoke-sub-fac988dd-jkdwx\" (UID: \"fd1b22e9-50b0-4359-ae9b-7302bbc4cff1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fac988dd-jkdwx" Nov 03 00:57:30.586797 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:30.586773 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd1b22e9-50b0-4359-ae9b-7302bbc4cff1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fac988dd-jkdwx\" (UID: \"fd1b22e9-50b0-4359-ae9b-7302bbc4cff1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fac988dd-jkdwx" Nov 03 00:57:30.693473 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:30.693342 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd1b22e9-50b0-4359-ae9b-7302bbc4cff1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fac988dd-jkdwx\" (UID: \"fd1b22e9-50b0-4359-ae9b-7302bbc4cff1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fac988dd-jkdwx" Nov 03 00:57:30.694180 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:30.694146 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd1b22e9-50b0-4359-ae9b-7302bbc4cff1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fac988dd-jkdwx\" (UID: \"fd1b22e9-50b0-4359-ae9b-7302bbc4cff1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fac988dd-jkdwx" Nov 03 00:57:30.694495 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:30.694457 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gz9j6\" (UniqueName: \"kubernetes.io/projected/fd1b22e9-50b0-4359-ae9b-7302bbc4cff1-kube-api-access-gz9j6\") pod \"maas-api-revoke-sub-fac988dd-jkdwx\" (UID: \"fd1b22e9-50b0-4359-ae9b-7302bbc4cff1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fac988dd-jkdwx" Nov 03 00:57:30.698049 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:30.698021 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd1b22e9-50b0-4359-ae9b-7302bbc4cff1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fac988dd-jkdwx\" (UID: \"fd1b22e9-50b0-4359-ae9b-7302bbc4cff1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fac988dd-jkdwx" Nov 03 00:57:30.727929 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:30.727843 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd1b22e9-50b0-4359-ae9b-7302bbc4cff1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fac988dd-jkdwx\" (UID: \"fd1b22e9-50b0-4359-ae9b-7302bbc4cff1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fac988dd-jkdwx" Nov 03 00:57:30.734378 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:30.734350 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gz9j6\" (UniqueName: \"kubernetes.io/projected/fd1b22e9-50b0-4359-ae9b-7302bbc4cff1-kube-api-access-gz9j6\") pod \"maas-api-revoke-sub-fac988dd-jkdwx\" (UID: \"fd1b22e9-50b0-4359-ae9b-7302bbc4cff1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fac988dd-jkdwx" Nov 03 00:57:30.812183 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:30.812144 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fac988dd-jkdwx" Nov 03 00:57:31.023688 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:31.023656 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-668b4d3c-r2qnq"] Nov 03 00:57:31.028012 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:31.027949 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-668b4d3c-r2qnq"] Nov 03 00:57:31.093417 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:31.093388 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fac988dd-jkdwx"] Nov 03 00:57:31.093516 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:57:31.093413 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd1b22e9_50b0_4359_ae9b_7302bbc4cff1.slice/crio-643fa9c2194eb84c3a509b255ded87fe5c67600cb9e0271bb39974357bc370d8 WatchSource:0}: Error finding container 643fa9c2194eb84c3a509b255ded87fe5c67600cb9e0271bb39974357bc370d8: Status 404 returned error can't find the container with id 643fa9c2194eb84c3a509b255ded87fe5c67600cb9e0271bb39974357bc370d8 Nov 03 00:57:31.437937 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:31.437875 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fac988dd-jkdwx" event={"ID":"fd1b22e9-50b0-4359-ae9b-7302bbc4cff1","Type":"ContainerStarted","Data":"643fa9c2194eb84c3a509b255ded87fe5c67600cb9e0271bb39974357bc370d8"} Nov 03 00:57:31.987240 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:31.982403 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8228b121-a23c-4757-8617-69b13be7edd6" path="/var/lib/kubelet/pods/8228b121-a23c-4757-8617-69b13be7edd6/volumes" Nov 03 00:57:32.448916 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:32.448882 4421 generic.go:356] "Generic (PLEG): container finished" podID="fd1b22e9-50b0-4359-ae9b-7302bbc4cff1" containerID="313f78e952d8a50e4f99f8c9af1526aa531bbadb03b67d5cd4744b498422217a" exitCode=0 Nov 03 00:57:32.449113 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:32.448941 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fac988dd-jkdwx" event={"ID":"fd1b22e9-50b0-4359-ae9b-7302bbc4cff1","Type":"ContainerDied","Data":"313f78e952d8a50e4f99f8c9af1526aa531bbadb03b67d5cd4744b498422217a"} Nov 03 00:57:33.151512 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:33.151479 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-a2d8e5-openshift-default-85655d8485-f4q8h"] Nov 03 00:57:33.156552 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:33.156524 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-a2d8e5-openshift-default-85655d8485-f4q8h" Nov 03 00:57:33.163469 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:33.163434 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-a2d8e5-openshift-default-85655d8485-f4q8h"] Nov 03 00:57:33.167119 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:33.167088 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-a2d8e5-openshift-default-dockercfg-prbqw\"" Nov 03 00:57:33.361326 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:33.360563 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0be0024f-24ee-4c93-81ef-8d9b3f2d0b69-istio-token\") pod \"e2e-api-b-a2d8e5-openshift-default-85655d8485-f4q8h\" (UID: \"0be0024f-24ee-4c93-81ef-8d9b3f2d0b69\") " pod="openshift-ingress/e2e-api-b-a2d8e5-openshift-default-85655d8485-f4q8h" Nov 03 00:57:33.361326 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:33.360672 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0be0024f-24ee-4c93-81ef-8d9b3f2d0b69-istio-envoy\") pod \"e2e-api-b-a2d8e5-openshift-default-85655d8485-f4q8h\" (UID: \"0be0024f-24ee-4c93-81ef-8d9b3f2d0b69\") " pod="openshift-ingress/e2e-api-b-a2d8e5-openshift-default-85655d8485-f4q8h" Nov 03 00:57:33.361326 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:33.360699 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0be0024f-24ee-4c93-81ef-8d9b3f2d0b69-credential-socket\") pod \"e2e-api-b-a2d8e5-openshift-default-85655d8485-f4q8h\" (UID: \"0be0024f-24ee-4c93-81ef-8d9b3f2d0b69\") " pod="openshift-ingress/e2e-api-b-a2d8e5-openshift-default-85655d8485-f4q8h" Nov 03 00:57:33.361326 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:33.360757 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0be0024f-24ee-4c93-81ef-8d9b3f2d0b69-workload-certs\") pod \"e2e-api-b-a2d8e5-openshift-default-85655d8485-f4q8h\" (UID: \"0be0024f-24ee-4c93-81ef-8d9b3f2d0b69\") " pod="openshift-ingress/e2e-api-b-a2d8e5-openshift-default-85655d8485-f4q8h" Nov 03 00:57:33.361326 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:33.360817 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0be0024f-24ee-4c93-81ef-8d9b3f2d0b69-workload-socket\") pod \"e2e-api-b-a2d8e5-openshift-default-85655d8485-f4q8h\" (UID: \"0be0024f-24ee-4c93-81ef-8d9b3f2d0b69\") " pod="openshift-ingress/e2e-api-b-a2d8e5-openshift-default-85655d8485-f4q8h" Nov 03 00:57:33.361326 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:33.360849 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0be0024f-24ee-4c93-81ef-8d9b3f2d0b69-istio-data\") pod \"e2e-api-b-a2d8e5-openshift-default-85655d8485-f4q8h\" (UID: \"0be0024f-24ee-4c93-81ef-8d9b3f2d0b69\") " pod="openshift-ingress/e2e-api-b-a2d8e5-openshift-default-85655d8485-f4q8h" Nov 03 00:57:33.361326 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:33.360919 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0be0024f-24ee-4c93-81ef-8d9b3f2d0b69-istiod-ca-cert\") pod \"e2e-api-b-a2d8e5-openshift-default-85655d8485-f4q8h\" (UID: \"0be0024f-24ee-4c93-81ef-8d9b3f2d0b69\") " pod="openshift-ingress/e2e-api-b-a2d8e5-openshift-default-85655d8485-f4q8h" Nov 03 00:57:33.361326 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:33.360953 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpbkw\" (UniqueName: \"kubernetes.io/projected/0be0024f-24ee-4c93-81ef-8d9b3f2d0b69-kube-api-access-lpbkw\") pod \"e2e-api-b-a2d8e5-openshift-default-85655d8485-f4q8h\" (UID: \"0be0024f-24ee-4c93-81ef-8d9b3f2d0b69\") " pod="openshift-ingress/e2e-api-b-a2d8e5-openshift-default-85655d8485-f4q8h" Nov 03 00:57:33.361326 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:33.360988 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0be0024f-24ee-4c93-81ef-8d9b3f2d0b69-istio-podinfo\") pod \"e2e-api-b-a2d8e5-openshift-default-85655d8485-f4q8h\" (UID: \"0be0024f-24ee-4c93-81ef-8d9b3f2d0b69\") " pod="openshift-ingress/e2e-api-b-a2d8e5-openshift-default-85655d8485-f4q8h" Nov 03 00:57:33.462333 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:33.462296 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0be0024f-24ee-4c93-81ef-8d9b3f2d0b69-istio-envoy\") pod \"e2e-api-b-a2d8e5-openshift-default-85655d8485-f4q8h\" (UID: \"0be0024f-24ee-4c93-81ef-8d9b3f2d0b69\") " pod="openshift-ingress/e2e-api-b-a2d8e5-openshift-default-85655d8485-f4q8h" Nov 03 00:57:33.462525 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:33.462344 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0be0024f-24ee-4c93-81ef-8d9b3f2d0b69-credential-socket\") pod \"e2e-api-b-a2d8e5-openshift-default-85655d8485-f4q8h\" (UID: \"0be0024f-24ee-4c93-81ef-8d9b3f2d0b69\") " pod="openshift-ingress/e2e-api-b-a2d8e5-openshift-default-85655d8485-f4q8h" Nov 03 00:57:33.462525 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:33.462426 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0be0024f-24ee-4c93-81ef-8d9b3f2d0b69-workload-certs\") pod \"e2e-api-b-a2d8e5-openshift-default-85655d8485-f4q8h\" (UID: \"0be0024f-24ee-4c93-81ef-8d9b3f2d0b69\") " pod="openshift-ingress/e2e-api-b-a2d8e5-openshift-default-85655d8485-f4q8h" Nov 03 00:57:33.462525 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:33.462506 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0be0024f-24ee-4c93-81ef-8d9b3f2d0b69-workload-socket\") pod \"e2e-api-b-a2d8e5-openshift-default-85655d8485-f4q8h\" (UID: \"0be0024f-24ee-4c93-81ef-8d9b3f2d0b69\") " pod="openshift-ingress/e2e-api-b-a2d8e5-openshift-default-85655d8485-f4q8h" Nov 03 00:57:33.462737 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:33.462542 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0be0024f-24ee-4c93-81ef-8d9b3f2d0b69-istio-data\") pod \"e2e-api-b-a2d8e5-openshift-default-85655d8485-f4q8h\" (UID: \"0be0024f-24ee-4c93-81ef-8d9b3f2d0b69\") " pod="openshift-ingress/e2e-api-b-a2d8e5-openshift-default-85655d8485-f4q8h" Nov 03 00:57:33.462737 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:33.462629 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0be0024f-24ee-4c93-81ef-8d9b3f2d0b69-istiod-ca-cert\") pod \"e2e-api-b-a2d8e5-openshift-default-85655d8485-f4q8h\" (UID: \"0be0024f-24ee-4c93-81ef-8d9b3f2d0b69\") " pod="openshift-ingress/e2e-api-b-a2d8e5-openshift-default-85655d8485-f4q8h" Nov 03 00:57:33.462737 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:33.462666 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpbkw\" (UniqueName: \"kubernetes.io/projected/0be0024f-24ee-4c93-81ef-8d9b3f2d0b69-kube-api-access-lpbkw\") pod \"e2e-api-b-a2d8e5-openshift-default-85655d8485-f4q8h\" (UID: \"0be0024f-24ee-4c93-81ef-8d9b3f2d0b69\") " pod="openshift-ingress/e2e-api-b-a2d8e5-openshift-default-85655d8485-f4q8h" Nov 03 00:57:33.462737 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:33.462701 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0be0024f-24ee-4c93-81ef-8d9b3f2d0b69-istio-podinfo\") pod \"e2e-api-b-a2d8e5-openshift-default-85655d8485-f4q8h\" (UID: \"0be0024f-24ee-4c93-81ef-8d9b3f2d0b69\") " pod="openshift-ingress/e2e-api-b-a2d8e5-openshift-default-85655d8485-f4q8h" Nov 03 00:57:33.462954 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:33.462820 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0be0024f-24ee-4c93-81ef-8d9b3f2d0b69-istio-token\") pod \"e2e-api-b-a2d8e5-openshift-default-85655d8485-f4q8h\" (UID: \"0be0024f-24ee-4c93-81ef-8d9b3f2d0b69\") " pod="openshift-ingress/e2e-api-b-a2d8e5-openshift-default-85655d8485-f4q8h" Nov 03 00:57:33.463716 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:33.463649 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0be0024f-24ee-4c93-81ef-8d9b3f2d0b69-istio-data\") pod \"e2e-api-b-a2d8e5-openshift-default-85655d8485-f4q8h\" (UID: \"0be0024f-24ee-4c93-81ef-8d9b3f2d0b69\") " pod="openshift-ingress/e2e-api-b-a2d8e5-openshift-default-85655d8485-f4q8h" Nov 03 00:57:33.463879 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:33.463693 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0be0024f-24ee-4c93-81ef-8d9b3f2d0b69-workload-socket\") pod \"e2e-api-b-a2d8e5-openshift-default-85655d8485-f4q8h\" (UID: \"0be0024f-24ee-4c93-81ef-8d9b3f2d0b69\") " pod="openshift-ingress/e2e-api-b-a2d8e5-openshift-default-85655d8485-f4q8h" Nov 03 00:57:33.464708 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:33.464613 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0be0024f-24ee-4c93-81ef-8d9b3f2d0b69-workload-certs\") pod \"e2e-api-b-a2d8e5-openshift-default-85655d8485-f4q8h\" (UID: \"0be0024f-24ee-4c93-81ef-8d9b3f2d0b69\") " pod="openshift-ingress/e2e-api-b-a2d8e5-openshift-default-85655d8485-f4q8h" Nov 03 00:57:33.464708 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:33.464687 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0be0024f-24ee-4c93-81ef-8d9b3f2d0b69-credential-socket\") pod \"e2e-api-b-a2d8e5-openshift-default-85655d8485-f4q8h\" (UID: \"0be0024f-24ee-4c93-81ef-8d9b3f2d0b69\") " pod="openshift-ingress/e2e-api-b-a2d8e5-openshift-default-85655d8485-f4q8h" Nov 03 00:57:33.464941 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:33.464919 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0be0024f-24ee-4c93-81ef-8d9b3f2d0b69-istiod-ca-cert\") pod \"e2e-api-b-a2d8e5-openshift-default-85655d8485-f4q8h\" (UID: \"0be0024f-24ee-4c93-81ef-8d9b3f2d0b69\") " pod="openshift-ingress/e2e-api-b-a2d8e5-openshift-default-85655d8485-f4q8h" Nov 03 00:57:33.466176 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:33.466160 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0be0024f-24ee-4c93-81ef-8d9b3f2d0b69-istio-envoy\") pod \"e2e-api-b-a2d8e5-openshift-default-85655d8485-f4q8h\" (UID: \"0be0024f-24ee-4c93-81ef-8d9b3f2d0b69\") " pod="openshift-ingress/e2e-api-b-a2d8e5-openshift-default-85655d8485-f4q8h" Nov 03 00:57:33.469979 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:33.469950 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0be0024f-24ee-4c93-81ef-8d9b3f2d0b69-istio-podinfo\") pod \"e2e-api-b-a2d8e5-openshift-default-85655d8485-f4q8h\" (UID: \"0be0024f-24ee-4c93-81ef-8d9b3f2d0b69\") " pod="openshift-ingress/e2e-api-b-a2d8e5-openshift-default-85655d8485-f4q8h" Nov 03 00:57:33.472256 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:33.472233 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpbkw\" (UniqueName: \"kubernetes.io/projected/0be0024f-24ee-4c93-81ef-8d9b3f2d0b69-kube-api-access-lpbkw\") pod \"e2e-api-b-a2d8e5-openshift-default-85655d8485-f4q8h\" (UID: \"0be0024f-24ee-4c93-81ef-8d9b3f2d0b69\") " pod="openshift-ingress/e2e-api-b-a2d8e5-openshift-default-85655d8485-f4q8h" Nov 03 00:57:33.482773 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:33.482744 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0be0024f-24ee-4c93-81ef-8d9b3f2d0b69-istio-token\") pod \"e2e-api-b-a2d8e5-openshift-default-85655d8485-f4q8h\" (UID: \"0be0024f-24ee-4c93-81ef-8d9b3f2d0b69\") " pod="openshift-ingress/e2e-api-b-a2d8e5-openshift-default-85655d8485-f4q8h" Nov 03 00:57:33.698822 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:33.698797 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fac988dd-jkdwx" Nov 03 00:57:33.795327 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:33.788583 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-a2d8e5-openshift-default-85655d8485-f4q8h" Nov 03 00:57:33.883296 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:33.874364 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd1b22e9-50b0-4359-ae9b-7302bbc4cff1-kube-api-access-gz9j6\" (UniqueName: \"kubernetes.io/projected/fd1b22e9-50b0-4359-ae9b-7302bbc4cff1-kube-api-access-gz9j6\") pod \"fd1b22e9-50b0-4359-ae9b-7302bbc4cff1\" (UID: \"fd1b22e9-50b0-4359-ae9b-7302bbc4cff1\") " Nov 03 00:57:33.883296 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:33.874602 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd1b22e9-50b0-4359-ae9b-7302bbc4cff1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd1b22e9-50b0-4359-ae9b-7302bbc4cff1-maas-api-xks-ca\") pod \"fd1b22e9-50b0-4359-ae9b-7302bbc4cff1\" (UID: \"fd1b22e9-50b0-4359-ae9b-7302bbc4cff1\") " Nov 03 00:57:33.883296 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:33.874650 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd1b22e9-50b0-4359-ae9b-7302bbc4cff1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd1b22e9-50b0-4359-ae9b-7302bbc4cff1-maas-api-openshift-ca\") pod \"fd1b22e9-50b0-4359-ae9b-7302bbc4cff1\" (UID: \"fd1b22e9-50b0-4359-ae9b-7302bbc4cff1\") " Nov 03 00:57:33.883296 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:33.881545 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd1b22e9-50b0-4359-ae9b-7302bbc4cff1-maas-api-openshift-ca" pod "fd1b22e9-50b0-4359-ae9b-7302bbc4cff1" (UID: "fd1b22e9-50b0-4359-ae9b-7302bbc4cff1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:57:33.910903 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:33.884671 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd1b22e9-50b0-4359-ae9b-7302bbc4cff1-kube-api-access-gz9j6" pod "fd1b22e9-50b0-4359-ae9b-7302bbc4cff1" (UID: "fd1b22e9-50b0-4359-ae9b-7302bbc4cff1"). InnerVolumeSpecName "kube-api-access-gz9j6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:57:33.916300 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:33.915003 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd1b22e9-50b0-4359-ae9b-7302bbc4cff1-maas-api-xks-ca" pod "fd1b22e9-50b0-4359-ae9b-7302bbc4cff1" (UID: "fd1b22e9-50b0-4359-ae9b-7302bbc4cff1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:57:33.989480 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:33.989442 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd1b22e9-50b0-4359-ae9b-7302bbc4cff1-maas-api-xks-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:57:33.989672 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:33.989493 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd1b22e9-50b0-4359-ae9b-7302bbc4cff1-maas-api-openshift-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:57:33.989672 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:33.989510 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gz9j6\" (UniqueName: \"kubernetes.io/projected/fd1b22e9-50b0-4359-ae9b-7302bbc4cff1-kube-api-access-gz9j6\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:57:34.074862 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:34.074773 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dd03e395-pbnvh"] Nov 03 00:57:34.079417 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:34.079383 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dd03e395-pbnvh"] Nov 03 00:57:34.284035 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:34.283970 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:57:34.284577 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:34.284161 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:57:34.284577 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:34.284265 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:57:34.297311 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:34.294716 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-a2d8e5-openshift-default-85655d8485-f4q8h"] Nov 03 00:57:34.473113 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:34.473075 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a2d8e5-openshift-default-85655d8485-f4q8h" event={"ID":"0be0024f-24ee-4c93-81ef-8d9b3f2d0b69","Type":"ContainerStarted","Data":"167b6ecaafc9a01a8ed6a0bdf239b0f6094c4caaec4e7a06831e0eaf9ec7f37b"} Nov 03 00:57:34.473338 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:34.473143 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a2d8e5-openshift-default-85655d8485-f4q8h" event={"ID":"0be0024f-24ee-4c93-81ef-8d9b3f2d0b69","Type":"ContainerStarted","Data":"6ea138688d6cfaa73cfc9135ac6c4419614aa94366e2971e5b6a0c08f1841c1a"} Nov 03 00:57:34.474979 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:34.474957 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fac988dd-jkdwx" Nov 03 00:57:34.474979 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:34.474966 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fac988dd-jkdwx" event={"ID":"fd1b22e9-50b0-4359-ae9b-7302bbc4cff1","Type":"ContainerDied","Data":"643fa9c2194eb84c3a509b255ded87fe5c67600cb9e0271bb39974357bc370d8"} Nov 03 00:57:34.475190 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:34.475013 4421 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="643fa9c2194eb84c3a509b255ded87fe5c67600cb9e0271bb39974357bc370d8" Nov 03 00:57:34.499642 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:34.499577 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-a2d8e5-openshift-default-85655d8485-f4q8h" podStartSLOduration=1.499560536 podStartE2EDuration="1.499560536s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:57:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:57:34.495450342 +0000 UTC m=+996.666453439" watchObservedRunningTime="2026-11-03 00:57:34.499560536 +0000 UTC m=+996.670563699" Nov 03 00:57:34.907111 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:34.907072 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-a2d8e5-openshift-default-85655d8485-f4q8h" Nov 03 00:57:35.912495 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:35.912462 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-a2d8e5-openshift-default-85655d8485-f4q8h" Nov 03 00:57:35.980304 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:35.980255 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0fc241d1-cb3f-4b97-af77-e850e30abcc7" path="/var/lib/kubelet/pods/0fc241d1-cb3f-4b97-af77-e850e30abcc7/volumes" Nov 03 00:57:36.490265 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:36.490235 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-a2d8e5-openshift-default-85655d8485-f4q8h" Nov 03 00:57:36.491191 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:36.491167 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-a2d8e5-openshift-default-85655d8485-f4q8h" Nov 03 00:57:37.985483 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:37.985400 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a2d8e5-f6bfcf94c-4jbsh"] Nov 03 00:57:37.986360 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:37.986336 4421 cpu_manager.go:414] "containerMap: removing container" podUID="fd1b22e9-50b0-4359-ae9b-7302bbc4cff1" containerName="revoke-keys" Nov 03 00:57:37.986360 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:37.986359 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd1b22e9-50b0-4359-ae9b-7302bbc4cff1" containerName="revoke-keys" Nov 03 00:57:37.986636 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:37.986618 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd1b22e9-50b0-4359-ae9b-7302bbc4cff1" containerName="revoke-keys" Nov 03 00:57:37.995353 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:37.995326 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a2d8e5-f6bfcf94c-4jbsh"] Nov 03 00:57:37.995480 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:37.995463 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a2d8e5-f6bfcf94c-4jbsh" Nov 03 00:57:38.000736 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:38.000709 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-a2d8e5-dockercfg-9m2wr\"" Nov 03 00:57:38.052750 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:38.052717 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-a2d8e5\"" Nov 03 00:57:38.083310 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:38.074670 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3521b6ee-8255-499f-ac8f-b32249341eb2-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-a2d8e5-f6bfcf94c-4jbsh\" (UID: \"3521b6ee-8255-499f-ac8f-b32249341eb2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a2d8e5-f6bfcf94c-4jbsh" Nov 03 00:57:38.083310 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:38.074866 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpgmc\" (UniqueName: \"kubernetes.io/projected/3521b6ee-8255-499f-ac8f-b32249341eb2-kube-api-access-rpgmc\") pod \"payload-pre-processing-e2e-api-b-a2d8e5-f6bfcf94c-4jbsh\" (UID: \"3521b6ee-8255-499f-ac8f-b32249341eb2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a2d8e5-f6bfcf94c-4jbsh" Nov 03 00:57:38.182363 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:38.176313 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpgmc\" (UniqueName: \"kubernetes.io/projected/3521b6ee-8255-499f-ac8f-b32249341eb2-kube-api-access-rpgmc\") pod \"payload-pre-processing-e2e-api-b-a2d8e5-f6bfcf94c-4jbsh\" (UID: \"3521b6ee-8255-499f-ac8f-b32249341eb2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a2d8e5-f6bfcf94c-4jbsh" Nov 03 00:57:38.182363 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:38.176642 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3521b6ee-8255-499f-ac8f-b32249341eb2-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-a2d8e5-f6bfcf94c-4jbsh\" (UID: \"3521b6ee-8255-499f-ac8f-b32249341eb2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a2d8e5-f6bfcf94c-4jbsh" Nov 03 00:57:38.182363 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:38.177738 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3521b6ee-8255-499f-ac8f-b32249341eb2-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-a2d8e5-f6bfcf94c-4jbsh\" (UID: \"3521b6ee-8255-499f-ac8f-b32249341eb2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a2d8e5-f6bfcf94c-4jbsh" Nov 03 00:57:38.223300 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:38.218267 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpgmc\" (UniqueName: \"kubernetes.io/projected/3521b6ee-8255-499f-ac8f-b32249341eb2-kube-api-access-rpgmc\") pod \"payload-pre-processing-e2e-api-b-a2d8e5-f6bfcf94c-4jbsh\" (UID: \"3521b6ee-8255-499f-ac8f-b32249341eb2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a2d8e5-f6bfcf94c-4jbsh" Nov 03 00:57:38.315205 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:38.315109 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a2d8e5-f6bfcf94c-4jbsh" Nov 03 00:57:38.609301 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:38.609226 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a2d8e5-f6bfcf94c-4jbsh"] Nov 03 00:57:39.515281 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:39.515246 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a2d8e5-f6bfcf94c-4jbsh" event={"ID":"3521b6ee-8255-499f-ac8f-b32249341eb2","Type":"ContainerStarted","Data":"a99a86d166d167294e3d6107a639e83f061d2db5d91e9f157d4243549e56be32"} Nov 03 00:57:39.515669 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:39.515319 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a2d8e5-f6bfcf94c-4jbsh" event={"ID":"3521b6ee-8255-499f-ac8f-b32249341eb2","Type":"ContainerStarted","Data":"823bbbaadb217d36e373e893a183dfb794241ffc74ee5f8ddaf8ca934871cd51"} Nov 03 00:57:39.515669 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:39.515349 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a2d8e5-f6bfcf94c-4jbsh" Nov 03 00:57:39.533850 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:39.533798 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a2d8e5-f6bfcf94c-4jbsh" podStartSLOduration=2.53378394 podStartE2EDuration="2.53378394s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:57:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:57:39.530053117 +0000 UTC m=+1001.701056214" watchObservedRunningTime="2026-11-03 00:57:39.53378394 +0000 UTC m=+1001.704787036" Nov 03 00:57:41.075615 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:41.075579 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b146b9f0-c5jsg"] Nov 03 00:57:41.084534 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:41.084500 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b146b9f0-c5jsg"] Nov 03 00:57:41.417547 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:41.417457 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-340b96-openshift-default-7846d978b9-8zqfs"] Nov 03 00:57:41.422042 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:41.422015 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-340b96-openshift-default-7846d978b9-8zqfs" Nov 03 00:57:41.438401 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:41.436160 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-340b96-openshift-default-dockercfg-bll8c\"" Nov 03 00:57:41.448632 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:41.448586 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-340b96-openshift-default-7846d978b9-8zqfs"] Nov 03 00:57:41.532049 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:41.532013 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90e63fe7-52ef-4fbd-ab37-32c6479abb7a-workload-socket\") pod \"e2e-api-a-340b96-openshift-default-7846d978b9-8zqfs\" (UID: \"90e63fe7-52ef-4fbd-ab37-32c6479abb7a\") " pod="openshift-ingress/e2e-api-a-340b96-openshift-default-7846d978b9-8zqfs" Nov 03 00:57:41.532245 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:41.532078 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90e63fe7-52ef-4fbd-ab37-32c6479abb7a-workload-certs\") pod \"e2e-api-a-340b96-openshift-default-7846d978b9-8zqfs\" (UID: \"90e63fe7-52ef-4fbd-ab37-32c6479abb7a\") " pod="openshift-ingress/e2e-api-a-340b96-openshift-default-7846d978b9-8zqfs" Nov 03 00:57:41.532245 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:41.532192 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vq57\" (UniqueName: \"kubernetes.io/projected/90e63fe7-52ef-4fbd-ab37-32c6479abb7a-kube-api-access-5vq57\") pod \"e2e-api-a-340b96-openshift-default-7846d978b9-8zqfs\" (UID: \"90e63fe7-52ef-4fbd-ab37-32c6479abb7a\") " pod="openshift-ingress/e2e-api-a-340b96-openshift-default-7846d978b9-8zqfs" Nov 03 00:57:41.532424 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:41.532242 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90e63fe7-52ef-4fbd-ab37-32c6479abb7a-istiod-ca-cert\") pod \"e2e-api-a-340b96-openshift-default-7846d978b9-8zqfs\" (UID: \"90e63fe7-52ef-4fbd-ab37-32c6479abb7a\") " pod="openshift-ingress/e2e-api-a-340b96-openshift-default-7846d978b9-8zqfs" Nov 03 00:57:41.532424 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:41.532360 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90e63fe7-52ef-4fbd-ab37-32c6479abb7a-istio-podinfo\") pod \"e2e-api-a-340b96-openshift-default-7846d978b9-8zqfs\" (UID: \"90e63fe7-52ef-4fbd-ab37-32c6479abb7a\") " pod="openshift-ingress/e2e-api-a-340b96-openshift-default-7846d978b9-8zqfs" Nov 03 00:57:41.532490 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:41.532425 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90e63fe7-52ef-4fbd-ab37-32c6479abb7a-istio-token\") pod \"e2e-api-a-340b96-openshift-default-7846d978b9-8zqfs\" (UID: \"90e63fe7-52ef-4fbd-ab37-32c6479abb7a\") " pod="openshift-ingress/e2e-api-a-340b96-openshift-default-7846d978b9-8zqfs" Nov 03 00:57:41.532567 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:41.532550 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90e63fe7-52ef-4fbd-ab37-32c6479abb7a-credential-socket\") pod \"e2e-api-a-340b96-openshift-default-7846d978b9-8zqfs\" (UID: \"90e63fe7-52ef-4fbd-ab37-32c6479abb7a\") " pod="openshift-ingress/e2e-api-a-340b96-openshift-default-7846d978b9-8zqfs" Nov 03 00:57:41.532709 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:41.532692 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90e63fe7-52ef-4fbd-ab37-32c6479abb7a-istio-data\") pod \"e2e-api-a-340b96-openshift-default-7846d978b9-8zqfs\" (UID: \"90e63fe7-52ef-4fbd-ab37-32c6479abb7a\") " pod="openshift-ingress/e2e-api-a-340b96-openshift-default-7846d978b9-8zqfs" Nov 03 00:57:41.532786 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:41.532772 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90e63fe7-52ef-4fbd-ab37-32c6479abb7a-istio-envoy\") pod \"e2e-api-a-340b96-openshift-default-7846d978b9-8zqfs\" (UID: \"90e63fe7-52ef-4fbd-ab37-32c6479abb7a\") " pod="openshift-ingress/e2e-api-a-340b96-openshift-default-7846d978b9-8zqfs" Nov 03 00:57:41.537476 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:41.537447 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-dd03e395-779465d96c-nppw5"] Nov 03 00:57:41.540829 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:41.540808 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-dd03e395-779465d96c-nppw5" Nov 03 00:57:41.553320 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:41.553295 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-dd03e395\"" Nov 03 00:57:41.553496 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:41.553360 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-dd03e395-dockercfg-n45nf\"" Nov 03 00:57:41.561167 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:41.561136 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-dd03e395-779465d96c-nppw5"] Nov 03 00:57:41.634623 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:41.634591 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vq57\" (UniqueName: \"kubernetes.io/projected/90e63fe7-52ef-4fbd-ab37-32c6479abb7a-kube-api-access-5vq57\") pod \"e2e-api-a-340b96-openshift-default-7846d978b9-8zqfs\" (UID: \"90e63fe7-52ef-4fbd-ab37-32c6479abb7a\") " pod="openshift-ingress/e2e-api-a-340b96-openshift-default-7846d978b9-8zqfs" Nov 03 00:57:41.634829 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:41.634631 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90e63fe7-52ef-4fbd-ab37-32c6479abb7a-istiod-ca-cert\") pod \"e2e-api-a-340b96-openshift-default-7846d978b9-8zqfs\" (UID: \"90e63fe7-52ef-4fbd-ab37-32c6479abb7a\") " pod="openshift-ingress/e2e-api-a-340b96-openshift-default-7846d978b9-8zqfs" Nov 03 00:57:41.634829 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:41.634674 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p74ng\" (UniqueName: \"kubernetes.io/projected/fa2a0390-dbfb-46e6-9a54-f2522988748f-kube-api-access-p74ng\") pod \"payload-pre-processing-e2e-mt-dd03e395-779465d96c-nppw5\" (UID: \"fa2a0390-dbfb-46e6-9a54-f2522988748f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-dd03e395-779465d96c-nppw5" Nov 03 00:57:41.634829 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:41.634702 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90e63fe7-52ef-4fbd-ab37-32c6479abb7a-istio-podinfo\") pod \"e2e-api-a-340b96-openshift-default-7846d978b9-8zqfs\" (UID: \"90e63fe7-52ef-4fbd-ab37-32c6479abb7a\") " pod="openshift-ingress/e2e-api-a-340b96-openshift-default-7846d978b9-8zqfs" Nov 03 00:57:41.634829 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:41.634743 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90e63fe7-52ef-4fbd-ab37-32c6479abb7a-istio-token\") pod \"e2e-api-a-340b96-openshift-default-7846d978b9-8zqfs\" (UID: \"90e63fe7-52ef-4fbd-ab37-32c6479abb7a\") " pod="openshift-ingress/e2e-api-a-340b96-openshift-default-7846d978b9-8zqfs" Nov 03 00:57:41.635055 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:41.634919 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90e63fe7-52ef-4fbd-ab37-32c6479abb7a-credential-socket\") pod \"e2e-api-a-340b96-openshift-default-7846d978b9-8zqfs\" (UID: \"90e63fe7-52ef-4fbd-ab37-32c6479abb7a\") " pod="openshift-ingress/e2e-api-a-340b96-openshift-default-7846d978b9-8zqfs" Nov 03 00:57:41.635055 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:41.634984 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa2a0390-dbfb-46e6-9a54-f2522988748f-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-dd03e395-779465d96c-nppw5\" (UID: \"fa2a0390-dbfb-46e6-9a54-f2522988748f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-dd03e395-779465d96c-nppw5" Nov 03 00:57:41.635197 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:41.635174 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90e63fe7-52ef-4fbd-ab37-32c6479abb7a-istio-data\") pod \"e2e-api-a-340b96-openshift-default-7846d978b9-8zqfs\" (UID: \"90e63fe7-52ef-4fbd-ab37-32c6479abb7a\") " pod="openshift-ingress/e2e-api-a-340b96-openshift-default-7846d978b9-8zqfs" Nov 03 00:57:41.635338 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:41.635315 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90e63fe7-52ef-4fbd-ab37-32c6479abb7a-istio-envoy\") pod \"e2e-api-a-340b96-openshift-default-7846d978b9-8zqfs\" (UID: \"90e63fe7-52ef-4fbd-ab37-32c6479abb7a\") " pod="openshift-ingress/e2e-api-a-340b96-openshift-default-7846d978b9-8zqfs" Nov 03 00:57:41.635424 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:41.635407 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90e63fe7-52ef-4fbd-ab37-32c6479abb7a-workload-socket\") pod \"e2e-api-a-340b96-openshift-default-7846d978b9-8zqfs\" (UID: \"90e63fe7-52ef-4fbd-ab37-32c6479abb7a\") " pod="openshift-ingress/e2e-api-a-340b96-openshift-default-7846d978b9-8zqfs" Nov 03 00:57:41.635488 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:41.635472 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90e63fe7-52ef-4fbd-ab37-32c6479abb7a-credential-socket\") pod \"e2e-api-a-340b96-openshift-default-7846d978b9-8zqfs\" (UID: \"90e63fe7-52ef-4fbd-ab37-32c6479abb7a\") " pod="openshift-ingress/e2e-api-a-340b96-openshift-default-7846d978b9-8zqfs" Nov 03 00:57:41.635643 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:41.635616 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90e63fe7-52ef-4fbd-ab37-32c6479abb7a-istio-data\") pod \"e2e-api-a-340b96-openshift-default-7846d978b9-8zqfs\" (UID: \"90e63fe7-52ef-4fbd-ab37-32c6479abb7a\") " pod="openshift-ingress/e2e-api-a-340b96-openshift-default-7846d978b9-8zqfs" Nov 03 00:57:41.635767 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:41.635750 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90e63fe7-52ef-4fbd-ab37-32c6479abb7a-istiod-ca-cert\") pod \"e2e-api-a-340b96-openshift-default-7846d978b9-8zqfs\" (UID: \"90e63fe7-52ef-4fbd-ab37-32c6479abb7a\") " pod="openshift-ingress/e2e-api-a-340b96-openshift-default-7846d978b9-8zqfs" Nov 03 00:57:41.635827 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:41.635765 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90e63fe7-52ef-4fbd-ab37-32c6479abb7a-workload-socket\") pod \"e2e-api-a-340b96-openshift-default-7846d978b9-8zqfs\" (UID: \"90e63fe7-52ef-4fbd-ab37-32c6479abb7a\") " pod="openshift-ingress/e2e-api-a-340b96-openshift-default-7846d978b9-8zqfs" Nov 03 00:57:41.635882 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:41.635823 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90e63fe7-52ef-4fbd-ab37-32c6479abb7a-workload-certs\") pod \"e2e-api-a-340b96-openshift-default-7846d978b9-8zqfs\" (UID: \"90e63fe7-52ef-4fbd-ab37-32c6479abb7a\") " pod="openshift-ingress/e2e-api-a-340b96-openshift-default-7846d978b9-8zqfs" Nov 03 00:57:41.636084 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:41.636066 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90e63fe7-52ef-4fbd-ab37-32c6479abb7a-workload-certs\") pod \"e2e-api-a-340b96-openshift-default-7846d978b9-8zqfs\" (UID: \"90e63fe7-52ef-4fbd-ab37-32c6479abb7a\") " pod="openshift-ingress/e2e-api-a-340b96-openshift-default-7846d978b9-8zqfs" Nov 03 00:57:41.637545 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:41.637519 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90e63fe7-52ef-4fbd-ab37-32c6479abb7a-istio-envoy\") pod \"e2e-api-a-340b96-openshift-default-7846d978b9-8zqfs\" (UID: \"90e63fe7-52ef-4fbd-ab37-32c6479abb7a\") " pod="openshift-ingress/e2e-api-a-340b96-openshift-default-7846d978b9-8zqfs" Nov 03 00:57:41.637829 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:41.637810 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90e63fe7-52ef-4fbd-ab37-32c6479abb7a-istio-podinfo\") pod \"e2e-api-a-340b96-openshift-default-7846d978b9-8zqfs\" (UID: \"90e63fe7-52ef-4fbd-ab37-32c6479abb7a\") " pod="openshift-ingress/e2e-api-a-340b96-openshift-default-7846d978b9-8zqfs" Nov 03 00:57:41.657522 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:41.645725 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vq57\" (UniqueName: \"kubernetes.io/projected/90e63fe7-52ef-4fbd-ab37-32c6479abb7a-kube-api-access-5vq57\") pod \"e2e-api-a-340b96-openshift-default-7846d978b9-8zqfs\" (UID: \"90e63fe7-52ef-4fbd-ab37-32c6479abb7a\") " pod="openshift-ingress/e2e-api-a-340b96-openshift-default-7846d978b9-8zqfs" Nov 03 00:57:41.657522 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:41.646197 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90e63fe7-52ef-4fbd-ab37-32c6479abb7a-istio-token\") pod \"e2e-api-a-340b96-openshift-default-7846d978b9-8zqfs\" (UID: \"90e63fe7-52ef-4fbd-ab37-32c6479abb7a\") " pod="openshift-ingress/e2e-api-a-340b96-openshift-default-7846d978b9-8zqfs" Nov 03 00:57:41.741366 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:41.740644 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa2a0390-dbfb-46e6-9a54-f2522988748f-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-dd03e395-779465d96c-nppw5\" (UID: \"fa2a0390-dbfb-46e6-9a54-f2522988748f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-dd03e395-779465d96c-nppw5" Nov 03 00:57:41.741366 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:41.740928 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p74ng\" (UniqueName: \"kubernetes.io/projected/fa2a0390-dbfb-46e6-9a54-f2522988748f-kube-api-access-p74ng\") pod \"payload-pre-processing-e2e-mt-dd03e395-779465d96c-nppw5\" (UID: \"fa2a0390-dbfb-46e6-9a54-f2522988748f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-dd03e395-779465d96c-nppw5" Nov 03 00:57:41.742439 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:41.742411 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa2a0390-dbfb-46e6-9a54-f2522988748f-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-dd03e395-779465d96c-nppw5\" (UID: \"fa2a0390-dbfb-46e6-9a54-f2522988748f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-dd03e395-779465d96c-nppw5" Nov 03 00:57:41.747293 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:41.746531 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-340b96-openshift-default-7846d978b9-8zqfs" Nov 03 00:57:41.831176 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:41.830749 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p74ng\" (UniqueName: \"kubernetes.io/projected/fa2a0390-dbfb-46e6-9a54-f2522988748f-kube-api-access-p74ng\") pod \"payload-pre-processing-e2e-mt-dd03e395-779465d96c-nppw5\" (UID: \"fa2a0390-dbfb-46e6-9a54-f2522988748f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-dd03e395-779465d96c-nppw5" Nov 03 00:57:41.864326 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:41.859995 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-dd03e395-779465d96c-nppw5" Nov 03 00:57:41.991847 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:41.991222 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1396b2d7-b7dc-4d5a-b897-480303f02972" path="/var/lib/kubelet/pods/1396b2d7-b7dc-4d5a-b897-480303f02972/volumes" Nov 03 00:57:42.177882 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:57:42.177822 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90e63fe7_52ef_4fbd_ab37_32c6479abb7a.slice/crio-13d623ebf0266bc1b8dac94584e3afcdd93d0a16b1fc4853856223b02b2a2fa7 WatchSource:0}: Error finding container 13d623ebf0266bc1b8dac94584e3afcdd93d0a16b1fc4853856223b02b2a2fa7: Status 404 returned error can't find the container with id 13d623ebf0266bc1b8dac94584e3afcdd93d0a16b1fc4853856223b02b2a2fa7 Nov 03 00:57:42.181038 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:42.180987 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:57:42.181158 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:42.181078 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:57:42.181158 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:42.181128 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:57:42.184815 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:42.184662 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-340b96-openshift-default-7846d978b9-8zqfs"] Nov 03 00:57:42.248991 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:57:42.248965 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa2a0390_dbfb_46e6_9a54_f2522988748f.slice/crio-d10309c916be06ec3bc6cdb6853e7325e6853d6e50cc8782ee89799c27117b06 WatchSource:0}: Error finding container d10309c916be06ec3bc6cdb6853e7325e6853d6e50cc8782ee89799c27117b06: Status 404 returned error can't find the container with id d10309c916be06ec3bc6cdb6853e7325e6853d6e50cc8782ee89799c27117b06 Nov 03 00:57:42.249086 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:42.249071 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-dd03e395-779465d96c-nppw5"] Nov 03 00:57:42.549041 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:42.548996 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-dd03e395-779465d96c-nppw5" event={"ID":"fa2a0390-dbfb-46e6-9a54-f2522988748f","Type":"ContainerStarted","Data":"b7580aacf21666a9deb4495ed376a7e4d913cb231095d3fefe7e6d89e0c7387e"} Nov 03 00:57:42.549235 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:42.549075 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-dd03e395-779465d96c-nppw5" Nov 03 00:57:42.549235 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:42.549101 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-dd03e395-779465d96c-nppw5" event={"ID":"fa2a0390-dbfb-46e6-9a54-f2522988748f","Type":"ContainerStarted","Data":"d10309c916be06ec3bc6cdb6853e7325e6853d6e50cc8782ee89799c27117b06"} Nov 03 00:57:42.551512 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:42.551474 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-340b96-openshift-default-7846d978b9-8zqfs" event={"ID":"90e63fe7-52ef-4fbd-ab37-32c6479abb7a","Type":"ContainerStarted","Data":"4727b4aed6f9fe13c30ac83ef9e469d024a700aaf2f264aea71481ee101489f8"} Nov 03 00:57:42.551646 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:42.551543 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-340b96-openshift-default-7846d978b9-8zqfs" event={"ID":"90e63fe7-52ef-4fbd-ab37-32c6479abb7a","Type":"ContainerStarted","Data":"13d623ebf0266bc1b8dac94584e3afcdd93d0a16b1fc4853856223b02b2a2fa7"} Nov 03 00:57:42.567003 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:42.566947 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-dd03e395-779465d96c-nppw5" podStartSLOduration=1.566929087 podStartE2EDuration="1.566929087s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:57:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:57:42.5650149 +0000 UTC m=+1004.736017996" watchObservedRunningTime="2026-11-03 00:57:42.566929087 +0000 UTC m=+1004.737932187" Nov 03 00:57:42.590846 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:42.590790 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-340b96-openshift-default-7846d978b9-8zqfs" podStartSLOduration=1.590775719 podStartE2EDuration="1.590775719s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:57:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:57:42.587169272 +0000 UTC m=+1004.758172369" watchObservedRunningTime="2026-11-03 00:57:42.590775719 +0000 UTC m=+1004.761778814" Nov 03 00:57:42.747730 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:42.747678 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-340b96-openshift-default-7846d978b9-8zqfs" Nov 03 00:57:43.753409 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:43.753378 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-340b96-openshift-default-7846d978b9-8zqfs" Nov 03 00:57:44.689219 ip-10-0-133-217 kubenswrapper[4421]: E1103 00:57:44.689172 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 00:57:44.738602 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:44.738570 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-340b96-openshift-default-7846d978b9-8zqfs" Nov 03 00:57:44.739706 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:44.739682 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-340b96-openshift-default-7846d978b9-8zqfs" Nov 03 00:57:50.526444 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:50.526412 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a2d8e5-f6bfcf94c-4jbsh" Nov 03 00:57:50.880531 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:50.880496 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9918c79-jtszs"] Nov 03 00:57:50.885069 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:50.885051 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9918c79-jtszs" Nov 03 00:57:50.888776 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:50.888749 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:57:50.888935 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:50.888851 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 00:57:50.888935 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:50.888883 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6q4xp\"" Nov 03 00:57:50.888935 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:50.888888 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 00:57:50.893760 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:50.893735 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9918c79-jtszs"] Nov 03 00:57:51.006858 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:51.006808 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lchr\" (UniqueName: \"kubernetes.io/projected/74cb1387-3dbb-4a02-8102-c0d86c5a9738-kube-api-access-8lchr\") pod \"maas-api-revoke-sub-c9918c79-jtszs\" (UID: \"74cb1387-3dbb-4a02-8102-c0d86c5a9738\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9918c79-jtszs" Nov 03 00:57:51.007131 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:51.007057 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74cb1387-3dbb-4a02-8102-c0d86c5a9738-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c9918c79-jtszs\" (UID: \"74cb1387-3dbb-4a02-8102-c0d86c5a9738\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9918c79-jtszs" Nov 03 00:57:51.007301 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:51.007192 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74cb1387-3dbb-4a02-8102-c0d86c5a9738-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c9918c79-jtszs\" (UID: \"74cb1387-3dbb-4a02-8102-c0d86c5a9738\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9918c79-jtszs" Nov 03 00:57:51.109113 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:51.109024 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74cb1387-3dbb-4a02-8102-c0d86c5a9738-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c9918c79-jtszs\" (UID: \"74cb1387-3dbb-4a02-8102-c0d86c5a9738\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9918c79-jtszs" Nov 03 00:57:51.109314 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:51.109137 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lchr\" (UniqueName: \"kubernetes.io/projected/74cb1387-3dbb-4a02-8102-c0d86c5a9738-kube-api-access-8lchr\") pod \"maas-api-revoke-sub-c9918c79-jtszs\" (UID: \"74cb1387-3dbb-4a02-8102-c0d86c5a9738\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9918c79-jtszs" Nov 03 00:57:51.109486 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:51.109456 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74cb1387-3dbb-4a02-8102-c0d86c5a9738-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c9918c79-jtszs\" (UID: \"74cb1387-3dbb-4a02-8102-c0d86c5a9738\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9918c79-jtszs" Nov 03 00:57:51.110151 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:51.110122 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74cb1387-3dbb-4a02-8102-c0d86c5a9738-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c9918c79-jtszs\" (UID: \"74cb1387-3dbb-4a02-8102-c0d86c5a9738\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9918c79-jtszs" Nov 03 00:57:51.111710 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:51.111676 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74cb1387-3dbb-4a02-8102-c0d86c5a9738-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c9918c79-jtszs\" (UID: \"74cb1387-3dbb-4a02-8102-c0d86c5a9738\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9918c79-jtszs" Nov 03 00:57:51.117823 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:51.117798 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lchr\" (UniqueName: \"kubernetes.io/projected/74cb1387-3dbb-4a02-8102-c0d86c5a9738-kube-api-access-8lchr\") pod \"maas-api-revoke-sub-c9918c79-jtszs\" (UID: \"74cb1387-3dbb-4a02-8102-c0d86c5a9738\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9918c79-jtszs" Nov 03 00:57:51.236919 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:51.236886 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9918c79-jtszs" Nov 03 00:57:51.632296 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:51.631330 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9918c79-jtszs"] Nov 03 00:57:51.824491 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:51.824451 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9918c79-jtszs" event={"ID":"74cb1387-3dbb-4a02-8102-c0d86c5a9738","Type":"ContainerStarted","Data":"4c2ff3a6e3fe89420681fe5f41c75a60e9fbcbf01a3daa3ca0ccae03602c46fd"} Nov 03 00:57:52.835868 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:52.835776 4421 generic.go:356] "Generic (PLEG): container finished" podID="74cb1387-3dbb-4a02-8102-c0d86c5a9738" containerID="9fac9d730ba79ac5c42913ff4ab87de7c119bc8ee5d510d3eb235d898c0585a5" exitCode=0 Nov 03 00:57:52.835868 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:52.835854 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9918c79-jtszs" event={"ID":"74cb1387-3dbb-4a02-8102-c0d86c5a9738","Type":"ContainerDied","Data":"9fac9d730ba79ac5c42913ff4ab87de7c119bc8ee5d510d3eb235d898c0585a5"} Nov 03 00:57:53.706233 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:53.706206 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-dd03e395-779465d96c-nppw5" Nov 03 00:57:54.091701 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:54.091664 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9918c79-jtszs" Nov 03 00:57:54.203465 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:54.203426 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/74cb1387-3dbb-4a02-8102-c0d86c5a9738-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74cb1387-3dbb-4a02-8102-c0d86c5a9738-maas-api-xks-ca\") pod \"74cb1387-3dbb-4a02-8102-c0d86c5a9738\" (UID: \"74cb1387-3dbb-4a02-8102-c0d86c5a9738\") " Nov 03 00:57:54.203630 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:54.203499 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74cb1387-3dbb-4a02-8102-c0d86c5a9738-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74cb1387-3dbb-4a02-8102-c0d86c5a9738-maas-api-openshift-ca\") pod \"74cb1387-3dbb-4a02-8102-c0d86c5a9738\" (UID: \"74cb1387-3dbb-4a02-8102-c0d86c5a9738\") " Nov 03 00:57:54.203630 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:54.203552 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74cb1387-3dbb-4a02-8102-c0d86c5a9738-kube-api-access-8lchr\" (UniqueName: \"kubernetes.io/projected/74cb1387-3dbb-4a02-8102-c0d86c5a9738-kube-api-access-8lchr\") pod \"74cb1387-3dbb-4a02-8102-c0d86c5a9738\" (UID: \"74cb1387-3dbb-4a02-8102-c0d86c5a9738\") " Nov 03 00:57:54.203951 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:54.203923 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74cb1387-3dbb-4a02-8102-c0d86c5a9738-maas-api-openshift-ca" pod "74cb1387-3dbb-4a02-8102-c0d86c5a9738" (UID: "74cb1387-3dbb-4a02-8102-c0d86c5a9738"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:57:54.204287 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:54.204249 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74cb1387-3dbb-4a02-8102-c0d86c5a9738-maas-api-openshift-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:57:54.205724 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:54.205665 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74cb1387-3dbb-4a02-8102-c0d86c5a9738-maas-api-xks-ca" pod "74cb1387-3dbb-4a02-8102-c0d86c5a9738" (UID: "74cb1387-3dbb-4a02-8102-c0d86c5a9738"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:57:54.206051 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:54.206034 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74cb1387-3dbb-4a02-8102-c0d86c5a9738-kube-api-access-8lchr" pod "74cb1387-3dbb-4a02-8102-c0d86c5a9738" (UID: "74cb1387-3dbb-4a02-8102-c0d86c5a9738"). InnerVolumeSpecName "kube-api-access-8lchr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:57:54.307698 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:54.307617 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74cb1387-3dbb-4a02-8102-c0d86c5a9738-maas-api-xks-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:57:54.307698 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:54.307648 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8lchr\" (UniqueName: \"kubernetes.io/projected/74cb1387-3dbb-4a02-8102-c0d86c5a9738-kube-api-access-8lchr\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:57:54.862651 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:54.862559 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9918c79-jtszs" Nov 03 00:57:54.862651 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:54.862585 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9918c79-jtszs" event={"ID":"74cb1387-3dbb-4a02-8102-c0d86c5a9738","Type":"ContainerDied","Data":"4c2ff3a6e3fe89420681fe5f41c75a60e9fbcbf01a3daa3ca0ccae03602c46fd"} Nov 03 00:57:54.862651 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:54.862645 4421 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4c2ff3a6e3fe89420681fe5f41c75a60e9fbcbf01a3daa3ca0ccae03602c46fd" Nov 03 00:57:54.864841 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:54.864810 4421 generic.go:356] "Generic (PLEG): container finished" podID="a6b01747-59d0-4ee6-9a96-4e3d1cad82df" containerID="c47e742c228c773354e827269a6ee024a997b9cd9dabf084161398b1ef7e0f75" exitCode=0 Nov 03 00:57:54.864985 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:54.864878 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-54df68-openshift-default-74b6dbc6cf-4kmpk" event={"ID":"a6b01747-59d0-4ee6-9a96-4e3d1cad82df","Type":"ContainerDied","Data":"c47e742c228c773354e827269a6ee024a997b9cd9dabf084161398b1ef7e0f75"} Nov 03 00:57:54.865368 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:54.865354 4421 scope.go:122] "RemoveContainer" containerID="c47e742c228c773354e827269a6ee024a997b9cd9dabf084161398b1ef7e0f75" Nov 03 00:57:54.866916 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:54.866885 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:57:54.867003 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:54.866959 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:57:54.867003 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:54.866992 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:57:55.861744 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:55.861699 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-54df68-openshift-default-74b6dbc6cf-4kmpk" Nov 03 00:57:55.875055 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:55.875015 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-54df68-openshift-default-74b6dbc6cf-4kmpk" event={"ID":"a6b01747-59d0-4ee6-9a96-4e3d1cad82df","Type":"ContainerStarted","Data":"fb41143b22956591601307bf987aa6e5f29323ac418b357d458ad6560ef0b918"} Nov 03 00:57:56.063140 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:56.063097 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-54df68-openshift-default-74b6dbc6cf-4kmpk" Nov 03 00:57:56.064368 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:56.064344 4421 patch_prober.go:29] interesting pod/e2e-api-a-54df68-openshift-default-74b6dbc6cf-4kmpk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Nov 03 00:57:56.064488 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:56.064404 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-54df68-openshift-default-74b6dbc6cf-4kmpk" podUID="a6b01747-59d0-4ee6-9a96-4e3d1cad82df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Nov 03 00:57:57.062860 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:57.062824 4421 patch_prober.go:29] interesting pod/e2e-api-a-54df68-openshift-default-74b6dbc6cf-4kmpk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Nov 03 00:57:57.063322 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:57.062878 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-54df68-openshift-default-74b6dbc6cf-4kmpk" podUID="a6b01747-59d0-4ee6-9a96-4e3d1cad82df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Nov 03 00:57:58.062750 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:58.062716 4421 patch_prober.go:29] interesting pod/e2e-api-a-54df68-openshift-default-74b6dbc6cf-4kmpk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Nov 03 00:57:58.062936 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:58.062769 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-54df68-openshift-default-74b6dbc6cf-4kmpk" podUID="a6b01747-59d0-4ee6-9a96-4e3d1cad82df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Nov 03 00:57:58.468773 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:58.468674 4421 scope.go:122] "RemoveContainer" containerID="98469d4892095fecbac19cf8a60b9f8684ce73fbb03a91921e4a2b8e3a6fad4a" Nov 03 00:57:59.063436 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:59.063396 4421 patch_prober.go:29] interesting pod/e2e-api-a-54df68-openshift-default-74b6dbc6cf-4kmpk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Nov 03 00:57:59.063857 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:57:59.063454 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-54df68-openshift-default-74b6dbc6cf-4kmpk" podUID="a6b01747-59d0-4ee6-9a96-4e3d1cad82df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Nov 03 00:58:00.063367 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:00.063333 4421 patch_prober.go:29] interesting pod/e2e-api-a-54df68-openshift-default-74b6dbc6cf-4kmpk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Nov 03 00:58:00.063554 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:00.063391 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-54df68-openshift-default-74b6dbc6cf-4kmpk" podUID="a6b01747-59d0-4ee6-9a96-4e3d1cad82df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Nov 03 00:58:01.063058 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:01.063021 4421 patch_prober.go:29] interesting pod/e2e-api-a-54df68-openshift-default-74b6dbc6cf-4kmpk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Nov 03 00:58:01.063240 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:01.063086 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-54df68-openshift-default-74b6dbc6cf-4kmpk" podUID="a6b01747-59d0-4ee6-9a96-4e3d1cad82df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Nov 03 00:58:02.066852 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:02.066821 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-54df68-openshift-default-74b6dbc6cf-4kmpk" Nov 03 00:58:02.067309 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:02.067265 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-54df68-openshift-default-74b6dbc6cf-4kmpk" Nov 03 00:58:02.067987 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:02.067957 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-54df68-openshift-default-74b6dbc6cf-4kmpk" Nov 03 00:58:05.251328 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:05.231252 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-54df68-59cb67b865-9d886"] Nov 03 00:58:05.887312 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:05.881525 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-650be0/simulator-w4-650be0-kserve-7bc588774f-gvz2k" podUID="a827fa88-f078-4312-8f12-2c799c11b461" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 00:58:05.887312 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:05.884817 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/premium-w3-83755a-kserve-7c9d66988b-lt4xd" podUID="c0c6b02e-2ecb-428e-a13d-be6d7cd1ba39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:58:06.070538 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:06.048413 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/distinct-w3-83755a-kserve-86bdbbdd85-2cnnl" podUID="f4ae9624-d127-4a55-837d-7c9ff0267b37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:58:06.249070 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:06.227457 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8gm8n" podUID="2d367e12-e46f-4a68-9825-9b1c934d4d12" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:58:06.285949 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:06.255938 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-650be0/premium-w4-650be0-kserve-7bf8555dfb-z97n5" podUID="40601dc5-808f-456f-826f-6e0c35966ada" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:58:06.313460 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:06.313396 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/simulator-w3-83755a-kserve-77d6664559-psqtg" podUID="b0a4c234-af58-4da0-a1a4-e1f167953f6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Nov 03 00:58:06.439585 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:06.415300 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/simulator-w2-f7680d-kserve-85bdc86cd6-8gxrg" podUID="2bdcf521-cebd-4d6b-8554-1b60db8b0f3c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:58:06.560360 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:06.553227 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-eed369-openshift-default-66698bcb95-pknkp"] Nov 03 00:58:06.698035 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:06.697999 4421 cpu_manager.go:414] "containerMap: removing container" podUID="74cb1387-3dbb-4a02-8102-c0d86c5a9738" containerName="revoke-keys" Nov 03 00:58:06.705352 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:06.705319 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74cb1387-3dbb-4a02-8102-c0d86c5a9738" containerName="revoke-keys" Nov 03 00:58:06.721305 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:06.706083 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="74cb1387-3dbb-4a02-8102-c0d86c5a9738" containerName="revoke-keys" Nov 03 00:58:06.784845 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:06.784807 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-eed369-openshift-default-66698bcb95-pknkp"] Nov 03 00:58:06.785197 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:06.785180 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-eed369-openshift-default-66698bcb95-pknkp" Nov 03 00:58:06.790988 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:06.790969 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-eed369-openshift-default-dockercfg-hnnnv\"" Nov 03 00:58:06.951450 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:06.951348 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b797c4ed-b8df-4e23-9188-3aeb3cf6c48f-istio-podinfo\") pod \"e2e-api-b-eed369-openshift-default-66698bcb95-pknkp\" (UID: \"b797c4ed-b8df-4e23-9188-3aeb3cf6c48f\") " pod="openshift-ingress/e2e-api-b-eed369-openshift-default-66698bcb95-pknkp" Nov 03 00:58:06.951696 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:06.951674 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b797c4ed-b8df-4e23-9188-3aeb3cf6c48f-workload-certs\") pod \"e2e-api-b-eed369-openshift-default-66698bcb95-pknkp\" (UID: \"b797c4ed-b8df-4e23-9188-3aeb3cf6c48f\") " pod="openshift-ingress/e2e-api-b-eed369-openshift-default-66698bcb95-pknkp" Nov 03 00:58:06.951842 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:06.951824 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b797c4ed-b8df-4e23-9188-3aeb3cf6c48f-credential-socket\") pod \"e2e-api-b-eed369-openshift-default-66698bcb95-pknkp\" (UID: \"b797c4ed-b8df-4e23-9188-3aeb3cf6c48f\") " pod="openshift-ingress/e2e-api-b-eed369-openshift-default-66698bcb95-pknkp" Nov 03 00:58:06.952027 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:06.952010 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b797c4ed-b8df-4e23-9188-3aeb3cf6c48f-istio-envoy\") pod \"e2e-api-b-eed369-openshift-default-66698bcb95-pknkp\" (UID: \"b797c4ed-b8df-4e23-9188-3aeb3cf6c48f\") " pod="openshift-ingress/e2e-api-b-eed369-openshift-default-66698bcb95-pknkp" Nov 03 00:58:06.952151 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:06.952136 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b797c4ed-b8df-4e23-9188-3aeb3cf6c48f-workload-socket\") pod \"e2e-api-b-eed369-openshift-default-66698bcb95-pknkp\" (UID: \"b797c4ed-b8df-4e23-9188-3aeb3cf6c48f\") " pod="openshift-ingress/e2e-api-b-eed369-openshift-default-66698bcb95-pknkp" Nov 03 00:58:06.952310 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:06.952294 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b797c4ed-b8df-4e23-9188-3aeb3cf6c48f-istio-data\") pod \"e2e-api-b-eed369-openshift-default-66698bcb95-pknkp\" (UID: \"b797c4ed-b8df-4e23-9188-3aeb3cf6c48f\") " pod="openshift-ingress/e2e-api-b-eed369-openshift-default-66698bcb95-pknkp" Nov 03 00:58:06.952581 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:06.952562 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d42j4\" (UniqueName: \"kubernetes.io/projected/b797c4ed-b8df-4e23-9188-3aeb3cf6c48f-kube-api-access-d42j4\") pod \"e2e-api-b-eed369-openshift-default-66698bcb95-pknkp\" (UID: \"b797c4ed-b8df-4e23-9188-3aeb3cf6c48f\") " pod="openshift-ingress/e2e-api-b-eed369-openshift-default-66698bcb95-pknkp" Nov 03 00:58:06.952728 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:06.952711 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b797c4ed-b8df-4e23-9188-3aeb3cf6c48f-istio-token\") pod \"e2e-api-b-eed369-openshift-default-66698bcb95-pknkp\" (UID: \"b797c4ed-b8df-4e23-9188-3aeb3cf6c48f\") " pod="openshift-ingress/e2e-api-b-eed369-openshift-default-66698bcb95-pknkp" Nov 03 00:58:06.952853 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:06.952838 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b797c4ed-b8df-4e23-9188-3aeb3cf6c48f-istiod-ca-cert\") pod \"e2e-api-b-eed369-openshift-default-66698bcb95-pknkp\" (UID: \"b797c4ed-b8df-4e23-9188-3aeb3cf6c48f\") " pod="openshift-ingress/e2e-api-b-eed369-openshift-default-66698bcb95-pknkp" Nov 03 00:58:07.060710 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:07.060645 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b797c4ed-b8df-4e23-9188-3aeb3cf6c48f-istio-podinfo\") pod \"e2e-api-b-eed369-openshift-default-66698bcb95-pknkp\" (UID: \"b797c4ed-b8df-4e23-9188-3aeb3cf6c48f\") " pod="openshift-ingress/e2e-api-b-eed369-openshift-default-66698bcb95-pknkp" Nov 03 00:58:07.062863 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:07.062823 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b797c4ed-b8df-4e23-9188-3aeb3cf6c48f-workload-certs\") pod \"e2e-api-b-eed369-openshift-default-66698bcb95-pknkp\" (UID: \"b797c4ed-b8df-4e23-9188-3aeb3cf6c48f\") " pod="openshift-ingress/e2e-api-b-eed369-openshift-default-66698bcb95-pknkp" Nov 03 00:58:07.063824 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:07.063799 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b797c4ed-b8df-4e23-9188-3aeb3cf6c48f-credential-socket\") pod \"e2e-api-b-eed369-openshift-default-66698bcb95-pknkp\" (UID: \"b797c4ed-b8df-4e23-9188-3aeb3cf6c48f\") " pod="openshift-ingress/e2e-api-b-eed369-openshift-default-66698bcb95-pknkp" Nov 03 00:58:07.064082 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:07.064062 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b797c4ed-b8df-4e23-9188-3aeb3cf6c48f-istio-envoy\") pod \"e2e-api-b-eed369-openshift-default-66698bcb95-pknkp\" (UID: \"b797c4ed-b8df-4e23-9188-3aeb3cf6c48f\") " pod="openshift-ingress/e2e-api-b-eed369-openshift-default-66698bcb95-pknkp" Nov 03 00:58:07.064221 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:07.064205 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b797c4ed-b8df-4e23-9188-3aeb3cf6c48f-workload-socket\") pod \"e2e-api-b-eed369-openshift-default-66698bcb95-pknkp\" (UID: \"b797c4ed-b8df-4e23-9188-3aeb3cf6c48f\") " pod="openshift-ingress/e2e-api-b-eed369-openshift-default-66698bcb95-pknkp" Nov 03 00:58:07.064390 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:07.064370 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b797c4ed-b8df-4e23-9188-3aeb3cf6c48f-istio-data\") pod \"e2e-api-b-eed369-openshift-default-66698bcb95-pknkp\" (UID: \"b797c4ed-b8df-4e23-9188-3aeb3cf6c48f\") " pod="openshift-ingress/e2e-api-b-eed369-openshift-default-66698bcb95-pknkp" Nov 03 00:58:07.064690 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:07.064669 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d42j4\" (UniqueName: \"kubernetes.io/projected/b797c4ed-b8df-4e23-9188-3aeb3cf6c48f-kube-api-access-d42j4\") pod \"e2e-api-b-eed369-openshift-default-66698bcb95-pknkp\" (UID: \"b797c4ed-b8df-4e23-9188-3aeb3cf6c48f\") " pod="openshift-ingress/e2e-api-b-eed369-openshift-default-66698bcb95-pknkp" Nov 03 00:58:07.064858 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:07.064843 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b797c4ed-b8df-4e23-9188-3aeb3cf6c48f-istio-token\") pod \"e2e-api-b-eed369-openshift-default-66698bcb95-pknkp\" (UID: \"b797c4ed-b8df-4e23-9188-3aeb3cf6c48f\") " pod="openshift-ingress/e2e-api-b-eed369-openshift-default-66698bcb95-pknkp" Nov 03 00:58:07.064999 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:07.064981 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b797c4ed-b8df-4e23-9188-3aeb3cf6c48f-istiod-ca-cert\") pod \"e2e-api-b-eed369-openshift-default-66698bcb95-pknkp\" (UID: \"b797c4ed-b8df-4e23-9188-3aeb3cf6c48f\") " pod="openshift-ingress/e2e-api-b-eed369-openshift-default-66698bcb95-pknkp" Nov 03 00:58:07.070299 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:07.065964 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b797c4ed-b8df-4e23-9188-3aeb3cf6c48f-istiod-ca-cert\") pod \"e2e-api-b-eed369-openshift-default-66698bcb95-pknkp\" (UID: \"b797c4ed-b8df-4e23-9188-3aeb3cf6c48f\") " pod="openshift-ingress/e2e-api-b-eed369-openshift-default-66698bcb95-pknkp" Nov 03 00:58:07.071296 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:07.063712 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b797c4ed-b8df-4e23-9188-3aeb3cf6c48f-workload-certs\") pod \"e2e-api-b-eed369-openshift-default-66698bcb95-pknkp\" (UID: \"b797c4ed-b8df-4e23-9188-3aeb3cf6c48f\") " pod="openshift-ingress/e2e-api-b-eed369-openshift-default-66698bcb95-pknkp" Nov 03 00:58:07.071437 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:07.070849 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b797c4ed-b8df-4e23-9188-3aeb3cf6c48f-credential-socket\") pod \"e2e-api-b-eed369-openshift-default-66698bcb95-pknkp\" (UID: \"b797c4ed-b8df-4e23-9188-3aeb3cf6c48f\") " pod="openshift-ingress/e2e-api-b-eed369-openshift-default-66698bcb95-pknkp" Nov 03 00:58:07.075683 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:07.075651 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b797c4ed-b8df-4e23-9188-3aeb3cf6c48f-workload-socket\") pod \"e2e-api-b-eed369-openshift-default-66698bcb95-pknkp\" (UID: \"b797c4ed-b8df-4e23-9188-3aeb3cf6c48f\") " pod="openshift-ingress/e2e-api-b-eed369-openshift-default-66698bcb95-pknkp" Nov 03 00:58:07.076367 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:07.076340 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b797c4ed-b8df-4e23-9188-3aeb3cf6c48f-istio-data\") pod \"e2e-api-b-eed369-openshift-default-66698bcb95-pknkp\" (UID: \"b797c4ed-b8df-4e23-9188-3aeb3cf6c48f\") " pod="openshift-ingress/e2e-api-b-eed369-openshift-default-66698bcb95-pknkp" Nov 03 00:58:07.088978 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:07.088948 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b797c4ed-b8df-4e23-9188-3aeb3cf6c48f-istio-envoy\") pod \"e2e-api-b-eed369-openshift-default-66698bcb95-pknkp\" (UID: \"b797c4ed-b8df-4e23-9188-3aeb3cf6c48f\") " pod="openshift-ingress/e2e-api-b-eed369-openshift-default-66698bcb95-pknkp" Nov 03 00:58:07.104215 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:07.104178 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b797c4ed-b8df-4e23-9188-3aeb3cf6c48f-istio-podinfo\") pod \"e2e-api-b-eed369-openshift-default-66698bcb95-pknkp\" (UID: \"b797c4ed-b8df-4e23-9188-3aeb3cf6c48f\") " pod="openshift-ingress/e2e-api-b-eed369-openshift-default-66698bcb95-pknkp" Nov 03 00:58:07.138297 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:07.124972 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b797c4ed-b8df-4e23-9188-3aeb3cf6c48f-istio-token\") pod \"e2e-api-b-eed369-openshift-default-66698bcb95-pknkp\" (UID: \"b797c4ed-b8df-4e23-9188-3aeb3cf6c48f\") " pod="openshift-ingress/e2e-api-b-eed369-openshift-default-66698bcb95-pknkp" Nov 03 00:58:07.140680 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:07.140645 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d42j4\" (UniqueName: \"kubernetes.io/projected/b797c4ed-b8df-4e23-9188-3aeb3cf6c48f-kube-api-access-d42j4\") pod \"e2e-api-b-eed369-openshift-default-66698bcb95-pknkp\" (UID: \"b797c4ed-b8df-4e23-9188-3aeb3cf6c48f\") " pod="openshift-ingress/e2e-api-b-eed369-openshift-default-66698bcb95-pknkp" Nov 03 00:58:07.141134 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:07.141113 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-eed369-openshift-default-66698bcb95-pknkp" Nov 03 00:58:07.280462 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:07.280418 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8gm8n" podUID="2d367e12-e46f-4a68-9825-9b1c934d4d12" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 00:58:07.281503 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:07.281480 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8gm8n" Nov 03 00:58:07.391449 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:07.391400 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/simulator-w3-83755a-kserve-77d6664559-psqtg" podUID="b0a4c234-af58-4da0-a1a4-e1f167953f6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 00:58:07.429435 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:07.429394 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-83755a/simulator-w3-83755a-kserve-77d6664559-psqtg" Nov 03 00:58:07.856299 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:07.826677 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-650be0/simulator-w4-650be0-kserve-7bc588774f-gvz2k" podUID="a827fa88-f078-4312-8f12-2c799c11b461" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:58:07.882300 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:07.868524 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/premium-w3-83755a-kserve-7c9d66988b-lt4xd" podUID="c0c6b02e-2ecb-428e-a13d-be6d7cd1ba39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:58:08.083359 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:08.067464 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/distinct-w3-83755a-kserve-86bdbbdd85-2cnnl" podUID="f4ae9624-d127-4a55-837d-7c9ff0267b37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded" Nov 03 00:58:08.247384 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:08.247337 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-650be0/premium-w4-650be0-kserve-7bf8555dfb-z97n5" podUID="40601dc5-808f-456f-826f-6e0c35966ada" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:58:08.307298 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:08.297367 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8gm8n" podUID="2d367e12-e46f-4a68-9825-9b1c934d4d12" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": context deadline exceeded" Nov 03 00:58:08.402404 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:08.400380 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/simulator-w3-83755a-kserve-77d6664559-psqtg" podUID="b0a4c234-af58-4da0-a1a4-e1f167953f6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:58:08.458619 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:08.458576 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/simulator-w2-f7680d-kserve-85bdc86cd6-8gxrg" podUID="2bdcf521-cebd-4d6b-8554-1b60db8b0f3c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 00:58:08.878518 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:08.878473 4421 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-t5bt4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 00:58:09.331352 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:09.233402 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-t5bt4" podUID="2b55c151-d235-4bcc-b602-eca0da4a7348" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 00:58:09.331352 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:08.958026 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-650be0/simulator-w4-650be0-kserve-7bc588774f-gvz2k" podUID="a827fa88-f078-4312-8f12-2c799c11b461" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:58:09.331352 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:09.069097 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/distinct-w3-83755a-kserve-86bdbbdd85-2cnnl" podUID="f4ae9624-d127-4a55-837d-7c9ff0267b37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:58:09.331352 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:08.957950 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/premium-w3-83755a-kserve-7c9d66988b-lt4xd" podUID="c0c6b02e-2ecb-428e-a13d-be6d7cd1ba39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Nov 03 00:58:09.331352 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:08.954451 4421 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-s4dc8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 00:58:09.331352 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:09.236149 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-s4dc8" podUID="ab5414b5-9d47-4c9f-9d0e-82242b9d6db2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 00:58:09.331352 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:09.236756 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-650be0/simulator-w4-650be0-kserve-7bc588774f-gvz2k" Nov 03 00:58:09.331352 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:09.236819 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-83755a/distinct-w3-83755a-kserve-86bdbbdd85-2cnnl" Nov 03 00:58:09.331352 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:09.236857 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-83755a/premium-w3-83755a-kserve-7c9d66988b-lt4xd" Nov 03 00:58:09.331352 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:09.287596 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-650be0/premium-w4-650be0-kserve-7bf8555dfb-z97n5" podUID="40601dc5-808f-456f-826f-6e0c35966ada" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Nov 03 00:58:09.331352 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:09.287884 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-650be0/premium-w4-650be0-kserve-7bf8555dfb-z97n5" Nov 03 00:58:09.331352 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:09.321137 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8gm8n" podUID="2d367e12-e46f-4a68-9825-9b1c934d4d12" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:58:09.422618 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:09.412030 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/simulator-w3-83755a-kserve-77d6664559-psqtg" podUID="b0a4c234-af58-4da0-a1a4-e1f167953f6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:58:09.473320 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:09.469474 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/simulator-w2-f7680d-kserve-85bdc86cd6-8gxrg" podUID="2bdcf521-cebd-4d6b-8554-1b60db8b0f3c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:58:09.473320 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:09.470570 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f7680d/simulator-w2-f7680d-kserve-85bdc86cd6-8gxrg" Nov 03 00:58:09.501566 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:09.496727 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:58:09.501566 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:09.496846 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:58:09.501566 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:09.496893 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:58:09.803207 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:09.802512 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-eed369-openshift-default-66698bcb95-pknkp"] Nov 03 00:58:10.240650 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:10.240526 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-650be0/simulator-w4-650be0-kserve-7bc588774f-gvz2k" podUID="a827fa88-f078-4312-8f12-2c799c11b461" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Nov 03 00:58:10.322360 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:10.319615 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-650be0/premium-w4-650be0-kserve-7bf8555dfb-z97n5" podUID="40601dc5-808f-456f-826f-6e0c35966ada" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Nov 03 00:58:10.378429 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:10.295398 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-54df68-59cb67b865-9d886" podUID="fc0615b0-2580-4c26-a028-616bc6042cc3" containerName="payload-processing" containerID="cri-o://7167c61602a6c290b365a586ea4e2a66b43d376ebc172fa8b7fc25e2e5a39657" gracePeriod=25 Nov 03 00:58:10.448330 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:10.434361 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8gm8n" podUID="2d367e12-e46f-4a68-9825-9b1c934d4d12" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:58:10.499655 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:10.474355 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/simulator-w2-f7680d-kserve-85bdc86cd6-8gxrg" podUID="2bdcf521-cebd-4d6b-8554-1b60db8b0f3c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:58:10.499655 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:10.476627 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/distinct-w3-83755a-kserve-86bdbbdd85-2cnnl" podUID="f4ae9624-d127-4a55-837d-7c9ff0267b37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded" Nov 03 00:58:10.499655 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:10.483124 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/premium-w3-83755a-kserve-7c9d66988b-lt4xd" podUID="c0c6b02e-2ecb-428e-a13d-be6d7cd1ba39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Nov 03 00:58:10.505091 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:10.505056 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/simulator-w3-83755a-kserve-77d6664559-psqtg" podUID="b0a4c234-af58-4da0-a1a4-e1f167953f6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:58:10.791413 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:10.790314 4421 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-f7680d/simulator-w2-f7680d-kserve-85bdc86cd6-8gxrg" podUID="2bdcf521-cebd-4d6b-8554-1b60db8b0f3c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:58:11.043739 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:11.043629 4421 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-83755a/premium-w3-83755a-kserve-7c9d66988b-lt4xd" podUID="c0c6b02e-2ecb-428e-a13d-be6d7cd1ba39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:58:11.083067 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:11.082847 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8gm8n" Nov 03 00:58:11.083419 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:11.083401 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-83755a/premium-w3-83755a-kserve-7c9d66988b-lt4xd" Nov 03 00:58:11.083580 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:11.083567 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f7680d/simulator-w2-f7680d-kserve-85bdc86cd6-8gxrg" Nov 03 00:58:11.083724 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:11.083713 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-650be0/premium-w4-650be0-kserve-7bf8555dfb-z97n5" Nov 03 00:58:11.083837 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:11.083827 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-83755a/distinct-w3-83755a-kserve-86bdbbdd85-2cnnl" Nov 03 00:58:11.083935 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:11.083916 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-eed369-openshift-default-66698bcb95-pknkp" event={"ID":"b797c4ed-b8df-4e23-9188-3aeb3cf6c48f","Type":"ContainerStarted","Data":"c478a537e618fb8d71c478fa5ca5bf32229158cc77be47271c4f611b1f348977"} Nov 03 00:58:11.084032 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:11.084021 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-eed369-openshift-default-66698bcb95-pknkp" event={"ID":"b797c4ed-b8df-4e23-9188-3aeb3cf6c48f","Type":"ContainerStarted","Data":"689fcb37f9f7b1d3f2a2c6c1cf87b8deded65d77670f30b5e5cf7b25a9f2a644"} Nov 03 00:58:11.084151 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:11.084140 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-83755a/simulator-w3-83755a-kserve-77d6664559-psqtg" Nov 03 00:58:11.091865 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:11.091839 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-650be0/simulator-w4-650be0-kserve-7bc588774f-gvz2k" Nov 03 00:58:11.190838 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:11.190803 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-eed369-openshift-default-66698bcb95-pknkp" Nov 03 00:58:11.193255 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:11.193220 4421 patch_prober.go:29] interesting pod/e2e-api-b-eed369-openshift-default-66698bcb95-pknkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body= Nov 03 00:58:11.193446 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:11.193290 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-eed369-openshift-default-66698bcb95-pknkp" podUID="b797c4ed-b8df-4e23-9188-3aeb3cf6c48f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" Nov 03 00:58:11.196790 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:11.196741 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-eed369-openshift-default-66698bcb95-pknkp" podStartSLOduration=5.196723426 podStartE2EDuration="5.196723426s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:58:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:58:11.16253519 +0000 UTC m=+1033.333538285" watchObservedRunningTime="2026-11-03 00:58:11.196723426 +0000 UTC m=+1033.367726524" Nov 03 00:58:11.816517 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:11.816479 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-54df68-59cb67b865-9d886" Nov 03 00:58:11.904923 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:11.904825 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc0615b0-2580-4c26-a028-616bc6042cc3-kube-api-access-qzcx2\" (UniqueName: \"kubernetes.io/projected/fc0615b0-2580-4c26-a028-616bc6042cc3-kube-api-access-qzcx2\") pod \"fc0615b0-2580-4c26-a028-616bc6042cc3\" (UID: \"fc0615b0-2580-4c26-a028-616bc6042cc3\") " Nov 03 00:58:11.905611 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:11.905587 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc0615b0-2580-4c26-a028-616bc6042cc3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc0615b0-2580-4c26-a028-616bc6042cc3-plugins-config-volume\") pod \"fc0615b0-2580-4c26-a028-616bc6042cc3\" (UID: \"fc0615b0-2580-4c26-a028-616bc6042cc3\") " Nov 03 00:58:11.906620 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:11.906117 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc0615b0-2580-4c26-a028-616bc6042cc3-plugins-config-volume" pod "fc0615b0-2580-4c26-a028-616bc6042cc3" (UID: "fc0615b0-2580-4c26-a028-616bc6042cc3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:58:11.907248 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:11.907225 4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc0615b0-2580-4c26-a028-616bc6042cc3-plugins-config-volume\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:58:12.181378 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:12.179521 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc0615b0-2580-4c26-a028-616bc6042cc3-kube-api-access-qzcx2" pod "fc0615b0-2580-4c26-a028-616bc6042cc3" (UID: "fc0615b0-2580-4c26-a028-616bc6042cc3"). InnerVolumeSpecName "kube-api-access-qzcx2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:58:12.284351 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:12.244622 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qzcx2\" (UniqueName: \"kubernetes.io/projected/fc0615b0-2580-4c26-a028-616bc6042cc3-kube-api-access-qzcx2\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:58:12.290310 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:12.290251 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-eed369-openshift-default-66698bcb95-pknkp" Nov 03 00:58:12.294609 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:12.294577 4421 generic.go:356] "Generic (PLEG): container finished" podID="fc0615b0-2580-4c26-a028-616bc6042cc3" containerID="7167c61602a6c290b365a586ea4e2a66b43d376ebc172fa8b7fc25e2e5a39657" exitCode=0 Nov 03 00:58:12.294871 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:12.294854 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-54df68-59cb67b865-9d886" Nov 03 00:58:12.294947 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:12.294910 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-54df68-59cb67b865-9d886" event={"ID":"fc0615b0-2580-4c26-a028-616bc6042cc3","Type":"ContainerDied","Data":"7167c61602a6c290b365a586ea4e2a66b43d376ebc172fa8b7fc25e2e5a39657"} Nov 03 00:58:12.295019 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:12.294951 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-54df68-59cb67b865-9d886" event={"ID":"fc0615b0-2580-4c26-a028-616bc6042cc3","Type":"ContainerDied","Data":"9a666883f95735ace5efff34d9804a6071ac12b40f737a35a6e16161e3d9ef77"} Nov 03 00:58:12.295019 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:12.294995 4421 scope.go:122] "RemoveContainer" containerID="7167c61602a6c290b365a586ea4e2a66b43d376ebc172fa8b7fc25e2e5a39657" Nov 03 00:58:12.303201 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:12.303172 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-eed369-openshift-default-66698bcb95-pknkp" Nov 03 00:58:12.304432 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:12.304409 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-eed369-openshift-default-66698bcb95-pknkp" Nov 03 00:58:12.344979 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:12.344948 4421 scope.go:122] "RemoveContainer" containerID="7167c61602a6c290b365a586ea4e2a66b43d376ebc172fa8b7fc25e2e5a39657" Nov 03 00:58:12.345900 ip-10-0-133-217 kubenswrapper[4421]: E1103 00:58:12.345800 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7167c61602a6c290b365a586ea4e2a66b43d376ebc172fa8b7fc25e2e5a39657\": container with ID starting with 7167c61602a6c290b365a586ea4e2a66b43d376ebc172fa8b7fc25e2e5a39657 not found: ID does not exist" containerID="7167c61602a6c290b365a586ea4e2a66b43d376ebc172fa8b7fc25e2e5a39657" Nov 03 00:58:12.345900 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:12.345850 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7167c61602a6c290b365a586ea4e2a66b43d376ebc172fa8b7fc25e2e5a39657"} err="failed to get container status \"7167c61602a6c290b365a586ea4e2a66b43d376ebc172fa8b7fc25e2e5a39657\": rpc error: code = NotFound desc = could not find container \"7167c61602a6c290b365a586ea4e2a66b43d376ebc172fa8b7fc25e2e5a39657\": container with ID starting with 7167c61602a6c290b365a586ea4e2a66b43d376ebc172fa8b7fc25e2e5a39657 not found: ID does not exist" Nov 03 00:58:12.376260 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:12.376221 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-54df68-59cb67b865-9d886"] Nov 03 00:58:12.383526 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:12.383472 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-54df68-59cb67b865-9d886"] Nov 03 00:58:13.985634 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:13.985589 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc0615b0-2580-4c26-a028-616bc6042cc3" path="/var/lib/kubelet/pods/fc0615b0-2580-4c26-a028-616bc6042cc3/volumes" Nov 03 00:58:14.139683 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:14.139642 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-54df68-openshift-default-74b6dbc6cf-4kmpk"] Nov 03 00:58:14.139969 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:14.139937 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-54df68-openshift-default-74b6dbc6cf-4kmpk" podUID="a6b01747-59d0-4ee6-9a96-4e3d1cad82df" containerName="istio-proxy" containerID="cri-o://fb41143b22956591601307bf987aa6e5f29323ac418b357d458ad6560ef0b918" gracePeriod=30 Nov 03 00:58:14.682362 ip-10-0-133-217 kubenswrapper[4421]: E1103 00:58:14.682322 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 00:58:15.299610 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:15.299575 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-eed369-688b4d87b7-wdrsx"] Nov 03 00:58:15.300493 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:15.300471 4421 cpu_manager.go:414] "containerMap: removing container" podUID="fc0615b0-2580-4c26-a028-616bc6042cc3" containerName="payload-processing" Nov 03 00:58:15.300560 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:15.300495 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc0615b0-2580-4c26-a028-616bc6042cc3" containerName="payload-processing" Nov 03 00:58:15.300680 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:15.300668 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc0615b0-2580-4c26-a028-616bc6042cc3" containerName="payload-processing" Nov 03 00:58:15.319019 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:15.317458 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-eed369-688b4d87b7-wdrsx"] Nov 03 00:58:15.319019 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:15.317623 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-eed369-688b4d87b7-wdrsx" Nov 03 00:58:15.328126 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:15.326591 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-eed369\"" Nov 03 00:58:15.328126 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:15.326665 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-eed369-dockercfg-z5xf5\"" Nov 03 00:58:15.451450 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:15.450791 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91b496b3-d1b2-475c-9ed5-a7ba4fe4d3a9-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-eed369-688b4d87b7-wdrsx\" (UID: \"91b496b3-d1b2-475c-9ed5-a7ba4fe4d3a9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-eed369-688b4d87b7-wdrsx" Nov 03 00:58:15.451450 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:15.451008 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nxfw\" (UniqueName: \"kubernetes.io/projected/91b496b3-d1b2-475c-9ed5-a7ba4fe4d3a9-kube-api-access-4nxfw\") pod \"payload-pre-processing-e2e-api-b-eed369-688b4d87b7-wdrsx\" (UID: \"91b496b3-d1b2-475c-9ed5-a7ba4fe4d3a9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-eed369-688b4d87b7-wdrsx" Nov 03 00:58:15.580652 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:15.580547 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91b496b3-d1b2-475c-9ed5-a7ba4fe4d3a9-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-eed369-688b4d87b7-wdrsx\" (UID: \"91b496b3-d1b2-475c-9ed5-a7ba4fe4d3a9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-eed369-688b4d87b7-wdrsx" Nov 03 00:58:15.580832 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:15.580744 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nxfw\" (UniqueName: \"kubernetes.io/projected/91b496b3-d1b2-475c-9ed5-a7ba4fe4d3a9-kube-api-access-4nxfw\") pod \"payload-pre-processing-e2e-api-b-eed369-688b4d87b7-wdrsx\" (UID: \"91b496b3-d1b2-475c-9ed5-a7ba4fe4d3a9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-eed369-688b4d87b7-wdrsx" Nov 03 00:58:15.612307 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:15.607390 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91b496b3-d1b2-475c-9ed5-a7ba4fe4d3a9-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-eed369-688b4d87b7-wdrsx\" (UID: \"91b496b3-d1b2-475c-9ed5-a7ba4fe4d3a9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-eed369-688b4d87b7-wdrsx" Nov 03 00:58:15.629109 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:15.629071 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nxfw\" (UniqueName: \"kubernetes.io/projected/91b496b3-d1b2-475c-9ed5-a7ba4fe4d3a9-kube-api-access-4nxfw\") pod \"payload-pre-processing-e2e-api-b-eed369-688b4d87b7-wdrsx\" (UID: \"91b496b3-d1b2-475c-9ed5-a7ba4fe4d3a9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-eed369-688b4d87b7-wdrsx" Nov 03 00:58:15.662672 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:15.662635 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-eed369-688b4d87b7-wdrsx" Nov 03 00:58:15.784455 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:15.784410 4421 patch_prober.go:29] interesting pod/e2e-api-a-340b96-openshift-default-7846d978b9-8zqfs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 00:58:15.784610 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:15.784481 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-340b96-openshift-default-7846d978b9-8zqfs" podUID="90e63fe7-52ef-4fbd-ab37-32c6479abb7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 00:58:16.053262 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:16.053228 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-eed369-688b4d87b7-wdrsx"] Nov 03 00:58:16.056215 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:58:16.056180 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91b496b3_d1b2_475c_9ed5_a7ba4fe4d3a9.slice/crio-5269ee9dfeb24e8663cd7385067a2ef8b43eaadf5185538be729d70a49bb9876 WatchSource:0}: Error finding container 5269ee9dfeb24e8663cd7385067a2ef8b43eaadf5185538be729d70a49bb9876: Status 404 returned error can't find the container with id 5269ee9dfeb24e8663cd7385067a2ef8b43eaadf5185538be729d70a49bb9876 Nov 03 00:58:16.380874 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:16.380789 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-eed369-688b4d87b7-wdrsx" event={"ID":"91b496b3-d1b2-475c-9ed5-a7ba4fe4d3a9","Type":"ContainerStarted","Data":"addccc54b9fb9e34fd686ca878c398e013ca166b2e27fc02ab6847dc1cb49049"} Nov 03 00:58:16.380874 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:16.380856 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-eed369-688b4d87b7-wdrsx" event={"ID":"91b496b3-d1b2-475c-9ed5-a7ba4fe4d3a9","Type":"ContainerStarted","Data":"5269ee9dfeb24e8663cd7385067a2ef8b43eaadf5185538be729d70a49bb9876"} Nov 03 00:58:16.381423 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:16.380901 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-eed369-688b4d87b7-wdrsx" Nov 03 00:58:16.399803 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:16.399742 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-eed369-688b4d87b7-wdrsx" podStartSLOduration=1.39972218 podStartE2EDuration="1.39972218s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:58:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:58:16.398055709 +0000 UTC m=+1038.569058807" watchObservedRunningTime="2026-11-03 00:58:16.39972218 +0000 UTC m=+1038.570725280" Nov 03 00:58:17.070757 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:17.070705 4421 patch_prober.go:29] interesting pod/e2e-api-a-54df68-openshift-default-74b6dbc6cf-4kmpk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 00:58:17.071035 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:17.070784 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-54df68-openshift-default-74b6dbc6cf-4kmpk" podUID="a6b01747-59d0-4ee6-9a96-4e3d1cad82df" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 00:58:17.086870 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:17.086835 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-07f5f6-78d57d48ff-tgdfc"] Nov 03 00:58:17.124151 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:17.124117 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-07f5f6-78d57d48ff-tgdfc" Nov 03 00:58:17.137611 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:17.137575 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-07f5f6-78d57d48ff-tgdfc"] Nov 03 00:58:17.212313 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:17.204751 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-07f5f6\"" Nov 03 00:58:17.212313 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:17.204921 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-07f5f6-dockercfg-q7wz8\"" Nov 03 00:58:17.288296 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:17.284824 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4wr2\" (UniqueName: \"kubernetes.io/projected/4f42ac06-fafe-49e4-bbd0-987fa37fe5ac-kube-api-access-p4wr2\") pod \"payload-pre-processing-e2e-ipp-a-07f5f6-78d57d48ff-tgdfc\" (UID: \"4f42ac06-fafe-49e4-bbd0-987fa37fe5ac\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-07f5f6-78d57d48ff-tgdfc" Nov 03 00:58:17.288296 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:17.284944 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f42ac06-fafe-49e4-bbd0-987fa37fe5ac-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-07f5f6-78d57d48ff-tgdfc\" (UID: \"4f42ac06-fafe-49e4-bbd0-987fa37fe5ac\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-07f5f6-78d57d48ff-tgdfc" Nov 03 00:58:17.406651 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:17.404666 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4wr2\" (UniqueName: \"kubernetes.io/projected/4f42ac06-fafe-49e4-bbd0-987fa37fe5ac-kube-api-access-p4wr2\") pod \"payload-pre-processing-e2e-ipp-a-07f5f6-78d57d48ff-tgdfc\" (UID: \"4f42ac06-fafe-49e4-bbd0-987fa37fe5ac\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-07f5f6-78d57d48ff-tgdfc" Nov 03 00:58:17.406651 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:17.404789 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f42ac06-fafe-49e4-bbd0-987fa37fe5ac-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-07f5f6-78d57d48ff-tgdfc\" (UID: \"4f42ac06-fafe-49e4-bbd0-987fa37fe5ac\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-07f5f6-78d57d48ff-tgdfc" Nov 03 00:58:17.431813 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:17.430021 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f42ac06-fafe-49e4-bbd0-987fa37fe5ac-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-07f5f6-78d57d48ff-tgdfc\" (UID: \"4f42ac06-fafe-49e4-bbd0-987fa37fe5ac\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-07f5f6-78d57d48ff-tgdfc" Nov 03 00:58:17.431813 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:17.430632 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4wr2\" (UniqueName: \"kubernetes.io/projected/4f42ac06-fafe-49e4-bbd0-987fa37fe5ac-kube-api-access-p4wr2\") pod \"payload-pre-processing-e2e-ipp-a-07f5f6-78d57d48ff-tgdfc\" (UID: \"4f42ac06-fafe-49e4-bbd0-987fa37fe5ac\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-07f5f6-78d57d48ff-tgdfc" Nov 03 00:58:17.569841 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:17.569802 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-07f5f6-78d57d48ff-tgdfc" Nov 03 00:58:17.872104 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:58:17.872065 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f42ac06_fafe_49e4_bbd0_987fa37fe5ac.slice/crio-1d19f2754f719725702fce8f67586d772875e84c954114f2f119bc41bdbcdcc7 WatchSource:0}: Error finding container 1d19f2754f719725702fce8f67586d772875e84c954114f2f119bc41bdbcdcc7: Status 404 returned error can't find the container with id 1d19f2754f719725702fce8f67586d772875e84c954114f2f119bc41bdbcdcc7 Nov 03 00:58:17.879129 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:17.875409 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-07f5f6-78d57d48ff-tgdfc"] Nov 03 00:58:18.450885 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:18.450857 4421 generic.go:356] "Generic (PLEG): container finished" podID="90e63fe7-52ef-4fbd-ab37-32c6479abb7a" containerID="4727b4aed6f9fe13c30ac83ef9e469d024a700aaf2f264aea71481ee101489f8" exitCode=0 Nov 03 00:58:18.451372 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:18.450932 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-340b96-openshift-default-7846d978b9-8zqfs" event={"ID":"90e63fe7-52ef-4fbd-ab37-32c6479abb7a","Type":"ContainerDied","Data":"4727b4aed6f9fe13c30ac83ef9e469d024a700aaf2f264aea71481ee101489f8"} Nov 03 00:58:18.451783 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:18.451761 4421 scope.go:122] "RemoveContainer" containerID="4727b4aed6f9fe13c30ac83ef9e469d024a700aaf2f264aea71481ee101489f8" Nov 03 00:58:18.452818 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:18.452794 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-07f5f6-78d57d48ff-tgdfc" event={"ID":"4f42ac06-fafe-49e4-bbd0-987fa37fe5ac","Type":"ContainerStarted","Data":"dc144c9412d9f56d657744743f0a7e67037f9380342c0b6299fe841a35b70545"} Nov 03 00:58:18.452939 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:18.452841 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-07f5f6-78d57d48ff-tgdfc" event={"ID":"4f42ac06-fafe-49e4-bbd0-987fa37fe5ac","Type":"ContainerStarted","Data":"1d19f2754f719725702fce8f67586d772875e84c954114f2f119bc41bdbcdcc7"} Nov 03 00:58:18.452939 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:18.452908 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-07f5f6-78d57d48ff-tgdfc" Nov 03 00:58:18.453548 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:18.453518 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:58:18.453621 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:18.453586 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:58:18.453661 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:18.453619 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:58:18.552168 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:18.552110 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-07f5f6-78d57d48ff-tgdfc" podStartSLOduration=1.5520934880000001 podStartE2EDuration="1.552093488s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:58:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:58:18.550362973 +0000 UTC m=+1040.721366072" watchObservedRunningTime="2026-11-03 00:58:18.552093488 +0000 UTC m=+1040.723096584" Nov 03 00:58:19.466138 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:19.465105 4421 generic.go:356] "Generic (PLEG): container finished" podID="0be0024f-24ee-4c93-81ef-8d9b3f2d0b69" containerID="167b6ecaafc9a01a8ed6a0bdf239b0f6094c4caaec4e7a06831e0eaf9ec7f37b" exitCode=0 Nov 03 00:58:19.466138 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:19.465185 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a2d8e5-openshift-default-85655d8485-f4q8h" event={"ID":"0be0024f-24ee-4c93-81ef-8d9b3f2d0b69","Type":"ContainerDied","Data":"167b6ecaafc9a01a8ed6a0bdf239b0f6094c4caaec4e7a06831e0eaf9ec7f37b"} Nov 03 00:58:19.466138 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:19.465774 4421 scope.go:122] "RemoveContainer" containerID="167b6ecaafc9a01a8ed6a0bdf239b0f6094c4caaec4e7a06831e0eaf9ec7f37b" Nov 03 00:58:19.468146 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:19.468103 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:58:19.468297 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:19.468199 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:58:19.468297 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:19.468288 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:58:19.474953 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:19.474919 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-340b96-openshift-default-7846d978b9-8zqfs" event={"ID":"90e63fe7-52ef-4fbd-ab37-32c6479abb7a","Type":"ContainerStarted","Data":"449161d828ac5cf3119b2d9c2232f2e6c7bfc9b3ca332c33155b19456a6492f5"} Nov 03 00:58:19.500161 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:19.500120 4421 generic.go:356] "Generic (PLEG): container finished" podID="a6b01747-59d0-4ee6-9a96-4e3d1cad82df" containerID="fb41143b22956591601307bf987aa6e5f29323ac418b357d458ad6560ef0b918" exitCode=0 Nov 03 00:58:19.501157 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:19.501055 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-54df68-openshift-default-74b6dbc6cf-4kmpk" event={"ID":"a6b01747-59d0-4ee6-9a96-4e3d1cad82df","Type":"ContainerDied","Data":"fb41143b22956591601307bf987aa6e5f29323ac418b357d458ad6560ef0b918"} Nov 03 00:58:19.501157 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:19.501118 4421 scope.go:122] "RemoveContainer" containerID="c47e742c228c773354e827269a6ee024a997b9cd9dabf084161398b1ef7e0f75" Nov 03 00:58:19.599401 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:19.599074 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-54df68-openshift-default-74b6dbc6cf-4kmpk" Nov 03 00:58:19.681202 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:19.680761 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a6b01747-59d0-4ee6-9a96-4e3d1cad82df-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6b01747-59d0-4ee6-9a96-4e3d1cad82df-istio-podinfo\") pod \"a6b01747-59d0-4ee6-9a96-4e3d1cad82df\" (UID: \"a6b01747-59d0-4ee6-9a96-4e3d1cad82df\") " Nov 03 00:58:19.681202 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:19.680836 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6b01747-59d0-4ee6-9a96-4e3d1cad82df-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6b01747-59d0-4ee6-9a96-4e3d1cad82df-credential-socket\") pod \"a6b01747-59d0-4ee6-9a96-4e3d1cad82df\" (UID: \"a6b01747-59d0-4ee6-9a96-4e3d1cad82df\") " Nov 03 00:58:19.681202 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:19.680869 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6b01747-59d0-4ee6-9a96-4e3d1cad82df-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6b01747-59d0-4ee6-9a96-4e3d1cad82df-istio-data\") pod \"a6b01747-59d0-4ee6-9a96-4e3d1cad82df\" (UID: \"a6b01747-59d0-4ee6-9a96-4e3d1cad82df\") " Nov 03 00:58:19.681202 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:19.680959 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6b01747-59d0-4ee6-9a96-4e3d1cad82df-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6b01747-59d0-4ee6-9a96-4e3d1cad82df-istio-envoy\") pod \"a6b01747-59d0-4ee6-9a96-4e3d1cad82df\" (UID: \"a6b01747-59d0-4ee6-9a96-4e3d1cad82df\") " Nov 03 00:58:19.681202 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:19.681003 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6b01747-59d0-4ee6-9a96-4e3d1cad82df-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6b01747-59d0-4ee6-9a96-4e3d1cad82df-istiod-ca-cert\") pod \"a6b01747-59d0-4ee6-9a96-4e3d1cad82df\" (UID: \"a6b01747-59d0-4ee6-9a96-4e3d1cad82df\") " Nov 03 00:58:19.681202 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:19.681104 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6b01747-59d0-4ee6-9a96-4e3d1cad82df-istio-token\" (UniqueName: \"kubernetes.io/projected/a6b01747-59d0-4ee6-9a96-4e3d1cad82df-istio-token\") pod \"a6b01747-59d0-4ee6-9a96-4e3d1cad82df\" (UID: \"a6b01747-59d0-4ee6-9a96-4e3d1cad82df\") " Nov 03 00:58:19.681202 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:19.681133 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6b01747-59d0-4ee6-9a96-4e3d1cad82df-kube-api-access-4j4lb\" (UniqueName: \"kubernetes.io/projected/a6b01747-59d0-4ee6-9a96-4e3d1cad82df-kube-api-access-4j4lb\") pod \"a6b01747-59d0-4ee6-9a96-4e3d1cad82df\" (UID: \"a6b01747-59d0-4ee6-9a96-4e3d1cad82df\") " Nov 03 00:58:19.681585 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:19.681216 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6b01747-59d0-4ee6-9a96-4e3d1cad82df-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6b01747-59d0-4ee6-9a96-4e3d1cad82df-workload-socket\") pod \"a6b01747-59d0-4ee6-9a96-4e3d1cad82df\" (UID: \"a6b01747-59d0-4ee6-9a96-4e3d1cad82df\") " Nov 03 00:58:19.681585 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:19.681423 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6b01747-59d0-4ee6-9a96-4e3d1cad82df-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6b01747-59d0-4ee6-9a96-4e3d1cad82df-workload-certs\") pod \"a6b01747-59d0-4ee6-9a96-4e3d1cad82df\" (UID: \"a6b01747-59d0-4ee6-9a96-4e3d1cad82df\") " Nov 03 00:58:19.683645 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:19.683433 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6b01747-59d0-4ee6-9a96-4e3d1cad82df-istio-data" pod "a6b01747-59d0-4ee6-9a96-4e3d1cad82df" (UID: "a6b01747-59d0-4ee6-9a96-4e3d1cad82df"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:58:19.683843 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:19.683524 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6b01747-59d0-4ee6-9a96-4e3d1cad82df-credential-socket" pod "a6b01747-59d0-4ee6-9a96-4e3d1cad82df" (UID: "a6b01747-59d0-4ee6-9a96-4e3d1cad82df"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:58:19.683843 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:19.683817 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6b01747-59d0-4ee6-9a96-4e3d1cad82df-workload-socket" pod "a6b01747-59d0-4ee6-9a96-4e3d1cad82df" (UID: "a6b01747-59d0-4ee6-9a96-4e3d1cad82df"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:58:19.684030 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:19.684002 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6b01747-59d0-4ee6-9a96-4e3d1cad82df-istiod-ca-cert" pod "a6b01747-59d0-4ee6-9a96-4e3d1cad82df" (UID: "a6b01747-59d0-4ee6-9a96-4e3d1cad82df"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:58:19.684097 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:19.684031 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6b01747-59d0-4ee6-9a96-4e3d1cad82df-workload-certs" pod "a6b01747-59d0-4ee6-9a96-4e3d1cad82df" (UID: "a6b01747-59d0-4ee6-9a96-4e3d1cad82df"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:58:19.684601 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:19.684579 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6b01747-59d0-4ee6-9a96-4e3d1cad82df-istio-token" pod "a6b01747-59d0-4ee6-9a96-4e3d1cad82df" (UID: "a6b01747-59d0-4ee6-9a96-4e3d1cad82df"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:58:19.684863 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:19.684843 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a6b01747-59d0-4ee6-9a96-4e3d1cad82df-istio-podinfo" pod "a6b01747-59d0-4ee6-9a96-4e3d1cad82df" (UID: "a6b01747-59d0-4ee6-9a96-4e3d1cad82df"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:58:19.684863 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:19.684851 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6b01747-59d0-4ee6-9a96-4e3d1cad82df-istio-envoy" pod "a6b01747-59d0-4ee6-9a96-4e3d1cad82df" (UID: "a6b01747-59d0-4ee6-9a96-4e3d1cad82df"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:58:19.686442 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:19.686412 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6b01747-59d0-4ee6-9a96-4e3d1cad82df-kube-api-access-4j4lb" pod "a6b01747-59d0-4ee6-9a96-4e3d1cad82df" (UID: "a6b01747-59d0-4ee6-9a96-4e3d1cad82df"). InnerVolumeSpecName "kube-api-access-4j4lb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:58:19.751810 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:19.751774 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-340b96-openshift-default-7846d978b9-8zqfs" Nov 03 00:58:19.752130 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:19.752105 4421 patch_prober.go:29] interesting pod/e2e-api-a-340b96-openshift-default-7846d978b9-8zqfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Nov 03 00:58:19.752345 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:19.752315 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-340b96-openshift-default-7846d978b9-8zqfs" podUID="90e63fe7-52ef-4fbd-ab37-32c6479abb7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Nov 03 00:58:19.784306 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:19.783568 4421 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6b01747-59d0-4ee6-9a96-4e3d1cad82df-istio-podinfo\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:58:19.784306 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:19.783601 4421 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6b01747-59d0-4ee6-9a96-4e3d1cad82df-credential-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:58:19.784306 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:19.783615 4421 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6b01747-59d0-4ee6-9a96-4e3d1cad82df-istio-data\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:58:19.784306 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:19.783631 4421 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6b01747-59d0-4ee6-9a96-4e3d1cad82df-istio-envoy\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:58:19.784306 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:19.783645 4421 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6b01747-59d0-4ee6-9a96-4e3d1cad82df-istiod-ca-cert\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:58:19.784306 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:19.783657 4421 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6b01747-59d0-4ee6-9a96-4e3d1cad82df-istio-token\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:58:19.784306 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:19.783671 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4j4lb\" (UniqueName: \"kubernetes.io/projected/a6b01747-59d0-4ee6-9a96-4e3d1cad82df-kube-api-access-4j4lb\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:58:19.784306 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:19.783685 4421 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6b01747-59d0-4ee6-9a96-4e3d1cad82df-workload-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:58:19.784306 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:19.783700 4421 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6b01747-59d0-4ee6-9a96-4e3d1cad82df-workload-certs\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:58:20.052053 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:20.051998 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77e7728f-74k8r"] Nov 03 00:58:20.057320 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:20.057290 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77e7728f-74k8r"] Nov 03 00:58:20.539414 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:20.539368 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-54df68-openshift-default-74b6dbc6cf-4kmpk" event={"ID":"a6b01747-59d0-4ee6-9a96-4e3d1cad82df","Type":"ContainerDied","Data":"c2daa186c29d0ad3c3d45918d9190b6e659428eb51410838703e92a9321fa9dd"} Nov 03 00:58:20.539414 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:20.539402 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-54df68-openshift-default-74b6dbc6cf-4kmpk" Nov 03 00:58:20.539996 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:20.539476 4421 scope.go:122] "RemoveContainer" containerID="fb41143b22956591601307bf987aa6e5f29323ac418b357d458ad6560ef0b918" Nov 03 00:58:20.541961 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:20.541934 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a2d8e5-f6bfcf94c-4jbsh"] Nov 03 00:58:20.544609 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:20.544582 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a2d8e5-openshift-default-85655d8485-f4q8h" event={"ID":"0be0024f-24ee-4c93-81ef-8d9b3f2d0b69","Type":"ContainerStarted","Data":"c146927fc9f30705a21dab905ddca8e4c3868db19bb45a898cf8e37692a299f9"} Nov 03 00:58:20.591954 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:20.591352 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-54df68-openshift-default-74b6dbc6cf-4kmpk"] Nov 03 00:58:20.599679 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:20.599639 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-54df68-openshift-default-74b6dbc6cf-4kmpk"] Nov 03 00:58:20.747252 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:20.747207 4421 patch_prober.go:29] interesting pod/e2e-api-a-340b96-openshift-default-7846d978b9-8zqfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Nov 03 00:58:20.747469 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:20.747326 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-340b96-openshift-default-7846d978b9-8zqfs" podUID="90e63fe7-52ef-4fbd-ab37-32c6479abb7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Nov 03 00:58:20.911515 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:20.910999 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-a2d8e5-openshift-default-85655d8485-f4q8h" Nov 03 00:58:20.911515 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:20.911138 4421 patch_prober.go:29] interesting pod/e2e-api-b-a2d8e5-openshift-default-85655d8485-f4q8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Nov 03 00:58:20.911515 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:20.911186 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a2d8e5-openshift-default-85655d8485-f4q8h" podUID="0be0024f-24ee-4c93-81ef-8d9b3f2d0b69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Nov 03 00:58:21.491586 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:21.491550 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-a2d8e5-openshift-default-85655d8485-f4q8h" Nov 03 00:58:21.763477 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:21.763429 4421 patch_prober.go:29] interesting pod/e2e-api-a-340b96-openshift-default-7846d978b9-8zqfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Nov 03 00:58:21.764165 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:21.764125 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-340b96-openshift-default-7846d978b9-8zqfs" podUID="90e63fe7-52ef-4fbd-ab37-32c6479abb7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Nov 03 00:58:21.910165 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:21.909703 4421 patch_prober.go:29] interesting pod/e2e-api-b-a2d8e5-openshift-default-85655d8485-f4q8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Nov 03 00:58:21.910165 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:21.909782 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a2d8e5-openshift-default-85655d8485-f4q8h" podUID="0be0024f-24ee-4c93-81ef-8d9b3f2d0b69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Nov 03 00:58:21.981675 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:21.981639 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6b01747-59d0-4ee6-9a96-4e3d1cad82df" path="/var/lib/kubelet/pods/a6b01747-59d0-4ee6-9a96-4e3d1cad82df/volumes" Nov 03 00:58:21.982225 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:21.982173 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cce43806-c404-486a-b604-8bdae6dd5986" path="/var/lib/kubelet/pods/cce43806-c404-486a-b604-8bdae6dd5986/volumes" Nov 03 00:58:22.751300 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:22.747786 4421 patch_prober.go:29] interesting pod/e2e-api-a-340b96-openshift-default-7846d978b9-8zqfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Nov 03 00:58:22.751300 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:22.747861 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-340b96-openshift-default-7846d978b9-8zqfs" podUID="90e63fe7-52ef-4fbd-ab37-32c6479abb7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Nov 03 00:58:22.917298 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:22.913837 4421 patch_prober.go:29] interesting pod/e2e-api-b-a2d8e5-openshift-default-85655d8485-f4q8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Nov 03 00:58:22.917298 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:22.913917 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a2d8e5-openshift-default-85655d8485-f4q8h" podUID="0be0024f-24ee-4c93-81ef-8d9b3f2d0b69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Nov 03 00:58:23.749391 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:23.747696 4421 patch_prober.go:29] interesting pod/e2e-api-a-340b96-openshift-default-7846d978b9-8zqfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Nov 03 00:58:23.749391 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:23.747779 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-340b96-openshift-default-7846d978b9-8zqfs" podUID="90e63fe7-52ef-4fbd-ab37-32c6479abb7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Nov 03 00:58:23.935297 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:23.913773 4421 patch_prober.go:29] interesting pod/e2e-api-b-a2d8e5-openshift-default-85655d8485-f4q8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Nov 03 00:58:23.935297 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:23.913886 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a2d8e5-openshift-default-85655d8485-f4q8h" podUID="0be0024f-24ee-4c93-81ef-8d9b3f2d0b69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Nov 03 00:58:24.749690 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:24.748091 4421 patch_prober.go:29] interesting pod/e2e-api-a-340b96-openshift-default-7846d978b9-8zqfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Nov 03 00:58:24.749690 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:24.748154 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-340b96-openshift-default-7846d978b9-8zqfs" podUID="90e63fe7-52ef-4fbd-ab37-32c6479abb7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Nov 03 00:58:24.913303 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:24.909835 4421 patch_prober.go:29] interesting pod/e2e-api-b-a2d8e5-openshift-default-85655d8485-f4q8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Nov 03 00:58:24.913303 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:24.909918 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a2d8e5-openshift-default-85655d8485-f4q8h" podUID="0be0024f-24ee-4c93-81ef-8d9b3f2d0b69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Nov 03 00:58:25.542674 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:25.542630 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a2d8e5-f6bfcf94c-4jbsh" podUID="3521b6ee-8255-499f-ac8f-b32249341eb2" containerName="payload-pre-processing" containerID="cri-o://a99a86d166d167294e3d6107a639e83f061d2db5d91e9f157d4243549e56be32" gracePeriod=25 Nov 03 00:58:25.722144 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:25.721788 4421 generic.go:356] "Generic (PLEG): container finished" podID="3521b6ee-8255-499f-ac8f-b32249341eb2" containerID="a99a86d166d167294e3d6107a639e83f061d2db5d91e9f157d4243549e56be32" exitCode=0 Nov 03 00:58:25.722144 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:25.722063 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a2d8e5-f6bfcf94c-4jbsh" event={"ID":"3521b6ee-8255-499f-ac8f-b32249341eb2","Type":"ContainerDied","Data":"a99a86d166d167294e3d6107a639e83f061d2db5d91e9f157d4243549e56be32"} Nov 03 00:58:25.761027 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:25.760324 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-340b96-openshift-default-7846d978b9-8zqfs" Nov 03 00:58:25.761181 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:25.761152 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-340b96-openshift-default-7846d978b9-8zqfs" Nov 03 00:58:25.761984 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:25.761961 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-340b96-openshift-default-7846d978b9-8zqfs" Nov 03 00:58:25.888420 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:25.888392 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a2d8e5-f6bfcf94c-4jbsh" Nov 03 00:58:25.949605 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:25.949566 4421 patch_prober.go:29] interesting pod/e2e-api-b-a2d8e5-openshift-default-85655d8485-f4q8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Nov 03 00:58:25.949769 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:25.949643 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a2d8e5-openshift-default-85655d8485-f4q8h" podUID="0be0024f-24ee-4c93-81ef-8d9b3f2d0b69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Nov 03 00:58:26.063207 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:26.060493 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3521b6ee-8255-499f-ac8f-b32249341eb2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3521b6ee-8255-499f-ac8f-b32249341eb2-plugins-config-volume\") pod \"3521b6ee-8255-499f-ac8f-b32249341eb2\" (UID: \"3521b6ee-8255-499f-ac8f-b32249341eb2\") " Nov 03 00:58:26.063207 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:26.060553 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3521b6ee-8255-499f-ac8f-b32249341eb2-kube-api-access-rpgmc\" (UniqueName: \"kubernetes.io/projected/3521b6ee-8255-499f-ac8f-b32249341eb2-kube-api-access-rpgmc\") pod \"3521b6ee-8255-499f-ac8f-b32249341eb2\" (UID: \"3521b6ee-8255-499f-ac8f-b32249341eb2\") " Nov 03 00:58:26.071562 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:26.071505 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3521b6ee-8255-499f-ac8f-b32249341eb2-plugins-config-volume" pod "3521b6ee-8255-499f-ac8f-b32249341eb2" (UID: "3521b6ee-8255-499f-ac8f-b32249341eb2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:58:26.076781 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:26.076740 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3521b6ee-8255-499f-ac8f-b32249341eb2-kube-api-access-rpgmc" pod "3521b6ee-8255-499f-ac8f-b32249341eb2" (UID: "3521b6ee-8255-499f-ac8f-b32249341eb2"). InnerVolumeSpecName "kube-api-access-rpgmc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:58:26.163768 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:26.163694 4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3521b6ee-8255-499f-ac8f-b32249341eb2-plugins-config-volume\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:58:26.163768 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:26.163721 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rpgmc\" (UniqueName: \"kubernetes.io/projected/3521b6ee-8255-499f-ac8f-b32249341eb2-kube-api-access-rpgmc\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:58:26.879032 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:26.879007 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a2d8e5-f6bfcf94c-4jbsh" Nov 03 00:58:26.879700 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:26.878963 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a2d8e5-f6bfcf94c-4jbsh" event={"ID":"3521b6ee-8255-499f-ac8f-b32249341eb2","Type":"ContainerDied","Data":"823bbbaadb217d36e373e893a183dfb794241ffc74ee5f8ddaf8ca934871cd51"} Nov 03 00:58:26.880235 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:26.880217 4421 scope.go:122] "RemoveContainer" containerID="a99a86d166d167294e3d6107a639e83f061d2db5d91e9f157d4243549e56be32" Nov 03 00:58:26.941298 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:26.919931 4421 patch_prober.go:29] interesting pod/e2e-api-b-a2d8e5-openshift-default-85655d8485-f4q8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Nov 03 00:58:26.941625 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:26.941584 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a2d8e5-openshift-default-85655d8485-f4q8h" podUID="0be0024f-24ee-4c93-81ef-8d9b3f2d0b69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Nov 03 00:58:27.064391 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:27.064343 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a2d8e5-f6bfcf94c-4jbsh"] Nov 03 00:58:27.067423 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:27.067394 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a2d8e5-f6bfcf94c-4jbsh"] Nov 03 00:58:27.442253 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:27.442224 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-eed369-688b4d87b7-wdrsx" Nov 03 00:58:27.911930 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:27.911898 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-a2d8e5-openshift-default-85655d8485-f4q8h" Nov 03 00:58:27.913135 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:27.913101 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-a2d8e5-openshift-default-85655d8485-f4q8h" Nov 03 00:58:27.983983 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:27.983939 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3521b6ee-8255-499f-ac8f-b32249341eb2" path="/var/lib/kubelet/pods/3521b6ee-8255-499f-ac8f-b32249341eb2/volumes" Nov 03 00:58:28.901323 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:28.899266 4421 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-api-b-a2d8e5-openshift-default-85655d8485-f4q8h" secret="" err="secret \"e2e-api-b-a2d8e5-openshift-default-dockercfg-prbqw\" not found" Nov 03 00:58:29.000224 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:29.000018 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-a2d8e5-openshift-default-85655d8485-f4q8h"] Nov 03 00:58:29.504168 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:29.504139 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-07f5f6-78d57d48ff-tgdfc" Nov 03 00:58:30.098787 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:30.098739 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-a2d8e5-openshift-default-85655d8485-f4q8h" podUID="0be0024f-24ee-4c93-81ef-8d9b3f2d0b69" containerName="istio-proxy" containerID="cri-o://c146927fc9f30705a21dab905ddca8e4c3868db19bb45a898cf8e37692a299f9" gracePeriod=30 Nov 03 00:58:30.301264 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:30.301220 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-7f43f9-openshift-default-74496bd585-lp65l"] Nov 03 00:58:30.302650 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:30.302613 4421 cpu_manager.go:414] "containerMap: removing container" podUID="3521b6ee-8255-499f-ac8f-b32249341eb2" containerName="payload-pre-processing" Nov 03 00:58:30.302841 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:30.302825 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3521b6ee-8255-499f-ac8f-b32249341eb2" containerName="payload-pre-processing" Nov 03 00:58:30.302963 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:30.302950 4421 cpu_manager.go:414] "containerMap: removing container" podUID="a6b01747-59d0-4ee6-9a96-4e3d1cad82df" containerName="istio-proxy" Nov 03 00:58:30.303059 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:30.303047 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6b01747-59d0-4ee6-9a96-4e3d1cad82df" containerName="istio-proxy" Nov 03 00:58:30.303171 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:30.303160 4421 cpu_manager.go:414] "containerMap: removing container" podUID="a6b01747-59d0-4ee6-9a96-4e3d1cad82df" containerName="istio-proxy" Nov 03 00:58:30.303268 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:30.303256 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6b01747-59d0-4ee6-9a96-4e3d1cad82df" containerName="istio-proxy" Nov 03 00:58:30.303635 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:30.303615 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="3521b6ee-8255-499f-ac8f-b32249341eb2" containerName="payload-pre-processing" Nov 03 00:58:30.303747 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:30.303735 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6b01747-59d0-4ee6-9a96-4e3d1cad82df" containerName="istio-proxy" Nov 03 00:58:30.303844 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:30.303834 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6b01747-59d0-4ee6-9a96-4e3d1cad82df" containerName="istio-proxy" Nov 03 00:58:30.364813 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:30.364732 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-7f43f9-openshift-default-74496bd585-lp65l" Nov 03 00:58:30.374705 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:30.374673 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-7f43f9-openshift-default-dockercfg-q62rv\"" Nov 03 00:58:30.386245 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:30.386212 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-7f43f9-openshift-default-74496bd585-lp65l"] Nov 03 00:58:30.405171 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:30.404778 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd400742-fae4-44b7-b4af-497f60943da9-credential-socket\") pod \"e2e-ipp-a-7f43f9-openshift-default-74496bd585-lp65l\" (UID: \"bd400742-fae4-44b7-b4af-497f60943da9\") " pod="openshift-ingress/e2e-ipp-a-7f43f9-openshift-default-74496bd585-lp65l" Nov 03 00:58:30.405171 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:30.404912 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd400742-fae4-44b7-b4af-497f60943da9-istiod-ca-cert\") pod \"e2e-ipp-a-7f43f9-openshift-default-74496bd585-lp65l\" (UID: \"bd400742-fae4-44b7-b4af-497f60943da9\") " pod="openshift-ingress/e2e-ipp-a-7f43f9-openshift-default-74496bd585-lp65l" Nov 03 00:58:30.405171 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:30.404979 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxffw\" (UniqueName: \"kubernetes.io/projected/bd400742-fae4-44b7-b4af-497f60943da9-kube-api-access-cxffw\") pod \"e2e-ipp-a-7f43f9-openshift-default-74496bd585-lp65l\" (UID: \"bd400742-fae4-44b7-b4af-497f60943da9\") " pod="openshift-ingress/e2e-ipp-a-7f43f9-openshift-default-74496bd585-lp65l" Nov 03 00:58:30.405171 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:30.405038 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd400742-fae4-44b7-b4af-497f60943da9-istio-podinfo\") pod \"e2e-ipp-a-7f43f9-openshift-default-74496bd585-lp65l\" (UID: \"bd400742-fae4-44b7-b4af-497f60943da9\") " pod="openshift-ingress/e2e-ipp-a-7f43f9-openshift-default-74496bd585-lp65l" Nov 03 00:58:30.405171 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:30.405076 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd400742-fae4-44b7-b4af-497f60943da9-workload-certs\") pod \"e2e-ipp-a-7f43f9-openshift-default-74496bd585-lp65l\" (UID: \"bd400742-fae4-44b7-b4af-497f60943da9\") " pod="openshift-ingress/e2e-ipp-a-7f43f9-openshift-default-74496bd585-lp65l" Nov 03 00:58:30.405576 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:30.405192 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd400742-fae4-44b7-b4af-497f60943da9-istio-envoy\") pod \"e2e-ipp-a-7f43f9-openshift-default-74496bd585-lp65l\" (UID: \"bd400742-fae4-44b7-b4af-497f60943da9\") " pod="openshift-ingress/e2e-ipp-a-7f43f9-openshift-default-74496bd585-lp65l" Nov 03 00:58:30.405576 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:30.405253 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd400742-fae4-44b7-b4af-497f60943da9-workload-socket\") pod \"e2e-ipp-a-7f43f9-openshift-default-74496bd585-lp65l\" (UID: \"bd400742-fae4-44b7-b4af-497f60943da9\") " pod="openshift-ingress/e2e-ipp-a-7f43f9-openshift-default-74496bd585-lp65l" Nov 03 00:58:30.405576 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:30.405333 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd400742-fae4-44b7-b4af-497f60943da9-istio-token\") pod \"e2e-ipp-a-7f43f9-openshift-default-74496bd585-lp65l\" (UID: \"bd400742-fae4-44b7-b4af-497f60943da9\") " pod="openshift-ingress/e2e-ipp-a-7f43f9-openshift-default-74496bd585-lp65l" Nov 03 00:58:30.405576 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:30.405428 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd400742-fae4-44b7-b4af-497f60943da9-istio-data\") pod \"e2e-ipp-a-7f43f9-openshift-default-74496bd585-lp65l\" (UID: \"bd400742-fae4-44b7-b4af-497f60943da9\") " pod="openshift-ingress/e2e-ipp-a-7f43f9-openshift-default-74496bd585-lp65l" Nov 03 00:58:30.515040 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:30.514890 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd400742-fae4-44b7-b4af-497f60943da9-istio-data\") pod \"e2e-ipp-a-7f43f9-openshift-default-74496bd585-lp65l\" (UID: \"bd400742-fae4-44b7-b4af-497f60943da9\") " pod="openshift-ingress/e2e-ipp-a-7f43f9-openshift-default-74496bd585-lp65l" Nov 03 00:58:30.515427 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:30.515400 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd400742-fae4-44b7-b4af-497f60943da9-credential-socket\") pod \"e2e-ipp-a-7f43f9-openshift-default-74496bd585-lp65l\" (UID: \"bd400742-fae4-44b7-b4af-497f60943da9\") " pod="openshift-ingress/e2e-ipp-a-7f43f9-openshift-default-74496bd585-lp65l" Nov 03 00:58:30.515566 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:30.515550 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd400742-fae4-44b7-b4af-497f60943da9-istiod-ca-cert\") pod \"e2e-ipp-a-7f43f9-openshift-default-74496bd585-lp65l\" (UID: \"bd400742-fae4-44b7-b4af-497f60943da9\") " pod="openshift-ingress/e2e-ipp-a-7f43f9-openshift-default-74496bd585-lp65l" Nov 03 00:58:30.515703 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:30.515684 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxffw\" (UniqueName: \"kubernetes.io/projected/bd400742-fae4-44b7-b4af-497f60943da9-kube-api-access-cxffw\") pod \"e2e-ipp-a-7f43f9-openshift-default-74496bd585-lp65l\" (UID: \"bd400742-fae4-44b7-b4af-497f60943da9\") " pod="openshift-ingress/e2e-ipp-a-7f43f9-openshift-default-74496bd585-lp65l" Nov 03 00:58:30.515817 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:30.515798 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd400742-fae4-44b7-b4af-497f60943da9-istio-podinfo\") pod \"e2e-ipp-a-7f43f9-openshift-default-74496bd585-lp65l\" (UID: \"bd400742-fae4-44b7-b4af-497f60943da9\") " pod="openshift-ingress/e2e-ipp-a-7f43f9-openshift-default-74496bd585-lp65l" Nov 03 00:58:30.515904 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:30.515833 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd400742-fae4-44b7-b4af-497f60943da9-workload-certs\") pod \"e2e-ipp-a-7f43f9-openshift-default-74496bd585-lp65l\" (UID: \"bd400742-fae4-44b7-b4af-497f60943da9\") " pod="openshift-ingress/e2e-ipp-a-7f43f9-openshift-default-74496bd585-lp65l" Nov 03 00:58:30.515962 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:30.515950 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd400742-fae4-44b7-b4af-497f60943da9-istio-envoy\") pod \"e2e-ipp-a-7f43f9-openshift-default-74496bd585-lp65l\" (UID: \"bd400742-fae4-44b7-b4af-497f60943da9\") " pod="openshift-ingress/e2e-ipp-a-7f43f9-openshift-default-74496bd585-lp65l" Nov 03 00:58:30.516015 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:30.516003 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd400742-fae4-44b7-b4af-497f60943da9-workload-socket\") pod \"e2e-ipp-a-7f43f9-openshift-default-74496bd585-lp65l\" (UID: \"bd400742-fae4-44b7-b4af-497f60943da9\") " pod="openshift-ingress/e2e-ipp-a-7f43f9-openshift-default-74496bd585-lp65l" Nov 03 00:58:30.516426 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:30.516059 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd400742-fae4-44b7-b4af-497f60943da9-istio-token\") pod \"e2e-ipp-a-7f43f9-openshift-default-74496bd585-lp65l\" (UID: \"bd400742-fae4-44b7-b4af-497f60943da9\") " pod="openshift-ingress/e2e-ipp-a-7f43f9-openshift-default-74496bd585-lp65l" Nov 03 00:58:30.516890 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:30.516866 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd400742-fae4-44b7-b4af-497f60943da9-istio-data\") pod \"e2e-ipp-a-7f43f9-openshift-default-74496bd585-lp65l\" (UID: \"bd400742-fae4-44b7-b4af-497f60943da9\") " pod="openshift-ingress/e2e-ipp-a-7f43f9-openshift-default-74496bd585-lp65l" Nov 03 00:58:30.517162 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:30.517137 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd400742-fae4-44b7-b4af-497f60943da9-credential-socket\") pod \"e2e-ipp-a-7f43f9-openshift-default-74496bd585-lp65l\" (UID: \"bd400742-fae4-44b7-b4af-497f60943da9\") " pod="openshift-ingress/e2e-ipp-a-7f43f9-openshift-default-74496bd585-lp65l" Nov 03 00:58:30.518161 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:30.518140 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd400742-fae4-44b7-b4af-497f60943da9-workload-certs\") pod \"e2e-ipp-a-7f43f9-openshift-default-74496bd585-lp65l\" (UID: \"bd400742-fae4-44b7-b4af-497f60943da9\") " pod="openshift-ingress/e2e-ipp-a-7f43f9-openshift-default-74496bd585-lp65l" Nov 03 00:58:30.558373 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:30.522183 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd400742-fae4-44b7-b4af-497f60943da9-istio-podinfo\") pod \"e2e-ipp-a-7f43f9-openshift-default-74496bd585-lp65l\" (UID: \"bd400742-fae4-44b7-b4af-497f60943da9\") " pod="openshift-ingress/e2e-ipp-a-7f43f9-openshift-default-74496bd585-lp65l" Nov 03 00:58:30.558373 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:30.523183 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd400742-fae4-44b7-b4af-497f60943da9-istiod-ca-cert\") pod \"e2e-ipp-a-7f43f9-openshift-default-74496bd585-lp65l\" (UID: \"bd400742-fae4-44b7-b4af-497f60943da9\") " pod="openshift-ingress/e2e-ipp-a-7f43f9-openshift-default-74496bd585-lp65l" Nov 03 00:58:30.558373 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:30.523793 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd400742-fae4-44b7-b4af-497f60943da9-workload-socket\") pod \"e2e-ipp-a-7f43f9-openshift-default-74496bd585-lp65l\" (UID: \"bd400742-fae4-44b7-b4af-497f60943da9\") " pod="openshift-ingress/e2e-ipp-a-7f43f9-openshift-default-74496bd585-lp65l" Nov 03 00:58:30.561481 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:30.561407 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd400742-fae4-44b7-b4af-497f60943da9-istio-envoy\") pod \"e2e-ipp-a-7f43f9-openshift-default-74496bd585-lp65l\" (UID: \"bd400742-fae4-44b7-b4af-497f60943da9\") " pod="openshift-ingress/e2e-ipp-a-7f43f9-openshift-default-74496bd585-lp65l" Nov 03 00:58:30.603850 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:30.603815 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd400742-fae4-44b7-b4af-497f60943da9-istio-token\") pod \"e2e-ipp-a-7f43f9-openshift-default-74496bd585-lp65l\" (UID: \"bd400742-fae4-44b7-b4af-497f60943da9\") " pod="openshift-ingress/e2e-ipp-a-7f43f9-openshift-default-74496bd585-lp65l" Nov 03 00:58:30.605049 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:30.604941 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxffw\" (UniqueName: \"kubernetes.io/projected/bd400742-fae4-44b7-b4af-497f60943da9-kube-api-access-cxffw\") pod \"e2e-ipp-a-7f43f9-openshift-default-74496bd585-lp65l\" (UID: \"bd400742-fae4-44b7-b4af-497f60943da9\") " pod="openshift-ingress/e2e-ipp-a-7f43f9-openshift-default-74496bd585-lp65l" Nov 03 00:58:30.722874 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:30.722782 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-7f43f9-openshift-default-74496bd585-lp65l" Nov 03 00:58:31.027985 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:31.027933 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e110f7cb-fzlzx"] Nov 03 00:58:31.044915 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:31.044849 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e110f7cb-fzlzx"] Nov 03 00:58:31.045810 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:58:31.045774 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd400742_fae4_44b7_b4af_497f60943da9.slice/crio-89715d8d27305ad07925c8e3c86d14bfee13984c87b2beeb2ff9e7f98d84a273 WatchSource:0}: Error finding container 89715d8d27305ad07925c8e3c86d14bfee13984c87b2beeb2ff9e7f98d84a273: Status 404 returned error can't find the container with id 89715d8d27305ad07925c8e3c86d14bfee13984c87b2beeb2ff9e7f98d84a273 Nov 03 00:58:31.050456 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:31.050379 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:58:31.050571 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:31.050486 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:58:31.050571 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:31.050533 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:58:31.051223 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:31.051201 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-7f43f9-openshift-default-74496bd585-lp65l"] Nov 03 00:58:31.113862 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:31.113813 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-7f43f9-openshift-default-74496bd585-lp65l" event={"ID":"bd400742-fae4-44b7-b4af-497f60943da9","Type":"ContainerStarted","Data":"89715d8d27305ad07925c8e3c86d14bfee13984c87b2beeb2ff9e7f98d84a273"} Nov 03 00:58:31.979336 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:31.979295 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff16a99b-497e-4503-ab96-d5e7063e679e" path="/var/lib/kubelet/pods/ff16a99b-497e-4503-ab96-d5e7063e679e/volumes" Nov 03 00:58:32.123526 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:32.123474 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-7f43f9-openshift-default-74496bd585-lp65l" event={"ID":"bd400742-fae4-44b7-b4af-497f60943da9","Type":"ContainerStarted","Data":"94b7c26d9b38f76f76ace804b3d83682116eae5a9a17db4d28bee6ac817f8911"} Nov 03 00:58:32.149381 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:32.149333 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-7f43f9-openshift-default-74496bd585-lp65l" podStartSLOduration=2.149320185 podStartE2EDuration="2.149320185s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:58:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:58:32.143559402 +0000 UTC m=+1054.314562512" watchObservedRunningTime="2026-11-03 00:58:32.149320185 +0000 UTC m=+1054.320323281" Nov 03 00:58:32.723624 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:32.723588 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-7f43f9-openshift-default-74496bd585-lp65l" Nov 03 00:58:32.729906 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:32.729876 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-7f43f9-openshift-default-74496bd585-lp65l" Nov 03 00:58:33.131902 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:33.131867 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-7f43f9-openshift-default-74496bd585-lp65l" Nov 03 00:58:33.132849 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:33.132830 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-7f43f9-openshift-default-74496bd585-lp65l" Nov 03 00:58:35.630971 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:35.630942 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-a2d8e5-openshift-default-85655d8485-f4q8h" Nov 03 00:58:35.771555 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:35.771521 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0be0024f-24ee-4c93-81ef-8d9b3f2d0b69-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0be0024f-24ee-4c93-81ef-8d9b3f2d0b69-istio-data\") pod \"0be0024f-24ee-4c93-81ef-8d9b3f2d0b69\" (UID: \"0be0024f-24ee-4c93-81ef-8d9b3f2d0b69\") " Nov 03 00:58:35.771731 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:35.771669 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0be0024f-24ee-4c93-81ef-8d9b3f2d0b69-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0be0024f-24ee-4c93-81ef-8d9b3f2d0b69-istio-envoy\") pod \"0be0024f-24ee-4c93-81ef-8d9b3f2d0b69\" (UID: \"0be0024f-24ee-4c93-81ef-8d9b3f2d0b69\") " Nov 03 00:58:35.771731 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:35.771706 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0be0024f-24ee-4c93-81ef-8d9b3f2d0b69-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0be0024f-24ee-4c93-81ef-8d9b3f2d0b69-credential-socket\") pod \"0be0024f-24ee-4c93-81ef-8d9b3f2d0b69\" (UID: \"0be0024f-24ee-4c93-81ef-8d9b3f2d0b69\") " Nov 03 00:58:35.771830 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:35.771767 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0be0024f-24ee-4c93-81ef-8d9b3f2d0b69-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0be0024f-24ee-4c93-81ef-8d9b3f2d0b69-istio-podinfo\") pod \"0be0024f-24ee-4c93-81ef-8d9b3f2d0b69\" (UID: \"0be0024f-24ee-4c93-81ef-8d9b3f2d0b69\") " Nov 03 00:58:35.771890 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:35.771822 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0be0024f-24ee-4c93-81ef-8d9b3f2d0b69-istio-data" pod "0be0024f-24ee-4c93-81ef-8d9b3f2d0b69" (UID: "0be0024f-24ee-4c93-81ef-8d9b3f2d0b69"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:58:35.771890 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:35.771837 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0be0024f-24ee-4c93-81ef-8d9b3f2d0b69-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0be0024f-24ee-4c93-81ef-8d9b3f2d0b69-workload-socket\") pod \"0be0024f-24ee-4c93-81ef-8d9b3f2d0b69\" (UID: \"0be0024f-24ee-4c93-81ef-8d9b3f2d0b69\") " Nov 03 00:58:35.772013 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:35.771935 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0be0024f-24ee-4c93-81ef-8d9b3f2d0b69-istio-token\" (UniqueName: \"kubernetes.io/projected/0be0024f-24ee-4c93-81ef-8d9b3f2d0b69-istio-token\") pod \"0be0024f-24ee-4c93-81ef-8d9b3f2d0b69\" (UID: \"0be0024f-24ee-4c93-81ef-8d9b3f2d0b69\") " Nov 03 00:58:35.772070 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:35.772022 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0be0024f-24ee-4c93-81ef-8d9b3f2d0b69-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0be0024f-24ee-4c93-81ef-8d9b3f2d0b69-workload-certs\") pod \"0be0024f-24ee-4c93-81ef-8d9b3f2d0b69\" (UID: \"0be0024f-24ee-4c93-81ef-8d9b3f2d0b69\") " Nov 03 00:58:35.772125 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:35.772071 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0be0024f-24ee-4c93-81ef-8d9b3f2d0b69-kube-api-access-lpbkw\" (UniqueName: \"kubernetes.io/projected/0be0024f-24ee-4c93-81ef-8d9b3f2d0b69-kube-api-access-lpbkw\") pod \"0be0024f-24ee-4c93-81ef-8d9b3f2d0b69\" (UID: \"0be0024f-24ee-4c93-81ef-8d9b3f2d0b69\") " Nov 03 00:58:35.772125 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:35.772091 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0be0024f-24ee-4c93-81ef-8d9b3f2d0b69-workload-socket" pod "0be0024f-24ee-4c93-81ef-8d9b3f2d0b69" (UID: "0be0024f-24ee-4c93-81ef-8d9b3f2d0b69"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:58:35.772125 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:35.772113 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0be0024f-24ee-4c93-81ef-8d9b3f2d0b69-credential-socket" pod "0be0024f-24ee-4c93-81ef-8d9b3f2d0b69" (UID: "0be0024f-24ee-4c93-81ef-8d9b3f2d0b69"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:58:35.772311 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:35.772228 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0be0024f-24ee-4c93-81ef-8d9b3f2d0b69-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0be0024f-24ee-4c93-81ef-8d9b3f2d0b69-istiod-ca-cert\") pod \"0be0024f-24ee-4c93-81ef-8d9b3f2d0b69\" (UID: \"0be0024f-24ee-4c93-81ef-8d9b3f2d0b69\") " Nov 03 00:58:35.772388 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:35.772364 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0be0024f-24ee-4c93-81ef-8d9b3f2d0b69-workload-certs" pod "0be0024f-24ee-4c93-81ef-8d9b3f2d0b69" (UID: "0be0024f-24ee-4c93-81ef-8d9b3f2d0b69"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:58:35.772703 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:35.772677 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0be0024f-24ee-4c93-81ef-8d9b3f2d0b69-istiod-ca-cert" pod "0be0024f-24ee-4c93-81ef-8d9b3f2d0b69" (UID: "0be0024f-24ee-4c93-81ef-8d9b3f2d0b69"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:58:35.773042 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:35.773019 4421 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0be0024f-24ee-4c93-81ef-8d9b3f2d0b69-credential-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:58:35.773118 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:35.773048 4421 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0be0024f-24ee-4c93-81ef-8d9b3f2d0b69-workload-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:58:35.773118 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:35.773064 4421 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0be0024f-24ee-4c93-81ef-8d9b3f2d0b69-workload-certs\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:58:35.773118 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:35.773080 4421 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0be0024f-24ee-4c93-81ef-8d9b3f2d0b69-istiod-ca-cert\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:58:35.773118 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:35.773097 4421 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0be0024f-24ee-4c93-81ef-8d9b3f2d0b69-istio-data\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:58:35.773967 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:35.773945 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0be0024f-24ee-4c93-81ef-8d9b3f2d0b69-istio-envoy" pod "0be0024f-24ee-4c93-81ef-8d9b3f2d0b69" (UID: "0be0024f-24ee-4c93-81ef-8d9b3f2d0b69"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:58:35.774179 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:35.774157 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0be0024f-24ee-4c93-81ef-8d9b3f2d0b69-istio-token" pod "0be0024f-24ee-4c93-81ef-8d9b3f2d0b69" (UID: "0be0024f-24ee-4c93-81ef-8d9b3f2d0b69"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:58:35.774219 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:35.774177 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0be0024f-24ee-4c93-81ef-8d9b3f2d0b69-istio-podinfo" pod "0be0024f-24ee-4c93-81ef-8d9b3f2d0b69" (UID: "0be0024f-24ee-4c93-81ef-8d9b3f2d0b69"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:58:35.774367 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:35.774349 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0be0024f-24ee-4c93-81ef-8d9b3f2d0b69-kube-api-access-lpbkw" pod "0be0024f-24ee-4c93-81ef-8d9b3f2d0b69" (UID: "0be0024f-24ee-4c93-81ef-8d9b3f2d0b69"). InnerVolumeSpecName "kube-api-access-lpbkw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:58:35.874449 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:35.874395 4421 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0be0024f-24ee-4c93-81ef-8d9b3f2d0b69-istio-token\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:58:35.874449 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:35.874427 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lpbkw\" (UniqueName: \"kubernetes.io/projected/0be0024f-24ee-4c93-81ef-8d9b3f2d0b69-kube-api-access-lpbkw\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:58:35.874449 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:35.874438 4421 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0be0024f-24ee-4c93-81ef-8d9b3f2d0b69-istio-envoy\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:58:35.874449 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:35.874449 4421 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0be0024f-24ee-4c93-81ef-8d9b3f2d0b69-istio-podinfo\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:58:35.966458 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:35.966422 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7f43f9-59dc7465d7-dnsxs"] Nov 03 00:58:35.967075 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:35.967061 4421 cpu_manager.go:414] "containerMap: removing container" podUID="0be0024f-24ee-4c93-81ef-8d9b3f2d0b69" containerName="istio-proxy" Nov 03 00:58:35.967120 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:35.967077 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0be0024f-24ee-4c93-81ef-8d9b3f2d0b69" containerName="istio-proxy" Nov 03 00:58:35.967153 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:35.967119 4421 cpu_manager.go:414] "containerMap: removing container" podUID="0be0024f-24ee-4c93-81ef-8d9b3f2d0b69" containerName="istio-proxy" Nov 03 00:58:35.967153 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:35.967125 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0be0024f-24ee-4c93-81ef-8d9b3f2d0b69" containerName="istio-proxy" Nov 03 00:58:35.967224 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:35.967215 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="0be0024f-24ee-4c93-81ef-8d9b3f2d0b69" containerName="istio-proxy" Nov 03 00:58:35.967256 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:35.967229 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="0be0024f-24ee-4c93-81ef-8d9b3f2d0b69" containerName="istio-proxy" Nov 03 00:58:35.971098 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:35.971073 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-7f43f9-59dc7465d7-dnsxs" Nov 03 00:58:35.976246 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:35.975634 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-7f43f9-dockercfg-jxxrr\"" Nov 03 00:58:35.976246 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:35.976009 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-7f43f9\"" Nov 03 00:58:35.984070 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:35.984038 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7f43f9-59dc7465d7-dnsxs"] Nov 03 00:58:36.130342 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:36.130213 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twdzd\" (UniqueName: \"kubernetes.io/projected/a99273cf-6ddb-4006-9207-9af01a04128c-kube-api-access-twdzd\") pod \"payload-processing-e2e-ipp-a-7f43f9-59dc7465d7-dnsxs\" (UID: \"a99273cf-6ddb-4006-9207-9af01a04128c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7f43f9-59dc7465d7-dnsxs" Nov 03 00:58:36.130657 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:36.130630 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a99273cf-6ddb-4006-9207-9af01a04128c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-7f43f9-59dc7465d7-dnsxs\" (UID: \"a99273cf-6ddb-4006-9207-9af01a04128c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7f43f9-59dc7465d7-dnsxs" Nov 03 00:58:36.254299 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:36.236185 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twdzd\" (UniqueName: \"kubernetes.io/projected/a99273cf-6ddb-4006-9207-9af01a04128c-kube-api-access-twdzd\") pod \"payload-processing-e2e-ipp-a-7f43f9-59dc7465d7-dnsxs\" (UID: \"a99273cf-6ddb-4006-9207-9af01a04128c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7f43f9-59dc7465d7-dnsxs" Nov 03 00:58:36.254299 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:36.236264 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a99273cf-6ddb-4006-9207-9af01a04128c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-7f43f9-59dc7465d7-dnsxs\" (UID: \"a99273cf-6ddb-4006-9207-9af01a04128c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7f43f9-59dc7465d7-dnsxs" Nov 03 00:58:36.254299 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:36.237491 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a99273cf-6ddb-4006-9207-9af01a04128c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-7f43f9-59dc7465d7-dnsxs\" (UID: \"a99273cf-6ddb-4006-9207-9af01a04128c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7f43f9-59dc7465d7-dnsxs" Nov 03 00:58:36.281722 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:36.269955 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-twdzd\" (UniqueName: \"kubernetes.io/projected/a99273cf-6ddb-4006-9207-9af01a04128c-kube-api-access-twdzd\") pod \"payload-processing-e2e-ipp-a-7f43f9-59dc7465d7-dnsxs\" (UID: \"a99273cf-6ddb-4006-9207-9af01a04128c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7f43f9-59dc7465d7-dnsxs" Nov 03 00:58:36.281722 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:36.270879 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-a2d8e5-openshift-default-85655d8485-f4q8h" Nov 03 00:58:36.281722 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:36.276649 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a2d8e5-openshift-default-85655d8485-f4q8h" event={"ID":"0be0024f-24ee-4c93-81ef-8d9b3f2d0b69","Type":"ContainerDied","Data":"c146927fc9f30705a21dab905ddca8e4c3868db19bb45a898cf8e37692a299f9"} Nov 03 00:58:36.281722 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:36.276717 4421 scope.go:122] "RemoveContainer" containerID="c146927fc9f30705a21dab905ddca8e4c3868db19bb45a898cf8e37692a299f9" Nov 03 00:58:36.282526 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:36.262011 4421 generic.go:356] "Generic (PLEG): container finished" podID="0be0024f-24ee-4c93-81ef-8d9b3f2d0b69" containerID="c146927fc9f30705a21dab905ddca8e4c3868db19bb45a898cf8e37692a299f9" exitCode=0 Nov 03 00:58:36.282671 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:36.282562 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a2d8e5-openshift-default-85655d8485-f4q8h" event={"ID":"0be0024f-24ee-4c93-81ef-8d9b3f2d0b69","Type":"ContainerDied","Data":"6ea138688d6cfaa73cfc9135ac6c4419614aa94366e2971e5b6a0c08f1841c1a"} Nov 03 00:58:36.291251 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:36.290686 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-7f43f9-59dc7465d7-dnsxs" Nov 03 00:58:36.311869 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:36.311791 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-a2d8e5-openshift-default-85655d8485-f4q8h"] Nov 03 00:58:36.318635 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:36.318604 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-a2d8e5-openshift-default-85655d8485-f4q8h"] Nov 03 00:58:36.321600 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:36.321567 4421 scope.go:122] "RemoveContainer" containerID="167b6ecaafc9a01a8ed6a0bdf239b0f6094c4caaec4e7a06831e0eaf9ec7f37b" Nov 03 00:58:36.350143 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:36.350113 4421 scope.go:122] "RemoveContainer" containerID="c146927fc9f30705a21dab905ddca8e4c3868db19bb45a898cf8e37692a299f9" Nov 03 00:58:36.350568 ip-10-0-133-217 kubenswrapper[4421]: E1103 00:58:36.350544 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c146927fc9f30705a21dab905ddca8e4c3868db19bb45a898cf8e37692a299f9\": container with ID starting with c146927fc9f30705a21dab905ddca8e4c3868db19bb45a898cf8e37692a299f9 not found: ID does not exist" containerID="c146927fc9f30705a21dab905ddca8e4c3868db19bb45a898cf8e37692a299f9" Nov 03 00:58:36.350671 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:36.350576 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c146927fc9f30705a21dab905ddca8e4c3868db19bb45a898cf8e37692a299f9"} err="failed to get container status \"c146927fc9f30705a21dab905ddca8e4c3868db19bb45a898cf8e37692a299f9\": rpc error: code = NotFound desc = could not find container \"c146927fc9f30705a21dab905ddca8e4c3868db19bb45a898cf8e37692a299f9\": container with ID starting with c146927fc9f30705a21dab905ddca8e4c3868db19bb45a898cf8e37692a299f9 not found: ID does not exist" Nov 03 00:58:36.350671 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:36.350599 4421 scope.go:122] "RemoveContainer" containerID="167b6ecaafc9a01a8ed6a0bdf239b0f6094c4caaec4e7a06831e0eaf9ec7f37b" Nov 03 00:58:36.350978 ip-10-0-133-217 kubenswrapper[4421]: E1103 00:58:36.350955 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"167b6ecaafc9a01a8ed6a0bdf239b0f6094c4caaec4e7a06831e0eaf9ec7f37b\": container with ID starting with 167b6ecaafc9a01a8ed6a0bdf239b0f6094c4caaec4e7a06831e0eaf9ec7f37b not found: ID does not exist" containerID="167b6ecaafc9a01a8ed6a0bdf239b0f6094c4caaec4e7a06831e0eaf9ec7f37b" Nov 03 00:58:36.351055 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:36.350987 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"167b6ecaafc9a01a8ed6a0bdf239b0f6094c4caaec4e7a06831e0eaf9ec7f37b"} err="failed to get container status \"167b6ecaafc9a01a8ed6a0bdf239b0f6094c4caaec4e7a06831e0eaf9ec7f37b\": rpc error: code = NotFound desc = could not find container \"167b6ecaafc9a01a8ed6a0bdf239b0f6094c4caaec4e7a06831e0eaf9ec7f37b\": container with ID starting with 167b6ecaafc9a01a8ed6a0bdf239b0f6094c4caaec4e7a06831e0eaf9ec7f37b not found: ID does not exist" Nov 03 00:58:36.645251 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:58:36.645210 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda99273cf_6ddb_4006_9207_9af01a04128c.slice/crio-5480439ae371d22271f57bbefa0cf683cb4279244de2270c2fba95004d48a542 WatchSource:0}: Error finding container 5480439ae371d22271f57bbefa0cf683cb4279244de2270c2fba95004d48a542: Status 404 returned error can't find the container with id 5480439ae371d22271f57bbefa0cf683cb4279244de2270c2fba95004d48a542 Nov 03 00:58:36.647608 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:36.647573 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7f43f9-59dc7465d7-dnsxs"] Nov 03 00:58:37.292659 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:37.292614 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-7f43f9-59dc7465d7-dnsxs" event={"ID":"a99273cf-6ddb-4006-9207-9af01a04128c","Type":"ContainerStarted","Data":"9ab37b8894d1dee432fa12e12a0d2f68974e3cec4f8e7455f7837d38ca816ad3"} Nov 03 00:58:37.292875 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:37.292692 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-7f43f9-59dc7465d7-dnsxs" event={"ID":"a99273cf-6ddb-4006-9207-9af01a04128c","Type":"ContainerStarted","Data":"5480439ae371d22271f57bbefa0cf683cb4279244de2270c2fba95004d48a542"} Nov 03 00:58:37.292875 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:37.292733 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-7f43f9-59dc7465d7-dnsxs" Nov 03 00:58:37.312759 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:37.312700 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-7f43f9-59dc7465d7-dnsxs" podStartSLOduration=2.312684592 podStartE2EDuration="2.312684592s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:58:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:58:37.311044791 +0000 UTC m=+1059.482047889" watchObservedRunningTime="2026-11-03 00:58:37.312684592 +0000 UTC m=+1059.483687688" Nov 03 00:58:37.979868 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:37.979836 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0be0024f-24ee-4c93-81ef-8d9b3f2d0b69" path="/var/lib/kubelet/pods/0be0024f-24ee-4c93-81ef-8d9b3f2d0b69/volumes" Nov 03 00:58:40.784249 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:40.784212 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-07f5f6-78d57d48ff-tgdfc"] Nov 03 00:58:41.965185 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:41.965145 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-80f70e-bfc75b495-7qxcs"] Nov 03 00:58:41.970260 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:41.970236 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-80f70e-bfc75b495-7qxcs" Nov 03 00:58:41.976685 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:41.976657 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-80f70e\"" Nov 03 00:58:41.976850 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:41.976801 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-80f70e-dockercfg-xvxcx\"" Nov 03 00:58:41.982991 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:41.982946 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-80f70e-bfc75b495-7qxcs"] Nov 03 00:58:41.993648 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:41.993620 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-80f70e-fb5785db7-l2v4s"] Nov 03 00:58:42.003568 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:42.003542 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-80f70e-fb5785db7-l2v4s" Nov 03 00:58:42.008102 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:42.008076 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-80f70e-fb5785db7-l2v4s"] Nov 03 00:58:42.029360 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:42.029330 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-513d2002-b9pm2"] Nov 03 00:58:42.038888 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:42.038858 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-513d2002-b9pm2"] Nov 03 00:58:42.077712 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:42.077678 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04b33c64-e869-43fa-81a6-57d5dacf927f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-80f70e-fb5785db7-l2v4s\" (UID: \"04b33c64-e869-43fa-81a6-57d5dacf927f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-80f70e-fb5785db7-l2v4s" Nov 03 00:58:42.077889 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:42.077867 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbqdd\" (UniqueName: \"kubernetes.io/projected/04b33c64-e869-43fa-81a6-57d5dacf927f-kube-api-access-lbqdd\") pod \"payload-pre-processing-e2e-api-a-80f70e-fb5785db7-l2v4s\" (UID: \"04b33c64-e869-43fa-81a6-57d5dacf927f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-80f70e-fb5785db7-l2v4s" Nov 03 00:58:42.077941 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:42.077928 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fs65\" (UniqueName: \"kubernetes.io/projected/d2320dab-9330-4dfd-b88c-eb5e952e8de2-kube-api-access-6fs65\") pod \"payload-processing-e2e-api-a-80f70e-bfc75b495-7qxcs\" (UID: \"d2320dab-9330-4dfd-b88c-eb5e952e8de2\") " pod="openshift-ingress/payload-processing-e2e-api-a-80f70e-bfc75b495-7qxcs" Nov 03 00:58:42.078053 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:42.078039 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2320dab-9330-4dfd-b88c-eb5e952e8de2-plugins-config-volume\") pod \"payload-processing-e2e-api-a-80f70e-bfc75b495-7qxcs\" (UID: \"d2320dab-9330-4dfd-b88c-eb5e952e8de2\") " pod="openshift-ingress/payload-processing-e2e-api-a-80f70e-bfc75b495-7qxcs" Nov 03 00:58:42.185306 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:42.181807 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04b33c64-e869-43fa-81a6-57d5dacf927f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-80f70e-fb5785db7-l2v4s\" (UID: \"04b33c64-e869-43fa-81a6-57d5dacf927f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-80f70e-fb5785db7-l2v4s" Nov 03 00:58:42.185306 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:42.182001 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbqdd\" (UniqueName: \"kubernetes.io/projected/04b33c64-e869-43fa-81a6-57d5dacf927f-kube-api-access-lbqdd\") pod \"payload-pre-processing-e2e-api-a-80f70e-fb5785db7-l2v4s\" (UID: \"04b33c64-e869-43fa-81a6-57d5dacf927f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-80f70e-fb5785db7-l2v4s" Nov 03 00:58:42.185306 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:42.182073 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fs65\" (UniqueName: \"kubernetes.io/projected/d2320dab-9330-4dfd-b88c-eb5e952e8de2-kube-api-access-6fs65\") pod \"payload-processing-e2e-api-a-80f70e-bfc75b495-7qxcs\" (UID: \"d2320dab-9330-4dfd-b88c-eb5e952e8de2\") " pod="openshift-ingress/payload-processing-e2e-api-a-80f70e-bfc75b495-7qxcs" Nov 03 00:58:42.185306 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:42.182193 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2320dab-9330-4dfd-b88c-eb5e952e8de2-plugins-config-volume\") pod \"payload-processing-e2e-api-a-80f70e-bfc75b495-7qxcs\" (UID: \"d2320dab-9330-4dfd-b88c-eb5e952e8de2\") " pod="openshift-ingress/payload-processing-e2e-api-a-80f70e-bfc75b495-7qxcs" Nov 03 00:58:42.185306 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:42.183165 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2320dab-9330-4dfd-b88c-eb5e952e8de2-plugins-config-volume\") pod \"payload-processing-e2e-api-a-80f70e-bfc75b495-7qxcs\" (UID: \"d2320dab-9330-4dfd-b88c-eb5e952e8de2\") " pod="openshift-ingress/payload-processing-e2e-api-a-80f70e-bfc75b495-7qxcs" Nov 03 00:58:42.185306 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:42.183928 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04b33c64-e869-43fa-81a6-57d5dacf927f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-80f70e-fb5785db7-l2v4s\" (UID: \"04b33c64-e869-43fa-81a6-57d5dacf927f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-80f70e-fb5785db7-l2v4s" Nov 03 00:58:42.216342 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:42.207973 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbqdd\" (UniqueName: \"kubernetes.io/projected/04b33c64-e869-43fa-81a6-57d5dacf927f-kube-api-access-lbqdd\") pod \"payload-pre-processing-e2e-api-a-80f70e-fb5785db7-l2v4s\" (UID: \"04b33c64-e869-43fa-81a6-57d5dacf927f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-80f70e-fb5785db7-l2v4s" Nov 03 00:58:42.238294 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:42.227978 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fs65\" (UniqueName: \"kubernetes.io/projected/d2320dab-9330-4dfd-b88c-eb5e952e8de2-kube-api-access-6fs65\") pod \"payload-processing-e2e-api-a-80f70e-bfc75b495-7qxcs\" (UID: \"d2320dab-9330-4dfd-b88c-eb5e952e8de2\") " pod="openshift-ingress/payload-processing-e2e-api-a-80f70e-bfc75b495-7qxcs" Nov 03 00:58:42.322300 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:42.311616 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-80f70e-bfc75b495-7qxcs" Nov 03 00:58:42.362297 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:42.361897 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-80f70e-fb5785db7-l2v4s" Nov 03 00:58:42.866331 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:42.866288 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-80f70e-bfc75b495-7qxcs"] Nov 03 00:58:42.879003 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:42.875393 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-80f70e-fb5785db7-l2v4s"] Nov 03 00:58:43.459470 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:43.459418 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-80f70e-bfc75b495-7qxcs" event={"ID":"d2320dab-9330-4dfd-b88c-eb5e952e8de2","Type":"ContainerStarted","Data":"41056dea8b065e76af98ba0eaa8fbf9566db00dd9c033963c206957692879944"} Nov 03 00:58:43.459948 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:43.459523 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-80f70e-bfc75b495-7qxcs" Nov 03 00:58:43.459948 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:43.459549 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-80f70e-bfc75b495-7qxcs" event={"ID":"d2320dab-9330-4dfd-b88c-eb5e952e8de2","Type":"ContainerStarted","Data":"cc964ba9457ada112cf6ef2b31f5df94f18dd2a556d71bbc03299858fc5b4907"} Nov 03 00:58:43.461344 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:43.461310 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-80f70e-fb5785db7-l2v4s" event={"ID":"04b33c64-e869-43fa-81a6-57d5dacf927f","Type":"ContainerStarted","Data":"8bf86768a33132a6bd43c06780f1b0d6ce416f7e4f40e0c83df8df8f54373eed"} Nov 03 00:58:43.461462 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:43.461359 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-80f70e-fb5785db7-l2v4s" event={"ID":"04b33c64-e869-43fa-81a6-57d5dacf927f","Type":"ContainerStarted","Data":"168ebacb29ed1550b1d75b270e18ae4eff860690660e808949a3239a2787a02b"} Nov 03 00:58:43.461462 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:43.461385 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-80f70e-fb5785db7-l2v4s" Nov 03 00:58:43.481851 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:43.481789 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-80f70e-bfc75b495-7qxcs" podStartSLOduration=2.481769495 podStartE2EDuration="2.481769495s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:58:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:58:43.475360954 +0000 UTC m=+1065.646364052" watchObservedRunningTime="2026-11-03 00:58:43.481769495 +0000 UTC m=+1065.652772592" Nov 03 00:58:43.506765 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:43.506710 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-80f70e-fb5785db7-l2v4s" podStartSLOduration=2.50669634 podStartE2EDuration="2.50669634s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:58:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:58:43.50313323 +0000 UTC m=+1065.674136328" watchObservedRunningTime="2026-11-03 00:58:43.50669634 +0000 UTC m=+1065.677699466" Nov 03 00:58:43.980513 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:43.980478 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aede371b-8f0e-41ed-baf4-09e0ac5afd50" path="/var/lib/kubelet/pods/aede371b-8f0e-41ed-baf4-09e0ac5afd50/volumes" Nov 03 00:58:44.670859 ip-10-0-133-217 kubenswrapper[4421]: E1103 00:58:44.670815 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 00:58:45.785039 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:45.784914 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-07f5f6-78d57d48ff-tgdfc" podUID="4f42ac06-fafe-49e4-bbd0-987fa37fe5ac" containerName="payload-pre-processing" containerID="cri-o://dc144c9412d9f56d657744743f0a7e67037f9380342c0b6299fe841a35b70545" gracePeriod=25 Nov 03 00:58:46.079681 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:46.079639 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-07f5f6-5b6bcc7f-bdvrj"] Nov 03 00:58:46.084869 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:46.084836 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-07f5f6-5b6bcc7f-bdvrj" Nov 03 00:58:46.090356 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:46.090326 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-07f5f6-dockercfg-fx4xh\"" Nov 03 00:58:46.090906 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:46.090881 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-07f5f6-5b6bcc7f-bdvrj"] Nov 03 00:58:46.096026 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:46.095998 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-07f5f6-78d57d48ff-tgdfc" Nov 03 00:58:46.226297 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:46.199450 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f42ac06-fafe-49e4-bbd0-987fa37fe5ac-kube-api-access-p4wr2\" (UniqueName: \"kubernetes.io/projected/4f42ac06-fafe-49e4-bbd0-987fa37fe5ac-kube-api-access-p4wr2\") pod \"4f42ac06-fafe-49e4-bbd0-987fa37fe5ac\" (UID: \"4f42ac06-fafe-49e4-bbd0-987fa37fe5ac\") " Nov 03 00:58:46.226297 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:46.199623 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f42ac06-fafe-49e4-bbd0-987fa37fe5ac-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f42ac06-fafe-49e4-bbd0-987fa37fe5ac-plugins-config-volume\") pod \"4f42ac06-fafe-49e4-bbd0-987fa37fe5ac\" (UID: \"4f42ac06-fafe-49e4-bbd0-987fa37fe5ac\") " Nov 03 00:58:46.226297 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:46.200124 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8d52857-85e2-4eb2-a223-33177d94ccfb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-07f5f6-5b6bcc7f-bdvrj\" (UID: \"f8d52857-85e2-4eb2-a223-33177d94ccfb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-07f5f6-5b6bcc7f-bdvrj" Nov 03 00:58:46.226297 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:46.200206 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5csc\" (UniqueName: \"kubernetes.io/projected/f8d52857-85e2-4eb2-a223-33177d94ccfb-kube-api-access-h5csc\") pod \"payload-pre-processing-e2e-ipp-a-07f5f6-5b6bcc7f-bdvrj\" (UID: \"f8d52857-85e2-4eb2-a223-33177d94ccfb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-07f5f6-5b6bcc7f-bdvrj" Nov 03 00:58:46.327479 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:46.295331 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f42ac06-fafe-49e4-bbd0-987fa37fe5ac-plugins-config-volume" pod "4f42ac06-fafe-49e4-bbd0-987fa37fe5ac" (UID: "4f42ac06-fafe-49e4-bbd0-987fa37fe5ac"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:58:46.327479 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:46.301691 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8d52857-85e2-4eb2-a223-33177d94ccfb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-07f5f6-5b6bcc7f-bdvrj\" (UID: \"f8d52857-85e2-4eb2-a223-33177d94ccfb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-07f5f6-5b6bcc7f-bdvrj" Nov 03 00:58:46.327479 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:46.301786 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5csc\" (UniqueName: \"kubernetes.io/projected/f8d52857-85e2-4eb2-a223-33177d94ccfb-kube-api-access-h5csc\") pod \"payload-pre-processing-e2e-ipp-a-07f5f6-5b6bcc7f-bdvrj\" (UID: \"f8d52857-85e2-4eb2-a223-33177d94ccfb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-07f5f6-5b6bcc7f-bdvrj" Nov 03 00:58:46.327479 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:46.302048 4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f42ac06-fafe-49e4-bbd0-987fa37fe5ac-plugins-config-volume\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:58:46.327479 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:46.303310 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8d52857-85e2-4eb2-a223-33177d94ccfb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-07f5f6-5b6bcc7f-bdvrj\" (UID: \"f8d52857-85e2-4eb2-a223-33177d94ccfb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-07f5f6-5b6bcc7f-bdvrj" Nov 03 00:58:46.370028 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:46.363129 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5csc\" (UniqueName: \"kubernetes.io/projected/f8d52857-85e2-4eb2-a223-33177d94ccfb-kube-api-access-h5csc\") pod \"payload-pre-processing-e2e-ipp-a-07f5f6-5b6bcc7f-bdvrj\" (UID: \"f8d52857-85e2-4eb2-a223-33177d94ccfb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-07f5f6-5b6bcc7f-bdvrj" Nov 03 00:58:46.380458 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:46.379149 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-07f5f6-6448498d8b-k8grc"] Nov 03 00:58:46.380458 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:46.380352 4421 cpu_manager.go:414] "containerMap: removing container" podUID="4f42ac06-fafe-49e4-bbd0-987fa37fe5ac" containerName="payload-pre-processing" Nov 03 00:58:46.380458 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:46.380371 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f42ac06-fafe-49e4-bbd0-987fa37fe5ac" containerName="payload-pre-processing" Nov 03 00:58:46.381573 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:46.380951 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f42ac06-fafe-49e4-bbd0-987fa37fe5ac" containerName="payload-pre-processing" Nov 03 00:58:46.387424 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:46.387389 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-07f5f6-6448498d8b-k8grc"] Nov 03 00:58:46.387549 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:46.387507 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-07f5f6-6448498d8b-k8grc" Nov 03 00:58:46.389055 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:46.389000 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f42ac06-fafe-49e4-bbd0-987fa37fe5ac-kube-api-access-p4wr2" pod "4f42ac06-fafe-49e4-bbd0-987fa37fe5ac" (UID: "4f42ac06-fafe-49e4-bbd0-987fa37fe5ac"). InnerVolumeSpecName "kube-api-access-p4wr2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:58:46.411490 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:46.411460 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4wr2\" (UniqueName: \"kubernetes.io/projected/4f42ac06-fafe-49e4-bbd0-987fa37fe5ac-kube-api-access-p4wr2\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:58:46.413482 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:46.413168 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-07f5f6-5b6bcc7f-bdvrj" Nov 03 00:58:46.506303 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:46.504585 4421 generic.go:356] "Generic (PLEG): container finished" podID="4f42ac06-fafe-49e4-bbd0-987fa37fe5ac" containerID="dc144c9412d9f56d657744743f0a7e67037f9380342c0b6299fe841a35b70545" exitCode=0 Nov 03 00:58:46.506303 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:46.504817 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-07f5f6-78d57d48ff-tgdfc" event={"ID":"4f42ac06-fafe-49e4-bbd0-987fa37fe5ac","Type":"ContainerDied","Data":"dc144c9412d9f56d657744743f0a7e67037f9380342c0b6299fe841a35b70545"} Nov 03 00:58:46.506303 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:46.504914 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-07f5f6-78d57d48ff-tgdfc" event={"ID":"4f42ac06-fafe-49e4-bbd0-987fa37fe5ac","Type":"ContainerDied","Data":"1d19f2754f719725702fce8f67586d772875e84c954114f2f119bc41bdbcdcc7"} Nov 03 00:58:46.506303 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:46.504948 4421 scope.go:122] "RemoveContainer" containerID="dc144c9412d9f56d657744743f0a7e67037f9380342c0b6299fe841a35b70545" Nov 03 00:58:46.506303 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:46.505127 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-07f5f6-78d57d48ff-tgdfc" Nov 03 00:58:46.513058 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:46.512727 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c232835-2f2e-4ee2-b646-9c56f413919c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-07f5f6-6448498d8b-k8grc\" (UID: \"4c232835-2f2e-4ee2-b646-9c56f413919c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-07f5f6-6448498d8b-k8grc" Nov 03 00:58:46.513058 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:46.512974 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjpnw\" (UniqueName: \"kubernetes.io/projected/4c232835-2f2e-4ee2-b646-9c56f413919c-kube-api-access-mjpnw\") pod \"payload-processing-e2e-ipp-a-07f5f6-6448498d8b-k8grc\" (UID: \"4c232835-2f2e-4ee2-b646-9c56f413919c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-07f5f6-6448498d8b-k8grc" Nov 03 00:58:46.519962 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:46.519938 4421 scope.go:122] "RemoveContainer" containerID="dc144c9412d9f56d657744743f0a7e67037f9380342c0b6299fe841a35b70545" Nov 03 00:58:46.520325 ip-10-0-133-217 kubenswrapper[4421]: E1103 00:58:46.520304 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc144c9412d9f56d657744743f0a7e67037f9380342c0b6299fe841a35b70545\": container with ID starting with dc144c9412d9f56d657744743f0a7e67037f9380342c0b6299fe841a35b70545 not found: ID does not exist" containerID="dc144c9412d9f56d657744743f0a7e67037f9380342c0b6299fe841a35b70545" Nov 03 00:58:46.520441 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:46.520337 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc144c9412d9f56d657744743f0a7e67037f9380342c0b6299fe841a35b70545"} err="failed to get container status \"dc144c9412d9f56d657744743f0a7e67037f9380342c0b6299fe841a35b70545\": rpc error: code = NotFound desc = could not find container \"dc144c9412d9f56d657744743f0a7e67037f9380342c0b6299fe841a35b70545\": container with ID starting with dc144c9412d9f56d657744743f0a7e67037f9380342c0b6299fe841a35b70545 not found: ID does not exist" Nov 03 00:58:46.534990 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:46.534953 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-07f5f6-78d57d48ff-tgdfc"] Nov 03 00:58:46.542470 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:46.542443 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-07f5f6-78d57d48ff-tgdfc"] Nov 03 00:58:46.614378 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:46.614253 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c232835-2f2e-4ee2-b646-9c56f413919c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-07f5f6-6448498d8b-k8grc\" (UID: \"4c232835-2f2e-4ee2-b646-9c56f413919c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-07f5f6-6448498d8b-k8grc" Nov 03 00:58:46.614529 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:46.614494 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjpnw\" (UniqueName: \"kubernetes.io/projected/4c232835-2f2e-4ee2-b646-9c56f413919c-kube-api-access-mjpnw\") pod \"payload-processing-e2e-ipp-a-07f5f6-6448498d8b-k8grc\" (UID: \"4c232835-2f2e-4ee2-b646-9c56f413919c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-07f5f6-6448498d8b-k8grc" Nov 03 00:58:46.614981 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:46.614962 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c232835-2f2e-4ee2-b646-9c56f413919c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-07f5f6-6448498d8b-k8grc\" (UID: \"4c232835-2f2e-4ee2-b646-9c56f413919c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-07f5f6-6448498d8b-k8grc" Nov 03 00:58:46.622894 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:46.622858 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjpnw\" (UniqueName: \"kubernetes.io/projected/4c232835-2f2e-4ee2-b646-9c56f413919c-kube-api-access-mjpnw\") pod \"payload-processing-e2e-ipp-a-07f5f6-6448498d8b-k8grc\" (UID: \"4c232835-2f2e-4ee2-b646-9c56f413919c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-07f5f6-6448498d8b-k8grc" Nov 03 00:58:46.742933 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:46.742898 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-07f5f6-6448498d8b-k8grc" Nov 03 00:58:46.746874 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:46.746829 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-07f5f6-5b6bcc7f-bdvrj"] Nov 03 00:58:46.753669 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:58:46.753618 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8d52857_85e2_4eb2_a223_33177d94ccfb.slice/crio-069c4d852f926644b561feb48114aa5dbc33d8449f0891035f9ab44c8a4c5c5a WatchSource:0}: Error finding container 069c4d852f926644b561feb48114aa5dbc33d8449f0891035f9ab44c8a4c5c5a: Status 404 returned error can't find the container with id 069c4d852f926644b561feb48114aa5dbc33d8449f0891035f9ab44c8a4c5c5a Nov 03 00:58:46.756777 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:46.756673 4421 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 00:58:47.048368 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:58:47.048338 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c232835_2f2e_4ee2_b646_9c56f413919c.slice/crio-b7fec3215156b8bf855044d1d34440f38f61c0ce57db21f2bf9bcb552d5d42e6 WatchSource:0}: Error finding container b7fec3215156b8bf855044d1d34440f38f61c0ce57db21f2bf9bcb552d5d42e6: Status 404 returned error can't find the container with id b7fec3215156b8bf855044d1d34440f38f61c0ce57db21f2bf9bcb552d5d42e6 Nov 03 00:58:47.049329 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:47.049303 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-07f5f6-6448498d8b-k8grc"] Nov 03 00:58:47.521267 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:47.521227 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-07f5f6-5b6bcc7f-bdvrj" event={"ID":"f8d52857-85e2-4eb2-a223-33177d94ccfb","Type":"ContainerStarted","Data":"069c4d852f926644b561feb48114aa5dbc33d8449f0891035f9ab44c8a4c5c5a"} Nov 03 00:58:47.523218 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:47.523187 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-07f5f6-6448498d8b-k8grc" event={"ID":"4c232835-2f2e-4ee2-b646-9c56f413919c","Type":"ContainerStarted","Data":"b7fec3215156b8bf855044d1d34440f38f61c0ce57db21f2bf9bcb552d5d42e6"} Nov 03 00:58:47.988242 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:47.988063 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f42ac06-fafe-49e4-bbd0-987fa37fe5ac" path="/var/lib/kubelet/pods/4f42ac06-fafe-49e4-bbd0-987fa37fe5ac/volumes" Nov 03 00:58:48.368480 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:48.368440 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-7f43f9-59dc7465d7-dnsxs" Nov 03 00:58:50.654301 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:50.654154 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-07f5f6-5b6bcc7f-bdvrj" event={"ID":"f8d52857-85e2-4eb2-a223-33177d94ccfb","Type":"ContainerStarted","Data":"3c1456e60721a6a7aaa70dc8debc699858a7eed15c77bf65a1cfaebde5b1f12b"} Nov 03 00:58:50.662298 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:50.660494 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-07f5f6-5b6bcc7f-bdvrj" Nov 03 00:58:50.684301 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:50.676972 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-07f5f6-6448498d8b-k8grc" event={"ID":"4c232835-2f2e-4ee2-b646-9c56f413919c","Type":"ContainerStarted","Data":"3efca6e4ec1a1f6cd2e9d42eb57fcd9f2b65e0de31c7b0d2d451c3c7c4d5f9f1"} Nov 03 00:58:50.684301 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:50.677948 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-07f5f6-6448498d8b-k8grc" Nov 03 00:58:50.699300 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:50.693393 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-07f5f6-5b6bcc7f-bdvrj" podStartSLOduration=1.79786907 podStartE2EDuration="4.693374357s" totalImagesPullingTime="2.895505287s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:58:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:58:50.691394099 +0000 UTC m=+1072.862397196" watchObservedRunningTime="2026-11-03 00:58:50.693374357 +0000 UTC m=+1072.864377455" Nov 03 00:58:54.500667 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:54.500638 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-80f70e-fb5785db7-l2v4s" Nov 03 00:58:54.502471 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:54.502446 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-80f70e-bfc75b495-7qxcs" Nov 03 00:58:54.507216 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:54.507157 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-07f5f6-6448498d8b-k8grc" podStartSLOduration=5.871109452 podStartE2EDuration="8.50713819s" totalImagesPullingTime="2.636028738s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:58:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:58:50.722375985 +0000 UTC m=+1072.893379084" watchObservedRunningTime="2026-11-03 00:58:54.50713819 +0000 UTC m=+1076.678141283" Nov 03 00:58:54.514298 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:54.508054 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7f43f9-59dc7465d7-dnsxs"] Nov 03 00:58:55.480719 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:55.480678 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-dd03e395-779465d96c-nppw5"] Nov 03 00:58:57.180681 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:57.180643 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-340b96-openshift-default-7846d978b9-8zqfs"] Nov 03 00:58:57.181795 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:57.181760 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-340b96-openshift-default-7846d978b9-8zqfs" podUID="90e63fe7-52ef-4fbd-ab37-32c6479abb7a" containerName="istio-proxy" containerID="cri-o://449161d828ac5cf3119b2d9c2232f2e6c7bfc9b3ca332c33155b19456a6492f5" gracePeriod=30 Nov 03 00:58:58.023458 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:58.023422 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d4167a9-92r25"] Nov 03 00:58:58.029517 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:58.029486 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d4167a9-92r25"] Nov 03 00:58:58.348527 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:58.348438 4421 patch_prober.go:29] interesting pod/e2e-api-b-eed369-openshift-default-66698bcb95-pknkp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 00:58:58.348527 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:58.348502 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-eed369-openshift-default-66698bcb95-pknkp" podUID="b797c4ed-b8df-4e23-9188-3aeb3cf6c48f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 00:58:58.591828 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:58.585348 4421 scope.go:122] "RemoveContainer" containerID="96393556fd395ff04001ef9b8ac7c2d2651c9982b95bac2f1543d82e8d528f40" Nov 03 00:58:58.643765 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:58.641742 4421 scope.go:122] "RemoveContainer" containerID="043524b93ace404e06e4d3723e9cf4c3e1c601c2cdda974ebd8d0796136b2277" Nov 03 00:58:58.656224 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:58.656198 4421 scope.go:122] "RemoveContainer" containerID="4db675d3d04c7be1ae90eaeeb7a9ca873355e2c34d2d13c6e3ec80911b9dd921" Nov 03 00:58:58.670544 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:58.670520 4421 scope.go:122] "RemoveContainer" containerID="6aa2daf6c740422a5530ea8b75de57bef5049c2917b73267579371d19be92b86" Nov 03 00:58:58.681965 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:58.681935 4421 scope.go:122] "RemoveContainer" containerID="ef21830034d3aef53dc42afb9884dfc7ecc4fdd546b17df82efb77572d0613a8" Nov 03 00:58:58.693149 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:58.693127 4421 scope.go:122] "RemoveContainer" containerID="467abd6290257c709088123048e4b3832847ee5e4f678e5cdb814ad21e2b3654" Nov 03 00:58:59.515828 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:59.515788 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-7f43f9-59dc7465d7-dnsxs" podUID="a99273cf-6ddb-4006-9207-9af01a04128c" containerName="payload-processing" containerID="cri-o://9ab37b8894d1dee432fa12e12a0d2f68974e3cec4f8e7455f7837d38ca816ad3" gracePeriod=25 Nov 03 00:58:59.859206 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:59.859170 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-7f43f9-59dc7465d7-dnsxs" Nov 03 00:58:59.952736 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:59.952693 4421 generic.go:356] "Generic (PLEG): container finished" podID="a99273cf-6ddb-4006-9207-9af01a04128c" containerID="9ab37b8894d1dee432fa12e12a0d2f68974e3cec4f8e7455f7837d38ca816ad3" exitCode=0 Nov 03 00:58:59.952736 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:59.952731 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-7f43f9-59dc7465d7-dnsxs" Nov 03 00:58:59.952961 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:59.952762 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-7f43f9-59dc7465d7-dnsxs" event={"ID":"a99273cf-6ddb-4006-9207-9af01a04128c","Type":"ContainerDied","Data":"9ab37b8894d1dee432fa12e12a0d2f68974e3cec4f8e7455f7837d38ca816ad3"} Nov 03 00:58:59.952961 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:59.952816 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-7f43f9-59dc7465d7-dnsxs" event={"ID":"a99273cf-6ddb-4006-9207-9af01a04128c","Type":"ContainerDied","Data":"5480439ae371d22271f57bbefa0cf683cb4279244de2270c2fba95004d48a542"} Nov 03 00:58:59.952961 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:59.952843 4421 scope.go:122] "RemoveContainer" containerID="9ab37b8894d1dee432fa12e12a0d2f68974e3cec4f8e7455f7837d38ca816ad3" Nov 03 00:58:59.964085 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:59.964062 4421 scope.go:122] "RemoveContainer" containerID="9ab37b8894d1dee432fa12e12a0d2f68974e3cec4f8e7455f7837d38ca816ad3" Nov 03 00:58:59.964468 ip-10-0-133-217 kubenswrapper[4421]: E1103 00:58:59.964449 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ab37b8894d1dee432fa12e12a0d2f68974e3cec4f8e7455f7837d38ca816ad3\": container with ID starting with 9ab37b8894d1dee432fa12e12a0d2f68974e3cec4f8e7455f7837d38ca816ad3 not found: ID does not exist" containerID="9ab37b8894d1dee432fa12e12a0d2f68974e3cec4f8e7455f7837d38ca816ad3" Nov 03 00:58:59.964533 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:59.964477 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ab37b8894d1dee432fa12e12a0d2f68974e3cec4f8e7455f7837d38ca816ad3"} err="failed to get container status \"9ab37b8894d1dee432fa12e12a0d2f68974e3cec4f8e7455f7837d38ca816ad3\": rpc error: code = NotFound desc = could not find container \"9ab37b8894d1dee432fa12e12a0d2f68974e3cec4f8e7455f7837d38ca816ad3\": container with ID starting with 9ab37b8894d1dee432fa12e12a0d2f68974e3cec4f8e7455f7837d38ca816ad3 not found: ID does not exist" Nov 03 00:58:59.974703 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:59.974674 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a99273cf-6ddb-4006-9207-9af01a04128c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a99273cf-6ddb-4006-9207-9af01a04128c-plugins-config-volume\") pod \"a99273cf-6ddb-4006-9207-9af01a04128c\" (UID: \"a99273cf-6ddb-4006-9207-9af01a04128c\") " Nov 03 00:58:59.974834 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:59.974766 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a99273cf-6ddb-4006-9207-9af01a04128c-kube-api-access-twdzd\" (UniqueName: \"kubernetes.io/projected/a99273cf-6ddb-4006-9207-9af01a04128c-kube-api-access-twdzd\") pod \"a99273cf-6ddb-4006-9207-9af01a04128c\" (UID: \"a99273cf-6ddb-4006-9207-9af01a04128c\") " Nov 03 00:58:59.975101 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:59.975077 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a99273cf-6ddb-4006-9207-9af01a04128c-plugins-config-volume" pod "a99273cf-6ddb-4006-9207-9af01a04128c" (UID: "a99273cf-6ddb-4006-9207-9af01a04128c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:58:59.975460 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:59.975436 4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a99273cf-6ddb-4006-9207-9af01a04128c-plugins-config-volume\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:58:59.977243 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:59.977216 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a99273cf-6ddb-4006-9207-9af01a04128c-kube-api-access-twdzd" pod "a99273cf-6ddb-4006-9207-9af01a04128c" (UID: "a99273cf-6ddb-4006-9207-9af01a04128c"). InnerVolumeSpecName "kube-api-access-twdzd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:58:59.981021 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:58:59.980998 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="430ade2a-8ebf-4b6c-bfaf-71eaba2a6a58" path="/var/lib/kubelet/pods/430ade2a-8ebf-4b6c-bfaf-71eaba2a6a58/volumes" Nov 03 00:59:00.077831 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:00.077745 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twdzd\" (UniqueName: \"kubernetes.io/projected/a99273cf-6ddb-4006-9207-9af01a04128c-kube-api-access-twdzd\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:59:00.272950 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:00.272918 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7f43f9-59dc7465d7-dnsxs"] Nov 03 00:59:00.277369 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:00.277342 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7f43f9-59dc7465d7-dnsxs"] Nov 03 00:59:00.481764 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:00.481717 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-dd03e395-779465d96c-nppw5" podUID="fa2a0390-dbfb-46e6-9a54-f2522988748f" containerName="payload-pre-processing" containerID="cri-o://b7580aacf21666a9deb4495ed376a7e4d913cb231095d3fefe7e6d89e0c7387e" gracePeriod=25 Nov 03 00:59:00.965840 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:00.965807 4421 generic.go:356] "Generic (PLEG): container finished" podID="b797c4ed-b8df-4e23-9188-3aeb3cf6c48f" containerID="c478a537e618fb8d71c478fa5ca5bf32229158cc77be47271c4f611b1f348977" exitCode=0 Nov 03 00:59:00.966255 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:00.965888 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-eed369-openshift-default-66698bcb95-pknkp" event={"ID":"b797c4ed-b8df-4e23-9188-3aeb3cf6c48f","Type":"ContainerDied","Data":"c478a537e618fb8d71c478fa5ca5bf32229158cc77be47271c4f611b1f348977"} Nov 03 00:59:00.966526 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:00.966504 4421 scope.go:122] "RemoveContainer" containerID="c478a537e618fb8d71c478fa5ca5bf32229158cc77be47271c4f611b1f348977" Nov 03 00:59:00.967574 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:00.967550 4421 generic.go:356] "Generic (PLEG): container finished" podID="fa2a0390-dbfb-46e6-9a54-f2522988748f" containerID="b7580aacf21666a9deb4495ed376a7e4d913cb231095d3fefe7e6d89e0c7387e" exitCode=0 Nov 03 00:59:00.967794 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:00.967595 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-dd03e395-779465d96c-nppw5" event={"ID":"fa2a0390-dbfb-46e6-9a54-f2522988748f","Type":"ContainerDied","Data":"b7580aacf21666a9deb4495ed376a7e4d913cb231095d3fefe7e6d89e0c7387e"} Nov 03 00:59:00.967794 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:00.967654 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-dd03e395-779465d96c-nppw5" event={"ID":"fa2a0390-dbfb-46e6-9a54-f2522988748f","Type":"ContainerDied","Data":"d10309c916be06ec3bc6cdb6853e7325e6853d6e50cc8782ee89799c27117b06"} Nov 03 00:59:00.967794 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:00.967679 4421 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d10309c916be06ec3bc6cdb6853e7325e6853d6e50cc8782ee89799c27117b06" Nov 03 00:59:00.968737 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:00.968696 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:59:00.968815 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:00.968790 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:59:00.968867 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:00.968838 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:59:00.989629 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:00.989602 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-dd03e395-779465d96c-nppw5" Nov 03 00:59:01.108348 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:01.108310 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa2a0390-dbfb-46e6-9a54-f2522988748f-kube-api-access-p74ng\" (UniqueName: \"kubernetes.io/projected/fa2a0390-dbfb-46e6-9a54-f2522988748f-kube-api-access-p74ng\") pod \"fa2a0390-dbfb-46e6-9a54-f2522988748f\" (UID: \"fa2a0390-dbfb-46e6-9a54-f2522988748f\") " Nov 03 00:59:01.108546 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:01.108522 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa2a0390-dbfb-46e6-9a54-f2522988748f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa2a0390-dbfb-46e6-9a54-f2522988748f-plugins-config-volume\") pod \"fa2a0390-dbfb-46e6-9a54-f2522988748f\" (UID: \"fa2a0390-dbfb-46e6-9a54-f2522988748f\") " Nov 03 00:59:01.110643 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:01.110593 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa2a0390-dbfb-46e6-9a54-f2522988748f-plugins-config-volume" pod "fa2a0390-dbfb-46e6-9a54-f2522988748f" (UID: "fa2a0390-dbfb-46e6-9a54-f2522988748f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:59:01.114451 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:01.112792 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa2a0390-dbfb-46e6-9a54-f2522988748f-kube-api-access-p74ng" pod "fa2a0390-dbfb-46e6-9a54-f2522988748f" (UID: "fa2a0390-dbfb-46e6-9a54-f2522988748f"). InnerVolumeSpecName "kube-api-access-p74ng". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:59:01.211238 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:01.211171 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p74ng\" (UniqueName: \"kubernetes.io/projected/fa2a0390-dbfb-46e6-9a54-f2522988748f-kube-api-access-p74ng\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:59:01.211238 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:01.211201 4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa2a0390-dbfb-46e6-9a54-f2522988748f-plugins-config-volume\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:59:01.980581 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:01.980548 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-dd03e395-779465d96c-nppw5" Nov 03 00:59:01.981079 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:01.980548 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a99273cf-6ddb-4006-9207-9af01a04128c" path="/var/lib/kubelet/pods/a99273cf-6ddb-4006-9207-9af01a04128c/volumes" Nov 03 00:59:01.981258 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:01.981235 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-eed369-openshift-default-66698bcb95-pknkp" event={"ID":"b797c4ed-b8df-4e23-9188-3aeb3cf6c48f","Type":"ContainerStarted","Data":"3efcc349025a233d05bdf085d3b12c572ed416e0da2d31daa76825d31d6c40e6"} Nov 03 00:59:02.039539 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:02.039507 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-dd03e395-779465d96c-nppw5"] Nov 03 00:59:02.044020 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:02.043990 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-dd03e395-779465d96c-nppw5"] Nov 03 00:59:02.149492 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:02.149462 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-eed369-openshift-default-66698bcb95-pknkp" Nov 03 00:59:02.149762 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:02.149631 4421 patch_prober.go:29] interesting pod/e2e-api-b-eed369-openshift-default-66698bcb95-pknkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body= Nov 03 00:59:02.149931 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:02.149891 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-eed369-openshift-default-66698bcb95-pknkp" podUID="b797c4ed-b8df-4e23-9188-3aeb3cf6c48f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" Nov 03 00:59:02.543148 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:02.543117 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-340b96-openshift-default-7846d978b9-8zqfs" Nov 03 00:59:02.660174 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:02.660077 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90e63fe7-52ef-4fbd-ab37-32c6479abb7a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90e63fe7-52ef-4fbd-ab37-32c6479abb7a-istiod-ca-cert\") pod \"90e63fe7-52ef-4fbd-ab37-32c6479abb7a\" (UID: \"90e63fe7-52ef-4fbd-ab37-32c6479abb7a\") " Nov 03 00:59:02.660174 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:02.660123 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/90e63fe7-52ef-4fbd-ab37-32c6479abb7a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90e63fe7-52ef-4fbd-ab37-32c6479abb7a-istio-podinfo\") pod \"90e63fe7-52ef-4fbd-ab37-32c6479abb7a\" (UID: \"90e63fe7-52ef-4fbd-ab37-32c6479abb7a\") " Nov 03 00:59:02.660174 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:02.660165 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90e63fe7-52ef-4fbd-ab37-32c6479abb7a-kube-api-access-5vq57\" (UniqueName: \"kubernetes.io/projected/90e63fe7-52ef-4fbd-ab37-32c6479abb7a-kube-api-access-5vq57\") pod \"90e63fe7-52ef-4fbd-ab37-32c6479abb7a\" (UID: \"90e63fe7-52ef-4fbd-ab37-32c6479abb7a\") " Nov 03 00:59:02.660469 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:02.660206 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90e63fe7-52ef-4fbd-ab37-32c6479abb7a-istio-token\" (UniqueName: \"kubernetes.io/projected/90e63fe7-52ef-4fbd-ab37-32c6479abb7a-istio-token\") pod \"90e63fe7-52ef-4fbd-ab37-32c6479abb7a\" (UID: \"90e63fe7-52ef-4fbd-ab37-32c6479abb7a\") " Nov 03 00:59:02.660469 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:02.660262 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90e63fe7-52ef-4fbd-ab37-32c6479abb7a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90e63fe7-52ef-4fbd-ab37-32c6479abb7a-workload-socket\") pod \"90e63fe7-52ef-4fbd-ab37-32c6479abb7a\" (UID: \"90e63fe7-52ef-4fbd-ab37-32c6479abb7a\") " Nov 03 00:59:02.660469 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:02.660345 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90e63fe7-52ef-4fbd-ab37-32c6479abb7a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90e63fe7-52ef-4fbd-ab37-32c6479abb7a-istio-envoy\") pod \"90e63fe7-52ef-4fbd-ab37-32c6479abb7a\" (UID: \"90e63fe7-52ef-4fbd-ab37-32c6479abb7a\") " Nov 03 00:59:02.660469 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:02.660445 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90e63fe7-52ef-4fbd-ab37-32c6479abb7a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90e63fe7-52ef-4fbd-ab37-32c6479abb7a-credential-socket\") pod \"90e63fe7-52ef-4fbd-ab37-32c6479abb7a\" (UID: \"90e63fe7-52ef-4fbd-ab37-32c6479abb7a\") " Nov 03 00:59:02.660688 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:02.660531 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90e63fe7-52ef-4fbd-ab37-32c6479abb7a-istiod-ca-cert" pod "90e63fe7-52ef-4fbd-ab37-32c6479abb7a" (UID: "90e63fe7-52ef-4fbd-ab37-32c6479abb7a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:59:02.660688 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:02.660562 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90e63fe7-52ef-4fbd-ab37-32c6479abb7a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90e63fe7-52ef-4fbd-ab37-32c6479abb7a-workload-certs\") pod \"90e63fe7-52ef-4fbd-ab37-32c6479abb7a\" (UID: \"90e63fe7-52ef-4fbd-ab37-32c6479abb7a\") " Nov 03 00:59:02.660688 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:02.660606 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90e63fe7-52ef-4fbd-ab37-32c6479abb7a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90e63fe7-52ef-4fbd-ab37-32c6479abb7a-istio-data\") pod \"90e63fe7-52ef-4fbd-ab37-32c6479abb7a\" (UID: \"90e63fe7-52ef-4fbd-ab37-32c6479abb7a\") " Nov 03 00:59:02.660872 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:02.660682 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90e63fe7-52ef-4fbd-ab37-32c6479abb7a-workload-socket" pod "90e63fe7-52ef-4fbd-ab37-32c6479abb7a" (UID: "90e63fe7-52ef-4fbd-ab37-32c6479abb7a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:59:02.660872 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:02.660761 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90e63fe7-52ef-4fbd-ab37-32c6479abb7a-credential-socket" pod "90e63fe7-52ef-4fbd-ab37-32c6479abb7a" (UID: "90e63fe7-52ef-4fbd-ab37-32c6479abb7a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:59:02.660989 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:02.660871 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90e63fe7-52ef-4fbd-ab37-32c6479abb7a-workload-certs" pod "90e63fe7-52ef-4fbd-ab37-32c6479abb7a" (UID: "90e63fe7-52ef-4fbd-ab37-32c6479abb7a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:59:02.660989 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:02.660949 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90e63fe7-52ef-4fbd-ab37-32c6479abb7a-istio-data" pod "90e63fe7-52ef-4fbd-ab37-32c6479abb7a" (UID: "90e63fe7-52ef-4fbd-ab37-32c6479abb7a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:59:02.661523 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:02.661485 4421 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90e63fe7-52ef-4fbd-ab37-32c6479abb7a-workload-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:59:02.661523 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:02.661516 4421 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90e63fe7-52ef-4fbd-ab37-32c6479abb7a-credential-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:59:02.661722 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:02.661532 4421 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90e63fe7-52ef-4fbd-ab37-32c6479abb7a-workload-certs\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:59:02.661722 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:02.661548 4421 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90e63fe7-52ef-4fbd-ab37-32c6479abb7a-istio-data\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:59:02.661722 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:02.661562 4421 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90e63fe7-52ef-4fbd-ab37-32c6479abb7a-istiod-ca-cert\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:59:02.662680 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:02.662631 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/90e63fe7-52ef-4fbd-ab37-32c6479abb7a-istio-podinfo" pod "90e63fe7-52ef-4fbd-ab37-32c6479abb7a" (UID: "90e63fe7-52ef-4fbd-ab37-32c6479abb7a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:59:02.662801 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:02.662702 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90e63fe7-52ef-4fbd-ab37-32c6479abb7a-istio-envoy" pod "90e63fe7-52ef-4fbd-ab37-32c6479abb7a" (UID: "90e63fe7-52ef-4fbd-ab37-32c6479abb7a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:59:02.662801 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:02.662718 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90e63fe7-52ef-4fbd-ab37-32c6479abb7a-istio-token" pod "90e63fe7-52ef-4fbd-ab37-32c6479abb7a" (UID: "90e63fe7-52ef-4fbd-ab37-32c6479abb7a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:59:02.662801 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:02.662754 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90e63fe7-52ef-4fbd-ab37-32c6479abb7a-kube-api-access-5vq57" pod "90e63fe7-52ef-4fbd-ab37-32c6479abb7a" (UID: "90e63fe7-52ef-4fbd-ab37-32c6479abb7a"). InnerVolumeSpecName "kube-api-access-5vq57". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:59:02.752783 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:02.752749 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-07f5f6-5b6bcc7f-bdvrj" Nov 03 00:59:02.752967 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:02.752838 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-07f5f6-6448498d8b-k8grc" Nov 03 00:59:02.763417 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:02.763384 4421 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90e63fe7-52ef-4fbd-ab37-32c6479abb7a-istio-podinfo\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:59:02.763417 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:02.763417 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vq57\" (UniqueName: \"kubernetes.io/projected/90e63fe7-52ef-4fbd-ab37-32c6479abb7a-kube-api-access-5vq57\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:59:02.763613 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:02.763431 4421 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90e63fe7-52ef-4fbd-ab37-32c6479abb7a-istio-token\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:59:02.763613 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:02.763444 4421 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90e63fe7-52ef-4fbd-ab37-32c6479abb7a-istio-envoy\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:59:02.990373 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:02.990292 4421 generic.go:356] "Generic (PLEG): container finished" podID="90e63fe7-52ef-4fbd-ab37-32c6479abb7a" containerID="449161d828ac5cf3119b2d9c2232f2e6c7bfc9b3ca332c33155b19456a6492f5" exitCode=0 Nov 03 00:59:02.990373 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:02.990364 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-340b96-openshift-default-7846d978b9-8zqfs" Nov 03 00:59:02.990835 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:02.990378 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-340b96-openshift-default-7846d978b9-8zqfs" event={"ID":"90e63fe7-52ef-4fbd-ab37-32c6479abb7a","Type":"ContainerDied","Data":"449161d828ac5cf3119b2d9c2232f2e6c7bfc9b3ca332c33155b19456a6492f5"} Nov 03 00:59:02.990835 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:02.990446 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-340b96-openshift-default-7846d978b9-8zqfs" event={"ID":"90e63fe7-52ef-4fbd-ab37-32c6479abb7a","Type":"ContainerDied","Data":"13d623ebf0266bc1b8dac94584e3afcdd93d0a16b1fc4853856223b02b2a2fa7"} Nov 03 00:59:02.990835 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:02.990473 4421 scope.go:122] "RemoveContainer" containerID="449161d828ac5cf3119b2d9c2232f2e6c7bfc9b3ca332c33155b19456a6492f5" Nov 03 00:59:03.002759 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:03.002732 4421 scope.go:122] "RemoveContainer" containerID="4727b4aed6f9fe13c30ac83ef9e469d024a700aaf2f264aea71481ee101489f8" Nov 03 00:59:03.062294 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:03.058409 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-340b96-openshift-default-7846d978b9-8zqfs"] Nov 03 00:59:03.062294 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:03.061658 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-340b96-openshift-default-7846d978b9-8zqfs"] Nov 03 00:59:03.064426 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:03.064117 4421 scope.go:122] "RemoveContainer" containerID="449161d828ac5cf3119b2d9c2232f2e6c7bfc9b3ca332c33155b19456a6492f5" Nov 03 00:59:03.064665 ip-10-0-133-217 kubenswrapper[4421]: E1103 00:59:03.064639 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"449161d828ac5cf3119b2d9c2232f2e6c7bfc9b3ca332c33155b19456a6492f5\": container with ID starting with 449161d828ac5cf3119b2d9c2232f2e6c7bfc9b3ca332c33155b19456a6492f5 not found: ID does not exist" containerID="449161d828ac5cf3119b2d9c2232f2e6c7bfc9b3ca332c33155b19456a6492f5" Nov 03 00:59:03.064707 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:03.064673 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"449161d828ac5cf3119b2d9c2232f2e6c7bfc9b3ca332c33155b19456a6492f5"} err="failed to get container status \"449161d828ac5cf3119b2d9c2232f2e6c7bfc9b3ca332c33155b19456a6492f5\": rpc error: code = NotFound desc = could not find container \"449161d828ac5cf3119b2d9c2232f2e6c7bfc9b3ca332c33155b19456a6492f5\": container with ID starting with 449161d828ac5cf3119b2d9c2232f2e6c7bfc9b3ca332c33155b19456a6492f5 not found: ID does not exist" Nov 03 00:59:03.064707 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:03.064695 4421 scope.go:122] "RemoveContainer" containerID="4727b4aed6f9fe13c30ac83ef9e469d024a700aaf2f264aea71481ee101489f8" Nov 03 00:59:03.065000 ip-10-0-133-217 kubenswrapper[4421]: E1103 00:59:03.064982 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4727b4aed6f9fe13c30ac83ef9e469d024a700aaf2f264aea71481ee101489f8\": container with ID starting with 4727b4aed6f9fe13c30ac83ef9e469d024a700aaf2f264aea71481ee101489f8 not found: ID does not exist" containerID="4727b4aed6f9fe13c30ac83ef9e469d024a700aaf2f264aea71481ee101489f8" Nov 03 00:59:03.065037 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:03.065010 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4727b4aed6f9fe13c30ac83ef9e469d024a700aaf2f264aea71481ee101489f8"} err="failed to get container status \"4727b4aed6f9fe13c30ac83ef9e469d024a700aaf2f264aea71481ee101489f8\": rpc error: code = NotFound desc = could not find container \"4727b4aed6f9fe13c30ac83ef9e469d024a700aaf2f264aea71481ee101489f8\": container with ID starting with 4727b4aed6f9fe13c30ac83ef9e469d024a700aaf2f264aea71481ee101489f8 not found: ID does not exist" Nov 03 00:59:03.157349 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:03.157305 4421 patch_prober.go:29] interesting pod/e2e-api-b-eed369-openshift-default-66698bcb95-pknkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body= Nov 03 00:59:03.157530 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:03.157370 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-eed369-openshift-default-66698bcb95-pknkp" podUID="b797c4ed-b8df-4e23-9188-3aeb3cf6c48f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" Nov 03 00:59:03.871648 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:03.871607 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-dd03e395-openshift-default-75799f6fc-kx54z"] Nov 03 00:59:03.871908 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:03.871883 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-dd03e395-openshift-default-75799f6fc-kx54z" podUID="01442ff5-1dab-4ed3-991b-ec0732c71305" containerName="istio-proxy" containerID="cri-o://07257ec70b22e16c24e895bbf7ddfea16c3a4228ef7975bb263a6ecb2c2b5086" gracePeriod=30 Nov 03 00:59:04.083746 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:04.064751 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90e63fe7-52ef-4fbd-ab37-32c6479abb7a" path="/var/lib/kubelet/pods/90e63fe7-52ef-4fbd-ab37-32c6479abb7a/volumes" Nov 03 00:59:04.096217 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:04.093072 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa2a0390-dbfb-46e6-9a54-f2522988748f" path="/var/lib/kubelet/pods/fa2a0390-dbfb-46e6-9a54-f2522988748f/volumes" Nov 03 00:59:04.156488 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:04.156384 4421 patch_prober.go:29] interesting pod/e2e-api-b-eed369-openshift-default-66698bcb95-pknkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body= Nov 03 00:59:04.156716 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:04.156594 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-eed369-openshift-default-66698bcb95-pknkp" podUID="b797c4ed-b8df-4e23-9188-3aeb3cf6c48f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" Nov 03 00:59:04.304334 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:04.304293 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-eed369-688b4d87b7-wdrsx"] Nov 03 00:59:04.990008 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:04.989967 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7f43f9-79d79dfff5-kpcxh"] Nov 03 00:59:04.990688 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:04.990669 4421 cpu_manager.go:414] "containerMap: removing container" podUID="a99273cf-6ddb-4006-9207-9af01a04128c" containerName="payload-processing" Nov 03 00:59:04.990688 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:04.990686 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a99273cf-6ddb-4006-9207-9af01a04128c" containerName="payload-processing" Nov 03 00:59:04.990885 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:04.990713 4421 cpu_manager.go:414] "containerMap: removing container" podUID="90e63fe7-52ef-4fbd-ab37-32c6479abb7a" containerName="istio-proxy" Nov 03 00:59:04.990885 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:04.990719 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90e63fe7-52ef-4fbd-ab37-32c6479abb7a" containerName="istio-proxy" Nov 03 00:59:04.990885 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:04.990738 4421 cpu_manager.go:414] "containerMap: removing container" podUID="fa2a0390-dbfb-46e6-9a54-f2522988748f" containerName="payload-pre-processing" Nov 03 00:59:04.990885 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:04.990744 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa2a0390-dbfb-46e6-9a54-f2522988748f" containerName="payload-pre-processing" Nov 03 00:59:04.990885 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:04.990754 4421 cpu_manager.go:414] "containerMap: removing container" podUID="90e63fe7-52ef-4fbd-ab37-32c6479abb7a" containerName="istio-proxy" Nov 03 00:59:04.990885 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:04.990759 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90e63fe7-52ef-4fbd-ab37-32c6479abb7a" containerName="istio-proxy" Nov 03 00:59:04.990885 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:04.990877 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="a99273cf-6ddb-4006-9207-9af01a04128c" containerName="payload-processing" Nov 03 00:59:04.990885 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:04.990888 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="90e63fe7-52ef-4fbd-ab37-32c6479abb7a" containerName="istio-proxy" Nov 03 00:59:04.991319 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:04.990898 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa2a0390-dbfb-46e6-9a54-f2522988748f" containerName="payload-pre-processing" Nov 03 00:59:04.991319 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:04.990905 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="90e63fe7-52ef-4fbd-ab37-32c6479abb7a" containerName="istio-proxy" Nov 03 00:59:04.994716 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:04.994693 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7f43f9-79d79dfff5-kpcxh" Nov 03 00:59:05.004379 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:05.004336 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7f43f9-79d79dfff5-kpcxh"] Nov 03 00:59:05.004634 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:05.004613 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-7f43f9\"" Nov 03 00:59:05.057946 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:05.047439 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-7f43f9-dockercfg-tvf85\"" Nov 03 00:59:05.057946 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:05.052527 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7f43f9-56467b6944-h8nzs"] Nov 03 00:59:05.059512 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:05.059490 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-7f43f9-56467b6944-h8nzs" Nov 03 00:59:05.071161 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:05.071127 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7f43f9-56467b6944-h8nzs"] Nov 03 00:59:05.175313 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:05.158997 4421 patch_prober.go:29] interesting pod/e2e-api-b-eed369-openshift-default-66698bcb95-pknkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body= Nov 03 00:59:05.195252 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:05.195204 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-eed369-openshift-default-66698bcb95-pknkp" podUID="b797c4ed-b8df-4e23-9188-3aeb3cf6c48f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" Nov 03 00:59:05.298475 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:05.160727 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0920df17-fa3c-497f-814e-0a1be8a141d5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-7f43f9-79d79dfff5-kpcxh\" (UID: \"0920df17-fa3c-497f-814e-0a1be8a141d5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7f43f9-79d79dfff5-kpcxh" Nov 03 00:59:05.299847 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:05.299729 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7f4xf\" (UniqueName: \"kubernetes.io/projected/0920df17-fa3c-497f-814e-0a1be8a141d5-kube-api-access-7f4xf\") pod \"payload-pre-processing-e2e-ipp-a-7f43f9-79d79dfff5-kpcxh\" (UID: \"0920df17-fa3c-497f-814e-0a1be8a141d5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7f43f9-79d79dfff5-kpcxh" Nov 03 00:59:05.413143 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:05.402625 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qn4q2\" (UniqueName: \"kubernetes.io/projected/18d503d0-9683-4d73-9c38-1991dcc19e44-kube-api-access-qn4q2\") pod \"payload-processing-e2e-ipp-a-7f43f9-56467b6944-h8nzs\" (UID: \"18d503d0-9683-4d73-9c38-1991dcc19e44\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7f43f9-56467b6944-h8nzs" Nov 03 00:59:05.413627 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:05.413595 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18d503d0-9683-4d73-9c38-1991dcc19e44-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-7f43f9-56467b6944-h8nzs\" (UID: \"18d503d0-9683-4d73-9c38-1991dcc19e44\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7f43f9-56467b6944-h8nzs" Nov 03 00:59:05.421724 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:05.413971 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7f4xf\" (UniqueName: \"kubernetes.io/projected/0920df17-fa3c-497f-814e-0a1be8a141d5-kube-api-access-7f4xf\") pod \"payload-pre-processing-e2e-ipp-a-7f43f9-79d79dfff5-kpcxh\" (UID: \"0920df17-fa3c-497f-814e-0a1be8a141d5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7f43f9-79d79dfff5-kpcxh" Nov 03 00:59:05.422137 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:05.422107 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0920df17-fa3c-497f-814e-0a1be8a141d5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-7f43f9-79d79dfff5-kpcxh\" (UID: \"0920df17-fa3c-497f-814e-0a1be8a141d5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7f43f9-79d79dfff5-kpcxh" Nov 03 00:59:05.448321 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:05.441741 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0920df17-fa3c-497f-814e-0a1be8a141d5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-7f43f9-79d79dfff5-kpcxh\" (UID: \"0920df17-fa3c-497f-814e-0a1be8a141d5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7f43f9-79d79dfff5-kpcxh" Nov 03 00:59:05.462360 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:05.462322 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7f4xf\" (UniqueName: \"kubernetes.io/projected/0920df17-fa3c-497f-814e-0a1be8a141d5-kube-api-access-7f4xf\") pod \"payload-pre-processing-e2e-ipp-a-7f43f9-79d79dfff5-kpcxh\" (UID: \"0920df17-fa3c-497f-814e-0a1be8a141d5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7f43f9-79d79dfff5-kpcxh" Nov 03 00:59:05.542818 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:05.542774 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qn4q2\" (UniqueName: \"kubernetes.io/projected/18d503d0-9683-4d73-9c38-1991dcc19e44-kube-api-access-qn4q2\") pod \"payload-processing-e2e-ipp-a-7f43f9-56467b6944-h8nzs\" (UID: \"18d503d0-9683-4d73-9c38-1991dcc19e44\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7f43f9-56467b6944-h8nzs" Nov 03 00:59:05.543000 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:05.542855 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18d503d0-9683-4d73-9c38-1991dcc19e44-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-7f43f9-56467b6944-h8nzs\" (UID: \"18d503d0-9683-4d73-9c38-1991dcc19e44\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7f43f9-56467b6944-h8nzs" Nov 03 00:59:05.543684 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:05.543662 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18d503d0-9683-4d73-9c38-1991dcc19e44-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-7f43f9-56467b6944-h8nzs\" (UID: \"18d503d0-9683-4d73-9c38-1991dcc19e44\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7f43f9-56467b6944-h8nzs" Nov 03 00:59:05.552047 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:05.551981 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qn4q2\" (UniqueName: \"kubernetes.io/projected/18d503d0-9683-4d73-9c38-1991dcc19e44-kube-api-access-qn4q2\") pod \"payload-processing-e2e-ipp-a-7f43f9-56467b6944-h8nzs\" (UID: \"18d503d0-9683-4d73-9c38-1991dcc19e44\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7f43f9-56467b6944-h8nzs" Nov 03 00:59:05.669582 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:05.669505 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7f43f9-79d79dfff5-kpcxh" Nov 03 00:59:05.685542 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:05.685509 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-7f43f9-56467b6944-h8nzs" Nov 03 00:59:06.149391 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:06.149263 4421 patch_prober.go:29] interesting pod/e2e-api-b-eed369-openshift-default-66698bcb95-pknkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body= Nov 03 00:59:06.149556 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:06.149407 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-eed369-openshift-default-66698bcb95-pknkp" podUID="b797c4ed-b8df-4e23-9188-3aeb3cf6c48f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" Nov 03 00:59:06.183248 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:59:06.183214 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0920df17_fa3c_497f_814e_0a1be8a141d5.slice/crio-dcb646a729527ad4ac2cf8e8cfd79f17b5628f0eb95292d9a77807208288337d WatchSource:0}: Error finding container dcb646a729527ad4ac2cf8e8cfd79f17b5628f0eb95292d9a77807208288337d: Status 404 returned error can't find the container with id dcb646a729527ad4ac2cf8e8cfd79f17b5628f0eb95292d9a77807208288337d Nov 03 00:59:06.183681 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:06.183477 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7f43f9-79d79dfff5-kpcxh"] Nov 03 00:59:06.241657 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:59:06.241623 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18d503d0_9683_4d73_9c38_1991dcc19e44.slice/crio-56f68c3b82c0e2722e12701aeb7b4c200a867e4c45d18c0c71d2e3df6e18d20c WatchSource:0}: Error finding container 56f68c3b82c0e2722e12701aeb7b4c200a867e4c45d18c0c71d2e3df6e18d20c: Status 404 returned error can't find the container with id 56f68c3b82c0e2722e12701aeb7b4c200a867e4c45d18c0c71d2e3df6e18d20c Nov 03 00:59:06.242943 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:06.242915 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7f43f9-56467b6944-h8nzs"] Nov 03 00:59:07.215849 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:07.209134 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7f43f9-79d79dfff5-kpcxh" event={"ID":"0920df17-fa3c-497f-814e-0a1be8a141d5","Type":"ContainerStarted","Data":"e2ef93acdb747314ee973e212940057c20e8aa28ee9c8d18c3569bdbe8c60f15"} Nov 03 00:59:07.215849 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:07.209206 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7f43f9-79d79dfff5-kpcxh" event={"ID":"0920df17-fa3c-497f-814e-0a1be8a141d5","Type":"ContainerStarted","Data":"dcb646a729527ad4ac2cf8e8cfd79f17b5628f0eb95292d9a77807208288337d"} Nov 03 00:59:07.215849 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:07.210440 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7f43f9-79d79dfff5-kpcxh" Nov 03 00:59:07.215849 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:07.213666 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-7f43f9-56467b6944-h8nzs" event={"ID":"18d503d0-9683-4d73-9c38-1991dcc19e44","Type":"ContainerStarted","Data":"d6491115d035f3201c64478f5ec3a1b40ff296f6921b008330295ff7586f50e0"} Nov 03 00:59:07.215849 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:07.213716 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-7f43f9-56467b6944-h8nzs" event={"ID":"18d503d0-9683-4d73-9c38-1991dcc19e44","Type":"ContainerStarted","Data":"56f68c3b82c0e2722e12701aeb7b4c200a867e4c45d18c0c71d2e3df6e18d20c"} Nov 03 00:59:07.215849 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:07.214579 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-7f43f9-56467b6944-h8nzs" Nov 03 00:59:07.220349 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:07.191261 4421 patch_prober.go:29] interesting pod/e2e-api-b-eed369-openshift-default-66698bcb95-pknkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body= Nov 03 00:59:07.220349 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:07.218446 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-eed369-openshift-default-66698bcb95-pknkp" podUID="b797c4ed-b8df-4e23-9188-3aeb3cf6c48f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" Nov 03 00:59:07.296134 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:07.283546 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7f43f9-79d79dfff5-kpcxh" podStartSLOduration=3.283523852 podStartE2EDuration="3.283523852s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:59:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:59:07.272728666 +0000 UTC m=+1089.443731774" watchObservedRunningTime="2026-11-03 00:59:07.283523852 +0000 UTC m=+1089.454526949" Nov 03 00:59:07.362845 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:07.362770 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-7f43f9-56467b6944-h8nzs" podStartSLOduration=3.362747757 podStartE2EDuration="3.362747757s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:59:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:59:07.356402136 +0000 UTC m=+1089.527405231" watchObservedRunningTime="2026-11-03 00:59:07.362747757 +0000 UTC m=+1089.533750855" Nov 03 00:59:08.052673 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:08.052638 4421 patch_prober.go:29] interesting pod/e2e-mt-dd03e395-openshift-default-75799f6fc-kx54z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 00:59:08.052848 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:08.052689 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-dd03e395-openshift-default-75799f6fc-kx54z" podUID="01442ff5-1dab-4ed3-991b-ec0732c71305" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 00:59:08.154975 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:08.154940 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-eed369-openshift-default-66698bcb95-pknkp" Nov 03 00:59:08.155392 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:08.155366 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-eed369-openshift-default-66698bcb95-pknkp" Nov 03 00:59:08.156079 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:08.156058 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-eed369-openshift-default-66698bcb95-pknkp" Nov 03 00:59:08.157378 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:08.157359 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-db5422-openshift-default-696cf4dc99-75p47"] Nov 03 00:59:08.160916 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:08.160901 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-db5422-openshift-default-696cf4dc99-75p47" Nov 03 00:59:08.184975 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:08.184941 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-db5422-openshift-default-696cf4dc99-75p47"] Nov 03 00:59:08.189504 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:08.189471 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-db5422-openshift-default-dockercfg-qbhcl\"" Nov 03 00:59:08.281296 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:08.240330 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33f4acce-9a60-4f9c-811d-81e8abfcabe8-istio-podinfo\") pod \"e2e-ipp-b-db5422-openshift-default-696cf4dc99-75p47\" (UID: \"33f4acce-9a60-4f9c-811d-81e8abfcabe8\") " pod="openshift-ingress/e2e-ipp-b-db5422-openshift-default-696cf4dc99-75p47" Nov 03 00:59:08.306366 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:08.296717 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33f4acce-9a60-4f9c-811d-81e8abfcabe8-istio-token\") pod \"e2e-ipp-b-db5422-openshift-default-696cf4dc99-75p47\" (UID: \"33f4acce-9a60-4f9c-811d-81e8abfcabe8\") " pod="openshift-ingress/e2e-ipp-b-db5422-openshift-default-696cf4dc99-75p47" Nov 03 00:59:08.306639 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:08.306606 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33f4acce-9a60-4f9c-811d-81e8abfcabe8-credential-socket\") pod \"e2e-ipp-b-db5422-openshift-default-696cf4dc99-75p47\" (UID: \"33f4acce-9a60-4f9c-811d-81e8abfcabe8\") " pod="openshift-ingress/e2e-ipp-b-db5422-openshift-default-696cf4dc99-75p47" Nov 03 00:59:08.314483 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:08.310336 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33f4acce-9a60-4f9c-811d-81e8abfcabe8-istio-data\") pod \"e2e-ipp-b-db5422-openshift-default-696cf4dc99-75p47\" (UID: \"33f4acce-9a60-4f9c-811d-81e8abfcabe8\") " pod="openshift-ingress/e2e-ipp-b-db5422-openshift-default-696cf4dc99-75p47" Nov 03 00:59:08.336298 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:08.314831 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33f4acce-9a60-4f9c-811d-81e8abfcabe8-istio-envoy\") pod \"e2e-ipp-b-db5422-openshift-default-696cf4dc99-75p47\" (UID: \"33f4acce-9a60-4f9c-811d-81e8abfcabe8\") " pod="openshift-ingress/e2e-ipp-b-db5422-openshift-default-696cf4dc99-75p47" Nov 03 00:59:08.336606 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:08.336575 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33f4acce-9a60-4f9c-811d-81e8abfcabe8-workload-certs\") pod \"e2e-ipp-b-db5422-openshift-default-696cf4dc99-75p47\" (UID: \"33f4acce-9a60-4f9c-811d-81e8abfcabe8\") " pod="openshift-ingress/e2e-ipp-b-db5422-openshift-default-696cf4dc99-75p47" Nov 03 00:59:08.336783 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:08.336761 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvj9w\" (UniqueName: \"kubernetes.io/projected/33f4acce-9a60-4f9c-811d-81e8abfcabe8-kube-api-access-rvj9w\") pod \"e2e-ipp-b-db5422-openshift-default-696cf4dc99-75p47\" (UID: \"33f4acce-9a60-4f9c-811d-81e8abfcabe8\") " pod="openshift-ingress/e2e-ipp-b-db5422-openshift-default-696cf4dc99-75p47" Nov 03 00:59:08.336976 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:08.336955 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33f4acce-9a60-4f9c-811d-81e8abfcabe8-istiod-ca-cert\") pod \"e2e-ipp-b-db5422-openshift-default-696cf4dc99-75p47\" (UID: \"33f4acce-9a60-4f9c-811d-81e8abfcabe8\") " pod="openshift-ingress/e2e-ipp-b-db5422-openshift-default-696cf4dc99-75p47" Nov 03 00:59:08.337308 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:08.337284 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33f4acce-9a60-4f9c-811d-81e8abfcabe8-workload-socket\") pod \"e2e-ipp-b-db5422-openshift-default-696cf4dc99-75p47\" (UID: \"33f4acce-9a60-4f9c-811d-81e8abfcabe8\") " pod="openshift-ingress/e2e-ipp-b-db5422-openshift-default-696cf4dc99-75p47" Nov 03 00:59:08.504232 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:08.503762 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33f4acce-9a60-4f9c-811d-81e8abfcabe8-workload-socket\") pod \"e2e-ipp-b-db5422-openshift-default-696cf4dc99-75p47\" (UID: \"33f4acce-9a60-4f9c-811d-81e8abfcabe8\") " pod="openshift-ingress/e2e-ipp-b-db5422-openshift-default-696cf4dc99-75p47" Nov 03 00:59:08.504548 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:08.504519 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33f4acce-9a60-4f9c-811d-81e8abfcabe8-istio-podinfo\") pod \"e2e-ipp-b-db5422-openshift-default-696cf4dc99-75p47\" (UID: \"33f4acce-9a60-4f9c-811d-81e8abfcabe8\") " pod="openshift-ingress/e2e-ipp-b-db5422-openshift-default-696cf4dc99-75p47" Nov 03 00:59:08.504818 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:08.504788 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33f4acce-9a60-4f9c-811d-81e8abfcabe8-istio-token\") pod \"e2e-ipp-b-db5422-openshift-default-696cf4dc99-75p47\" (UID: \"33f4acce-9a60-4f9c-811d-81e8abfcabe8\") " pod="openshift-ingress/e2e-ipp-b-db5422-openshift-default-696cf4dc99-75p47" Nov 03 00:59:08.504927 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:08.504831 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33f4acce-9a60-4f9c-811d-81e8abfcabe8-credential-socket\") pod \"e2e-ipp-b-db5422-openshift-default-696cf4dc99-75p47\" (UID: \"33f4acce-9a60-4f9c-811d-81e8abfcabe8\") " pod="openshift-ingress/e2e-ipp-b-db5422-openshift-default-696cf4dc99-75p47" Nov 03 00:59:08.505115 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:08.505093 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33f4acce-9a60-4f9c-811d-81e8abfcabe8-credential-socket\") pod \"e2e-ipp-b-db5422-openshift-default-696cf4dc99-75p47\" (UID: \"33f4acce-9a60-4f9c-811d-81e8abfcabe8\") " pod="openshift-ingress/e2e-ipp-b-db5422-openshift-default-696cf4dc99-75p47" Nov 03 00:59:08.505187 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:08.504202 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33f4acce-9a60-4f9c-811d-81e8abfcabe8-workload-socket\") pod \"e2e-ipp-b-db5422-openshift-default-696cf4dc99-75p47\" (UID: \"33f4acce-9a60-4f9c-811d-81e8abfcabe8\") " pod="openshift-ingress/e2e-ipp-b-db5422-openshift-default-696cf4dc99-75p47" Nov 03 00:59:08.505338 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:08.505307 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33f4acce-9a60-4f9c-811d-81e8abfcabe8-istio-data\") pod \"e2e-ipp-b-db5422-openshift-default-696cf4dc99-75p47\" (UID: \"33f4acce-9a60-4f9c-811d-81e8abfcabe8\") " pod="openshift-ingress/e2e-ipp-b-db5422-openshift-default-696cf4dc99-75p47" Nov 03 00:59:08.505471 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:08.505374 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33f4acce-9a60-4f9c-811d-81e8abfcabe8-istio-envoy\") pod \"e2e-ipp-b-db5422-openshift-default-696cf4dc99-75p47\" (UID: \"33f4acce-9a60-4f9c-811d-81e8abfcabe8\") " pod="openshift-ingress/e2e-ipp-b-db5422-openshift-default-696cf4dc99-75p47" Nov 03 00:59:08.505471 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:08.505409 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33f4acce-9a60-4f9c-811d-81e8abfcabe8-workload-certs\") pod \"e2e-ipp-b-db5422-openshift-default-696cf4dc99-75p47\" (UID: \"33f4acce-9a60-4f9c-811d-81e8abfcabe8\") " pod="openshift-ingress/e2e-ipp-b-db5422-openshift-default-696cf4dc99-75p47" Nov 03 00:59:08.505754 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:08.505470 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvj9w\" (UniqueName: \"kubernetes.io/projected/33f4acce-9a60-4f9c-811d-81e8abfcabe8-kube-api-access-rvj9w\") pod \"e2e-ipp-b-db5422-openshift-default-696cf4dc99-75p47\" (UID: \"33f4acce-9a60-4f9c-811d-81e8abfcabe8\") " pod="openshift-ingress/e2e-ipp-b-db5422-openshift-default-696cf4dc99-75p47" Nov 03 00:59:08.505754 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:08.505535 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33f4acce-9a60-4f9c-811d-81e8abfcabe8-istiod-ca-cert\") pod \"e2e-ipp-b-db5422-openshift-default-696cf4dc99-75p47\" (UID: \"33f4acce-9a60-4f9c-811d-81e8abfcabe8\") " pod="openshift-ingress/e2e-ipp-b-db5422-openshift-default-696cf4dc99-75p47" Nov 03 00:59:08.505754 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:08.505673 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33f4acce-9a60-4f9c-811d-81e8abfcabe8-istio-data\") pod \"e2e-ipp-b-db5422-openshift-default-696cf4dc99-75p47\" (UID: \"33f4acce-9a60-4f9c-811d-81e8abfcabe8\") " pod="openshift-ingress/e2e-ipp-b-db5422-openshift-default-696cf4dc99-75p47" Nov 03 00:59:08.506032 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:08.505998 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33f4acce-9a60-4f9c-811d-81e8abfcabe8-workload-certs\") pod \"e2e-ipp-b-db5422-openshift-default-696cf4dc99-75p47\" (UID: \"33f4acce-9a60-4f9c-811d-81e8abfcabe8\") " pod="openshift-ingress/e2e-ipp-b-db5422-openshift-default-696cf4dc99-75p47" Nov 03 00:59:08.507247 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:08.507204 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33f4acce-9a60-4f9c-811d-81e8abfcabe8-istiod-ca-cert\") pod \"e2e-ipp-b-db5422-openshift-default-696cf4dc99-75p47\" (UID: \"33f4acce-9a60-4f9c-811d-81e8abfcabe8\") " pod="openshift-ingress/e2e-ipp-b-db5422-openshift-default-696cf4dc99-75p47" Nov 03 00:59:08.513958 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:08.513633 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33f4acce-9a60-4f9c-811d-81e8abfcabe8-istio-envoy\") pod \"e2e-ipp-b-db5422-openshift-default-696cf4dc99-75p47\" (UID: \"33f4acce-9a60-4f9c-811d-81e8abfcabe8\") " pod="openshift-ingress/e2e-ipp-b-db5422-openshift-default-696cf4dc99-75p47" Nov 03 00:59:08.514884 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:08.514857 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33f4acce-9a60-4f9c-811d-81e8abfcabe8-istio-podinfo\") pod \"e2e-ipp-b-db5422-openshift-default-696cf4dc99-75p47\" (UID: \"33f4acce-9a60-4f9c-811d-81e8abfcabe8\") " pod="openshift-ingress/e2e-ipp-b-db5422-openshift-default-696cf4dc99-75p47" Nov 03 00:59:08.519767 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:08.519736 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33f4acce-9a60-4f9c-811d-81e8abfcabe8-istio-token\") pod \"e2e-ipp-b-db5422-openshift-default-696cf4dc99-75p47\" (UID: \"33f4acce-9a60-4f9c-811d-81e8abfcabe8\") " pod="openshift-ingress/e2e-ipp-b-db5422-openshift-default-696cf4dc99-75p47" Nov 03 00:59:08.531360 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:08.526068 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvj9w\" (UniqueName: \"kubernetes.io/projected/33f4acce-9a60-4f9c-811d-81e8abfcabe8-kube-api-access-rvj9w\") pod \"e2e-ipp-b-db5422-openshift-default-696cf4dc99-75p47\" (UID: \"33f4acce-9a60-4f9c-811d-81e8abfcabe8\") " pod="openshift-ingress/e2e-ipp-b-db5422-openshift-default-696cf4dc99-75p47" Nov 03 00:59:08.774866 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:08.774828 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-db5422-openshift-default-696cf4dc99-75p47" Nov 03 00:59:09.141393 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:09.141136 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-db5422-openshift-default-696cf4dc99-75p47"] Nov 03 00:59:09.168074 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:09.168012 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:59:09.168227 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:09.168173 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:59:09.168554 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:09.168262 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:59:09.256841 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:09.256753 4421 generic.go:356] "Generic (PLEG): container finished" podID="01442ff5-1dab-4ed3-991b-ec0732c71305" containerID="07257ec70b22e16c24e895bbf7ddfea16c3a4228ef7975bb263a6ecb2c2b5086" exitCode=0 Nov 03 00:59:09.257364 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:09.257020 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-dd03e395-openshift-default-75799f6fc-kx54z" event={"ID":"01442ff5-1dab-4ed3-991b-ec0732c71305","Type":"ContainerDied","Data":"07257ec70b22e16c24e895bbf7ddfea16c3a4228ef7975bb263a6ecb2c2b5086"} Nov 03 00:59:09.260520 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:09.260489 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-db5422-openshift-default-696cf4dc99-75p47" event={"ID":"33f4acce-9a60-4f9c-811d-81e8abfcabe8","Type":"ContainerStarted","Data":"a55d07aed794ed8c392803da3d41f0277377785c34923f3086cf8839617cafae"} Nov 03 00:59:09.260730 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:09.260675 4421 scope.go:122] "RemoveContainer" containerID="5f145538a80b3358ab75e1c1021050da283eed8d6d1f9782ad088f6ef764e4a4" Nov 03 00:59:09.305852 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:09.305700 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-eed369-688b4d87b7-wdrsx" podUID="91b496b3-d1b2-475c-9ed5-a7ba4fe4d3a9" containerName="payload-pre-processing" containerID="cri-o://addccc54b9fb9e34fd686ca878c398e013ca166b2e27fc02ab6847dc1cb49049" gracePeriod=25 Nov 03 00:59:09.315876 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:09.311807 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-dd03e395-openshift-default-75799f6fc-kx54z" Nov 03 00:59:09.366828 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:09.365543 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01442ff5-1dab-4ed3-991b-ec0732c71305-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01442ff5-1dab-4ed3-991b-ec0732c71305-workload-socket\") pod \"01442ff5-1dab-4ed3-991b-ec0732c71305\" (UID: \"01442ff5-1dab-4ed3-991b-ec0732c71305\") " Nov 03 00:59:09.366828 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:09.365622 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01442ff5-1dab-4ed3-991b-ec0732c71305-istio-token\" (UniqueName: \"kubernetes.io/projected/01442ff5-1dab-4ed3-991b-ec0732c71305-istio-token\") pod \"01442ff5-1dab-4ed3-991b-ec0732c71305\" (UID: \"01442ff5-1dab-4ed3-991b-ec0732c71305\") " Nov 03 00:59:09.366828 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:09.365681 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01442ff5-1dab-4ed3-991b-ec0732c71305-kube-api-access-wgxqr\" (UniqueName: \"kubernetes.io/projected/01442ff5-1dab-4ed3-991b-ec0732c71305-kube-api-access-wgxqr\") pod \"01442ff5-1dab-4ed3-991b-ec0732c71305\" (UID: \"01442ff5-1dab-4ed3-991b-ec0732c71305\") " Nov 03 00:59:09.366828 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:09.365823 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01442ff5-1dab-4ed3-991b-ec0732c71305-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01442ff5-1dab-4ed3-991b-ec0732c71305-istio-envoy\") pod \"01442ff5-1dab-4ed3-991b-ec0732c71305\" (UID: \"01442ff5-1dab-4ed3-991b-ec0732c71305\") " Nov 03 00:59:09.366828 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:09.365896 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01442ff5-1dab-4ed3-991b-ec0732c71305-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01442ff5-1dab-4ed3-991b-ec0732c71305-istiod-ca-cert\") pod \"01442ff5-1dab-4ed3-991b-ec0732c71305\" (UID: \"01442ff5-1dab-4ed3-991b-ec0732c71305\") " Nov 03 00:59:09.366828 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:09.366005 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/01442ff5-1dab-4ed3-991b-ec0732c71305-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01442ff5-1dab-4ed3-991b-ec0732c71305-istio-podinfo\") pod \"01442ff5-1dab-4ed3-991b-ec0732c71305\" (UID: \"01442ff5-1dab-4ed3-991b-ec0732c71305\") " Nov 03 00:59:09.366828 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:09.366069 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01442ff5-1dab-4ed3-991b-ec0732c71305-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01442ff5-1dab-4ed3-991b-ec0732c71305-workload-certs\") pod \"01442ff5-1dab-4ed3-991b-ec0732c71305\" (UID: \"01442ff5-1dab-4ed3-991b-ec0732c71305\") " Nov 03 00:59:09.366828 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:09.366111 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01442ff5-1dab-4ed3-991b-ec0732c71305-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01442ff5-1dab-4ed3-991b-ec0732c71305-credential-socket\") pod \"01442ff5-1dab-4ed3-991b-ec0732c71305\" (UID: \"01442ff5-1dab-4ed3-991b-ec0732c71305\") " Nov 03 00:59:09.366828 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:09.366190 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01442ff5-1dab-4ed3-991b-ec0732c71305-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01442ff5-1dab-4ed3-991b-ec0732c71305-istio-data\") pod \"01442ff5-1dab-4ed3-991b-ec0732c71305\" (UID: \"01442ff5-1dab-4ed3-991b-ec0732c71305\") " Nov 03 00:59:09.367650 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:09.367410 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01442ff5-1dab-4ed3-991b-ec0732c71305-workload-certs" pod "01442ff5-1dab-4ed3-991b-ec0732c71305" (UID: "01442ff5-1dab-4ed3-991b-ec0732c71305"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:59:09.367650 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:09.367503 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01442ff5-1dab-4ed3-991b-ec0732c71305-istiod-ca-cert" pod "01442ff5-1dab-4ed3-991b-ec0732c71305" (UID: "01442ff5-1dab-4ed3-991b-ec0732c71305"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:59:09.367844 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:09.367710 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01442ff5-1dab-4ed3-991b-ec0732c71305-credential-socket" pod "01442ff5-1dab-4ed3-991b-ec0732c71305" (UID: "01442ff5-1dab-4ed3-991b-ec0732c71305"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:59:09.367921 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:09.367869 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01442ff5-1dab-4ed3-991b-ec0732c71305-istio-data" pod "01442ff5-1dab-4ed3-991b-ec0732c71305" (UID: "01442ff5-1dab-4ed3-991b-ec0732c71305"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:59:09.368076 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:09.368022 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01442ff5-1dab-4ed3-991b-ec0732c71305-workload-socket" pod "01442ff5-1dab-4ed3-991b-ec0732c71305" (UID: "01442ff5-1dab-4ed3-991b-ec0732c71305"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:59:09.369994 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:09.369874 4421 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01442ff5-1dab-4ed3-991b-ec0732c71305-workload-certs\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:59:09.369994 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:09.369902 4421 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01442ff5-1dab-4ed3-991b-ec0732c71305-credential-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:59:09.369994 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:09.369917 4421 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01442ff5-1dab-4ed3-991b-ec0732c71305-istio-data\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:59:09.369994 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:09.369932 4421 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01442ff5-1dab-4ed3-991b-ec0732c71305-workload-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:59:09.369994 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:09.369946 4421 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01442ff5-1dab-4ed3-991b-ec0732c71305-istiod-ca-cert\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:59:09.372008 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:09.371978 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01442ff5-1dab-4ed3-991b-ec0732c71305-istio-envoy" pod "01442ff5-1dab-4ed3-991b-ec0732c71305" (UID: "01442ff5-1dab-4ed3-991b-ec0732c71305"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:59:09.372592 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:09.372566 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01442ff5-1dab-4ed3-991b-ec0732c71305-istio-token" pod "01442ff5-1dab-4ed3-991b-ec0732c71305" (UID: "01442ff5-1dab-4ed3-991b-ec0732c71305"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:59:09.373002 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:09.372958 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/01442ff5-1dab-4ed3-991b-ec0732c71305-istio-podinfo" pod "01442ff5-1dab-4ed3-991b-ec0732c71305" (UID: "01442ff5-1dab-4ed3-991b-ec0732c71305"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:59:09.374147 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:09.374125 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01442ff5-1dab-4ed3-991b-ec0732c71305-kube-api-access-wgxqr" pod "01442ff5-1dab-4ed3-991b-ec0732c71305" (UID: "01442ff5-1dab-4ed3-991b-ec0732c71305"). InnerVolumeSpecName "kube-api-access-wgxqr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:59:09.473028 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:09.472952 4421 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01442ff5-1dab-4ed3-991b-ec0732c71305-istio-token\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:59:09.473645 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:09.473365 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wgxqr\" (UniqueName: \"kubernetes.io/projected/01442ff5-1dab-4ed3-991b-ec0732c71305-kube-api-access-wgxqr\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:59:09.473645 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:09.473405 4421 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01442ff5-1dab-4ed3-991b-ec0732c71305-istio-envoy\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:59:09.473645 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:09.473421 4421 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01442ff5-1dab-4ed3-991b-ec0732c71305-istio-podinfo\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:59:09.828596 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:09.828523 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-eed369-688b4d87b7-wdrsx" Nov 03 00:59:09.942106 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:09.941390 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91b496b3-d1b2-475c-9ed5-a7ba4fe4d3a9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91b496b3-d1b2-475c-9ed5-a7ba4fe4d3a9-plugins-config-volume\") pod \"91b496b3-d1b2-475c-9ed5-a7ba4fe4d3a9\" (UID: \"91b496b3-d1b2-475c-9ed5-a7ba4fe4d3a9\") " Nov 03 00:59:09.942106 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:09.941597 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91b496b3-d1b2-475c-9ed5-a7ba4fe4d3a9-kube-api-access-4nxfw\" (UniqueName: \"kubernetes.io/projected/91b496b3-d1b2-475c-9ed5-a7ba4fe4d3a9-kube-api-access-4nxfw\") pod \"91b496b3-d1b2-475c-9ed5-a7ba4fe4d3a9\" (UID: \"91b496b3-d1b2-475c-9ed5-a7ba4fe4d3a9\") " Nov 03 00:59:09.980319 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:09.969758 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91b496b3-d1b2-475c-9ed5-a7ba4fe4d3a9-plugins-config-volume" pod "91b496b3-d1b2-475c-9ed5-a7ba4fe4d3a9" (UID: "91b496b3-d1b2-475c-9ed5-a7ba4fe4d3a9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:59:10.015444 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:10.014014 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91b496b3-d1b2-475c-9ed5-a7ba4fe4d3a9-kube-api-access-4nxfw" pod "91b496b3-d1b2-475c-9ed5-a7ba4fe4d3a9" (UID: "91b496b3-d1b2-475c-9ed5-a7ba4fe4d3a9"). InnerVolumeSpecName "kube-api-access-4nxfw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:59:10.045522 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:10.044978 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4nxfw\" (UniqueName: \"kubernetes.io/projected/91b496b3-d1b2-475c-9ed5-a7ba4fe4d3a9-kube-api-access-4nxfw\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:59:10.045522 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:10.045018 4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91b496b3-d1b2-475c-9ed5-a7ba4fe4d3a9-plugins-config-volume\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:59:10.275938 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:10.275903 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-dd03e395-openshift-default-75799f6fc-kx54z" Nov 03 00:59:10.276162 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:10.275985 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-dd03e395-openshift-default-75799f6fc-kx54z" event={"ID":"01442ff5-1dab-4ed3-991b-ec0732c71305","Type":"ContainerDied","Data":"70d1cea667022cdb8212eb6278d1ef71a0d1a5286d2b8ab4786b76cbee3ed3e1"} Nov 03 00:59:10.276162 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:10.276037 4421 scope.go:122] "RemoveContainer" containerID="07257ec70b22e16c24e895bbf7ddfea16c3a4228ef7975bb263a6ecb2c2b5086" Nov 03 00:59:10.278153 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:10.278123 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-db5422-openshift-default-696cf4dc99-75p47" event={"ID":"33f4acce-9a60-4f9c-811d-81e8abfcabe8","Type":"ContainerStarted","Data":"88750a243ff2f3dd38bac3810635e35888160a711beb60db021fceed8d99aeb1"} Nov 03 00:59:10.283967 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:10.283944 4421 generic.go:356] "Generic (PLEG): container finished" podID="91b496b3-d1b2-475c-9ed5-a7ba4fe4d3a9" containerID="addccc54b9fb9e34fd686ca878c398e013ca166b2e27fc02ab6847dc1cb49049" exitCode=0 Nov 03 00:59:10.284065 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:10.284022 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-eed369-688b4d87b7-wdrsx" event={"ID":"91b496b3-d1b2-475c-9ed5-a7ba4fe4d3a9","Type":"ContainerDied","Data":"addccc54b9fb9e34fd686ca878c398e013ca166b2e27fc02ab6847dc1cb49049"} Nov 03 00:59:10.284136 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:10.284081 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-eed369-688b4d87b7-wdrsx" event={"ID":"91b496b3-d1b2-475c-9ed5-a7ba4fe4d3a9","Type":"ContainerDied","Data":"5269ee9dfeb24e8663cd7385067a2ef8b43eaadf5185538be729d70a49bb9876"} Nov 03 00:59:10.284136 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:10.284132 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-eed369-688b4d87b7-wdrsx" Nov 03 00:59:10.292425 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:10.292396 4421 scope.go:122] "RemoveContainer" containerID="addccc54b9fb9e34fd686ca878c398e013ca166b2e27fc02ab6847dc1cb49049" Nov 03 00:59:10.322472 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:10.322423 4421 scope.go:122] "RemoveContainer" containerID="addccc54b9fb9e34fd686ca878c398e013ca166b2e27fc02ab6847dc1cb49049" Nov 03 00:59:10.334363 ip-10-0-133-217 kubenswrapper[4421]: E1103 00:59:10.334247 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"addccc54b9fb9e34fd686ca878c398e013ca166b2e27fc02ab6847dc1cb49049\": container with ID starting with addccc54b9fb9e34fd686ca878c398e013ca166b2e27fc02ab6847dc1cb49049 not found: ID does not exist" containerID="addccc54b9fb9e34fd686ca878c398e013ca166b2e27fc02ab6847dc1cb49049" Nov 03 00:59:10.335132 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:10.335093 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"addccc54b9fb9e34fd686ca878c398e013ca166b2e27fc02ab6847dc1cb49049"} err="failed to get container status \"addccc54b9fb9e34fd686ca878c398e013ca166b2e27fc02ab6847dc1cb49049\": rpc error: code = NotFound desc = could not find container \"addccc54b9fb9e34fd686ca878c398e013ca166b2e27fc02ab6847dc1cb49049\": container with ID starting with addccc54b9fb9e34fd686ca878c398e013ca166b2e27fc02ab6847dc1cb49049 not found: ID does not exist" Nov 03 00:59:10.347604 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:10.344360 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-dd03e395-openshift-default-75799f6fc-kx54z"] Nov 03 00:59:10.352507 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:10.351763 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-dd03e395-openshift-default-75799f6fc-kx54z"] Nov 03 00:59:10.357211 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:10.357158 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-db5422-openshift-default-696cf4dc99-75p47" podStartSLOduration=2.357142639 podStartE2EDuration="2.357142639s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:59:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:59:10.345957375 +0000 UTC m=+1092.516960468" watchObservedRunningTime="2026-11-03 00:59:10.357142639 +0000 UTC m=+1092.528145842" Nov 03 00:59:10.375104 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:10.375070 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-eed369-688b4d87b7-wdrsx"] Nov 03 00:59:10.381554 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:10.381514 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-eed369-688b4d87b7-wdrsx"] Nov 03 00:59:10.842305 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:10.841532 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-db5422-openshift-default-696cf4dc99-75p47" Nov 03 00:59:10.969116 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:10.968805 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-db5422-openshift-default-696cf4dc99-75p47" Nov 03 00:59:11.035390 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:11.035344 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1866d6d9-lr7c7"] Nov 03 00:59:11.045740 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:11.042865 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1866d6d9-lr7c7"] Nov 03 00:59:11.294873 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:11.294842 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-db5422-openshift-default-696cf4dc99-75p47" Nov 03 00:59:11.295889 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:11.295869 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-db5422-openshift-default-696cf4dc99-75p47" Nov 03 00:59:11.982610 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:11.982574 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01442ff5-1dab-4ed3-991b-ec0732c71305" path="/var/lib/kubelet/pods/01442ff5-1dab-4ed3-991b-ec0732c71305/volumes" Nov 03 00:59:11.983386 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:11.983360 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91b01607-71bd-4c79-805d-1437b55de97f" path="/var/lib/kubelet/pods/91b01607-71bd-4c79-805d-1437b55de97f/volumes" Nov 03 00:59:11.983936 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:11.983916 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91b496b3-d1b2-475c-9ed5-a7ba4fe4d3a9" path="/var/lib/kubelet/pods/91b496b3-d1b2-475c-9ed5-a7ba4fe4d3a9/volumes" Nov 03 00:59:12.070747 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:12.070711 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f372f7f-9zxcl"] Nov 03 00:59:12.078075 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:12.078041 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f372f7f-9zxcl"] Nov 03 00:59:13.603971 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:13.603934 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-47369d-7c8f86b795-hclwx"] Nov 03 00:59:13.604662 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:13.604646 4421 cpu_manager.go:414] "containerMap: removing container" podUID="01442ff5-1dab-4ed3-991b-ec0732c71305" containerName="istio-proxy" Nov 03 00:59:13.604662 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:13.604664 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01442ff5-1dab-4ed3-991b-ec0732c71305" containerName="istio-proxy" Nov 03 00:59:13.604783 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:13.604680 4421 cpu_manager.go:414] "containerMap: removing container" podUID="91b496b3-d1b2-475c-9ed5-a7ba4fe4d3a9" containerName="payload-pre-processing" Nov 03 00:59:13.604783 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:13.604685 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91b496b3-d1b2-475c-9ed5-a7ba4fe4d3a9" containerName="payload-pre-processing" Nov 03 00:59:13.604862 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:13.604846 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="01442ff5-1dab-4ed3-991b-ec0732c71305" containerName="istio-proxy" Nov 03 00:59:13.604862 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:13.604859 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="91b496b3-d1b2-475c-9ed5-a7ba4fe4d3a9" containerName="payload-pre-processing" Nov 03 00:59:13.604937 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:13.604869 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="01442ff5-1dab-4ed3-991b-ec0732c71305" containerName="istio-proxy" Nov 03 00:59:13.633816 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:13.625430 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-47369d-7c8f86b795-hclwx" Nov 03 00:59:13.644186 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:13.644155 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-47369d-6fbdd5b9b7-h78wk"] Nov 03 00:59:13.645955 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:13.645459 4421 cpu_manager.go:414] "containerMap: removing container" podUID="01442ff5-1dab-4ed3-991b-ec0732c71305" containerName="istio-proxy" Nov 03 00:59:13.646161 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:13.646137 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-47369d\"" Nov 03 00:59:13.646319 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:13.646300 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-47369d-dockercfg-9b24h\"" Nov 03 00:59:13.646400 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:13.646136 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01442ff5-1dab-4ed3-991b-ec0732c71305" containerName="istio-proxy" Nov 03 00:59:13.653404 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:13.653376 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-47369d-6fbdd5b9b7-h78wk" Nov 03 00:59:13.658219 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:13.658185 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-47369d-7c8f86b795-hclwx"] Nov 03 00:59:13.671634 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:13.671598 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-47369d-6fbdd5b9b7-h78wk"] Nov 03 00:59:13.833297 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:13.798560 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4145d1f-cba3-4b46-a0ee-5ac18eec30ea-plugins-config-volume\") pod \"payload-processing-e2e-api-b-47369d-7c8f86b795-hclwx\" (UID: \"f4145d1f-cba3-4b46-a0ee-5ac18eec30ea\") " pod="openshift-ingress/payload-processing-e2e-api-b-47369d-7c8f86b795-hclwx" Nov 03 00:59:13.844309 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:13.833704 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46ntg\" (UniqueName: \"kubernetes.io/projected/f465a3d3-72ba-40a8-a6a2-dc5dd77efcce-kube-api-access-46ntg\") pod \"payload-pre-processing-e2e-api-b-47369d-6fbdd5b9b7-h78wk\" (UID: \"f465a3d3-72ba-40a8-a6a2-dc5dd77efcce\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-47369d-6fbdd5b9b7-h78wk" Nov 03 00:59:13.856358 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:13.844692 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2cfj\" (UniqueName: \"kubernetes.io/projected/f4145d1f-cba3-4b46-a0ee-5ac18eec30ea-kube-api-access-p2cfj\") pod \"payload-processing-e2e-api-b-47369d-7c8f86b795-hclwx\" (UID: \"f4145d1f-cba3-4b46-a0ee-5ac18eec30ea\") " pod="openshift-ingress/payload-processing-e2e-api-b-47369d-7c8f86b795-hclwx" Nov 03 00:59:13.865496 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:13.865454 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f465a3d3-72ba-40a8-a6a2-dc5dd77efcce-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-47369d-6fbdd5b9b7-h78wk\" (UID: \"f465a3d3-72ba-40a8-a6a2-dc5dd77efcce\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-47369d-6fbdd5b9b7-h78wk" Nov 03 00:59:13.971139 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:13.970633 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2cfj\" (UniqueName: \"kubernetes.io/projected/f4145d1f-cba3-4b46-a0ee-5ac18eec30ea-kube-api-access-p2cfj\") pod \"payload-processing-e2e-api-b-47369d-7c8f86b795-hclwx\" (UID: \"f4145d1f-cba3-4b46-a0ee-5ac18eec30ea\") " pod="openshift-ingress/payload-processing-e2e-api-b-47369d-7c8f86b795-hclwx" Nov 03 00:59:13.976829 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:13.972457 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f465a3d3-72ba-40a8-a6a2-dc5dd77efcce-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-47369d-6fbdd5b9b7-h78wk\" (UID: \"f465a3d3-72ba-40a8-a6a2-dc5dd77efcce\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-47369d-6fbdd5b9b7-h78wk" Nov 03 00:59:13.976829 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:13.972628 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4145d1f-cba3-4b46-a0ee-5ac18eec30ea-plugins-config-volume\") pod \"payload-processing-e2e-api-b-47369d-7c8f86b795-hclwx\" (UID: \"f4145d1f-cba3-4b46-a0ee-5ac18eec30ea\") " pod="openshift-ingress/payload-processing-e2e-api-b-47369d-7c8f86b795-hclwx" Nov 03 00:59:13.976829 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:13.972785 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46ntg\" (UniqueName: \"kubernetes.io/projected/f465a3d3-72ba-40a8-a6a2-dc5dd77efcce-kube-api-access-46ntg\") pod \"payload-pre-processing-e2e-api-b-47369d-6fbdd5b9b7-h78wk\" (UID: \"f465a3d3-72ba-40a8-a6a2-dc5dd77efcce\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-47369d-6fbdd5b9b7-h78wk" Nov 03 00:59:13.976829 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:13.974147 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f465a3d3-72ba-40a8-a6a2-dc5dd77efcce-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-47369d-6fbdd5b9b7-h78wk\" (UID: \"f465a3d3-72ba-40a8-a6a2-dc5dd77efcce\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-47369d-6fbdd5b9b7-h78wk" Nov 03 00:59:13.976829 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:13.975033 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4145d1f-cba3-4b46-a0ee-5ac18eec30ea-plugins-config-volume\") pod \"payload-processing-e2e-api-b-47369d-7c8f86b795-hclwx\" (UID: \"f4145d1f-cba3-4b46-a0ee-5ac18eec30ea\") " pod="openshift-ingress/payload-processing-e2e-api-b-47369d-7c8f86b795-hclwx" Nov 03 00:59:13.997399 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:13.997360 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c855b88-d5c6-4c09-b6e7-3685dd1e57a1" path="/var/lib/kubelet/pods/6c855b88-d5c6-4c09-b6e7-3685dd1e57a1/volumes" Nov 03 00:59:14.000506 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:14.000475 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2cfj\" (UniqueName: \"kubernetes.io/projected/f4145d1f-cba3-4b46-a0ee-5ac18eec30ea-kube-api-access-p2cfj\") pod \"payload-processing-e2e-api-b-47369d-7c8f86b795-hclwx\" (UID: \"f4145d1f-cba3-4b46-a0ee-5ac18eec30ea\") " pod="openshift-ingress/payload-processing-e2e-api-b-47369d-7c8f86b795-hclwx" Nov 03 00:59:14.004368 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:14.004329 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-46ntg\" (UniqueName: \"kubernetes.io/projected/f465a3d3-72ba-40a8-a6a2-dc5dd77efcce-kube-api-access-46ntg\") pod \"payload-pre-processing-e2e-api-b-47369d-6fbdd5b9b7-h78wk\" (UID: \"f465a3d3-72ba-40a8-a6a2-dc5dd77efcce\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-47369d-6fbdd5b9b7-h78wk" Nov 03 00:59:14.077924 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:14.077895 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-47369d-6fbdd5b9b7-h78wk" Nov 03 00:59:14.268388 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:14.267774 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-47369d-7c8f86b795-hclwx" Nov 03 00:59:14.452408 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:14.452267 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-47369d-6fbdd5b9b7-h78wk"] Nov 03 00:59:14.682802 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:14.682695 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-47369d-7c8f86b795-hclwx"] Nov 03 00:59:14.683210 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:59:14.683153 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4145d1f_cba3_4b46_a0ee_5ac18eec30ea.slice/crio-0206c65eae78025839f73f77c44890d1cac2d89d8e025aaed631323bbc4f13ac WatchSource:0}: Error finding container 0206c65eae78025839f73f77c44890d1cac2d89d8e025aaed631323bbc4f13ac: Status 404 returned error can't find the container with id 0206c65eae78025839f73f77c44890d1cac2d89d8e025aaed631323bbc4f13ac Nov 03 00:59:14.742757 ip-10-0-133-217 kubenswrapper[4421]: E1103 00:59:14.742718 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 00:59:15.145848 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:15.145803 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-db5422-5f7b97b655-4nwn5"] Nov 03 00:59:15.151407 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:15.151381 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-db5422-5f7b97b655-4nwn5" Nov 03 00:59:15.152565 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:15.152537 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-db5422-5f7b97b655-4nwn5"] Nov 03 00:59:15.157577 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:15.157557 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-db5422\"" Nov 03 00:59:15.194648 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:15.194614 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldgnc\" (UniqueName: \"kubernetes.io/projected/e362c3ac-70d7-487a-a181-47d9ffd3dd17-kube-api-access-ldgnc\") pod \"payload-pre-processing-e2e-ipp-b-db5422-5f7b97b655-4nwn5\" (UID: \"e362c3ac-70d7-487a-a181-47d9ffd3dd17\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-db5422-5f7b97b655-4nwn5" Nov 03 00:59:15.194812 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:15.194661 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e362c3ac-70d7-487a-a181-47d9ffd3dd17-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-db5422-5f7b97b655-4nwn5\" (UID: \"e362c3ac-70d7-487a-a181-47d9ffd3dd17\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-db5422-5f7b97b655-4nwn5" Nov 03 00:59:15.338899 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:15.330593 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldgnc\" (UniqueName: \"kubernetes.io/projected/e362c3ac-70d7-487a-a181-47d9ffd3dd17-kube-api-access-ldgnc\") pod \"payload-pre-processing-e2e-ipp-b-db5422-5f7b97b655-4nwn5\" (UID: \"e362c3ac-70d7-487a-a181-47d9ffd3dd17\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-db5422-5f7b97b655-4nwn5" Nov 03 00:59:15.338899 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:15.330655 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e362c3ac-70d7-487a-a181-47d9ffd3dd17-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-db5422-5f7b97b655-4nwn5\" (UID: \"e362c3ac-70d7-487a-a181-47d9ffd3dd17\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-db5422-5f7b97b655-4nwn5" Nov 03 00:59:15.361309 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:15.360037 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e362c3ac-70d7-487a-a181-47d9ffd3dd17-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-db5422-5f7b97b655-4nwn5\" (UID: \"e362c3ac-70d7-487a-a181-47d9ffd3dd17\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-db5422-5f7b97b655-4nwn5" Nov 03 00:59:15.418356 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:15.417555 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldgnc\" (UniqueName: \"kubernetes.io/projected/e362c3ac-70d7-487a-a181-47d9ffd3dd17-kube-api-access-ldgnc\") pod \"payload-pre-processing-e2e-ipp-b-db5422-5f7b97b655-4nwn5\" (UID: \"e362c3ac-70d7-487a-a181-47d9ffd3dd17\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-db5422-5f7b97b655-4nwn5" Nov 03 00:59:15.451298 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:15.447813 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-47369d-6fbdd5b9b7-h78wk" event={"ID":"f465a3d3-72ba-40a8-a6a2-dc5dd77efcce","Type":"ContainerStarted","Data":"68fa0f326c088c4f670c1b58a9a650eb38bb6c945d71865802da38d00d796fc3"} Nov 03 00:59:15.451298 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:15.447885 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-47369d-6fbdd5b9b7-h78wk" event={"ID":"f465a3d3-72ba-40a8-a6a2-dc5dd77efcce","Type":"ContainerStarted","Data":"8b44f29c12a455add0a423506f796212ff67957d9de4a5df3375096cdd6b023b"} Nov 03 00:59:15.451298 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:15.449050 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-47369d-6fbdd5b9b7-h78wk" Nov 03 00:59:15.476296 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:15.461131 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-47369d-7c8f86b795-hclwx" event={"ID":"f4145d1f-cba3-4b46-a0ee-5ac18eec30ea","Type":"ContainerStarted","Data":"07b096b853ae222023a3437fe30feffde013cea4508ca41ad51ce5800e3de357"} Nov 03 00:59:15.476296 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:15.461200 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-47369d-7c8f86b795-hclwx" event={"ID":"f4145d1f-cba3-4b46-a0ee-5ac18eec30ea","Type":"ContainerStarted","Data":"0206c65eae78025839f73f77c44890d1cac2d89d8e025aaed631323bbc4f13ac"} Nov 03 00:59:15.476296 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:15.462205 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-47369d-7c8f86b795-hclwx" Nov 03 00:59:15.509625 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:15.509591 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-db5422-5f7b97b655-4nwn5" Nov 03 00:59:15.511329 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:15.510291 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-47369d-6fbdd5b9b7-h78wk" podStartSLOduration=2.510254802 podStartE2EDuration="2.510254802s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:59:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:59:15.509578251 +0000 UTC m=+1097.680581352" watchObservedRunningTime="2026-11-03 00:59:15.510254802 +0000 UTC m=+1097.681257899" Nov 03 00:59:15.553844 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:15.541994 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-47369d-7c8f86b795-hclwx" podStartSLOduration=2.541973582 podStartE2EDuration="2.541973582s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:59:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:59:15.533204656 +0000 UTC m=+1097.704207755" watchObservedRunningTime="2026-11-03 00:59:15.541973582 +0000 UTC m=+1097.712976689" Nov 03 00:59:15.916436 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:59:15.916397 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode362c3ac_70d7_487a_a181_47d9ffd3dd17.slice/crio-7b67da6bf5019f594c9a99d2c28f284c7d03fc2b33877f2255291873ffddcf89 WatchSource:0}: Error finding container 7b67da6bf5019f594c9a99d2c28f284c7d03fc2b33877f2255291873ffddcf89: Status 404 returned error can't find the container with id 7b67da6bf5019f594c9a99d2c28f284c7d03fc2b33877f2255291873ffddcf89 Nov 03 00:59:15.916881 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:15.916647 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-db5422-5f7b97b655-4nwn5"] Nov 03 00:59:16.480055 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:16.480018 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-db5422-5f7b97b655-4nwn5" event={"ID":"e362c3ac-70d7-487a-a181-47d9ffd3dd17","Type":"ContainerStarted","Data":"86519838f2277e3171f298065620810c0d8537bdcf6beac1a8c3223193212196"} Nov 03 00:59:16.480257 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:16.480087 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-db5422-5f7b97b655-4nwn5" event={"ID":"e362c3ac-70d7-487a-a181-47d9ffd3dd17","Type":"ContainerStarted","Data":"7b67da6bf5019f594c9a99d2c28f284c7d03fc2b33877f2255291873ffddcf89"} Nov 03 00:59:16.480396 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:16.480374 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-db5422-5f7b97b655-4nwn5" Nov 03 00:59:19.172449 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:19.172411 4421 patch_prober.go:29] interesting pod/e2e-ipp-a-7f43f9-openshift-default-74496bd585-lp65l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 00:59:19.172838 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:19.172477 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-7f43f9-openshift-default-74496bd585-lp65l" podUID="bd400742-fae4-44b7-b4af-497f60943da9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 00:59:19.262925 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:19.262892 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7f43f9-79d79dfff5-kpcxh" Nov 03 00:59:19.263090 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:19.262968 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-7f43f9-56467b6944-h8nzs" Nov 03 00:59:19.292368 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:19.292256 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-db5422-5f7b97b655-4nwn5" podStartSLOduration=4.292234964 podStartE2EDuration="4.292234964s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:59:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:59:16.502624028 +0000 UTC m=+1098.673627125" watchObservedRunningTime="2026-11-03 00:59:19.292234964 +0000 UTC m=+1101.463238061" Nov 03 00:59:20.563560 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:20.563528 4421 generic.go:356] "Generic (PLEG): container finished" podID="bd400742-fae4-44b7-b4af-497f60943da9" containerID="94b7c26d9b38f76f76ace804b3d83682116eae5a9a17db4d28bee6ac817f8911" exitCode=0 Nov 03 00:59:20.564024 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:20.563597 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-7f43f9-openshift-default-74496bd585-lp65l" event={"ID":"bd400742-fae4-44b7-b4af-497f60943da9","Type":"ContainerDied","Data":"94b7c26d9b38f76f76ace804b3d83682116eae5a9a17db4d28bee6ac817f8911"} Nov 03 00:59:20.564098 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:20.564040 4421 scope.go:122] "RemoveContainer" containerID="94b7c26d9b38f76f76ace804b3d83682116eae5a9a17db4d28bee6ac817f8911" Nov 03 00:59:20.565681 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:20.565649 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:59:20.565773 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:20.565723 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:59:20.565773 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:20.565758 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:59:21.263507 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:21.263464 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-eed369-openshift-default-66698bcb95-pknkp"] Nov 03 00:59:21.263741 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:21.263711 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-eed369-openshift-default-66698bcb95-pknkp" podUID="b797c4ed-b8df-4e23-9188-3aeb3cf6c48f" containerName="istio-proxy" containerID="cri-o://3efcc349025a233d05bdf085d3b12c572ed416e0da2d31daa76825d31d6c40e6" gracePeriod=30 Nov 03 00:59:21.648834 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:21.640038 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-7f43f9-openshift-default-74496bd585-lp65l" event={"ID":"bd400742-fae4-44b7-b4af-497f60943da9","Type":"ContainerStarted","Data":"bd0f6c1c324c5dc66867d59f2d88c35222b174e4aff852802b90f1693b9cddc2"} Nov 03 00:59:21.741337 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:21.736693 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-7f43f9-openshift-default-74496bd585-lp65l" Nov 03 00:59:21.741337 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:21.736824 4421 patch_prober.go:29] interesting pod/e2e-ipp-a-7f43f9-openshift-default-74496bd585-lp65l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body= Nov 03 00:59:21.741337 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:21.736870 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-7f43f9-openshift-default-74496bd585-lp65l" podUID="bd400742-fae4-44b7-b4af-497f60943da9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" Nov 03 00:59:22.731788 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:22.731736 4421 patch_prober.go:29] interesting pod/e2e-ipp-a-7f43f9-openshift-default-74496bd585-lp65l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body= Nov 03 00:59:22.732361 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:22.731806 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-7f43f9-openshift-default-74496bd585-lp65l" podUID="bd400742-fae4-44b7-b4af-497f60943da9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" Nov 03 00:59:23.157936 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:23.157897 4421 patch_prober.go:29] interesting pod/e2e-api-b-eed369-openshift-default-66698bcb95-pknkp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused Nov 03 00:59:23.158151 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:23.157970 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-eed369-openshift-default-66698bcb95-pknkp" podUID="b797c4ed-b8df-4e23-9188-3aeb3cf6c48f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 00:59:23.498113 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:23.498074 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-f88f87-openshift-default-7b6c9d6885-g92w2"] Nov 03 00:59:23.504496 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:23.504471 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f88f87-openshift-default-7b6c9d6885-g92w2" Nov 03 00:59:23.538581 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:23.538471 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-f88f87-openshift-default-dockercfg-mx2kj\"" Nov 03 00:59:23.544043 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:23.544013 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-f88f87-openshift-default-7b6c9d6885-g92w2"] Nov 03 00:59:23.678363 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:23.675403 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7200ec5-9b07-4003-9652-a92c5a1ab5de-istio-envoy\") pod \"e2e-ipp-a-f88f87-openshift-default-7b6c9d6885-g92w2\" (UID: \"e7200ec5-9b07-4003-9652-a92c5a1ab5de\") " pod="openshift-ingress/e2e-ipp-a-f88f87-openshift-default-7b6c9d6885-g92w2" Nov 03 00:59:23.678363 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:23.675481 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7200ec5-9b07-4003-9652-a92c5a1ab5de-workload-certs\") pod \"e2e-ipp-a-f88f87-openshift-default-7b6c9d6885-g92w2\" (UID: \"e7200ec5-9b07-4003-9652-a92c5a1ab5de\") " pod="openshift-ingress/e2e-ipp-a-f88f87-openshift-default-7b6c9d6885-g92w2" Nov 03 00:59:23.678363 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:23.675545 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7200ec5-9b07-4003-9652-a92c5a1ab5de-istiod-ca-cert\") pod \"e2e-ipp-a-f88f87-openshift-default-7b6c9d6885-g92w2\" (UID: \"e7200ec5-9b07-4003-9652-a92c5a1ab5de\") " pod="openshift-ingress/e2e-ipp-a-f88f87-openshift-default-7b6c9d6885-g92w2" Nov 03 00:59:23.678363 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:23.675657 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7200ec5-9b07-4003-9652-a92c5a1ab5de-istio-data\") pod \"e2e-ipp-a-f88f87-openshift-default-7b6c9d6885-g92w2\" (UID: \"e7200ec5-9b07-4003-9652-a92c5a1ab5de\") " pod="openshift-ingress/e2e-ipp-a-f88f87-openshift-default-7b6c9d6885-g92w2" Nov 03 00:59:23.678363 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:23.675705 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7200ec5-9b07-4003-9652-a92c5a1ab5de-credential-socket\") pod \"e2e-ipp-a-f88f87-openshift-default-7b6c9d6885-g92w2\" (UID: \"e7200ec5-9b07-4003-9652-a92c5a1ab5de\") " pod="openshift-ingress/e2e-ipp-a-f88f87-openshift-default-7b6c9d6885-g92w2" Nov 03 00:59:23.678363 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:23.675734 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7200ec5-9b07-4003-9652-a92c5a1ab5de-istio-token\") pod \"e2e-ipp-a-f88f87-openshift-default-7b6c9d6885-g92w2\" (UID: \"e7200ec5-9b07-4003-9652-a92c5a1ab5de\") " pod="openshift-ingress/e2e-ipp-a-f88f87-openshift-default-7b6c9d6885-g92w2" Nov 03 00:59:23.678363 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:23.675861 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7200ec5-9b07-4003-9652-a92c5a1ab5de-workload-socket\") pod \"e2e-ipp-a-f88f87-openshift-default-7b6c9d6885-g92w2\" (UID: \"e7200ec5-9b07-4003-9652-a92c5a1ab5de\") " pod="openshift-ingress/e2e-ipp-a-f88f87-openshift-default-7b6c9d6885-g92w2" Nov 03 00:59:23.678363 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:23.675983 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7200ec5-9b07-4003-9652-a92c5a1ab5de-istio-podinfo\") pod \"e2e-ipp-a-f88f87-openshift-default-7b6c9d6885-g92w2\" (UID: \"e7200ec5-9b07-4003-9652-a92c5a1ab5de\") " pod="openshift-ingress/e2e-ipp-a-f88f87-openshift-default-7b6c9d6885-g92w2" Nov 03 00:59:23.678363 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:23.676015 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzwj4\" (UniqueName: \"kubernetes.io/projected/e7200ec5-9b07-4003-9652-a92c5a1ab5de-kube-api-access-vzwj4\") pod \"e2e-ipp-a-f88f87-openshift-default-7b6c9d6885-g92w2\" (UID: \"e7200ec5-9b07-4003-9652-a92c5a1ab5de\") " pod="openshift-ingress/e2e-ipp-a-f88f87-openshift-default-7b6c9d6885-g92w2" Nov 03 00:59:23.726728 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:23.726689 4421 patch_prober.go:29] interesting pod/e2e-ipp-a-7f43f9-openshift-default-74496bd585-lp65l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body= Nov 03 00:59:23.727048 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:23.727014 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-7f43f9-openshift-default-74496bd585-lp65l" podUID="bd400742-fae4-44b7-b4af-497f60943da9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" Nov 03 00:59:23.777942 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:23.777845 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7200ec5-9b07-4003-9652-a92c5a1ab5de-istio-envoy\") pod \"e2e-ipp-a-f88f87-openshift-default-7b6c9d6885-g92w2\" (UID: \"e7200ec5-9b07-4003-9652-a92c5a1ab5de\") " pod="openshift-ingress/e2e-ipp-a-f88f87-openshift-default-7b6c9d6885-g92w2" Nov 03 00:59:23.779065 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:23.779038 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7200ec5-9b07-4003-9652-a92c5a1ab5de-workload-certs\") pod \"e2e-ipp-a-f88f87-openshift-default-7b6c9d6885-g92w2\" (UID: \"e7200ec5-9b07-4003-9652-a92c5a1ab5de\") " pod="openshift-ingress/e2e-ipp-a-f88f87-openshift-default-7b6c9d6885-g92w2" Nov 03 00:59:23.779253 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:23.779236 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7200ec5-9b07-4003-9652-a92c5a1ab5de-istiod-ca-cert\") pod \"e2e-ipp-a-f88f87-openshift-default-7b6c9d6885-g92w2\" (UID: \"e7200ec5-9b07-4003-9652-a92c5a1ab5de\") " pod="openshift-ingress/e2e-ipp-a-f88f87-openshift-default-7b6c9d6885-g92w2" Nov 03 00:59:23.779494 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:23.779473 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7200ec5-9b07-4003-9652-a92c5a1ab5de-istio-data\") pod \"e2e-ipp-a-f88f87-openshift-default-7b6c9d6885-g92w2\" (UID: \"e7200ec5-9b07-4003-9652-a92c5a1ab5de\") " pod="openshift-ingress/e2e-ipp-a-f88f87-openshift-default-7b6c9d6885-g92w2" Nov 03 00:59:23.779633 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:23.779619 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7200ec5-9b07-4003-9652-a92c5a1ab5de-credential-socket\") pod \"e2e-ipp-a-f88f87-openshift-default-7b6c9d6885-g92w2\" (UID: \"e7200ec5-9b07-4003-9652-a92c5a1ab5de\") " pod="openshift-ingress/e2e-ipp-a-f88f87-openshift-default-7b6c9d6885-g92w2" Nov 03 00:59:23.779744 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:23.779731 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7200ec5-9b07-4003-9652-a92c5a1ab5de-istio-token\") pod \"e2e-ipp-a-f88f87-openshift-default-7b6c9d6885-g92w2\" (UID: \"e7200ec5-9b07-4003-9652-a92c5a1ab5de\") " pod="openshift-ingress/e2e-ipp-a-f88f87-openshift-default-7b6c9d6885-g92w2" Nov 03 00:59:23.779956 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:23.779942 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7200ec5-9b07-4003-9652-a92c5a1ab5de-workload-socket\") pod \"e2e-ipp-a-f88f87-openshift-default-7b6c9d6885-g92w2\" (UID: \"e7200ec5-9b07-4003-9652-a92c5a1ab5de\") " pod="openshift-ingress/e2e-ipp-a-f88f87-openshift-default-7b6c9d6885-g92w2" Nov 03 00:59:23.799672 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:23.780144 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7200ec5-9b07-4003-9652-a92c5a1ab5de-istio-podinfo\") pod \"e2e-ipp-a-f88f87-openshift-default-7b6c9d6885-g92w2\" (UID: \"e7200ec5-9b07-4003-9652-a92c5a1ab5de\") " pod="openshift-ingress/e2e-ipp-a-f88f87-openshift-default-7b6c9d6885-g92w2" Nov 03 00:59:23.799926 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:23.799897 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzwj4\" (UniqueName: \"kubernetes.io/projected/e7200ec5-9b07-4003-9652-a92c5a1ab5de-kube-api-access-vzwj4\") pod \"e2e-ipp-a-f88f87-openshift-default-7b6c9d6885-g92w2\" (UID: \"e7200ec5-9b07-4003-9652-a92c5a1ab5de\") " pod="openshift-ingress/e2e-ipp-a-f88f87-openshift-default-7b6c9d6885-g92w2" Nov 03 00:59:23.800255 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:23.780925 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7200ec5-9b07-4003-9652-a92c5a1ab5de-workload-certs\") pod \"e2e-ipp-a-f88f87-openshift-default-7b6c9d6885-g92w2\" (UID: \"e7200ec5-9b07-4003-9652-a92c5a1ab5de\") " pod="openshift-ingress/e2e-ipp-a-f88f87-openshift-default-7b6c9d6885-g92w2" Nov 03 00:59:23.800444 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:23.781572 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7200ec5-9b07-4003-9652-a92c5a1ab5de-istiod-ca-cert\") pod \"e2e-ipp-a-f88f87-openshift-default-7b6c9d6885-g92w2\" (UID: \"e7200ec5-9b07-4003-9652-a92c5a1ab5de\") " pod="openshift-ingress/e2e-ipp-a-f88f87-openshift-default-7b6c9d6885-g92w2" Nov 03 00:59:23.800545 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:23.781800 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7200ec5-9b07-4003-9652-a92c5a1ab5de-workload-socket\") pod \"e2e-ipp-a-f88f87-openshift-default-7b6c9d6885-g92w2\" (UID: \"e7200ec5-9b07-4003-9652-a92c5a1ab5de\") " pod="openshift-ingress/e2e-ipp-a-f88f87-openshift-default-7b6c9d6885-g92w2" Nov 03 00:59:23.800635 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:23.782077 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7200ec5-9b07-4003-9652-a92c5a1ab5de-credential-socket\") pod \"e2e-ipp-a-f88f87-openshift-default-7b6c9d6885-g92w2\" (UID: \"e7200ec5-9b07-4003-9652-a92c5a1ab5de\") " pod="openshift-ingress/e2e-ipp-a-f88f87-openshift-default-7b6c9d6885-g92w2" Nov 03 00:59:23.800711 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:23.782123 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7200ec5-9b07-4003-9652-a92c5a1ab5de-istio-envoy\") pod \"e2e-ipp-a-f88f87-openshift-default-7b6c9d6885-g92w2\" (UID: \"e7200ec5-9b07-4003-9652-a92c5a1ab5de\") " pod="openshift-ingress/e2e-ipp-a-f88f87-openshift-default-7b6c9d6885-g92w2" Nov 03 00:59:23.800786 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:23.792319 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7200ec5-9b07-4003-9652-a92c5a1ab5de-istio-podinfo\") pod \"e2e-ipp-a-f88f87-openshift-default-7b6c9d6885-g92w2\" (UID: \"e7200ec5-9b07-4003-9652-a92c5a1ab5de\") " pod="openshift-ingress/e2e-ipp-a-f88f87-openshift-default-7b6c9d6885-g92w2" Nov 03 00:59:23.800856 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:23.780679 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7200ec5-9b07-4003-9652-a92c5a1ab5de-istio-data\") pod \"e2e-ipp-a-f88f87-openshift-default-7b6c9d6885-g92w2\" (UID: \"e7200ec5-9b07-4003-9652-a92c5a1ab5de\") " pod="openshift-ingress/e2e-ipp-a-f88f87-openshift-default-7b6c9d6885-g92w2" Nov 03 00:59:23.828831 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:23.828794 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzwj4\" (UniqueName: \"kubernetes.io/projected/e7200ec5-9b07-4003-9652-a92c5a1ab5de-kube-api-access-vzwj4\") pod \"e2e-ipp-a-f88f87-openshift-default-7b6c9d6885-g92w2\" (UID: \"e7200ec5-9b07-4003-9652-a92c5a1ab5de\") " pod="openshift-ingress/e2e-ipp-a-f88f87-openshift-default-7b6c9d6885-g92w2" Nov 03 00:59:23.834351 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:23.829957 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7200ec5-9b07-4003-9652-a92c5a1ab5de-istio-token\") pod \"e2e-ipp-a-f88f87-openshift-default-7b6c9d6885-g92w2\" (UID: \"e7200ec5-9b07-4003-9652-a92c5a1ab5de\") " pod="openshift-ingress/e2e-ipp-a-f88f87-openshift-default-7b6c9d6885-g92w2" Nov 03 00:59:23.857531 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:23.857500 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f88f87-openshift-default-7b6c9d6885-g92w2" Nov 03 00:59:24.238647 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:24.238612 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-f88f87-openshift-default-7b6c9d6885-g92w2"] Nov 03 00:59:24.240066 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:59:24.240038 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7200ec5_9b07_4003_9652_a92c5a1ab5de.slice/crio-e407daa31fa4f08e39db89fd6e765cd176e74c0f9322d2e5f1d6b821ba18e78a WatchSource:0}: Error finding container e407daa31fa4f08e39db89fd6e765cd176e74c0f9322d2e5f1d6b821ba18e78a: Status 404 returned error can't find the container with id e407daa31fa4f08e39db89fd6e765cd176e74c0f9322d2e5f1d6b821ba18e78a Nov 03 00:59:24.243159 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:24.243117 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:59:24.243262 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:24.243200 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:59:24.243262 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:24.243231 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:59:24.690125 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:24.690090 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f88f87-openshift-default-7b6c9d6885-g92w2" event={"ID":"e7200ec5-9b07-4003-9652-a92c5a1ab5de","Type":"ContainerStarted","Data":"9e94379bd48d6349e7e47ddc3391920faa9dfe975bd5118e072fb0fd1d235b6a"} Nov 03 00:59:24.690321 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:24.690152 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f88f87-openshift-default-7b6c9d6885-g92w2" event={"ID":"e7200ec5-9b07-4003-9652-a92c5a1ab5de","Type":"ContainerStarted","Data":"e407daa31fa4f08e39db89fd6e765cd176e74c0f9322d2e5f1d6b821ba18e78a"} Nov 03 00:59:24.723251 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:24.723215 4421 patch_prober.go:29] interesting pod/e2e-ipp-a-7f43f9-openshift-default-74496bd585-lp65l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body= Nov 03 00:59:24.723535 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:24.723504 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-7f43f9-openshift-default-74496bd585-lp65l" podUID="bd400742-fae4-44b7-b4af-497f60943da9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" Nov 03 00:59:24.725120 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:24.725067 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-f88f87-openshift-default-7b6c9d6885-g92w2" podStartSLOduration=1.725050872 podStartE2EDuration="1.725050872s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:59:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:59:24.722906844 +0000 UTC m=+1106.893909941" watchObservedRunningTime="2026-11-03 00:59:24.725050872 +0000 UTC m=+1106.896053969" Nov 03 00:59:24.859982 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:24.859907 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-f88f87-openshift-default-7b6c9d6885-g92w2" Nov 03 00:59:25.730301 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:25.730028 4421 patch_prober.go:29] interesting pod/e2e-ipp-a-7f43f9-openshift-default-74496bd585-lp65l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body= Nov 03 00:59:25.730301 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:25.730098 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-7f43f9-openshift-default-74496bd585-lp65l" podUID="bd400742-fae4-44b7-b4af-497f60943da9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" Nov 03 00:59:25.959583 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:25.959543 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-f88f87-openshift-default-7b6c9d6885-g92w2" Nov 03 00:59:26.055837 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:26.055797 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2de4fbe5-gpm2r"] Nov 03 00:59:26.060372 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:26.060337 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2de4fbe5-gpm2r"] Nov 03 00:59:26.695358 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:26.695290 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-eed369-openshift-default-66698bcb95-pknkp" Nov 03 00:59:26.739191 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:26.736703 4421 patch_prober.go:29] interesting pod/e2e-ipp-a-7f43f9-openshift-default-74496bd585-lp65l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body= Nov 03 00:59:26.739191 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:26.736764 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-7f43f9-openshift-default-74496bd585-lp65l" podUID="bd400742-fae4-44b7-b4af-497f60943da9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" Nov 03 00:59:26.739771 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:26.739684 4421 generic.go:356] "Generic (PLEG): container finished" podID="b797c4ed-b8df-4e23-9188-3aeb3cf6c48f" containerID="3efcc349025a233d05bdf085d3b12c572ed416e0da2d31daa76825d31d6c40e6" exitCode=0 Nov 03 00:59:26.741670 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:26.741147 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-eed369-openshift-default-66698bcb95-pknkp" Nov 03 00:59:26.741670 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:26.741480 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-eed369-openshift-default-66698bcb95-pknkp" event={"ID":"b797c4ed-b8df-4e23-9188-3aeb3cf6c48f","Type":"ContainerDied","Data":"3efcc349025a233d05bdf085d3b12c572ed416e0da2d31daa76825d31d6c40e6"} Nov 03 00:59:26.741670 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:26.741526 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-f88f87-openshift-default-7b6c9d6885-g92w2" Nov 03 00:59:26.741670 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:26.741546 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-eed369-openshift-default-66698bcb95-pknkp" event={"ID":"b797c4ed-b8df-4e23-9188-3aeb3cf6c48f","Type":"ContainerDied","Data":"689fcb37f9f7b1d3f2a2c6c1cf87b8deded65d77670f30b5e5cf7b25a9f2a644"} Nov 03 00:59:26.741670 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:26.741565 4421 scope.go:122] "RemoveContainer" containerID="3efcc349025a233d05bdf085d3b12c572ed416e0da2d31daa76825d31d6c40e6" Nov 03 00:59:26.742608 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:26.742585 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-f88f87-openshift-default-7b6c9d6885-g92w2" Nov 03 00:59:26.762533 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:26.762503 4421 scope.go:122] "RemoveContainer" containerID="c478a537e618fb8d71c478fa5ca5bf32229158cc77be47271c4f611b1f348977" Nov 03 00:59:26.778981 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:26.778958 4421 scope.go:122] "RemoveContainer" containerID="3efcc349025a233d05bdf085d3b12c572ed416e0da2d31daa76825d31d6c40e6" Nov 03 00:59:26.779340 ip-10-0-133-217 kubenswrapper[4421]: E1103 00:59:26.779308 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3efcc349025a233d05bdf085d3b12c572ed416e0da2d31daa76825d31d6c40e6\": container with ID starting with 3efcc349025a233d05bdf085d3b12c572ed416e0da2d31daa76825d31d6c40e6 not found: ID does not exist" containerID="3efcc349025a233d05bdf085d3b12c572ed416e0da2d31daa76825d31d6c40e6" Nov 03 00:59:26.779455 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:26.779346 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3efcc349025a233d05bdf085d3b12c572ed416e0da2d31daa76825d31d6c40e6"} err="failed to get container status \"3efcc349025a233d05bdf085d3b12c572ed416e0da2d31daa76825d31d6c40e6\": rpc error: code = NotFound desc = could not find container \"3efcc349025a233d05bdf085d3b12c572ed416e0da2d31daa76825d31d6c40e6\": container with ID starting with 3efcc349025a233d05bdf085d3b12c572ed416e0da2d31daa76825d31d6c40e6 not found: ID does not exist" Nov 03 00:59:26.779455 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:26.779363 4421 scope.go:122] "RemoveContainer" containerID="c478a537e618fb8d71c478fa5ca5bf32229158cc77be47271c4f611b1f348977" Nov 03 00:59:26.779700 ip-10-0-133-217 kubenswrapper[4421]: E1103 00:59:26.779673 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c478a537e618fb8d71c478fa5ca5bf32229158cc77be47271c4f611b1f348977\": container with ID starting with c478a537e618fb8d71c478fa5ca5bf32229158cc77be47271c4f611b1f348977 not found: ID does not exist" containerID="c478a537e618fb8d71c478fa5ca5bf32229158cc77be47271c4f611b1f348977" Nov 03 00:59:26.779775 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:26.779710 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c478a537e618fb8d71c478fa5ca5bf32229158cc77be47271c4f611b1f348977"} err="failed to get container status \"c478a537e618fb8d71c478fa5ca5bf32229158cc77be47271c4f611b1f348977\": rpc error: code = NotFound desc = could not find container \"c478a537e618fb8d71c478fa5ca5bf32229158cc77be47271c4f611b1f348977\": container with ID starting with c478a537e618fb8d71c478fa5ca5bf32229158cc77be47271c4f611b1f348977 not found: ID does not exist" Nov 03 00:59:26.839264 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:26.839233 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b797c4ed-b8df-4e23-9188-3aeb3cf6c48f-kube-api-access-d42j4\" (UniqueName: \"kubernetes.io/projected/b797c4ed-b8df-4e23-9188-3aeb3cf6c48f-kube-api-access-d42j4\") pod \"b797c4ed-b8df-4e23-9188-3aeb3cf6c48f\" (UID: \"b797c4ed-b8df-4e23-9188-3aeb3cf6c48f\") " Nov 03 00:59:26.839453 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:26.839335 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b797c4ed-b8df-4e23-9188-3aeb3cf6c48f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b797c4ed-b8df-4e23-9188-3aeb3cf6c48f-credential-socket\") pod \"b797c4ed-b8df-4e23-9188-3aeb3cf6c48f\" (UID: \"b797c4ed-b8df-4e23-9188-3aeb3cf6c48f\") " Nov 03 00:59:26.839453 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:26.839430 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b797c4ed-b8df-4e23-9188-3aeb3cf6c48f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b797c4ed-b8df-4e23-9188-3aeb3cf6c48f-istio-data\") pod \"b797c4ed-b8df-4e23-9188-3aeb3cf6c48f\" (UID: \"b797c4ed-b8df-4e23-9188-3aeb3cf6c48f\") " Nov 03 00:59:26.839543 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:26.839502 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b797c4ed-b8df-4e23-9188-3aeb3cf6c48f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b797c4ed-b8df-4e23-9188-3aeb3cf6c48f-istio-envoy\") pod \"b797c4ed-b8df-4e23-9188-3aeb3cf6c48f\" (UID: \"b797c4ed-b8df-4e23-9188-3aeb3cf6c48f\") " Nov 03 00:59:26.839604 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:26.839567 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b797c4ed-b8df-4e23-9188-3aeb3cf6c48f-istio-token\" (UniqueName: \"kubernetes.io/projected/b797c4ed-b8df-4e23-9188-3aeb3cf6c48f-istio-token\") pod \"b797c4ed-b8df-4e23-9188-3aeb3cf6c48f\" (UID: \"b797c4ed-b8df-4e23-9188-3aeb3cf6c48f\") " Nov 03 00:59:26.839658 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:26.839621 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b797c4ed-b8df-4e23-9188-3aeb3cf6c48f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b797c4ed-b8df-4e23-9188-3aeb3cf6c48f-workload-socket\") pod \"b797c4ed-b8df-4e23-9188-3aeb3cf6c48f\" (UID: \"b797c4ed-b8df-4e23-9188-3aeb3cf6c48f\") " Nov 03 00:59:26.839716 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:26.839658 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b797c4ed-b8df-4e23-9188-3aeb3cf6c48f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b797c4ed-b8df-4e23-9188-3aeb3cf6c48f-workload-certs\") pod \"b797c4ed-b8df-4e23-9188-3aeb3cf6c48f\" (UID: \"b797c4ed-b8df-4e23-9188-3aeb3cf6c48f\") " Nov 03 00:59:26.839770 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:26.839761 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b797c4ed-b8df-4e23-9188-3aeb3cf6c48f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b797c4ed-b8df-4e23-9188-3aeb3cf6c48f-istiod-ca-cert\") pod \"b797c4ed-b8df-4e23-9188-3aeb3cf6c48f\" (UID: \"b797c4ed-b8df-4e23-9188-3aeb3cf6c48f\") " Nov 03 00:59:26.839926 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:26.839905 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b797c4ed-b8df-4e23-9188-3aeb3cf6c48f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b797c4ed-b8df-4e23-9188-3aeb3cf6c48f-istio-podinfo\") pod \"b797c4ed-b8df-4e23-9188-3aeb3cf6c48f\" (UID: \"b797c4ed-b8df-4e23-9188-3aeb3cf6c48f\") " Nov 03 00:59:26.840585 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:26.840452 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b797c4ed-b8df-4e23-9188-3aeb3cf6c48f-credential-socket" pod "b797c4ed-b8df-4e23-9188-3aeb3cf6c48f" (UID: "b797c4ed-b8df-4e23-9188-3aeb3cf6c48f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:59:26.840687 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:26.840405 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b797c4ed-b8df-4e23-9188-3aeb3cf6c48f-workload-certs" pod "b797c4ed-b8df-4e23-9188-3aeb3cf6c48f" (UID: "b797c4ed-b8df-4e23-9188-3aeb3cf6c48f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:59:26.840687 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:26.840656 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b797c4ed-b8df-4e23-9188-3aeb3cf6c48f-istiod-ca-cert" pod "b797c4ed-b8df-4e23-9188-3aeb3cf6c48f" (UID: "b797c4ed-b8df-4e23-9188-3aeb3cf6c48f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:59:26.840809 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:26.840739 4421 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b797c4ed-b8df-4e23-9188-3aeb3cf6c48f-workload-certs\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:59:26.840809 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:26.840759 4421 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b797c4ed-b8df-4e23-9188-3aeb3cf6c48f-credential-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:59:26.840923 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:26.840844 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b797c4ed-b8df-4e23-9188-3aeb3cf6c48f-workload-socket" pod "b797c4ed-b8df-4e23-9188-3aeb3cf6c48f" (UID: "b797c4ed-b8df-4e23-9188-3aeb3cf6c48f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:59:26.841088 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:26.841067 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b797c4ed-b8df-4e23-9188-3aeb3cf6c48f-istio-data" pod "b797c4ed-b8df-4e23-9188-3aeb3cf6c48f" (UID: "b797c4ed-b8df-4e23-9188-3aeb3cf6c48f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:59:26.843314 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:26.843224 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b797c4ed-b8df-4e23-9188-3aeb3cf6c48f-istio-envoy" pod "b797c4ed-b8df-4e23-9188-3aeb3cf6c48f" (UID: "b797c4ed-b8df-4e23-9188-3aeb3cf6c48f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:59:26.843611 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:26.843572 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b797c4ed-b8df-4e23-9188-3aeb3cf6c48f-istio-podinfo" pod "b797c4ed-b8df-4e23-9188-3aeb3cf6c48f" (UID: "b797c4ed-b8df-4e23-9188-3aeb3cf6c48f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:59:26.843701 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:26.843602 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b797c4ed-b8df-4e23-9188-3aeb3cf6c48f-kube-api-access-d42j4" pod "b797c4ed-b8df-4e23-9188-3aeb3cf6c48f" (UID: "b797c4ed-b8df-4e23-9188-3aeb3cf6c48f"). InnerVolumeSpecName "kube-api-access-d42j4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:59:26.844546 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:26.844513 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b797c4ed-b8df-4e23-9188-3aeb3cf6c48f-istio-token" pod "b797c4ed-b8df-4e23-9188-3aeb3cf6c48f" (UID: "b797c4ed-b8df-4e23-9188-3aeb3cf6c48f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:59:26.951351 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:26.943111 4421 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b797c4ed-b8df-4e23-9188-3aeb3cf6c48f-istio-data\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:59:26.951351 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:26.943158 4421 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b797c4ed-b8df-4e23-9188-3aeb3cf6c48f-istio-envoy\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:59:26.951351 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:26.943175 4421 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b797c4ed-b8df-4e23-9188-3aeb3cf6c48f-istio-token\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:59:26.951351 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:26.943193 4421 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b797c4ed-b8df-4e23-9188-3aeb3cf6c48f-workload-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:59:26.951351 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:26.943208 4421 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b797c4ed-b8df-4e23-9188-3aeb3cf6c48f-istiod-ca-cert\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:59:26.951351 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:26.943221 4421 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b797c4ed-b8df-4e23-9188-3aeb3cf6c48f-istio-podinfo\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:59:26.951351 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:26.943234 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d42j4\" (UniqueName: \"kubernetes.io/projected/b797c4ed-b8df-4e23-9188-3aeb3cf6c48f-kube-api-access-d42j4\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:59:27.099752 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:27.099378 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-eed369-openshift-default-66698bcb95-pknkp"] Nov 03 00:59:27.105889 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:27.105857 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-eed369-openshift-default-66698bcb95-pknkp"] Nov 03 00:59:27.536583 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:27.536549 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-47369d-7c8f86b795-hclwx" Nov 03 00:59:27.536789 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:27.536644 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-47369d-6fbdd5b9b7-h78wk" Nov 03 00:59:27.536992 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:27.536970 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-db5422-5f7b97b655-4nwn5" Nov 03 00:59:27.737101 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:27.733430 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-7f43f9-openshift-default-74496bd585-lp65l" Nov 03 00:59:27.737101 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:27.734586 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-7f43f9-openshift-default-74496bd585-lp65l" Nov 03 00:59:27.737101 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:27.736104 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-7f43f9-openshift-default-74496bd585-lp65l" Nov 03 00:59:27.981134 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:27.981055 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b797c4ed-b8df-4e23-9188-3aeb3cf6c48f" path="/var/lib/kubelet/pods/b797c4ed-b8df-4e23-9188-3aeb3cf6c48f/volumes" Nov 03 00:59:27.981600 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:27.981584 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc8f0e43-471d-4339-a877-719e2b5061fa" path="/var/lib/kubelet/pods/cc8f0e43-471d-4339-a877-719e2b5061fa/volumes" Nov 03 00:59:28.042047 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:28.042013 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0533525-shntr"] Nov 03 00:59:28.046552 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:28.046521 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0533525-shntr"] Nov 03 00:59:28.309684 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:28.309645 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-def754-7788f9b985-ppxwb"] Nov 03 00:59:28.310384 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:28.310367 4421 cpu_manager.go:414] "containerMap: removing container" podUID="b797c4ed-b8df-4e23-9188-3aeb3cf6c48f" containerName="istio-proxy" Nov 03 00:59:28.310455 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:28.310385 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b797c4ed-b8df-4e23-9188-3aeb3cf6c48f" containerName="istio-proxy" Nov 03 00:59:28.310455 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:28.310406 4421 cpu_manager.go:414] "containerMap: removing container" podUID="b797c4ed-b8df-4e23-9188-3aeb3cf6c48f" containerName="istio-proxy" Nov 03 00:59:28.310455 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:28.310411 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b797c4ed-b8df-4e23-9188-3aeb3cf6c48f" containerName="istio-proxy" Nov 03 00:59:28.310552 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:28.310548 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="b797c4ed-b8df-4e23-9188-3aeb3cf6c48f" containerName="istio-proxy" Nov 03 00:59:28.310583 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:28.310558 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="b797c4ed-b8df-4e23-9188-3aeb3cf6c48f" containerName="istio-proxy" Nov 03 00:59:28.315922 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:28.315900 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-def754-7788f9b985-ppxwb" Nov 03 00:59:28.348325 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:28.344460 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-def754\"" Nov 03 00:59:28.348325 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:28.344542 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-def754-dockercfg-c66w7\"" Nov 03 00:59:28.349031 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:28.349004 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-def754-7788f9b985-ppxwb"] Nov 03 00:59:28.484451 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:28.484412 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4vhw\" (UniqueName: \"kubernetes.io/projected/1423f3de-68b7-430d-9c0d-9e957468fedf-kube-api-access-n4vhw\") pod \"payload-processing-e2e-ipp-b-def754-7788f9b985-ppxwb\" (UID: \"1423f3de-68b7-430d-9c0d-9e957468fedf\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-def754-7788f9b985-ppxwb" Nov 03 00:59:28.484646 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:28.484562 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1423f3de-68b7-430d-9c0d-9e957468fedf-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-def754-7788f9b985-ppxwb\" (UID: \"1423f3de-68b7-430d-9c0d-9e957468fedf\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-def754-7788f9b985-ppxwb" Nov 03 00:59:28.586378 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:28.586256 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1423f3de-68b7-430d-9c0d-9e957468fedf-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-def754-7788f9b985-ppxwb\" (UID: \"1423f3de-68b7-430d-9c0d-9e957468fedf\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-def754-7788f9b985-ppxwb" Nov 03 00:59:28.586637 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:28.586606 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4vhw\" (UniqueName: \"kubernetes.io/projected/1423f3de-68b7-430d-9c0d-9e957468fedf-kube-api-access-n4vhw\") pod \"payload-processing-e2e-ipp-b-def754-7788f9b985-ppxwb\" (UID: \"1423f3de-68b7-430d-9c0d-9e957468fedf\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-def754-7788f9b985-ppxwb" Nov 03 00:59:28.594981 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:28.587346 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1423f3de-68b7-430d-9c0d-9e957468fedf-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-def754-7788f9b985-ppxwb\" (UID: \"1423f3de-68b7-430d-9c0d-9e957468fedf\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-def754-7788f9b985-ppxwb" Nov 03 00:59:28.624304 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:28.623148 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4vhw\" (UniqueName: \"kubernetes.io/projected/1423f3de-68b7-430d-9c0d-9e957468fedf-kube-api-access-n4vhw\") pod \"payload-processing-e2e-ipp-b-def754-7788f9b985-ppxwb\" (UID: \"1423f3de-68b7-430d-9c0d-9e957468fedf\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-def754-7788f9b985-ppxwb" Nov 03 00:59:28.641874 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:28.640711 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-def754-7788f9b985-ppxwb" Nov 03 00:59:28.995763 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:59:28.995736 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1423f3de_68b7_430d_9c0d_9e957468fedf.slice/crio-0e429ec9c2e5dd850068139f396cee411546b029018e45758c0eb921103e4174 WatchSource:0}: Error finding container 0e429ec9c2e5dd850068139f396cee411546b029018e45758c0eb921103e4174: Status 404 returned error can't find the container with id 0e429ec9c2e5dd850068139f396cee411546b029018e45758c0eb921103e4174 Nov 03 00:59:28.997959 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:28.997930 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-def754-7788f9b985-ppxwb"] Nov 03 00:59:29.779243 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:29.779203 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-def754-7788f9b985-ppxwb" event={"ID":"1423f3de-68b7-430d-9c0d-9e957468fedf","Type":"ContainerStarted","Data":"58704160dca57f419fa862ccb1f0c33d5258b5a501d1c7c49d9805293921f482"} Nov 03 00:59:29.779686 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:29.779266 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-def754-7788f9b985-ppxwb" event={"ID":"1423f3de-68b7-430d-9c0d-9e957468fedf","Type":"ContainerStarted","Data":"0e429ec9c2e5dd850068139f396cee411546b029018e45758c0eb921103e4174"} Nov 03 00:59:29.779686 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:29.779326 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-def754-7788f9b985-ppxwb" Nov 03 00:59:29.820342 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:29.817430 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-def754-7788f9b985-ppxwb" podStartSLOduration=1.8174066199999999 podStartE2EDuration="1.81740662s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:59:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:59:29.812950879 +0000 UTC m=+1111.983953977" watchObservedRunningTime="2026-11-03 00:59:29.81740662 +0000 UTC m=+1111.988409723" Nov 03 00:59:29.980133 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:29.980101 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c364576-1756-471d-9f70-01e1d90ae1a0" path="/var/lib/kubelet/pods/2c364576-1756-471d-9f70-01e1d90ae1a0/volumes" Nov 03 00:59:30.035391 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:30.035303 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f88f87-5d58f64779-pwc7n"] Nov 03 00:59:30.041527 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:30.041499 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f88f87-5d58f64779-pwc7n" Nov 03 00:59:30.052207 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:30.051952 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f88f87-5d58f64779-pwc7n"] Nov 03 00:59:30.122100 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:30.122068 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-f88f87\"" Nov 03 00:59:30.122565 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:30.122545 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-f88f87-dockercfg-clk5s\"" Nov 03 00:59:30.231614 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:30.231010 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x995d\" (UniqueName: \"kubernetes.io/projected/c0fb5e39-8b3d-4bc4-8791-01042c8ab304-kube-api-access-x995d\") pod \"payload-pre-processing-e2e-ipp-a-f88f87-5d58f64779-pwc7n\" (UID: \"c0fb5e39-8b3d-4bc4-8791-01042c8ab304\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f88f87-5d58f64779-pwc7n" Nov 03 00:59:30.231614 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:30.231132 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0fb5e39-8b3d-4bc4-8791-01042c8ab304-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f88f87-5d58f64779-pwc7n\" (UID: \"c0fb5e39-8b3d-4bc4-8791-01042c8ab304\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f88f87-5d58f64779-pwc7n" Nov 03 00:59:30.334521 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:30.334420 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x995d\" (UniqueName: \"kubernetes.io/projected/c0fb5e39-8b3d-4bc4-8791-01042c8ab304-kube-api-access-x995d\") pod \"payload-pre-processing-e2e-ipp-a-f88f87-5d58f64779-pwc7n\" (UID: \"c0fb5e39-8b3d-4bc4-8791-01042c8ab304\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f88f87-5d58f64779-pwc7n" Nov 03 00:59:30.334713 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:30.334545 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0fb5e39-8b3d-4bc4-8791-01042c8ab304-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f88f87-5d58f64779-pwc7n\" (UID: \"c0fb5e39-8b3d-4bc4-8791-01042c8ab304\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f88f87-5d58f64779-pwc7n" Nov 03 00:59:30.335587 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:30.335562 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0fb5e39-8b3d-4bc4-8791-01042c8ab304-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f88f87-5d58f64779-pwc7n\" (UID: \"c0fb5e39-8b3d-4bc4-8791-01042c8ab304\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f88f87-5d58f64779-pwc7n" Nov 03 00:59:30.344602 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:30.344557 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x995d\" (UniqueName: \"kubernetes.io/projected/c0fb5e39-8b3d-4bc4-8791-01042c8ab304-kube-api-access-x995d\") pod \"payload-pre-processing-e2e-ipp-a-f88f87-5d58f64779-pwc7n\" (UID: \"c0fb5e39-8b3d-4bc4-8791-01042c8ab304\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f88f87-5d58f64779-pwc7n" Nov 03 00:59:30.388419 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:30.388378 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f88f87-5d58f64779-pwc7n" Nov 03 00:59:30.799724 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:59:30.799690 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0fb5e39_8b3d_4bc4_8791_01042c8ab304.slice/crio-33e304068221cce15efc794f98f46956ece2934bde7b646eaa4c7e7de165b62c WatchSource:0}: Error finding container 33e304068221cce15efc794f98f46956ece2934bde7b646eaa4c7e7de165b62c: Status 404 returned error can't find the container with id 33e304068221cce15efc794f98f46956ece2934bde7b646eaa4c7e7de165b62c Nov 03 00:59:30.801313 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:30.801268 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f88f87-5d58f64779-pwc7n"] Nov 03 00:59:31.030907 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:31.030865 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc90ad3d-8drjr"] Nov 03 00:59:31.035468 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:31.035433 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc90ad3d-8drjr"] Nov 03 00:59:31.798261 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:31.798208 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f88f87-5d58f64779-pwc7n" event={"ID":"c0fb5e39-8b3d-4bc4-8791-01042c8ab304","Type":"ContainerStarted","Data":"463b91c090caf30164eb2588e844de5b5b602eb8002b658dd515fb59bb78492c"} Nov 03 00:59:31.798637 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:31.798323 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f88f87-5d58f64779-pwc7n" Nov 03 00:59:31.798637 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:31.798353 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f88f87-5d58f64779-pwc7n" event={"ID":"c0fb5e39-8b3d-4bc4-8791-01042c8ab304","Type":"ContainerStarted","Data":"33e304068221cce15efc794f98f46956ece2934bde7b646eaa4c7e7de165b62c"} Nov 03 00:59:31.819901 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:31.819844 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f88f87-5d58f64779-pwc7n" podStartSLOduration=1.819830308 podStartE2EDuration="1.819830308s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:59:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:59:31.817569587 +0000 UTC m=+1113.988572683" watchObservedRunningTime="2026-11-03 00:59:31.819830308 +0000 UTC m=+1113.990833432" Nov 03 00:59:31.981258 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:31.981206 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c50f5d52-bbc4-4fd8-b894-8af10e0454bd" path="/var/lib/kubelet/pods/c50f5d52-bbc4-4fd8-b894-8af10e0454bd/volumes" Nov 03 00:59:38.030126 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:38.030088 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-259597d3-lcqhj"] Nov 03 00:59:38.035217 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:38.035190 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c09f2e44-xqjtm"] Nov 03 00:59:38.039185 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:38.039160 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-259597d3-lcqhj"] Nov 03 00:59:38.042866 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:38.042846 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c09f2e44-xqjtm"] Nov 03 00:59:39.982535 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:39.982502 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7197ee03-efec-4b33-be93-2853619c6215" path="/var/lib/kubelet/pods/7197ee03-efec-4b33-be93-2853619c6215/volumes" Nov 03 00:59:39.983078 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:39.983062 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9dcc5fdc-d37a-47f7-8b6f-2359a8a377bd" path="/var/lib/kubelet/pods/9dcc5fdc-d37a-47f7-8b6f-2359a8a377bd/volumes" Nov 03 00:59:40.124772 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:40.124698 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-ffdea6-openshift-default-85977f6bff-jw7rs"] Nov 03 00:59:40.145859 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:40.145821 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-ffdea6-openshift-default-85977f6bff-jw7rs" Nov 03 00:59:40.153373 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:40.153329 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-ffdea6-openshift-default-85977f6bff-jw7rs"] Nov 03 00:59:40.153901 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:40.153876 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-ffdea6-openshift-default-dockercfg-2224w\"" Nov 03 00:59:40.250842 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:40.250800 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c95ca370-7e69-4c57-912c-e953469cb21f-workload-certs\") pod \"e2e-api-b-ffdea6-openshift-default-85977f6bff-jw7rs\" (UID: \"c95ca370-7e69-4c57-912c-e953469cb21f\") " pod="openshift-ingress/e2e-api-b-ffdea6-openshift-default-85977f6bff-jw7rs" Nov 03 00:59:40.250842 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:40.250838 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c95ca370-7e69-4c57-912c-e953469cb21f-istio-podinfo\") pod \"e2e-api-b-ffdea6-openshift-default-85977f6bff-jw7rs\" (UID: \"c95ca370-7e69-4c57-912c-e953469cb21f\") " pod="openshift-ingress/e2e-api-b-ffdea6-openshift-default-85977f6bff-jw7rs" Nov 03 00:59:40.251131 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:40.250928 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcv4w\" (UniqueName: \"kubernetes.io/projected/c95ca370-7e69-4c57-912c-e953469cb21f-kube-api-access-fcv4w\") pod \"e2e-api-b-ffdea6-openshift-default-85977f6bff-jw7rs\" (UID: \"c95ca370-7e69-4c57-912c-e953469cb21f\") " pod="openshift-ingress/e2e-api-b-ffdea6-openshift-default-85977f6bff-jw7rs" Nov 03 00:59:40.251131 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:40.250972 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c95ca370-7e69-4c57-912c-e953469cb21f-workload-socket\") pod \"e2e-api-b-ffdea6-openshift-default-85977f6bff-jw7rs\" (UID: \"c95ca370-7e69-4c57-912c-e953469cb21f\") " pod="openshift-ingress/e2e-api-b-ffdea6-openshift-default-85977f6bff-jw7rs" Nov 03 00:59:40.251131 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:40.251079 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c95ca370-7e69-4c57-912c-e953469cb21f-istiod-ca-cert\") pod \"e2e-api-b-ffdea6-openshift-default-85977f6bff-jw7rs\" (UID: \"c95ca370-7e69-4c57-912c-e953469cb21f\") " pod="openshift-ingress/e2e-api-b-ffdea6-openshift-default-85977f6bff-jw7rs" Nov 03 00:59:40.251131 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:40.251127 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c95ca370-7e69-4c57-912c-e953469cb21f-istio-envoy\") pod \"e2e-api-b-ffdea6-openshift-default-85977f6bff-jw7rs\" (UID: \"c95ca370-7e69-4c57-912c-e953469cb21f\") " pod="openshift-ingress/e2e-api-b-ffdea6-openshift-default-85977f6bff-jw7rs" Nov 03 00:59:40.251424 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:40.251232 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c95ca370-7e69-4c57-912c-e953469cb21f-credential-socket\") pod \"e2e-api-b-ffdea6-openshift-default-85977f6bff-jw7rs\" (UID: \"c95ca370-7e69-4c57-912c-e953469cb21f\") " pod="openshift-ingress/e2e-api-b-ffdea6-openshift-default-85977f6bff-jw7rs" Nov 03 00:59:40.251424 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:40.251267 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c95ca370-7e69-4c57-912c-e953469cb21f-istio-data\") pod \"e2e-api-b-ffdea6-openshift-default-85977f6bff-jw7rs\" (UID: \"c95ca370-7e69-4c57-912c-e953469cb21f\") " pod="openshift-ingress/e2e-api-b-ffdea6-openshift-default-85977f6bff-jw7rs" Nov 03 00:59:40.251424 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:40.251347 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c95ca370-7e69-4c57-912c-e953469cb21f-istio-token\") pod \"e2e-api-b-ffdea6-openshift-default-85977f6bff-jw7rs\" (UID: \"c95ca370-7e69-4c57-912c-e953469cb21f\") " pod="openshift-ingress/e2e-api-b-ffdea6-openshift-default-85977f6bff-jw7rs" Nov 03 00:59:40.385299 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:40.377852 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c95ca370-7e69-4c57-912c-e953469cb21f-workload-certs\") pod \"e2e-api-b-ffdea6-openshift-default-85977f6bff-jw7rs\" (UID: \"c95ca370-7e69-4c57-912c-e953469cb21f\") " pod="openshift-ingress/e2e-api-b-ffdea6-openshift-default-85977f6bff-jw7rs" Nov 03 00:59:40.385299 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:40.377932 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c95ca370-7e69-4c57-912c-e953469cb21f-istio-podinfo\") pod \"e2e-api-b-ffdea6-openshift-default-85977f6bff-jw7rs\" (UID: \"c95ca370-7e69-4c57-912c-e953469cb21f\") " pod="openshift-ingress/e2e-api-b-ffdea6-openshift-default-85977f6bff-jw7rs" Nov 03 00:59:40.385299 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:40.377993 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcv4w\" (UniqueName: \"kubernetes.io/projected/c95ca370-7e69-4c57-912c-e953469cb21f-kube-api-access-fcv4w\") pod \"e2e-api-b-ffdea6-openshift-default-85977f6bff-jw7rs\" (UID: \"c95ca370-7e69-4c57-912c-e953469cb21f\") " pod="openshift-ingress/e2e-api-b-ffdea6-openshift-default-85977f6bff-jw7rs" Nov 03 00:59:40.385299 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:40.378037 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c95ca370-7e69-4c57-912c-e953469cb21f-workload-socket\") pod \"e2e-api-b-ffdea6-openshift-default-85977f6bff-jw7rs\" (UID: \"c95ca370-7e69-4c57-912c-e953469cb21f\") " pod="openshift-ingress/e2e-api-b-ffdea6-openshift-default-85977f6bff-jw7rs" Nov 03 00:59:40.385299 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:40.378121 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c95ca370-7e69-4c57-912c-e953469cb21f-istiod-ca-cert\") pod \"e2e-api-b-ffdea6-openshift-default-85977f6bff-jw7rs\" (UID: \"c95ca370-7e69-4c57-912c-e953469cb21f\") " pod="openshift-ingress/e2e-api-b-ffdea6-openshift-default-85977f6bff-jw7rs" Nov 03 00:59:40.385299 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:40.378181 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c95ca370-7e69-4c57-912c-e953469cb21f-istio-envoy\") pod \"e2e-api-b-ffdea6-openshift-default-85977f6bff-jw7rs\" (UID: \"c95ca370-7e69-4c57-912c-e953469cb21f\") " pod="openshift-ingress/e2e-api-b-ffdea6-openshift-default-85977f6bff-jw7rs" Nov 03 00:59:40.385299 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:40.378442 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c95ca370-7e69-4c57-912c-e953469cb21f-credential-socket\") pod \"e2e-api-b-ffdea6-openshift-default-85977f6bff-jw7rs\" (UID: \"c95ca370-7e69-4c57-912c-e953469cb21f\") " pod="openshift-ingress/e2e-api-b-ffdea6-openshift-default-85977f6bff-jw7rs" Nov 03 00:59:40.385299 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:40.378522 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c95ca370-7e69-4c57-912c-e953469cb21f-istio-data\") pod \"e2e-api-b-ffdea6-openshift-default-85977f6bff-jw7rs\" (UID: \"c95ca370-7e69-4c57-912c-e953469cb21f\") " pod="openshift-ingress/e2e-api-b-ffdea6-openshift-default-85977f6bff-jw7rs" Nov 03 00:59:40.385299 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:40.378551 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c95ca370-7e69-4c57-912c-e953469cb21f-istio-token\") pod \"e2e-api-b-ffdea6-openshift-default-85977f6bff-jw7rs\" (UID: \"c95ca370-7e69-4c57-912c-e953469cb21f\") " pod="openshift-ingress/e2e-api-b-ffdea6-openshift-default-85977f6bff-jw7rs" Nov 03 00:59:40.385299 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:40.378972 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c95ca370-7e69-4c57-912c-e953469cb21f-workload-socket\") pod \"e2e-api-b-ffdea6-openshift-default-85977f6bff-jw7rs\" (UID: \"c95ca370-7e69-4c57-912c-e953469cb21f\") " pod="openshift-ingress/e2e-api-b-ffdea6-openshift-default-85977f6bff-jw7rs" Nov 03 00:59:40.385299 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:40.379219 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c95ca370-7e69-4c57-912c-e953469cb21f-workload-certs\") pod \"e2e-api-b-ffdea6-openshift-default-85977f6bff-jw7rs\" (UID: \"c95ca370-7e69-4c57-912c-e953469cb21f\") " pod="openshift-ingress/e2e-api-b-ffdea6-openshift-default-85977f6bff-jw7rs" Nov 03 00:59:40.385299 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:40.380338 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c95ca370-7e69-4c57-912c-e953469cb21f-credential-socket\") pod \"e2e-api-b-ffdea6-openshift-default-85977f6bff-jw7rs\" (UID: \"c95ca370-7e69-4c57-912c-e953469cb21f\") " pod="openshift-ingress/e2e-api-b-ffdea6-openshift-default-85977f6bff-jw7rs" Nov 03 00:59:40.385299 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:40.380615 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c95ca370-7e69-4c57-912c-e953469cb21f-istio-data\") pod \"e2e-api-b-ffdea6-openshift-default-85977f6bff-jw7rs\" (UID: \"c95ca370-7e69-4c57-912c-e953469cb21f\") " pod="openshift-ingress/e2e-api-b-ffdea6-openshift-default-85977f6bff-jw7rs" Nov 03 00:59:40.405689 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:40.405650 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c95ca370-7e69-4c57-912c-e953469cb21f-istio-envoy\") pod \"e2e-api-b-ffdea6-openshift-default-85977f6bff-jw7rs\" (UID: \"c95ca370-7e69-4c57-912c-e953469cb21f\") " pod="openshift-ingress/e2e-api-b-ffdea6-openshift-default-85977f6bff-jw7rs" Nov 03 00:59:40.430303 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:40.424639 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcv4w\" (UniqueName: \"kubernetes.io/projected/c95ca370-7e69-4c57-912c-e953469cb21f-kube-api-access-fcv4w\") pod \"e2e-api-b-ffdea6-openshift-default-85977f6bff-jw7rs\" (UID: \"c95ca370-7e69-4c57-912c-e953469cb21f\") " pod="openshift-ingress/e2e-api-b-ffdea6-openshift-default-85977f6bff-jw7rs" Nov 03 00:59:40.451299 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:40.430557 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c95ca370-7e69-4c57-912c-e953469cb21f-istiod-ca-cert\") pod \"e2e-api-b-ffdea6-openshift-default-85977f6bff-jw7rs\" (UID: \"c95ca370-7e69-4c57-912c-e953469cb21f\") " pod="openshift-ingress/e2e-api-b-ffdea6-openshift-default-85977f6bff-jw7rs" Nov 03 00:59:40.460300 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:40.454301 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c95ca370-7e69-4c57-912c-e953469cb21f-istio-token\") pod \"e2e-api-b-ffdea6-openshift-default-85977f6bff-jw7rs\" (UID: \"c95ca370-7e69-4c57-912c-e953469cb21f\") " pod="openshift-ingress/e2e-api-b-ffdea6-openshift-default-85977f6bff-jw7rs" Nov 03 00:59:40.460300 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:40.454926 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c95ca370-7e69-4c57-912c-e953469cb21f-istio-podinfo\") pod \"e2e-api-b-ffdea6-openshift-default-85977f6bff-jw7rs\" (UID: \"c95ca370-7e69-4c57-912c-e953469cb21f\") " pod="openshift-ingress/e2e-api-b-ffdea6-openshift-default-85977f6bff-jw7rs" Nov 03 00:59:40.472297 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:40.467991 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-ffdea6-openshift-default-85977f6bff-jw7rs" Nov 03 00:59:40.792018 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:40.791932 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-def754-7788f9b985-ppxwb" Nov 03 00:59:40.906768 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:59:40.906735 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc95ca370_7e69_4c57_912c_e953469cb21f.slice/crio-35b05066d3652409fb297ccf114610302f9fa0f7816a3393be30f7a163ac22b3 WatchSource:0}: Error finding container 35b05066d3652409fb297ccf114610302f9fa0f7816a3393be30f7a163ac22b3: Status 404 returned error can't find the container with id 35b05066d3652409fb297ccf114610302f9fa0f7816a3393be30f7a163ac22b3 Nov 03 00:59:40.907353 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:40.907330 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-ffdea6-openshift-default-85977f6bff-jw7rs"] Nov 03 00:59:40.910492 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:40.910445 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:59:40.910593 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:40.910532 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:59:40.910593 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:40.910562 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:59:41.020571 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:41.020534 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ffdea6-openshift-default-85977f6bff-jw7rs" event={"ID":"c95ca370-7e69-4c57-912c-e953469cb21f","Type":"ContainerStarted","Data":"35b05066d3652409fb297ccf114610302f9fa0f7816a3393be30f7a163ac22b3"} Nov 03 00:59:42.032595 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:42.032558 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ffdea6-openshift-default-85977f6bff-jw7rs" event={"ID":"c95ca370-7e69-4c57-912c-e953469cb21f","Type":"ContainerStarted","Data":"b070fb3205bb6c1922e6d768b4dffd8cfca9496e661b97284d1307009e4c1bbf"} Nov 03 00:59:42.063927 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:42.063855 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-ffdea6-openshift-default-85977f6bff-jw7rs" podStartSLOduration=2.063835779 podStartE2EDuration="2.063835779s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:59:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:59:42.056715916 +0000 UTC m=+1124.227719012" watchObservedRunningTime="2026-11-03 00:59:42.063835779 +0000 UTC m=+1124.234838878" Nov 03 00:59:42.469356 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:42.469258 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-ffdea6-openshift-default-85977f6bff-jw7rs" Nov 03 00:59:42.475411 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:42.475385 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-ffdea6-openshift-default-85977f6bff-jw7rs" Nov 03 00:59:42.810852 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:42.810823 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f88f87-5d58f64779-pwc7n" Nov 03 00:59:43.040459 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:43.040431 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-ffdea6-openshift-default-85977f6bff-jw7rs" Nov 03 00:59:43.041533 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:43.041515 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-ffdea6-openshift-default-85977f6bff-jw7rs" Nov 03 00:59:44.673203 ip-10-0-133-217 kubenswrapper[4421]: E1103 00:59:44.673163 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 00:59:45.667024 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:45.666980 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-ffdea6-67f595b459-k49vb"] Nov 03 00:59:45.672004 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:45.671979 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ffdea6-67f595b459-k49vb" Nov 03 00:59:45.685154 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:45.685125 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-ffdea6-dockercfg-dpfcx\"" Nov 03 00:59:45.685648 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:45.685297 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-ffdea6\"" Nov 03 00:59:45.689892 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:45.689865 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-ffdea6-67f595b459-k49vb"] Nov 03 00:59:45.804719 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:45.804680 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc932a32-c09c-4792-a90c-c4f906034e9f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-ffdea6-67f595b459-k49vb\" (UID: \"dc932a32-c09c-4792-a90c-c4f906034e9f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-ffdea6-67f595b459-k49vb" Nov 03 00:59:45.805569 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:45.805534 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldzc2\" (UniqueName: \"kubernetes.io/projected/dc932a32-c09c-4792-a90c-c4f906034e9f-kube-api-access-ldzc2\") pod \"payload-pre-processing-e2e-api-b-ffdea6-67f595b459-k49vb\" (UID: \"dc932a32-c09c-4792-a90c-c4f906034e9f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-ffdea6-67f595b459-k49vb" Nov 03 00:59:45.907332 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:45.907290 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldzc2\" (UniqueName: \"kubernetes.io/projected/dc932a32-c09c-4792-a90c-c4f906034e9f-kube-api-access-ldzc2\") pod \"payload-pre-processing-e2e-api-b-ffdea6-67f595b459-k49vb\" (UID: \"dc932a32-c09c-4792-a90c-c4f906034e9f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-ffdea6-67f595b459-k49vb" Nov 03 00:59:45.907909 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:45.907880 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc932a32-c09c-4792-a90c-c4f906034e9f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-ffdea6-67f595b459-k49vb\" (UID: \"dc932a32-c09c-4792-a90c-c4f906034e9f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-ffdea6-67f595b459-k49vb" Nov 03 00:59:45.909525 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:45.909480 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc932a32-c09c-4792-a90c-c4f906034e9f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-ffdea6-67f595b459-k49vb\" (UID: \"dc932a32-c09c-4792-a90c-c4f906034e9f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-ffdea6-67f595b459-k49vb" Nov 03 00:59:45.933785 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:45.933709 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldzc2\" (UniqueName: \"kubernetes.io/projected/dc932a32-c09c-4792-a90c-c4f906034e9f-kube-api-access-ldzc2\") pod \"payload-pre-processing-e2e-api-b-ffdea6-67f595b459-k49vb\" (UID: \"dc932a32-c09c-4792-a90c-c4f906034e9f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-ffdea6-67f595b459-k49vb" Nov 03 00:59:46.001132 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:46.001101 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ffdea6-67f595b459-k49vb" Nov 03 00:59:46.311195 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:46.311161 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-ffdea6-67f595b459-k49vb"] Nov 03 00:59:46.311610 ip-10-0-133-217 kubenswrapper[4421]: W1103 00:59:46.311589 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc932a32_c09c_4792_a90c_c4f906034e9f.slice/crio-0b0638261804f2f8e0d52d12bf2d63da0195a73272e40c985a8dd57f28e38bc1 WatchSource:0}: Error finding container 0b0638261804f2f8e0d52d12bf2d63da0195a73272e40c985a8dd57f28e38bc1: Status 404 returned error can't find the container with id 0b0638261804f2f8e0d52d12bf2d63da0195a73272e40c985a8dd57f28e38bc1 Nov 03 00:59:47.100198 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:47.100161 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ffdea6-67f595b459-k49vb" event={"ID":"dc932a32-c09c-4792-a90c-c4f906034e9f","Type":"ContainerStarted","Data":"251ffbe73b31080419d31d3f13c1aa7eac8413d8298fdbbc471f9e11568d7990"} Nov 03 00:59:47.100754 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:47.100218 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ffdea6-67f595b459-k49vb" event={"ID":"dc932a32-c09c-4792-a90c-c4f906034e9f","Type":"ContainerStarted","Data":"0b0638261804f2f8e0d52d12bf2d63da0195a73272e40c985a8dd57f28e38bc1"} Nov 03 00:59:47.100754 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:47.100305 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ffdea6-67f595b459-k49vb" Nov 03 00:59:47.143072 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:47.143004 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ffdea6-67f595b459-k49vb" podStartSLOduration=2.142983832 podStartE2EDuration="2.142983832s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:59:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:59:47.11989584 +0000 UTC m=+1129.290898937" watchObservedRunningTime="2026-11-03 00:59:47.142983832 +0000 UTC m=+1129.313986929" Nov 03 00:59:47.823881 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:47.823841 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-80f70e-bfc75b495-7qxcs"] Nov 03 00:59:47.843012 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:47.842971 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-80f70e-fb5785db7-l2v4s"] Nov 03 00:59:48.318976 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:48.318942 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-db5422-5f7b97b655-4nwn5"] Nov 03 00:59:49.027170 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:49.027137 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b529cf1-z7d5k"] Nov 03 00:59:49.031293 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:49.031253 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b529cf1-z7d5k"] Nov 03 00:59:49.980642 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:49.980611 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="848ee430-f2ec-41a6-aeb4-ac2bd369980f" path="/var/lib/kubelet/pods/848ee430-f2ec-41a6-aeb4-ac2bd369980f/volumes" Nov 03 00:59:52.824830 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:52.824759 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-80f70e-bfc75b495-7qxcs" podUID="d2320dab-9330-4dfd-b88c-eb5e952e8de2" containerName="payload-processing" containerID="cri-o://41056dea8b065e76af98ba0eaa8fbf9566db00dd9c033963c206957692879944" gracePeriod=25 Nov 03 00:59:52.843848 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:52.843785 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-80f70e-fb5785db7-l2v4s" podUID="04b33c64-e869-43fa-81a6-57d5dacf927f" containerName="payload-pre-processing" containerID="cri-o://8bf86768a33132a6bd43c06780f1b0d6ce416f7e4f40e0c83df8df8f54373eed" gracePeriod=25 Nov 03 00:59:53.152450 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:53.152399 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-80f70e-bfc75b495-7qxcs" Nov 03 00:59:53.159607 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:53.159571 4421 generic.go:356] "Generic (PLEG): container finished" podID="04b33c64-e869-43fa-81a6-57d5dacf927f" containerID="8bf86768a33132a6bd43c06780f1b0d6ce416f7e4f40e0c83df8df8f54373eed" exitCode=0 Nov 03 00:59:53.159751 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:53.159649 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-80f70e-fb5785db7-l2v4s" event={"ID":"04b33c64-e869-43fa-81a6-57d5dacf927f","Type":"ContainerDied","Data":"8bf86768a33132a6bd43c06780f1b0d6ce416f7e4f40e0c83df8df8f54373eed"} Nov 03 00:59:53.161265 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:53.161241 4421 generic.go:356] "Generic (PLEG): container finished" podID="d2320dab-9330-4dfd-b88c-eb5e952e8de2" containerID="41056dea8b065e76af98ba0eaa8fbf9566db00dd9c033963c206957692879944" exitCode=0 Nov 03 00:59:53.161412 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:53.161295 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-80f70e-bfc75b495-7qxcs" Nov 03 00:59:53.161412 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:53.161293 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-80f70e-bfc75b495-7qxcs" event={"ID":"d2320dab-9330-4dfd-b88c-eb5e952e8de2","Type":"ContainerDied","Data":"41056dea8b065e76af98ba0eaa8fbf9566db00dd9c033963c206957692879944"} Nov 03 00:59:53.161412 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:53.161408 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-80f70e-bfc75b495-7qxcs" event={"ID":"d2320dab-9330-4dfd-b88c-eb5e952e8de2","Type":"ContainerDied","Data":"cc964ba9457ada112cf6ef2b31f5df94f18dd2a556d71bbc03299858fc5b4907"} Nov 03 00:59:53.161520 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:53.161432 4421 scope.go:122] "RemoveContainer" containerID="41056dea8b065e76af98ba0eaa8fbf9566db00dd9c033963c206957692879944" Nov 03 00:59:53.174659 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:53.174641 4421 scope.go:122] "RemoveContainer" containerID="41056dea8b065e76af98ba0eaa8fbf9566db00dd9c033963c206957692879944" Nov 03 00:59:53.175134 ip-10-0-133-217 kubenswrapper[4421]: E1103 00:59:53.175082 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41056dea8b065e76af98ba0eaa8fbf9566db00dd9c033963c206957692879944\": container with ID starting with 41056dea8b065e76af98ba0eaa8fbf9566db00dd9c033963c206957692879944 not found: ID does not exist" containerID="41056dea8b065e76af98ba0eaa8fbf9566db00dd9c033963c206957692879944" Nov 03 00:59:53.175134 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:53.175125 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41056dea8b065e76af98ba0eaa8fbf9566db00dd9c033963c206957692879944"} err="failed to get container status \"41056dea8b065e76af98ba0eaa8fbf9566db00dd9c033963c206957692879944\": rpc error: code = NotFound desc = could not find container \"41056dea8b065e76af98ba0eaa8fbf9566db00dd9c033963c206957692879944\": container with ID starting with 41056dea8b065e76af98ba0eaa8fbf9566db00dd9c033963c206957692879944 not found: ID does not exist" Nov 03 00:59:53.205076 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:53.205047 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-80f70e-fb5785db7-l2v4s" Nov 03 00:59:53.246543 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:53.246504 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2320dab-9330-4dfd-b88c-eb5e952e8de2-kube-api-access-6fs65\" (UniqueName: \"kubernetes.io/projected/d2320dab-9330-4dfd-b88c-eb5e952e8de2-kube-api-access-6fs65\") pod \"d2320dab-9330-4dfd-b88c-eb5e952e8de2\" (UID: \"d2320dab-9330-4dfd-b88c-eb5e952e8de2\") " Nov 03 00:59:53.247100 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:53.246748 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2320dab-9330-4dfd-b88c-eb5e952e8de2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2320dab-9330-4dfd-b88c-eb5e952e8de2-plugins-config-volume\") pod \"d2320dab-9330-4dfd-b88c-eb5e952e8de2\" (UID: \"d2320dab-9330-4dfd-b88c-eb5e952e8de2\") " Nov 03 00:59:53.247326 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:53.247255 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2320dab-9330-4dfd-b88c-eb5e952e8de2-plugins-config-volume" pod "d2320dab-9330-4dfd-b88c-eb5e952e8de2" (UID: "d2320dab-9330-4dfd-b88c-eb5e952e8de2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:59:53.247519 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:53.247493 4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2320dab-9330-4dfd-b88c-eb5e952e8de2-plugins-config-volume\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:59:53.249771 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:53.249739 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2320dab-9330-4dfd-b88c-eb5e952e8de2-kube-api-access-6fs65" pod "d2320dab-9330-4dfd-b88c-eb5e952e8de2" (UID: "d2320dab-9330-4dfd-b88c-eb5e952e8de2"). InnerVolumeSpecName "kube-api-access-6fs65". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:59:53.320070 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:53.320024 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-db5422-5f7b97b655-4nwn5" podUID="e362c3ac-70d7-487a-a181-47d9ffd3dd17" containerName="payload-pre-processing" containerID="cri-o://86519838f2277e3171f298065620810c0d8537bdcf6beac1a8c3223193212196" gracePeriod=25 Nov 03 00:59:53.348537 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:53.348505 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04b33c64-e869-43fa-81a6-57d5dacf927f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04b33c64-e869-43fa-81a6-57d5dacf927f-plugins-config-volume\") pod \"04b33c64-e869-43fa-81a6-57d5dacf927f\" (UID: \"04b33c64-e869-43fa-81a6-57d5dacf927f\") " Nov 03 00:59:53.348663 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:53.348632 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04b33c64-e869-43fa-81a6-57d5dacf927f-kube-api-access-lbqdd\" (UniqueName: \"kubernetes.io/projected/04b33c64-e869-43fa-81a6-57d5dacf927f-kube-api-access-lbqdd\") pod \"04b33c64-e869-43fa-81a6-57d5dacf927f\" (UID: \"04b33c64-e869-43fa-81a6-57d5dacf927f\") " Nov 03 00:59:53.348823 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:53.348802 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04b33c64-e869-43fa-81a6-57d5dacf927f-plugins-config-volume" pod "04b33c64-e869-43fa-81a6-57d5dacf927f" (UID: "04b33c64-e869-43fa-81a6-57d5dacf927f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:59:53.349245 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:53.349230 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6fs65\" (UniqueName: \"kubernetes.io/projected/d2320dab-9330-4dfd-b88c-eb5e952e8de2-kube-api-access-6fs65\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:59:53.349301 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:53.349248 4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04b33c64-e869-43fa-81a6-57d5dacf927f-plugins-config-volume\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:59:53.350838 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:53.350821 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04b33c64-e869-43fa-81a6-57d5dacf927f-kube-api-access-lbqdd" pod "04b33c64-e869-43fa-81a6-57d5dacf927f" (UID: "04b33c64-e869-43fa-81a6-57d5dacf927f"). InnerVolumeSpecName "kube-api-access-lbqdd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:59:53.451174 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:53.451144 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbqdd\" (UniqueName: \"kubernetes.io/projected/04b33c64-e869-43fa-81a6-57d5dacf927f-kube-api-access-lbqdd\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:59:53.491547 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:53.491345 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-80f70e-bfc75b495-7qxcs"] Nov 03 00:59:53.495863 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:53.495832 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-80f70e-bfc75b495-7qxcs"] Nov 03 00:59:53.645671 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:53.645641 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-db5422-5f7b97b655-4nwn5" Nov 03 00:59:53.756818 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:53.756787 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e362c3ac-70d7-487a-a181-47d9ffd3dd17-kube-api-access-ldgnc\" (UniqueName: \"kubernetes.io/projected/e362c3ac-70d7-487a-a181-47d9ffd3dd17-kube-api-access-ldgnc\") pod \"e362c3ac-70d7-487a-a181-47d9ffd3dd17\" (UID: \"e362c3ac-70d7-487a-a181-47d9ffd3dd17\") " Nov 03 00:59:53.756989 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:53.756928 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e362c3ac-70d7-487a-a181-47d9ffd3dd17-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e362c3ac-70d7-487a-a181-47d9ffd3dd17-plugins-config-volume\") pod \"e362c3ac-70d7-487a-a181-47d9ffd3dd17\" (UID: \"e362c3ac-70d7-487a-a181-47d9ffd3dd17\") " Nov 03 00:59:53.757347 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:53.757318 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e362c3ac-70d7-487a-a181-47d9ffd3dd17-plugins-config-volume" pod "e362c3ac-70d7-487a-a181-47d9ffd3dd17" (UID: "e362c3ac-70d7-487a-a181-47d9ffd3dd17"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:59:53.757592 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:53.757571 4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e362c3ac-70d7-487a-a181-47d9ffd3dd17-plugins-config-volume\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:59:53.758957 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:53.758935 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e362c3ac-70d7-487a-a181-47d9ffd3dd17-kube-api-access-ldgnc" pod "e362c3ac-70d7-487a-a181-47d9ffd3dd17" (UID: "e362c3ac-70d7-487a-a181-47d9ffd3dd17"). InnerVolumeSpecName "kube-api-access-ldgnc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:59:53.859228 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:53.859199 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ldgnc\" (UniqueName: \"kubernetes.io/projected/e362c3ac-70d7-487a-a181-47d9ffd3dd17-kube-api-access-ldgnc\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:59:53.980417 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:53.980384 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2320dab-9330-4dfd-b88c-eb5e952e8de2" path="/var/lib/kubelet/pods/d2320dab-9330-4dfd-b88c-eb5e952e8de2/volumes" Nov 03 00:59:54.171954 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:54.171861 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-80f70e-fb5785db7-l2v4s" Nov 03 00:59:54.171954 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:54.171898 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-80f70e-fb5785db7-l2v4s" event={"ID":"04b33c64-e869-43fa-81a6-57d5dacf927f","Type":"ContainerDied","Data":"168ebacb29ed1550b1d75b270e18ae4eff860690660e808949a3239a2787a02b"} Nov 03 00:59:54.172255 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:54.171963 4421 scope.go:122] "RemoveContainer" containerID="8bf86768a33132a6bd43c06780f1b0d6ce416f7e4f40e0c83df8df8f54373eed" Nov 03 00:59:54.173947 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:54.173919 4421 generic.go:356] "Generic (PLEG): container finished" podID="e362c3ac-70d7-487a-a181-47d9ffd3dd17" containerID="86519838f2277e3171f298065620810c0d8537bdcf6beac1a8c3223193212196" exitCode=0 Nov 03 00:59:54.174111 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:54.173956 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-db5422-5f7b97b655-4nwn5" Nov 03 00:59:54.174111 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:54.173988 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-db5422-5f7b97b655-4nwn5" event={"ID":"e362c3ac-70d7-487a-a181-47d9ffd3dd17","Type":"ContainerDied","Data":"86519838f2277e3171f298065620810c0d8537bdcf6beac1a8c3223193212196"} Nov 03 00:59:54.174111 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:54.174034 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-db5422-5f7b97b655-4nwn5" event={"ID":"e362c3ac-70d7-487a-a181-47d9ffd3dd17","Type":"ContainerDied","Data":"7b67da6bf5019f594c9a99d2c28f284c7d03fc2b33877f2255291873ffddcf89"} Nov 03 00:59:54.188441 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:54.188414 4421 scope.go:122] "RemoveContainer" containerID="86519838f2277e3171f298065620810c0d8537bdcf6beac1a8c3223193212196" Nov 03 00:59:54.198079 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:54.198049 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-80f70e-fb5785db7-l2v4s"] Nov 03 00:59:54.202171 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:54.202150 4421 scope.go:122] "RemoveContainer" containerID="86519838f2277e3171f298065620810c0d8537bdcf6beac1a8c3223193212196" Nov 03 00:59:54.202626 ip-10-0-133-217 kubenswrapper[4421]: E1103 00:59:54.202596 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86519838f2277e3171f298065620810c0d8537bdcf6beac1a8c3223193212196\": container with ID starting with 86519838f2277e3171f298065620810c0d8537bdcf6beac1a8c3223193212196 not found: ID does not exist" containerID="86519838f2277e3171f298065620810c0d8537bdcf6beac1a8c3223193212196" Nov 03 00:59:54.202710 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:54.202670 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86519838f2277e3171f298065620810c0d8537bdcf6beac1a8c3223193212196"} err="failed to get container status \"86519838f2277e3171f298065620810c0d8537bdcf6beac1a8c3223193212196\": rpc error: code = NotFound desc = could not find container \"86519838f2277e3171f298065620810c0d8537bdcf6beac1a8c3223193212196\": container with ID starting with 86519838f2277e3171f298065620810c0d8537bdcf6beac1a8c3223193212196 not found: ID does not exist" Nov 03 00:59:54.204867 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:54.204839 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-80f70e-fb5785db7-l2v4s"] Nov 03 00:59:54.215749 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:54.215724 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-db5422-5f7b97b655-4nwn5"] Nov 03 00:59:54.219869 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:54.219845 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-db5422-5f7b97b655-4nwn5"] Nov 03 00:59:55.022891 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:55.022858 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-937c6484-5zwbf"] Nov 03 00:59:55.027511 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:55.027474 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-937c6484-5zwbf"] Nov 03 00:59:55.984753 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:55.984714 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04b33c64-e869-43fa-81a6-57d5dacf927f" path="/var/lib/kubelet/pods/04b33c64-e869-43fa-81a6-57d5dacf927f/volumes" Nov 03 00:59:55.985697 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:55.985668 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8e3f769-362b-4f1a-993f-2cc73903fa25" path="/var/lib/kubelet/pods/a8e3f769-362b-4f1a-993f-2cc73903fa25/volumes" Nov 03 00:59:55.986843 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:55.986377 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e362c3ac-70d7-487a-a181-47d9ffd3dd17" path="/var/lib/kubelet/pods/e362c3ac-70d7-487a-a181-47d9ffd3dd17/volumes" Nov 03 00:59:56.792290 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:56.792243 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-def754-7788f9b985-ppxwb"] Nov 03 00:59:57.028385 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:57.028348 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1759d292-np6cp"] Nov 03 00:59:57.037461 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:57.037431 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1759d292-np6cp"] Nov 03 00:59:57.563958 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:57.563913 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f88f87-5d58f64779-pwc7n"] Nov 03 00:59:58.270228 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:58.270187 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00ef641b-db94-41d3-8c7e-684daa5d7d67" path="/var/lib/kubelet/pods/00ef641b-db94-41d3-8c7e-684daa5d7d67/volumes" Nov 03 00:59:58.271484 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:58.271464 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ffdea6-67f595b459-k49vb" Nov 03 00:59:58.869322 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:58.869250 4421 scope.go:122] "RemoveContainer" containerID="8903fcef0796c3ca9930ea8d9c034f25c65954b7895d92ff8d0389816970dea4" Nov 03 00:59:58.898675 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:58.898653 4421 scope.go:122] "RemoveContainer" containerID="e39af9db817d96030b05d6c25d7129eb1e8fae324f6bf23be3ee03f9f4b6172c" Nov 03 00:59:58.950368 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:58.950339 4421 scope.go:122] "RemoveContainer" containerID="bebc01dab7ff808c002663bc543eda3dde680929dc6661539aa1bc642cdc9a4f" Nov 03 00:59:58.964264 ip-10-0-133-217 kubenswrapper[4421]: I1103 00:59:58.964207 4421 scope.go:122] "RemoveContainer" containerID="fc5cce0c7d248ef8b3db78c5df2bf13d5126acc1cb7cb8a4c3c727c03fbc2d03" Nov 03 01:00:00.167353 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:00.166531 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7f43f9-29894460-bfjlr"] Nov 03 01:00:00.167771 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:00.167653 4421 cpu_manager.go:414] "containerMap: removing container" podUID="d2320dab-9330-4dfd-b88c-eb5e952e8de2" containerName="payload-processing" Nov 03 01:00:00.167771 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:00.167676 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2320dab-9330-4dfd-b88c-eb5e952e8de2" containerName="payload-processing" Nov 03 01:00:00.167771 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:00.167705 4421 cpu_manager.go:414] "containerMap: removing container" podUID="04b33c64-e869-43fa-81a6-57d5dacf927f" containerName="payload-pre-processing" Nov 03 01:00:00.167771 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:00.167713 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04b33c64-e869-43fa-81a6-57d5dacf927f" containerName="payload-pre-processing" Nov 03 01:00:00.167939 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:00.167795 4421 cpu_manager.go:414] "containerMap: removing container" podUID="e362c3ac-70d7-487a-a181-47d9ffd3dd17" containerName="payload-pre-processing" Nov 03 01:00:00.167939 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:00.167804 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e362c3ac-70d7-487a-a181-47d9ffd3dd17" containerName="payload-pre-processing" Nov 03 01:00:00.168025 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:00.167993 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2320dab-9330-4dfd-b88c-eb5e952e8de2" containerName="payload-processing" Nov 03 01:00:00.168068 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:00.168008 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="04b33c64-e869-43fa-81a6-57d5dacf927f" containerName="payload-pre-processing" Nov 03 01:00:00.168068 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:00.168048 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="e362c3ac-70d7-487a-a181-47d9ffd3dd17" containerName="payload-pre-processing" Nov 03 01:00:00.180580 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:00.180546 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f9a3a-29894460-sj5kk"] Nov 03 01:00:00.181445 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:00.181420 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7f43f9-29894460-bfjlr" Nov 03 01:00:00.186840 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:00.186651 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f9a3a-29894460-sj5kk" Nov 03 01:00:00.187072 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:00.187046 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894460-btsdn"] Nov 03 01:00:00.195830 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:00.194355 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6q4xp\"" Nov 03 01:00:00.252654 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:00.237692 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 01:00:00.252654 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:00.237831 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 01:00:00.288302 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:00.286439 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f7680d-29894460-qpglw"] Nov 03 01:00:00.288302 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:00.286547 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894460-btsdn" Nov 03 01:00:00.301591 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:00.301563 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-83755a-29894460-7z6xz"] Nov 03 01:00:00.365964 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:00.365499 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f7680d-29894460-qpglw" Nov 03 01:00:00.368776 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:00.368330 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/84fca73a-b1e2-4fee-892f-b2f162bd75b3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-2f9a3a-29894460-sj5kk\" (UID: \"84fca73a-b1e2-4fee-892f-b2f162bd75b3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f9a3a-29894460-sj5kk" Nov 03 01:00:00.368776 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:00.368390 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qd9s\" (UniqueName: \"kubernetes.io/projected/84fca73a-b1e2-4fee-892f-b2f162bd75b3-kube-api-access-9qd9s\") pod \"maas-api-key-cleanup-e2e-worker-w0-2f9a3a-29894460-sj5kk\" (UID: \"84fca73a-b1e2-4fee-892f-b2f162bd75b3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f9a3a-29894460-sj5kk" Nov 03 01:00:00.368776 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:00.368677 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64ad274e-6de7-4604-a8e4-958f67dbd00f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-7f43f9-29894460-bfjlr\" (UID: \"64ad274e-6de7-4604-a8e4-958f67dbd00f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7f43f9-29894460-bfjlr" Nov 03 01:00:00.369081 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:00.368788 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5mxt\" (UniqueName: \"kubernetes.io/projected/64ad274e-6de7-4604-a8e4-958f67dbd00f-kube-api-access-n5mxt\") pod \"maas-api-key-cleanup-e2e-ipp-a-7f43f9-29894460-bfjlr\" (UID: \"64ad274e-6de7-4604-a8e4-958f67dbd00f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7f43f9-29894460-bfjlr" Nov 03 01:00:00.394830 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:00.394709 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7f43f9-29894460-bfjlr"] Nov 03 01:00:00.394830 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:00.394812 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f9a3a-29894460-sj5kk"] Nov 03 01:00:00.394992 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:00.394846 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894460-btsdn"] Nov 03 01:00:00.394992 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:00.394902 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f7680d-29894460-qpglw"] Nov 03 01:00:00.394992 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:00.394941 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-83755a-29894460-7z6xz"] Nov 03 01:00:00.394992 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:00.394971 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-7406a9-29894460-vnwfw"] Nov 03 01:00:00.395692 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:00.395668 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-83755a-29894460-7z6xz" Nov 03 01:00:00.418645 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:00.418555 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-80f70e-29894460-kh466"] Nov 03 01:00:00.455555 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:00.443470 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-7406a9-29894460-vnwfw" Nov 03 01:00:00.472264 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:00.470122 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/600dc1bf-8abc-480d-8503-3942ecd9d48b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-f7680d-29894460-qpglw\" (UID: \"600dc1bf-8abc-480d-8503-3942ecd9d48b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f7680d-29894460-qpglw" Nov 03 01:00:00.472264 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:00.470212 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k42d5\" (UniqueName: \"kubernetes.io/projected/569546b1-7662-4df6-a849-23fb4095797b-kube-api-access-k42d5\") pod \"maas-api-key-cleanup-29894460-btsdn\" (UID: \"569546b1-7662-4df6-a849-23fb4095797b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894460-btsdn" Nov 03 01:00:00.472264 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:00.470399 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/569546b1-7662-4df6-a849-23fb4095797b-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894460-btsdn\" (UID: \"569546b1-7662-4df6-a849-23fb4095797b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894460-btsdn" Nov 03 01:00:00.472264 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:00.470451 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64ad274e-6de7-4604-a8e4-958f67dbd00f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-7f43f9-29894460-bfjlr\" (UID: \"64ad274e-6de7-4604-a8e4-958f67dbd00f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7f43f9-29894460-bfjlr" Nov 03 01:00:00.472264 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:00.470560 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5mxt\" (UniqueName: \"kubernetes.io/projected/64ad274e-6de7-4604-a8e4-958f67dbd00f-kube-api-access-n5mxt\") pod \"maas-api-key-cleanup-e2e-ipp-a-7f43f9-29894460-bfjlr\" (UID: \"64ad274e-6de7-4604-a8e4-958f67dbd00f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7f43f9-29894460-bfjlr" Nov 03 01:00:00.472264 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:00.470639 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hff44\" (UniqueName: \"kubernetes.io/projected/600dc1bf-8abc-480d-8503-3942ecd9d48b-kube-api-access-hff44\") pod \"maas-api-key-cleanup-e2e-worker-w2-f7680d-29894460-qpglw\" (UID: \"600dc1bf-8abc-480d-8503-3942ecd9d48b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f7680d-29894460-qpglw" Nov 03 01:00:00.472264 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:00.470675 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/84fca73a-b1e2-4fee-892f-b2f162bd75b3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-2f9a3a-29894460-sj5kk\" (UID: \"84fca73a-b1e2-4fee-892f-b2f162bd75b3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f9a3a-29894460-sj5kk" Nov 03 01:00:00.472264 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:00.470702 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qd9s\" (UniqueName: \"kubernetes.io/projected/84fca73a-b1e2-4fee-892f-b2f162bd75b3-kube-api-access-9qd9s\") pod \"maas-api-key-cleanup-e2e-worker-w0-2f9a3a-29894460-sj5kk\" (UID: \"84fca73a-b1e2-4fee-892f-b2f162bd75b3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f9a3a-29894460-sj5kk" Nov 03 01:00:00.472264 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:00.471961 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/84fca73a-b1e2-4fee-892f-b2f162bd75b3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-2f9a3a-29894460-sj5kk\" (UID: \"84fca73a-b1e2-4fee-892f-b2f162bd75b3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f9a3a-29894460-sj5kk" Nov 03 01:00:00.472264 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:00.472025 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64ad274e-6de7-4604-a8e4-958f67dbd00f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-7f43f9-29894460-bfjlr\" (UID: \"64ad274e-6de7-4604-a8e4-958f67dbd00f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7f43f9-29894460-bfjlr" Nov 03 01:00:00.474561 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:00.474543 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-db5422-29894460-w4ksr"] Nov 03 01:00:00.487916 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:00.474830 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-80f70e-29894460-kh466" Nov 03 01:00:00.488603 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:00.488570 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5mxt\" (UniqueName: \"kubernetes.io/projected/64ad274e-6de7-4604-a8e4-958f67dbd00f-kube-api-access-n5mxt\") pod \"maas-api-key-cleanup-e2e-ipp-a-7f43f9-29894460-bfjlr\" (UID: \"64ad274e-6de7-4604-a8e4-958f67dbd00f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7f43f9-29894460-bfjlr" Nov 03 01:00:00.489622 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:00.489542 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qd9s\" (UniqueName: \"kubernetes.io/projected/84fca73a-b1e2-4fee-892f-b2f162bd75b3-kube-api-access-9qd9s\") pod \"maas-api-key-cleanup-e2e-worker-w0-2f9a3a-29894460-sj5kk\" (UID: \"84fca73a-b1e2-4fee-892f-b2f162bd75b3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f9a3a-29894460-sj5kk" Nov 03 01:00:00.498661 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:00.498623 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-7406a9-29894460-vnwfw"] Nov 03 01:00:00.498845 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:00.498677 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-80f70e-29894460-kh466"] Nov 03 01:00:00.498845 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:00.498707 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-db5422-29894460-w4ksr"] Nov 03 01:00:00.498845 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:00.498732 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-db5422-29894460-w4ksr" Nov 03 01:00:00.499029 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:00.498734 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-07f5f6-29894460-5lb9h"] Nov 03 01:00:00.520323 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:00.518504 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f88f87-29894460-p26bw"] Nov 03 01:00:00.582374 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:00.577858 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91abc456-0b19-4c43-9a94-57ef27f96c4a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-7406a9-29894460-vnwfw\" (UID: \"91abc456-0b19-4c43-9a94-57ef27f96c4a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-7406a9-29894460-vnwfw" Nov 03 01:00:00.582374 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:00.577928 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hff44\" (UniqueName: \"kubernetes.io/projected/600dc1bf-8abc-480d-8503-3942ecd9d48b-kube-api-access-hff44\") pod \"maas-api-key-cleanup-e2e-worker-w2-f7680d-29894460-qpglw\" (UID: \"600dc1bf-8abc-480d-8503-3942ecd9d48b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f7680d-29894460-qpglw" Nov 03 01:00:00.582374 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:00.578022 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/600dc1bf-8abc-480d-8503-3942ecd9d48b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-f7680d-29894460-qpglw\" (UID: \"600dc1bf-8abc-480d-8503-3942ecd9d48b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f7680d-29894460-qpglw" Nov 03 01:00:00.582374 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:00.578065 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0370c2db-8545-4266-a4f4-719aebbd925b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-83755a-29894460-7z6xz\" (UID: \"0370c2db-8545-4266-a4f4-719aebbd925b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-83755a-29894460-7z6xz" Nov 03 01:00:00.582374 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:00.578117 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k42d5\" (UniqueName: \"kubernetes.io/projected/569546b1-7662-4df6-a849-23fb4095797b-kube-api-access-k42d5\") pod \"maas-api-key-cleanup-29894460-btsdn\" (UID: \"569546b1-7662-4df6-a849-23fb4095797b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894460-btsdn" Nov 03 01:00:00.582374 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:00.578149 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtfh7\" (UniqueName: \"kubernetes.io/projected/7d930dca-a318-4fb3-9a22-2a0dff4de5b4-kube-api-access-vtfh7\") pod \"maas-api-key-cleanup-e2e-api-a-80f70e-29894460-kh466\" (UID: \"7d930dca-a318-4fb3-9a22-2a0dff4de5b4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-80f70e-29894460-kh466" Nov 03 01:00:00.582374 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:00.578734 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/569546b1-7662-4df6-a849-23fb4095797b-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894460-btsdn\" (UID: \"569546b1-7662-4df6-a849-23fb4095797b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894460-btsdn" Nov 03 01:00:00.582374 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:00.578812 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d930dca-a318-4fb3-9a22-2a0dff4de5b4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-80f70e-29894460-kh466\" (UID: \"7d930dca-a318-4fb3-9a22-2a0dff4de5b4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-80f70e-29894460-kh466" Nov 03 01:00:00.582374 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:00.578876 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbg45\" (UniqueName: \"kubernetes.io/projected/0370c2db-8545-4266-a4f4-719aebbd925b-kube-api-access-gbg45\") pod \"maas-api-key-cleanup-e2e-worker-w3-83755a-29894460-7z6xz\" (UID: \"0370c2db-8545-4266-a4f4-719aebbd925b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-83755a-29894460-7z6xz" Nov 03 01:00:00.582374 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:00.578918 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qssk2\" (UniqueName: \"kubernetes.io/projected/91abc456-0b19-4c43-9a94-57ef27f96c4a-kube-api-access-qssk2\") pod \"maas-api-key-cleanup-e2e-api-a-7406a9-29894460-vnwfw\" (UID: \"91abc456-0b19-4c43-9a94-57ef27f96c4a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-7406a9-29894460-vnwfw" Nov 03 01:00:00.582374 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:00.579203 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/600dc1bf-8abc-480d-8503-3942ecd9d48b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-f7680d-29894460-qpglw\" (UID: \"600dc1bf-8abc-480d-8503-3942ecd9d48b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f7680d-29894460-qpglw" Nov 03 01:00:00.582374 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:00.580016 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/569546b1-7662-4df6-a849-23fb4095797b-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894460-btsdn\" (UID: \"569546b1-7662-4df6-a849-23fb4095797b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894460-btsdn" Nov 03 01:00:00.582374 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:00.580365 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-07f5f6-29894460-5lb9h" Nov 03 01:00:00.585237 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:00.585211 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7f43f9-29894460-bfjlr" Nov 03 01:00:00.587194 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:00.587172 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-07f5f6-29894460-5lb9h"] Nov 03 01:00:00.587397 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:00.587222 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f88f87-29894460-p26bw"] Nov 03 01:00:00.587397 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:00.587309 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f88f87-29894460-p26bw" Nov 03 01:00:00.604326 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:00.603587 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k42d5\" (UniqueName: \"kubernetes.io/projected/569546b1-7662-4df6-a849-23fb4095797b-kube-api-access-k42d5\") pod \"maas-api-key-cleanup-29894460-btsdn\" (UID: \"569546b1-7662-4df6-a849-23fb4095797b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894460-btsdn" Nov 03 01:00:00.614362 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:00.607365 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hff44\" (UniqueName: \"kubernetes.io/projected/600dc1bf-8abc-480d-8503-3942ecd9d48b-kube-api-access-hff44\") pod \"maas-api-key-cleanup-e2e-worker-w2-f7680d-29894460-qpglw\" (UID: \"600dc1bf-8abc-480d-8503-3942ecd9d48b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f7680d-29894460-qpglw" Nov 03 01:00:00.679403 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:00.678814 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f9a3a-29894460-sj5kk" Nov 03 01:00:00.697087 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:00.695920 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894460-btsdn" Nov 03 01:00:00.699825 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:00.699792 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9bb1557e-955c-41d1-8513-e1f8c4f341f3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-07f5f6-29894460-5lb9h\" (UID: \"9bb1557e-955c-41d1-8513-e1f8c4f341f3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-07f5f6-29894460-5lb9h" Nov 03 01:00:00.699825 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:00.699826 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgckj\" (UniqueName: \"kubernetes.io/projected/c8c543e5-5eda-4993-92af-8da6a226cfa6-kube-api-access-cgckj\") pod \"maas-api-key-cleanup-e2e-ipp-b-db5422-29894460-w4ksr\" (UID: \"c8c543e5-5eda-4993-92af-8da6a226cfa6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-db5422-29894460-w4ksr" Nov 03 01:00:00.700196 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:00.700163 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d930dca-a318-4fb3-9a22-2a0dff4de5b4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-80f70e-29894460-kh466\" (UID: \"7d930dca-a318-4fb3-9a22-2a0dff4de5b4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-80f70e-29894460-kh466" Nov 03 01:00:00.705038 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:00.703987 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbg45\" (UniqueName: \"kubernetes.io/projected/0370c2db-8545-4266-a4f4-719aebbd925b-kube-api-access-gbg45\") pod \"maas-api-key-cleanup-e2e-worker-w3-83755a-29894460-7z6xz\" (UID: \"0370c2db-8545-4266-a4f4-719aebbd925b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-83755a-29894460-7z6xz" Nov 03 01:00:00.705038 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:00.704034 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qssk2\" (UniqueName: \"kubernetes.io/projected/91abc456-0b19-4c43-9a94-57ef27f96c4a-kube-api-access-qssk2\") pod \"maas-api-key-cleanup-e2e-api-a-7406a9-29894460-vnwfw\" (UID: \"91abc456-0b19-4c43-9a94-57ef27f96c4a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-7406a9-29894460-vnwfw" Nov 03 01:00:00.705038 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:00.704092 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxlwn\" (UniqueName: \"kubernetes.io/projected/9bb1557e-955c-41d1-8513-e1f8c4f341f3-kube-api-access-nxlwn\") pod \"maas-api-key-cleanup-e2e-ipp-a-07f5f6-29894460-5lb9h\" (UID: \"9bb1557e-955c-41d1-8513-e1f8c4f341f3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-07f5f6-29894460-5lb9h" Nov 03 01:00:00.705038 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:00.704126 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8c543e5-5eda-4993-92af-8da6a226cfa6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-db5422-29894460-w4ksr\" (UID: \"c8c543e5-5eda-4993-92af-8da6a226cfa6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-db5422-29894460-w4ksr" Nov 03 01:00:00.705038 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:00.704198 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91abc456-0b19-4c43-9a94-57ef27f96c4a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-7406a9-29894460-vnwfw\" (UID: \"91abc456-0b19-4c43-9a94-57ef27f96c4a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-7406a9-29894460-vnwfw" Nov 03 01:00:00.705038 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:00.704927 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0370c2db-8545-4266-a4f4-719aebbd925b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-83755a-29894460-7z6xz\" (UID: \"0370c2db-8545-4266-a4f4-719aebbd925b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-83755a-29894460-7z6xz" Nov 03 01:00:00.728420 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:00.705658 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0370c2db-8545-4266-a4f4-719aebbd925b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-83755a-29894460-7z6xz\" (UID: \"0370c2db-8545-4266-a4f4-719aebbd925b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-83755a-29894460-7z6xz" Nov 03 01:00:00.728420 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:00.709982 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91abc456-0b19-4c43-9a94-57ef27f96c4a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-7406a9-29894460-vnwfw\" (UID: \"91abc456-0b19-4c43-9a94-57ef27f96c4a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-7406a9-29894460-vnwfw" Nov 03 01:00:00.757947 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:00.755417 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f7680d-29894460-qpglw" Nov 03 01:00:00.757947 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:00.755828 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtfh7\" (UniqueName: \"kubernetes.io/projected/7d930dca-a318-4fb3-9a22-2a0dff4de5b4-kube-api-access-vtfh7\") pod \"maas-api-key-cleanup-e2e-api-a-80f70e-29894460-kh466\" (UID: \"7d930dca-a318-4fb3-9a22-2a0dff4de5b4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-80f70e-29894460-kh466" Nov 03 01:00:00.885316 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:00.885254 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9bb1557e-955c-41d1-8513-e1f8c4f341f3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-07f5f6-29894460-5lb9h\" (UID: \"9bb1557e-955c-41d1-8513-e1f8c4f341f3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-07f5f6-29894460-5lb9h" Nov 03 01:00:00.885500 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:00.885329 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgckj\" (UniqueName: \"kubernetes.io/projected/c8c543e5-5eda-4993-92af-8da6a226cfa6-kube-api-access-cgckj\") pod \"maas-api-key-cleanup-e2e-ipp-b-db5422-29894460-w4ksr\" (UID: \"c8c543e5-5eda-4993-92af-8da6a226cfa6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-db5422-29894460-w4ksr" Nov 03 01:00:00.885500 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:00.885443 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxlwn\" (UniqueName: \"kubernetes.io/projected/9bb1557e-955c-41d1-8513-e1f8c4f341f3-kube-api-access-nxlwn\") pod \"maas-api-key-cleanup-e2e-ipp-a-07f5f6-29894460-5lb9h\" (UID: \"9bb1557e-955c-41d1-8513-e1f8c4f341f3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-07f5f6-29894460-5lb9h" Nov 03 01:00:00.885500 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:00.885475 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8c543e5-5eda-4993-92af-8da6a226cfa6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-db5422-29894460-w4ksr\" (UID: \"c8c543e5-5eda-4993-92af-8da6a226cfa6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-db5422-29894460-w4ksr" Nov 03 01:00:00.886588 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:00.886559 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8c543e5-5eda-4993-92af-8da6a226cfa6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-db5422-29894460-w4ksr\" (UID: \"c8c543e5-5eda-4993-92af-8da6a226cfa6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-db5422-29894460-w4ksr" Nov 03 01:00:00.913697 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:00.911451 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9bb1557e-955c-41d1-8513-e1f8c4f341f3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-07f5f6-29894460-5lb9h\" (UID: \"9bb1557e-955c-41d1-8513-e1f8c4f341f3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-07f5f6-29894460-5lb9h" Nov 03 01:00:00.932138 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:00.932043 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d930dca-a318-4fb3-9a22-2a0dff4de5b4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-80f70e-29894460-kh466\" (UID: \"7d930dca-a318-4fb3-9a22-2a0dff4de5b4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-80f70e-29894460-kh466" Nov 03 01:00:01.188499 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:01.188467 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qssk2\" (UniqueName: \"kubernetes.io/projected/91abc456-0b19-4c43-9a94-57ef27f96c4a-kube-api-access-qssk2\") pod \"maas-api-key-cleanup-e2e-api-a-7406a9-29894460-vnwfw\" (UID: \"91abc456-0b19-4c43-9a94-57ef27f96c4a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-7406a9-29894460-vnwfw" Nov 03 01:00:01.188933 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:01.188567 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxlwn\" (UniqueName: \"kubernetes.io/projected/9bb1557e-955c-41d1-8513-e1f8c4f341f3-kube-api-access-nxlwn\") pod \"maas-api-key-cleanup-e2e-ipp-a-07f5f6-29894460-5lb9h\" (UID: \"9bb1557e-955c-41d1-8513-e1f8c4f341f3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-07f5f6-29894460-5lb9h" Nov 03 01:00:01.191250 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:01.191218 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtfh7\" (UniqueName: \"kubernetes.io/projected/7d930dca-a318-4fb3-9a22-2a0dff4de5b4-kube-api-access-vtfh7\") pod \"maas-api-key-cleanup-e2e-api-a-80f70e-29894460-kh466\" (UID: \"7d930dca-a318-4fb3-9a22-2a0dff4de5b4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-80f70e-29894460-kh466" Nov 03 01:00:01.191388 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:01.191329 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgckj\" (UniqueName: \"kubernetes.io/projected/c8c543e5-5eda-4993-92af-8da6a226cfa6-kube-api-access-cgckj\") pod \"maas-api-key-cleanup-e2e-ipp-b-db5422-29894460-w4ksr\" (UID: \"c8c543e5-5eda-4993-92af-8da6a226cfa6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-db5422-29894460-w4ksr" Nov 03 01:00:01.193642 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:01.193614 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbg45\" (UniqueName: \"kubernetes.io/projected/0370c2db-8545-4266-a4f4-719aebbd925b-kube-api-access-gbg45\") pod \"maas-api-key-cleanup-e2e-worker-w3-83755a-29894460-7z6xz\" (UID: \"0370c2db-8545-4266-a4f4-719aebbd925b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-83755a-29894460-7z6xz" Nov 03 01:00:01.204794 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:01.204756 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-80f70e-29894460-kh466" Nov 03 01:00:01.266442 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:01.265419 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/03c838c5-5695-434a-b78f-a338c14f3365-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-f88f87-29894460-p26bw\" (UID: \"03c838c5-5695-434a-b78f-a338c14f3365\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f88f87-29894460-p26bw" Nov 03 01:00:01.266442 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:01.265591 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6clrj\" (UniqueName: \"kubernetes.io/projected/03c838c5-5695-434a-b78f-a338c14f3365-kube-api-access-6clrj\") pod \"maas-api-key-cleanup-e2e-ipp-a-f88f87-29894460-p26bw\" (UID: \"03c838c5-5695-434a-b78f-a338c14f3365\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f88f87-29894460-p26bw" Nov 03 01:00:01.267844 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:01.267820 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-db5422-7f6b57dbd7-b5nvk"] Nov 03 01:00:01.276289 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:01.275188 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-db5422-29894460-w4ksr" Nov 03 01:00:01.276834 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:01.276800 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-db5422-7f6b57dbd7-b5nvk"] Nov 03 01:00:01.276939 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:01.276928 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-db5422-7f6b57dbd7-b5nvk" Nov 03 01:00:01.283518 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:01.283438 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-db5422\"" Nov 03 01:00:01.283786 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:01.283771 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-db5422-dockercfg-m2q5w\"" Nov 03 01:00:01.294977 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:01.292068 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-db5422-8464b57b74-pgngf"] Nov 03 01:00:01.301812 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:01.301610 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-db5422-8464b57b74-pgngf" Nov 03 01:00:01.302703 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:01.302677 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-db5422-8464b57b74-pgngf"] Nov 03 01:00:01.367932 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:01.367899 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6m9ht\" (UniqueName: \"kubernetes.io/projected/7a3242e9-467f-4c9f-b145-0ae70423d5cb-kube-api-access-6m9ht\") pod \"payload-pre-processing-e2e-ipp-b-db5422-7f6b57dbd7-b5nvk\" (UID: \"7a3242e9-467f-4c9f-b145-0ae70423d5cb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-db5422-7f6b57dbd7-b5nvk" Nov 03 01:00:01.368087 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:01.367974 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6clrj\" (UniqueName: \"kubernetes.io/projected/03c838c5-5695-434a-b78f-a338c14f3365-kube-api-access-6clrj\") pod \"maas-api-key-cleanup-e2e-ipp-a-f88f87-29894460-p26bw\" (UID: \"03c838c5-5695-434a-b78f-a338c14f3365\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f88f87-29894460-p26bw" Nov 03 01:00:01.369380 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:01.368285 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a3242e9-467f-4c9f-b145-0ae70423d5cb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-db5422-7f6b57dbd7-b5nvk\" (UID: \"7a3242e9-467f-4c9f-b145-0ae70423d5cb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-db5422-7f6b57dbd7-b5nvk" Nov 03 01:00:01.369380 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:01.368376 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/03c838c5-5695-434a-b78f-a338c14f3365-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-f88f87-29894460-p26bw\" (UID: \"03c838c5-5695-434a-b78f-a338c14f3365\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f88f87-29894460-p26bw" Nov 03 01:00:01.377218 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:01.373351 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/03c838c5-5695-434a-b78f-a338c14f3365-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-f88f87-29894460-p26bw\" (UID: \"03c838c5-5695-434a-b78f-a338c14f3365\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f88f87-29894460-p26bw" Nov 03 01:00:01.382704 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:01.382179 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-83755a-29894460-7z6xz" Nov 03 01:00:01.400896 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:01.400593 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-7406a9-29894460-vnwfw" Nov 03 01:00:01.436550 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:01.424455 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6clrj\" (UniqueName: \"kubernetes.io/projected/03c838c5-5695-434a-b78f-a338c14f3365-kube-api-access-6clrj\") pod \"maas-api-key-cleanup-e2e-ipp-a-f88f87-29894460-p26bw\" (UID: \"03c838c5-5695-434a-b78f-a338c14f3365\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f88f87-29894460-p26bw" Nov 03 01:00:01.436550 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:01.424724 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-07f5f6-29894460-5lb9h" Nov 03 01:00:01.559467 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:01.509686 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a3242e9-467f-4c9f-b145-0ae70423d5cb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-db5422-7f6b57dbd7-b5nvk\" (UID: \"7a3242e9-467f-4c9f-b145-0ae70423d5cb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-db5422-7f6b57dbd7-b5nvk" Nov 03 01:00:01.559467 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:01.509844 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82a61b7a-409c-4939-8715-1114dab89b24-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-db5422-8464b57b74-pgngf\" (UID: \"82a61b7a-409c-4939-8715-1114dab89b24\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-db5422-8464b57b74-pgngf" Nov 03 01:00:01.559467 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:01.511319 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a3242e9-467f-4c9f-b145-0ae70423d5cb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-db5422-7f6b57dbd7-b5nvk\" (UID: \"7a3242e9-467f-4c9f-b145-0ae70423d5cb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-db5422-7f6b57dbd7-b5nvk" Nov 03 01:00:01.559467 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:01.521499 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6m9ht\" (UniqueName: \"kubernetes.io/projected/7a3242e9-467f-4c9f-b145-0ae70423d5cb-kube-api-access-6m9ht\") pod \"payload-pre-processing-e2e-ipp-b-db5422-7f6b57dbd7-b5nvk\" (UID: \"7a3242e9-467f-4c9f-b145-0ae70423d5cb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-db5422-7f6b57dbd7-b5nvk" Nov 03 01:00:01.559467 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:01.521784 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9cc2\" (UniqueName: \"kubernetes.io/projected/82a61b7a-409c-4939-8715-1114dab89b24-kube-api-access-z9cc2\") pod \"payload-processing-e2e-ipp-b-db5422-8464b57b74-pgngf\" (UID: \"82a61b7a-409c-4939-8715-1114dab89b24\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-db5422-8464b57b74-pgngf" Nov 03 01:00:01.613297 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:01.574153 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f88f87-29894460-p26bw" Nov 03 01:00:01.613297 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:01.575979 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6m9ht\" (UniqueName: \"kubernetes.io/projected/7a3242e9-467f-4c9f-b145-0ae70423d5cb-kube-api-access-6m9ht\") pod \"payload-pre-processing-e2e-ipp-b-db5422-7f6b57dbd7-b5nvk\" (UID: \"7a3242e9-467f-4c9f-b145-0ae70423d5cb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-db5422-7f6b57dbd7-b5nvk" Nov 03 01:00:01.613297 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:01.611776 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-db5422-7f6b57dbd7-b5nvk" Nov 03 01:00:01.742357 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:01.702120 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9cc2\" (UniqueName: \"kubernetes.io/projected/82a61b7a-409c-4939-8715-1114dab89b24-kube-api-access-z9cc2\") pod \"payload-processing-e2e-ipp-b-db5422-8464b57b74-pgngf\" (UID: \"82a61b7a-409c-4939-8715-1114dab89b24\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-db5422-8464b57b74-pgngf" Nov 03 01:00:01.742357 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:01.702321 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82a61b7a-409c-4939-8715-1114dab89b24-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-db5422-8464b57b74-pgngf\" (UID: \"82a61b7a-409c-4939-8715-1114dab89b24\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-db5422-8464b57b74-pgngf" Nov 03 01:00:01.754334 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:01.750496 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82a61b7a-409c-4939-8715-1114dab89b24-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-db5422-8464b57b74-pgngf\" (UID: \"82a61b7a-409c-4939-8715-1114dab89b24\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-db5422-8464b57b74-pgngf" Nov 03 01:00:01.786161 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:01.786124 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9cc2\" (UniqueName: \"kubernetes.io/projected/82a61b7a-409c-4939-8715-1114dab89b24-kube-api-access-z9cc2\") pod \"payload-processing-e2e-ipp-b-db5422-8464b57b74-pgngf\" (UID: \"82a61b7a-409c-4939-8715-1114dab89b24\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-db5422-8464b57b74-pgngf" Nov 03 01:00:01.793634 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:01.793463 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-def754-7788f9b985-ppxwb" podUID="1423f3de-68b7-430d-9c0d-9e957468fedf" containerName="payload-processing" containerID="cri-o://58704160dca57f419fa862ccb1f0c33d5258b5a501d1c7c49d9805293921f482" gracePeriod=25 Nov 03 01:00:01.872904 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:01.872810 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7f43f9-29894460-bfjlr"] Nov 03 01:00:01.881451 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:00:01.881143 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64ad274e_6de7_4604_a8e4_958f67dbd00f.slice/crio-12b59779be49227646fd96e59f7abeab4105bc0e1079ed57e631e6a41119021c WatchSource:0}: Error finding container 12b59779be49227646fd96e59f7abeab4105bc0e1079ed57e631e6a41119021c: Status 404 returned error can't find the container with id 12b59779be49227646fd96e59f7abeab4105bc0e1079ed57e631e6a41119021c Nov 03 01:00:02.005010 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:02.004579 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-db5422-8464b57b74-pgngf" Nov 03 01:00:02.203075 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:02.202982 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-80f70e-29894460-kh466"] Nov 03 01:00:02.565648 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:02.565542 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f88f87-5d58f64779-pwc7n" podUID="c0fb5e39-8b3d-4bc4-8791-01042c8ab304" containerName="payload-pre-processing" containerID="cri-o://463b91c090caf30164eb2588e844de5b5b602eb8002b658dd515fb59bb78492c" gracePeriod=25 Nov 03 01:00:02.668672 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:02.666331 4421 generic.go:356] "Generic (PLEG): container finished" podID="1423f3de-68b7-430d-9c0d-9e957468fedf" containerID="58704160dca57f419fa862ccb1f0c33d5258b5a501d1c7c49d9805293921f482" exitCode=0 Nov 03 01:00:02.668672 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:02.666508 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-def754-7788f9b985-ppxwb" event={"ID":"1423f3de-68b7-430d-9c0d-9e957468fedf","Type":"ContainerDied","Data":"58704160dca57f419fa862ccb1f0c33d5258b5a501d1c7c49d9805293921f482"} Nov 03 01:00:02.668672 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:02.668385 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7f43f9-29894460-bfjlr" event={"ID":"64ad274e-6de7-4604-a8e4-958f67dbd00f","Type":"ContainerStarted","Data":"12b59779be49227646fd96e59f7abeab4105bc0e1079ed57e631e6a41119021c"} Nov 03 01:00:02.796251 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:02.796111 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894460-btsdn"] Nov 03 01:00:02.847473 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:00:02.837916 4421 logging.go:55] [core] [Channel #1604 SubChannel #1605]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.146:50052", ServerName: "10.134.0.146:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.146:50052: connect: connection refused" Nov 03 01:00:03.188863 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:00:03.187901 4421 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0fb5e39_8b3d_4bc4_8791_01042c8ab304.slice/crio-463b91c090caf30164eb2588e844de5b5b602eb8002b658dd515fb59bb78492c.scope\": RecentStats: unable to find data in memory cache]" Nov 03 01:00:03.385776 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:03.385675 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-80f70e-29894460-kh466"] Nov 03 01:00:03.391085 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:03.391049 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f9a3a-29894460-sj5kk"] Nov 03 01:00:03.399657 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:03.399633 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-def754-7788f9b985-ppxwb" Nov 03 01:00:03.488155 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:03.487131 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-db5422-29894460-w4ksr"] Nov 03 01:00:03.489915 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:03.489887 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1423f3de-68b7-430d-9c0d-9e957468fedf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1423f3de-68b7-430d-9c0d-9e957468fedf-plugins-config-volume\") pod \"1423f3de-68b7-430d-9c0d-9e957468fedf\" (UID: \"1423f3de-68b7-430d-9c0d-9e957468fedf\") " Nov 03 01:00:03.490252 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:03.490076 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1423f3de-68b7-430d-9c0d-9e957468fedf-kube-api-access-n4vhw\" (UniqueName: \"kubernetes.io/projected/1423f3de-68b7-430d-9c0d-9e957468fedf-kube-api-access-n4vhw\") pod \"1423f3de-68b7-430d-9c0d-9e957468fedf\" (UID: \"1423f3de-68b7-430d-9c0d-9e957468fedf\") " Nov 03 01:00:03.490779 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:03.490505 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1423f3de-68b7-430d-9c0d-9e957468fedf-plugins-config-volume" pod "1423f3de-68b7-430d-9c0d-9e957468fedf" (UID: "1423f3de-68b7-430d-9c0d-9e957468fedf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:00:03.490779 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:03.490718 4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1423f3de-68b7-430d-9c0d-9e957468fedf-plugins-config-volume\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:00:03.492777 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:03.492727 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1423f3de-68b7-430d-9c0d-9e957468fedf-kube-api-access-n4vhw" pod "1423f3de-68b7-430d-9c0d-9e957468fedf" (UID: "1423f3de-68b7-430d-9c0d-9e957468fedf"). InnerVolumeSpecName "kube-api-access-n4vhw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:00:03.656044 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:03.611546 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4vhw\" (UniqueName: \"kubernetes.io/projected/1423f3de-68b7-430d-9c0d-9e957468fedf-kube-api-access-n4vhw\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:00:03.785687 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:03.785654 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f9a3a-29894460-sj5kk" event={"ID":"84fca73a-b1e2-4fee-892f-b2f162bd75b3","Type":"ContainerStarted","Data":"aa539b67894d9ad5095562c732c9966949dd147c2840b016c41c14d19ef2eb97"} Nov 03 01:00:03.789869 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:03.789837 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-def754-7788f9b985-ppxwb" event={"ID":"1423f3de-68b7-430d-9c0d-9e957468fedf","Type":"ContainerDied","Data":"0e429ec9c2e5dd850068139f396cee411546b029018e45758c0eb921103e4174"} Nov 03 01:00:03.789989 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:03.789899 4421 scope.go:122] "RemoveContainer" containerID="58704160dca57f419fa862ccb1f0c33d5258b5a501d1c7c49d9805293921f482" Nov 03 01:00:03.790058 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:03.790022 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-def754-7788f9b985-ppxwb" Nov 03 01:00:03.797636 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:03.797600 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-db5422-29894460-w4ksr" event={"ID":"c8c543e5-5eda-4993-92af-8da6a226cfa6","Type":"ContainerStarted","Data":"ab0484440b11a8f71f2c3588df044fd2305aaa6914535a4c66d1dbd405761fcf"} Nov 03 01:00:03.830791 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:03.830138 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7f43f9-29894460-bfjlr" event={"ID":"64ad274e-6de7-4604-a8e4-958f67dbd00f","Type":"ContainerStarted","Data":"0d2c97d2c478a353f658ce0549a18c60013b7df0eac034c84ee8efb4beff6f32"} Nov 03 01:00:03.830968 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:03.830930 4421 scope.go:122] "RemoveContainer" containerID="0d2c97d2c478a353f658ce0549a18c60013b7df0eac034c84ee8efb4beff6f32" Nov 03 01:00:03.839770 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:03.838961 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f88f87-5d58f64779-pwc7n" podUID="c0fb5e39-8b3d-4bc4-8791-01042c8ab304" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.146:50052\" within 1s: context deadline exceeded" Nov 03 01:00:03.846933 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:03.843365 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894460-btsdn" event={"ID":"569546b1-7662-4df6-a849-23fb4095797b","Type":"ContainerStarted","Data":"c7873861cf42d761ec3137912ae6379d1a88b0217a5abb6bb89ccb446e87e8a1"} Nov 03 01:00:03.851850 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:03.851816 4421 generic.go:356] "Generic (PLEG): container finished" podID="c0fb5e39-8b3d-4bc4-8791-01042c8ab304" containerID="463b91c090caf30164eb2588e844de5b5b602eb8002b658dd515fb59bb78492c" exitCode=0 Nov 03 01:00:03.851971 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:03.851900 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f88f87-5d58f64779-pwc7n" event={"ID":"c0fb5e39-8b3d-4bc4-8791-01042c8ab304","Type":"ContainerDied","Data":"463b91c090caf30164eb2588e844de5b5b602eb8002b658dd515fb59bb78492c"} Nov 03 01:00:03.866802 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:03.866733 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-80f70e-29894460-kh466" event={"ID":"7d930dca-a318-4fb3-9a22-2a0dff4de5b4","Type":"ContainerStarted","Data":"3a57c4f0b0c0efd68ff2851be4fe599fa02ed4c1ad10a32a36288767e93bf88b"} Nov 03 01:00:03.879537 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:03.879431 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-7406a9-29894460-vnwfw"] Nov 03 01:00:03.884152 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:03.884093 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894460-btsdn" podStartSLOduration=3.884073446 podStartE2EDuration="3.884073446s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:00:03.883694585 +0000 UTC m=+1146.054697682" watchObservedRunningTime="2026-11-03 01:00:03.884073446 +0000 UTC m=+1146.055076542" Nov 03 01:00:04.043394 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:04.042348 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-db5422-7f6b57dbd7-b5nvk"] Nov 03 01:00:04.064611 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:04.064568 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f7680d-29894460-qpglw"] Nov 03 01:00:04.067008 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:00:04.066972 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod600dc1bf_8abc_480d_8503_3942ecd9d48b.slice/crio-db1afae8c78cf220ccf4f68ff1b7737a373ccb6885f717bc33f89324e087b0f3 WatchSource:0}: Error finding container db1afae8c78cf220ccf4f68ff1b7737a373ccb6885f717bc33f89324e087b0f3: Status 404 returned error can't find the container with id db1afae8c78cf220ccf4f68ff1b7737a373ccb6885f717bc33f89324e087b0f3 Nov 03 01:00:04.073467 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:04.073387 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f88f87-5d58f64779-pwc7n" Nov 03 01:00:04.094864 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:04.094832 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-def754-7788f9b985-ppxwb"] Nov 03 01:00:04.102202 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:04.102172 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-def754-7788f9b985-ppxwb"] Nov 03 01:00:04.134508 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:04.127833 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f88f87-29894460-p26bw"] Nov 03 01:00:04.134508 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:04.132388 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-db5422-8464b57b74-pgngf"] Nov 03 01:00:04.155398 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:04.155366 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0fb5e39-8b3d-4bc4-8791-01042c8ab304-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0fb5e39-8b3d-4bc4-8791-01042c8ab304-plugins-config-volume\") pod \"c0fb5e39-8b3d-4bc4-8791-01042c8ab304\" (UID: \"c0fb5e39-8b3d-4bc4-8791-01042c8ab304\") " Nov 03 01:00:04.155572 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:04.155412 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0fb5e39-8b3d-4bc4-8791-01042c8ab304-kube-api-access-x995d\" (UniqueName: \"kubernetes.io/projected/c0fb5e39-8b3d-4bc4-8791-01042c8ab304-kube-api-access-x995d\") pod \"c0fb5e39-8b3d-4bc4-8791-01042c8ab304\" (UID: \"c0fb5e39-8b3d-4bc4-8791-01042c8ab304\") " Nov 03 01:00:04.155793 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:04.155768 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0fb5e39-8b3d-4bc4-8791-01042c8ab304-plugins-config-volume" pod "c0fb5e39-8b3d-4bc4-8791-01042c8ab304" (UID: "c0fb5e39-8b3d-4bc4-8791-01042c8ab304"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:00:04.156059 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:04.156038 4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0fb5e39-8b3d-4bc4-8791-01042c8ab304-plugins-config-volume\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:00:04.200200 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:04.200176 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-83755a-29894460-7z6xz"] Nov 03 01:00:04.204401 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:04.204376 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-07f5f6-29894460-5lb9h"] Nov 03 01:00:04.307734 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:04.307690 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0fb5e39-8b3d-4bc4-8791-01042c8ab304-kube-api-access-x995d" pod "c0fb5e39-8b3d-4bc4-8791-01042c8ab304" (UID: "c0fb5e39-8b3d-4bc4-8791-01042c8ab304"). InnerVolumeSpecName "kube-api-access-x995d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:00:04.326964 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:00:04.326771 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03c838c5_5695_434a_b78f_a338c14f3365.slice/crio-2477b35d77103a9b7271529947d029a46d0eb35039caf7fa0b5397be7e3165f2 WatchSource:0}: Error finding container 2477b35d77103a9b7271529947d029a46d0eb35039caf7fa0b5397be7e3165f2: Status 404 returned error can't find the container with id 2477b35d77103a9b7271529947d029a46d0eb35039caf7fa0b5397be7e3165f2 Nov 03 01:00:04.327061 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:00:04.326974 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82a61b7a_409c_4939_8715_1114dab89b24.slice/crio-c3d689f812180bd501a0a874f52833d6e81290c39cc5bce23e079920c616b227 WatchSource:0}: Error finding container c3d689f812180bd501a0a874f52833d6e81290c39cc5bce23e079920c616b227: Status 404 returned error can't find the container with id c3d689f812180bd501a0a874f52833d6e81290c39cc5bce23e079920c616b227 Nov 03 01:00:04.330839 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:00:04.328712 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0370c2db_8545_4266_a4f4_719aebbd925b.slice/crio-edc668facbe2d8ad99732a56a8a5a4e754e3ed7a28c1a3e879f8eb2046515472 WatchSource:0}: Error finding container edc668facbe2d8ad99732a56a8a5a4e754e3ed7a28c1a3e879f8eb2046515472: Status 404 returned error can't find the container with id edc668facbe2d8ad99732a56a8a5a4e754e3ed7a28c1a3e879f8eb2046515472 Nov 03 01:00:04.358731 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:04.358684 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x995d\" (UniqueName: \"kubernetes.io/projected/c0fb5e39-8b3d-4bc4-8791-01042c8ab304-kube-api-access-x995d\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:00:04.879285 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:04.879231 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-db5422-8464b57b74-pgngf" event={"ID":"82a61b7a-409c-4939-8715-1114dab89b24","Type":"ContainerStarted","Data":"fdba2e2f24937aa6033c40b03dbe4b800edf82fea08052bc7b1c6aca540e93df"} Nov 03 01:00:04.879733 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:04.879307 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-db5422-8464b57b74-pgngf" event={"ID":"82a61b7a-409c-4939-8715-1114dab89b24","Type":"ContainerStarted","Data":"c3d689f812180bd501a0a874f52833d6e81290c39cc5bce23e079920c616b227"} Nov 03 01:00:04.879733 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:04.879339 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-db5422-8464b57b74-pgngf" Nov 03 01:00:04.881510 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:04.881475 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f7680d-29894460-qpglw" event={"ID":"600dc1bf-8abc-480d-8503-3942ecd9d48b","Type":"ContainerStarted","Data":"b48ff27a49985a542b09994a61270210fb715f54a69d6033cde173079b81715a"} Nov 03 01:00:04.881644 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:04.881546 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f7680d-29894460-qpglw" event={"ID":"600dc1bf-8abc-480d-8503-3942ecd9d48b","Type":"ContainerStarted","Data":"db1afae8c78cf220ccf4f68ff1b7737a373ccb6885f717bc33f89324e087b0f3"} Nov 03 01:00:04.883400 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:04.883370 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-7406a9-29894460-vnwfw" event={"ID":"91abc456-0b19-4c43-9a94-57ef27f96c4a","Type":"ContainerStarted","Data":"d5b99aaa518248bb6c7224d70faf1d845bc97bf228da53bb2b75a61973e3fe9f"} Nov 03 01:00:04.883525 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:04.883426 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-7406a9-29894460-vnwfw" event={"ID":"91abc456-0b19-4c43-9a94-57ef27f96c4a","Type":"ContainerStarted","Data":"bbd3f758933261a61f90ac0a6ee789f171195ebde8cbcb9f8334a9b53b506dab"} Nov 03 01:00:04.885288 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:04.885245 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-db5422-7f6b57dbd7-b5nvk" event={"ID":"7a3242e9-467f-4c9f-b145-0ae70423d5cb","Type":"ContainerStarted","Data":"d69307c467ec7d304c28ce8fcae8dd0dcdea9e7fadfb4ffbaa49079ffe68d44f"} Nov 03 01:00:04.885396 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:04.885335 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-db5422-7f6b57dbd7-b5nvk" event={"ID":"7a3242e9-467f-4c9f-b145-0ae70423d5cb","Type":"ContainerStarted","Data":"83e0c006fcc1888418b2c60b9691c8b5d3b6e82a1226c97c58cb46cf1dc4e166"} Nov 03 01:00:04.885396 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:04.885367 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-db5422-7f6b57dbd7-b5nvk" Nov 03 01:00:04.887553 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:04.887529 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-07f5f6-29894460-5lb9h" event={"ID":"9bb1557e-955c-41d1-8513-e1f8c4f341f3","Type":"ContainerStarted","Data":"60aade7b8f2b1eb66e208d1995130feece3900a54db72e29861d298167e00802"} Nov 03 01:00:04.889068 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:04.889046 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f88f87-29894460-p26bw" event={"ID":"03c838c5-5695-434a-b78f-a338c14f3365","Type":"ContainerStarted","Data":"2477b35d77103a9b7271529947d029a46d0eb35039caf7fa0b5397be7e3165f2"} Nov 03 01:00:04.890948 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:04.890924 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f9a3a-29894460-sj5kk" event={"ID":"84fca73a-b1e2-4fee-892f-b2f162bd75b3","Type":"ContainerStarted","Data":"6647a921578d2a84a13ef6b4ba8e63414d05409772038e083868e9947c55b906"} Nov 03 01:00:04.893734 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:04.893707 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-83755a-29894460-7z6xz" event={"ID":"0370c2db-8545-4266-a4f4-719aebbd925b","Type":"ContainerStarted","Data":"edc668facbe2d8ad99732a56a8a5a4e754e3ed7a28c1a3e879f8eb2046515472"} Nov 03 01:00:04.895290 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:04.895256 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-db5422-29894460-w4ksr" event={"ID":"c8c543e5-5eda-4993-92af-8da6a226cfa6","Type":"ContainerStarted","Data":"3007c3d9d74b1e76d96e4c3eda4885c99d21843e140f6de9b57987d5b441d81f"} Nov 03 01:00:04.897780 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:04.897756 4421 generic.go:356] "Generic (PLEG): container finished" podID="64ad274e-6de7-4604-a8e4-958f67dbd00f" containerID="0d2c97d2c478a353f658ce0549a18c60013b7df0eac034c84ee8efb4beff6f32" exitCode=60 Nov 03 01:00:04.897905 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:04.897831 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7f43f9-29894460-bfjlr" event={"ID":"64ad274e-6de7-4604-a8e4-958f67dbd00f","Type":"ContainerDied","Data":"0d2c97d2c478a353f658ce0549a18c60013b7df0eac034c84ee8efb4beff6f32"} Nov 03 01:00:04.897905 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:04.897869 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7f43f9-29894460-bfjlr" event={"ID":"64ad274e-6de7-4604-a8e4-958f67dbd00f","Type":"ContainerStarted","Data":"ad3a057f372b419a664cc22c8b3d1ec3e2915a9b0176fd0b63152a9ce8de178e"} Nov 03 01:00:04.899756 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:04.899732 4421 generic.go:356] "Generic (PLEG): container finished" podID="569546b1-7662-4df6-a849-23fb4095797b" containerID="0cece1d6b8f111c22cddfd890db27f1b2013525919f3c85ab6e7eed23fc18224" exitCode=60 Nov 03 01:00:04.900226 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:04.899997 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894460-btsdn" event={"ID":"569546b1-7662-4df6-a849-23fb4095797b","Type":"ContainerDied","Data":"0cece1d6b8f111c22cddfd890db27f1b2013525919f3c85ab6e7eed23fc18224"} Nov 03 01:00:04.900226 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:04.900187 4421 scope.go:122] "RemoveContainer" containerID="0cece1d6b8f111c22cddfd890db27f1b2013525919f3c85ab6e7eed23fc18224" Nov 03 01:00:04.901875 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:04.901855 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f88f87-5d58f64779-pwc7n" Nov 03 01:00:04.901981 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:04.901934 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f88f87-5d58f64779-pwc7n" event={"ID":"c0fb5e39-8b3d-4bc4-8791-01042c8ab304","Type":"ContainerDied","Data":"33e304068221cce15efc794f98f46956ece2934bde7b646eaa4c7e7de165b62c"} Nov 03 01:00:04.901981 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:04.901979 4421 scope.go:122] "RemoveContainer" containerID="463b91c090caf30164eb2588e844de5b5b602eb8002b658dd515fb59bb78492c" Nov 03 01:00:04.905254 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:04.905221 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-80f70e-29894460-kh466" event={"ID":"7d930dca-a318-4fb3-9a22-2a0dff4de5b4","Type":"ContainerStarted","Data":"e86835258b9e403da9e44ee17e6bf69be76aca9a6e7ff35c0cb020595adaf866"} Nov 03 01:00:04.912901 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:04.912799 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-80f70e-29894460-kh466" podUID="7d930dca-a318-4fb3-9a22-2a0dff4de5b4" containerName="cleanup" containerID="cri-o://e86835258b9e403da9e44ee17e6bf69be76aca9a6e7ff35c0cb020595adaf866" gracePeriod=30 Nov 03 01:00:04.915513 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:04.915459 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-db5422-8464b57b74-pgngf" podStartSLOduration=4.915440126 podStartE2EDuration="4.915440126s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:00:04.902975437 +0000 UTC m=+1147.073978551" watchObservedRunningTime="2026-11-03 01:00:04.915440126 +0000 UTC m=+1147.086443223" Nov 03 01:00:04.925684 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:04.925631 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-db5422-29894460-w4ksr" podStartSLOduration=4.925614705 podStartE2EDuration="4.925614705s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:00:04.925205076 +0000 UTC m=+1147.096208176" watchObservedRunningTime="2026-11-03 01:00:04.925614705 +0000 UTC m=+1147.096617802" Nov 03 01:00:04.946546 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:04.946489 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-7406a9-29894460-vnwfw" podStartSLOduration=4.94646114 podStartE2EDuration="4.94646114s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:00:04.943915758 +0000 UTC m=+1147.114918856" watchObservedRunningTime="2026-11-03 01:00:04.94646114 +0000 UTC m=+1147.117464241" Nov 03 01:00:04.969113 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:04.968548 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-db5422-7f6b57dbd7-b5nvk" podStartSLOduration=4.968529531 podStartE2EDuration="4.968529531s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:00:04.966560277 +0000 UTC m=+1147.137563374" watchObservedRunningTime="2026-11-03 01:00:04.968529531 +0000 UTC m=+1147.139532627" Nov 03 01:00:05.001743 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:05.001689 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7f43f9-29894460-bfjlr" podStartSLOduration=5.001675277 podStartE2EDuration="5.001675277s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:00:04.999164828 +0000 UTC m=+1147.170167924" watchObservedRunningTime="2026-11-03 01:00:05.001675277 +0000 UTC m=+1147.172678372" Nov 03 01:00:05.016422 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:05.016156 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f9a3a-29894460-sj5kk" podStartSLOduration=5.016136268 podStartE2EDuration="5.016136268s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:00:05.015118319 +0000 UTC m=+1147.186121419" watchObservedRunningTime="2026-11-03 01:00:05.016136268 +0000 UTC m=+1147.187139367" Nov 03 01:00:05.032676 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:05.032632 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5fb83c19-bnnjc"] Nov 03 01:00:05.037466 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:05.037433 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5fb83c19-bnnjc"] Nov 03 01:00:05.055706 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:05.055655 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f7680d-29894460-qpglw" podStartSLOduration=5.05564204 podStartE2EDuration="5.05564204s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:00:05.050335426 +0000 UTC m=+1147.221338522" watchObservedRunningTime="2026-11-03 01:00:05.05564204 +0000 UTC m=+1147.226645162" Nov 03 01:00:05.114297 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:05.109677 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-80f70e-29894460-kh466" podStartSLOduration=5.10965547 podStartE2EDuration="5.10965547s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:00:05.075244505 +0000 UTC m=+1147.246247602" watchObservedRunningTime="2026-11-03 01:00:05.10965547 +0000 UTC m=+1147.280658567" Nov 03 01:00:05.176459 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:05.176373 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f88f87-5d58f64779-pwc7n"] Nov 03 01:00:05.203537 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:05.203504 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f88f87-5d58f64779-pwc7n"] Nov 03 01:00:05.608783 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:05.608461 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-80f70e-29894460-kh466" Nov 03 01:00:05.681662 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:05.681619 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d930dca-a318-4fb3-9a22-2a0dff4de5b4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d930dca-a318-4fb3-9a22-2a0dff4de5b4-maas-api-service-ca\") pod \"7d930dca-a318-4fb3-9a22-2a0dff4de5b4\" (UID: \"7d930dca-a318-4fb3-9a22-2a0dff4de5b4\") " Nov 03 01:00:05.682047 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:05.682018 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d930dca-a318-4fb3-9a22-2a0dff4de5b4-maas-api-service-ca" pod "7d930dca-a318-4fb3-9a22-2a0dff4de5b4" (UID: "7d930dca-a318-4fb3-9a22-2a0dff4de5b4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:00:05.682151 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:05.682025 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d930dca-a318-4fb3-9a22-2a0dff4de5b4-kube-api-access-vtfh7\" (UniqueName: \"kubernetes.io/projected/7d930dca-a318-4fb3-9a22-2a0dff4de5b4-kube-api-access-vtfh7\") pod \"7d930dca-a318-4fb3-9a22-2a0dff4de5b4\" (UID: \"7d930dca-a318-4fb3-9a22-2a0dff4de5b4\") " Nov 03 01:00:05.682765 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:05.682740 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d930dca-a318-4fb3-9a22-2a0dff4de5b4-maas-api-service-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:00:05.684386 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:05.684360 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d930dca-a318-4fb3-9a22-2a0dff4de5b4-kube-api-access-vtfh7" pod "7d930dca-a318-4fb3-9a22-2a0dff4de5b4" (UID: "7d930dca-a318-4fb3-9a22-2a0dff4de5b4"). InnerVolumeSpecName "kube-api-access-vtfh7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:00:05.785112 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:05.785015 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vtfh7\" (UniqueName: \"kubernetes.io/projected/7d930dca-a318-4fb3-9a22-2a0dff4de5b4-kube-api-access-vtfh7\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:00:05.921803 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:05.921767 4421 generic.go:356] "Generic (PLEG): container finished" podID="c8c543e5-5eda-4993-92af-8da6a226cfa6" containerID="3007c3d9d74b1e76d96e4c3eda4885c99d21843e140f6de9b57987d5b441d81f" exitCode=60 Nov 03 01:00:05.922443 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:05.921889 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-db5422-29894460-w4ksr" event={"ID":"c8c543e5-5eda-4993-92af-8da6a226cfa6","Type":"ContainerDied","Data":"3007c3d9d74b1e76d96e4c3eda4885c99d21843e140f6de9b57987d5b441d81f"} Nov 03 01:00:05.922443 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:05.922343 4421 scope.go:122] "RemoveContainer" containerID="3007c3d9d74b1e76d96e4c3eda4885c99d21843e140f6de9b57987d5b441d81f" Nov 03 01:00:05.924936 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:05.924876 4421 generic.go:356] "Generic (PLEG): container finished" podID="64ad274e-6de7-4604-a8e4-958f67dbd00f" containerID="ad3a057f372b419a664cc22c8b3d1ec3e2915a9b0176fd0b63152a9ce8de178e" exitCode=60 Nov 03 01:00:05.925015 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:05.924956 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7f43f9-29894460-bfjlr" event={"ID":"64ad274e-6de7-4604-a8e4-958f67dbd00f","Type":"ContainerDied","Data":"ad3a057f372b419a664cc22c8b3d1ec3e2915a9b0176fd0b63152a9ce8de178e"} Nov 03 01:00:05.925015 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:05.925006 4421 scope.go:122] "RemoveContainer" containerID="0d2c97d2c478a353f658ce0549a18c60013b7df0eac034c84ee8efb4beff6f32" Nov 03 01:00:05.925572 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:05.925507 4421 scope.go:122] "RemoveContainer" containerID="ad3a057f372b419a664cc22c8b3d1ec3e2915a9b0176fd0b63152a9ce8de178e" Nov 03 01:00:05.925874 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:00:05.925852 4421 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-7f43f9-29894460-bfjlr_odh-ai-gateway-infra(64ad274e-6de7-4604-a8e4-958f67dbd00f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7f43f9-29894460-bfjlr" podUID="64ad274e-6de7-4604-a8e4-958f67dbd00f" Nov 03 01:00:05.929250 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:05.929188 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894460-btsdn" event={"ID":"569546b1-7662-4df6-a849-23fb4095797b","Type":"ContainerStarted","Data":"ca926a9c32f02b3235f64fcd8bad103a2c5bef74fb7104c863ea4c418e64b0c9"} Nov 03 01:00:05.932565 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:05.932539 4421 generic.go:356] "Generic (PLEG): container finished" podID="7d930dca-a318-4fb3-9a22-2a0dff4de5b4" containerID="e86835258b9e403da9e44ee17e6bf69be76aca9a6e7ff35c0cb020595adaf866" exitCode=6 Nov 03 01:00:05.932698 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:05.932681 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-80f70e-29894460-kh466" event={"ID":"7d930dca-a318-4fb3-9a22-2a0dff4de5b4","Type":"ContainerDied","Data":"e86835258b9e403da9e44ee17e6bf69be76aca9a6e7ff35c0cb020595adaf866"} Nov 03 01:00:05.932768 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:05.932727 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-80f70e-29894460-kh466" event={"ID":"7d930dca-a318-4fb3-9a22-2a0dff4de5b4","Type":"ContainerDied","Data":"3a57c4f0b0c0efd68ff2851be4fe599fa02ed4c1ad10a32a36288767e93bf88b"} Nov 03 01:00:05.932931 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:05.932899 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-80f70e-29894460-kh466" Nov 03 01:00:05.934909 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:05.934882 4421 generic.go:356] "Generic (PLEG): container finished" podID="600dc1bf-8abc-480d-8503-3942ecd9d48b" containerID="b48ff27a49985a542b09994a61270210fb715f54a69d6033cde173079b81715a" exitCode=60 Nov 03 01:00:05.935035 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:05.934959 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f7680d-29894460-qpglw" event={"ID":"600dc1bf-8abc-480d-8503-3942ecd9d48b","Type":"ContainerDied","Data":"b48ff27a49985a542b09994a61270210fb715f54a69d6033cde173079b81715a"} Nov 03 01:00:05.935415 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:05.935396 4421 scope.go:122] "RemoveContainer" containerID="b48ff27a49985a542b09994a61270210fb715f54a69d6033cde173079b81715a" Nov 03 01:00:05.944789 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:05.944758 4421 generic.go:356] "Generic (PLEG): container finished" podID="91abc456-0b19-4c43-9a94-57ef27f96c4a" containerID="d5b99aaa518248bb6c7224d70faf1d845bc97bf228da53bb2b75a61973e3fe9f" exitCode=60 Nov 03 01:00:05.944930 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:05.944884 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-7406a9-29894460-vnwfw" event={"ID":"91abc456-0b19-4c43-9a94-57ef27f96c4a","Type":"ContainerDied","Data":"d5b99aaa518248bb6c7224d70faf1d845bc97bf228da53bb2b75a61973e3fe9f"} Nov 03 01:00:05.945308 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:05.945288 4421 scope.go:122] "RemoveContainer" containerID="d5b99aaa518248bb6c7224d70faf1d845bc97bf228da53bb2b75a61973e3fe9f" Nov 03 01:00:05.954217 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:05.954057 4421 generic.go:356] "Generic (PLEG): container finished" podID="9bb1557e-955c-41d1-8513-e1f8c4f341f3" containerID="b75663971452088072a6e9f4ea4521199c213c2b1ecdf02dda78b84fc7ffae64" exitCode=60 Nov 03 01:00:05.954217 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:05.954192 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-07f5f6-29894460-5lb9h" event={"ID":"9bb1557e-955c-41d1-8513-e1f8c4f341f3","Type":"ContainerDied","Data":"b75663971452088072a6e9f4ea4521199c213c2b1ecdf02dda78b84fc7ffae64"} Nov 03 01:00:05.954681 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:05.954658 4421 scope.go:122] "RemoveContainer" containerID="b75663971452088072a6e9f4ea4521199c213c2b1ecdf02dda78b84fc7ffae64" Nov 03 01:00:05.961132 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:05.961095 4421 generic.go:356] "Generic (PLEG): container finished" podID="03c838c5-5695-434a-b78f-a338c14f3365" containerID="d79a7d4a6643c83471dd0aec12616d1d5fe428374706fca4b23ec6c0e01f9260" exitCode=60 Nov 03 01:00:05.961264 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:05.961242 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f88f87-29894460-p26bw" event={"ID":"03c838c5-5695-434a-b78f-a338c14f3365","Type":"ContainerDied","Data":"d79a7d4a6643c83471dd0aec12616d1d5fe428374706fca4b23ec6c0e01f9260"} Nov 03 01:00:05.961521 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:05.961500 4421 scope.go:122] "RemoveContainer" containerID="d79a7d4a6643c83471dd0aec12616d1d5fe428374706fca4b23ec6c0e01f9260" Nov 03 01:00:05.963368 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:05.963344 4421 generic.go:356] "Generic (PLEG): container finished" podID="84fca73a-b1e2-4fee-892f-b2f162bd75b3" containerID="6647a921578d2a84a13ef6b4ba8e63414d05409772038e083868e9947c55b906" exitCode=60 Nov 03 01:00:05.963473 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:05.963454 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f9a3a-29894460-sj5kk" event={"ID":"84fca73a-b1e2-4fee-892f-b2f162bd75b3","Type":"ContainerDied","Data":"6647a921578d2a84a13ef6b4ba8e63414d05409772038e083868e9947c55b906"} Nov 03 01:00:05.963733 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:05.963713 4421 scope.go:122] "RemoveContainer" containerID="6647a921578d2a84a13ef6b4ba8e63414d05409772038e083868e9947c55b906" Nov 03 01:00:05.965742 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:05.965726 4421 scope.go:122] "RemoveContainer" containerID="e86835258b9e403da9e44ee17e6bf69be76aca9a6e7ff35c0cb020595adaf866" Nov 03 01:00:05.966663 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:05.966642 4421 generic.go:356] "Generic (PLEG): container finished" podID="0370c2db-8545-4266-a4f4-719aebbd925b" containerID="16fcad589126ee525216b2fe49ca62fdb7f944bc86520d28684818a20ab201ed" exitCode=60 Nov 03 01:00:05.967202 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:05.966793 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-83755a-29894460-7z6xz" event={"ID":"0370c2db-8545-4266-a4f4-719aebbd925b","Type":"ContainerDied","Data":"16fcad589126ee525216b2fe49ca62fdb7f944bc86520d28684818a20ab201ed"} Nov 03 01:00:05.967339 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:05.967054 4421 scope.go:122] "RemoveContainer" containerID="16fcad589126ee525216b2fe49ca62fdb7f944bc86520d28684818a20ab201ed" Nov 03 01:00:05.995518 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:05.995149 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1423f3de-68b7-430d-9c0d-9e957468fedf" path="/var/lib/kubelet/pods/1423f3de-68b7-430d-9c0d-9e957468fedf/volumes" Nov 03 01:00:05.995874 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:05.995850 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ca48327-61e7-4ea8-828e-57ce1e72845e" path="/var/lib/kubelet/pods/3ca48327-61e7-4ea8-828e-57ce1e72845e/volumes" Nov 03 01:00:05.996398 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:05.996381 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0fb5e39-8b3d-4bc4-8791-01042c8ab304" path="/var/lib/kubelet/pods/c0fb5e39-8b3d-4bc4-8791-01042c8ab304/volumes" Nov 03 01:00:06.000226 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:05.999865 4421 scope.go:122] "RemoveContainer" containerID="e86835258b9e403da9e44ee17e6bf69be76aca9a6e7ff35c0cb020595adaf866" Nov 03 01:00:06.000226 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:00:06.000147 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e86835258b9e403da9e44ee17e6bf69be76aca9a6e7ff35c0cb020595adaf866\": container with ID starting with e86835258b9e403da9e44ee17e6bf69be76aca9a6e7ff35c0cb020595adaf866 not found: ID does not exist" containerID="e86835258b9e403da9e44ee17e6bf69be76aca9a6e7ff35c0cb020595adaf866" Nov 03 01:00:06.000226 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:06.000186 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e86835258b9e403da9e44ee17e6bf69be76aca9a6e7ff35c0cb020595adaf866"} err="failed to get container status \"e86835258b9e403da9e44ee17e6bf69be76aca9a6e7ff35c0cb020595adaf866\": rpc error: code = NotFound desc = could not find container \"e86835258b9e403da9e44ee17e6bf69be76aca9a6e7ff35c0cb020595adaf866\": container with ID starting with e86835258b9e403da9e44ee17e6bf69be76aca9a6e7ff35c0cb020595adaf866 not found: ID does not exist" Nov 03 01:00:06.070046 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:06.069961 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-80f70e-29894460-kh466"] Nov 03 01:00:06.074352 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:06.074326 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-80f70e-29894460-kh466"] Nov 03 01:00:06.993484 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:06.993350 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-07f5f6-29894460-5lb9h" event={"ID":"9bb1557e-955c-41d1-8513-e1f8c4f341f3","Type":"ContainerStarted","Data":"6b649f0f038f254fe4910bc23e1ad3d5bd9d8442e94506cf305db76654a0bde8"} Nov 03 01:00:06.996495 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:06.996461 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f88f87-29894460-p26bw" event={"ID":"03c838c5-5695-434a-b78f-a338c14f3365","Type":"ContainerStarted","Data":"2d9afba2766dae0563b1322cbe1ebb8173a6d2289ce6f20e733f5a65a6d2fd26"} Nov 03 01:00:06.998736 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:06.998712 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f9a3a-29894460-sj5kk" event={"ID":"84fca73a-b1e2-4fee-892f-b2f162bd75b3","Type":"ContainerStarted","Data":"25edee7d461b7aeda9e0548939876d994839442bd21015cd8406839acf561e9e"} Nov 03 01:00:07.001232 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:07.001195 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-83755a-29894460-7z6xz" event={"ID":"0370c2db-8545-4266-a4f4-719aebbd925b","Type":"ContainerStarted","Data":"13bb8e2dbd9cd36c11e590c2ac62f3a5c5974b9dfdc09cd37fe871f66245f48e"} Nov 03 01:00:07.004000 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:07.003972 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-db5422-29894460-w4ksr" event={"ID":"c8c543e5-5eda-4993-92af-8da6a226cfa6","Type":"ContainerStarted","Data":"428bbf3fffdde8124c0eabda8092609f044c249f9e48fcdd6389a9251f531e85"} Nov 03 01:00:07.007715 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:07.007692 4421 scope.go:122] "RemoveContainer" containerID="ad3a057f372b419a664cc22c8b3d1ec3e2915a9b0176fd0b63152a9ce8de178e" Nov 03 01:00:07.008197 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:00:07.008172 4421 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-7f43f9-29894460-bfjlr_odh-ai-gateway-infra(64ad274e-6de7-4604-a8e4-958f67dbd00f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7f43f9-29894460-bfjlr" podUID="64ad274e-6de7-4604-a8e4-958f67dbd00f" Nov 03 01:00:07.013808 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:07.013738 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-07f5f6-29894460-5lb9h" podStartSLOduration=7.013718844 podStartE2EDuration="7.013718844s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:00:07.009975595 +0000 UTC m=+1149.180978697" watchObservedRunningTime="2026-11-03 01:00:07.013718844 +0000 UTC m=+1149.184721944" Nov 03 01:00:07.022416 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:07.021411 4421 generic.go:356] "Generic (PLEG): container finished" podID="569546b1-7662-4df6-a849-23fb4095797b" containerID="ca926a9c32f02b3235f64fcd8bad103a2c5bef74fb7104c863ea4c418e64b0c9" exitCode=60 Nov 03 01:00:07.022416 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:07.021455 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894460-btsdn" event={"ID":"569546b1-7662-4df6-a849-23fb4095797b","Type":"ContainerDied","Data":"ca926a9c32f02b3235f64fcd8bad103a2c5bef74fb7104c863ea4c418e64b0c9"} Nov 03 01:00:07.022416 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:07.021535 4421 scope.go:122] "RemoveContainer" containerID="0cece1d6b8f111c22cddfd890db27f1b2013525919f3c85ab6e7eed23fc18224" Nov 03 01:00:07.022416 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:07.021913 4421 scope.go:122] "RemoveContainer" containerID="ca926a9c32f02b3235f64fcd8bad103a2c5bef74fb7104c863ea4c418e64b0c9" Nov 03 01:00:07.022416 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:00:07.022376 4421 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894460-btsdn_odh-ai-gateway-infra(569546b1-7662-4df6-a849-23fb4095797b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894460-btsdn" podUID="569546b1-7662-4df6-a849-23fb4095797b" Nov 03 01:00:07.025724 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:07.025701 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f7680d-29894460-qpglw" event={"ID":"600dc1bf-8abc-480d-8503-3942ecd9d48b","Type":"ContainerStarted","Data":"0dcc01371515db5627d6dfbb1004a6e291d6628645f068c7265993f534e4bd2d"} Nov 03 01:00:07.028183 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:07.028159 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-7406a9-29894460-vnwfw" event={"ID":"91abc456-0b19-4c43-9a94-57ef27f96c4a","Type":"ContainerStarted","Data":"7607543d27c817ec8133397877f65bdd5455bea0e6157724455240a1db9d1197"} Nov 03 01:00:07.071040 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:07.070964 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-83755a-29894460-7z6xz" podStartSLOduration=7.07094559 podStartE2EDuration="7.07094559s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:00:07.066058547 +0000 UTC m=+1149.237061644" watchObservedRunningTime="2026-11-03 01:00:07.07094559 +0000 UTC m=+1149.241948684" Nov 03 01:00:07.113534 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:07.113459 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f88f87-29894460-p26bw" podStartSLOduration=7.113433758 podStartE2EDuration="7.113433758s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:00:07.099997748 +0000 UTC m=+1149.271000845" watchObservedRunningTime="2026-11-03 01:00:07.113433758 +0000 UTC m=+1149.284436855" Nov 03 01:00:07.985051 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:07.984976 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d930dca-a318-4fb3-9a22-2a0dff4de5b4" path="/var/lib/kubelet/pods/7d930dca-a318-4fb3-9a22-2a0dff4de5b4/volumes" Nov 03 01:00:08.056308 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:08.056188 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64f6987b-jwc7c"] Nov 03 01:00:08.066457 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:08.066409 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-45019300-xjtwx"] Nov 03 01:00:08.076130 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:08.076091 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64f6987b-jwc7c"] Nov 03 01:00:08.083807 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:08.083117 4421 generic.go:356] "Generic (PLEG): container finished" podID="84fca73a-b1e2-4fee-892f-b2f162bd75b3" containerID="25edee7d461b7aeda9e0548939876d994839442bd21015cd8406839acf561e9e" exitCode=60 Nov 03 01:00:08.083807 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:08.083246 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f9a3a-29894460-sj5kk" event={"ID":"84fca73a-b1e2-4fee-892f-b2f162bd75b3","Type":"ContainerDied","Data":"25edee7d461b7aeda9e0548939876d994839442bd21015cd8406839acf561e9e"} Nov 03 01:00:08.083807 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:08.083329 4421 scope.go:122] "RemoveContainer" containerID="6647a921578d2a84a13ef6b4ba8e63414d05409772038e083868e9947c55b906" Nov 03 01:00:08.084148 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:08.083997 4421 scope.go:122] "RemoveContainer" containerID="25edee7d461b7aeda9e0548939876d994839442bd21015cd8406839acf561e9e" Nov 03 01:00:08.084509 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:00:08.084480 4421 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-2f9a3a-29894460-sj5kk_odh-ai-gateway-infra(84fca73a-b1e2-4fee-892f-b2f162bd75b3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f9a3a-29894460-sj5kk" podUID="84fca73a-b1e2-4fee-892f-b2f162bd75b3" Nov 03 01:00:08.116477 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:08.092150 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-45019300-xjtwx"] Nov 03 01:00:08.119992 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:08.118050 4421 generic.go:356] "Generic (PLEG): container finished" podID="0370c2db-8545-4266-a4f4-719aebbd925b" containerID="13bb8e2dbd9cd36c11e590c2ac62f3a5c5974b9dfdc09cd37fe871f66245f48e" exitCode=60 Nov 03 01:00:08.119992 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:08.118168 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-83755a-29894460-7z6xz" event={"ID":"0370c2db-8545-4266-a4f4-719aebbd925b","Type":"ContainerDied","Data":"13bb8e2dbd9cd36c11e590c2ac62f3a5c5974b9dfdc09cd37fe871f66245f48e"} Nov 03 01:00:08.119992 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:08.118716 4421 scope.go:122] "RemoveContainer" containerID="13bb8e2dbd9cd36c11e590c2ac62f3a5c5974b9dfdc09cd37fe871f66245f48e" Nov 03 01:00:08.119992 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:00:08.119162 4421 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-83755a-29894460-7z6xz_odh-ai-gateway-infra(0370c2db-8545-4266-a4f4-719aebbd925b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-83755a-29894460-7z6xz" podUID="0370c2db-8545-4266-a4f4-719aebbd925b" Nov 03 01:00:08.133846 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:08.133790 4421 scope.go:122] "RemoveContainer" containerID="16fcad589126ee525216b2fe49ca62fdb7f944bc86520d28684818a20ab201ed" Nov 03 01:00:08.134710 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:08.134679 4421 generic.go:356] "Generic (PLEG): container finished" podID="c8c543e5-5eda-4993-92af-8da6a226cfa6" containerID="428bbf3fffdde8124c0eabda8092609f044c249f9e48fcdd6389a9251f531e85" exitCode=60 Nov 03 01:00:08.134844 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:08.134781 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-db5422-29894460-w4ksr" event={"ID":"c8c543e5-5eda-4993-92af-8da6a226cfa6","Type":"ContainerDied","Data":"428bbf3fffdde8124c0eabda8092609f044c249f9e48fcdd6389a9251f531e85"} Nov 03 01:00:08.135352 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:08.135210 4421 scope.go:122] "RemoveContainer" containerID="428bbf3fffdde8124c0eabda8092609f044c249f9e48fcdd6389a9251f531e85" Nov 03 01:00:08.136100 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:00:08.135705 4421 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-db5422-29894460-w4ksr_odh-ai-gateway-infra(c8c543e5-5eda-4993-92af-8da6a226cfa6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-db5422-29894460-w4ksr" podUID="c8c543e5-5eda-4993-92af-8da6a226cfa6" Nov 03 01:00:08.138517 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:08.138481 4421 scope.go:122] "RemoveContainer" containerID="ca926a9c32f02b3235f64fcd8bad103a2c5bef74fb7104c863ea4c418e64b0c9" Nov 03 01:00:08.138931 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:00:08.138903 4421 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894460-btsdn_odh-ai-gateway-infra(569546b1-7662-4df6-a849-23fb4095797b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894460-btsdn" podUID="569546b1-7662-4df6-a849-23fb4095797b" Nov 03 01:00:08.141439 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:08.141409 4421 generic.go:356] "Generic (PLEG): container finished" podID="600dc1bf-8abc-480d-8503-3942ecd9d48b" containerID="0dcc01371515db5627d6dfbb1004a6e291d6628645f068c7265993f534e4bd2d" exitCode=60 Nov 03 01:00:08.141576 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:08.141498 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f7680d-29894460-qpglw" event={"ID":"600dc1bf-8abc-480d-8503-3942ecd9d48b","Type":"ContainerDied","Data":"0dcc01371515db5627d6dfbb1004a6e291d6628645f068c7265993f534e4bd2d"} Nov 03 01:00:08.141996 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:08.141975 4421 scope.go:122] "RemoveContainer" containerID="0dcc01371515db5627d6dfbb1004a6e291d6628645f068c7265993f534e4bd2d" Nov 03 01:00:08.142427 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:00:08.142402 4421 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-f7680d-29894460-qpglw_odh-ai-gateway-infra(600dc1bf-8abc-480d-8503-3942ecd9d48b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f7680d-29894460-qpglw" podUID="600dc1bf-8abc-480d-8503-3942ecd9d48b" Nov 03 01:00:08.145894 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:08.145869 4421 generic.go:356] "Generic (PLEG): container finished" podID="91abc456-0b19-4c43-9a94-57ef27f96c4a" containerID="7607543d27c817ec8133397877f65bdd5455bea0e6157724455240a1db9d1197" exitCode=60 Nov 03 01:00:08.146021 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:08.145954 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-7406a9-29894460-vnwfw" event={"ID":"91abc456-0b19-4c43-9a94-57ef27f96c4a","Type":"ContainerDied","Data":"7607543d27c817ec8133397877f65bdd5455bea0e6157724455240a1db9d1197"} Nov 03 01:00:08.146529 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:08.146510 4421 scope.go:122] "RemoveContainer" containerID="7607543d27c817ec8133397877f65bdd5455bea0e6157724455240a1db9d1197" Nov 03 01:00:08.146986 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:00:08.146938 4421 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-7406a9-29894460-vnwfw_odh-ai-gateway-infra(91abc456-0b19-4c43-9a94-57ef27f96c4a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-7406a9-29894460-vnwfw" podUID="91abc456-0b19-4c43-9a94-57ef27f96c4a" Nov 03 01:00:08.149206 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:08.149180 4421 generic.go:356] "Generic (PLEG): container finished" podID="9bb1557e-955c-41d1-8513-e1f8c4f341f3" containerID="6b649f0f038f254fe4910bc23e1ad3d5bd9d8442e94506cf305db76654a0bde8" exitCode=60 Nov 03 01:00:08.150313 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:08.149513 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-07f5f6-29894460-5lb9h" event={"ID":"9bb1557e-955c-41d1-8513-e1f8c4f341f3","Type":"ContainerDied","Data":"6b649f0f038f254fe4910bc23e1ad3d5bd9d8442e94506cf305db76654a0bde8"} Nov 03 01:00:08.150313 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:08.149669 4421 scope.go:122] "RemoveContainer" containerID="6b649f0f038f254fe4910bc23e1ad3d5bd9d8442e94506cf305db76654a0bde8" Nov 03 01:00:08.150313 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:00:08.149938 4421 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-07f5f6-29894460-5lb9h_odh-ai-gateway-infra(9bb1557e-955c-41d1-8513-e1f8c4f341f3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-07f5f6-29894460-5lb9h" podUID="9bb1557e-955c-41d1-8513-e1f8c4f341f3" Nov 03 01:00:08.164170 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:08.163510 4421 scope.go:122] "RemoveContainer" containerID="3007c3d9d74b1e76d96e4c3eda4885c99d21843e140f6de9b57987d5b441d81f" Nov 03 01:00:08.170158 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:08.170114 4421 generic.go:356] "Generic (PLEG): container finished" podID="03c838c5-5695-434a-b78f-a338c14f3365" containerID="2d9afba2766dae0563b1322cbe1ebb8173a6d2289ce6f20e733f5a65a6d2fd26" exitCode=60 Nov 03 01:00:08.170453 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:08.170427 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f88f87-29894460-p26bw" event={"ID":"03c838c5-5695-434a-b78f-a338c14f3365","Type":"ContainerDied","Data":"2d9afba2766dae0563b1322cbe1ebb8173a6d2289ce6f20e733f5a65a6d2fd26"} Nov 03 01:00:08.170660 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:08.170640 4421 scope.go:122] "RemoveContainer" containerID="2d9afba2766dae0563b1322cbe1ebb8173a6d2289ce6f20e733f5a65a6d2fd26" Nov 03 01:00:08.171176 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:00:08.171110 4421 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-f88f87-29894460-p26bw_odh-ai-gateway-infra(03c838c5-5695-434a-b78f-a338c14f3365)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f88f87-29894460-p26bw" podUID="03c838c5-5695-434a-b78f-a338c14f3365" Nov 03 01:00:08.192949 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:08.192923 4421 scope.go:122] "RemoveContainer" containerID="b48ff27a49985a542b09994a61270210fb715f54a69d6033cde173079b81715a" Nov 03 01:00:08.214707 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:08.214672 4421 scope.go:122] "RemoveContainer" containerID="d5b99aaa518248bb6c7224d70faf1d845bc97bf228da53bb2b75a61973e3fe9f" Nov 03 01:00:08.236073 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:08.235963 4421 scope.go:122] "RemoveContainer" containerID="b75663971452088072a6e9f4ea4521199c213c2b1ecdf02dda78b84fc7ffae64" Nov 03 01:00:08.260369 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:08.260209 4421 scope.go:122] "RemoveContainer" containerID="d79a7d4a6643c83471dd0aec12616d1d5fe428374706fca4b23ec6c0e01f9260" Nov 03 01:00:09.027631 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:09.027596 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3dc652a0-mzw8v"] Nov 03 01:00:09.032831 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:09.032780 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3dc652a0-mzw8v"] Nov 03 01:00:09.182701 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:09.182663 4421 scope.go:122] "RemoveContainer" containerID="0dcc01371515db5627d6dfbb1004a6e291d6628645f068c7265993f534e4bd2d" Nov 03 01:00:09.183174 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:00:09.182986 4421 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-f7680d-29894460-qpglw_odh-ai-gateway-infra(600dc1bf-8abc-480d-8503-3942ecd9d48b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f7680d-29894460-qpglw" podUID="600dc1bf-8abc-480d-8503-3942ecd9d48b" Nov 03 01:00:09.184632 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:09.184612 4421 scope.go:122] "RemoveContainer" containerID="7607543d27c817ec8133397877f65bdd5455bea0e6157724455240a1db9d1197" Nov 03 01:00:09.184934 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:00:09.184911 4421 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-7406a9-29894460-vnwfw_odh-ai-gateway-infra(91abc456-0b19-4c43-9a94-57ef27f96c4a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-7406a9-29894460-vnwfw" podUID="91abc456-0b19-4c43-9a94-57ef27f96c4a" Nov 03 01:00:09.186444 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:09.186423 4421 scope.go:122] "RemoveContainer" containerID="6b649f0f038f254fe4910bc23e1ad3d5bd9d8442e94506cf305db76654a0bde8" Nov 03 01:00:09.186748 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:00:09.186726 4421 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-07f5f6-29894460-5lb9h_odh-ai-gateway-infra(9bb1557e-955c-41d1-8513-e1f8c4f341f3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-07f5f6-29894460-5lb9h" podUID="9bb1557e-955c-41d1-8513-e1f8c4f341f3" Nov 03 01:00:09.188311 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:09.188291 4421 scope.go:122] "RemoveContainer" containerID="2d9afba2766dae0563b1322cbe1ebb8173a6d2289ce6f20e733f5a65a6d2fd26" Nov 03 01:00:09.188548 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:00:09.188524 4421 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-f88f87-29894460-p26bw_odh-ai-gateway-infra(03c838c5-5695-434a-b78f-a338c14f3365)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f88f87-29894460-p26bw" podUID="03c838c5-5695-434a-b78f-a338c14f3365" Nov 03 01:00:09.190317 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:09.190254 4421 scope.go:122] "RemoveContainer" containerID="25edee7d461b7aeda9e0548939876d994839442bd21015cd8406839acf561e9e" Nov 03 01:00:09.190540 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:00:09.190520 4421 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-2f9a3a-29894460-sj5kk_odh-ai-gateway-infra(84fca73a-b1e2-4fee-892f-b2f162bd75b3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f9a3a-29894460-sj5kk" podUID="84fca73a-b1e2-4fee-892f-b2f162bd75b3" Nov 03 01:00:09.192118 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:09.192101 4421 scope.go:122] "RemoveContainer" containerID="13bb8e2dbd9cd36c11e590c2ac62f3a5c5974b9dfdc09cd37fe871f66245f48e" Nov 03 01:00:09.192370 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:00:09.192348 4421 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-83755a-29894460-7z6xz_odh-ai-gateway-infra(0370c2db-8545-4266-a4f4-719aebbd925b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-83755a-29894460-7z6xz" podUID="0370c2db-8545-4266-a4f4-719aebbd925b" Nov 03 01:00:09.193955 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:09.193938 4421 scope.go:122] "RemoveContainer" containerID="428bbf3fffdde8124c0eabda8092609f044c249f9e48fcdd6389a9251f531e85" Nov 03 01:00:09.194197 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:00:09.194179 4421 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-db5422-29894460-w4ksr_odh-ai-gateway-infra(c8c543e5-5eda-4993-92af-8da6a226cfa6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-db5422-29894460-w4ksr" podUID="c8c543e5-5eda-4993-92af-8da6a226cfa6" Nov 03 01:00:09.983184 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:09.983148 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8cef01a5-6822-48dd-93b7-77f8d58f6c1b" path="/var/lib/kubelet/pods/8cef01a5-6822-48dd-93b7-77f8d58f6c1b/volumes" Nov 03 01:00:09.983724 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:09.983709 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e4f1a8e-7ab7-45e3-8057-29a447b79a28" path="/var/lib/kubelet/pods/8e4f1a8e-7ab7-45e3-8057-29a447b79a28/volumes" Nov 03 01:00:09.984143 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:09.984130 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9af830e-c054-4944-91d1-0b96e9a90f7c" path="/var/lib/kubelet/pods/d9af830e-c054-4944-91d1-0b96e9a90f7c/volumes" Nov 03 01:00:10.458337 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:10.458296 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-def754-555d7884c4-k8sh9"] Nov 03 01:00:10.459114 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:10.459097 4421 cpu_manager.go:414] "containerMap: removing container" podUID="c0fb5e39-8b3d-4bc4-8791-01042c8ab304" containerName="payload-pre-processing" Nov 03 01:00:10.459159 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:10.459115 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0fb5e39-8b3d-4bc4-8791-01042c8ab304" containerName="payload-pre-processing" Nov 03 01:00:10.459159 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:10.459135 4421 cpu_manager.go:414] "containerMap: removing container" podUID="7d930dca-a318-4fb3-9a22-2a0dff4de5b4" containerName="cleanup" Nov 03 01:00:10.459159 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:10.459140 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d930dca-a318-4fb3-9a22-2a0dff4de5b4" containerName="cleanup" Nov 03 01:00:10.459159 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:10.459156 4421 cpu_manager.go:414] "containerMap: removing container" podUID="1423f3de-68b7-430d-9c0d-9e957468fedf" containerName="payload-processing" Nov 03 01:00:10.459329 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:10.459163 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1423f3de-68b7-430d-9c0d-9e957468fedf" containerName="payload-processing" Nov 03 01:00:10.459329 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:10.459322 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d930dca-a318-4fb3-9a22-2a0dff4de5b4" containerName="cleanup" Nov 03 01:00:10.459404 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:10.459332 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="1423f3de-68b7-430d-9c0d-9e957468fedf" containerName="payload-processing" Nov 03 01:00:10.459404 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:10.459348 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0fb5e39-8b3d-4bc4-8791-01042c8ab304" containerName="payload-pre-processing" Nov 03 01:00:10.502046 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:10.501997 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-def754-555d7884c4-k8sh9"] Nov 03 01:00:10.502252 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:10.502125 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-def754-555d7884c4-k8sh9" Nov 03 01:00:10.511455 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:10.510935 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df69fbbc-b8cb-4681-9f8d-c3c6f788cde5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-def754-555d7884c4-k8sh9\" (UID: \"df69fbbc-b8cb-4681-9f8d-c3c6f788cde5\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-def754-555d7884c4-k8sh9" Nov 03 01:00:10.511455 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:10.511017 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfn9q\" (UniqueName: \"kubernetes.io/projected/df69fbbc-b8cb-4681-9f8d-c3c6f788cde5-kube-api-access-pfn9q\") pod \"payload-processing-e2e-ipp-b-def754-555d7884c4-k8sh9\" (UID: \"df69fbbc-b8cb-4681-9f8d-c3c6f788cde5\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-def754-555d7884c4-k8sh9" Nov 03 01:00:10.541198 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:10.541168 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-def754-dockercfg-nhcmv\"" Nov 03 01:00:10.543352 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:10.543324 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-def754\"" Nov 03 01:00:10.772916 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:10.759558 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df69fbbc-b8cb-4681-9f8d-c3c6f788cde5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-def754-555d7884c4-k8sh9\" (UID: \"df69fbbc-b8cb-4681-9f8d-c3c6f788cde5\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-def754-555d7884c4-k8sh9" Nov 03 01:00:10.772916 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:10.759639 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfn9q\" (UniqueName: \"kubernetes.io/projected/df69fbbc-b8cb-4681-9f8d-c3c6f788cde5-kube-api-access-pfn9q\") pod \"payload-processing-e2e-ipp-b-def754-555d7884c4-k8sh9\" (UID: \"df69fbbc-b8cb-4681-9f8d-c3c6f788cde5\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-def754-555d7884c4-k8sh9" Nov 03 01:00:10.772916 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:10.761182 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df69fbbc-b8cb-4681-9f8d-c3c6f788cde5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-def754-555d7884c4-k8sh9\" (UID: \"df69fbbc-b8cb-4681-9f8d-c3c6f788cde5\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-def754-555d7884c4-k8sh9" Nov 03 01:00:10.779749 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:10.779606 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-def754-744dbd79f-dm4jt"] Nov 03 01:00:10.802146 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:10.802101 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-def754-744dbd79f-dm4jt"] Nov 03 01:00:10.802555 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:10.802234 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-def754-744dbd79f-dm4jt" Nov 03 01:00:10.818246 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:10.818209 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfn9q\" (UniqueName: \"kubernetes.io/projected/df69fbbc-b8cb-4681-9f8d-c3c6f788cde5-kube-api-access-pfn9q\") pod \"payload-processing-e2e-ipp-b-def754-555d7884c4-k8sh9\" (UID: \"df69fbbc-b8cb-4681-9f8d-c3c6f788cde5\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-def754-555d7884c4-k8sh9" Nov 03 01:00:10.849727 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:10.846715 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-def754-555d7884c4-k8sh9" Nov 03 01:00:10.862501 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:10.861844 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/048aae80-64d2-41b2-9e6a-6b6431d41bd9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-def754-744dbd79f-dm4jt\" (UID: \"048aae80-64d2-41b2-9e6a-6b6431d41bd9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-def754-744dbd79f-dm4jt" Nov 03 01:00:10.862501 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:10.862012 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4q58\" (UniqueName: \"kubernetes.io/projected/048aae80-64d2-41b2-9e6a-6b6431d41bd9-kube-api-access-f4q58\") pod \"payload-pre-processing-e2e-ipp-b-def754-744dbd79f-dm4jt\" (UID: \"048aae80-64d2-41b2-9e6a-6b6431d41bd9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-def754-744dbd79f-dm4jt" Nov 03 01:00:10.964132 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:10.964083 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/048aae80-64d2-41b2-9e6a-6b6431d41bd9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-def754-744dbd79f-dm4jt\" (UID: \"048aae80-64d2-41b2-9e6a-6b6431d41bd9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-def754-744dbd79f-dm4jt" Nov 03 01:00:10.964363 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:10.964156 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4q58\" (UniqueName: \"kubernetes.io/projected/048aae80-64d2-41b2-9e6a-6b6431d41bd9-kube-api-access-f4q58\") pod \"payload-pre-processing-e2e-ipp-b-def754-744dbd79f-dm4jt\" (UID: \"048aae80-64d2-41b2-9e6a-6b6431d41bd9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-def754-744dbd79f-dm4jt" Nov 03 01:00:10.964875 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:10.964853 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/048aae80-64d2-41b2-9e6a-6b6431d41bd9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-def754-744dbd79f-dm4jt\" (UID: \"048aae80-64d2-41b2-9e6a-6b6431d41bd9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-def754-744dbd79f-dm4jt" Nov 03 01:00:10.972814 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:10.972790 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4q58\" (UniqueName: \"kubernetes.io/projected/048aae80-64d2-41b2-9e6a-6b6431d41bd9-kube-api-access-f4q58\") pod \"payload-pre-processing-e2e-ipp-b-def754-744dbd79f-dm4jt\" (UID: \"048aae80-64d2-41b2-9e6a-6b6431d41bd9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-def754-744dbd79f-dm4jt" Nov 03 01:00:11.255199 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:11.253220 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-def754-744dbd79f-dm4jt" Nov 03 01:00:11.325747 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:11.325037 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-def754-555d7884c4-k8sh9"] Nov 03 01:00:11.626847 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:11.626802 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-def754-744dbd79f-dm4jt"] Nov 03 01:00:12.241483 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:12.241430 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-def754-555d7884c4-k8sh9" event={"ID":"df69fbbc-b8cb-4681-9f8d-c3c6f788cde5","Type":"ContainerStarted","Data":"b623172352dd7a8ba50470b32d857820acbeaa047f75d8bda1a59f8568998f57"} Nov 03 01:00:12.241763 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:12.241513 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-def754-555d7884c4-k8sh9" event={"ID":"df69fbbc-b8cb-4681-9f8d-c3c6f788cde5","Type":"ContainerStarted","Data":"39351cc089b7c575ccd0ce235c75ad2ce17b8e144fc436764901cec7d728769f"} Nov 03 01:00:12.242861 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:12.242837 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-def754-555d7884c4-k8sh9" Nov 03 01:00:12.245585 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:12.245554 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-def754-744dbd79f-dm4jt" event={"ID":"048aae80-64d2-41b2-9e6a-6b6431d41bd9","Type":"ContainerStarted","Data":"f59c1c4d16212ada56120d62dd80a5c5b3b8c2ad566f46a4bbcea5f129430786"} Nov 03 01:00:12.245713 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:12.245612 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-def754-744dbd79f-dm4jt" event={"ID":"048aae80-64d2-41b2-9e6a-6b6431d41bd9","Type":"ContainerStarted","Data":"5df08e1904e03615077d2cf9f0db0432bb18abf2a078fe92eeb7cce78aa43dc6"} Nov 03 01:00:12.246618 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:12.246591 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-def754-744dbd79f-dm4jt" Nov 03 01:00:12.273663 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:12.273604 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-def754-555d7884c4-k8sh9" podStartSLOduration=2.273588111 podStartE2EDuration="2.273588111s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:00:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:00:12.269484369 +0000 UTC m=+1154.440487467" watchObservedRunningTime="2026-11-03 01:00:12.273588111 +0000 UTC m=+1154.444591206" Nov 03 01:00:12.292659 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:12.292385 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-def754-744dbd79f-dm4jt" podStartSLOduration=2.292365651 podStartE2EDuration="2.292365651s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:00:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:00:12.290288708 +0000 UTC m=+1154.461291810" watchObservedRunningTime="2026-11-03 01:00:12.292365651 +0000 UTC m=+1154.463368744" Nov 03 01:00:12.357763 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:12.342577 4421 patch_prober.go:29] interesting pod/e2e-ipp-b-db5422-openshift-default-696cf4dc99-75p47 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 01:00:12.357763 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:12.342654 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-db5422-openshift-default-696cf4dc99-75p47" podUID="33f4acce-9a60-4f9c-811d-81e8abfcabe8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 01:00:12.803782 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:12.803439 4421 patch_prober.go:29] interesting pod/e2e-ipp-a-f88f87-openshift-default-7b6c9d6885-g92w2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 01:00:12.803782 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:12.803503 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-f88f87-openshift-default-7b6c9d6885-g92w2" podUID="e7200ec5-9b07-4003-9652-a92c5a1ab5de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 01:00:13.267378 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:13.267330 4421 generic.go:356] "Generic (PLEG): container finished" podID="e7200ec5-9b07-4003-9652-a92c5a1ab5de" containerID="9e94379bd48d6349e7e47ddc3391920faa9dfe975bd5118e072fb0fd1d235b6a" exitCode=0 Nov 03 01:00:13.336782 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:13.267412 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f88f87-openshift-default-7b6c9d6885-g92w2" event={"ID":"e7200ec5-9b07-4003-9652-a92c5a1ab5de","Type":"ContainerDied","Data":"9e94379bd48d6349e7e47ddc3391920faa9dfe975bd5118e072fb0fd1d235b6a"} Nov 03 01:00:13.336782 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:13.268233 4421 scope.go:122] "RemoveContainer" containerID="9e94379bd48d6349e7e47ddc3391920faa9dfe975bd5118e072fb0fd1d235b6a" Nov 03 01:00:13.336782 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:13.270057 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:00:13.336782 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:13.270125 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:00:13.336782 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:13.270157 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:00:14.384509 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:14.384470 4421 generic.go:356] "Generic (PLEG): container finished" podID="33f4acce-9a60-4f9c-811d-81e8abfcabe8" containerID="88750a243ff2f3dd38bac3810635e35888160a711beb60db021fceed8d99aeb1" exitCode=0 Nov 03 01:00:14.385249 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:14.385219 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-db5422-openshift-default-696cf4dc99-75p47" event={"ID":"33f4acce-9a60-4f9c-811d-81e8abfcabe8","Type":"ContainerDied","Data":"88750a243ff2f3dd38bac3810635e35888160a711beb60db021fceed8d99aeb1"} Nov 03 01:00:14.389087 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:14.389055 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f88f87-openshift-default-7b6c9d6885-g92w2" event={"ID":"e7200ec5-9b07-4003-9652-a92c5a1ab5de","Type":"ContainerStarted","Data":"b1011411c1aa9b70e8cfa688e51cb1e3f6c168a17dd72271a89cb0f90f7a38e7"} Nov 03 01:00:14.389530 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:14.389507 4421 scope.go:122] "RemoveContainer" containerID="88750a243ff2f3dd38bac3810635e35888160a711beb60db021fceed8d99aeb1" Nov 03 01:00:14.391876 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:14.391833 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:00:14.391959 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:14.391935 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:00:14.392000 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:14.391984 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:00:14.870295 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:14.859840 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-f88f87-openshift-default-7b6c9d6885-g92w2" Nov 03 01:00:14.870295 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:14.859973 4421 patch_prober.go:29] interesting pod/e2e-ipp-a-f88f87-openshift-default-7b6c9d6885-g92w2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body= Nov 03 01:00:14.870295 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:14.860019 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f88f87-openshift-default-7b6c9d6885-g92w2" podUID="e7200ec5-9b07-4003-9652-a92c5a1ab5de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" Nov 03 01:00:14.996368 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:00:14.988023 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 01:00:15.073602 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:15.073384 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd1c415e-62vfr"] Nov 03 01:00:15.081225 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:15.081188 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd1c415e-62vfr"] Nov 03 01:00:15.416360 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:15.415344 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-db5422-openshift-default-696cf4dc99-75p47" event={"ID":"33f4acce-9a60-4f9c-811d-81e8abfcabe8","Type":"ContainerStarted","Data":"d2a15723d7b6701815942f97a5762ea7f6fa989487c86467b5ff61c932816e5e"} Nov 03 01:00:15.775755 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:15.775721 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-db5422-openshift-default-696cf4dc99-75p47" Nov 03 01:00:15.776023 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:15.775996 4421 patch_prober.go:29] interesting pod/e2e-ipp-b-db5422-openshift-default-696cf4dc99-75p47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body= Nov 03 01:00:15.776228 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:15.776201 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-db5422-openshift-default-696cf4dc99-75p47" podUID="33f4acce-9a60-4f9c-811d-81e8abfcabe8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" Nov 03 01:00:15.866748 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:15.864186 4421 patch_prober.go:29] interesting pod/e2e-ipp-a-f88f87-openshift-default-7b6c9d6885-g92w2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body= Nov 03 01:00:15.869183 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:15.869144 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f88f87-openshift-default-7b6c9d6885-g92w2" podUID="e7200ec5-9b07-4003-9652-a92c5a1ab5de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" Nov 03 01:00:15.970633 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:15.970424 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-db5422-7f6b57dbd7-b5nvk" Nov 03 01:00:15.970633 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:15.970516 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-db5422-8464b57b74-pgngf" Nov 03 01:00:15.984517 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:15.984488 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45eb170b-568c-4bf2-a54b-2132272783ea" path="/var/lib/kubelet/pods/45eb170b-568c-4bf2-a54b-2132272783ea/volumes" Nov 03 01:00:16.776762 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:16.776726 4421 patch_prober.go:29] interesting pod/e2e-ipp-b-db5422-openshift-default-696cf4dc99-75p47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body= Nov 03 01:00:16.777443 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:16.777409 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-db5422-openshift-default-696cf4dc99-75p47" podUID="33f4acce-9a60-4f9c-811d-81e8abfcabe8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" Nov 03 01:00:16.860989 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:16.860950 4421 patch_prober.go:29] interesting pod/e2e-ipp-a-f88f87-openshift-default-7b6c9d6885-g92w2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body= Nov 03 01:00:16.861299 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:16.861242 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f88f87-openshift-default-7b6c9d6885-g92w2" podUID="e7200ec5-9b07-4003-9652-a92c5a1ab5de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" Nov 03 01:00:16.973299 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:16.973118 4421 scope.go:122] "RemoveContainer" containerID="ca926a9c32f02b3235f64fcd8bad103a2c5bef74fb7104c863ea4c418e64b0c9" Nov 03 01:00:17.813978 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:17.813894 4421 patch_prober.go:29] interesting pod/e2e-ipp-b-db5422-openshift-default-696cf4dc99-75p47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body= Nov 03 01:00:17.814655 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:17.814618 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-db5422-openshift-default-696cf4dc99-75p47" podUID="33f4acce-9a60-4f9c-811d-81e8abfcabe8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" Nov 03 01:00:17.870716 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:17.870678 4421 patch_prober.go:29] interesting pod/e2e-ipp-a-f88f87-openshift-default-7b6c9d6885-g92w2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body= Nov 03 01:00:17.888092 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:17.888051 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f88f87-openshift-default-7b6c9d6885-g92w2" podUID="e7200ec5-9b07-4003-9652-a92c5a1ab5de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" Nov 03 01:00:18.185816 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:18.185712 4421 scope.go:122] "RemoveContainer" containerID="2d9afba2766dae0563b1322cbe1ebb8173a6d2289ce6f20e733f5a65a6d2fd26" Nov 03 01:00:18.187867 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:18.187813 4421 scope.go:122] "RemoveContainer" containerID="7607543d27c817ec8133397877f65bdd5455bea0e6157724455240a1db9d1197" Nov 03 01:00:18.789325 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:18.788864 4421 patch_prober.go:29] interesting pod/e2e-ipp-b-db5422-openshift-default-696cf4dc99-75p47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body= Nov 03 01:00:18.789325 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:18.788938 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-db5422-openshift-default-696cf4dc99-75p47" podUID="33f4acce-9a60-4f9c-811d-81e8abfcabe8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" Nov 03 01:00:18.888340 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:18.884474 4421 patch_prober.go:29] interesting pod/e2e-ipp-a-f88f87-openshift-default-7b6c9d6885-g92w2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body= Nov 03 01:00:18.888340 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:18.884536 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f88f87-openshift-default-7b6c9d6885-g92w2" podUID="e7200ec5-9b07-4003-9652-a92c5a1ab5de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" Nov 03 01:00:18.891561 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:18.891533 4421 generic.go:356] "Generic (PLEG): container finished" podID="569546b1-7662-4df6-a849-23fb4095797b" containerID="9361847a7a8ac89f798f02aa6ecb6f6c3456c4c7803ab051cafcbc9f38fcbec3" exitCode=60 Nov 03 01:00:18.891687 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:18.891588 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894460-btsdn" event={"ID":"569546b1-7662-4df6-a849-23fb4095797b","Type":"ContainerDied","Data":"9361847a7a8ac89f798f02aa6ecb6f6c3456c4c7803ab051cafcbc9f38fcbec3"} Nov 03 01:00:18.891687 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:18.891659 4421 scope.go:122] "RemoveContainer" containerID="ca926a9c32f02b3235f64fcd8bad103a2c5bef74fb7104c863ea4c418e64b0c9" Nov 03 01:00:18.892043 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:18.892027 4421 scope.go:122] "RemoveContainer" containerID="9361847a7a8ac89f798f02aa6ecb6f6c3456c4c7803ab051cafcbc9f38fcbec3" Nov 03 01:00:18.892356 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:00:18.892338 4421 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29894460-btsdn_odh-ai-gateway-infra(569546b1-7662-4df6-a849-23fb4095797b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894460-btsdn" podUID="569546b1-7662-4df6-a849-23fb4095797b" Nov 03 01:00:18.988355 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:18.983557 4421 scope.go:122] "RemoveContainer" containerID="ad3a057f372b419a664cc22c8b3d1ec3e2915a9b0176fd0b63152a9ce8de178e" Nov 03 01:00:19.058299 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:19.056518 4421 scope.go:122] "RemoveContainer" containerID="428bbf3fffdde8124c0eabda8092609f044c249f9e48fcdd6389a9251f531e85" Nov 03 01:00:19.058299 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:19.057151 4421 scope.go:122] "RemoveContainer" containerID="6b649f0f038f254fe4910bc23e1ad3d5bd9d8442e94506cf305db76654a0bde8" Nov 03 01:00:19.775186 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:19.775156 4421 patch_prober.go:29] interesting pod/e2e-ipp-b-db5422-openshift-default-696cf4dc99-75p47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body= Nov 03 01:00:19.775321 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:19.775209 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-db5422-openshift-default-696cf4dc99-75p47" podUID="33f4acce-9a60-4f9c-811d-81e8abfcabe8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" Nov 03 01:00:19.866375 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:19.862742 4421 patch_prober.go:29] interesting pod/e2e-ipp-a-f88f87-openshift-default-7b6c9d6885-g92w2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body= Nov 03 01:00:19.866375 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:19.862804 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f88f87-openshift-default-7b6c9d6885-g92w2" podUID="e7200ec5-9b07-4003-9652-a92c5a1ab5de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" Nov 03 01:00:19.901999 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:19.901960 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7f43f9-29894460-bfjlr" event={"ID":"64ad274e-6de7-4604-a8e4-958f67dbd00f","Type":"ContainerStarted","Data":"1cbbdc9c739f10dc61de9285170c17ada0e1f868e06330d95746de6c917a962a"} Nov 03 01:00:20.051795 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:20.002682 4421 scope.go:122] "RemoveContainer" containerID="0dcc01371515db5627d6dfbb1004a6e291d6628645f068c7265993f534e4bd2d" Nov 03 01:00:20.051795 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:20.013648 4421 scope.go:122] "RemoveContainer" containerID="25edee7d461b7aeda9e0548939876d994839442bd21015cd8406839acf561e9e" Nov 03 01:00:20.175740 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:20.175707 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-7406a9-29894460-vnwfw" event={"ID":"91abc456-0b19-4c43-9a94-57ef27f96c4a","Type":"ContainerStarted","Data":"f9096cbc3c1c3168c651983ba650a37017723bb36dc63da4ec7d6ec065483b9f"} Nov 03 01:00:20.175909 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:20.175768 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894460-btsdn"] Nov 03 01:00:20.175909 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:20.175810 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-47369d-7c8f86b795-hclwx"] Nov 03 01:00:20.180168 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:20.180131 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-47369d-6fbdd5b9b7-h78wk"] Nov 03 01:00:20.203512 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:20.202253 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f88f87-29894460-p26bw" event={"ID":"03c838c5-5695-434a-b78f-a338c14f3365","Type":"ContainerStarted","Data":"30b2dbfa98e87a46dbccc04ad6429200979af1b3225d305938bd69a03f4fc288"} Nov 03 01:00:20.203512 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:20.202947 4421 scope.go:122] "RemoveContainer" containerID="30b2dbfa98e87a46dbccc04ad6429200979af1b3225d305938bd69a03f4fc288" Nov 03 01:00:20.203512 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:00:20.203432 4421 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-f88f87-29894460-p26bw_odh-ai-gateway-infra(03c838c5-5695-434a-b78f-a338c14f3365)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f88f87-29894460-p26bw" podUID="03c838c5-5695-434a-b78f-a338c14f3365" Nov 03 01:00:20.777332 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:20.776989 4421 patch_prober.go:29] interesting pod/e2e-ipp-b-db5422-openshift-default-696cf4dc99-75p47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body= Nov 03 01:00:20.777332 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:20.777045 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-db5422-openshift-default-696cf4dc99-75p47" podUID="33f4acce-9a60-4f9c-811d-81e8abfcabe8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" Nov 03 01:00:20.897285 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:20.897227 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-f88f87-openshift-default-7b6c9d6885-g92w2" Nov 03 01:00:20.898736 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:20.898666 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-f88f87-openshift-default-7b6c9d6885-g92w2" Nov 03 01:00:20.900235 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:20.900201 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-f88f87-openshift-default-7b6c9d6885-g92w2" Nov 03 01:00:20.955985 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:20.955954 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7f43f9-29894460-bfjlr"] Nov 03 01:00:20.970739 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:20.970708 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894460-btsdn" Nov 03 01:00:20.971688 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:20.971609 4421 scope.go:122] "RemoveContainer" containerID="13bb8e2dbd9cd36c11e590c2ac62f3a5c5974b9dfdc09cd37fe871f66245f48e" Nov 03 01:00:21.050523 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:21.050032 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/569546b1-7662-4df6-a849-23fb4095797b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/569546b1-7662-4df6-a849-23fb4095797b-maas-api-service-ca\") pod \"569546b1-7662-4df6-a849-23fb4095797b\" (UID: \"569546b1-7662-4df6-a849-23fb4095797b\") " Nov 03 01:00:21.050523 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:21.050400 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/569546b1-7662-4df6-a849-23fb4095797b-kube-api-access-k42d5\" (UniqueName: \"kubernetes.io/projected/569546b1-7662-4df6-a849-23fb4095797b-kube-api-access-k42d5\") pod \"569546b1-7662-4df6-a849-23fb4095797b\" (UID: \"569546b1-7662-4df6-a849-23fb4095797b\") " Nov 03 01:00:21.053095 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:21.053055 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/569546b1-7662-4df6-a849-23fb4095797b-maas-api-service-ca" pod "569546b1-7662-4df6-a849-23fb4095797b" (UID: "569546b1-7662-4df6-a849-23fb4095797b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:00:21.107429 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:21.107351 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/569546b1-7662-4df6-a849-23fb4095797b-kube-api-access-k42d5" pod "569546b1-7662-4df6-a849-23fb4095797b" (UID: "569546b1-7662-4df6-a849-23fb4095797b"). InnerVolumeSpecName "kube-api-access-k42d5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:00:21.152450 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:21.152412 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/569546b1-7662-4df6-a849-23fb4095797b-maas-api-service-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:00:21.152450 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:21.152442 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k42d5\" (UniqueName: \"kubernetes.io/projected/569546b1-7662-4df6-a849-23fb4095797b-kube-api-access-k42d5\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:00:21.251943 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:21.251903 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f7680d-29894460-qpglw" event={"ID":"600dc1bf-8abc-480d-8503-3942ecd9d48b","Type":"ContainerStarted","Data":"94d619dd062e625c52ed1534a58cb990722374793da5ba415cc8c4323e5e8ced"} Nov 03 01:00:21.254222 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:21.254194 4421 generic.go:356] "Generic (PLEG): container finished" podID="91abc456-0b19-4c43-9a94-57ef27f96c4a" containerID="f9096cbc3c1c3168c651983ba650a37017723bb36dc63da4ec7d6ec065483b9f" exitCode=60 Nov 03 01:00:21.254387 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:21.254266 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-7406a9-29894460-vnwfw" event={"ID":"91abc456-0b19-4c43-9a94-57ef27f96c4a","Type":"ContainerDied","Data":"f9096cbc3c1c3168c651983ba650a37017723bb36dc63da4ec7d6ec065483b9f"} Nov 03 01:00:21.254387 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:21.254350 4421 scope.go:122] "RemoveContainer" containerID="7607543d27c817ec8133397877f65bdd5455bea0e6157724455240a1db9d1197" Nov 03 01:00:21.254728 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:21.254708 4421 scope.go:122] "RemoveContainer" containerID="f9096cbc3c1c3168c651983ba650a37017723bb36dc63da4ec7d6ec065483b9f" Nov 03 01:00:21.255091 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:00:21.255064 4421 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-7406a9-29894460-vnwfw_odh-ai-gateway-infra(91abc456-0b19-4c43-9a94-57ef27f96c4a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-7406a9-29894460-vnwfw" podUID="91abc456-0b19-4c43-9a94-57ef27f96c4a" Nov 03 01:00:21.256810 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:21.256785 4421 generic.go:356] "Generic (PLEG): container finished" podID="9bb1557e-955c-41d1-8513-e1f8c4f341f3" containerID="fe0c019c296ed4b4848b448592b1ca49a1be0eac4d1bf204d912627ff2f0ad0a" exitCode=60 Nov 03 01:00:21.256914 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:21.256862 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-07f5f6-29894460-5lb9h" event={"ID":"9bb1557e-955c-41d1-8513-e1f8c4f341f3","Type":"ContainerDied","Data":"fe0c019c296ed4b4848b448592b1ca49a1be0eac4d1bf204d912627ff2f0ad0a"} Nov 03 01:00:21.257237 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:21.257221 4421 scope.go:122] "RemoveContainer" containerID="fe0c019c296ed4b4848b448592b1ca49a1be0eac4d1bf204d912627ff2f0ad0a" Nov 03 01:00:21.257634 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:00:21.257615 4421 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-07f5f6-29894460-5lb9h_odh-ai-gateway-infra(9bb1557e-955c-41d1-8513-e1f8c4f341f3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-07f5f6-29894460-5lb9h" podUID="9bb1557e-955c-41d1-8513-e1f8c4f341f3" Nov 03 01:00:21.259153 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:21.259129 4421 generic.go:356] "Generic (PLEG): container finished" podID="03c838c5-5695-434a-b78f-a338c14f3365" containerID="30b2dbfa98e87a46dbccc04ad6429200979af1b3225d305938bd69a03f4fc288" exitCode=60 Nov 03 01:00:21.259245 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:21.259193 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f88f87-29894460-p26bw" event={"ID":"03c838c5-5695-434a-b78f-a338c14f3365","Type":"ContainerDied","Data":"30b2dbfa98e87a46dbccc04ad6429200979af1b3225d305938bd69a03f4fc288"} Nov 03 01:00:21.259579 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:21.259564 4421 scope.go:122] "RemoveContainer" containerID="30b2dbfa98e87a46dbccc04ad6429200979af1b3225d305938bd69a03f4fc288" Nov 03 01:00:21.259879 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:00:21.259856 4421 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-f88f87-29894460-p26bw_odh-ai-gateway-infra(03c838c5-5695-434a-b78f-a338c14f3365)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f88f87-29894460-p26bw" podUID="03c838c5-5695-434a-b78f-a338c14f3365" Nov 03 01:00:21.261934 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:21.261906 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f9a3a-29894460-sj5kk" event={"ID":"84fca73a-b1e2-4fee-892f-b2f162bd75b3","Type":"ContainerStarted","Data":"d74f8ff625e1b767998edb3629f5fb880841fc623d4564104cab029d8a1c3333"} Nov 03 01:00:21.264150 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:21.264128 4421 generic.go:356] "Generic (PLEG): container finished" podID="c8c543e5-5eda-4993-92af-8da6a226cfa6" containerID="4a8c19ab2f102e699bcaa6b70786d05c622f962e4b30ede3510ebe5f1cf92a84" exitCode=60 Nov 03 01:00:21.264296 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:21.264195 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-db5422-29894460-w4ksr" event={"ID":"c8c543e5-5eda-4993-92af-8da6a226cfa6","Type":"ContainerDied","Data":"4a8c19ab2f102e699bcaa6b70786d05c622f962e4b30ede3510ebe5f1cf92a84"} Nov 03 01:00:21.264647 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:21.264618 4421 scope.go:122] "RemoveContainer" containerID="4a8c19ab2f102e699bcaa6b70786d05c622f962e4b30ede3510ebe5f1cf92a84" Nov 03 01:00:21.264952 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:00:21.264932 4421 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-db5422-29894460-w4ksr_odh-ai-gateway-infra(c8c543e5-5eda-4993-92af-8da6a226cfa6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-db5422-29894460-w4ksr" podUID="c8c543e5-5eda-4993-92af-8da6a226cfa6" Nov 03 01:00:21.266435 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:21.266403 4421 generic.go:356] "Generic (PLEG): container finished" podID="64ad274e-6de7-4604-a8e4-958f67dbd00f" containerID="1cbbdc9c739f10dc61de9285170c17ada0e1f868e06330d95746de6c917a962a" exitCode=60 Nov 03 01:00:21.266552 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:21.266482 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7f43f9-29894460-bfjlr" event={"ID":"64ad274e-6de7-4604-a8e4-958f67dbd00f","Type":"ContainerDied","Data":"1cbbdc9c739f10dc61de9285170c17ada0e1f868e06330d95746de6c917a962a"} Nov 03 01:00:21.268261 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:21.268240 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894460-btsdn" Nov 03 01:00:21.268400 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:21.268257 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894460-btsdn" event={"ID":"569546b1-7662-4df6-a849-23fb4095797b","Type":"ContainerDied","Data":"c7873861cf42d761ec3137912ae6379d1a88b0217a5abb6bb89ccb446e87e8a1"} Nov 03 01:00:21.273147 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:21.273125 4421 scope.go:122] "RemoveContainer" containerID="6b649f0f038f254fe4910bc23e1ad3d5bd9d8442e94506cf305db76654a0bde8" Nov 03 01:00:21.296694 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:21.296660 4421 scope.go:122] "RemoveContainer" containerID="2d9afba2766dae0563b1322cbe1ebb8173a6d2289ce6f20e733f5a65a6d2fd26" Nov 03 01:00:21.386219 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:21.386183 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-7406a9-29894460-vnwfw"] Nov 03 01:00:21.403540 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:21.403086 4421 scope.go:122] "RemoveContainer" containerID="428bbf3fffdde8124c0eabda8092609f044c249f9e48fcdd6389a9251f531e85" Nov 03 01:00:21.479325 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:21.479209 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f88f87-29894460-p26bw"] Nov 03 01:00:21.490729 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:21.490591 4421 scope.go:122] "RemoveContainer" containerID="ad3a057f372b419a664cc22c8b3d1ec3e2915a9b0176fd0b63152a9ce8de178e" Nov 03 01:00:21.560187 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:21.560043 4421 scope.go:122] "RemoveContainer" containerID="9361847a7a8ac89f798f02aa6ecb6f6c3456c4c7803ab051cafcbc9f38fcbec3" Nov 03 01:00:21.647930 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:21.647898 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894460-btsdn"] Nov 03 01:00:21.650910 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:21.649400 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7f43f9-29894460-bfjlr" Nov 03 01:00:21.657134 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:21.657093 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894460-btsdn"] Nov 03 01:00:21.777002 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:21.776956 4421 patch_prober.go:29] interesting pod/e2e-ipp-b-db5422-openshift-default-696cf4dc99-75p47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body= Nov 03 01:00:21.777193 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:21.777018 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-db5422-openshift-default-696cf4dc99-75p47" podUID="33f4acce-9a60-4f9c-811d-81e8abfcabe8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" Nov 03 01:00:21.781735 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:21.781697 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64ad274e-6de7-4604-a8e4-958f67dbd00f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64ad274e-6de7-4604-a8e4-958f67dbd00f-maas-api-service-ca\") pod \"64ad274e-6de7-4604-a8e4-958f67dbd00f\" (UID: \"64ad274e-6de7-4604-a8e4-958f67dbd00f\") " Nov 03 01:00:21.781959 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:21.781920 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64ad274e-6de7-4604-a8e4-958f67dbd00f-kube-api-access-n5mxt\" (UniqueName: \"kubernetes.io/projected/64ad274e-6de7-4604-a8e4-958f67dbd00f-kube-api-access-n5mxt\") pod \"64ad274e-6de7-4604-a8e4-958f67dbd00f\" (UID: \"64ad274e-6de7-4604-a8e4-958f67dbd00f\") " Nov 03 01:00:21.782170 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:21.782140 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64ad274e-6de7-4604-a8e4-958f67dbd00f-maas-api-service-ca" pod "64ad274e-6de7-4604-a8e4-958f67dbd00f" (UID: "64ad274e-6de7-4604-a8e4-958f67dbd00f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:00:21.782949 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:21.782930 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64ad274e-6de7-4604-a8e4-958f67dbd00f-maas-api-service-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:00:21.784344 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:21.784320 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64ad274e-6de7-4604-a8e4-958f67dbd00f-kube-api-access-n5mxt" pod "64ad274e-6de7-4604-a8e4-958f67dbd00f" (UID: "64ad274e-6de7-4604-a8e4-958f67dbd00f"). InnerVolumeSpecName "kube-api-access-n5mxt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:00:21.884073 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:21.884031 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5mxt\" (UniqueName: \"kubernetes.io/projected/64ad274e-6de7-4604-a8e4-958f67dbd00f-kube-api-access-n5mxt\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:00:21.988397 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:21.988355 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="569546b1-7662-4df6-a849-23fb4095797b" path="/var/lib/kubelet/pods/569546b1-7662-4df6-a849-23fb4095797b/volumes" Nov 03 01:00:22.286606 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:22.286572 4421 generic.go:356] "Generic (PLEG): container finished" podID="600dc1bf-8abc-480d-8503-3942ecd9d48b" containerID="94d619dd062e625c52ed1534a58cb990722374793da5ba415cc8c4323e5e8ced" exitCode=60 Nov 03 01:00:22.286786 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:22.286671 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f7680d-29894460-qpglw" event={"ID":"600dc1bf-8abc-480d-8503-3942ecd9d48b","Type":"ContainerDied","Data":"94d619dd062e625c52ed1534a58cb990722374793da5ba415cc8c4323e5e8ced"} Nov 03 01:00:22.286786 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:22.286728 4421 scope.go:122] "RemoveContainer" containerID="0dcc01371515db5627d6dfbb1004a6e291d6628645f068c7265993f534e4bd2d" Nov 03 01:00:22.287255 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:22.287222 4421 scope.go:122] "RemoveContainer" containerID="94d619dd062e625c52ed1534a58cb990722374793da5ba415cc8c4323e5e8ced" Nov 03 01:00:22.287731 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:00:22.287701 4421 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-f7680d-29894460-qpglw_odh-ai-gateway-infra(600dc1bf-8abc-480d-8503-3942ecd9d48b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f7680d-29894460-qpglw" podUID="600dc1bf-8abc-480d-8503-3942ecd9d48b" Nov 03 01:00:22.290150 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:22.290125 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f7680d-29894460-qpglw"] Nov 03 01:00:22.325150 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:22.324975 4421 generic.go:356] "Generic (PLEG): container finished" podID="84fca73a-b1e2-4fee-892f-b2f162bd75b3" containerID="d74f8ff625e1b767998edb3629f5fb880841fc623d4564104cab029d8a1c3333" exitCode=60 Nov 03 01:00:22.325387 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:22.325164 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f9a3a-29894460-sj5kk" event={"ID":"84fca73a-b1e2-4fee-892f-b2f162bd75b3","Type":"ContainerDied","Data":"d74f8ff625e1b767998edb3629f5fb880841fc623d4564104cab029d8a1c3333"} Nov 03 01:00:22.325387 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:22.325286 4421 scope.go:122] "RemoveContainer" containerID="25edee7d461b7aeda9e0548939876d994839442bd21015cd8406839acf561e9e" Nov 03 01:00:22.326002 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:22.325958 4421 scope.go:122] "RemoveContainer" containerID="d74f8ff625e1b767998edb3629f5fb880841fc623d4564104cab029d8a1c3333" Nov 03 01:00:22.326760 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:00:22.326714 4421 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-2f9a3a-29894460-sj5kk_odh-ai-gateway-infra(84fca73a-b1e2-4fee-892f-b2f162bd75b3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f9a3a-29894460-sj5kk" podUID="84fca73a-b1e2-4fee-892f-b2f162bd75b3" Nov 03 01:00:22.334927 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:22.334870 4421 generic.go:356] "Generic (PLEG): container finished" podID="0370c2db-8545-4266-a4f4-719aebbd925b" containerID="84c768b3b488413a494675264154acb0aec3a3a7ce9ada255a3368e09079d7dc" exitCode=60 Nov 03 01:00:22.337626 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:22.336837 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-83755a-29894460-7z6xz" event={"ID":"0370c2db-8545-4266-a4f4-719aebbd925b","Type":"ContainerDied","Data":"84c768b3b488413a494675264154acb0aec3a3a7ce9ada255a3368e09079d7dc"} Nov 03 01:00:22.349966 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:22.349918 4421 scope.go:122] "RemoveContainer" containerID="84c768b3b488413a494675264154acb0aec3a3a7ce9ada255a3368e09079d7dc" Nov 03 01:00:22.355089 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:00:22.355050 4421 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-83755a-29894460-7z6xz_odh-ai-gateway-infra(0370c2db-8545-4266-a4f4-719aebbd925b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-83755a-29894460-7z6xz" podUID="0370c2db-8545-4266-a4f4-719aebbd925b" Nov 03 01:00:22.355753 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:22.355729 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-db5422-29894460-w4ksr"] Nov 03 01:00:22.373858 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:22.372640 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7f43f9-29894460-bfjlr" event={"ID":"64ad274e-6de7-4604-a8e4-958f67dbd00f","Type":"ContainerDied","Data":"12b59779be49227646fd96e59f7abeab4105bc0e1079ed57e631e6a41119021c"} Nov 03 01:00:22.373858 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:22.372798 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7f43f9-29894460-bfjlr" Nov 03 01:00:22.417065 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:22.417035 4421 scope.go:122] "RemoveContainer" containerID="13bb8e2dbd9cd36c11e590c2ac62f3a5c5974b9dfdc09cd37fe871f66245f48e" Nov 03 01:00:22.466845 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:22.461840 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7f43f9-29894460-bfjlr"] Nov 03 01:00:22.469972 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:22.469947 4421 scope.go:122] "RemoveContainer" containerID="1cbbdc9c739f10dc61de9285170c17ada0e1f868e06330d95746de6c917a962a" Nov 03 01:00:22.481382 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:22.481347 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7f43f9-29894460-bfjlr"] Nov 03 01:00:22.481603 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:22.481585 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-07f5f6-29894460-5lb9h"] Nov 03 01:00:22.619626 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:22.619596 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f9a3a-29894460-sj5kk"] Nov 03 01:00:22.763698 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:22.763436 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-7406a9-29894460-vnwfw" Nov 03 01:00:22.800963 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:22.800934 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-db5422-openshift-default-696cf4dc99-75p47" Nov 03 01:00:22.801793 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:22.801723 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-db5422-openshift-default-696cf4dc99-75p47" Nov 03 01:00:22.802677 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:22.802623 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-db5422-openshift-default-696cf4dc99-75p47" Nov 03 01:00:22.836719 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:22.836693 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f88f87-29894460-p26bw" Nov 03 01:00:22.875609 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:22.875530 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-db5422-29894460-w4ksr" Nov 03 01:00:22.918525 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:22.917660 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91abc456-0b19-4c43-9a94-57ef27f96c4a-kube-api-access-qssk2\" (UniqueName: \"kubernetes.io/projected/91abc456-0b19-4c43-9a94-57ef27f96c4a-kube-api-access-qssk2\") pod \"91abc456-0b19-4c43-9a94-57ef27f96c4a\" (UID: \"91abc456-0b19-4c43-9a94-57ef27f96c4a\") " Nov 03 01:00:22.918525 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:22.917724 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03c838c5-5695-434a-b78f-a338c14f3365-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/03c838c5-5695-434a-b78f-a338c14f3365-maas-api-service-ca\") pod \"03c838c5-5695-434a-b78f-a338c14f3365\" (UID: \"03c838c5-5695-434a-b78f-a338c14f3365\") " Nov 03 01:00:22.918525 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:22.917820 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03c838c5-5695-434a-b78f-a338c14f3365-kube-api-access-6clrj\" (UniqueName: \"kubernetes.io/projected/03c838c5-5695-434a-b78f-a338c14f3365-kube-api-access-6clrj\") pod \"03c838c5-5695-434a-b78f-a338c14f3365\" (UID: \"03c838c5-5695-434a-b78f-a338c14f3365\") " Nov 03 01:00:22.918525 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:22.917857 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91abc456-0b19-4c43-9a94-57ef27f96c4a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91abc456-0b19-4c43-9a94-57ef27f96c4a-maas-api-service-ca\") pod \"91abc456-0b19-4c43-9a94-57ef27f96c4a\" (UID: \"91abc456-0b19-4c43-9a94-57ef27f96c4a\") " Nov 03 01:00:22.925770 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:22.921700 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91abc456-0b19-4c43-9a94-57ef27f96c4a-maas-api-service-ca" pod "91abc456-0b19-4c43-9a94-57ef27f96c4a" (UID: "91abc456-0b19-4c43-9a94-57ef27f96c4a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:00:22.925770 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:22.922051 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03c838c5-5695-434a-b78f-a338c14f3365-maas-api-service-ca" pod "03c838c5-5695-434a-b78f-a338c14f3365" (UID: "03c838c5-5695-434a-b78f-a338c14f3365"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:00:22.938589 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:22.938234 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91abc456-0b19-4c43-9a94-57ef27f96c4a-kube-api-access-qssk2" pod "91abc456-0b19-4c43-9a94-57ef27f96c4a" (UID: "91abc456-0b19-4c43-9a94-57ef27f96c4a"). InnerVolumeSpecName "kube-api-access-qssk2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:00:22.942517 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:22.942473 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03c838c5-5695-434a-b78f-a338c14f3365-kube-api-access-6clrj" pod "03c838c5-5695-434a-b78f-a338c14f3365" (UID: "03c838c5-5695-434a-b78f-a338c14f3365"). InnerVolumeSpecName "kube-api-access-6clrj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:00:22.976605 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:22.976578 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-07f5f6-29894460-5lb9h" Nov 03 01:00:23.020905 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:23.020419 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8c543e5-5eda-4993-92af-8da6a226cfa6-kube-api-access-cgckj\" (UniqueName: \"kubernetes.io/projected/c8c543e5-5eda-4993-92af-8da6a226cfa6-kube-api-access-cgckj\") pod \"c8c543e5-5eda-4993-92af-8da6a226cfa6\" (UID: \"c8c543e5-5eda-4993-92af-8da6a226cfa6\") " Nov 03 01:00:23.020905 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:23.020646 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8c543e5-5eda-4993-92af-8da6a226cfa6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8c543e5-5eda-4993-92af-8da6a226cfa6-maas-api-service-ca\") pod \"c8c543e5-5eda-4993-92af-8da6a226cfa6\" (UID: \"c8c543e5-5eda-4993-92af-8da6a226cfa6\") " Nov 03 01:00:23.021477 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:23.021025 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8c543e5-5eda-4993-92af-8da6a226cfa6-maas-api-service-ca" pod "c8c543e5-5eda-4993-92af-8da6a226cfa6" (UID: "c8c543e5-5eda-4993-92af-8da6a226cfa6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:00:23.021535 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:23.021516 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8c543e5-5eda-4993-92af-8da6a226cfa6-maas-api-service-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:00:23.021590 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:23.021545 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qssk2\" (UniqueName: \"kubernetes.io/projected/91abc456-0b19-4c43-9a94-57ef27f96c4a-kube-api-access-qssk2\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:00:23.021590 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:23.021563 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/03c838c5-5695-434a-b78f-a338c14f3365-maas-api-service-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:00:23.021590 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:23.021578 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6clrj\" (UniqueName: \"kubernetes.io/projected/03c838c5-5695-434a-b78f-a338c14f3365-kube-api-access-6clrj\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:00:23.021725 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:23.021593 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91abc456-0b19-4c43-9a94-57ef27f96c4a-maas-api-service-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:00:23.023866 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:23.023834 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8c543e5-5eda-4993-92af-8da6a226cfa6-kube-api-access-cgckj" pod "c8c543e5-5eda-4993-92af-8da6a226cfa6" (UID: "c8c543e5-5eda-4993-92af-8da6a226cfa6"). InnerVolumeSpecName "kube-api-access-cgckj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:00:23.123187 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:23.123148 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9bb1557e-955c-41d1-8513-e1f8c4f341f3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9bb1557e-955c-41d1-8513-e1f8c4f341f3-maas-api-service-ca\") pod \"9bb1557e-955c-41d1-8513-e1f8c4f341f3\" (UID: \"9bb1557e-955c-41d1-8513-e1f8c4f341f3\") " Nov 03 01:00:23.123474 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:23.123454 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bb1557e-955c-41d1-8513-e1f8c4f341f3-kube-api-access-nxlwn\" (UniqueName: \"kubernetes.io/projected/9bb1557e-955c-41d1-8513-e1f8c4f341f3-kube-api-access-nxlwn\") pod \"9bb1557e-955c-41d1-8513-e1f8c4f341f3\" (UID: \"9bb1557e-955c-41d1-8513-e1f8c4f341f3\") " Nov 03 01:00:23.123590 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:23.123562 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bb1557e-955c-41d1-8513-e1f8c4f341f3-maas-api-service-ca" pod "9bb1557e-955c-41d1-8513-e1f8c4f341f3" (UID: "9bb1557e-955c-41d1-8513-e1f8c4f341f3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:00:23.126504 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:23.126461 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bb1557e-955c-41d1-8513-e1f8c4f341f3-kube-api-access-nxlwn" pod "9bb1557e-955c-41d1-8513-e1f8c4f341f3" (UID: "9bb1557e-955c-41d1-8513-e1f8c4f341f3"). InnerVolumeSpecName "kube-api-access-nxlwn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:00:23.127368 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:23.127340 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9bb1557e-955c-41d1-8513-e1f8c4f341f3-maas-api-service-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:00:23.127506 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:23.127370 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxlwn\" (UniqueName: \"kubernetes.io/projected/9bb1557e-955c-41d1-8513-e1f8c4f341f3-kube-api-access-nxlwn\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:00:23.127506 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:23.127387 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cgckj\" (UniqueName: \"kubernetes.io/projected/c8c543e5-5eda-4993-92af-8da6a226cfa6-kube-api-access-cgckj\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:00:23.396135 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:23.396045 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-7406a9-29894460-vnwfw" Nov 03 01:00:23.396339 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:23.396132 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-7406a9-29894460-vnwfw" event={"ID":"91abc456-0b19-4c43-9a94-57ef27f96c4a","Type":"ContainerDied","Data":"bbd3f758933261a61f90ac0a6ee789f171195ebde8cbcb9f8334a9b53b506dab"} Nov 03 01:00:23.396339 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:23.396215 4421 scope.go:122] "RemoveContainer" containerID="f9096cbc3c1c3168c651983ba650a37017723bb36dc63da4ec7d6ec065483b9f" Nov 03 01:00:23.410572 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:23.409210 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-07f5f6-29894460-5lb9h" event={"ID":"9bb1557e-955c-41d1-8513-e1f8c4f341f3","Type":"ContainerDied","Data":"60aade7b8f2b1eb66e208d1995130feece3900a54db72e29861d298167e00802"} Nov 03 01:00:23.410572 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:23.409343 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-07f5f6-29894460-5lb9h" Nov 03 01:00:23.413663 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:23.413641 4421 scope.go:122] "RemoveContainer" containerID="fe0c019c296ed4b4848b448592b1ca49a1be0eac4d1bf204d912627ff2f0ad0a" Nov 03 01:00:23.417216 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:23.417188 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f88f87-29894460-p26bw" event={"ID":"03c838c5-5695-434a-b78f-a338c14f3365","Type":"ContainerDied","Data":"2477b35d77103a9b7271529947d029a46d0eb35039caf7fa0b5397be7e3165f2"} Nov 03 01:00:23.417345 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:23.417313 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f88f87-29894460-p26bw" Nov 03 01:00:23.432345 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:23.431786 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-db5422-29894460-w4ksr" event={"ID":"c8c543e5-5eda-4993-92af-8da6a226cfa6","Type":"ContainerDied","Data":"ab0484440b11a8f71f2c3588df044fd2305aaa6914535a4c66d1dbd405761fcf"} Nov 03 01:00:23.432345 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:23.431880 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-db5422-29894460-w4ksr" Nov 03 01:00:23.442911 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:23.442880 4421 scope.go:122] "RemoveContainer" containerID="30b2dbfa98e87a46dbccc04ad6429200979af1b3225d305938bd69a03f4fc288" Nov 03 01:00:23.482449 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:23.482398 4421 scope.go:122] "RemoveContainer" containerID="4a8c19ab2f102e699bcaa6b70786d05c622f962e4b30ede3510ebe5f1cf92a84" Nov 03 01:00:23.483849 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:23.483657 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-7406a9-29894460-vnwfw"] Nov 03 01:00:23.498867 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:23.498835 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-7406a9-29894460-vnwfw"] Nov 03 01:00:23.509434 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:23.509391 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-83755a-29894460-7z6xz"] Nov 03 01:00:23.525446 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:23.525412 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-07f5f6-29894460-5lb9h"] Nov 03 01:00:23.539241 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:23.539155 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-07f5f6-29894460-5lb9h"] Nov 03 01:00:23.554693 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:23.554616 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-db5422-29894460-w4ksr"] Nov 03 01:00:23.563233 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:23.563177 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-db5422-29894460-w4ksr"] Nov 03 01:00:23.573414 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:23.573341 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f88f87-29894460-p26bw"] Nov 03 01:00:23.579432 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:23.579346 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f88f87-29894460-p26bw"] Nov 03 01:00:23.622383 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:23.622265 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f7680d-29894460-qpglw" Nov 03 01:00:23.763589 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:23.762399 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/600dc1bf-8abc-480d-8503-3942ecd9d48b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/600dc1bf-8abc-480d-8503-3942ecd9d48b-maas-api-service-ca\") pod \"600dc1bf-8abc-480d-8503-3942ecd9d48b\" (UID: \"600dc1bf-8abc-480d-8503-3942ecd9d48b\") " Nov 03 01:00:23.763589 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:23.762721 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/600dc1bf-8abc-480d-8503-3942ecd9d48b-maas-api-service-ca" pod "600dc1bf-8abc-480d-8503-3942ecd9d48b" (UID: "600dc1bf-8abc-480d-8503-3942ecd9d48b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:00:23.763589 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:23.762740 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/600dc1bf-8abc-480d-8503-3942ecd9d48b-kube-api-access-hff44\" (UniqueName: \"kubernetes.io/projected/600dc1bf-8abc-480d-8503-3942ecd9d48b-kube-api-access-hff44\") pod \"600dc1bf-8abc-480d-8503-3942ecd9d48b\" (UID: \"600dc1bf-8abc-480d-8503-3942ecd9d48b\") " Nov 03 01:00:23.763589 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:23.763464 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/600dc1bf-8abc-480d-8503-3942ecd9d48b-maas-api-service-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:00:23.765799 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:23.765750 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/600dc1bf-8abc-480d-8503-3942ecd9d48b-kube-api-access-hff44" pod "600dc1bf-8abc-480d-8503-3942ecd9d48b" (UID: "600dc1bf-8abc-480d-8503-3942ecd9d48b"). InnerVolumeSpecName "kube-api-access-hff44". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:00:23.814099 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:23.814072 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f9a3a-29894460-sj5kk" Nov 03 01:00:23.817816 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:23.817790 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-83755a-29894460-7z6xz" Nov 03 01:00:23.865604 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:23.865554 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hff44\" (UniqueName: \"kubernetes.io/projected/600dc1bf-8abc-480d-8503-3942ecd9d48b-kube-api-access-hff44\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:00:23.966593 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:23.966515 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84fca73a-b1e2-4fee-892f-b2f162bd75b3-kube-api-access-9qd9s\" (UniqueName: \"kubernetes.io/projected/84fca73a-b1e2-4fee-892f-b2f162bd75b3-kube-api-access-9qd9s\") pod \"84fca73a-b1e2-4fee-892f-b2f162bd75b3\" (UID: \"84fca73a-b1e2-4fee-892f-b2f162bd75b3\") " Nov 03 01:00:23.966736 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:23.966709 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0370c2db-8545-4266-a4f4-719aebbd925b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0370c2db-8545-4266-a4f4-719aebbd925b-maas-api-service-ca\") pod \"0370c2db-8545-4266-a4f4-719aebbd925b\" (UID: \"0370c2db-8545-4266-a4f4-719aebbd925b\") " Nov 03 01:00:23.966800 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:23.966762 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0370c2db-8545-4266-a4f4-719aebbd925b-kube-api-access-gbg45\" (UniqueName: \"kubernetes.io/projected/0370c2db-8545-4266-a4f4-719aebbd925b-kube-api-access-gbg45\") pod \"0370c2db-8545-4266-a4f4-719aebbd925b\" (UID: \"0370c2db-8545-4266-a4f4-719aebbd925b\") " Nov 03 01:00:23.966861 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:23.966808 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84fca73a-b1e2-4fee-892f-b2f162bd75b3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/84fca73a-b1e2-4fee-892f-b2f162bd75b3-maas-api-service-ca\") pod \"84fca73a-b1e2-4fee-892f-b2f162bd75b3\" (UID: \"84fca73a-b1e2-4fee-892f-b2f162bd75b3\") " Nov 03 01:00:23.967282 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:23.967230 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0370c2db-8545-4266-a4f4-719aebbd925b-maas-api-service-ca" pod "0370c2db-8545-4266-a4f4-719aebbd925b" (UID: "0370c2db-8545-4266-a4f4-719aebbd925b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:00:23.967381 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:23.967290 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84fca73a-b1e2-4fee-892f-b2f162bd75b3-maas-api-service-ca" pod "84fca73a-b1e2-4fee-892f-b2f162bd75b3" (UID: "84fca73a-b1e2-4fee-892f-b2f162bd75b3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:00:23.967720 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:23.967699 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0370c2db-8545-4266-a4f4-719aebbd925b-maas-api-service-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:00:23.967765 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:23.967728 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/84fca73a-b1e2-4fee-892f-b2f162bd75b3-maas-api-service-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:00:23.970066 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:23.969049 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84fca73a-b1e2-4fee-892f-b2f162bd75b3-kube-api-access-9qd9s" pod "84fca73a-b1e2-4fee-892f-b2f162bd75b3" (UID: "84fca73a-b1e2-4fee-892f-b2f162bd75b3"). InnerVolumeSpecName "kube-api-access-9qd9s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:00:23.974245 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:23.974214 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0370c2db-8545-4266-a4f4-719aebbd925b-kube-api-access-gbg45" pod "0370c2db-8545-4266-a4f4-719aebbd925b" (UID: "0370c2db-8545-4266-a4f4-719aebbd925b"). InnerVolumeSpecName "kube-api-access-gbg45". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:00:23.984348 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:23.984319 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03c838c5-5695-434a-b78f-a338c14f3365" path="/var/lib/kubelet/pods/03c838c5-5695-434a-b78f-a338c14f3365/volumes" Nov 03 01:00:23.984876 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:23.984862 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64ad274e-6de7-4604-a8e4-958f67dbd00f" path="/var/lib/kubelet/pods/64ad274e-6de7-4604-a8e4-958f67dbd00f/volumes" Nov 03 01:00:23.985293 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:23.985264 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91abc456-0b19-4c43-9a94-57ef27f96c4a" path="/var/lib/kubelet/pods/91abc456-0b19-4c43-9a94-57ef27f96c4a/volumes" Nov 03 01:00:23.985967 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:23.985951 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9bb1557e-955c-41d1-8513-e1f8c4f341f3" path="/var/lib/kubelet/pods/9bb1557e-955c-41d1-8513-e1f8c4f341f3/volumes" Nov 03 01:00:23.986361 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:23.986348 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8c543e5-5eda-4993-92af-8da6a226cfa6" path="/var/lib/kubelet/pods/c8c543e5-5eda-4993-92af-8da6a226cfa6/volumes" Nov 03 01:00:24.069045 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:24.069000 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qd9s\" (UniqueName: \"kubernetes.io/projected/84fca73a-b1e2-4fee-892f-b2f162bd75b3-kube-api-access-9qd9s\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:00:24.069045 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:24.069042 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbg45\" (UniqueName: \"kubernetes.io/projected/0370c2db-8545-4266-a4f4-719aebbd925b-kube-api-access-gbg45\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:00:24.391491 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:24.391452 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-def754-744dbd79f-dm4jt" Nov 03 01:00:24.391932 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:24.391906 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-def754-555d7884c4-k8sh9" Nov 03 01:00:24.464159 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:24.464087 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f9a3a-29894460-sj5kk" Nov 03 01:00:24.464159 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:24.464099 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f9a3a-29894460-sj5kk" event={"ID":"84fca73a-b1e2-4fee-892f-b2f162bd75b3","Type":"ContainerDied","Data":"aa539b67894d9ad5095562c732c9966949dd147c2840b016c41c14d19ef2eb97"} Nov 03 01:00:24.464630 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:24.464176 4421 scope.go:122] "RemoveContainer" containerID="d74f8ff625e1b767998edb3629f5fb880841fc623d4564104cab029d8a1c3333" Nov 03 01:00:24.468517 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:24.467363 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-83755a-29894460-7z6xz" event={"ID":"0370c2db-8545-4266-a4f4-719aebbd925b","Type":"ContainerDied","Data":"edc668facbe2d8ad99732a56a8a5a4e754e3ed7a28c1a3e879f8eb2046515472"} Nov 03 01:00:24.468517 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:24.467474 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-83755a-29894460-7z6xz" Nov 03 01:00:24.473080 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:24.472898 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f7680d-29894460-qpglw" Nov 03 01:00:24.473080 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:24.473002 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f7680d-29894460-qpglw" event={"ID":"600dc1bf-8abc-480d-8503-3942ecd9d48b","Type":"ContainerDied","Data":"db1afae8c78cf220ccf4f68ff1b7737a373ccb6885f717bc33f89324e087b0f3"} Nov 03 01:00:24.491149 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:24.484184 4421 scope.go:122] "RemoveContainer" containerID="84c768b3b488413a494675264154acb0aec3a3a7ce9ada255a3368e09079d7dc" Nov 03 01:00:24.496587 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:24.496553 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-83755a-29894460-7z6xz"] Nov 03 01:00:24.500258 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:24.500232 4421 scope.go:122] "RemoveContainer" containerID="94d619dd062e625c52ed1534a58cb990722374793da5ba415cc8c4323e5e8ced" Nov 03 01:00:24.504207 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:24.504180 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-83755a-29894460-7z6xz"] Nov 03 01:00:24.517103 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:24.517074 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f9a3a-29894460-sj5kk"] Nov 03 01:00:24.521263 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:24.521236 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f9a3a-29894460-sj5kk"] Nov 03 01:00:24.530226 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:24.530188 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f7680d-29894460-qpglw"] Nov 03 01:00:24.534387 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:24.534357 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f7680d-29894460-qpglw"] Nov 03 01:00:25.177083 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:25.177042 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-47369d-7c8f86b795-hclwx" podUID="f4145d1f-cba3-4b46-a0ee-5ac18eec30ea" containerName="payload-processing" containerID="cri-o://07b096b853ae222023a3437fe30feffde013cea4508ca41ad51ce5800e3de357" gracePeriod=25 Nov 03 01:00:25.180635 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:25.180601 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-47369d-6fbdd5b9b7-h78wk" podUID="f465a3d3-72ba-40a8-a6a2-dc5dd77efcce" containerName="payload-pre-processing" containerID="cri-o://68fa0f326c088c4f670c1b58a9a650eb38bb6c945d71865802da38d00d796fc3" gracePeriod=25 Nov 03 01:00:25.552704 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:25.552669 4421 generic.go:356] "Generic (PLEG): container finished" podID="f4145d1f-cba3-4b46-a0ee-5ac18eec30ea" containerID="07b096b853ae222023a3437fe30feffde013cea4508ca41ad51ce5800e3de357" exitCode=0 Nov 03 01:00:25.553007 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:25.552985 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-47369d-7c8f86b795-hclwx" event={"ID":"f4145d1f-cba3-4b46-a0ee-5ac18eec30ea","Type":"ContainerDied","Data":"07b096b853ae222023a3437fe30feffde013cea4508ca41ad51ce5800e3de357"} Nov 03 01:00:25.562390 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:25.562360 4421 generic.go:356] "Generic (PLEG): container finished" podID="f465a3d3-72ba-40a8-a6a2-dc5dd77efcce" containerID="68fa0f326c088c4f670c1b58a9a650eb38bb6c945d71865802da38d00d796fc3" exitCode=0 Nov 03 01:00:25.562627 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:25.562579 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-47369d-6fbdd5b9b7-h78wk" event={"ID":"f465a3d3-72ba-40a8-a6a2-dc5dd77efcce","Type":"ContainerDied","Data":"68fa0f326c088c4f670c1b58a9a650eb38bb6c945d71865802da38d00d796fc3"} Nov 03 01:00:25.566515 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:25.566452 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-47369d-7c8f86b795-hclwx" Nov 03 01:00:25.629886 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:25.629855 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-47369d-6fbdd5b9b7-h78wk" Nov 03 01:00:25.645052 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:25.645017 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4145d1f-cba3-4b46-a0ee-5ac18eec30ea-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4145d1f-cba3-4b46-a0ee-5ac18eec30ea-plugins-config-volume\") pod \"f4145d1f-cba3-4b46-a0ee-5ac18eec30ea\" (UID: \"f4145d1f-cba3-4b46-a0ee-5ac18eec30ea\") " Nov 03 01:00:25.645261 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:25.645169 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4145d1f-cba3-4b46-a0ee-5ac18eec30ea-kube-api-access-p2cfj\" (UniqueName: \"kubernetes.io/projected/f4145d1f-cba3-4b46-a0ee-5ac18eec30ea-kube-api-access-p2cfj\") pod \"f4145d1f-cba3-4b46-a0ee-5ac18eec30ea\" (UID: \"f4145d1f-cba3-4b46-a0ee-5ac18eec30ea\") " Nov 03 01:00:25.645351 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:25.645331 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f465a3d3-72ba-40a8-a6a2-dc5dd77efcce-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f465a3d3-72ba-40a8-a6a2-dc5dd77efcce-plugins-config-volume\") pod \"f465a3d3-72ba-40a8-a6a2-dc5dd77efcce\" (UID: \"f465a3d3-72ba-40a8-a6a2-dc5dd77efcce\") " Nov 03 01:00:25.645430 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:25.645415 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f465a3d3-72ba-40a8-a6a2-dc5dd77efcce-kube-api-access-46ntg\" (UniqueName: \"kubernetes.io/projected/f465a3d3-72ba-40a8-a6a2-dc5dd77efcce-kube-api-access-46ntg\") pod \"f465a3d3-72ba-40a8-a6a2-dc5dd77efcce\" (UID: \"f465a3d3-72ba-40a8-a6a2-dc5dd77efcce\") " Nov 03 01:00:25.645652 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:25.645615 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4145d1f-cba3-4b46-a0ee-5ac18eec30ea-plugins-config-volume" pod "f4145d1f-cba3-4b46-a0ee-5ac18eec30ea" (UID: "f4145d1f-cba3-4b46-a0ee-5ac18eec30ea"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:00:25.645918 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:25.645728 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f465a3d3-72ba-40a8-a6a2-dc5dd77efcce-plugins-config-volume" pod "f465a3d3-72ba-40a8-a6a2-dc5dd77efcce" (UID: "f465a3d3-72ba-40a8-a6a2-dc5dd77efcce"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:00:25.647565 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:25.647530 4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4145d1f-cba3-4b46-a0ee-5ac18eec30ea-plugins-config-volume\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:00:25.647785 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:25.647764 4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f465a3d3-72ba-40a8-a6a2-dc5dd77efcce-plugins-config-volume\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:00:25.648903 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:25.648865 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4145d1f-cba3-4b46-a0ee-5ac18eec30ea-kube-api-access-p2cfj" pod "f4145d1f-cba3-4b46-a0ee-5ac18eec30ea" (UID: "f4145d1f-cba3-4b46-a0ee-5ac18eec30ea"). InnerVolumeSpecName "kube-api-access-p2cfj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:00:25.650440 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:25.650324 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f465a3d3-72ba-40a8-a6a2-dc5dd77efcce-kube-api-access-46ntg" pod "f465a3d3-72ba-40a8-a6a2-dc5dd77efcce" (UID: "f465a3d3-72ba-40a8-a6a2-dc5dd77efcce"). InnerVolumeSpecName "kube-api-access-46ntg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:00:25.749072 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:25.749038 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2cfj\" (UniqueName: \"kubernetes.io/projected/f4145d1f-cba3-4b46-a0ee-5ac18eec30ea-kube-api-access-p2cfj\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:00:25.749072 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:25.749072 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46ntg\" (UniqueName: \"kubernetes.io/projected/f465a3d3-72ba-40a8-a6a2-dc5dd77efcce-kube-api-access-46ntg\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:00:25.986456 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:25.986417 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0370c2db-8545-4266-a4f4-719aebbd925b" path="/var/lib/kubelet/pods/0370c2db-8545-4266-a4f4-719aebbd925b/volumes" Nov 03 01:00:25.999436 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:25.986963 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="600dc1bf-8abc-480d-8503-3942ecd9d48b" path="/var/lib/kubelet/pods/600dc1bf-8abc-480d-8503-3942ecd9d48b/volumes" Nov 03 01:00:25.999436 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:25.987390 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84fca73a-b1e2-4fee-892f-b2f162bd75b3" path="/var/lib/kubelet/pods/84fca73a-b1e2-4fee-892f-b2f162bd75b3/volumes" Nov 03 01:00:26.573339 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:26.573309 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-47369d-6fbdd5b9b7-h78wk" Nov 03 01:00:26.573339 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:26.573312 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-47369d-6fbdd5b9b7-h78wk" event={"ID":"f465a3d3-72ba-40a8-a6a2-dc5dd77efcce","Type":"ContainerDied","Data":"8b44f29c12a455add0a423506f796212ff67957d9de4a5df3375096cdd6b023b"} Nov 03 01:00:26.573880 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:26.573410 4421 scope.go:122] "RemoveContainer" containerID="68fa0f326c088c4f670c1b58a9a650eb38bb6c945d71865802da38d00d796fc3" Nov 03 01:00:26.575647 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:26.575616 4421 generic.go:356] "Generic (PLEG): container finished" podID="c95ca370-7e69-4c57-912c-e953469cb21f" containerID="b070fb3205bb6c1922e6d768b4dffd8cfca9496e661b97284d1307009e4c1bbf" exitCode=0 Nov 03 01:00:26.575971 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:26.575666 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ffdea6-openshift-default-85977f6bff-jw7rs" event={"ID":"c95ca370-7e69-4c57-912c-e953469cb21f","Type":"ContainerDied","Data":"b070fb3205bb6c1922e6d768b4dffd8cfca9496e661b97284d1307009e4c1bbf"} Nov 03 01:00:26.576523 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:26.576498 4421 scope.go:122] "RemoveContainer" containerID="b070fb3205bb6c1922e6d768b4dffd8cfca9496e661b97284d1307009e4c1bbf" Nov 03 01:00:26.577704 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:26.577676 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-47369d-7c8f86b795-hclwx" Nov 03 01:00:26.577810 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:26.577674 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-47369d-7c8f86b795-hclwx" event={"ID":"f4145d1f-cba3-4b46-a0ee-5ac18eec30ea","Type":"ContainerDied","Data":"0206c65eae78025839f73f77c44890d1cac2d89d8e025aaed631323bbc4f13ac"} Nov 03 01:00:26.578231 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:26.578201 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:00:26.578328 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:26.578309 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:00:26.578398 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:26.578365 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:00:26.586200 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:26.586177 4421 scope.go:122] "RemoveContainer" containerID="07b096b853ae222023a3437fe30feffde013cea4508ca41ad51ce5800e3de357" Nov 03 01:00:26.600791 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:26.600761 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-47369d-6fbdd5b9b7-h78wk"] Nov 03 01:00:26.605830 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:26.605804 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-47369d-6fbdd5b9b7-h78wk"] Nov 03 01:00:26.641815 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:26.641757 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-47369d-7c8f86b795-hclwx"] Nov 03 01:00:26.647099 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:26.647002 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-47369d-7c8f86b795-hclwx"] Nov 03 01:00:27.778994 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:27.778957 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ffdea6-openshift-default-85977f6bff-jw7rs" event={"ID":"c95ca370-7e69-4c57-912c-e953469cb21f","Type":"ContainerStarted","Data":"9cb5d78c8764bddf5f3bd53560fd8ea50f34914fec2c212619fad9a74589f64b"} Nov 03 01:00:27.990994 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:27.990962 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4145d1f-cba3-4b46-a0ee-5ac18eec30ea" path="/var/lib/kubelet/pods/f4145d1f-cba3-4b46-a0ee-5ac18eec30ea/volumes" Nov 03 01:00:27.991517 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:27.991490 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f465a3d3-72ba-40a8-a6a2-dc5dd77efcce" path="/var/lib/kubelet/pods/f465a3d3-72ba-40a8-a6a2-dc5dd77efcce/volumes" Nov 03 01:00:28.057915 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:28.044064 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-ffdea6-openshift-default-85977f6bff-jw7rs" Nov 03 01:00:28.469493 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:28.469406 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-ffdea6-openshift-default-85977f6bff-jw7rs" Nov 03 01:00:28.470832 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:28.470805 4421 patch_prober.go:29] interesting pod/e2e-api-b-ffdea6-openshift-default-85977f6bff-jw7rs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Nov 03 01:00:28.470974 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:28.470861 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ffdea6-openshift-default-85977f6bff-jw7rs" podUID="c95ca370-7e69-4c57-912c-e953469cb21f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Nov 03 01:00:29.481331 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:29.481021 4421 patch_prober.go:29] interesting pod/e2e-api-b-ffdea6-openshift-default-85977f6bff-jw7rs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Nov 03 01:00:29.481331 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:29.481082 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ffdea6-openshift-default-85977f6bff-jw7rs" podUID="c95ca370-7e69-4c57-912c-e953469cb21f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Nov 03 01:00:30.475295 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:30.475244 4421 patch_prober.go:29] interesting pod/e2e-api-b-ffdea6-openshift-default-85977f6bff-jw7rs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Nov 03 01:00:30.475608 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:30.475572 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ffdea6-openshift-default-85977f6bff-jw7rs" podUID="c95ca370-7e69-4c57-912c-e953469cb21f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Nov 03 01:00:31.469457 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:31.469423 4421 patch_prober.go:29] interesting pod/e2e-api-b-ffdea6-openshift-default-85977f6bff-jw7rs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Nov 03 01:00:31.469867 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:31.469481 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ffdea6-openshift-default-85977f6bff-jw7rs" podUID="c95ca370-7e69-4c57-912c-e953469cb21f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Nov 03 01:00:32.471704 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:32.471657 4421 patch_prober.go:29] interesting pod/e2e-api-b-ffdea6-openshift-default-85977f6bff-jw7rs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Nov 03 01:00:32.472379 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:32.471735 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ffdea6-openshift-default-85977f6bff-jw7rs" podUID="c95ca370-7e69-4c57-912c-e953469cb21f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Nov 03 01:00:33.475681 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:33.475639 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-ffdea6-openshift-default-85977f6bff-jw7rs" Nov 03 01:00:33.477081 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:33.477060 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-ffdea6-openshift-default-85977f6bff-jw7rs" Nov 03 01:00:39.062167 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:39.062128 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ce7eb93-bpnr2"] Nov 03 01:00:39.065885 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:39.065853 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ce7eb93-bpnr2"] Nov 03 01:00:39.980062 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:39.980019 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="318c5891-d03d-462f-91f1-de6c5a4a20f6" path="/var/lib/kubelet/pods/318c5891-d03d-462f-91f1-de6c5a4a20f6/volumes" Nov 03 01:00:44.671009 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:00:44.670969 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 01:00:48.307729 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:48.307697 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7f43f9-56467b6944-h8nzs"] Nov 03 01:00:48.311076 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:48.311037 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-7f43f9-56467b6944-h8nzs" podUID="18d503d0-9683-4d73-9c38-1991dcc19e44" containerName="payload-processing" containerID="cri-o://d6491115d035f3201c64478f5ec3a1b40ff296f6921b008330295ff7586f50e0" gracePeriod=30 Nov 03 01:00:48.320055 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:48.320019 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7f43f9-79d79dfff5-kpcxh"] Nov 03 01:00:48.320792 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:48.320726 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7f43f9-79d79dfff5-kpcxh" podUID="0920df17-fa3c-497f-814e-0a1be8a141d5" containerName="payload-pre-processing" containerID="cri-o://e2ef93acdb747314ee973e212940057c20e8aa28ee9c8d18c3569bdbe8c60f15" gracePeriod=30 Nov 03 01:00:48.833356 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:48.821262 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-7f43f9-56467b6944-h8nzs" Nov 03 01:00:48.948973 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:48.948940 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18d503d0-9683-4d73-9c38-1991dcc19e44-kube-api-access-qn4q2\" (UniqueName: \"kubernetes.io/projected/18d503d0-9683-4d73-9c38-1991dcc19e44-kube-api-access-qn4q2\") pod \"18d503d0-9683-4d73-9c38-1991dcc19e44\" (UID: \"18d503d0-9683-4d73-9c38-1991dcc19e44\") " Nov 03 01:00:48.949171 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:48.949151 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18d503d0-9683-4d73-9c38-1991dcc19e44-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18d503d0-9683-4d73-9c38-1991dcc19e44-plugins-config-volume\") pod \"18d503d0-9683-4d73-9c38-1991dcc19e44\" (UID: \"18d503d0-9683-4d73-9c38-1991dcc19e44\") " Nov 03 01:00:48.949744 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:48.949682 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18d503d0-9683-4d73-9c38-1991dcc19e44-plugins-config-volume" pod "18d503d0-9683-4d73-9c38-1991dcc19e44" (UID: "18d503d0-9683-4d73-9c38-1991dcc19e44"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:00:48.951244 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:48.951222 4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18d503d0-9683-4d73-9c38-1991dcc19e44-plugins-config-volume\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:00:48.951968 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:48.951941 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18d503d0-9683-4d73-9c38-1991dcc19e44-kube-api-access-qn4q2" pod "18d503d0-9683-4d73-9c38-1991dcc19e44" (UID: "18d503d0-9683-4d73-9c38-1991dcc19e44"). InnerVolumeSpecName "kube-api-access-qn4q2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:00:48.976346 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:48.976313 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7f43f9-79d79dfff5-kpcxh" Nov 03 01:00:49.052954 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:49.052876 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qn4q2\" (UniqueName: \"kubernetes.io/projected/18d503d0-9683-4d73-9c38-1991dcc19e44-kube-api-access-qn4q2\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:00:49.145039 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:49.145001 4421 generic.go:356] "Generic (PLEG): container finished" podID="0920df17-fa3c-497f-814e-0a1be8a141d5" containerID="e2ef93acdb747314ee973e212940057c20e8aa28ee9c8d18c3569bdbe8c60f15" exitCode=0 Nov 03 01:00:49.145253 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:49.145039 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7f43f9-79d79dfff5-kpcxh" event={"ID":"0920df17-fa3c-497f-814e-0a1be8a141d5","Type":"ContainerDied","Data":"e2ef93acdb747314ee973e212940057c20e8aa28ee9c8d18c3569bdbe8c60f15"} Nov 03 01:00:49.145253 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:49.145099 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7f43f9-79d79dfff5-kpcxh" event={"ID":"0920df17-fa3c-497f-814e-0a1be8a141d5","Type":"ContainerDied","Data":"dcb646a729527ad4ac2cf8e8cfd79f17b5628f0eb95292d9a77807208288337d"} Nov 03 01:00:49.145253 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:49.145125 4421 scope.go:122] "RemoveContainer" containerID="e2ef93acdb747314ee973e212940057c20e8aa28ee9c8d18c3569bdbe8c60f15" Nov 03 01:00:49.145253 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:49.145049 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7f43f9-79d79dfff5-kpcxh" Nov 03 01:00:49.146987 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:49.146961 4421 generic.go:356] "Generic (PLEG): container finished" podID="18d503d0-9683-4d73-9c38-1991dcc19e44" containerID="d6491115d035f3201c64478f5ec3a1b40ff296f6921b008330295ff7586f50e0" exitCode=0 Nov 03 01:00:49.147136 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:49.147004 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-7f43f9-56467b6944-h8nzs" Nov 03 01:00:49.147136 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:49.147032 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-7f43f9-56467b6944-h8nzs" event={"ID":"18d503d0-9683-4d73-9c38-1991dcc19e44","Type":"ContainerDied","Data":"d6491115d035f3201c64478f5ec3a1b40ff296f6921b008330295ff7586f50e0"} Nov 03 01:00:49.147136 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:49.147065 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-7f43f9-56467b6944-h8nzs" event={"ID":"18d503d0-9683-4d73-9c38-1991dcc19e44","Type":"ContainerDied","Data":"56f68c3b82c0e2722e12701aeb7b4c200a867e4c45d18c0c71d2e3df6e18d20c"} Nov 03 01:00:49.153710 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:49.153684 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0920df17-fa3c-497f-814e-0a1be8a141d5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0920df17-fa3c-497f-814e-0a1be8a141d5-plugins-config-volume\") pod \"0920df17-fa3c-497f-814e-0a1be8a141d5\" (UID: \"0920df17-fa3c-497f-814e-0a1be8a141d5\") " Nov 03 01:00:49.154040 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:49.154021 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0920df17-fa3c-497f-814e-0a1be8a141d5-kube-api-access-7f4xf\" (UniqueName: \"kubernetes.io/projected/0920df17-fa3c-497f-814e-0a1be8a141d5-kube-api-access-7f4xf\") pod \"0920df17-fa3c-497f-814e-0a1be8a141d5\" (UID: \"0920df17-fa3c-497f-814e-0a1be8a141d5\") " Nov 03 01:00:49.154105 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:49.154040 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0920df17-fa3c-497f-814e-0a1be8a141d5-plugins-config-volume" pod "0920df17-fa3c-497f-814e-0a1be8a141d5" (UID: "0920df17-fa3c-497f-814e-0a1be8a141d5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:00:49.155415 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:49.155394 4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0920df17-fa3c-497f-814e-0a1be8a141d5-plugins-config-volume\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:00:49.156576 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:49.156547 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0920df17-fa3c-497f-814e-0a1be8a141d5-kube-api-access-7f4xf" pod "0920df17-fa3c-497f-814e-0a1be8a141d5" (UID: "0920df17-fa3c-497f-814e-0a1be8a141d5"). InnerVolumeSpecName "kube-api-access-7f4xf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:00:49.158039 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:49.158020 4421 scope.go:122] "RemoveContainer" containerID="e2ef93acdb747314ee973e212940057c20e8aa28ee9c8d18c3569bdbe8c60f15" Nov 03 01:00:49.158361 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:00:49.158342 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2ef93acdb747314ee973e212940057c20e8aa28ee9c8d18c3569bdbe8c60f15\": container with ID starting with e2ef93acdb747314ee973e212940057c20e8aa28ee9c8d18c3569bdbe8c60f15 not found: ID does not exist" containerID="e2ef93acdb747314ee973e212940057c20e8aa28ee9c8d18c3569bdbe8c60f15" Nov 03 01:00:49.158418 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:49.158369 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2ef93acdb747314ee973e212940057c20e8aa28ee9c8d18c3569bdbe8c60f15"} err="failed to get container status \"e2ef93acdb747314ee973e212940057c20e8aa28ee9c8d18c3569bdbe8c60f15\": rpc error: code = NotFound desc = could not find container \"e2ef93acdb747314ee973e212940057c20e8aa28ee9c8d18c3569bdbe8c60f15\": container with ID starting with e2ef93acdb747314ee973e212940057c20e8aa28ee9c8d18c3569bdbe8c60f15 not found: ID does not exist" Nov 03 01:00:49.158418 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:49.158385 4421 scope.go:122] "RemoveContainer" containerID="d6491115d035f3201c64478f5ec3a1b40ff296f6921b008330295ff7586f50e0" Nov 03 01:00:49.170035 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:49.170007 4421 scope.go:122] "RemoveContainer" containerID="d6491115d035f3201c64478f5ec3a1b40ff296f6921b008330295ff7586f50e0" Nov 03 01:00:49.170411 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:00:49.170385 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6491115d035f3201c64478f5ec3a1b40ff296f6921b008330295ff7586f50e0\": container with ID starting with d6491115d035f3201c64478f5ec3a1b40ff296f6921b008330295ff7586f50e0 not found: ID does not exist" containerID="d6491115d035f3201c64478f5ec3a1b40ff296f6921b008330295ff7586f50e0" Nov 03 01:00:49.170517 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:49.170416 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6491115d035f3201c64478f5ec3a1b40ff296f6921b008330295ff7586f50e0"} err="failed to get container status \"d6491115d035f3201c64478f5ec3a1b40ff296f6921b008330295ff7586f50e0\": rpc error: code = NotFound desc = could not find container \"d6491115d035f3201c64478f5ec3a1b40ff296f6921b008330295ff7586f50e0\": container with ID starting with d6491115d035f3201c64478f5ec3a1b40ff296f6921b008330295ff7586f50e0 not found: ID does not exist" Nov 03 01:00:49.174056 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:49.174030 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7f43f9-56467b6944-h8nzs"] Nov 03 01:00:49.177809 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:49.177791 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7f43f9-56467b6944-h8nzs"] Nov 03 01:00:49.256721 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:49.256691 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7f4xf\" (UniqueName: \"kubernetes.io/projected/0920df17-fa3c-497f-814e-0a1be8a141d5-kube-api-access-7f4xf\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:00:49.471493 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:49.471457 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7f43f9-79d79dfff5-kpcxh"] Nov 03 01:00:49.475108 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:49.475085 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7f43f9-79d79dfff5-kpcxh"] Nov 03 01:00:49.980440 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:49.980405 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0920df17-fa3c-497f-814e-0a1be8a141d5" path="/var/lib/kubelet/pods/0920df17-fa3c-497f-814e-0a1be8a141d5/volumes" Nov 03 01:00:49.980894 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:49.980879 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18d503d0-9683-4d73-9c38-1991dcc19e44" path="/var/lib/kubelet/pods/18d503d0-9683-4d73-9c38-1991dcc19e44/volumes" Nov 03 01:00:51.177656 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:51.177562 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-ffdea6-67f595b459-k49vb"] Nov 03 01:00:53.032738 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:53.032695 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f13c8836-ns6nf"] Nov 03 01:00:53.072242 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:53.070765 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f13c8836-ns6nf"] Nov 03 01:00:53.979912 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:53.979878 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a00a109-67c5-46a5-94ec-3bc37e97bf18" path="/var/lib/kubelet/pods/0a00a109-67c5-46a5-94ec-3bc37e97bf18/volumes" Nov 03 01:00:54.975993 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:54.975958 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-7f43f9-openshift-default-74496bd585-lp65l"] Nov 03 01:00:54.976968 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:54.976148 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-7f43f9-openshift-default-74496bd585-lp65l" podUID="bd400742-fae4-44b7-b4af-497f60943da9" containerName="istio-proxy" containerID="cri-o://bd0f6c1c324c5dc66867d59f2d88c35222b174e4aff852802b90f1693b9cddc2" gracePeriod=30 Nov 03 01:00:56.178211 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:56.178168 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ffdea6-67f595b459-k49vb" podUID="dc932a32-c09c-4792-a90c-c4f906034e9f" containerName="payload-pre-processing" containerID="cri-o://251ffbe73b31080419d31d3f13c1aa7eac8413d8298fdbbc471f9e11568d7990" gracePeriod=25 Nov 03 01:00:56.302247 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:56.302217 4421 generic.go:356] "Generic (PLEG): container finished" podID="dc932a32-c09c-4792-a90c-c4f906034e9f" containerID="251ffbe73b31080419d31d3f13c1aa7eac8413d8298fdbbc471f9e11568d7990" exitCode=0 Nov 03 01:00:56.302396 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:56.302329 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ffdea6-67f595b459-k49vb" event={"ID":"dc932a32-c09c-4792-a90c-c4f906034e9f","Type":"ContainerDied","Data":"251ffbe73b31080419d31d3f13c1aa7eac8413d8298fdbbc471f9e11568d7990"} Nov 03 01:00:56.731613 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:56.731581 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ffdea6-67f595b459-k49vb" Nov 03 01:00:56.828629 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:56.828593 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc932a32-c09c-4792-a90c-c4f906034e9f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc932a32-c09c-4792-a90c-c4f906034e9f-plugins-config-volume\") pod \"dc932a32-c09c-4792-a90c-c4f906034e9f\" (UID: \"dc932a32-c09c-4792-a90c-c4f906034e9f\") " Nov 03 01:00:56.828848 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:56.828821 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc932a32-c09c-4792-a90c-c4f906034e9f-kube-api-access-ldzc2\" (UniqueName: \"kubernetes.io/projected/dc932a32-c09c-4792-a90c-c4f906034e9f-kube-api-access-ldzc2\") pod \"dc932a32-c09c-4792-a90c-c4f906034e9f\" (UID: \"dc932a32-c09c-4792-a90c-c4f906034e9f\") " Nov 03 01:00:56.829066 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:56.829038 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc932a32-c09c-4792-a90c-c4f906034e9f-plugins-config-volume" pod "dc932a32-c09c-4792-a90c-c4f906034e9f" (UID: "dc932a32-c09c-4792-a90c-c4f906034e9f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:00:56.829436 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:56.829412 4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc932a32-c09c-4792-a90c-c4f906034e9f-plugins-config-volume\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:00:56.830958 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:56.830932 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc932a32-c09c-4792-a90c-c4f906034e9f-kube-api-access-ldzc2" pod "dc932a32-c09c-4792-a90c-c4f906034e9f" (UID: "dc932a32-c09c-4792-a90c-c4f906034e9f"). InnerVolumeSpecName "kube-api-access-ldzc2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:00:56.931647 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:56.931543 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ldzc2\" (UniqueName: \"kubernetes.io/projected/dc932a32-c09c-4792-a90c-c4f906034e9f-kube-api-access-ldzc2\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:00:57.311481 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:57.311450 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ffdea6-67f595b459-k49vb" Nov 03 01:00:57.311481 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:57.311477 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ffdea6-67f595b459-k49vb" event={"ID":"dc932a32-c09c-4792-a90c-c4f906034e9f","Type":"ContainerDied","Data":"0b0638261804f2f8e0d52d12bf2d63da0195a73272e40c985a8dd57f28e38bc1"} Nov 03 01:00:57.311977 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:57.311537 4421 scope.go:122] "RemoveContainer" containerID="251ffbe73b31080419d31d3f13c1aa7eac8413d8298fdbbc471f9e11568d7990" Nov 03 01:00:57.335798 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:57.335761 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-ffdea6-67f595b459-k49vb"] Nov 03 01:00:57.340747 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:57.340705 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-ffdea6-67f595b459-k49vb"] Nov 03 01:00:57.737450 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:57.737364 4421 patch_prober.go:29] interesting pod/e2e-ipp-a-7f43f9-openshift-default-74496bd585-lp65l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 01:00:57.737450 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:57.737437 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-7f43f9-openshift-default-74496bd585-lp65l" podUID="bd400742-fae4-44b7-b4af-497f60943da9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 01:00:57.982197 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:57.982164 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc932a32-c09c-4792-a90c-c4f906034e9f" path="/var/lib/kubelet/pods/dc932a32-c09c-4792-a90c-c4f906034e9f/volumes" Nov 03 01:00:59.179981 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:59.179940 4421 scope.go:122] "RemoveContainer" containerID="4fcdbd7ece1ecaf43d8c78c867cf761abecdd48fefbbb4524cda5413fff3bc03" Nov 03 01:00:59.228789 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:59.228758 4421 scope.go:122] "RemoveContainer" containerID="af0e0c4f4e3f353329d8e239340e6b61ce586b82f7e77922eac28d14931b17cd" Nov 03 01:00:59.242093 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:59.242070 4421 scope.go:122] "RemoveContainer" containerID="52a0a2f219f234870cf453e39508097a97c9367606e664061d427f88c751be1f" Nov 03 01:00:59.252750 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:59.252719 4421 scope.go:122] "RemoveContainer" containerID="f92827d6d35408a50593a9073cb31be4e570868cf8a5a8ddf16ed0c2fcde204b" Nov 03 01:00:59.263509 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:59.263481 4421 scope.go:122] "RemoveContainer" containerID="07735d553edacf36d3b8e4c14ef0d411e5c6837f6bbff59e126af149fc469d0c" Nov 03 01:00:59.274960 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:59.274937 4421 scope.go:122] "RemoveContainer" containerID="196edae792056b59afcf0389ba8152d37b0562c1de6016b4c020e14abd490f96" Nov 03 01:00:59.286229 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:59.286204 4421 scope.go:122] "RemoveContainer" containerID="0c2310530b2a0f532ad97b7881ff2aec365fbd7394fe12cbf77114f326e32d52" Nov 03 01:00:59.297329 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:59.297302 4421 scope.go:122] "RemoveContainer" containerID="b253436f1a8653c42495e7ebb7616b92eaa7886cb6ca1155eb942193f785e209" Nov 03 01:00:59.309582 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:59.309556 4421 scope.go:122] "RemoveContainer" containerID="b4262ea71982eb2994361392030856f81abb26f3ed7a421901b4fe285eab25e2" Nov 03 01:00:59.329510 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:59.329190 4421 scope.go:122] "RemoveContainer" containerID="6e95a8f5d7f8b6ce32701138996de2565427fafa98af69932d0a557f423ed790" Nov 03 01:00:59.341357 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:59.341327 4421 scope.go:122] "RemoveContainer" containerID="5012a85874dc216f7e8893aaea622d83d12dc3c34dc2a82db385f1b55e700cda" Nov 03 01:00:59.507192 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:59.507150 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-ffdea6-openshift-default-85977f6bff-jw7rs"] Nov 03 01:00:59.507488 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:00:59.507458 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-ffdea6-openshift-default-85977f6bff-jw7rs" podUID="c95ca370-7e69-4c57-912c-e953469cb21f" containerName="istio-proxy" containerID="cri-o://9cb5d78c8764bddf5f3bd53560fd8ea50f34914fec2c212619fad9a74589f64b" gracePeriod=30 Nov 03 01:01:00.422313 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:00.400489 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-7f43f9-openshift-default-74496bd585-lp65l" Nov 03 01:01:00.466321 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:00.465761 4421 generic.go:356] "Generic (PLEG): container finished" podID="bd400742-fae4-44b7-b4af-497f60943da9" containerID="bd0f6c1c324c5dc66867d59f2d88c35222b174e4aff852802b90f1693b9cddc2" exitCode=0 Nov 03 01:01:00.466321 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:00.465978 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-7f43f9-openshift-default-74496bd585-lp65l" event={"ID":"bd400742-fae4-44b7-b4af-497f60943da9","Type":"ContainerDied","Data":"bd0f6c1c324c5dc66867d59f2d88c35222b174e4aff852802b90f1693b9cddc2"} Nov 03 01:01:00.466321 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:00.466034 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-7f43f9-openshift-default-74496bd585-lp65l" event={"ID":"bd400742-fae4-44b7-b4af-497f60943da9","Type":"ContainerDied","Data":"89715d8d27305ad07925c8e3c86d14bfee13984c87b2beeb2ff9e7f98d84a273"} Nov 03 01:01:00.466321 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:00.466093 4421 scope.go:122] "RemoveContainer" containerID="bd0f6c1c324c5dc66867d59f2d88c35222b174e4aff852802b90f1693b9cddc2" Nov 03 01:01:00.466709 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:00.466335 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-7f43f9-openshift-default-74496bd585-lp65l" Nov 03 01:01:00.514528 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:00.512058 4421 scope.go:122] "RemoveContainer" containerID="94b7c26d9b38f76f76ace804b3d83682116eae5a9a17db4d28bee6ac817f8911" Nov 03 01:01:00.519365 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:00.519333 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd400742-fae4-44b7-b4af-497f60943da9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd400742-fae4-44b7-b4af-497f60943da9-workload-certs\") pod \"bd400742-fae4-44b7-b4af-497f60943da9\" (UID: \"bd400742-fae4-44b7-b4af-497f60943da9\") " Nov 03 01:01:00.519491 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:00.519380 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd400742-fae4-44b7-b4af-497f60943da9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd400742-fae4-44b7-b4af-497f60943da9-istiod-ca-cert\") pod \"bd400742-fae4-44b7-b4af-497f60943da9\" (UID: \"bd400742-fae4-44b7-b4af-497f60943da9\") " Nov 03 01:01:00.519491 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:00.519433 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd400742-fae4-44b7-b4af-497f60943da9-istio-token\" (UniqueName: \"kubernetes.io/projected/bd400742-fae4-44b7-b4af-497f60943da9-istio-token\") pod \"bd400742-fae4-44b7-b4af-497f60943da9\" (UID: \"bd400742-fae4-44b7-b4af-497f60943da9\") " Nov 03 01:01:00.519602 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:00.519506 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd400742-fae4-44b7-b4af-497f60943da9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd400742-fae4-44b7-b4af-497f60943da9-credential-socket\") pod \"bd400742-fae4-44b7-b4af-497f60943da9\" (UID: \"bd400742-fae4-44b7-b4af-497f60943da9\") " Nov 03 01:01:00.519657 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:00.519629 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd400742-fae4-44b7-b4af-497f60943da9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd400742-fae4-44b7-b4af-497f60943da9-workload-socket\") pod \"bd400742-fae4-44b7-b4af-497f60943da9\" (UID: \"bd400742-fae4-44b7-b4af-497f60943da9\") " Nov 03 01:01:00.519711 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:00.519673 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bd400742-fae4-44b7-b4af-497f60943da9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd400742-fae4-44b7-b4af-497f60943da9-istio-podinfo\") pod \"bd400742-fae4-44b7-b4af-497f60943da9\" (UID: \"bd400742-fae4-44b7-b4af-497f60943da9\") " Nov 03 01:01:00.519769 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:00.519734 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd400742-fae4-44b7-b4af-497f60943da9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd400742-fae4-44b7-b4af-497f60943da9-istio-envoy\") pod \"bd400742-fae4-44b7-b4af-497f60943da9\" (UID: \"bd400742-fae4-44b7-b4af-497f60943da9\") " Nov 03 01:01:00.519826 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:00.519781 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd400742-fae4-44b7-b4af-497f60943da9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd400742-fae4-44b7-b4af-497f60943da9-istio-data\") pod \"bd400742-fae4-44b7-b4af-497f60943da9\" (UID: \"bd400742-fae4-44b7-b4af-497f60943da9\") " Nov 03 01:01:00.519967 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:00.519942 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd400742-fae4-44b7-b4af-497f60943da9-kube-api-access-cxffw\" (UniqueName: \"kubernetes.io/projected/bd400742-fae4-44b7-b4af-497f60943da9-kube-api-access-cxffw\") pod \"bd400742-fae4-44b7-b4af-497f60943da9\" (UID: \"bd400742-fae4-44b7-b4af-497f60943da9\") " Nov 03 01:01:00.520401 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:00.520262 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd400742-fae4-44b7-b4af-497f60943da9-workload-certs" pod "bd400742-fae4-44b7-b4af-497f60943da9" (UID: "bd400742-fae4-44b7-b4af-497f60943da9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:01:00.520785 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:00.520759 4421 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd400742-fae4-44b7-b4af-497f60943da9-workload-certs\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:01:00.521269 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:00.521240 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd400742-fae4-44b7-b4af-497f60943da9-istiod-ca-cert" pod "bd400742-fae4-44b7-b4af-497f60943da9" (UID: "bd400742-fae4-44b7-b4af-497f60943da9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:01:00.523836 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:00.523628 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd400742-fae4-44b7-b4af-497f60943da9-istio-token" pod "bd400742-fae4-44b7-b4af-497f60943da9" (UID: "bd400742-fae4-44b7-b4af-497f60943da9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:01:00.524403 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:00.524328 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd400742-fae4-44b7-b4af-497f60943da9-credential-socket" pod "bd400742-fae4-44b7-b4af-497f60943da9" (UID: "bd400742-fae4-44b7-b4af-497f60943da9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:01:00.524403 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:00.524342 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd400742-fae4-44b7-b4af-497f60943da9-istio-data" pod "bd400742-fae4-44b7-b4af-497f60943da9" (UID: "bd400742-fae4-44b7-b4af-497f60943da9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:01:00.524403 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:00.524375 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd400742-fae4-44b7-b4af-497f60943da9-workload-socket" pod "bd400742-fae4-44b7-b4af-497f60943da9" (UID: "bd400742-fae4-44b7-b4af-497f60943da9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:01:00.526314 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:00.525681 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd400742-fae4-44b7-b4af-497f60943da9-kube-api-access-cxffw" pod "bd400742-fae4-44b7-b4af-497f60943da9" (UID: "bd400742-fae4-44b7-b4af-497f60943da9"). InnerVolumeSpecName "kube-api-access-cxffw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:01:00.526509 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:00.526470 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd400742-fae4-44b7-b4af-497f60943da9-istio-envoy" pod "bd400742-fae4-44b7-b4af-497f60943da9" (UID: "bd400742-fae4-44b7-b4af-497f60943da9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:01:00.526988 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:00.526963 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bd400742-fae4-44b7-b4af-497f60943da9-istio-podinfo" pod "bd400742-fae4-44b7-b4af-497f60943da9" (UID: "bd400742-fae4-44b7-b4af-497f60943da9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:01:00.561584 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:00.561548 4421 scope.go:122] "RemoveContainer" containerID="bd0f6c1c324c5dc66867d59f2d88c35222b174e4aff852802b90f1693b9cddc2" Nov 03 01:01:00.562021 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:01:00.561990 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd0f6c1c324c5dc66867d59f2d88c35222b174e4aff852802b90f1693b9cddc2\": container with ID starting with bd0f6c1c324c5dc66867d59f2d88c35222b174e4aff852802b90f1693b9cddc2 not found: ID does not exist" containerID="bd0f6c1c324c5dc66867d59f2d88c35222b174e4aff852802b90f1693b9cddc2" Nov 03 01:01:00.562160 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:00.562029 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd0f6c1c324c5dc66867d59f2d88c35222b174e4aff852802b90f1693b9cddc2"} err="failed to get container status \"bd0f6c1c324c5dc66867d59f2d88c35222b174e4aff852802b90f1693b9cddc2\": rpc error: code = NotFound desc = could not find container \"bd0f6c1c324c5dc66867d59f2d88c35222b174e4aff852802b90f1693b9cddc2\": container with ID starting with bd0f6c1c324c5dc66867d59f2d88c35222b174e4aff852802b90f1693b9cddc2 not found: ID does not exist" Nov 03 01:01:00.562160 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:00.562052 4421 scope.go:122] "RemoveContainer" containerID="94b7c26d9b38f76f76ace804b3d83682116eae5a9a17db4d28bee6ac817f8911" Nov 03 01:01:00.562464 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:01:00.562444 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94b7c26d9b38f76f76ace804b3d83682116eae5a9a17db4d28bee6ac817f8911\": container with ID starting with 94b7c26d9b38f76f76ace804b3d83682116eae5a9a17db4d28bee6ac817f8911 not found: ID does not exist" containerID="94b7c26d9b38f76f76ace804b3d83682116eae5a9a17db4d28bee6ac817f8911" Nov 03 01:01:00.562574 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:00.562470 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94b7c26d9b38f76f76ace804b3d83682116eae5a9a17db4d28bee6ac817f8911"} err="failed to get container status \"94b7c26d9b38f76f76ace804b3d83682116eae5a9a17db4d28bee6ac817f8911\": rpc error: code = NotFound desc = could not find container \"94b7c26d9b38f76f76ace804b3d83682116eae5a9a17db4d28bee6ac817f8911\": container with ID starting with 94b7c26d9b38f76f76ace804b3d83682116eae5a9a17db4d28bee6ac817f8911 not found: ID does not exist" Nov 03 01:01:00.621945 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:00.621907 4421 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd400742-fae4-44b7-b4af-497f60943da9-istiod-ca-cert\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:01:00.621945 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:00.621938 4421 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd400742-fae4-44b7-b4af-497f60943da9-istio-token\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:01:00.621945 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:00.621950 4421 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd400742-fae4-44b7-b4af-497f60943da9-credential-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:01:00.622258 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:00.621961 4421 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd400742-fae4-44b7-b4af-497f60943da9-workload-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:01:00.622258 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:00.621970 4421 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd400742-fae4-44b7-b4af-497f60943da9-istio-podinfo\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:01:00.622258 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:00.621979 4421 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd400742-fae4-44b7-b4af-497f60943da9-istio-envoy\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:01:00.622258 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:00.621987 4421 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd400742-fae4-44b7-b4af-497f60943da9-istio-data\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:01:00.622258 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:00.621998 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxffw\" (UniqueName: \"kubernetes.io/projected/bd400742-fae4-44b7-b4af-497f60943da9-kube-api-access-cxffw\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:01:00.804342 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:00.804291 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-7f43f9-openshift-default-74496bd585-lp65l"] Nov 03 01:01:00.815762 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:00.815718 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-7f43f9-openshift-default-74496bd585-lp65l"] Nov 03 01:01:01.981836 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:01.981801 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd400742-fae4-44b7-b4af-497f60943da9" path="/var/lib/kubelet/pods/bd400742-fae4-44b7-b4af-497f60943da9/volumes" Nov 03 01:01:03.478798 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:03.478761 4421 patch_prober.go:29] interesting pod/e2e-api-b-ffdea6-openshift-default-85977f6bff-jw7rs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 01:01:03.479183 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:03.478835 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-ffdea6-openshift-default-85977f6bff-jw7rs" podUID="c95ca370-7e69-4c57-912c-e953469cb21f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 01:01:04.841454 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:04.841424 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-ffdea6-openshift-default-85977f6bff-jw7rs" Nov 03 01:01:04.869228 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:04.869191 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c95ca370-7e69-4c57-912c-e953469cb21f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c95ca370-7e69-4c57-912c-e953469cb21f-istio-envoy\") pod \"c95ca370-7e69-4c57-912c-e953469cb21f\" (UID: \"c95ca370-7e69-4c57-912c-e953469cb21f\") " Nov 03 01:01:04.869403 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:04.869244 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c95ca370-7e69-4c57-912c-e953469cb21f-kube-api-access-fcv4w\" (UniqueName: \"kubernetes.io/projected/c95ca370-7e69-4c57-912c-e953469cb21f-kube-api-access-fcv4w\") pod \"c95ca370-7e69-4c57-912c-e953469cb21f\" (UID: \"c95ca370-7e69-4c57-912c-e953469cb21f\") " Nov 03 01:01:04.869403 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:04.869328 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c95ca370-7e69-4c57-912c-e953469cb21f-istio-token\" (UniqueName: \"kubernetes.io/projected/c95ca370-7e69-4c57-912c-e953469cb21f-istio-token\") pod \"c95ca370-7e69-4c57-912c-e953469cb21f\" (UID: \"c95ca370-7e69-4c57-912c-e953469cb21f\") " Nov 03 01:01:04.869403 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:04.869377 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c95ca370-7e69-4c57-912c-e953469cb21f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c95ca370-7e69-4c57-912c-e953469cb21f-istio-data\") pod \"c95ca370-7e69-4c57-912c-e953469cb21f\" (UID: \"c95ca370-7e69-4c57-912c-e953469cb21f\") " Nov 03 01:01:04.869604 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:04.869415 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c95ca370-7e69-4c57-912c-e953469cb21f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c95ca370-7e69-4c57-912c-e953469cb21f-istio-podinfo\") pod \"c95ca370-7e69-4c57-912c-e953469cb21f\" (UID: \"c95ca370-7e69-4c57-912c-e953469cb21f\") " Nov 03 01:01:04.869604 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:04.869467 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c95ca370-7e69-4c57-912c-e953469cb21f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c95ca370-7e69-4c57-912c-e953469cb21f-istiod-ca-cert\") pod \"c95ca370-7e69-4c57-912c-e953469cb21f\" (UID: \"c95ca370-7e69-4c57-912c-e953469cb21f\") " Nov 03 01:01:04.869604 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:04.869534 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c95ca370-7e69-4c57-912c-e953469cb21f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c95ca370-7e69-4c57-912c-e953469cb21f-workload-certs\") pod \"c95ca370-7e69-4c57-912c-e953469cb21f\" (UID: \"c95ca370-7e69-4c57-912c-e953469cb21f\") " Nov 03 01:01:04.869604 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:04.869578 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c95ca370-7e69-4c57-912c-e953469cb21f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c95ca370-7e69-4c57-912c-e953469cb21f-workload-socket\") pod \"c95ca370-7e69-4c57-912c-e953469cb21f\" (UID: \"c95ca370-7e69-4c57-912c-e953469cb21f\") " Nov 03 01:01:04.869827 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:04.869632 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c95ca370-7e69-4c57-912c-e953469cb21f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c95ca370-7e69-4c57-912c-e953469cb21f-credential-socket\") pod \"c95ca370-7e69-4c57-912c-e953469cb21f\" (UID: \"c95ca370-7e69-4c57-912c-e953469cb21f\") " Nov 03 01:01:04.870560 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:04.870093 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c95ca370-7e69-4c57-912c-e953469cb21f-istiod-ca-cert" pod "c95ca370-7e69-4c57-912c-e953469cb21f" (UID: "c95ca370-7e69-4c57-912c-e953469cb21f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:01:04.870560 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:04.870144 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c95ca370-7e69-4c57-912c-e953469cb21f-istio-data" pod "c95ca370-7e69-4c57-912c-e953469cb21f" (UID: "c95ca370-7e69-4c57-912c-e953469cb21f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:01:04.870560 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:04.870333 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c95ca370-7e69-4c57-912c-e953469cb21f-credential-socket" pod "c95ca370-7e69-4c57-912c-e953469cb21f" (UID: "c95ca370-7e69-4c57-912c-e953469cb21f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:01:04.870560 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:04.870488 4421 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c95ca370-7e69-4c57-912c-e953469cb21f-istio-data\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:01:04.870560 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:04.870496 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c95ca370-7e69-4c57-912c-e953469cb21f-workload-certs" pod "c95ca370-7e69-4c57-912c-e953469cb21f" (UID: "c95ca370-7e69-4c57-912c-e953469cb21f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:01:04.870560 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:04.870513 4421 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c95ca370-7e69-4c57-912c-e953469cb21f-istiod-ca-cert\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:01:04.870560 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:04.870532 4421 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c95ca370-7e69-4c57-912c-e953469cb21f-credential-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:01:04.871027 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:04.870670 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c95ca370-7e69-4c57-912c-e953469cb21f-workload-socket" pod "c95ca370-7e69-4c57-912c-e953469cb21f" (UID: "c95ca370-7e69-4c57-912c-e953469cb21f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:01:04.872439 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:04.872127 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c95ca370-7e69-4c57-912c-e953469cb21f-kube-api-access-fcv4w" pod "c95ca370-7e69-4c57-912c-e953469cb21f" (UID: "c95ca370-7e69-4c57-912c-e953469cb21f"). InnerVolumeSpecName "kube-api-access-fcv4w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:01:04.872809 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:04.872783 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c95ca370-7e69-4c57-912c-e953469cb21f-istio-token" pod "c95ca370-7e69-4c57-912c-e953469cb21f" (UID: "c95ca370-7e69-4c57-912c-e953469cb21f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:01:04.873002 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:04.872974 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c95ca370-7e69-4c57-912c-e953469cb21f-istio-podinfo" pod "c95ca370-7e69-4c57-912c-e953469cb21f" (UID: "c95ca370-7e69-4c57-912c-e953469cb21f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:01:04.873185 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:04.873161 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c95ca370-7e69-4c57-912c-e953469cb21f-istio-envoy" pod "c95ca370-7e69-4c57-912c-e953469cb21f" (UID: "c95ca370-7e69-4c57-912c-e953469cb21f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:01:04.971949 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:04.971916 4421 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c95ca370-7e69-4c57-912c-e953469cb21f-istio-envoy\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:01:04.971949 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:04.971947 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fcv4w\" (UniqueName: \"kubernetes.io/projected/c95ca370-7e69-4c57-912c-e953469cb21f-kube-api-access-fcv4w\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:01:04.972161 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:04.971959 4421 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c95ca370-7e69-4c57-912c-e953469cb21f-istio-token\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:01:04.972161 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:04.971968 4421 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c95ca370-7e69-4c57-912c-e953469cb21f-istio-podinfo\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:01:04.972161 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:04.971976 4421 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c95ca370-7e69-4c57-912c-e953469cb21f-workload-certs\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:01:04.972161 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:04.971985 4421 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c95ca370-7e69-4c57-912c-e953469cb21f-workload-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:01:05.530726 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:05.530689 4421 generic.go:356] "Generic (PLEG): container finished" podID="c95ca370-7e69-4c57-912c-e953469cb21f" containerID="9cb5d78c8764bddf5f3bd53560fd8ea50f34914fec2c212619fad9a74589f64b" exitCode=0 Nov 03 01:01:05.530904 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:05.530733 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-ffdea6-openshift-default-85977f6bff-jw7rs" Nov 03 01:01:05.530904 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:05.530760 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ffdea6-openshift-default-85977f6bff-jw7rs" event={"ID":"c95ca370-7e69-4c57-912c-e953469cb21f","Type":"ContainerDied","Data":"9cb5d78c8764bddf5f3bd53560fd8ea50f34914fec2c212619fad9a74589f64b"} Nov 03 01:01:05.530904 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:05.530817 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ffdea6-openshift-default-85977f6bff-jw7rs" event={"ID":"c95ca370-7e69-4c57-912c-e953469cb21f","Type":"ContainerDied","Data":"35b05066d3652409fb297ccf114610302f9fa0f7816a3393be30f7a163ac22b3"} Nov 03 01:01:05.530904 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:05.530844 4421 scope.go:122] "RemoveContainer" containerID="9cb5d78c8764bddf5f3bd53560fd8ea50f34914fec2c212619fad9a74589f64b" Nov 03 01:01:05.545105 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:05.545082 4421 scope.go:122] "RemoveContainer" containerID="b070fb3205bb6c1922e6d768b4dffd8cfca9496e661b97284d1307009e4c1bbf" Nov 03 01:01:05.568048 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:05.567263 4421 scope.go:122] "RemoveContainer" containerID="9cb5d78c8764bddf5f3bd53560fd8ea50f34914fec2c212619fad9a74589f64b" Nov 03 01:01:05.568048 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:01:05.567767 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9cb5d78c8764bddf5f3bd53560fd8ea50f34914fec2c212619fad9a74589f64b\": container with ID starting with 9cb5d78c8764bddf5f3bd53560fd8ea50f34914fec2c212619fad9a74589f64b not found: ID does not exist" containerID="9cb5d78c8764bddf5f3bd53560fd8ea50f34914fec2c212619fad9a74589f64b" Nov 03 01:01:05.568048 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:05.567803 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9cb5d78c8764bddf5f3bd53560fd8ea50f34914fec2c212619fad9a74589f64b"} err="failed to get container status \"9cb5d78c8764bddf5f3bd53560fd8ea50f34914fec2c212619fad9a74589f64b\": rpc error: code = NotFound desc = could not find container \"9cb5d78c8764bddf5f3bd53560fd8ea50f34914fec2c212619fad9a74589f64b\": container with ID starting with 9cb5d78c8764bddf5f3bd53560fd8ea50f34914fec2c212619fad9a74589f64b not found: ID does not exist" Nov 03 01:01:05.568048 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:05.567826 4421 scope.go:122] "RemoveContainer" containerID="b070fb3205bb6c1922e6d768b4dffd8cfca9496e661b97284d1307009e4c1bbf" Nov 03 01:01:05.574541 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:01:05.574511 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b070fb3205bb6c1922e6d768b4dffd8cfca9496e661b97284d1307009e4c1bbf\": container with ID starting with b070fb3205bb6c1922e6d768b4dffd8cfca9496e661b97284d1307009e4c1bbf not found: ID does not exist" containerID="b070fb3205bb6c1922e6d768b4dffd8cfca9496e661b97284d1307009e4c1bbf" Nov 03 01:01:05.574676 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:05.574549 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b070fb3205bb6c1922e6d768b4dffd8cfca9496e661b97284d1307009e4c1bbf"} err="failed to get container status \"b070fb3205bb6c1922e6d768b4dffd8cfca9496e661b97284d1307009e4c1bbf\": rpc error: code = NotFound desc = could not find container \"b070fb3205bb6c1922e6d768b4dffd8cfca9496e661b97284d1307009e4c1bbf\": container with ID starting with b070fb3205bb6c1922e6d768b4dffd8cfca9496e661b97284d1307009e4c1bbf not found: ID does not exist" Nov 03 01:01:05.574751 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:05.574733 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-def754-555d7884c4-k8sh9"] Nov 03 01:01:05.574977 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:05.574955 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-def754-555d7884c4-k8sh9" podUID="df69fbbc-b8cb-4681-9f8d-c3c6f788cde5" containerName="payload-processing" containerID="cri-o://b623172352dd7a8ba50470b32d857820acbeaa047f75d8bda1a59f8568998f57" gracePeriod=30 Nov 03 01:01:05.580330 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:05.580301 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-def754-744dbd79f-dm4jt"] Nov 03 01:01:05.580545 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:05.580521 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-def754-744dbd79f-dm4jt" podUID="048aae80-64d2-41b2-9e6a-6b6431d41bd9" containerName="payload-pre-processing" containerID="cri-o://f59c1c4d16212ada56120d62dd80a5c5b3b8c2ad566f46a4bbcea5f129430786" gracePeriod=30 Nov 03 01:01:05.589377 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:05.589247 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-ffdea6-openshift-default-85977f6bff-jw7rs"] Nov 03 01:01:05.623197 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:05.623158 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-ffdea6-openshift-default-85977f6bff-jw7rs"] Nov 03 01:01:06.047537 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:06.047484 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c95ca370-7e69-4c57-912c-e953469cb21f" path="/var/lib/kubelet/pods/c95ca370-7e69-4c57-912c-e953469cb21f/volumes" Nov 03 01:01:06.197646 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:06.197613 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-def754-555d7884c4-k8sh9" Nov 03 01:01:06.240054 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:06.240019 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-def754-744dbd79f-dm4jt" Nov 03 01:01:06.322061 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:06.321969 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/048aae80-64d2-41b2-9e6a-6b6431d41bd9-kube-api-access-f4q58\" (UniqueName: \"kubernetes.io/projected/048aae80-64d2-41b2-9e6a-6b6431d41bd9-kube-api-access-f4q58\") pod \"048aae80-64d2-41b2-9e6a-6b6431d41bd9\" (UID: \"048aae80-64d2-41b2-9e6a-6b6431d41bd9\") " Nov 03 01:01:06.322244 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:06.322084 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df69fbbc-b8cb-4681-9f8d-c3c6f788cde5-kube-api-access-pfn9q\" (UniqueName: \"kubernetes.io/projected/df69fbbc-b8cb-4681-9f8d-c3c6f788cde5-kube-api-access-pfn9q\") pod \"df69fbbc-b8cb-4681-9f8d-c3c6f788cde5\" (UID: \"df69fbbc-b8cb-4681-9f8d-c3c6f788cde5\") " Nov 03 01:01:06.322451 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:06.322426 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/048aae80-64d2-41b2-9e6a-6b6431d41bd9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/048aae80-64d2-41b2-9e6a-6b6431d41bd9-plugins-config-volume\") pod \"048aae80-64d2-41b2-9e6a-6b6431d41bd9\" (UID: \"048aae80-64d2-41b2-9e6a-6b6431d41bd9\") " Nov 03 01:01:06.322565 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:06.322489 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df69fbbc-b8cb-4681-9f8d-c3c6f788cde5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df69fbbc-b8cb-4681-9f8d-c3c6f788cde5-plugins-config-volume\") pod \"df69fbbc-b8cb-4681-9f8d-c3c6f788cde5\" (UID: \"df69fbbc-b8cb-4681-9f8d-c3c6f788cde5\") " Nov 03 01:01:06.323050 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:06.323021 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/048aae80-64d2-41b2-9e6a-6b6431d41bd9-plugins-config-volume" pod "048aae80-64d2-41b2-9e6a-6b6431d41bd9" (UID: "048aae80-64d2-41b2-9e6a-6b6431d41bd9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:01:06.323396 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:06.323371 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df69fbbc-b8cb-4681-9f8d-c3c6f788cde5-plugins-config-volume" pod "df69fbbc-b8cb-4681-9f8d-c3c6f788cde5" (UID: "df69fbbc-b8cb-4681-9f8d-c3c6f788cde5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:01:06.323574 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:06.323541 4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/048aae80-64d2-41b2-9e6a-6b6431d41bd9-plugins-config-volume\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:01:06.323574 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:06.323568 4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df69fbbc-b8cb-4681-9f8d-c3c6f788cde5-plugins-config-volume\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:01:06.325389 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:06.324937 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/048aae80-64d2-41b2-9e6a-6b6431d41bd9-kube-api-access-f4q58" pod "048aae80-64d2-41b2-9e6a-6b6431d41bd9" (UID: "048aae80-64d2-41b2-9e6a-6b6431d41bd9"). InnerVolumeSpecName "kube-api-access-f4q58". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:01:06.325732 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:06.325710 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df69fbbc-b8cb-4681-9f8d-c3c6f788cde5-kube-api-access-pfn9q" pod "df69fbbc-b8cb-4681-9f8d-c3c6f788cde5" (UID: "df69fbbc-b8cb-4681-9f8d-c3c6f788cde5"). InnerVolumeSpecName "kube-api-access-pfn9q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:01:06.425355 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:06.425318 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4q58\" (UniqueName: \"kubernetes.io/projected/048aae80-64d2-41b2-9e6a-6b6431d41bd9-kube-api-access-f4q58\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:01:06.425355 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:06.425348 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pfn9q\" (UniqueName: \"kubernetes.io/projected/df69fbbc-b8cb-4681-9f8d-c3c6f788cde5-kube-api-access-pfn9q\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:01:06.452527 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:06.452487 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-208a0d-openshift-default-67457fb65c-ksfzw"] Nov 03 01:01:06.453519 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:06.453492 4421 cpu_manager.go:414] "containerMap: removing container" podUID="18d503d0-9683-4d73-9c38-1991dcc19e44" containerName="payload-processing" Nov 03 01:01:06.453519 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:06.453516 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18d503d0-9683-4d73-9c38-1991dcc19e44" containerName="payload-processing" Nov 03 01:01:06.453712 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:06.453536 4421 cpu_manager.go:414] "containerMap: removing container" podUID="84fca73a-b1e2-4fee-892f-b2f162bd75b3" containerName="cleanup" Nov 03 01:01:06.453712 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:06.453544 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84fca73a-b1e2-4fee-892f-b2f162bd75b3" containerName="cleanup" Nov 03 01:01:06.453712 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:06.453555 4421 cpu_manager.go:414] "containerMap: removing container" podUID="0370c2db-8545-4266-a4f4-719aebbd925b" containerName="cleanup" Nov 03 01:01:06.453712 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:06.453564 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0370c2db-8545-4266-a4f4-719aebbd925b" containerName="cleanup" Nov 03 01:01:06.453712 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:06.453573 4421 cpu_manager.go:414] "containerMap: removing container" podUID="569546b1-7662-4df6-a849-23fb4095797b" containerName="cleanup" Nov 03 01:01:06.453712 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:06.453580 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="569546b1-7662-4df6-a849-23fb4095797b" containerName="cleanup" Nov 03 01:01:06.453712 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:06.453598 4421 cpu_manager.go:414] "containerMap: removing container" podUID="569546b1-7662-4df6-a849-23fb4095797b" containerName="cleanup" Nov 03 01:01:06.453712 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:06.453605 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="569546b1-7662-4df6-a849-23fb4095797b" containerName="cleanup" Nov 03 01:01:06.453712 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:06.453619 4421 cpu_manager.go:414] "containerMap: removing container" podUID="03c838c5-5695-434a-b78f-a338c14f3365" containerName="cleanup" Nov 03 01:01:06.453712 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:06.453627 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03c838c5-5695-434a-b78f-a338c14f3365" containerName="cleanup" Nov 03 01:01:06.453712 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:06.453647 4421 cpu_manager.go:414] "containerMap: removing container" podUID="600dc1bf-8abc-480d-8503-3942ecd9d48b" containerName="cleanup" Nov 03 01:01:06.453712 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:06.453654 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="600dc1bf-8abc-480d-8503-3942ecd9d48b" containerName="cleanup" Nov 03 01:01:06.453712 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:06.453667 4421 cpu_manager.go:414] "containerMap: removing container" podUID="569546b1-7662-4df6-a849-23fb4095797b" containerName="cleanup" Nov 03 01:01:06.453712 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:06.453674 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="569546b1-7662-4df6-a849-23fb4095797b" containerName="cleanup" Nov 03 01:01:06.453712 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:06.453682 4421 cpu_manager.go:414] "containerMap: removing container" podUID="91abc456-0b19-4c43-9a94-57ef27f96c4a" containerName="cleanup" Nov 03 01:01:06.453712 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:06.453690 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91abc456-0b19-4c43-9a94-57ef27f96c4a" containerName="cleanup" Nov 03 01:01:06.453712 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:06.453701 4421 cpu_manager.go:414] "containerMap: removing container" podUID="048aae80-64d2-41b2-9e6a-6b6431d41bd9" containerName="payload-pre-processing" Nov 03 01:01:06.453712 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:06.453708 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="048aae80-64d2-41b2-9e6a-6b6431d41bd9" containerName="payload-pre-processing" Nov 03 01:01:06.453712 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:06.453722 4421 cpu_manager.go:414] "containerMap: removing container" podUID="c95ca370-7e69-4c57-912c-e953469cb21f" containerName="istio-proxy" Nov 03 01:01:06.454741 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:06.453729 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c95ca370-7e69-4c57-912c-e953469cb21f" containerName="istio-proxy" Nov 03 01:01:06.454741 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:06.453738 4421 cpu_manager.go:414] "containerMap: removing container" podUID="c8c543e5-5eda-4993-92af-8da6a226cfa6" containerName="cleanup" Nov 03 01:01:06.454741 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:06.453744 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8c543e5-5eda-4993-92af-8da6a226cfa6" containerName="cleanup" Nov 03 01:01:06.454741 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:06.453756 4421 cpu_manager.go:414] "containerMap: removing container" podUID="df69fbbc-b8cb-4681-9f8d-c3c6f788cde5" containerName="payload-processing" Nov 03 01:01:06.454741 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:06.453764 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df69fbbc-b8cb-4681-9f8d-c3c6f788cde5" containerName="payload-processing" Nov 03 01:01:06.454741 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:06.453775 4421 cpu_manager.go:414] "containerMap: removing container" podUID="0370c2db-8545-4266-a4f4-719aebbd925b" containerName="cleanup" Nov 03 01:01:06.454741 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:06.453782 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0370c2db-8545-4266-a4f4-719aebbd925b" containerName="cleanup" Nov 03 01:01:06.454741 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:06.453797 4421 cpu_manager.go:414] "containerMap: removing container" podUID="9bb1557e-955c-41d1-8513-e1f8c4f341f3" containerName="cleanup" Nov 03 01:01:06.454741 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:06.453803 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bb1557e-955c-41d1-8513-e1f8c4f341f3" containerName="cleanup" Nov 03 01:01:06.454741 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:06.453812 4421 cpu_manager.go:414] "containerMap: removing container" podUID="c8c543e5-5eda-4993-92af-8da6a226cfa6" containerName="cleanup" Nov 03 01:01:06.454741 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:06.453819 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8c543e5-5eda-4993-92af-8da6a226cfa6" containerName="cleanup" Nov 03 01:01:06.454741 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:06.453831 4421 cpu_manager.go:414] "containerMap: removing container" podUID="9bb1557e-955c-41d1-8513-e1f8c4f341f3" containerName="cleanup" Nov 03 01:01:06.454741 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:06.453838 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bb1557e-955c-41d1-8513-e1f8c4f341f3" containerName="cleanup" Nov 03 01:01:06.454741 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:06.453849 4421 cpu_manager.go:414] "containerMap: removing container" podUID="c8c543e5-5eda-4993-92af-8da6a226cfa6" containerName="cleanup" Nov 03 01:01:06.454741 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:06.453855 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8c543e5-5eda-4993-92af-8da6a226cfa6" containerName="cleanup" Nov 03 01:01:06.454741 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:06.453872 4421 cpu_manager.go:414] "containerMap: removing container" podUID="0370c2db-8545-4266-a4f4-719aebbd925b" containerName="cleanup" Nov 03 01:01:06.454741 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:06.453878 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0370c2db-8545-4266-a4f4-719aebbd925b" containerName="cleanup" Nov 03 01:01:06.454741 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:06.453894 4421 cpu_manager.go:414] "containerMap: removing container" podUID="91abc456-0b19-4c43-9a94-57ef27f96c4a" containerName="cleanup" Nov 03 01:01:06.454741 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:06.453900 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91abc456-0b19-4c43-9a94-57ef27f96c4a" containerName="cleanup" Nov 03 01:01:06.454741 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:06.453914 4421 cpu_manager.go:414] "containerMap: removing container" podUID="91abc456-0b19-4c43-9a94-57ef27f96c4a" containerName="cleanup" Nov 03 01:01:06.454741 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:06.453920 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91abc456-0b19-4c43-9a94-57ef27f96c4a" containerName="cleanup" Nov 03 01:01:06.454741 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:06.453936 4421 cpu_manager.go:414] "containerMap: removing container" podUID="64ad274e-6de7-4604-a8e4-958f67dbd00f" containerName="cleanup" Nov 03 01:01:06.454741 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:06.453943 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64ad274e-6de7-4604-a8e4-958f67dbd00f" containerName="cleanup" Nov 03 01:01:06.454741 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:06.453955 4421 cpu_manager.go:414] "containerMap: removing container" podUID="bd400742-fae4-44b7-b4af-497f60943da9" containerName="istio-proxy" Nov 03 01:01:06.454741 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:06.453962 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd400742-fae4-44b7-b4af-497f60943da9" containerName="istio-proxy" Nov 03 01:01:06.454741 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:06.453973 4421 cpu_manager.go:414] "containerMap: removing container" podUID="c95ca370-7e69-4c57-912c-e953469cb21f" containerName="istio-proxy" Nov 03 01:01:06.454741 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:06.453981 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c95ca370-7e69-4c57-912c-e953469cb21f" containerName="istio-proxy" Nov 03 01:01:06.454741 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:06.454001 4421 cpu_manager.go:414] "containerMap: removing container" podUID="64ad274e-6de7-4604-a8e4-958f67dbd00f" containerName="cleanup" Nov 03 01:01:06.454741 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:06.454009 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64ad274e-6de7-4604-a8e4-958f67dbd00f" containerName="cleanup" Nov 03 01:01:06.454741 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:06.454019 4421 cpu_manager.go:414] "containerMap: removing container" podUID="0920df17-fa3c-497f-814e-0a1be8a141d5" containerName="payload-pre-processing" Nov 03 01:01:06.454741 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:06.454026 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0920df17-fa3c-497f-814e-0a1be8a141d5" containerName="payload-pre-processing" Nov 03 01:01:06.454741 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:06.454037 4421 cpu_manager.go:414] "containerMap: removing container" podUID="84fca73a-b1e2-4fee-892f-b2f162bd75b3" containerName="cleanup" Nov 03 01:01:06.454741 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:06.454044 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84fca73a-b1e2-4fee-892f-b2f162bd75b3" containerName="cleanup" Nov 03 01:01:06.454741 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:06.454058 4421 cpu_manager.go:414] "containerMap: removing container" podUID="600dc1bf-8abc-480d-8503-3942ecd9d48b" containerName="cleanup" Nov 03 01:01:06.454741 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:06.454065 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="600dc1bf-8abc-480d-8503-3942ecd9d48b" containerName="cleanup" Nov 03 01:01:06.454741 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:06.454079 4421 cpu_manager.go:414] "containerMap: removing container" podUID="64ad274e-6de7-4604-a8e4-958f67dbd00f" containerName="cleanup" Nov 03 01:01:06.454741 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:06.454087 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64ad274e-6de7-4604-a8e4-958f67dbd00f" containerName="cleanup" Nov 03 01:01:06.454741 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:06.454097 4421 cpu_manager.go:414] "containerMap: removing container" podUID="f465a3d3-72ba-40a8-a6a2-dc5dd77efcce" containerName="payload-pre-processing" Nov 03 01:01:06.454741 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:06.454104 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f465a3d3-72ba-40a8-a6a2-dc5dd77efcce" containerName="payload-pre-processing" Nov 03 01:01:06.454741 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:06.454113 4421 cpu_manager.go:414] "containerMap: removing container" podUID="f4145d1f-cba3-4b46-a0ee-5ac18eec30ea" containerName="payload-processing" Nov 03 01:01:06.454741 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:06.454120 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4145d1f-cba3-4b46-a0ee-5ac18eec30ea" containerName="payload-processing" Nov 03 01:01:06.454741 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:06.454143 4421 cpu_manager.go:414] "containerMap: removing container" podUID="dc932a32-c09c-4792-a90c-c4f906034e9f" containerName="payload-pre-processing" Nov 03 01:01:06.454741 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:06.454150 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc932a32-c09c-4792-a90c-c4f906034e9f" containerName="payload-pre-processing" Nov 03 01:01:06.454741 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:06.454163 4421 cpu_manager.go:414] "containerMap: removing container" podUID="03c838c5-5695-434a-b78f-a338c14f3365" containerName="cleanup" Nov 03 01:01:06.454741 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:06.454170 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03c838c5-5695-434a-b78f-a338c14f3365" containerName="cleanup" Nov 03 01:01:06.454741 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:06.454342 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="569546b1-7662-4df6-a849-23fb4095797b" containerName="cleanup" Nov 03 01:01:06.454741 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:06.454360 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="03c838c5-5695-434a-b78f-a338c14f3365" containerName="cleanup" Nov 03 01:01:06.454741 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:06.454370 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="600dc1bf-8abc-480d-8503-3942ecd9d48b" containerName="cleanup" Nov 03 01:01:06.454741 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:06.454380 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="569546b1-7662-4df6-a849-23fb4095797b" containerName="cleanup" Nov 03 01:01:06.454741 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:06.454388 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="91abc456-0b19-4c43-9a94-57ef27f96c4a" containerName="cleanup" Nov 03 01:01:06.454741 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:06.454397 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="048aae80-64d2-41b2-9e6a-6b6431d41bd9" containerName="payload-pre-processing" Nov 03 01:01:06.454741 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:06.454406 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="c95ca370-7e69-4c57-912c-e953469cb21f" containerName="istio-proxy" Nov 03 01:01:06.454741 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:06.454416 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8c543e5-5eda-4993-92af-8da6a226cfa6" containerName="cleanup" Nov 03 01:01:06.454741 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:06.454425 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="df69fbbc-b8cb-4681-9f8d-c3c6f788cde5" containerName="payload-processing" Nov 03 01:01:06.454741 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:06.454434 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="0370c2db-8545-4266-a4f4-719aebbd925b" containerName="cleanup" Nov 03 01:01:06.454741 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:06.454445 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bb1557e-955c-41d1-8513-e1f8c4f341f3" containerName="cleanup" Nov 03 01:01:06.454741 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:06.454453 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8c543e5-5eda-4993-92af-8da6a226cfa6" containerName="cleanup" Nov 03 01:01:06.454741 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:06.454461 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bb1557e-955c-41d1-8513-e1f8c4f341f3" containerName="cleanup" Nov 03 01:01:06.454741 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:06.454483 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8c543e5-5eda-4993-92af-8da6a226cfa6" containerName="cleanup" Nov 03 01:01:06.454741 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:06.454492 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="600dc1bf-8abc-480d-8503-3942ecd9d48b" containerName="cleanup" Nov 03 01:01:06.454741 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:06.454502 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="0370c2db-8545-4266-a4f4-719aebbd925b" containerName="cleanup" Nov 03 01:01:06.454741 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:06.454513 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="91abc456-0b19-4c43-9a94-57ef27f96c4a" containerName="cleanup" Nov 03 01:01:06.454741 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:06.454520 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="91abc456-0b19-4c43-9a94-57ef27f96c4a" containerName="cleanup" Nov 03 01:01:06.454741 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:06.454529 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="64ad274e-6de7-4604-a8e4-958f67dbd00f" containerName="cleanup" Nov 03 01:01:06.454741 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:06.454538 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd400742-fae4-44b7-b4af-497f60943da9" containerName="istio-proxy" Nov 03 01:01:06.454741 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:06.454545 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="c95ca370-7e69-4c57-912c-e953469cb21f" containerName="istio-proxy" Nov 03 01:01:06.454741 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:06.454554 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="84fca73a-b1e2-4fee-892f-b2f162bd75b3" containerName="cleanup" Nov 03 01:01:06.454741 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:06.454563 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="64ad274e-6de7-4604-a8e4-958f67dbd00f" containerName="cleanup" Nov 03 01:01:06.454741 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:06.454570 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="0920df17-fa3c-497f-814e-0a1be8a141d5" containerName="payload-pre-processing" Nov 03 01:01:06.454741 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:06.454579 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="84fca73a-b1e2-4fee-892f-b2f162bd75b3" containerName="cleanup" Nov 03 01:01:06.454741 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:06.454589 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="64ad274e-6de7-4604-a8e4-958f67dbd00f" containerName="cleanup" Nov 03 01:01:06.454741 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:06.454599 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="f465a3d3-72ba-40a8-a6a2-dc5dd77efcce" containerName="payload-pre-processing" Nov 03 01:01:06.454741 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:06.454608 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4145d1f-cba3-4b46-a0ee-5ac18eec30ea" containerName="payload-processing" Nov 03 01:01:06.454741 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:06.454620 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc932a32-c09c-4792-a90c-c4f906034e9f" containerName="payload-pre-processing" Nov 03 01:01:06.454741 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:06.454629 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="03c838c5-5695-434a-b78f-a338c14f3365" containerName="cleanup" Nov 03 01:01:06.454741 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:06.454638 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="18d503d0-9683-4d73-9c38-1991dcc19e44" containerName="payload-processing" Nov 03 01:01:06.458652 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:06.457887 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-208a0d-openshift-default-67457fb65c-ksfzw" Nov 03 01:01:06.467823 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:06.464569 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-208a0d-openshift-default-dockercfg-ldk2l\"" Nov 03 01:01:06.484802 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:06.477593 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-208a0d-openshift-default-67457fb65c-ksfzw"] Nov 03 01:01:06.526977 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:06.526942 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a47649ab-48b3-4de0-b057-5445c923c427-workload-certs\") pod \"e2e-praxis-b-208a0d-openshift-default-67457fb65c-ksfzw\" (UID: \"a47649ab-48b3-4de0-b057-5445c923c427\") " pod="openshift-ingress/e2e-praxis-b-208a0d-openshift-default-67457fb65c-ksfzw" Nov 03 01:01:06.527185 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:06.527021 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a47649ab-48b3-4de0-b057-5445c923c427-workload-socket\") pod \"e2e-praxis-b-208a0d-openshift-default-67457fb65c-ksfzw\" (UID: \"a47649ab-48b3-4de0-b057-5445c923c427\") " pod="openshift-ingress/e2e-praxis-b-208a0d-openshift-default-67457fb65c-ksfzw" Nov 03 01:01:06.527185 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:06.527078 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a47649ab-48b3-4de0-b057-5445c923c427-istiod-ca-cert\") pod \"e2e-praxis-b-208a0d-openshift-default-67457fb65c-ksfzw\" (UID: \"a47649ab-48b3-4de0-b057-5445c923c427\") " pod="openshift-ingress/e2e-praxis-b-208a0d-openshift-default-67457fb65c-ksfzw" Nov 03 01:01:06.527185 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:06.527168 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rm65l\" (UniqueName: \"kubernetes.io/projected/a47649ab-48b3-4de0-b057-5445c923c427-kube-api-access-rm65l\") pod \"e2e-praxis-b-208a0d-openshift-default-67457fb65c-ksfzw\" (UID: \"a47649ab-48b3-4de0-b057-5445c923c427\") " pod="openshift-ingress/e2e-praxis-b-208a0d-openshift-default-67457fb65c-ksfzw" Nov 03 01:01:06.527381 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:06.527237 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a47649ab-48b3-4de0-b057-5445c923c427-istio-token\") pod \"e2e-praxis-b-208a0d-openshift-default-67457fb65c-ksfzw\" (UID: \"a47649ab-48b3-4de0-b057-5445c923c427\") " pod="openshift-ingress/e2e-praxis-b-208a0d-openshift-default-67457fb65c-ksfzw" Nov 03 01:01:06.527465 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:06.527441 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a47649ab-48b3-4de0-b057-5445c923c427-credential-socket\") pod \"e2e-praxis-b-208a0d-openshift-default-67457fb65c-ksfzw\" (UID: \"a47649ab-48b3-4de0-b057-5445c923c427\") " pod="openshift-ingress/e2e-praxis-b-208a0d-openshift-default-67457fb65c-ksfzw" Nov 03 01:01:06.527573 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:06.527558 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a47649ab-48b3-4de0-b057-5445c923c427-istio-podinfo\") pod \"e2e-praxis-b-208a0d-openshift-default-67457fb65c-ksfzw\" (UID: \"a47649ab-48b3-4de0-b057-5445c923c427\") " pod="openshift-ingress/e2e-praxis-b-208a0d-openshift-default-67457fb65c-ksfzw" Nov 03 01:01:06.527646 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:06.527631 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a47649ab-48b3-4de0-b057-5445c923c427-istio-envoy\") pod \"e2e-praxis-b-208a0d-openshift-default-67457fb65c-ksfzw\" (UID: \"a47649ab-48b3-4de0-b057-5445c923c427\") " pod="openshift-ingress/e2e-praxis-b-208a0d-openshift-default-67457fb65c-ksfzw" Nov 03 01:01:06.527784 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:06.527748 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a47649ab-48b3-4de0-b057-5445c923c427-istio-data\") pod \"e2e-praxis-b-208a0d-openshift-default-67457fb65c-ksfzw\" (UID: \"a47649ab-48b3-4de0-b057-5445c923c427\") " pod="openshift-ingress/e2e-praxis-b-208a0d-openshift-default-67457fb65c-ksfzw" Nov 03 01:01:06.545014 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:06.544261 4421 generic.go:356] "Generic (PLEG): container finished" podID="df69fbbc-b8cb-4681-9f8d-c3c6f788cde5" containerID="b623172352dd7a8ba50470b32d857820acbeaa047f75d8bda1a59f8568998f57" exitCode=0 Nov 03 01:01:06.545014 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:06.544463 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-def754-555d7884c4-k8sh9" Nov 03 01:01:06.545349 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:06.545318 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-def754-555d7884c4-k8sh9" event={"ID":"df69fbbc-b8cb-4681-9f8d-c3c6f788cde5","Type":"ContainerDied","Data":"b623172352dd7a8ba50470b32d857820acbeaa047f75d8bda1a59f8568998f57"} Nov 03 01:01:06.545419 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:06.545393 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-def754-555d7884c4-k8sh9" event={"ID":"df69fbbc-b8cb-4681-9f8d-c3c6f788cde5","Type":"ContainerDied","Data":"39351cc089b7c575ccd0ce235c75ad2ce17b8e144fc436764901cec7d728769f"} Nov 03 01:01:06.545462 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:06.545429 4421 scope.go:122] "RemoveContainer" containerID="b623172352dd7a8ba50470b32d857820acbeaa047f75d8bda1a59f8568998f57" Nov 03 01:01:06.549211 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:06.547570 4421 generic.go:356] "Generic (PLEG): container finished" podID="048aae80-64d2-41b2-9e6a-6b6431d41bd9" containerID="f59c1c4d16212ada56120d62dd80a5c5b3b8c2ad566f46a4bbcea5f129430786" exitCode=0 Nov 03 01:01:06.549211 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:06.547636 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-def754-744dbd79f-dm4jt" Nov 03 01:01:06.549211 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:06.547683 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-def754-744dbd79f-dm4jt" event={"ID":"048aae80-64d2-41b2-9e6a-6b6431d41bd9","Type":"ContainerDied","Data":"f59c1c4d16212ada56120d62dd80a5c5b3b8c2ad566f46a4bbcea5f129430786"} Nov 03 01:01:06.549211 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:06.547745 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-def754-744dbd79f-dm4jt" event={"ID":"048aae80-64d2-41b2-9e6a-6b6431d41bd9","Type":"ContainerDied","Data":"5df08e1904e03615077d2cf9f0db0432bb18abf2a078fe92eeb7cce78aa43dc6"} Nov 03 01:01:06.572842 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:06.572777 4421 scope.go:122] "RemoveContainer" containerID="b623172352dd7a8ba50470b32d857820acbeaa047f75d8bda1a59f8568998f57" Nov 03 01:01:06.573992 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:01:06.573832 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b623172352dd7a8ba50470b32d857820acbeaa047f75d8bda1a59f8568998f57\": container with ID starting with b623172352dd7a8ba50470b32d857820acbeaa047f75d8bda1a59f8568998f57 not found: ID does not exist" containerID="b623172352dd7a8ba50470b32d857820acbeaa047f75d8bda1a59f8568998f57" Nov 03 01:01:06.573992 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:06.573874 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b623172352dd7a8ba50470b32d857820acbeaa047f75d8bda1a59f8568998f57"} err="failed to get container status \"b623172352dd7a8ba50470b32d857820acbeaa047f75d8bda1a59f8568998f57\": rpc error: code = NotFound desc = could not find container \"b623172352dd7a8ba50470b32d857820acbeaa047f75d8bda1a59f8568998f57\": container with ID starting with b623172352dd7a8ba50470b32d857820acbeaa047f75d8bda1a59f8568998f57 not found: ID does not exist" Nov 03 01:01:06.573992 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:06.573899 4421 scope.go:122] "RemoveContainer" containerID="f59c1c4d16212ada56120d62dd80a5c5b3b8c2ad566f46a4bbcea5f129430786" Nov 03 01:01:06.698773 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:06.684805 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a47649ab-48b3-4de0-b057-5445c923c427-credential-socket\") pod \"e2e-praxis-b-208a0d-openshift-default-67457fb65c-ksfzw\" (UID: \"a47649ab-48b3-4de0-b057-5445c923c427\") " pod="openshift-ingress/e2e-praxis-b-208a0d-openshift-default-67457fb65c-ksfzw" Nov 03 01:01:06.723296 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:06.718763 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a47649ab-48b3-4de0-b057-5445c923c427-credential-socket\") pod \"e2e-praxis-b-208a0d-openshift-default-67457fb65c-ksfzw\" (UID: \"a47649ab-48b3-4de0-b057-5445c923c427\") " pod="openshift-ingress/e2e-praxis-b-208a0d-openshift-default-67457fb65c-ksfzw" Nov 03 01:01:06.723296 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:06.718824 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-def754-555d7884c4-k8sh9"] Nov 03 01:01:06.723296 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:06.684928 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a47649ab-48b3-4de0-b057-5445c923c427-istio-podinfo\") pod \"e2e-praxis-b-208a0d-openshift-default-67457fb65c-ksfzw\" (UID: \"a47649ab-48b3-4de0-b057-5445c923c427\") " pod="openshift-ingress/e2e-praxis-b-208a0d-openshift-default-67457fb65c-ksfzw" Nov 03 01:01:06.723296 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:06.721496 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a47649ab-48b3-4de0-b057-5445c923c427-istio-envoy\") pod \"e2e-praxis-b-208a0d-openshift-default-67457fb65c-ksfzw\" (UID: \"a47649ab-48b3-4de0-b057-5445c923c427\") " pod="openshift-ingress/e2e-praxis-b-208a0d-openshift-default-67457fb65c-ksfzw" Nov 03 01:01:06.723296 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:06.721547 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a47649ab-48b3-4de0-b057-5445c923c427-istio-data\") pod \"e2e-praxis-b-208a0d-openshift-default-67457fb65c-ksfzw\" (UID: \"a47649ab-48b3-4de0-b057-5445c923c427\") " pod="openshift-ingress/e2e-praxis-b-208a0d-openshift-default-67457fb65c-ksfzw" Nov 03 01:01:06.723296 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:06.721595 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a47649ab-48b3-4de0-b057-5445c923c427-workload-certs\") pod \"e2e-praxis-b-208a0d-openshift-default-67457fb65c-ksfzw\" (UID: \"a47649ab-48b3-4de0-b057-5445c923c427\") " pod="openshift-ingress/e2e-praxis-b-208a0d-openshift-default-67457fb65c-ksfzw" Nov 03 01:01:06.723296 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:06.721689 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a47649ab-48b3-4de0-b057-5445c923c427-workload-socket\") pod \"e2e-praxis-b-208a0d-openshift-default-67457fb65c-ksfzw\" (UID: \"a47649ab-48b3-4de0-b057-5445c923c427\") " pod="openshift-ingress/e2e-praxis-b-208a0d-openshift-default-67457fb65c-ksfzw" Nov 03 01:01:06.723296 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:06.721714 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a47649ab-48b3-4de0-b057-5445c923c427-istiod-ca-cert\") pod \"e2e-praxis-b-208a0d-openshift-default-67457fb65c-ksfzw\" (UID: \"a47649ab-48b3-4de0-b057-5445c923c427\") " pod="openshift-ingress/e2e-praxis-b-208a0d-openshift-default-67457fb65c-ksfzw" Nov 03 01:01:06.723296 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:06.721764 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rm65l\" (UniqueName: \"kubernetes.io/projected/a47649ab-48b3-4de0-b057-5445c923c427-kube-api-access-rm65l\") pod \"e2e-praxis-b-208a0d-openshift-default-67457fb65c-ksfzw\" (UID: \"a47649ab-48b3-4de0-b057-5445c923c427\") " pod="openshift-ingress/e2e-praxis-b-208a0d-openshift-default-67457fb65c-ksfzw" Nov 03 01:01:06.723296 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:06.721875 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a47649ab-48b3-4de0-b057-5445c923c427-istio-token\") pod \"e2e-praxis-b-208a0d-openshift-default-67457fb65c-ksfzw\" (UID: \"a47649ab-48b3-4de0-b057-5445c923c427\") " pod="openshift-ingress/e2e-praxis-b-208a0d-openshift-default-67457fb65c-ksfzw" Nov 03 01:01:06.724449 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:06.722587 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-def754-555d7884c4-k8sh9"] Nov 03 01:01:06.724449 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:06.722889 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a47649ab-48b3-4de0-b057-5445c923c427-workload-socket\") pod \"e2e-praxis-b-208a0d-openshift-default-67457fb65c-ksfzw\" (UID: \"a47649ab-48b3-4de0-b057-5445c923c427\") " pod="openshift-ingress/e2e-praxis-b-208a0d-openshift-default-67457fb65c-ksfzw" Nov 03 01:01:06.724643 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:06.723190 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a47649ab-48b3-4de0-b057-5445c923c427-istio-data\") pod \"e2e-praxis-b-208a0d-openshift-default-67457fb65c-ksfzw\" (UID: \"a47649ab-48b3-4de0-b057-5445c923c427\") " pod="openshift-ingress/e2e-praxis-b-208a0d-openshift-default-67457fb65c-ksfzw" Nov 03 01:01:06.724643 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:06.723430 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a47649ab-48b3-4de0-b057-5445c923c427-workload-certs\") pod \"e2e-praxis-b-208a0d-openshift-default-67457fb65c-ksfzw\" (UID: \"a47649ab-48b3-4de0-b057-5445c923c427\") " pod="openshift-ingress/e2e-praxis-b-208a0d-openshift-default-67457fb65c-ksfzw" Nov 03 01:01:06.724643 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:06.724014 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a47649ab-48b3-4de0-b057-5445c923c427-istiod-ca-cert\") pod \"e2e-praxis-b-208a0d-openshift-default-67457fb65c-ksfzw\" (UID: \"a47649ab-48b3-4de0-b057-5445c923c427\") " pod="openshift-ingress/e2e-praxis-b-208a0d-openshift-default-67457fb65c-ksfzw" Nov 03 01:01:06.731298 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:06.730044 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-def754-744dbd79f-dm4jt"] Nov 03 01:01:06.736297 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:06.735560 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-def754-744dbd79f-dm4jt"] Nov 03 01:01:06.742292 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:06.739558 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rm65l\" (UniqueName: \"kubernetes.io/projected/a47649ab-48b3-4de0-b057-5445c923c427-kube-api-access-rm65l\") pod \"e2e-praxis-b-208a0d-openshift-default-67457fb65c-ksfzw\" (UID: \"a47649ab-48b3-4de0-b057-5445c923c427\") " pod="openshift-ingress/e2e-praxis-b-208a0d-openshift-default-67457fb65c-ksfzw" Nov 03 01:01:06.742292 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:06.742136 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a47649ab-48b3-4de0-b057-5445c923c427-istio-token\") pod \"e2e-praxis-b-208a0d-openshift-default-67457fb65c-ksfzw\" (UID: \"a47649ab-48b3-4de0-b057-5445c923c427\") " pod="openshift-ingress/e2e-praxis-b-208a0d-openshift-default-67457fb65c-ksfzw" Nov 03 01:01:06.773297 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:06.763710 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a47649ab-48b3-4de0-b057-5445c923c427-istio-podinfo\") pod \"e2e-praxis-b-208a0d-openshift-default-67457fb65c-ksfzw\" (UID: \"a47649ab-48b3-4de0-b057-5445c923c427\") " pod="openshift-ingress/e2e-praxis-b-208a0d-openshift-default-67457fb65c-ksfzw" Nov 03 01:01:06.773297 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:06.767540 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a47649ab-48b3-4de0-b057-5445c923c427-istio-envoy\") pod \"e2e-praxis-b-208a0d-openshift-default-67457fb65c-ksfzw\" (UID: \"a47649ab-48b3-4de0-b057-5445c923c427\") " pod="openshift-ingress/e2e-praxis-b-208a0d-openshift-default-67457fb65c-ksfzw" Nov 03 01:01:06.790299 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:06.790155 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-208a0d-openshift-default-67457fb65c-ksfzw" Nov 03 01:01:06.799739 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:06.798179 4421 scope.go:122] "RemoveContainer" containerID="f59c1c4d16212ada56120d62dd80a5c5b3b8c2ad566f46a4bbcea5f129430786" Nov 03 01:01:06.799739 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:01:06.799534 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f59c1c4d16212ada56120d62dd80a5c5b3b8c2ad566f46a4bbcea5f129430786\": container with ID starting with f59c1c4d16212ada56120d62dd80a5c5b3b8c2ad566f46a4bbcea5f129430786 not found: ID does not exist" containerID="f59c1c4d16212ada56120d62dd80a5c5b3b8c2ad566f46a4bbcea5f129430786" Nov 03 01:01:06.799739 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:06.799593 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f59c1c4d16212ada56120d62dd80a5c5b3b8c2ad566f46a4bbcea5f129430786"} err="failed to get container status \"f59c1c4d16212ada56120d62dd80a5c5b3b8c2ad566f46a4bbcea5f129430786\": rpc error: code = NotFound desc = could not find container \"f59c1c4d16212ada56120d62dd80a5c5b3b8c2ad566f46a4bbcea5f129430786\": container with ID starting with f59c1c4d16212ada56120d62dd80a5c5b3b8c2ad566f46a4bbcea5f129430786 not found: ID does not exist" Nov 03 01:01:07.102438 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:07.102407 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-208a0d-openshift-default-67457fb65c-ksfzw"] Nov 03 01:01:07.103025 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:01:07.102994 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda47649ab_48b3_4de0_b057_5445c923c427.slice/crio-fbd3340cda97898e99d665c9b51ed26e349f1775c95a8a507ff0b9d9eec97852 WatchSource:0}: Error finding container fbd3340cda97898e99d665c9b51ed26e349f1775c95a8a507ff0b9d9eec97852: Status 404 returned error can't find the container with id fbd3340cda97898e99d665c9b51ed26e349f1775c95a8a507ff0b9d9eec97852 Nov 03 01:01:07.106448 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:07.106407 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:01:07.106558 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:07.106507 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:01:07.106598 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:07.106554 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:01:07.557099 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:07.557056 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-208a0d-openshift-default-67457fb65c-ksfzw" event={"ID":"a47649ab-48b3-4de0-b057-5445c923c427","Type":"ContainerStarted","Data":"e68dbf918c3fed3dce2f889f1e30293534ea755b4403b621185c3638abd9173a"} Nov 03 01:01:07.557339 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:07.557131 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-208a0d-openshift-default-67457fb65c-ksfzw" event={"ID":"a47649ab-48b3-4de0-b057-5445c923c427","Type":"ContainerStarted","Data":"fbd3340cda97898e99d665c9b51ed26e349f1775c95a8a507ff0b9d9eec97852"} Nov 03 01:01:07.582229 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:07.582176 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-208a0d-openshift-default-67457fb65c-ksfzw" podStartSLOduration=1.582161683 podStartE2EDuration="1.582161683s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:01:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:01:07.579121366 +0000 UTC m=+1209.750124466" watchObservedRunningTime="2026-11-03 01:01:07.582161683 +0000 UTC m=+1209.753164778" Nov 03 01:01:07.790603 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:07.790571 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-208a0d-openshift-default-67457fb65c-ksfzw" Nov 03 01:01:07.980256 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:07.980142 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="048aae80-64d2-41b2-9e6a-6b6431d41bd9" path="/var/lib/kubelet/pods/048aae80-64d2-41b2-9e6a-6b6431d41bd9/volumes" Nov 03 01:01:07.981100 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:07.980911 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df69fbbc-b8cb-4681-9f8d-c3c6f788cde5" path="/var/lib/kubelet/pods/df69fbbc-b8cb-4681-9f8d-c3c6f788cde5/volumes" Nov 03 01:01:08.797436 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:08.797403 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-208a0d-openshift-default-67457fb65c-ksfzw" Nov 03 01:01:09.576448 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:09.576418 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-208a0d-openshift-default-67457fb65c-ksfzw" Nov 03 01:01:09.577492 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:09.577472 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-208a0d-openshift-default-67457fb65c-ksfzw" Nov 03 01:01:14.672536 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:01:14.672458 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 01:01:16.120379 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:16.120340 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-fgwv5"] Nov 03 01:01:16.120953 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:16.120554 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-fgwv5" podUID="126b600b-80c4-4377-a1f8-fd5936d40bf4" containerName="istio-proxy" containerID="cri-o://7b20ee6ba3354f9d00feb66eec971a36f11fdd3f2ca069652dbfe3fb1b67714c" gracePeriod=30 Nov 03 01:01:17.445310 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:17.445251 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-d5f78f-6f874c67f9-llvzf"] Nov 03 01:01:17.446128 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:17.446106 4421 cpu_manager.go:414] "containerMap: removing container" podUID="9bb1557e-955c-41d1-8513-e1f8c4f341f3" containerName="cleanup" Nov 03 01:01:17.446237 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:17.446130 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bb1557e-955c-41d1-8513-e1f8c4f341f3" containerName="cleanup" Nov 03 01:01:17.446237 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:17.446158 4421 cpu_manager.go:414] "containerMap: removing container" podUID="bd400742-fae4-44b7-b4af-497f60943da9" containerName="istio-proxy" Nov 03 01:01:17.446237 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:17.446167 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd400742-fae4-44b7-b4af-497f60943da9" containerName="istio-proxy" Nov 03 01:01:17.446237 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:17.446188 4421 cpu_manager.go:414] "containerMap: removing container" podUID="600dc1bf-8abc-480d-8503-3942ecd9d48b" containerName="cleanup" Nov 03 01:01:17.446237 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:17.446196 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="600dc1bf-8abc-480d-8503-3942ecd9d48b" containerName="cleanup" Nov 03 01:01:17.446237 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:17.446234 4421 cpu_manager.go:414] "containerMap: removing container" podUID="84fca73a-b1e2-4fee-892f-b2f162bd75b3" containerName="cleanup" Nov 03 01:01:17.446632 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:17.446241 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84fca73a-b1e2-4fee-892f-b2f162bd75b3" containerName="cleanup" Nov 03 01:01:17.446632 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:17.446261 4421 cpu_manager.go:414] "containerMap: removing container" podUID="03c838c5-5695-434a-b78f-a338c14f3365" containerName="cleanup" Nov 03 01:01:17.446632 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:17.446286 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03c838c5-5695-434a-b78f-a338c14f3365" containerName="cleanup" Nov 03 01:01:17.446632 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:17.446441 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="03c838c5-5695-434a-b78f-a338c14f3365" containerName="cleanup" Nov 03 01:01:17.446632 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:17.446468 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="569546b1-7662-4df6-a849-23fb4095797b" containerName="cleanup" Nov 03 01:01:17.446632 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:17.446481 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bb1557e-955c-41d1-8513-e1f8c4f341f3" containerName="cleanup" Nov 03 01:01:17.446632 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:17.446492 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="600dc1bf-8abc-480d-8503-3942ecd9d48b" containerName="cleanup" Nov 03 01:01:17.446632 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:17.446503 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="0370c2db-8545-4266-a4f4-719aebbd925b" containerName="cleanup" Nov 03 01:01:17.446632 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:17.446513 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd400742-fae4-44b7-b4af-497f60943da9" containerName="istio-proxy" Nov 03 01:01:17.446632 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:17.446524 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="84fca73a-b1e2-4fee-892f-b2f162bd75b3" containerName="cleanup" Nov 03 01:01:17.450253 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:17.450231 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d5f78f-6f874c67f9-llvzf" Nov 03 01:01:17.455827 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:17.455769 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-d5f78f\"" Nov 03 01:01:17.456799 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:17.456780 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-d5f78f-dockercfg-kbxtl\"" Nov 03 01:01:17.459906 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:17.459880 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-d5f78f-6f874c67f9-llvzf"] Nov 03 01:01:17.484615 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:17.484578 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-d5f78f-d665ddc99-pgdkv"] Nov 03 01:01:17.489466 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:17.489442 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-d5f78f-d665ddc99-pgdkv" Nov 03 01:01:17.502639 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:17.502605 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-d5f78f-d665ddc99-pgdkv"] Nov 03 01:01:17.548547 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:17.548510 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rwdh\" (UniqueName: \"kubernetes.io/projected/1695554d-050f-4de0-bced-428c9325bead-kube-api-access-5rwdh\") pod \"payload-processing-e2e-praxis-a-d5f78f-d665ddc99-pgdkv\" (UID: \"1695554d-050f-4de0-bced-428c9325bead\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-d5f78f-d665ddc99-pgdkv" Nov 03 01:01:17.548873 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:17.548846 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b72d8af9-6f28-4ca8-a032-54a60293a7b3-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-d5f78f-6f874c67f9-llvzf\" (UID: \"b72d8af9-6f28-4ca8-a032-54a60293a7b3\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d5f78f-6f874c67f9-llvzf" Nov 03 01:01:17.549144 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:17.549126 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1695554d-050f-4de0-bced-428c9325bead-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-d5f78f-d665ddc99-pgdkv\" (UID: \"1695554d-050f-4de0-bced-428c9325bead\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-d5f78f-d665ddc99-pgdkv" Nov 03 01:01:17.549315 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:17.549292 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bn4vw\" (UniqueName: \"kubernetes.io/projected/b72d8af9-6f28-4ca8-a032-54a60293a7b3-kube-api-access-bn4vw\") pod \"payload-pre-processing-e2e-praxis-a-d5f78f-6f874c67f9-llvzf\" (UID: \"b72d8af9-6f28-4ca8-a032-54a60293a7b3\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d5f78f-6f874c67f9-llvzf" Nov 03 01:01:17.736005 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:17.735906 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rwdh\" (UniqueName: \"kubernetes.io/projected/1695554d-050f-4de0-bced-428c9325bead-kube-api-access-5rwdh\") pod \"payload-processing-e2e-praxis-a-d5f78f-d665ddc99-pgdkv\" (UID: \"1695554d-050f-4de0-bced-428c9325bead\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-d5f78f-d665ddc99-pgdkv" Nov 03 01:01:17.736338 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:17.736304 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b72d8af9-6f28-4ca8-a032-54a60293a7b3-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-d5f78f-6f874c67f9-llvzf\" (UID: \"b72d8af9-6f28-4ca8-a032-54a60293a7b3\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d5f78f-6f874c67f9-llvzf" Nov 03 01:01:17.736621 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:17.736603 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1695554d-050f-4de0-bced-428c9325bead-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-d5f78f-d665ddc99-pgdkv\" (UID: \"1695554d-050f-4de0-bced-428c9325bead\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-d5f78f-d665ddc99-pgdkv" Nov 03 01:01:17.736771 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:17.736752 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bn4vw\" (UniqueName: \"kubernetes.io/projected/b72d8af9-6f28-4ca8-a032-54a60293a7b3-kube-api-access-bn4vw\") pod \"payload-pre-processing-e2e-praxis-a-d5f78f-6f874c67f9-llvzf\" (UID: \"b72d8af9-6f28-4ca8-a032-54a60293a7b3\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d5f78f-6f874c67f9-llvzf" Nov 03 01:01:17.746599 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:17.746569 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b72d8af9-6f28-4ca8-a032-54a60293a7b3-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-d5f78f-6f874c67f9-llvzf\" (UID: \"b72d8af9-6f28-4ca8-a032-54a60293a7b3\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d5f78f-6f874c67f9-llvzf" Nov 03 01:01:17.747635 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:17.747610 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1695554d-050f-4de0-bced-428c9325bead-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-d5f78f-d665ddc99-pgdkv\" (UID: \"1695554d-050f-4de0-bced-428c9325bead\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-d5f78f-d665ddc99-pgdkv" Nov 03 01:01:17.802325 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:17.790974 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rwdh\" (UniqueName: \"kubernetes.io/projected/1695554d-050f-4de0-bced-428c9325bead-kube-api-access-5rwdh\") pod \"payload-processing-e2e-praxis-a-d5f78f-d665ddc99-pgdkv\" (UID: \"1695554d-050f-4de0-bced-428c9325bead\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-d5f78f-d665ddc99-pgdkv" Nov 03 01:01:17.860310 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:17.847310 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bn4vw\" (UniqueName: \"kubernetes.io/projected/b72d8af9-6f28-4ca8-a032-54a60293a7b3-kube-api-access-bn4vw\") pod \"payload-pre-processing-e2e-praxis-a-d5f78f-6f874c67f9-llvzf\" (UID: \"b72d8af9-6f28-4ca8-a032-54a60293a7b3\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d5f78f-6f874c67f9-llvzf" Nov 03 01:01:18.074438 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:18.074397 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d5f78f-6f874c67f9-llvzf" Nov 03 01:01:18.115947 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:18.115916 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-d5f78f-d665ddc99-pgdkv" Nov 03 01:01:18.386949 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:18.386909 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-d5f78f-6f874c67f9-llvzf"] Nov 03 01:01:18.388978 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:01:18.388944 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb72d8af9_6f28_4ca8_a032_54a60293a7b3.slice/crio-90600d5debbb5082923bfa38cb734a047556cd312e005bcb8a2c4d04a0f878a7 WatchSource:0}: Error finding container 90600d5debbb5082923bfa38cb734a047556cd312e005bcb8a2c4d04a0f878a7: Status 404 returned error can't find the container with id 90600d5debbb5082923bfa38cb734a047556cd312e005bcb8a2c4d04a0f878a7 Nov 03 01:01:18.421462 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:18.421430 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-d5f78f-d665ddc99-pgdkv"] Nov 03 01:01:18.426750 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:01:18.426721 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1695554d_050f_4de0_bced_428c9325bead.slice/crio-83e62eebbdf2194714c755d0bce4b56ba5ccf1bf92067c7d4cc9c307423a6f86 WatchSource:0}: Error finding container 83e62eebbdf2194714c755d0bce4b56ba5ccf1bf92067c7d4cc9c307423a6f86: Status 404 returned error can't find the container with id 83e62eebbdf2194714c755d0bce4b56ba5ccf1bf92067c7d4cc9c307423a6f86 Nov 03 01:01:18.852348 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:18.852303 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-d5f78f-d665ddc99-pgdkv" event={"ID":"1695554d-050f-4de0-bced-428c9325bead","Type":"ContainerStarted","Data":"1cea72e08e4324325f47ba1bf67dbc49ebeba890183393763ebb63b980d760e8"} Nov 03 01:01:18.852819 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:18.852375 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-d5f78f-d665ddc99-pgdkv" event={"ID":"1695554d-050f-4de0-bced-428c9325bead","Type":"ContainerStarted","Data":"83e62eebbdf2194714c755d0bce4b56ba5ccf1bf92067c7d4cc9c307423a6f86"} Nov 03 01:01:18.852819 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:18.852401 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-d5f78f-d665ddc99-pgdkv" Nov 03 01:01:18.854143 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:18.854118 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d5f78f-6f874c67f9-llvzf" event={"ID":"b72d8af9-6f28-4ca8-a032-54a60293a7b3","Type":"ContainerStarted","Data":"45c8b8632a8a87bc7ab3d849afc6e226092b2222b2db75528e8e32b3e20a342c"} Nov 03 01:01:18.854237 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:18.854167 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d5f78f-6f874c67f9-llvzf" event={"ID":"b72d8af9-6f28-4ca8-a032-54a60293a7b3","Type":"ContainerStarted","Data":"90600d5debbb5082923bfa38cb734a047556cd312e005bcb8a2c4d04a0f878a7"} Nov 03 01:01:18.854237 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:18.854189 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d5f78f-6f874c67f9-llvzf" Nov 03 01:01:18.872492 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:18.872439 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-d5f78f-d665ddc99-pgdkv" podStartSLOduration=1.872425712 podStartE2EDuration="1.872425712s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:01:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:01:18.868405227 +0000 UTC m=+1221.039408325" watchObservedRunningTime="2026-11-03 01:01:18.872425712 +0000 UTC m=+1221.043428807" Nov 03 01:01:18.886550 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:18.886509 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d5f78f-6f874c67f9-llvzf" podStartSLOduration=1.886496247 podStartE2EDuration="1.886496247s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:01:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:01:18.884617193 +0000 UTC m=+1221.055620290" watchObservedRunningTime="2026-11-03 01:01:18.886496247 +0000 UTC m=+1221.057499342" Nov 03 01:01:21.404239 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:21.404169 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-fgwv5" Nov 03 01:01:21.417672 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:21.417640 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/126b600b-80c4-4377-a1f8-fd5936d40bf4-kube-api-access-cmxn8\" (UniqueName: \"kubernetes.io/projected/126b600b-80c4-4377-a1f8-fd5936d40bf4-kube-api-access-cmxn8\") pod \"126b600b-80c4-4377-a1f8-fd5936d40bf4\" (UID: \"126b600b-80c4-4377-a1f8-fd5936d40bf4\") " Nov 03 01:01:21.417806 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:21.417747 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/126b600b-80c4-4377-a1f8-fd5936d40bf4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/126b600b-80c4-4377-a1f8-fd5936d40bf4-credential-socket\") pod \"126b600b-80c4-4377-a1f8-fd5936d40bf4\" (UID: \"126b600b-80c4-4377-a1f8-fd5936d40bf4\") " Nov 03 01:01:21.417806 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:21.417788 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/126b600b-80c4-4377-a1f8-fd5936d40bf4-istio-token\" (UniqueName: \"kubernetes.io/projected/126b600b-80c4-4377-a1f8-fd5936d40bf4-istio-token\") pod \"126b600b-80c4-4377-a1f8-fd5936d40bf4\" (UID: \"126b600b-80c4-4377-a1f8-fd5936d40bf4\") " Nov 03 01:01:21.417884 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:21.417827 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/126b600b-80c4-4377-a1f8-fd5936d40bf4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/126b600b-80c4-4377-a1f8-fd5936d40bf4-istio-podinfo\") pod \"126b600b-80c4-4377-a1f8-fd5936d40bf4\" (UID: \"126b600b-80c4-4377-a1f8-fd5936d40bf4\") " Nov 03 01:01:21.417884 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:21.417866 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/126b600b-80c4-4377-a1f8-fd5936d40bf4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/126b600b-80c4-4377-a1f8-fd5936d40bf4-istio-data\") pod \"126b600b-80c4-4377-a1f8-fd5936d40bf4\" (UID: \"126b600b-80c4-4377-a1f8-fd5936d40bf4\") " Nov 03 01:01:21.417975 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:21.417905 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/126b600b-80c4-4377-a1f8-fd5936d40bf4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/126b600b-80c4-4377-a1f8-fd5936d40bf4-workload-socket\") pod \"126b600b-80c4-4377-a1f8-fd5936d40bf4\" (UID: \"126b600b-80c4-4377-a1f8-fd5936d40bf4\") " Nov 03 01:01:21.418033 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:21.418022 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/126b600b-80c4-4377-a1f8-fd5936d40bf4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/126b600b-80c4-4377-a1f8-fd5936d40bf4-istiod-ca-cert\") pod \"126b600b-80c4-4377-a1f8-fd5936d40bf4\" (UID: \"126b600b-80c4-4377-a1f8-fd5936d40bf4\") " Nov 03 01:01:21.418098 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:21.418067 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/126b600b-80c4-4377-a1f8-fd5936d40bf4-credential-socket" pod "126b600b-80c4-4377-a1f8-fd5936d40bf4" (UID: "126b600b-80c4-4377-a1f8-fd5936d40bf4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:01:21.418156 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:21.418129 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/126b600b-80c4-4377-a1f8-fd5936d40bf4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/126b600b-80c4-4377-a1f8-fd5936d40bf4-istio-envoy\") pod \"126b600b-80c4-4377-a1f8-fd5936d40bf4\" (UID: \"126b600b-80c4-4377-a1f8-fd5936d40bf4\") " Nov 03 01:01:21.418240 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:21.418176 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/126b600b-80c4-4377-a1f8-fd5936d40bf4-istio-data" pod "126b600b-80c4-4377-a1f8-fd5936d40bf4" (UID: "126b600b-80c4-4377-a1f8-fd5936d40bf4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:01:21.418240 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:21.418227 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/126b600b-80c4-4377-a1f8-fd5936d40bf4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/126b600b-80c4-4377-a1f8-fd5936d40bf4-workload-certs\") pod \"126b600b-80c4-4377-a1f8-fd5936d40bf4\" (UID: \"126b600b-80c4-4377-a1f8-fd5936d40bf4\") " Nov 03 01:01:21.418397 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:21.418249 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/126b600b-80c4-4377-a1f8-fd5936d40bf4-workload-socket" pod "126b600b-80c4-4377-a1f8-fd5936d40bf4" (UID: "126b600b-80c4-4377-a1f8-fd5936d40bf4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:01:21.418470 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:21.418422 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/126b600b-80c4-4377-a1f8-fd5936d40bf4-istiod-ca-cert" pod "126b600b-80c4-4377-a1f8-fd5936d40bf4" (UID: "126b600b-80c4-4377-a1f8-fd5936d40bf4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:01:21.418729 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:21.418687 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/126b600b-80c4-4377-a1f8-fd5936d40bf4-workload-certs" pod "126b600b-80c4-4377-a1f8-fd5936d40bf4" (UID: "126b600b-80c4-4377-a1f8-fd5936d40bf4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:01:21.419031 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:21.419009 4421 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/126b600b-80c4-4377-a1f8-fd5936d40bf4-credential-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:01:21.419083 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:21.419040 4421 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/126b600b-80c4-4377-a1f8-fd5936d40bf4-istio-data\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:01:21.419083 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:21.419055 4421 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/126b600b-80c4-4377-a1f8-fd5936d40bf4-workload-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:01:21.419083 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:21.419069 4421 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/126b600b-80c4-4377-a1f8-fd5936d40bf4-istiod-ca-cert\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:01:21.419182 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:21.419084 4421 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/126b600b-80c4-4377-a1f8-fd5936d40bf4-workload-certs\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:01:21.420096 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:21.420074 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/126b600b-80c4-4377-a1f8-fd5936d40bf4-istio-token" pod "126b600b-80c4-4377-a1f8-fd5936d40bf4" (UID: "126b600b-80c4-4377-a1f8-fd5936d40bf4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:01:21.420181 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:21.420160 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/126b600b-80c4-4377-a1f8-fd5936d40bf4-kube-api-access-cmxn8" pod "126b600b-80c4-4377-a1f8-fd5936d40bf4" (UID: "126b600b-80c4-4377-a1f8-fd5936d40bf4"). InnerVolumeSpecName "kube-api-access-cmxn8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:01:21.420493 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:21.420477 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/126b600b-80c4-4377-a1f8-fd5936d40bf4-istio-podinfo" pod "126b600b-80c4-4377-a1f8-fd5936d40bf4" (UID: "126b600b-80c4-4377-a1f8-fd5936d40bf4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:01:21.420595 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:21.420574 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/126b600b-80c4-4377-a1f8-fd5936d40bf4-istio-envoy" pod "126b600b-80c4-4377-a1f8-fd5936d40bf4" (UID: "126b600b-80c4-4377-a1f8-fd5936d40bf4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:01:21.539583 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:21.539548 4421 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/126b600b-80c4-4377-a1f8-fd5936d40bf4-istio-token\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:01:21.539823 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:21.539803 4421 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/126b600b-80c4-4377-a1f8-fd5936d40bf4-istio-podinfo\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:01:21.539976 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:21.539962 4421 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/126b600b-80c4-4377-a1f8-fd5936d40bf4-istio-envoy\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:01:21.540081 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:21.540066 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmxn8\" (UniqueName: \"kubernetes.io/projected/126b600b-80c4-4377-a1f8-fd5936d40bf4-kube-api-access-cmxn8\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:01:21.879648 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:21.879607 4421 generic.go:356] "Generic (PLEG): container finished" podID="126b600b-80c4-4377-a1f8-fd5936d40bf4" containerID="7b20ee6ba3354f9d00feb66eec971a36f11fdd3f2ca069652dbfe3fb1b67714c" exitCode=0 Nov 03 01:01:21.879812 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:21.879664 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-fgwv5" Nov 03 01:01:21.879812 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:21.879680 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-fgwv5" event={"ID":"126b600b-80c4-4377-a1f8-fd5936d40bf4","Type":"ContainerDied","Data":"7b20ee6ba3354f9d00feb66eec971a36f11fdd3f2ca069652dbfe3fb1b67714c"} Nov 03 01:01:21.879812 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:21.879736 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-fgwv5" event={"ID":"126b600b-80c4-4377-a1f8-fd5936d40bf4","Type":"ContainerDied","Data":"cc2c7a4927155f4124ca05dfb2c4c4d0afb354f924f7e11bede43efa57d1c455"} Nov 03 01:01:21.879812 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:21.879768 4421 scope.go:122] "RemoveContainer" containerID="7b20ee6ba3354f9d00feb66eec971a36f11fdd3f2ca069652dbfe3fb1b67714c" Nov 03 01:01:21.891107 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:21.891082 4421 scope.go:122] "RemoveContainer" containerID="7b20ee6ba3354f9d00feb66eec971a36f11fdd3f2ca069652dbfe3fb1b67714c" Nov 03 01:01:21.891381 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:01:21.891360 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b20ee6ba3354f9d00feb66eec971a36f11fdd3f2ca069652dbfe3fb1b67714c\": container with ID starting with 7b20ee6ba3354f9d00feb66eec971a36f11fdd3f2ca069652dbfe3fb1b67714c not found: ID does not exist" containerID="7b20ee6ba3354f9d00feb66eec971a36f11fdd3f2ca069652dbfe3fb1b67714c" Nov 03 01:01:21.891459 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:21.891389 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b20ee6ba3354f9d00feb66eec971a36f11fdd3f2ca069652dbfe3fb1b67714c"} err="failed to get container status \"7b20ee6ba3354f9d00feb66eec971a36f11fdd3f2ca069652dbfe3fb1b67714c\": rpc error: code = NotFound desc = could not find container \"7b20ee6ba3354f9d00feb66eec971a36f11fdd3f2ca069652dbfe3fb1b67714c\": container with ID starting with 7b20ee6ba3354f9d00feb66eec971a36f11fdd3f2ca069652dbfe3fb1b67714c not found: ID does not exist" Nov 03 01:01:21.906641 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:21.906606 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-fgwv5"] Nov 03 01:01:21.911302 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:21.911253 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-fgwv5"] Nov 03 01:01:21.980113 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:21.980084 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="126b600b-80c4-4377-a1f8-fd5936d40bf4" path="/var/lib/kubelet/pods/126b600b-80c4-4377-a1f8-fd5936d40bf4/volumes" Nov 03 01:01:29.865365 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:29.865330 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-d5f78f-d665ddc99-pgdkv" Nov 03 01:01:29.865956 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:29.865932 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d5f78f-6f874c67f9-llvzf" Nov 03 01:01:42.671710 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:42.671677 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-dc0f3f-7b78d8bfdb-4s767"] Nov 03 01:01:42.672366 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:42.672350 4421 cpu_manager.go:414] "containerMap: removing container" podUID="126b600b-80c4-4377-a1f8-fd5936d40bf4" containerName="istio-proxy" Nov 03 01:01:42.672428 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:42.672367 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="126b600b-80c4-4377-a1f8-fd5936d40bf4" containerName="istio-proxy" Nov 03 01:01:42.672556 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:42.672545 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="126b600b-80c4-4377-a1f8-fd5936d40bf4" containerName="istio-proxy" Nov 03 01:01:42.677811 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:42.677790 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-dc0f3f-7b78d8bfdb-4s767" Nov 03 01:01:42.682394 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:42.682366 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-dc0f3f\"" Nov 03 01:01:42.682520 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:42.682472 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-dc0f3f-dockercfg-2vwcs\"" Nov 03 01:01:42.686900 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:42.686863 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-dc0f3f-7b78d8bfdb-4s767"] Nov 03 01:01:42.719253 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:42.719223 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-dc0f3f-59446658cf-n6msk"] Nov 03 01:01:42.723652 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:42.723629 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dc0f3f-59446658cf-n6msk" Nov 03 01:01:42.734675 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:42.734651 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-dc0f3f-59446658cf-n6msk"] Nov 03 01:01:42.764721 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:42.764680 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqrrr\" (UniqueName: \"kubernetes.io/projected/f894ad12-9d4e-4133-860c-b053bbbaf136-kube-api-access-xqrrr\") pod \"payload-processing-e2e-ipp-a-dc0f3f-7b78d8bfdb-4s767\" (UID: \"f894ad12-9d4e-4133-860c-b053bbbaf136\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-dc0f3f-7b78d8bfdb-4s767" Nov 03 01:01:42.765012 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:42.764988 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ffa6158-c563-4d0e-91db-d5b6665bebb3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-dc0f3f-59446658cf-n6msk\" (UID: \"9ffa6158-c563-4d0e-91db-d5b6665bebb3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dc0f3f-59446658cf-n6msk" Nov 03 01:01:42.765128 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:42.765104 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f894ad12-9d4e-4133-860c-b053bbbaf136-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-dc0f3f-7b78d8bfdb-4s767\" (UID: \"f894ad12-9d4e-4133-860c-b053bbbaf136\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-dc0f3f-7b78d8bfdb-4s767" Nov 03 01:01:42.765259 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:42.765237 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjgsk\" (UniqueName: \"kubernetes.io/projected/9ffa6158-c563-4d0e-91db-d5b6665bebb3-kube-api-access-sjgsk\") pod \"payload-pre-processing-e2e-ipp-a-dc0f3f-59446658cf-n6msk\" (UID: \"9ffa6158-c563-4d0e-91db-d5b6665bebb3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dc0f3f-59446658cf-n6msk" Nov 03 01:01:42.912301 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:42.895719 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ffa6158-c563-4d0e-91db-d5b6665bebb3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-dc0f3f-59446658cf-n6msk\" (UID: \"9ffa6158-c563-4d0e-91db-d5b6665bebb3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dc0f3f-59446658cf-n6msk" Nov 03 01:01:42.946072 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:42.903290 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ffa6158-c563-4d0e-91db-d5b6665bebb3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-dc0f3f-59446658cf-n6msk\" (UID: \"9ffa6158-c563-4d0e-91db-d5b6665bebb3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dc0f3f-59446658cf-n6msk" Nov 03 01:01:42.947996 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:42.946521 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f894ad12-9d4e-4133-860c-b053bbbaf136-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-dc0f3f-7b78d8bfdb-4s767\" (UID: \"f894ad12-9d4e-4133-860c-b053bbbaf136\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-dc0f3f-7b78d8bfdb-4s767" Nov 03 01:01:42.948309 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:42.947966 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f894ad12-9d4e-4133-860c-b053bbbaf136-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-dc0f3f-7b78d8bfdb-4s767\" (UID: \"f894ad12-9d4e-4133-860c-b053bbbaf136\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-dc0f3f-7b78d8bfdb-4s767" Nov 03 01:01:42.949130 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:42.948684 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjgsk\" (UniqueName: \"kubernetes.io/projected/9ffa6158-c563-4d0e-91db-d5b6665bebb3-kube-api-access-sjgsk\") pod \"payload-pre-processing-e2e-ipp-a-dc0f3f-59446658cf-n6msk\" (UID: \"9ffa6158-c563-4d0e-91db-d5b6665bebb3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dc0f3f-59446658cf-n6msk" Nov 03 01:01:42.949443 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:42.949405 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqrrr\" (UniqueName: \"kubernetes.io/projected/f894ad12-9d4e-4133-860c-b053bbbaf136-kube-api-access-xqrrr\") pod \"payload-processing-e2e-ipp-a-dc0f3f-7b78d8bfdb-4s767\" (UID: \"f894ad12-9d4e-4133-860c-b053bbbaf136\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-dc0f3f-7b78d8bfdb-4s767" Nov 03 01:01:42.976049 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:42.976011 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqrrr\" (UniqueName: \"kubernetes.io/projected/f894ad12-9d4e-4133-860c-b053bbbaf136-kube-api-access-xqrrr\") pod \"payload-processing-e2e-ipp-a-dc0f3f-7b78d8bfdb-4s767\" (UID: \"f894ad12-9d4e-4133-860c-b053bbbaf136\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-dc0f3f-7b78d8bfdb-4s767" Nov 03 01:01:42.991779 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:42.991742 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjgsk\" (UniqueName: \"kubernetes.io/projected/9ffa6158-c563-4d0e-91db-d5b6665bebb3-kube-api-access-sjgsk\") pod \"payload-pre-processing-e2e-ipp-a-dc0f3f-59446658cf-n6msk\" (UID: \"9ffa6158-c563-4d0e-91db-d5b6665bebb3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dc0f3f-59446658cf-n6msk" Nov 03 01:01:43.000254 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:43.000217 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-dc0f3f-7b78d8bfdb-4s767" Nov 03 01:01:43.044501 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:43.044467 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dc0f3f-59446658cf-n6msk" Nov 03 01:01:43.317977 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:43.317950 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-dc0f3f-7b78d8bfdb-4s767"] Nov 03 01:01:43.319934 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:01:43.319905 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf894ad12_9d4e_4133_860c_b053bbbaf136.slice/crio-f98d231c2bab6183dba1b9d2a0f56dcd83d34a1d5c9d2d51a06d58ba39cf4f28 WatchSource:0}: Error finding container f98d231c2bab6183dba1b9d2a0f56dcd83d34a1d5c9d2d51a06d58ba39cf4f28: Status 404 returned error can't find the container with id f98d231c2bab6183dba1b9d2a0f56dcd83d34a1d5c9d2d51a06d58ba39cf4f28 Nov 03 01:01:43.363201 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:43.363160 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-dc0f3f-59446658cf-n6msk"] Nov 03 01:01:44.159175 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:44.158861 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dc0f3f-59446658cf-n6msk" event={"ID":"9ffa6158-c563-4d0e-91db-d5b6665bebb3","Type":"ContainerStarted","Data":"98198872aec313e9d9129ebc5355a7602365a4de9803af2abcf7eed8e86247ce"} Nov 03 01:01:44.159175 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:44.158941 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dc0f3f-59446658cf-n6msk" event={"ID":"9ffa6158-c563-4d0e-91db-d5b6665bebb3","Type":"ContainerStarted","Data":"c4a973da8f02b50af97187ff36fac67f37541e832c8a5f7b1e5ed3c5d2182a6c"} Nov 03 01:01:44.159777 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:44.159664 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dc0f3f-59446658cf-n6msk" Nov 03 01:01:44.161673 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:44.161627 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-dc0f3f-7b78d8bfdb-4s767" event={"ID":"f894ad12-9d4e-4133-860c-b053bbbaf136","Type":"ContainerStarted","Data":"d63dccf3bb816de8905099f315280f1fd57d15720dbcec16e154fb0bb410f98b"} Nov 03 01:01:44.161673 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:44.161677 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-dc0f3f-7b78d8bfdb-4s767" event={"ID":"f894ad12-9d4e-4133-860c-b053bbbaf136","Type":"ContainerStarted","Data":"f98d231c2bab6183dba1b9d2a0f56dcd83d34a1d5c9d2d51a06d58ba39cf4f28"} Nov 03 01:01:44.161885 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:44.161712 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-dc0f3f-7b78d8bfdb-4s767" Nov 03 01:01:44.189807 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:44.189258 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dc0f3f-59446658cf-n6msk" podStartSLOduration=2.189240812 podStartE2EDuration="2.189240812s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:01:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:01:44.185743517 +0000 UTC m=+1246.356746609" watchObservedRunningTime="2026-11-03 01:01:44.189240812 +0000 UTC m=+1246.360243911" Nov 03 01:01:44.212632 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:44.212573 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-dc0f3f-7b78d8bfdb-4s767" podStartSLOduration=2.21255392 podStartE2EDuration="2.21255392s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:01:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:01:44.209614649 +0000 UTC m=+1246.380617745" watchObservedRunningTime="2026-11-03 01:01:44.21255392 +0000 UTC m=+1246.383557020" Nov 03 01:01:44.673478 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:01:44.673439 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 01:01:49.892383 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:49.892341 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-f88f87-openshift-default-7b6c9d6885-g92w2"] Nov 03 01:01:49.892906 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:49.892619 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-f88f87-openshift-default-7b6c9d6885-g92w2" podUID="e7200ec5-9b07-4003-9652-a92c5a1ab5de" containerName="istio-proxy" containerID="cri-o://b1011411c1aa9b70e8cfa688e51cb1e3f6c168a17dd72271a89cb0f90f7a38e7" gracePeriod=30 Nov 03 01:01:50.901160 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:50.901112 4421 patch_prober.go:29] interesting pod/e2e-ipp-a-f88f87-openshift-default-7b6c9d6885-g92w2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 01:01:50.901596 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:50.901185 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-f88f87-openshift-default-7b6c9d6885-g92w2" podUID="e7200ec5-9b07-4003-9652-a92c5a1ab5de" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 01:01:52.631094 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:52.630867 4421 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-07f5f6-6448498d8b-k8grc_4c232835-2f2e-4ee2-b646-9c56f413919c/payload-processing/0.log" Nov 03 01:01:53.250308 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:53.234069 4421 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-db5422-8464b57b74-pgngf_82a61b7a-409c-4939-8715-1114dab89b24/payload-processing/0.log" Nov 03 01:01:53.418296 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:53.418248 4421 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-db5422-8464b57b74-pgngf_82a61b7a-409c-4939-8715-1114dab89b24/payload-processing/0.log" Nov 03 01:01:54.402215 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:54.402175 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-cmdm8"] Nov 03 01:01:54.402724 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:54.402393 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-cmdm8" podUID="9916b060-ab92-4836-afab-cd531100dd69" containerName="istio-proxy" containerID="cri-o://b03ef5608292522d186a3c306168b9ba29d7732b01214764dd988da139dcef5c" gracePeriod=30 Nov 03 01:01:54.407503 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:54.407471 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-zrdp5"] Nov 03 01:01:54.407739 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:54.407702 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-zrdp5" podUID="7b6779b2-2b64-40e9-821c-5444681cea09" containerName="istio-proxy" containerID="cri-o://817dc1736b03a03d7b7981117dcc9a84c8fbe8722105c1052ea00962ca3ee5e1" gracePeriod=30 Nov 03 01:01:55.171842 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:55.170811 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-dc0f3f-7b78d8bfdb-4s767" Nov 03 01:01:55.216338 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:55.216262 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f88f87-openshift-default-7b6c9d6885-g92w2" Nov 03 01:01:55.318843 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:55.318786 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7200ec5-9b07-4003-9652-a92c5a1ab5de-istio-token\" (UniqueName: \"kubernetes.io/projected/e7200ec5-9b07-4003-9652-a92c5a1ab5de-istio-token\") pod \"e7200ec5-9b07-4003-9652-a92c5a1ab5de\" (UID: \"e7200ec5-9b07-4003-9652-a92c5a1ab5de\") " Nov 03 01:01:55.319071 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:55.318999 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7200ec5-9b07-4003-9652-a92c5a1ab5de-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7200ec5-9b07-4003-9652-a92c5a1ab5de-istio-data\") pod \"e7200ec5-9b07-4003-9652-a92c5a1ab5de\" (UID: \"e7200ec5-9b07-4003-9652-a92c5a1ab5de\") " Nov 03 01:01:55.319146 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:55.319072 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7200ec5-9b07-4003-9652-a92c5a1ab5de-kube-api-access-vzwj4\" (UniqueName: \"kubernetes.io/projected/e7200ec5-9b07-4003-9652-a92c5a1ab5de-kube-api-access-vzwj4\") pod \"e7200ec5-9b07-4003-9652-a92c5a1ab5de\" (UID: \"e7200ec5-9b07-4003-9652-a92c5a1ab5de\") " Nov 03 01:01:55.319146 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:55.319125 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7200ec5-9b07-4003-9652-a92c5a1ab5de-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7200ec5-9b07-4003-9652-a92c5a1ab5de-workload-certs\") pod \"e7200ec5-9b07-4003-9652-a92c5a1ab5de\" (UID: \"e7200ec5-9b07-4003-9652-a92c5a1ab5de\") " Nov 03 01:01:55.319261 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:55.319199 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7200ec5-9b07-4003-9652-a92c5a1ab5de-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7200ec5-9b07-4003-9652-a92c5a1ab5de-istio-envoy\") pod \"e7200ec5-9b07-4003-9652-a92c5a1ab5de\" (UID: \"e7200ec5-9b07-4003-9652-a92c5a1ab5de\") " Nov 03 01:01:55.319361 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:55.319315 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7200ec5-9b07-4003-9652-a92c5a1ab5de-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7200ec5-9b07-4003-9652-a92c5a1ab5de-credential-socket\") pod \"e7200ec5-9b07-4003-9652-a92c5a1ab5de\" (UID: \"e7200ec5-9b07-4003-9652-a92c5a1ab5de\") " Nov 03 01:01:55.319425 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:55.319366 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7200ec5-9b07-4003-9652-a92c5a1ab5de-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7200ec5-9b07-4003-9652-a92c5a1ab5de-istiod-ca-cert\") pod \"e7200ec5-9b07-4003-9652-a92c5a1ab5de\" (UID: \"e7200ec5-9b07-4003-9652-a92c5a1ab5de\") " Nov 03 01:01:55.319425 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:55.319403 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7200ec5-9b07-4003-9652-a92c5a1ab5de-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7200ec5-9b07-4003-9652-a92c5a1ab5de-workload-socket\") pod \"e7200ec5-9b07-4003-9652-a92c5a1ab5de\" (UID: \"e7200ec5-9b07-4003-9652-a92c5a1ab5de\") " Nov 03 01:01:55.319539 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:55.319487 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e7200ec5-9b07-4003-9652-a92c5a1ab5de-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7200ec5-9b07-4003-9652-a92c5a1ab5de-istio-podinfo\") pod \"e7200ec5-9b07-4003-9652-a92c5a1ab5de\" (UID: \"e7200ec5-9b07-4003-9652-a92c5a1ab5de\") " Nov 03 01:01:55.320545 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:55.319812 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7200ec5-9b07-4003-9652-a92c5a1ab5de-workload-socket" pod "e7200ec5-9b07-4003-9652-a92c5a1ab5de" (UID: "e7200ec5-9b07-4003-9652-a92c5a1ab5de"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:01:55.320545 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:55.319857 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7200ec5-9b07-4003-9652-a92c5a1ab5de-workload-certs" pod "e7200ec5-9b07-4003-9652-a92c5a1ab5de" (UID: "e7200ec5-9b07-4003-9652-a92c5a1ab5de"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:01:55.320545 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:55.319906 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7200ec5-9b07-4003-9652-a92c5a1ab5de-istiod-ca-cert" pod "e7200ec5-9b07-4003-9652-a92c5a1ab5de" (UID: "e7200ec5-9b07-4003-9652-a92c5a1ab5de"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:01:55.320545 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:55.320123 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7200ec5-9b07-4003-9652-a92c5a1ab5de-istio-data" pod "e7200ec5-9b07-4003-9652-a92c5a1ab5de" (UID: "e7200ec5-9b07-4003-9652-a92c5a1ab5de"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:01:55.320545 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:55.320362 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7200ec5-9b07-4003-9652-a92c5a1ab5de-credential-socket" pod "e7200ec5-9b07-4003-9652-a92c5a1ab5de" (UID: "e7200ec5-9b07-4003-9652-a92c5a1ab5de"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:01:55.320930 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:55.320660 4421 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7200ec5-9b07-4003-9652-a92c5a1ab5de-istio-data\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:01:55.320930 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:55.320682 4421 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7200ec5-9b07-4003-9652-a92c5a1ab5de-workload-certs\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:01:55.320930 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:55.320696 4421 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7200ec5-9b07-4003-9652-a92c5a1ab5de-credential-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:01:55.320930 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:55.320709 4421 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7200ec5-9b07-4003-9652-a92c5a1ab5de-istiod-ca-cert\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:01:55.320930 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:55.320723 4421 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7200ec5-9b07-4003-9652-a92c5a1ab5de-workload-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:01:55.323470 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:55.323435 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7200ec5-9b07-4003-9652-a92c5a1ab5de-istio-envoy" pod "e7200ec5-9b07-4003-9652-a92c5a1ab5de" (UID: "e7200ec5-9b07-4003-9652-a92c5a1ab5de"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:01:55.323594 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:55.323445 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e7200ec5-9b07-4003-9652-a92c5a1ab5de-istio-podinfo" pod "e7200ec5-9b07-4003-9652-a92c5a1ab5de" (UID: "e7200ec5-9b07-4003-9652-a92c5a1ab5de"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:01:55.323594 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:55.323482 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7200ec5-9b07-4003-9652-a92c5a1ab5de-kube-api-access-vzwj4" pod "e7200ec5-9b07-4003-9652-a92c5a1ab5de" (UID: "e7200ec5-9b07-4003-9652-a92c5a1ab5de"). InnerVolumeSpecName "kube-api-access-vzwj4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:01:55.323594 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:55.323521 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7200ec5-9b07-4003-9652-a92c5a1ab5de-istio-token" pod "e7200ec5-9b07-4003-9652-a92c5a1ab5de" (UID: "e7200ec5-9b07-4003-9652-a92c5a1ab5de"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:01:55.408876 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:55.408840 4421 generic.go:356] "Generic (PLEG): container finished" podID="e7200ec5-9b07-4003-9652-a92c5a1ab5de" containerID="b1011411c1aa9b70e8cfa688e51cb1e3f6c168a17dd72271a89cb0f90f7a38e7" exitCode=0 Nov 03 01:01:55.409562 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:55.408905 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f88f87-openshift-default-7b6c9d6885-g92w2" Nov 03 01:01:55.409562 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:55.408930 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f88f87-openshift-default-7b6c9d6885-g92w2" event={"ID":"e7200ec5-9b07-4003-9652-a92c5a1ab5de","Type":"ContainerDied","Data":"b1011411c1aa9b70e8cfa688e51cb1e3f6c168a17dd72271a89cb0f90f7a38e7"} Nov 03 01:01:55.409562 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:55.408988 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f88f87-openshift-default-7b6c9d6885-g92w2" event={"ID":"e7200ec5-9b07-4003-9652-a92c5a1ab5de","Type":"ContainerDied","Data":"e407daa31fa4f08e39db89fd6e765cd176e74c0f9322d2e5f1d6b821ba18e78a"} Nov 03 01:01:55.409562 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:55.409014 4421 scope.go:122] "RemoveContainer" containerID="b1011411c1aa9b70e8cfa688e51cb1e3f6c168a17dd72271a89cb0f90f7a38e7" Nov 03 01:01:55.422187 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:55.422159 4421 scope.go:122] "RemoveContainer" containerID="9e94379bd48d6349e7e47ddc3391920faa9dfe975bd5118e072fb0fd1d235b6a" Nov 03 01:01:55.422477 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:55.422458 4421 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7200ec5-9b07-4003-9652-a92c5a1ab5de-istio-token\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:01:55.422562 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:55.422480 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vzwj4\" (UniqueName: \"kubernetes.io/projected/e7200ec5-9b07-4003-9652-a92c5a1ab5de-kube-api-access-vzwj4\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:01:55.422562 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:55.422492 4421 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7200ec5-9b07-4003-9652-a92c5a1ab5de-istio-envoy\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:01:55.422562 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:55.422501 4421 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7200ec5-9b07-4003-9652-a92c5a1ab5de-istio-podinfo\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:01:55.432808 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:55.432787 4421 scope.go:122] "RemoveContainer" containerID="b1011411c1aa9b70e8cfa688e51cb1e3f6c168a17dd72271a89cb0f90f7a38e7" Nov 03 01:01:55.433141 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:01:55.433120 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1011411c1aa9b70e8cfa688e51cb1e3f6c168a17dd72271a89cb0f90f7a38e7\": container with ID starting with b1011411c1aa9b70e8cfa688e51cb1e3f6c168a17dd72271a89cb0f90f7a38e7 not found: ID does not exist" containerID="b1011411c1aa9b70e8cfa688e51cb1e3f6c168a17dd72271a89cb0f90f7a38e7" Nov 03 01:01:55.433213 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:55.433151 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1011411c1aa9b70e8cfa688e51cb1e3f6c168a17dd72271a89cb0f90f7a38e7"} err="failed to get container status \"b1011411c1aa9b70e8cfa688e51cb1e3f6c168a17dd72271a89cb0f90f7a38e7\": rpc error: code = NotFound desc = could not find container \"b1011411c1aa9b70e8cfa688e51cb1e3f6c168a17dd72271a89cb0f90f7a38e7\": container with ID starting with b1011411c1aa9b70e8cfa688e51cb1e3f6c168a17dd72271a89cb0f90f7a38e7 not found: ID does not exist" Nov 03 01:01:55.433213 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:55.433172 4421 scope.go:122] "RemoveContainer" containerID="9e94379bd48d6349e7e47ddc3391920faa9dfe975bd5118e072fb0fd1d235b6a" Nov 03 01:01:55.434691 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:01:55.434652 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e94379bd48d6349e7e47ddc3391920faa9dfe975bd5118e072fb0fd1d235b6a\": container with ID starting with 9e94379bd48d6349e7e47ddc3391920faa9dfe975bd5118e072fb0fd1d235b6a not found: ID does not exist" containerID="9e94379bd48d6349e7e47ddc3391920faa9dfe975bd5118e072fb0fd1d235b6a" Nov 03 01:01:55.434814 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:55.434686 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e94379bd48d6349e7e47ddc3391920faa9dfe975bd5118e072fb0fd1d235b6a"} err="failed to get container status \"9e94379bd48d6349e7e47ddc3391920faa9dfe975bd5118e072fb0fd1d235b6a\": rpc error: code = NotFound desc = could not find container \"9e94379bd48d6349e7e47ddc3391920faa9dfe975bd5118e072fb0fd1d235b6a\": container with ID starting with 9e94379bd48d6349e7e47ddc3391920faa9dfe975bd5118e072fb0fd1d235b6a not found: ID does not exist" Nov 03 01:01:55.440117 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:55.440090 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-f88f87-openshift-default-7b6c9d6885-g92w2"] Nov 03 01:01:55.444069 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:55.444043 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-f88f87-openshift-default-7b6c9d6885-g92w2"] Nov 03 01:01:55.618574 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:55.618526 4421 patch_prober.go:29] interesting pod/e2e-praxis-b-208a0d-openshift-default-67457fb65c-ksfzw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 01:01:55.618909 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:55.618600 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-208a0d-openshift-default-67457fb65c-ksfzw" podUID="a47649ab-48b3-4de0-b057-5445c923c427" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 01:01:55.984401 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:55.984366 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7200ec5-9b07-4003-9652-a92c5a1ab5de" path="/var/lib/kubelet/pods/e7200ec5-9b07-4003-9652-a92c5a1ab5de/volumes" Nov 03 01:01:56.181170 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:56.181140 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dc0f3f-59446658cf-n6msk" Nov 03 01:01:57.429439 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:57.429344 4421 generic.go:356] "Generic (PLEG): container finished" podID="a47649ab-48b3-4de0-b057-5445c923c427" containerID="e68dbf918c3fed3dce2f889f1e30293534ea755b4403b621185c3638abd9173a" exitCode=0 Nov 03 01:01:57.429439 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:57.429403 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-208a0d-openshift-default-67457fb65c-ksfzw" event={"ID":"a47649ab-48b3-4de0-b057-5445c923c427","Type":"ContainerDied","Data":"e68dbf918c3fed3dce2f889f1e30293534ea755b4403b621185c3638abd9173a"} Nov 03 01:01:57.429880 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:57.429865 4421 scope.go:122] "RemoveContainer" containerID="e68dbf918c3fed3dce2f889f1e30293534ea755b4403b621185c3638abd9173a" Nov 03 01:01:57.431536 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:57.431510 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:01:57.431622 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:57.431579 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:01:57.431661 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:57.431619 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:01:57.909298 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:57.907134 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-ff4113-openshift-default-6b94f7db5f-4p87g"] Nov 03 01:01:57.909298 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:57.908208 4421 cpu_manager.go:414] "containerMap: removing container" podUID="e7200ec5-9b07-4003-9652-a92c5a1ab5de" containerName="istio-proxy" Nov 03 01:01:57.909298 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:57.908228 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7200ec5-9b07-4003-9652-a92c5a1ab5de" containerName="istio-proxy" Nov 03 01:01:57.909298 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:57.908476 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7200ec5-9b07-4003-9652-a92c5a1ab5de" containerName="istio-proxy" Nov 03 01:01:57.909298 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:57.908496 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7200ec5-9b07-4003-9652-a92c5a1ab5de" containerName="istio-proxy" Nov 03 01:01:57.922300 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:57.919129 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-ff4113-openshift-default-6b94f7db5f-4p87g" Nov 03 01:01:57.939323 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:57.936944 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-ff4113-openshift-default-6b94f7db5f-4p87g"] Nov 03 01:01:57.946671 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:57.946641 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-ff4113-openshift-default-dockercfg-4nkm6\"" Nov 03 01:01:58.081840 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:58.081802 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ab8d9c7-c645-4569-81a8-c5b1edaa7857-istiod-ca-cert\") pod \"e2e-ipp-b-ff4113-openshift-default-6b94f7db5f-4p87g\" (UID: \"6ab8d9c7-c645-4569-81a8-c5b1edaa7857\") " pod="openshift-ingress/e2e-ipp-b-ff4113-openshift-default-6b94f7db5f-4p87g" Nov 03 01:01:58.082082 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:58.082064 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ab8d9c7-c645-4569-81a8-c5b1edaa7857-workload-socket\") pod \"e2e-ipp-b-ff4113-openshift-default-6b94f7db5f-4p87g\" (UID: \"6ab8d9c7-c645-4569-81a8-c5b1edaa7857\") " pod="openshift-ingress/e2e-ipp-b-ff4113-openshift-default-6b94f7db5f-4p87g" Nov 03 01:01:58.082338 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:58.082317 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6z7g\" (UniqueName: \"kubernetes.io/projected/6ab8d9c7-c645-4569-81a8-c5b1edaa7857-kube-api-access-k6z7g\") pod \"e2e-ipp-b-ff4113-openshift-default-6b94f7db5f-4p87g\" (UID: \"6ab8d9c7-c645-4569-81a8-c5b1edaa7857\") " pod="openshift-ingress/e2e-ipp-b-ff4113-openshift-default-6b94f7db5f-4p87g" Nov 03 01:01:58.082530 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:58.082511 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ab8d9c7-c645-4569-81a8-c5b1edaa7857-istio-podinfo\") pod \"e2e-ipp-b-ff4113-openshift-default-6b94f7db5f-4p87g\" (UID: \"6ab8d9c7-c645-4569-81a8-c5b1edaa7857\") " pod="openshift-ingress/e2e-ipp-b-ff4113-openshift-default-6b94f7db5f-4p87g" Nov 03 01:01:58.082678 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:58.082663 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ab8d9c7-c645-4569-81a8-c5b1edaa7857-workload-certs\") pod \"e2e-ipp-b-ff4113-openshift-default-6b94f7db5f-4p87g\" (UID: \"6ab8d9c7-c645-4569-81a8-c5b1edaa7857\") " pod="openshift-ingress/e2e-ipp-b-ff4113-openshift-default-6b94f7db5f-4p87g" Nov 03 01:01:58.082863 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:58.082846 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ab8d9c7-c645-4569-81a8-c5b1edaa7857-credential-socket\") pod \"e2e-ipp-b-ff4113-openshift-default-6b94f7db5f-4p87g\" (UID: \"6ab8d9c7-c645-4569-81a8-c5b1edaa7857\") " pod="openshift-ingress/e2e-ipp-b-ff4113-openshift-default-6b94f7db5f-4p87g" Nov 03 01:01:58.083032 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:58.083013 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ab8d9c7-c645-4569-81a8-c5b1edaa7857-istio-envoy\") pod \"e2e-ipp-b-ff4113-openshift-default-6b94f7db5f-4p87g\" (UID: \"6ab8d9c7-c645-4569-81a8-c5b1edaa7857\") " pod="openshift-ingress/e2e-ipp-b-ff4113-openshift-default-6b94f7db5f-4p87g" Nov 03 01:01:58.083193 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:58.083177 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ab8d9c7-c645-4569-81a8-c5b1edaa7857-istio-data\") pod \"e2e-ipp-b-ff4113-openshift-default-6b94f7db5f-4p87g\" (UID: \"6ab8d9c7-c645-4569-81a8-c5b1edaa7857\") " pod="openshift-ingress/e2e-ipp-b-ff4113-openshift-default-6b94f7db5f-4p87g" Nov 03 01:01:58.083343 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:58.083326 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ab8d9c7-c645-4569-81a8-c5b1edaa7857-istio-token\") pod \"e2e-ipp-b-ff4113-openshift-default-6b94f7db5f-4p87g\" (UID: \"6ab8d9c7-c645-4569-81a8-c5b1edaa7857\") " pod="openshift-ingress/e2e-ipp-b-ff4113-openshift-default-6b94f7db5f-4p87g" Nov 03 01:01:58.187362 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:58.187236 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ab8d9c7-c645-4569-81a8-c5b1edaa7857-istiod-ca-cert\") pod \"e2e-ipp-b-ff4113-openshift-default-6b94f7db5f-4p87g\" (UID: \"6ab8d9c7-c645-4569-81a8-c5b1edaa7857\") " pod="openshift-ingress/e2e-ipp-b-ff4113-openshift-default-6b94f7db5f-4p87g" Nov 03 01:01:58.187362 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:58.187308 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ab8d9c7-c645-4569-81a8-c5b1edaa7857-workload-socket\") pod \"e2e-ipp-b-ff4113-openshift-default-6b94f7db5f-4p87g\" (UID: \"6ab8d9c7-c645-4569-81a8-c5b1edaa7857\") " pod="openshift-ingress/e2e-ipp-b-ff4113-openshift-default-6b94f7db5f-4p87g" Nov 03 01:01:58.187614 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:58.187381 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6z7g\" (UniqueName: \"kubernetes.io/projected/6ab8d9c7-c645-4569-81a8-c5b1edaa7857-kube-api-access-k6z7g\") pod \"e2e-ipp-b-ff4113-openshift-default-6b94f7db5f-4p87g\" (UID: \"6ab8d9c7-c645-4569-81a8-c5b1edaa7857\") " pod="openshift-ingress/e2e-ipp-b-ff4113-openshift-default-6b94f7db5f-4p87g" Nov 03 01:01:58.187614 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:58.187410 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ab8d9c7-c645-4569-81a8-c5b1edaa7857-istio-podinfo\") pod \"e2e-ipp-b-ff4113-openshift-default-6b94f7db5f-4p87g\" (UID: \"6ab8d9c7-c645-4569-81a8-c5b1edaa7857\") " pod="openshift-ingress/e2e-ipp-b-ff4113-openshift-default-6b94f7db5f-4p87g" Nov 03 01:01:58.187614 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:58.187442 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ab8d9c7-c645-4569-81a8-c5b1edaa7857-workload-certs\") pod \"e2e-ipp-b-ff4113-openshift-default-6b94f7db5f-4p87g\" (UID: \"6ab8d9c7-c645-4569-81a8-c5b1edaa7857\") " pod="openshift-ingress/e2e-ipp-b-ff4113-openshift-default-6b94f7db5f-4p87g" Nov 03 01:01:58.187614 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:58.187494 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ab8d9c7-c645-4569-81a8-c5b1edaa7857-credential-socket\") pod \"e2e-ipp-b-ff4113-openshift-default-6b94f7db5f-4p87g\" (UID: \"6ab8d9c7-c645-4569-81a8-c5b1edaa7857\") " pod="openshift-ingress/e2e-ipp-b-ff4113-openshift-default-6b94f7db5f-4p87g" Nov 03 01:01:58.187614 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:58.187567 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ab8d9c7-c645-4569-81a8-c5b1edaa7857-istio-envoy\") pod \"e2e-ipp-b-ff4113-openshift-default-6b94f7db5f-4p87g\" (UID: \"6ab8d9c7-c645-4569-81a8-c5b1edaa7857\") " pod="openshift-ingress/e2e-ipp-b-ff4113-openshift-default-6b94f7db5f-4p87g" Nov 03 01:01:58.187893 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:58.187620 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ab8d9c7-c645-4569-81a8-c5b1edaa7857-istio-data\") pod \"e2e-ipp-b-ff4113-openshift-default-6b94f7db5f-4p87g\" (UID: \"6ab8d9c7-c645-4569-81a8-c5b1edaa7857\") " pod="openshift-ingress/e2e-ipp-b-ff4113-openshift-default-6b94f7db5f-4p87g" Nov 03 01:01:58.187893 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:58.187651 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ab8d9c7-c645-4569-81a8-c5b1edaa7857-istio-token\") pod \"e2e-ipp-b-ff4113-openshift-default-6b94f7db5f-4p87g\" (UID: \"6ab8d9c7-c645-4569-81a8-c5b1edaa7857\") " pod="openshift-ingress/e2e-ipp-b-ff4113-openshift-default-6b94f7db5f-4p87g" Nov 03 01:01:58.187893 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:58.187801 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ab8d9c7-c645-4569-81a8-c5b1edaa7857-workload-socket\") pod \"e2e-ipp-b-ff4113-openshift-default-6b94f7db5f-4p87g\" (UID: \"6ab8d9c7-c645-4569-81a8-c5b1edaa7857\") " pod="openshift-ingress/e2e-ipp-b-ff4113-openshift-default-6b94f7db5f-4p87g" Nov 03 01:01:58.187893 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:58.187825 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ab8d9c7-c645-4569-81a8-c5b1edaa7857-workload-certs\") pod \"e2e-ipp-b-ff4113-openshift-default-6b94f7db5f-4p87g\" (UID: \"6ab8d9c7-c645-4569-81a8-c5b1edaa7857\") " pod="openshift-ingress/e2e-ipp-b-ff4113-openshift-default-6b94f7db5f-4p87g" Nov 03 01:01:58.187893 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:58.187890 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ab8d9c7-c645-4569-81a8-c5b1edaa7857-credential-socket\") pod \"e2e-ipp-b-ff4113-openshift-default-6b94f7db5f-4p87g\" (UID: \"6ab8d9c7-c645-4569-81a8-c5b1edaa7857\") " pod="openshift-ingress/e2e-ipp-b-ff4113-openshift-default-6b94f7db5f-4p87g" Nov 03 01:01:58.188170 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:58.188014 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ab8d9c7-c645-4569-81a8-c5b1edaa7857-istiod-ca-cert\") pod \"e2e-ipp-b-ff4113-openshift-default-6b94f7db5f-4p87g\" (UID: \"6ab8d9c7-c645-4569-81a8-c5b1edaa7857\") " pod="openshift-ingress/e2e-ipp-b-ff4113-openshift-default-6b94f7db5f-4p87g" Nov 03 01:01:58.188170 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:58.188089 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ab8d9c7-c645-4569-81a8-c5b1edaa7857-istio-data\") pod \"e2e-ipp-b-ff4113-openshift-default-6b94f7db5f-4p87g\" (UID: \"6ab8d9c7-c645-4569-81a8-c5b1edaa7857\") " pod="openshift-ingress/e2e-ipp-b-ff4113-openshift-default-6b94f7db5f-4p87g" Nov 03 01:01:58.190023 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:58.189999 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ab8d9c7-c645-4569-81a8-c5b1edaa7857-istio-podinfo\") pod \"e2e-ipp-b-ff4113-openshift-default-6b94f7db5f-4p87g\" (UID: \"6ab8d9c7-c645-4569-81a8-c5b1edaa7857\") " pod="openshift-ingress/e2e-ipp-b-ff4113-openshift-default-6b94f7db5f-4p87g" Nov 03 01:01:58.190144 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:58.190033 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ab8d9c7-c645-4569-81a8-c5b1edaa7857-istio-envoy\") pod \"e2e-ipp-b-ff4113-openshift-default-6b94f7db5f-4p87g\" (UID: \"6ab8d9c7-c645-4569-81a8-c5b1edaa7857\") " pod="openshift-ingress/e2e-ipp-b-ff4113-openshift-default-6b94f7db5f-4p87g" Nov 03 01:01:58.196098 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:58.196068 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6z7g\" (UniqueName: \"kubernetes.io/projected/6ab8d9c7-c645-4569-81a8-c5b1edaa7857-kube-api-access-k6z7g\") pod \"e2e-ipp-b-ff4113-openshift-default-6b94f7db5f-4p87g\" (UID: \"6ab8d9c7-c645-4569-81a8-c5b1edaa7857\") " pod="openshift-ingress/e2e-ipp-b-ff4113-openshift-default-6b94f7db5f-4p87g" Nov 03 01:01:58.196250 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:58.196230 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ab8d9c7-c645-4569-81a8-c5b1edaa7857-istio-token\") pod \"e2e-ipp-b-ff4113-openshift-default-6b94f7db5f-4p87g\" (UID: \"6ab8d9c7-c645-4569-81a8-c5b1edaa7857\") " pod="openshift-ingress/e2e-ipp-b-ff4113-openshift-default-6b94f7db5f-4p87g" Nov 03 01:01:58.261846 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:58.261798 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-ff4113-openshift-default-dockercfg-4nkm6\"" Nov 03 01:01:58.267000 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:58.266948 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-ff4113-openshift-default-6b94f7db5f-4p87g" Nov 03 01:01:58.459546 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:58.459441 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-208a0d-openshift-default-67457fb65c-ksfzw" event={"ID":"a47649ab-48b3-4de0-b057-5445c923c427","Type":"ContainerStarted","Data":"646434756c4cd65ff57a10037ec12e3886ac199876cf56718af7ea6ea311f17e"} Nov 03 01:01:58.586685 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:58.571125 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-ff4113-openshift-default-6b94f7db5f-4p87g"] Nov 03 01:01:58.586685 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:01:58.571715 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ab8d9c7_c645_4569_81a8_c5b1edaa7857.slice/crio-1db82aab5dc5f2f49bb7aa828097dc67f390672ba74e9e1cda4a78594fb0b472 WatchSource:0}: Error finding container 1db82aab5dc5f2f49bb7aa828097dc67f390672ba74e9e1cda4a78594fb0b472: Status 404 returned error can't find the container with id 1db82aab5dc5f2f49bb7aa828097dc67f390672ba74e9e1cda4a78594fb0b472 Nov 03 01:01:58.655569 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:58.639318 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:01:58.655569 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:58.639425 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:01:58.655569 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:58.639473 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:01:58.791299 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:58.790711 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-208a0d-openshift-default-67457fb65c-ksfzw" Nov 03 01:01:58.791299 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:58.790851 4421 patch_prober.go:29] interesting pod/e2e-praxis-b-208a0d-openshift-default-67457fb65c-ksfzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body= Nov 03 01:01:58.791299 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:58.790915 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-208a0d-openshift-default-67457fb65c-ksfzw" podUID="a47649ab-48b3-4de0-b057-5445c923c427" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" Nov 03 01:01:59.066686 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:59.066546 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21e770b3-gcrlb"] Nov 03 01:01:59.085296 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:59.077852 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21e770b3-gcrlb"] Nov 03 01:01:59.468033 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:59.467999 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-ff4113-openshift-default-6b94f7db5f-4p87g" event={"ID":"6ab8d9c7-c645-4569-81a8-c5b1edaa7857","Type":"ContainerStarted","Data":"b478cd4bdd25d7b2802f37b37f38112c582f06cbd4121a2c311eb0d585a60df9"} Nov 03 01:01:59.468486 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:59.468063 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-ff4113-openshift-default-6b94f7db5f-4p87g" event={"ID":"6ab8d9c7-c645-4569-81a8-c5b1edaa7857","Type":"ContainerStarted","Data":"1db82aab5dc5f2f49bb7aa828097dc67f390672ba74e9e1cda4a78594fb0b472"} Nov 03 01:01:59.501880 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:59.501810 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-ff4113-openshift-default-6b94f7db5f-4p87g" podStartSLOduration=2.501795038 podStartE2EDuration="2.501795038s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:01:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:01:59.497345017 +0000 UTC m=+1261.668348115" watchObservedRunningTime="2026-11-03 01:01:59.501795038 +0000 UTC m=+1261.672798134" Nov 03 01:01:59.582809 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:59.582779 4421 scope.go:122] "RemoveContainer" containerID="29083d52d0a8d8c8cf5d78ba7f9d91d237512c5a746199b20f8868990efced29" Nov 03 01:01:59.595190 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:59.595166 4421 scope.go:122] "RemoveContainer" containerID="3a5fcbbc9a4d597b850c0031e022ad946d1532f1b4a4773021d3be6b63e84c05" Nov 03 01:01:59.669919 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:59.667088 4421 scope.go:122] "RemoveContainer" containerID="9ee0e0d2bff70cd6872ba72d3c2cd2bd853bab6f4b8402e732862e525f1d89f5" Nov 03 01:01:59.691536 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:59.691503 4421 scope.go:122] "RemoveContainer" containerID="551d3fdadc50e76d83b20c94806a4c04c5b24280969cf7fd3aa8a6223e7d634e" Nov 03 01:01:59.703718 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:59.703649 4421 scope.go:122] "RemoveContainer" containerID="0fe4d66becec984da80267f73baf1cf8a7b8d9be8b15963dd7c694ed47cf0c45" Nov 03 01:01:59.743167 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:59.741517 4421 scope.go:122] "RemoveContainer" containerID="45b723ad0c55830daa39637d038e3a4b886001a5af48a72c2e24c5737452e8ed" Nov 03 01:01:59.793954 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:59.793499 4421 patch_prober.go:29] interesting pod/e2e-praxis-b-208a0d-openshift-default-67457fb65c-ksfzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body= Nov 03 01:01:59.793954 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:59.793555 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-208a0d-openshift-default-67457fb65c-ksfzw" podUID="a47649ab-48b3-4de0-b057-5445c923c427" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" Nov 03 01:01:59.900164 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:59.900137 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-cmdm8" Nov 03 01:01:59.901448 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:59.901423 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-zrdp5" Nov 03 01:01:59.984601 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:01:59.984392 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a091a850-4bf7-4272-b9ea-3fbd5b7b3df4" path="/var/lib/kubelet/pods/a091a850-4bf7-4272-b9ea-3fbd5b7b3df4/volumes" Nov 03 01:02:00.071126 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:00.071089 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b6779b2-2b64-40e9-821c-5444681cea09-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b6779b2-2b64-40e9-821c-5444681cea09-istio-envoy\") pod \"7b6779b2-2b64-40e9-821c-5444681cea09\" (UID: \"7b6779b2-2b64-40e9-821c-5444681cea09\") " Nov 03 01:02:00.071438 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:00.071139 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9916b060-ab92-4836-afab-cd531100dd69-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9916b060-ab92-4836-afab-cd531100dd69-istio-podinfo\") pod \"9916b060-ab92-4836-afab-cd531100dd69\" (UID: \"9916b060-ab92-4836-afab-cd531100dd69\") " Nov 03 01:02:00.071438 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:00.071186 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b6779b2-2b64-40e9-821c-5444681cea09-kube-api-access-5qtns\" (UniqueName: \"kubernetes.io/projected/7b6779b2-2b64-40e9-821c-5444681cea09-kube-api-access-5qtns\") pod \"7b6779b2-2b64-40e9-821c-5444681cea09\" (UID: \"7b6779b2-2b64-40e9-821c-5444681cea09\") " Nov 03 01:02:00.071438 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:00.071216 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9916b060-ab92-4836-afab-cd531100dd69-istio-token\" (UniqueName: \"kubernetes.io/projected/9916b060-ab92-4836-afab-cd531100dd69-istio-token\") pod \"9916b060-ab92-4836-afab-cd531100dd69\" (UID: \"9916b060-ab92-4836-afab-cd531100dd69\") " Nov 03 01:02:00.071438 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:00.071243 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9916b060-ab92-4836-afab-cd531100dd69-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9916b060-ab92-4836-afab-cd531100dd69-workload-socket\") pod \"9916b060-ab92-4836-afab-cd531100dd69\" (UID: \"9916b060-ab92-4836-afab-cd531100dd69\") " Nov 03 01:02:00.071438 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:00.071330 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9916b060-ab92-4836-afab-cd531100dd69-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9916b060-ab92-4836-afab-cd531100dd69-istio-envoy\") pod \"9916b060-ab92-4836-afab-cd531100dd69\" (UID: \"9916b060-ab92-4836-afab-cd531100dd69\") " Nov 03 01:02:00.071438 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:00.071360 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9916b060-ab92-4836-afab-cd531100dd69-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9916b060-ab92-4836-afab-cd531100dd69-workload-certs\") pod \"9916b060-ab92-4836-afab-cd531100dd69\" (UID: \"9916b060-ab92-4836-afab-cd531100dd69\") " Nov 03 01:02:00.071438 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:00.071389 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b6779b2-2b64-40e9-821c-5444681cea09-istio-token\" (UniqueName: \"kubernetes.io/projected/7b6779b2-2b64-40e9-821c-5444681cea09-istio-token\") pod \"7b6779b2-2b64-40e9-821c-5444681cea09\" (UID: \"7b6779b2-2b64-40e9-821c-5444681cea09\") " Nov 03 01:02:00.071438 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:00.071422 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b6779b2-2b64-40e9-821c-5444681cea09-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b6779b2-2b64-40e9-821c-5444681cea09-workload-socket\") pod \"7b6779b2-2b64-40e9-821c-5444681cea09\" (UID: \"7b6779b2-2b64-40e9-821c-5444681cea09\") " Nov 03 01:02:00.071908 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:00.071519 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b6779b2-2b64-40e9-821c-5444681cea09-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b6779b2-2b64-40e9-821c-5444681cea09-workload-certs\") pod \"7b6779b2-2b64-40e9-821c-5444681cea09\" (UID: \"7b6779b2-2b64-40e9-821c-5444681cea09\") " Nov 03 01:02:00.071908 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:00.071558 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9916b060-ab92-4836-afab-cd531100dd69-kube-api-access-wczfd\" (UniqueName: \"kubernetes.io/projected/9916b060-ab92-4836-afab-cd531100dd69-kube-api-access-wczfd\") pod \"9916b060-ab92-4836-afab-cd531100dd69\" (UID: \"9916b060-ab92-4836-afab-cd531100dd69\") " Nov 03 01:02:00.071908 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:00.071634 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b6779b2-2b64-40e9-821c-5444681cea09-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b6779b2-2b64-40e9-821c-5444681cea09-credential-socket\") pod \"7b6779b2-2b64-40e9-821c-5444681cea09\" (UID: \"7b6779b2-2b64-40e9-821c-5444681cea09\") " Nov 03 01:02:00.071908 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:00.071696 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9916b060-ab92-4836-afab-cd531100dd69-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9916b060-ab92-4836-afab-cd531100dd69-istiod-ca-cert\") pod \"9916b060-ab92-4836-afab-cd531100dd69\" (UID: \"9916b060-ab92-4836-afab-cd531100dd69\") " Nov 03 01:02:00.071908 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:00.071704 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9916b060-ab92-4836-afab-cd531100dd69-workload-certs" pod "9916b060-ab92-4836-afab-cd531100dd69" (UID: "9916b060-ab92-4836-afab-cd531100dd69"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:02:00.071908 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:00.071742 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9916b060-ab92-4836-afab-cd531100dd69-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9916b060-ab92-4836-afab-cd531100dd69-istio-data\") pod \"9916b060-ab92-4836-afab-cd531100dd69\" (UID: \"9916b060-ab92-4836-afab-cd531100dd69\") " Nov 03 01:02:00.071908 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:00.071799 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b6779b2-2b64-40e9-821c-5444681cea09-workload-socket" pod "7b6779b2-2b64-40e9-821c-5444681cea09" (UID: "7b6779b2-2b64-40e9-821c-5444681cea09"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:02:00.071908 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:00.071849 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9916b060-ab92-4836-afab-cd531100dd69-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9916b060-ab92-4836-afab-cd531100dd69-credential-socket\") pod \"9916b060-ab92-4836-afab-cd531100dd69\" (UID: \"9916b060-ab92-4836-afab-cd531100dd69\") " Nov 03 01:02:00.071908 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:00.071891 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9916b060-ab92-4836-afab-cd531100dd69-workload-socket" pod "9916b060-ab92-4836-afab-cd531100dd69" (UID: "9916b060-ab92-4836-afab-cd531100dd69"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:02:00.072400 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:00.071922 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b6779b2-2b64-40e9-821c-5444681cea09-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b6779b2-2b64-40e9-821c-5444681cea09-istiod-ca-cert\") pod \"7b6779b2-2b64-40e9-821c-5444681cea09\" (UID: \"7b6779b2-2b64-40e9-821c-5444681cea09\") " Nov 03 01:02:00.072400 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:00.071958 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7b6779b2-2b64-40e9-821c-5444681cea09-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b6779b2-2b64-40e9-821c-5444681cea09-istio-podinfo\") pod \"7b6779b2-2b64-40e9-821c-5444681cea09\" (UID: \"7b6779b2-2b64-40e9-821c-5444681cea09\") " Nov 03 01:02:00.072400 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:00.072020 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b6779b2-2b64-40e9-821c-5444681cea09-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b6779b2-2b64-40e9-821c-5444681cea09-istio-data\") pod \"7b6779b2-2b64-40e9-821c-5444681cea09\" (UID: \"7b6779b2-2b64-40e9-821c-5444681cea09\") " Nov 03 01:02:00.072400 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:00.072122 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b6779b2-2b64-40e9-821c-5444681cea09-workload-certs" pod "7b6779b2-2b64-40e9-821c-5444681cea09" (UID: "7b6779b2-2b64-40e9-821c-5444681cea09"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:02:00.072400 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:00.072359 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9916b060-ab92-4836-afab-cd531100dd69-istio-data" pod "9916b060-ab92-4836-afab-cd531100dd69" (UID: "9916b060-ab92-4836-afab-cd531100dd69"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:02:00.072765 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:00.072570 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b6779b2-2b64-40e9-821c-5444681cea09-credential-socket" pod "7b6779b2-2b64-40e9-821c-5444681cea09" (UID: "7b6779b2-2b64-40e9-821c-5444681cea09"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:02:00.072765 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:00.072611 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b6779b2-2b64-40e9-821c-5444681cea09-istio-data" pod "7b6779b2-2b64-40e9-821c-5444681cea09" (UID: "7b6779b2-2b64-40e9-821c-5444681cea09"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:02:00.072890 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:00.072783 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9916b060-ab92-4836-afab-cd531100dd69-credential-socket" pod "9916b060-ab92-4836-afab-cd531100dd69" (UID: "9916b060-ab92-4836-afab-cd531100dd69"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:02:00.072952 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:00.072895 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9916b060-ab92-4836-afab-cd531100dd69-istiod-ca-cert" pod "9916b060-ab92-4836-afab-cd531100dd69" (UID: "9916b060-ab92-4836-afab-cd531100dd69"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:02:00.073119 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:00.073098 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b6779b2-2b64-40e9-821c-5444681cea09-istiod-ca-cert" pod "7b6779b2-2b64-40e9-821c-5444681cea09" (UID: "7b6779b2-2b64-40e9-821c-5444681cea09"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:02:00.073509 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:00.073486 4421 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b6779b2-2b64-40e9-821c-5444681cea09-credential-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:02:00.073632 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:00.073516 4421 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9916b060-ab92-4836-afab-cd531100dd69-istiod-ca-cert\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:02:00.073632 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:00.073533 4421 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9916b060-ab92-4836-afab-cd531100dd69-istio-data\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:02:00.073632 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:00.073547 4421 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9916b060-ab92-4836-afab-cd531100dd69-credential-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:02:00.073632 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:00.073561 4421 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b6779b2-2b64-40e9-821c-5444681cea09-istiod-ca-cert\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:02:00.073632 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:00.073577 4421 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b6779b2-2b64-40e9-821c-5444681cea09-istio-data\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:02:00.073632 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:00.073591 4421 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9916b060-ab92-4836-afab-cd531100dd69-workload-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:02:00.073632 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:00.073609 4421 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9916b060-ab92-4836-afab-cd531100dd69-workload-certs\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:02:00.073632 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:00.073626 4421 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b6779b2-2b64-40e9-821c-5444681cea09-workload-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:02:00.074061 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:00.073642 4421 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b6779b2-2b64-40e9-821c-5444681cea09-workload-certs\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:02:00.074883 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:00.074841 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b6779b2-2b64-40e9-821c-5444681cea09-istio-envoy" pod "7b6779b2-2b64-40e9-821c-5444681cea09" (UID: "7b6779b2-2b64-40e9-821c-5444681cea09"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:02:00.075126 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:00.075094 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9916b060-ab92-4836-afab-cd531100dd69-istio-podinfo" pod "9916b060-ab92-4836-afab-cd531100dd69" (UID: "9916b060-ab92-4836-afab-cd531100dd69"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:02:00.075380 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:00.075351 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b6779b2-2b64-40e9-821c-5444681cea09-istio-token" pod "7b6779b2-2b64-40e9-821c-5444681cea09" (UID: "7b6779b2-2b64-40e9-821c-5444681cea09"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:02:00.075380 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:00.075371 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9916b060-ab92-4836-afab-cd531100dd69-istio-token" pod "9916b060-ab92-4836-afab-cd531100dd69" (UID: "9916b060-ab92-4836-afab-cd531100dd69"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:02:00.075541 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:00.075481 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b6779b2-2b64-40e9-821c-5444681cea09-kube-api-access-5qtns" pod "7b6779b2-2b64-40e9-821c-5444681cea09" (UID: "7b6779b2-2b64-40e9-821c-5444681cea09"). InnerVolumeSpecName "kube-api-access-5qtns". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:02:00.075597 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:00.075554 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9916b060-ab92-4836-afab-cd531100dd69-istio-envoy" pod "9916b060-ab92-4836-afab-cd531100dd69" (UID: "9916b060-ab92-4836-afab-cd531100dd69"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:02:00.075770 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:00.075746 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7b6779b2-2b64-40e9-821c-5444681cea09-istio-podinfo" pod "7b6779b2-2b64-40e9-821c-5444681cea09" (UID: "7b6779b2-2b64-40e9-821c-5444681cea09"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:02:00.076402 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:00.076387 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9916b060-ab92-4836-afab-cd531100dd69-kube-api-access-wczfd" pod "9916b060-ab92-4836-afab-cd531100dd69" (UID: "9916b060-ab92-4836-afab-cd531100dd69"). InnerVolumeSpecName "kube-api-access-wczfd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:02:00.175132 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:00.175095 4421 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9916b060-ab92-4836-afab-cd531100dd69-istio-envoy\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:02:00.175132 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:00.175125 4421 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b6779b2-2b64-40e9-821c-5444681cea09-istio-token\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:02:00.175132 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:00.175139 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wczfd\" (UniqueName: \"kubernetes.io/projected/9916b060-ab92-4836-afab-cd531100dd69-kube-api-access-wczfd\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:02:00.175410 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:00.175154 4421 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b6779b2-2b64-40e9-821c-5444681cea09-istio-podinfo\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:02:00.175410 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:00.175167 4421 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b6779b2-2b64-40e9-821c-5444681cea09-istio-envoy\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:02:00.175410 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:00.175179 4421 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9916b060-ab92-4836-afab-cd531100dd69-istio-podinfo\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:02:00.175410 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:00.175193 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qtns\" (UniqueName: \"kubernetes.io/projected/7b6779b2-2b64-40e9-821c-5444681cea09-kube-api-access-5qtns\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:02:00.175410 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:00.175208 4421 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9916b060-ab92-4836-afab-cd531100dd69-istio-token\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:02:00.267926 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:00.267891 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-ff4113-openshift-default-6b94f7db5f-4p87g" Nov 03 01:02:00.274561 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:00.274529 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-ff4113-openshift-default-6b94f7db5f-4p87g" Nov 03 01:02:00.477905 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:00.477812 4421 generic.go:356] "Generic (PLEG): container finished" podID="9916b060-ab92-4836-afab-cd531100dd69" containerID="b03ef5608292522d186a3c306168b9ba29d7732b01214764dd988da139dcef5c" exitCode=0 Nov 03 01:02:00.477905 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:00.477860 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-cmdm8" event={"ID":"9916b060-ab92-4836-afab-cd531100dd69","Type":"ContainerDied","Data":"b03ef5608292522d186a3c306168b9ba29d7732b01214764dd988da139dcef5c"} Nov 03 01:02:00.478462 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:00.477919 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-cmdm8" event={"ID":"9916b060-ab92-4836-afab-cd531100dd69","Type":"ContainerDied","Data":"79dd8b4e3a1dfa127919acbbae7bb94d210ca157d3facc91a2d0ad9188983ac6"} Nov 03 01:02:00.478462 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:00.477868 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-cmdm8" Nov 03 01:02:00.478462 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:00.477954 4421 scope.go:122] "RemoveContainer" containerID="b03ef5608292522d186a3c306168b9ba29d7732b01214764dd988da139dcef5c" Nov 03 01:02:00.479828 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:00.479803 4421 generic.go:356] "Generic (PLEG): container finished" podID="7b6779b2-2b64-40e9-821c-5444681cea09" containerID="817dc1736b03a03d7b7981117dcc9a84c8fbe8722105c1052ea00962ca3ee5e1" exitCode=0 Nov 03 01:02:00.480038 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:00.480002 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-zrdp5" event={"ID":"7b6779b2-2b64-40e9-821c-5444681cea09","Type":"ContainerDied","Data":"817dc1736b03a03d7b7981117dcc9a84c8fbe8722105c1052ea00962ca3ee5e1"} Nov 03 01:02:00.480156 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:00.480056 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-zrdp5" event={"ID":"7b6779b2-2b64-40e9-821c-5444681cea09","Type":"ContainerDied","Data":"23e5d69af785381e97a8ee5bfffd6b53f000f908bc7858cb3ea4cffb3c605c4f"} Nov 03 01:02:00.480156 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:00.480079 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-zrdp5" Nov 03 01:02:00.480906 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:00.480885 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-ff4113-openshift-default-6b94f7db5f-4p87g" Nov 03 01:02:00.482177 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:00.482157 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-ff4113-openshift-default-6b94f7db5f-4p87g" Nov 03 01:02:00.490748 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:00.490726 4421 scope.go:122] "RemoveContainer" containerID="b03ef5608292522d186a3c306168b9ba29d7732b01214764dd988da139dcef5c" Nov 03 01:02:00.491078 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:02:00.491056 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b03ef5608292522d186a3c306168b9ba29d7732b01214764dd988da139dcef5c\": container with ID starting with b03ef5608292522d186a3c306168b9ba29d7732b01214764dd988da139dcef5c not found: ID does not exist" containerID="b03ef5608292522d186a3c306168b9ba29d7732b01214764dd988da139dcef5c" Nov 03 01:02:00.491165 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:00.491084 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b03ef5608292522d186a3c306168b9ba29d7732b01214764dd988da139dcef5c"} err="failed to get container status \"b03ef5608292522d186a3c306168b9ba29d7732b01214764dd988da139dcef5c\": rpc error: code = NotFound desc = could not find container \"b03ef5608292522d186a3c306168b9ba29d7732b01214764dd988da139dcef5c\": container with ID starting with b03ef5608292522d186a3c306168b9ba29d7732b01214764dd988da139dcef5c not found: ID does not exist" Nov 03 01:02:00.491165 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:00.491100 4421 scope.go:122] "RemoveContainer" containerID="817dc1736b03a03d7b7981117dcc9a84c8fbe8722105c1052ea00962ca3ee5e1" Nov 03 01:02:00.500772 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:00.500754 4421 scope.go:122] "RemoveContainer" containerID="817dc1736b03a03d7b7981117dcc9a84c8fbe8722105c1052ea00962ca3ee5e1" Nov 03 01:02:00.501068 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:02:00.501042 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"817dc1736b03a03d7b7981117dcc9a84c8fbe8722105c1052ea00962ca3ee5e1\": container with ID starting with 817dc1736b03a03d7b7981117dcc9a84c8fbe8722105c1052ea00962ca3ee5e1 not found: ID does not exist" containerID="817dc1736b03a03d7b7981117dcc9a84c8fbe8722105c1052ea00962ca3ee5e1" Nov 03 01:02:00.501157 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:00.501074 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"817dc1736b03a03d7b7981117dcc9a84c8fbe8722105c1052ea00962ca3ee5e1"} err="failed to get container status \"817dc1736b03a03d7b7981117dcc9a84c8fbe8722105c1052ea00962ca3ee5e1\": rpc error: code = NotFound desc = could not find container \"817dc1736b03a03d7b7981117dcc9a84c8fbe8722105c1052ea00962ca3ee5e1\": container with ID starting with 817dc1736b03a03d7b7981117dcc9a84c8fbe8722105c1052ea00962ca3ee5e1 not found: ID does not exist" Nov 03 01:02:00.555423 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:00.555377 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-cmdm8"] Nov 03 01:02:00.560453 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:00.560416 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-cmdm8"] Nov 03 01:02:00.572811 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:00.572780 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-zrdp5"] Nov 03 01:02:00.576503 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:00.576476 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-zrdp5"] Nov 03 01:02:00.790712 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:00.790670 4421 patch_prober.go:29] interesting pod/e2e-praxis-b-208a0d-openshift-default-67457fb65c-ksfzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body= Nov 03 01:02:00.790907 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:00.790731 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-208a0d-openshift-default-67457fb65c-ksfzw" podUID="a47649ab-48b3-4de0-b057-5445c923c427" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" Nov 03 01:02:01.796380 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:01.796343 4421 patch_prober.go:29] interesting pod/e2e-praxis-b-208a0d-openshift-default-67457fb65c-ksfzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body= Nov 03 01:02:01.796961 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:01.796932 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-208a0d-openshift-default-67457fb65c-ksfzw" podUID="a47649ab-48b3-4de0-b057-5445c923c427" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" Nov 03 01:02:01.980552 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:01.980511 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b6779b2-2b64-40e9-821c-5444681cea09" path="/var/lib/kubelet/pods/7b6779b2-2b64-40e9-821c-5444681cea09/volumes" Nov 03 01:02:01.981115 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:01.981099 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9916b060-ab92-4836-afab-cd531100dd69" path="/var/lib/kubelet/pods/9916b060-ab92-4836-afab-cd531100dd69/volumes" Nov 03 01:02:02.791410 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:02.791361 4421 patch_prober.go:29] interesting pod/e2e-praxis-b-208a0d-openshift-default-67457fb65c-ksfzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body= Nov 03 01:02:02.791635 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:02.791439 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-208a0d-openshift-default-67457fb65c-ksfzw" podUID="a47649ab-48b3-4de0-b057-5445c923c427" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" Nov 03 01:02:03.790699 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:03.790652 4421 patch_prober.go:29] interesting pod/e2e-praxis-b-208a0d-openshift-default-67457fb65c-ksfzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body= Nov 03 01:02:03.791151 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:03.790737 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-208a0d-openshift-default-67457fb65c-ksfzw" podUID="a47649ab-48b3-4de0-b057-5445c923c427" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" Nov 03 01:02:03.794760 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:03.794725 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ff4113-7756b5c9bf-9slj7"] Nov 03 01:02:03.795403 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:03.795388 4421 cpu_manager.go:414] "containerMap: removing container" podUID="e7200ec5-9b07-4003-9652-a92c5a1ab5de" containerName="istio-proxy" Nov 03 01:02:03.795403 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:03.795404 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7200ec5-9b07-4003-9652-a92c5a1ab5de" containerName="istio-proxy" Nov 03 01:02:03.795521 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:03.795422 4421 cpu_manager.go:414] "containerMap: removing container" podUID="7b6779b2-2b64-40e9-821c-5444681cea09" containerName="istio-proxy" Nov 03 01:02:03.795521 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:03.795428 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b6779b2-2b64-40e9-821c-5444681cea09" containerName="istio-proxy" Nov 03 01:02:03.795521 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:03.795451 4421 cpu_manager.go:414] "containerMap: removing container" podUID="9916b060-ab92-4836-afab-cd531100dd69" containerName="istio-proxy" Nov 03 01:02:03.795521 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:03.795456 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9916b060-ab92-4836-afab-cd531100dd69" containerName="istio-proxy" Nov 03 01:02:03.795648 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:03.795576 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b6779b2-2b64-40e9-821c-5444681cea09" containerName="istio-proxy" Nov 03 01:02:03.795648 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:03.795587 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="9916b060-ab92-4836-afab-cd531100dd69" containerName="istio-proxy" Nov 03 01:02:03.799542 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:03.799521 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ff4113-7756b5c9bf-9slj7" Nov 03 01:02:03.804779 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:03.804738 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-ff4113\"" Nov 03 01:02:03.804929 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:03.804871 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-ff4113-dockercfg-gzxfg\"" Nov 03 01:02:03.841319 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:03.841234 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ff4113-7756b5c9bf-9slj7"] Nov 03 01:02:03.861330 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:03.860314 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhq9w\" (UniqueName: \"kubernetes.io/projected/ae55f712-b9f8-4b20-bf6f-6edd47ae643c-kube-api-access-fhq9w\") pod \"payload-processing-e2e-ipp-b-ff4113-7756b5c9bf-9slj7\" (UID: \"ae55f712-b9f8-4b20-bf6f-6edd47ae643c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ff4113-7756b5c9bf-9slj7" Nov 03 01:02:03.861330 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:03.860461 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae55f712-b9f8-4b20-bf6f-6edd47ae643c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-ff4113-7756b5c9bf-9slj7\" (UID: \"ae55f712-b9f8-4b20-bf6f-6edd47ae643c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ff4113-7756b5c9bf-9slj7" Nov 03 01:02:03.863497 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:03.863470 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ff4113-784c56fbc9-82ttz"] Nov 03 01:02:03.866729 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:03.866712 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ff4113-784c56fbc9-82ttz" Nov 03 01:02:03.877834 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:03.877806 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ff4113-784c56fbc9-82ttz"] Nov 03 01:02:04.006362 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:03.988247 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhq9w\" (UniqueName: \"kubernetes.io/projected/ae55f712-b9f8-4b20-bf6f-6edd47ae643c-kube-api-access-fhq9w\") pod \"payload-processing-e2e-ipp-b-ff4113-7756b5c9bf-9slj7\" (UID: \"ae55f712-b9f8-4b20-bf6f-6edd47ae643c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ff4113-7756b5c9bf-9slj7" Nov 03 01:02:04.030189 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:04.019359 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bp2b\" (UniqueName: \"kubernetes.io/projected/73c99eb0-c089-4d8c-9d83-4163fb106ded-kube-api-access-8bp2b\") pod \"payload-pre-processing-e2e-ipp-b-ff4113-784c56fbc9-82ttz\" (UID: \"73c99eb0-c089-4d8c-9d83-4163fb106ded\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ff4113-784c56fbc9-82ttz" Nov 03 01:02:04.030189 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:04.019559 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae55f712-b9f8-4b20-bf6f-6edd47ae643c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-ff4113-7756b5c9bf-9slj7\" (UID: \"ae55f712-b9f8-4b20-bf6f-6edd47ae643c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ff4113-7756b5c9bf-9slj7" Nov 03 01:02:04.030189 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:04.019772 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73c99eb0-c089-4d8c-9d83-4163fb106ded-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-ff4113-784c56fbc9-82ttz\" (UID: \"73c99eb0-c089-4d8c-9d83-4163fb106ded\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ff4113-784c56fbc9-82ttz" Nov 03 01:02:04.031217 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:04.031184 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae55f712-b9f8-4b20-bf6f-6edd47ae643c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-ff4113-7756b5c9bf-9slj7\" (UID: \"ae55f712-b9f8-4b20-bf6f-6edd47ae643c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ff4113-7756b5c9bf-9slj7" Nov 03 01:02:04.093051 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:04.062714 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhq9w\" (UniqueName: \"kubernetes.io/projected/ae55f712-b9f8-4b20-bf6f-6edd47ae643c-kube-api-access-fhq9w\") pod \"payload-processing-e2e-ipp-b-ff4113-7756b5c9bf-9slj7\" (UID: \"ae55f712-b9f8-4b20-bf6f-6edd47ae643c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ff4113-7756b5c9bf-9slj7" Nov 03 01:02:04.122834 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:04.122797 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73c99eb0-c089-4d8c-9d83-4163fb106ded-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-ff4113-784c56fbc9-82ttz\" (UID: \"73c99eb0-c089-4d8c-9d83-4163fb106ded\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ff4113-784c56fbc9-82ttz" Nov 03 01:02:04.124583 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:04.124555 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bp2b\" (UniqueName: \"kubernetes.io/projected/73c99eb0-c089-4d8c-9d83-4163fb106ded-kube-api-access-8bp2b\") pod \"payload-pre-processing-e2e-ipp-b-ff4113-784c56fbc9-82ttz\" (UID: \"73c99eb0-c089-4d8c-9d83-4163fb106ded\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ff4113-784c56fbc9-82ttz" Nov 03 01:02:04.125187 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:04.124375 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73c99eb0-c089-4d8c-9d83-4163fb106ded-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-ff4113-784c56fbc9-82ttz\" (UID: \"73c99eb0-c089-4d8c-9d83-4163fb106ded\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ff4113-784c56fbc9-82ttz" Nov 03 01:02:04.149875 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:04.149839 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ff4113-7756b5c9bf-9slj7" Nov 03 01:02:04.175042 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:04.175002 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bp2b\" (UniqueName: \"kubernetes.io/projected/73c99eb0-c089-4d8c-9d83-4163fb106ded-kube-api-access-8bp2b\") pod \"payload-pre-processing-e2e-ipp-b-ff4113-784c56fbc9-82ttz\" (UID: \"73c99eb0-c089-4d8c-9d83-4163fb106ded\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ff4113-784c56fbc9-82ttz" Nov 03 01:02:04.182394 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:04.182028 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ff4113-784c56fbc9-82ttz" Nov 03 01:02:04.469370 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:02:04.469331 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae55f712_b9f8_4b20_bf6f_6edd47ae643c.slice/crio-0f1f21de2ce6ae1f95a1314a2057fa1955c1deacc333a777acb175c128e0ab4d WatchSource:0}: Error finding container 0f1f21de2ce6ae1f95a1314a2057fa1955c1deacc333a777acb175c128e0ab4d: Status 404 returned error can't find the container with id 0f1f21de2ce6ae1f95a1314a2057fa1955c1deacc333a777acb175c128e0ab4d Nov 03 01:02:04.471369 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:04.471341 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ff4113-7756b5c9bf-9slj7"] Nov 03 01:02:04.514594 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:02:04.514559 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73c99eb0_c089_4d8c_9d83_4163fb106ded.slice/crio-27b6f486b839da2bf54b1db486a0af4c327729b07cf2d2f7757c8a347aa6930a WatchSource:0}: Error finding container 27b6f486b839da2bf54b1db486a0af4c327729b07cf2d2f7757c8a347aa6930a: Status 404 returned error can't find the container with id 27b6f486b839da2bf54b1db486a0af4c327729b07cf2d2f7757c8a347aa6930a Nov 03 01:02:04.514824 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:04.514799 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ff4113-784c56fbc9-82ttz"] Nov 03 01:02:04.517981 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:04.517889 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ff4113-7756b5c9bf-9slj7" event={"ID":"ae55f712-b9f8-4b20-bf6f-6edd47ae643c","Type":"ContainerStarted","Data":"0f1f21de2ce6ae1f95a1314a2057fa1955c1deacc333a777acb175c128e0ab4d"} Nov 03 01:02:04.795773 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:04.795737 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-208a0d-openshift-default-67457fb65c-ksfzw" Nov 03 01:02:04.796346 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:04.796308 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-208a0d-openshift-default-67457fb65c-ksfzw" Nov 03 01:02:04.797311 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:04.797251 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-208a0d-openshift-default-67457fb65c-ksfzw" Nov 03 01:02:05.529418 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:05.529377 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ff4113-7756b5c9bf-9slj7" event={"ID":"ae55f712-b9f8-4b20-bf6f-6edd47ae643c","Type":"ContainerStarted","Data":"5022aaaaba94362d9a312545a9e4df5324edbd14aeb56231705f0b6ca22a22e4"} Nov 03 01:02:05.529605 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:05.529464 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-ff4113-7756b5c9bf-9slj7" Nov 03 01:02:05.531788 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:05.531755 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ff4113-784c56fbc9-82ttz" event={"ID":"73c99eb0-c089-4d8c-9d83-4163fb106ded","Type":"ContainerStarted","Data":"6371c6ca37a701de9f9efc78f0525ba8b50fe441b287fd9cb5d96792174863ef"} Nov 03 01:02:05.531921 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:05.531810 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ff4113-784c56fbc9-82ttz" event={"ID":"73c99eb0-c089-4d8c-9d83-4163fb106ded","Type":"ContainerStarted","Data":"27b6f486b839da2bf54b1db486a0af4c327729b07cf2d2f7757c8a347aa6930a"} Nov 03 01:02:05.531921 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:05.531861 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ff4113-784c56fbc9-82ttz" Nov 03 01:02:05.549340 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:05.549286 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-ff4113-7756b5c9bf-9slj7" podStartSLOduration=2.549257008 podStartE2EDuration="2.549257008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:02:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:02:05.548444116 +0000 UTC m=+1267.719447213" watchObservedRunningTime="2026-11-03 01:02:05.549257008 +0000 UTC m=+1267.720260103" Nov 03 01:02:05.571142 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:05.571065 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ff4113-784c56fbc9-82ttz" podStartSLOduration=2.571047191 podStartE2EDuration="2.571047191s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:02:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:02:05.566776334 +0000 UTC m=+1267.737779431" watchObservedRunningTime="2026-11-03 01:02:05.571047191 +0000 UTC m=+1267.742050288" Nov 03 01:02:12.156658 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:12.156623 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-07f5f6-openshift-default-98746c89f-mr8c7"] Nov 03 01:02:12.162078 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:12.162048 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-07f5f6-openshift-default-98746c89f-mr8c7" Nov 03 01:02:12.169115 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:12.169092 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-07f5f6-openshift-default-dockercfg-6m8gj\"" Nov 03 01:02:12.174631 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:12.174608 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-07f5f6-openshift-default-98746c89f-mr8c7"] Nov 03 01:02:12.237391 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:12.237356 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/648ce4f9-456f-48ff-a857-6d9bc7f61dc5-workload-socket\") pod \"e2e-ipp-a-07f5f6-openshift-default-98746c89f-mr8c7\" (UID: \"648ce4f9-456f-48ff-a857-6d9bc7f61dc5\") " pod="openshift-ingress/e2e-ipp-a-07f5f6-openshift-default-98746c89f-mr8c7" Nov 03 01:02:12.237578 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:12.237417 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/648ce4f9-456f-48ff-a857-6d9bc7f61dc5-workload-certs\") pod \"e2e-ipp-a-07f5f6-openshift-default-98746c89f-mr8c7\" (UID: \"648ce4f9-456f-48ff-a857-6d9bc7f61dc5\") " pod="openshift-ingress/e2e-ipp-a-07f5f6-openshift-default-98746c89f-mr8c7" Nov 03 01:02:12.237578 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:12.237530 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/648ce4f9-456f-48ff-a857-6d9bc7f61dc5-istiod-ca-cert\") pod \"e2e-ipp-a-07f5f6-openshift-default-98746c89f-mr8c7\" (UID: \"648ce4f9-456f-48ff-a857-6d9bc7f61dc5\") " pod="openshift-ingress/e2e-ipp-a-07f5f6-openshift-default-98746c89f-mr8c7" Nov 03 01:02:12.237713 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:12.237661 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/648ce4f9-456f-48ff-a857-6d9bc7f61dc5-istio-podinfo\") pod \"e2e-ipp-a-07f5f6-openshift-default-98746c89f-mr8c7\" (UID: \"648ce4f9-456f-48ff-a857-6d9bc7f61dc5\") " pod="openshift-ingress/e2e-ipp-a-07f5f6-openshift-default-98746c89f-mr8c7" Nov 03 01:02:12.237763 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:12.237727 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/648ce4f9-456f-48ff-a857-6d9bc7f61dc5-credential-socket\") pod \"e2e-ipp-a-07f5f6-openshift-default-98746c89f-mr8c7\" (UID: \"648ce4f9-456f-48ff-a857-6d9bc7f61dc5\") " pod="openshift-ingress/e2e-ipp-a-07f5f6-openshift-default-98746c89f-mr8c7" Nov 03 01:02:12.237811 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:12.237796 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/648ce4f9-456f-48ff-a857-6d9bc7f61dc5-istio-token\") pod \"e2e-ipp-a-07f5f6-openshift-default-98746c89f-mr8c7\" (UID: \"648ce4f9-456f-48ff-a857-6d9bc7f61dc5\") " pod="openshift-ingress/e2e-ipp-a-07f5f6-openshift-default-98746c89f-mr8c7" Nov 03 01:02:12.237852 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:12.237825 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqq2m\" (UniqueName: \"kubernetes.io/projected/648ce4f9-456f-48ff-a857-6d9bc7f61dc5-kube-api-access-qqq2m\") pod \"e2e-ipp-a-07f5f6-openshift-default-98746c89f-mr8c7\" (UID: \"648ce4f9-456f-48ff-a857-6d9bc7f61dc5\") " pod="openshift-ingress/e2e-ipp-a-07f5f6-openshift-default-98746c89f-mr8c7" Nov 03 01:02:12.237943 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:12.237854 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/648ce4f9-456f-48ff-a857-6d9bc7f61dc5-istio-data\") pod \"e2e-ipp-a-07f5f6-openshift-default-98746c89f-mr8c7\" (UID: \"648ce4f9-456f-48ff-a857-6d9bc7f61dc5\") " pod="openshift-ingress/e2e-ipp-a-07f5f6-openshift-default-98746c89f-mr8c7" Nov 03 01:02:12.237982 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:12.237946 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/648ce4f9-456f-48ff-a857-6d9bc7f61dc5-istio-envoy\") pod \"e2e-ipp-a-07f5f6-openshift-default-98746c89f-mr8c7\" (UID: \"648ce4f9-456f-48ff-a857-6d9bc7f61dc5\") " pod="openshift-ingress/e2e-ipp-a-07f5f6-openshift-default-98746c89f-mr8c7" Nov 03 01:02:12.339405 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:12.339365 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/648ce4f9-456f-48ff-a857-6d9bc7f61dc5-istio-podinfo\") pod \"e2e-ipp-a-07f5f6-openshift-default-98746c89f-mr8c7\" (UID: \"648ce4f9-456f-48ff-a857-6d9bc7f61dc5\") " pod="openshift-ingress/e2e-ipp-a-07f5f6-openshift-default-98746c89f-mr8c7" Nov 03 01:02:12.339624 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:12.339421 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/648ce4f9-456f-48ff-a857-6d9bc7f61dc5-credential-socket\") pod \"e2e-ipp-a-07f5f6-openshift-default-98746c89f-mr8c7\" (UID: \"648ce4f9-456f-48ff-a857-6d9bc7f61dc5\") " pod="openshift-ingress/e2e-ipp-a-07f5f6-openshift-default-98746c89f-mr8c7" Nov 03 01:02:12.339624 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:12.339507 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/648ce4f9-456f-48ff-a857-6d9bc7f61dc5-istio-token\") pod \"e2e-ipp-a-07f5f6-openshift-default-98746c89f-mr8c7\" (UID: \"648ce4f9-456f-48ff-a857-6d9bc7f61dc5\") " pod="openshift-ingress/e2e-ipp-a-07f5f6-openshift-default-98746c89f-mr8c7" Nov 03 01:02:12.339624 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:12.339529 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqq2m\" (UniqueName: \"kubernetes.io/projected/648ce4f9-456f-48ff-a857-6d9bc7f61dc5-kube-api-access-qqq2m\") pod \"e2e-ipp-a-07f5f6-openshift-default-98746c89f-mr8c7\" (UID: \"648ce4f9-456f-48ff-a857-6d9bc7f61dc5\") " pod="openshift-ingress/e2e-ipp-a-07f5f6-openshift-default-98746c89f-mr8c7" Nov 03 01:02:12.339624 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:12.339568 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/648ce4f9-456f-48ff-a857-6d9bc7f61dc5-istio-data\") pod \"e2e-ipp-a-07f5f6-openshift-default-98746c89f-mr8c7\" (UID: \"648ce4f9-456f-48ff-a857-6d9bc7f61dc5\") " pod="openshift-ingress/e2e-ipp-a-07f5f6-openshift-default-98746c89f-mr8c7" Nov 03 01:02:12.339865 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:12.339651 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/648ce4f9-456f-48ff-a857-6d9bc7f61dc5-istio-envoy\") pod \"e2e-ipp-a-07f5f6-openshift-default-98746c89f-mr8c7\" (UID: \"648ce4f9-456f-48ff-a857-6d9bc7f61dc5\") " pod="openshift-ingress/e2e-ipp-a-07f5f6-openshift-default-98746c89f-mr8c7" Nov 03 01:02:12.339865 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:12.339729 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/648ce4f9-456f-48ff-a857-6d9bc7f61dc5-workload-socket\") pod \"e2e-ipp-a-07f5f6-openshift-default-98746c89f-mr8c7\" (UID: \"648ce4f9-456f-48ff-a857-6d9bc7f61dc5\") " pod="openshift-ingress/e2e-ipp-a-07f5f6-openshift-default-98746c89f-mr8c7" Nov 03 01:02:12.339865 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:12.339789 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/648ce4f9-456f-48ff-a857-6d9bc7f61dc5-workload-certs\") pod \"e2e-ipp-a-07f5f6-openshift-default-98746c89f-mr8c7\" (UID: \"648ce4f9-456f-48ff-a857-6d9bc7f61dc5\") " pod="openshift-ingress/e2e-ipp-a-07f5f6-openshift-default-98746c89f-mr8c7" Nov 03 01:02:12.339865 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:12.339839 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/648ce4f9-456f-48ff-a857-6d9bc7f61dc5-credential-socket\") pod \"e2e-ipp-a-07f5f6-openshift-default-98746c89f-mr8c7\" (UID: \"648ce4f9-456f-48ff-a857-6d9bc7f61dc5\") " pod="openshift-ingress/e2e-ipp-a-07f5f6-openshift-default-98746c89f-mr8c7" Nov 03 01:02:12.340083 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:12.339844 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/648ce4f9-456f-48ff-a857-6d9bc7f61dc5-istiod-ca-cert\") pod \"e2e-ipp-a-07f5f6-openshift-default-98746c89f-mr8c7\" (UID: \"648ce4f9-456f-48ff-a857-6d9bc7f61dc5\") " pod="openshift-ingress/e2e-ipp-a-07f5f6-openshift-default-98746c89f-mr8c7" Nov 03 01:02:12.340083 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:12.339926 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/648ce4f9-456f-48ff-a857-6d9bc7f61dc5-istio-data\") pod \"e2e-ipp-a-07f5f6-openshift-default-98746c89f-mr8c7\" (UID: \"648ce4f9-456f-48ff-a857-6d9bc7f61dc5\") " pod="openshift-ingress/e2e-ipp-a-07f5f6-openshift-default-98746c89f-mr8c7" Nov 03 01:02:12.340192 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:12.340163 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/648ce4f9-456f-48ff-a857-6d9bc7f61dc5-workload-socket\") pod \"e2e-ipp-a-07f5f6-openshift-default-98746c89f-mr8c7\" (UID: \"648ce4f9-456f-48ff-a857-6d9bc7f61dc5\") " pod="openshift-ingress/e2e-ipp-a-07f5f6-openshift-default-98746c89f-mr8c7" Nov 03 01:02:12.340259 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:12.340167 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/648ce4f9-456f-48ff-a857-6d9bc7f61dc5-workload-certs\") pod \"e2e-ipp-a-07f5f6-openshift-default-98746c89f-mr8c7\" (UID: \"648ce4f9-456f-48ff-a857-6d9bc7f61dc5\") " pod="openshift-ingress/e2e-ipp-a-07f5f6-openshift-default-98746c89f-mr8c7" Nov 03 01:02:12.340500 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:12.340479 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/648ce4f9-456f-48ff-a857-6d9bc7f61dc5-istiod-ca-cert\") pod \"e2e-ipp-a-07f5f6-openshift-default-98746c89f-mr8c7\" (UID: \"648ce4f9-456f-48ff-a857-6d9bc7f61dc5\") " pod="openshift-ingress/e2e-ipp-a-07f5f6-openshift-default-98746c89f-mr8c7" Nov 03 01:02:12.341880 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:12.341861 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/648ce4f9-456f-48ff-a857-6d9bc7f61dc5-istio-envoy\") pod \"e2e-ipp-a-07f5f6-openshift-default-98746c89f-mr8c7\" (UID: \"648ce4f9-456f-48ff-a857-6d9bc7f61dc5\") " pod="openshift-ingress/e2e-ipp-a-07f5f6-openshift-default-98746c89f-mr8c7" Nov 03 01:02:12.342111 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:12.342090 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/648ce4f9-456f-48ff-a857-6d9bc7f61dc5-istio-podinfo\") pod \"e2e-ipp-a-07f5f6-openshift-default-98746c89f-mr8c7\" (UID: \"648ce4f9-456f-48ff-a857-6d9bc7f61dc5\") " pod="openshift-ingress/e2e-ipp-a-07f5f6-openshift-default-98746c89f-mr8c7" Nov 03 01:02:12.347777 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:12.347758 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqq2m\" (UniqueName: \"kubernetes.io/projected/648ce4f9-456f-48ff-a857-6d9bc7f61dc5-kube-api-access-qqq2m\") pod \"e2e-ipp-a-07f5f6-openshift-default-98746c89f-mr8c7\" (UID: \"648ce4f9-456f-48ff-a857-6d9bc7f61dc5\") " pod="openshift-ingress/e2e-ipp-a-07f5f6-openshift-default-98746c89f-mr8c7" Nov 03 01:02:12.347881 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:12.347861 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/648ce4f9-456f-48ff-a857-6d9bc7f61dc5-istio-token\") pod \"e2e-ipp-a-07f5f6-openshift-default-98746c89f-mr8c7\" (UID: \"648ce4f9-456f-48ff-a857-6d9bc7f61dc5\") " pod="openshift-ingress/e2e-ipp-a-07f5f6-openshift-default-98746c89f-mr8c7" Nov 03 01:02:12.480684 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:12.480595 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-07f5f6-openshift-default-98746c89f-mr8c7" Nov 03 01:02:12.806300 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:12.798641 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-07f5f6-openshift-default-98746c89f-mr8c7"] Nov 03 01:02:12.832501 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:12.829655 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:02:12.832501 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:12.829771 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:02:12.832501 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:12.829814 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:02:13.798835 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:13.798794 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-07f5f6-openshift-default-98746c89f-mr8c7" event={"ID":"648ce4f9-456f-48ff-a857-6d9bc7f61dc5","Type":"ContainerStarted","Data":"e8d36fcabdc7131a5fe2cd29b7f0be64d86820085a902770d892bccf5421627e"} Nov 03 01:02:13.799311 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:13.798856 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-07f5f6-openshift-default-98746c89f-mr8c7" event={"ID":"648ce4f9-456f-48ff-a857-6d9bc7f61dc5","Type":"ContainerStarted","Data":"befd15367093e14b6f83c51a9fd3f73503d178d8dbffa64d662a654767714d3f"} Nov 03 01:02:13.839780 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:13.839723 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-07f5f6-openshift-default-98746c89f-mr8c7" podStartSLOduration=1.839702105 podStartE2EDuration="1.839702105s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:02:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:02:13.836644117 +0000 UTC m=+1276.007647213" watchObservedRunningTime="2026-11-03 01:02:13.839702105 +0000 UTC m=+1276.010705202" Nov 03 01:02:14.481202 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:14.481169 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-07f5f6-openshift-default-98746c89f-mr8c7" Nov 03 01:02:14.483918 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:14.483888 4421 patch_prober.go:29] interesting pod/e2e-ipp-a-07f5f6-openshift-default-98746c89f-mr8c7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" start-of-body= Nov 03 01:02:14.484074 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:14.483936 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-07f5f6-openshift-default-98746c89f-mr8c7" podUID="648ce4f9-456f-48ff-a857-6d9bc7f61dc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" Nov 03 01:02:14.672667 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:02:14.672626 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 01:02:15.482740 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:15.482699 4421 patch_prober.go:29] interesting pod/e2e-ipp-a-07f5f6-openshift-default-98746c89f-mr8c7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" start-of-body= Nov 03 01:02:15.483258 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:15.482780 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-07f5f6-openshift-default-98746c89f-mr8c7" podUID="648ce4f9-456f-48ff-a857-6d9bc7f61dc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" Nov 03 01:02:16.485300 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:16.485148 4421 patch_prober.go:29] interesting pod/e2e-ipp-a-07f5f6-openshift-default-98746c89f-mr8c7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" start-of-body= Nov 03 01:02:16.485300 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:16.485226 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-07f5f6-openshift-default-98746c89f-mr8c7" podUID="648ce4f9-456f-48ff-a857-6d9bc7f61dc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" Nov 03 01:02:16.556710 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:16.556671 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-ff4113-7756b5c9bf-9slj7" Nov 03 01:02:16.556903 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:16.556882 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ff4113-784c56fbc9-82ttz" Nov 03 01:02:17.481998 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:17.481964 4421 patch_prober.go:29] interesting pod/e2e-ipp-a-07f5f6-openshift-default-98746c89f-mr8c7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" start-of-body= Nov 03 01:02:17.482200 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:17.482024 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-07f5f6-openshift-default-98746c89f-mr8c7" podUID="648ce4f9-456f-48ff-a857-6d9bc7f61dc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" Nov 03 01:02:18.482312 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:18.482259 4421 patch_prober.go:29] interesting pod/e2e-ipp-a-07f5f6-openshift-default-98746c89f-mr8c7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" start-of-body= Nov 03 01:02:18.482717 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:18.482366 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-07f5f6-openshift-default-98746c89f-mr8c7" podUID="648ce4f9-456f-48ff-a857-6d9bc7f61dc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" Nov 03 01:02:19.652925 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:19.652892 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-07f5f6-openshift-default-98746c89f-mr8c7" Nov 03 01:02:19.653446 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:19.653235 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-07f5f6-openshift-default-98746c89f-mr8c7" Nov 03 01:02:19.654184 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:19.654166 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-07f5f6-openshift-default-98746c89f-mr8c7" Nov 03 01:02:26.031951 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:26.031915 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1ea8b2e-k2nfc"] Nov 03 01:02:26.035339 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:26.035315 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1ea8b2e-k2nfc"] Nov 03 01:02:28.007119 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:28.007078 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db84ab91-9005-4838-9c29-baad4bb6739b" path="/var/lib/kubelet/pods/db84ab91-9005-4838-9c29-baad4bb6739b/volumes" Nov 03 01:02:32.233792 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:32.233750 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-d5f78f-d665ddc99-pgdkv"] Nov 03 01:02:32.248532 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:32.248499 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-d5f78f-6f874c67f9-llvzf"] Nov 03 01:02:32.482498 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:32.482463 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ff4113-7756b5c9bf-9slj7"] Nov 03 01:02:32.572502 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:32.572465 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ff4113-784c56fbc9-82ttz"] Nov 03 01:02:34.034629 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:34.034581 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fac988dd-jkdwx"] Nov 03 01:02:34.039862 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:34.039833 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fac988dd-jkdwx"] Nov 03 01:02:35.981467 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:35.981433 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd1b22e9-50b0-4359-ae9b-7302bbc4cff1" path="/var/lib/kubelet/pods/fd1b22e9-50b0-4359-ae9b-7302bbc4cff1/volumes" Nov 03 01:02:36.400463 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:36.400424 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28b319b3-ch7k5"] Nov 03 01:02:36.410405 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:36.410367 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28b319b3-ch7k5" Nov 03 01:02:36.413933 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:36.413908 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28b319b3-ch7k5"] Nov 03 01:02:36.414801 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:36.414739 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:02:36.415871 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:36.415849 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 01:02:36.415871 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:36.415848 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6q4xp\"" Nov 03 01:02:36.416162 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:36.415985 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 01:02:36.511520 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:36.510878 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vq75c\" (UniqueName: \"kubernetes.io/projected/e13eb472-c996-46dd-b804-774416975943-kube-api-access-vq75c\") pod \"maas-api-revoke-sub-28b319b3-ch7k5\" (UID: \"e13eb472-c996-46dd-b804-774416975943\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28b319b3-ch7k5" Nov 03 01:02:36.511520 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:36.510951 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e13eb472-c996-46dd-b804-774416975943-maas-api-xks-ca\") pod \"maas-api-revoke-sub-28b319b3-ch7k5\" (UID: \"e13eb472-c996-46dd-b804-774416975943\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28b319b3-ch7k5" Nov 03 01:02:36.512319 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:36.512109 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e13eb472-c996-46dd-b804-774416975943-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-28b319b3-ch7k5\" (UID: \"e13eb472-c996-46dd-b804-774416975943\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28b319b3-ch7k5" Nov 03 01:02:36.613210 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:36.613170 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e13eb472-c996-46dd-b804-774416975943-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-28b319b3-ch7k5\" (UID: \"e13eb472-c996-46dd-b804-774416975943\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28b319b3-ch7k5" Nov 03 01:02:36.613701 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:36.613672 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vq75c\" (UniqueName: \"kubernetes.io/projected/e13eb472-c996-46dd-b804-774416975943-kube-api-access-vq75c\") pod \"maas-api-revoke-sub-28b319b3-ch7k5\" (UID: \"e13eb472-c996-46dd-b804-774416975943\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28b319b3-ch7k5" Nov 03 01:02:36.614982 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:36.614954 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e13eb472-c996-46dd-b804-774416975943-maas-api-xks-ca\") pod \"maas-api-revoke-sub-28b319b3-ch7k5\" (UID: \"e13eb472-c996-46dd-b804-774416975943\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28b319b3-ch7k5" Nov 03 01:02:36.615420 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:36.614590 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e13eb472-c996-46dd-b804-774416975943-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-28b319b3-ch7k5\" (UID: \"e13eb472-c996-46dd-b804-774416975943\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28b319b3-ch7k5" Nov 03 01:02:36.619720 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:36.619668 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e13eb472-c996-46dd-b804-774416975943-maas-api-xks-ca\") pod \"maas-api-revoke-sub-28b319b3-ch7k5\" (UID: \"e13eb472-c996-46dd-b804-774416975943\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28b319b3-ch7k5" Nov 03 01:02:36.655354 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:36.654374 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vq75c\" (UniqueName: \"kubernetes.io/projected/e13eb472-c996-46dd-b804-774416975943-kube-api-access-vq75c\") pod \"maas-api-revoke-sub-28b319b3-ch7k5\" (UID: \"e13eb472-c996-46dd-b804-774416975943\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28b319b3-ch7k5" Nov 03 01:02:36.740881 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:36.740843 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28b319b3-ch7k5" Nov 03 01:02:37.068512 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:37.067575 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28b319b3-ch7k5"] Nov 03 01:02:37.138706 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:37.138671 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28b319b3-ch7k5" event={"ID":"e13eb472-c996-46dd-b804-774416975943","Type":"ContainerStarted","Data":"ce8bd38a505eff7dc16dc01c7a9f3b8b124497def82887f27cf916971d8ae691"} Nov 03 01:02:37.234924 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:37.234869 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-d5f78f-d665ddc99-pgdkv" podUID="1695554d-050f-4de0-bced-428c9325bead" containerName="payload-processing" containerID="cri-o://1cea72e08e4324325f47ba1bf67dbc49ebeba890183393763ebb63b980d760e8" gracePeriod=25 Nov 03 01:02:37.249527 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:37.249488 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d5f78f-6f874c67f9-llvzf" podUID="b72d8af9-6f28-4ca8-a032-54a60293a7b3" containerName="payload-pre-processing" containerID="cri-o://45c8b8632a8a87bc7ab3d849afc6e226092b2222b2db75528e8e32b3e20a342c" gracePeriod=25 Nov 03 01:02:37.483847 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:37.483638 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-ff4113-7756b5c9bf-9slj7" podUID="ae55f712-b9f8-4b20-bf6f-6edd47ae643c" containerName="payload-processing" containerID="cri-o://5022aaaaba94362d9a312545a9e4df5324edbd14aeb56231705f0b6ca22a22e4" gracePeriod=25 Nov 03 01:02:37.574132 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:37.574081 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ff4113-784c56fbc9-82ttz" podUID="73c99eb0-c089-4d8c-9d83-4163fb106ded" containerName="payload-pre-processing" containerID="cri-o://6371c6ca37a701de9f9efc78f0525ba8b50fe441b287fd9cb5d96792174863ef" gracePeriod=25 Nov 03 01:02:37.613405 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:37.613223 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-d5f78f-d665ddc99-pgdkv" Nov 03 01:02:37.640509 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:37.638993 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1695554d-050f-4de0-bced-428c9325bead-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1695554d-050f-4de0-bced-428c9325bead-plugins-config-volume\") pod \"1695554d-050f-4de0-bced-428c9325bead\" (UID: \"1695554d-050f-4de0-bced-428c9325bead\") " Nov 03 01:02:37.640509 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:37.639284 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1695554d-050f-4de0-bced-428c9325bead-kube-api-access-5rwdh\" (UniqueName: \"kubernetes.io/projected/1695554d-050f-4de0-bced-428c9325bead-kube-api-access-5rwdh\") pod \"1695554d-050f-4de0-bced-428c9325bead\" (UID: \"1695554d-050f-4de0-bced-428c9325bead\") " Nov 03 01:02:37.641567 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:37.641540 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1695554d-050f-4de0-bced-428c9325bead-plugins-config-volume" pod "1695554d-050f-4de0-bced-428c9325bead" (UID: "1695554d-050f-4de0-bced-428c9325bead"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:02:37.647126 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:37.645800 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1695554d-050f-4de0-bced-428c9325bead-kube-api-access-5rwdh" pod "1695554d-050f-4de0-bced-428c9325bead" (UID: "1695554d-050f-4de0-bced-428c9325bead"). InnerVolumeSpecName "kube-api-access-5rwdh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:02:37.723694 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:37.723657 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d5f78f-6f874c67f9-llvzf" Nov 03 01:02:37.744855 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:37.744774 4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1695554d-050f-4de0-bced-428c9325bead-plugins-config-volume\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:02:37.744855 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:37.744813 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rwdh\" (UniqueName: \"kubernetes.io/projected/1695554d-050f-4de0-bced-428c9325bead-kube-api-access-5rwdh\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:02:37.915183 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:37.915158 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ff4113-7756b5c9bf-9slj7" Nov 03 01:02:37.961703 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:37.961669 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae55f712-b9f8-4b20-bf6f-6edd47ae643c-kube-api-access-fhq9w\" (UniqueName: \"kubernetes.io/projected/ae55f712-b9f8-4b20-bf6f-6edd47ae643c-kube-api-access-fhq9w\") pod \"ae55f712-b9f8-4b20-bf6f-6edd47ae643c\" (UID: \"ae55f712-b9f8-4b20-bf6f-6edd47ae643c\") " Nov 03 01:02:37.961868 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:37.961853 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b72d8af9-6f28-4ca8-a032-54a60293a7b3-kube-api-access-bn4vw\" (UniqueName: \"kubernetes.io/projected/b72d8af9-6f28-4ca8-a032-54a60293a7b3-kube-api-access-bn4vw\") pod \"b72d8af9-6f28-4ca8-a032-54a60293a7b3\" (UID: \"b72d8af9-6f28-4ca8-a032-54a60293a7b3\") " Nov 03 01:02:37.961956 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:37.961939 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b72d8af9-6f28-4ca8-a032-54a60293a7b3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b72d8af9-6f28-4ca8-a032-54a60293a7b3-plugins-config-volume\") pod \"b72d8af9-6f28-4ca8-a032-54a60293a7b3\" (UID: \"b72d8af9-6f28-4ca8-a032-54a60293a7b3\") " Nov 03 01:02:37.962090 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:37.962071 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae55f712-b9f8-4b20-bf6f-6edd47ae643c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae55f712-b9f8-4b20-bf6f-6edd47ae643c-plugins-config-volume\") pod \"ae55f712-b9f8-4b20-bf6f-6edd47ae643c\" (UID: \"ae55f712-b9f8-4b20-bf6f-6edd47ae643c\") " Nov 03 01:02:37.962403 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:37.962371 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b72d8af9-6f28-4ca8-a032-54a60293a7b3-plugins-config-volume" pod "b72d8af9-6f28-4ca8-a032-54a60293a7b3" (UID: "b72d8af9-6f28-4ca8-a032-54a60293a7b3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:02:37.962646 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:37.962617 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae55f712-b9f8-4b20-bf6f-6edd47ae643c-plugins-config-volume" pod "ae55f712-b9f8-4b20-bf6f-6edd47ae643c" (UID: "ae55f712-b9f8-4b20-bf6f-6edd47ae643c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:02:37.962939 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:37.962916 4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b72d8af9-6f28-4ca8-a032-54a60293a7b3-plugins-config-volume\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:02:37.963022 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:37.962948 4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae55f712-b9f8-4b20-bf6f-6edd47ae643c-plugins-config-volume\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:02:37.964331 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:37.964304 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae55f712-b9f8-4b20-bf6f-6edd47ae643c-kube-api-access-fhq9w" pod "ae55f712-b9f8-4b20-bf6f-6edd47ae643c" (UID: "ae55f712-b9f8-4b20-bf6f-6edd47ae643c"). InnerVolumeSpecName "kube-api-access-fhq9w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:02:37.964423 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:37.964326 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b72d8af9-6f28-4ca8-a032-54a60293a7b3-kube-api-access-bn4vw" pod "b72d8af9-6f28-4ca8-a032-54a60293a7b3" (UID: "b72d8af9-6f28-4ca8-a032-54a60293a7b3"). InnerVolumeSpecName "kube-api-access-bn4vw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:02:38.007901 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:38.007872 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ff4113-784c56fbc9-82ttz" Nov 03 01:02:38.064571 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:38.064534 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73c99eb0-c089-4d8c-9d83-4163fb106ded-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73c99eb0-c089-4d8c-9d83-4163fb106ded-plugins-config-volume\") pod \"73c99eb0-c089-4d8c-9d83-4163fb106ded\" (UID: \"73c99eb0-c089-4d8c-9d83-4163fb106ded\") " Nov 03 01:02:38.064776 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:38.064741 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73c99eb0-c089-4d8c-9d83-4163fb106ded-kube-api-access-8bp2b\" (UniqueName: \"kubernetes.io/projected/73c99eb0-c089-4d8c-9d83-4163fb106ded-kube-api-access-8bp2b\") pod \"73c99eb0-c089-4d8c-9d83-4163fb106ded\" (UID: \"73c99eb0-c089-4d8c-9d83-4163fb106ded\") " Nov 03 01:02:38.065066 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:38.065018 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73c99eb0-c089-4d8c-9d83-4163fb106ded-plugins-config-volume" pod "73c99eb0-c089-4d8c-9d83-4163fb106ded" (UID: "73c99eb0-c089-4d8c-9d83-4163fb106ded"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:02:38.065372 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:38.065349 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhq9w\" (UniqueName: \"kubernetes.io/projected/ae55f712-b9f8-4b20-bf6f-6edd47ae643c-kube-api-access-fhq9w\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:02:38.065470 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:38.065376 4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73c99eb0-c089-4d8c-9d83-4163fb106ded-plugins-config-volume\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:02:38.065470 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:38.065394 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bn4vw\" (UniqueName: \"kubernetes.io/projected/b72d8af9-6f28-4ca8-a032-54a60293a7b3-kube-api-access-bn4vw\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:02:38.068480 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:38.068452 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73c99eb0-c089-4d8c-9d83-4163fb106ded-kube-api-access-8bp2b" pod "73c99eb0-c089-4d8c-9d83-4163fb106ded" (UID: "73c99eb0-c089-4d8c-9d83-4163fb106ded"). InnerVolumeSpecName "kube-api-access-8bp2b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:02:38.149027 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:38.148947 4421 generic.go:356] "Generic (PLEG): container finished" podID="73c99eb0-c089-4d8c-9d83-4163fb106ded" containerID="6371c6ca37a701de9f9efc78f0525ba8b50fe441b287fd9cb5d96792174863ef" exitCode=0 Nov 03 01:02:38.149027 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:38.148992 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ff4113-784c56fbc9-82ttz" Nov 03 01:02:38.149613 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:38.149027 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ff4113-784c56fbc9-82ttz" event={"ID":"73c99eb0-c089-4d8c-9d83-4163fb106ded","Type":"ContainerDied","Data":"6371c6ca37a701de9f9efc78f0525ba8b50fe441b287fd9cb5d96792174863ef"} Nov 03 01:02:38.149613 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:38.149081 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ff4113-784c56fbc9-82ttz" event={"ID":"73c99eb0-c089-4d8c-9d83-4163fb106ded","Type":"ContainerDied","Data":"27b6f486b839da2bf54b1db486a0af4c327729b07cf2d2f7757c8a347aa6930a"} Nov 03 01:02:38.149613 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:38.149111 4421 scope.go:122] "RemoveContainer" containerID="6371c6ca37a701de9f9efc78f0525ba8b50fe441b287fd9cb5d96792174863ef" Nov 03 01:02:38.150855 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:38.150832 4421 generic.go:356] "Generic (PLEG): container finished" podID="1695554d-050f-4de0-bced-428c9325bead" containerID="1cea72e08e4324325f47ba1bf67dbc49ebeba890183393763ebb63b980d760e8" exitCode=0 Nov 03 01:02:38.151064 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:38.150884 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-d5f78f-d665ddc99-pgdkv" Nov 03 01:02:38.151586 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:38.150901 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-d5f78f-d665ddc99-pgdkv" event={"ID":"1695554d-050f-4de0-bced-428c9325bead","Type":"ContainerDied","Data":"1cea72e08e4324325f47ba1bf67dbc49ebeba890183393763ebb63b980d760e8"} Nov 03 01:02:38.151703 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:38.151600 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-d5f78f-d665ddc99-pgdkv" event={"ID":"1695554d-050f-4de0-bced-428c9325bead","Type":"ContainerDied","Data":"83e62eebbdf2194714c755d0bce4b56ba5ccf1bf92067c7d4cc9c307423a6f86"} Nov 03 01:02:38.153208 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:38.153187 4421 generic.go:356] "Generic (PLEG): container finished" podID="b72d8af9-6f28-4ca8-a032-54a60293a7b3" containerID="45c8b8632a8a87bc7ab3d849afc6e226092b2222b2db75528e8e32b3e20a342c" exitCode=0 Nov 03 01:02:38.153375 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:38.153227 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d5f78f-6f874c67f9-llvzf" Nov 03 01:02:38.153455 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:38.153311 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d5f78f-6f874c67f9-llvzf" event={"ID":"b72d8af9-6f28-4ca8-a032-54a60293a7b3","Type":"ContainerDied","Data":"45c8b8632a8a87bc7ab3d849afc6e226092b2222b2db75528e8e32b3e20a342c"} Nov 03 01:02:38.153581 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:38.153544 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d5f78f-6f874c67f9-llvzf" event={"ID":"b72d8af9-6f28-4ca8-a032-54a60293a7b3","Type":"ContainerDied","Data":"90600d5debbb5082923bfa38cb734a047556cd312e005bcb8a2c4d04a0f878a7"} Nov 03 01:02:38.158669 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:38.158644 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28b319b3-ch7k5" event={"ID":"e13eb472-c996-46dd-b804-774416975943","Type":"ContainerStarted","Data":"c70e9a0de0311127a9a7c6f499cae8d29fba148e24b91ea9421b96d87359d62a"} Nov 03 01:02:38.160180 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:38.160158 4421 generic.go:356] "Generic (PLEG): container finished" podID="ae55f712-b9f8-4b20-bf6f-6edd47ae643c" containerID="5022aaaaba94362d9a312545a9e4df5324edbd14aeb56231705f0b6ca22a22e4" exitCode=0 Nov 03 01:02:38.160248 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:38.160198 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ff4113-7756b5c9bf-9slj7" event={"ID":"ae55f712-b9f8-4b20-bf6f-6edd47ae643c","Type":"ContainerDied","Data":"5022aaaaba94362d9a312545a9e4df5324edbd14aeb56231705f0b6ca22a22e4"} Nov 03 01:02:38.160248 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:38.160234 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ff4113-7756b5c9bf-9slj7" event={"ID":"ae55f712-b9f8-4b20-bf6f-6edd47ae643c","Type":"ContainerDied","Data":"0f1f21de2ce6ae1f95a1314a2057fa1955c1deacc333a777acb175c128e0ab4d"} Nov 03 01:02:38.160416 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:38.160406 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ff4113-7756b5c9bf-9slj7" Nov 03 01:02:38.166373 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:38.166344 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8bp2b\" (UniqueName: \"kubernetes.io/projected/73c99eb0-c089-4d8c-9d83-4163fb106ded-kube-api-access-8bp2b\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:02:38.167902 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:38.167889 4421 scope.go:122] "RemoveContainer" containerID="6371c6ca37a701de9f9efc78f0525ba8b50fe441b287fd9cb5d96792174863ef" Nov 03 01:02:38.168194 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:02:38.168178 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6371c6ca37a701de9f9efc78f0525ba8b50fe441b287fd9cb5d96792174863ef\": container with ID starting with 6371c6ca37a701de9f9efc78f0525ba8b50fe441b287fd9cb5d96792174863ef not found: ID does not exist" containerID="6371c6ca37a701de9f9efc78f0525ba8b50fe441b287fd9cb5d96792174863ef" Nov 03 01:02:38.168247 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:38.168202 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6371c6ca37a701de9f9efc78f0525ba8b50fe441b287fd9cb5d96792174863ef"} err="failed to get container status \"6371c6ca37a701de9f9efc78f0525ba8b50fe441b287fd9cb5d96792174863ef\": rpc error: code = NotFound desc = could not find container \"6371c6ca37a701de9f9efc78f0525ba8b50fe441b287fd9cb5d96792174863ef\": container with ID starting with 6371c6ca37a701de9f9efc78f0525ba8b50fe441b287fd9cb5d96792174863ef not found: ID does not exist" Nov 03 01:02:38.168247 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:38.168217 4421 scope.go:122] "RemoveContainer" containerID="1cea72e08e4324325f47ba1bf67dbc49ebeba890183393763ebb63b980d760e8" Nov 03 01:02:38.176542 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:38.176505 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28b319b3-ch7k5" podStartSLOduration=2.176493586 podStartE2EDuration="2.176493586s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:02:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:02:38.174940929 +0000 UTC m=+1300.345944028" watchObservedRunningTime="2026-11-03 01:02:38.176493586 +0000 UTC m=+1300.347496681" Nov 03 01:02:38.188293 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:38.188256 4421 scope.go:122] "RemoveContainer" containerID="1cea72e08e4324325f47ba1bf67dbc49ebeba890183393763ebb63b980d760e8" Nov 03 01:02:38.188691 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:02:38.188663 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1cea72e08e4324325f47ba1bf67dbc49ebeba890183393763ebb63b980d760e8\": container with ID starting with 1cea72e08e4324325f47ba1bf67dbc49ebeba890183393763ebb63b980d760e8 not found: ID does not exist" containerID="1cea72e08e4324325f47ba1bf67dbc49ebeba890183393763ebb63b980d760e8" Nov 03 01:02:38.188785 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:38.188693 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1cea72e08e4324325f47ba1bf67dbc49ebeba890183393763ebb63b980d760e8"} err="failed to get container status \"1cea72e08e4324325f47ba1bf67dbc49ebeba890183393763ebb63b980d760e8\": rpc error: code = NotFound desc = could not find container \"1cea72e08e4324325f47ba1bf67dbc49ebeba890183393763ebb63b980d760e8\": container with ID starting with 1cea72e08e4324325f47ba1bf67dbc49ebeba890183393763ebb63b980d760e8 not found: ID does not exist" Nov 03 01:02:38.188785 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:38.188712 4421 scope.go:122] "RemoveContainer" containerID="45c8b8632a8a87bc7ab3d849afc6e226092b2222b2db75528e8e32b3e20a342c" Nov 03 01:02:38.189883 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:38.189811 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ff4113-7756b5c9bf-9slj7"] Nov 03 01:02:38.195981 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:38.195953 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ff4113-7756b5c9bf-9slj7"] Nov 03 01:02:38.206416 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:38.206243 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-d5f78f-d665ddc99-pgdkv"] Nov 03 01:02:38.209876 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:38.209852 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-d5f78f-d665ddc99-pgdkv"] Nov 03 01:02:38.210374 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:38.210357 4421 scope.go:122] "RemoveContainer" containerID="45c8b8632a8a87bc7ab3d849afc6e226092b2222b2db75528e8e32b3e20a342c" Nov 03 01:02:38.210752 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:02:38.210719 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45c8b8632a8a87bc7ab3d849afc6e226092b2222b2db75528e8e32b3e20a342c\": container with ID starting with 45c8b8632a8a87bc7ab3d849afc6e226092b2222b2db75528e8e32b3e20a342c not found: ID does not exist" containerID="45c8b8632a8a87bc7ab3d849afc6e226092b2222b2db75528e8e32b3e20a342c" Nov 03 01:02:38.210855 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:38.210758 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45c8b8632a8a87bc7ab3d849afc6e226092b2222b2db75528e8e32b3e20a342c"} err="failed to get container status \"45c8b8632a8a87bc7ab3d849afc6e226092b2222b2db75528e8e32b3e20a342c\": rpc error: code = NotFound desc = could not find container \"45c8b8632a8a87bc7ab3d849afc6e226092b2222b2db75528e8e32b3e20a342c\": container with ID starting with 45c8b8632a8a87bc7ab3d849afc6e226092b2222b2db75528e8e32b3e20a342c not found: ID does not exist" Nov 03 01:02:38.210855 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:38.210778 4421 scope.go:122] "RemoveContainer" containerID="5022aaaaba94362d9a312545a9e4df5324edbd14aeb56231705f0b6ca22a22e4" Nov 03 01:02:38.223064 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:38.223036 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ff4113-784c56fbc9-82ttz"] Nov 03 01:02:38.227911 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:38.227885 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ff4113-784c56fbc9-82ttz"] Nov 03 01:02:38.228079 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:38.228055 4421 scope.go:122] "RemoveContainer" containerID="5022aaaaba94362d9a312545a9e4df5324edbd14aeb56231705f0b6ca22a22e4" Nov 03 01:02:38.228467 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:02:38.228441 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5022aaaaba94362d9a312545a9e4df5324edbd14aeb56231705f0b6ca22a22e4\": container with ID starting with 5022aaaaba94362d9a312545a9e4df5324edbd14aeb56231705f0b6ca22a22e4 not found: ID does not exist" containerID="5022aaaaba94362d9a312545a9e4df5324edbd14aeb56231705f0b6ca22a22e4" Nov 03 01:02:38.228577 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:38.228481 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5022aaaaba94362d9a312545a9e4df5324edbd14aeb56231705f0b6ca22a22e4"} err="failed to get container status \"5022aaaaba94362d9a312545a9e4df5324edbd14aeb56231705f0b6ca22a22e4\": rpc error: code = NotFound desc = could not find container \"5022aaaaba94362d9a312545a9e4df5324edbd14aeb56231705f0b6ca22a22e4\": container with ID starting with 5022aaaaba94362d9a312545a9e4df5324edbd14aeb56231705f0b6ca22a22e4 not found: ID does not exist" Nov 03 01:02:38.240863 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:38.240836 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-d5f78f-6f874c67f9-llvzf"] Nov 03 01:02:38.244460 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:38.244428 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-d5f78f-6f874c67f9-llvzf"] Nov 03 01:02:39.169236 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:39.169199 4421 generic.go:356] "Generic (PLEG): container finished" podID="e13eb472-c996-46dd-b804-774416975943" containerID="c70e9a0de0311127a9a7c6f499cae8d29fba148e24b91ea9421b96d87359d62a" exitCode=0 Nov 03 01:02:39.171817 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:39.169308 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28b319b3-ch7k5" event={"ID":"e13eb472-c996-46dd-b804-774416975943","Type":"ContainerDied","Data":"c70e9a0de0311127a9a7c6f499cae8d29fba148e24b91ea9421b96d87359d62a"} Nov 03 01:02:39.988327 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:39.984937 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1695554d-050f-4de0-bced-428c9325bead" path="/var/lib/kubelet/pods/1695554d-050f-4de0-bced-428c9325bead/volumes" Nov 03 01:02:39.988327 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:39.985602 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73c99eb0-c089-4d8c-9d83-4163fb106ded" path="/var/lib/kubelet/pods/73c99eb0-c089-4d8c-9d83-4163fb106ded/volumes" Nov 03 01:02:39.988327 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:39.986440 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae55f712-b9f8-4b20-bf6f-6edd47ae643c" path="/var/lib/kubelet/pods/ae55f712-b9f8-4b20-bf6f-6edd47ae643c/volumes" Nov 03 01:02:39.988327 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:39.986981 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b72d8af9-6f28-4ca8-a032-54a60293a7b3" path="/var/lib/kubelet/pods/b72d8af9-6f28-4ca8-a032-54a60293a7b3/volumes" Nov 03 01:02:40.186574 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:40.186489 4421 generic.go:356] "Generic (PLEG): container finished" podID="6ab8d9c7-c645-4569-81a8-c5b1edaa7857" containerID="b478cd4bdd25d7b2802f37b37f38112c582f06cbd4121a2c311eb0d585a60df9" exitCode=0 Nov 03 01:02:40.186574 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:40.186559 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-ff4113-openshift-default-6b94f7db5f-4p87g" event={"ID":"6ab8d9c7-c645-4569-81a8-c5b1edaa7857","Type":"ContainerDied","Data":"b478cd4bdd25d7b2802f37b37f38112c582f06cbd4121a2c311eb0d585a60df9"} Nov 03 01:02:40.187099 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:40.186983 4421 scope.go:122] "RemoveContainer" containerID="b478cd4bdd25d7b2802f37b37f38112c582f06cbd4121a2c311eb0d585a60df9" Nov 03 01:02:40.188804 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:40.188775 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:02:40.189063 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:40.188839 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:02:40.189063 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:40.188879 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:02:40.291154 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:40.291091 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ff4113-679c7cdbdb-cgkkl"] Nov 03 01:02:40.292177 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:40.292148 4421 cpu_manager.go:414] "containerMap: removing container" podUID="1695554d-050f-4de0-bced-428c9325bead" containerName="payload-processing" Nov 03 01:02:40.292177 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:40.292173 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1695554d-050f-4de0-bced-428c9325bead" containerName="payload-processing" Nov 03 01:02:40.292429 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:40.292193 4421 cpu_manager.go:414] "containerMap: removing container" podUID="73c99eb0-c089-4d8c-9d83-4163fb106ded" containerName="payload-pre-processing" Nov 03 01:02:40.292429 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:40.292201 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73c99eb0-c089-4d8c-9d83-4163fb106ded" containerName="payload-pre-processing" Nov 03 01:02:40.292429 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:40.292227 4421 cpu_manager.go:414] "containerMap: removing container" podUID="b72d8af9-6f28-4ca8-a032-54a60293a7b3" containerName="payload-pre-processing" Nov 03 01:02:40.292429 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:40.292235 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b72d8af9-6f28-4ca8-a032-54a60293a7b3" containerName="payload-pre-processing" Nov 03 01:02:40.292429 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:40.292248 4421 cpu_manager.go:414] "containerMap: removing container" podUID="ae55f712-b9f8-4b20-bf6f-6edd47ae643c" containerName="payload-processing" Nov 03 01:02:40.292429 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:40.292256 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae55f712-b9f8-4b20-bf6f-6edd47ae643c" containerName="payload-processing" Nov 03 01:02:40.293020 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:40.293000 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="1695554d-050f-4de0-bced-428c9325bead" containerName="payload-processing" Nov 03 01:02:40.293075 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:40.293027 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="73c99eb0-c089-4d8c-9d83-4163fb106ded" containerName="payload-pre-processing" Nov 03 01:02:40.293075 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:40.293041 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="b72d8af9-6f28-4ca8-a032-54a60293a7b3" containerName="payload-pre-processing" Nov 03 01:02:40.293075 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:40.293051 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae55f712-b9f8-4b20-bf6f-6edd47ae643c" containerName="payload-processing" Nov 03 01:02:40.312767 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:40.309766 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ff4113-679c7cdbdb-cgkkl"] Nov 03 01:02:40.312767 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:40.309916 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ff4113-679c7cdbdb-cgkkl" Nov 03 01:02:40.330383 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:40.328510 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-ff4113\"" Nov 03 01:02:40.378508 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:40.373579 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-ff4113-dockercfg-95nfb\"" Nov 03 01:02:40.557464 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:40.550924 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2t55v\" (UniqueName: \"kubernetes.io/projected/e35bd864-89b4-4a36-b59f-b623da7a86ac-kube-api-access-2t55v\") pod \"payload-processing-e2e-ipp-b-ff4113-679c7cdbdb-cgkkl\" (UID: \"e35bd864-89b4-4a36-b59f-b623da7a86ac\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ff4113-679c7cdbdb-cgkkl" Nov 03 01:02:40.557464 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:40.550987 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e35bd864-89b4-4a36-b59f-b623da7a86ac-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-ff4113-679c7cdbdb-cgkkl\" (UID: \"e35bd864-89b4-4a36-b59f-b623da7a86ac\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ff4113-679c7cdbdb-cgkkl" Nov 03 01:02:40.626332 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:40.625118 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28b319b3-ch7k5" Nov 03 01:02:40.651919 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:40.651888 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e13eb472-c996-46dd-b804-774416975943-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e13eb472-c996-46dd-b804-774416975943-maas-api-xks-ca\") pod \"e13eb472-c996-46dd-b804-774416975943\" (UID: \"e13eb472-c996-46dd-b804-774416975943\") " Nov 03 01:02:40.652113 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:40.651940 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e13eb472-c996-46dd-b804-774416975943-kube-api-access-vq75c\" (UniqueName: \"kubernetes.io/projected/e13eb472-c996-46dd-b804-774416975943-kube-api-access-vq75c\") pod \"e13eb472-c996-46dd-b804-774416975943\" (UID: \"e13eb472-c996-46dd-b804-774416975943\") " Nov 03 01:02:40.652113 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:40.651977 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e13eb472-c996-46dd-b804-774416975943-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e13eb472-c996-46dd-b804-774416975943-maas-api-openshift-ca\") pod \"e13eb472-c996-46dd-b804-774416975943\" (UID: \"e13eb472-c996-46dd-b804-774416975943\") " Nov 03 01:02:40.652817 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:40.652786 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e13eb472-c996-46dd-b804-774416975943-maas-api-openshift-ca" pod "e13eb472-c996-46dd-b804-774416975943" (UID: "e13eb472-c996-46dd-b804-774416975943"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:02:40.652817 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:40.652792 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2t55v\" (UniqueName: \"kubernetes.io/projected/e35bd864-89b4-4a36-b59f-b623da7a86ac-kube-api-access-2t55v\") pod \"payload-processing-e2e-ipp-b-ff4113-679c7cdbdb-cgkkl\" (UID: \"e35bd864-89b4-4a36-b59f-b623da7a86ac\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ff4113-679c7cdbdb-cgkkl" Nov 03 01:02:40.653026 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:40.652875 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e35bd864-89b4-4a36-b59f-b623da7a86ac-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-ff4113-679c7cdbdb-cgkkl\" (UID: \"e35bd864-89b4-4a36-b59f-b623da7a86ac\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ff4113-679c7cdbdb-cgkkl" Nov 03 01:02:40.653192 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:40.653175 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e13eb472-c996-46dd-b804-774416975943-maas-api-openshift-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:02:40.654072 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:40.654033 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e35bd864-89b4-4a36-b59f-b623da7a86ac-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-ff4113-679c7cdbdb-cgkkl\" (UID: \"e35bd864-89b4-4a36-b59f-b623da7a86ac\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ff4113-679c7cdbdb-cgkkl" Nov 03 01:02:40.656407 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:40.654601 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e13eb472-c996-46dd-b804-774416975943-maas-api-xks-ca" pod "e13eb472-c996-46dd-b804-774416975943" (UID: "e13eb472-c996-46dd-b804-774416975943"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:02:40.656407 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:40.654923 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e13eb472-c996-46dd-b804-774416975943-kube-api-access-vq75c" pod "e13eb472-c996-46dd-b804-774416975943" (UID: "e13eb472-c996-46dd-b804-774416975943"). InnerVolumeSpecName "kube-api-access-vq75c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:02:40.661010 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:40.660980 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2t55v\" (UniqueName: \"kubernetes.io/projected/e35bd864-89b4-4a36-b59f-b623da7a86ac-kube-api-access-2t55v\") pod \"payload-processing-e2e-ipp-b-ff4113-679c7cdbdb-cgkkl\" (UID: \"e35bd864-89b4-4a36-b59f-b623da7a86ac\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ff4113-679c7cdbdb-cgkkl" Nov 03 01:02:40.754797 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:40.754758 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e13eb472-c996-46dd-b804-774416975943-maas-api-xks-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:02:40.754797 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:40.754791 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vq75c\" (UniqueName: \"kubernetes.io/projected/e13eb472-c996-46dd-b804-774416975943-kube-api-access-vq75c\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:02:40.947161 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:40.946780 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ff4113-679c7cdbdb-cgkkl" Nov 03 01:02:41.238902 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:41.205756 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28b319b3-ch7k5" event={"ID":"e13eb472-c996-46dd-b804-774416975943","Type":"ContainerDied","Data":"ce8bd38a505eff7dc16dc01c7a9f3b8b124497def82887f27cf916971d8ae691"} Nov 03 01:02:41.238902 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:41.205821 4421 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ce8bd38a505eff7dc16dc01c7a9f3b8b124497def82887f27cf916971d8ae691" Nov 03 01:02:41.238902 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:41.205899 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28b319b3-ch7k5" Nov 03 01:02:41.238902 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:41.220954 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-ff4113-openshift-default-6b94f7db5f-4p87g" event={"ID":"6ab8d9c7-c645-4569-81a8-c5b1edaa7857","Type":"ContainerStarted","Data":"666f452f17f45ddc14cc8cde9b4809fc6a78eb9255dcc158e2021173ba609ffa"} Nov 03 01:02:41.268189 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:41.268159 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-ff4113-openshift-default-6b94f7db5f-4p87g" Nov 03 01:02:41.269062 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:41.269036 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ff4113-679c7cdbdb-cgkkl"] Nov 03 01:02:41.269479 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:41.269443 4421 patch_prober.go:29] interesting pod/e2e-ipp-b-ff4113-openshift-default-6b94f7db5f-4p87g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body= Nov 03 01:02:41.269565 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:41.269498 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-ff4113-openshift-default-6b94f7db5f-4p87g" podUID="6ab8d9c7-c645-4569-81a8-c5b1edaa7857" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" Nov 03 01:02:41.269565 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:02:41.269511 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode35bd864_89b4_4a36_b59f_b623da7a86ac.slice/crio-7a0e6c3a4d4eac6bc523b05fc22135da5e8629bf192782ea3b6b437a7981c7a9 WatchSource:0}: Error finding container 7a0e6c3a4d4eac6bc523b05fc22135da5e8629bf192782ea3b6b437a7981c7a9: Status 404 returned error can't find the container with id 7a0e6c3a4d4eac6bc523b05fc22135da5e8629bf192782ea3b6b437a7981c7a9 Nov 03 01:02:42.251044 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:42.250373 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ff4113-679c7cdbdb-cgkkl" event={"ID":"e35bd864-89b4-4a36-b59f-b623da7a86ac","Type":"ContainerStarted","Data":"6222b760dcc5b9f9fa1adadb50368f14bd2193512c749f3c7dfe3bc326eb6c6f"} Nov 03 01:02:42.251044 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:42.250440 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ff4113-679c7cdbdb-cgkkl" event={"ID":"e35bd864-89b4-4a36-b59f-b623da7a86ac","Type":"ContainerStarted","Data":"7a0e6c3a4d4eac6bc523b05fc22135da5e8629bf192782ea3b6b437a7981c7a9"} Nov 03 01:02:42.251044 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:42.250476 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-ff4113-679c7cdbdb-cgkkl" Nov 03 01:02:42.268333 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:42.268254 4421 patch_prober.go:29] interesting pod/e2e-ipp-b-ff4113-openshift-default-6b94f7db5f-4p87g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body= Nov 03 01:02:42.268636 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:42.268609 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-ff4113-openshift-default-6b94f7db5f-4p87g" podUID="6ab8d9c7-c645-4569-81a8-c5b1edaa7857" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" Nov 03 01:02:42.282607 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:42.282548 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-ff4113-679c7cdbdb-cgkkl" podStartSLOduration=2.282530183 podStartE2EDuration="2.282530183s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:02:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:02:42.276348853 +0000 UTC m=+1304.447351950" watchObservedRunningTime="2026-11-03 01:02:42.282530183 +0000 UTC m=+1304.453533271" Nov 03 01:02:43.269214 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:43.269172 4421 patch_prober.go:29] interesting pod/e2e-ipp-b-ff4113-openshift-default-6b94f7db5f-4p87g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body= Nov 03 01:02:43.270000 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:43.269957 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-ff4113-openshift-default-6b94f7db5f-4p87g" podUID="6ab8d9c7-c645-4569-81a8-c5b1edaa7857" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" Nov 03 01:02:44.268416 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:44.268378 4421 patch_prober.go:29] interesting pod/e2e-ipp-b-ff4113-openshift-default-6b94f7db5f-4p87g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body= Nov 03 01:02:44.268614 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:44.268437 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-ff4113-openshift-default-6b94f7db5f-4p87g" podUID="6ab8d9c7-c645-4569-81a8-c5b1edaa7857" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" Nov 03 01:02:44.672735 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:02:44.672643 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 01:02:45.268860 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:45.268824 4421 patch_prober.go:29] interesting pod/e2e-ipp-b-ff4113-openshift-default-6b94f7db5f-4p87g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body= Nov 03 01:02:45.269051 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:45.268879 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-ff4113-openshift-default-6b94f7db5f-4p87g" podUID="6ab8d9c7-c645-4569-81a8-c5b1edaa7857" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" Nov 03 01:02:45.483465 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:45.483162 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-ff4113-openshift-default-6b94f7db5f-4p87g" Nov 03 01:02:46.268446 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:46.268405 4421 patch_prober.go:29] interesting pod/e2e-ipp-b-ff4113-openshift-default-6b94f7db5f-4p87g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body= Nov 03 01:02:46.268954 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:46.268464 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-ff4113-openshift-default-6b94f7db5f-4p87g" podUID="6ab8d9c7-c645-4569-81a8-c5b1edaa7857" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" Nov 03 01:02:47.274510 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:47.274478 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-ff4113-openshift-default-6b94f7db5f-4p87g" Nov 03 01:02:47.275801 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:47.275779 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-ff4113-openshift-default-6b94f7db5f-4p87g" Nov 03 01:02:49.440332 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:49.439336 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-07f5f6-6448498d8b-k8grc"] Nov 03 01:02:49.440332 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:49.439728 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-07f5f6-6448498d8b-k8grc" podUID="4c232835-2f2e-4ee2-b646-9c56f413919c" containerName="payload-processing" containerID="cri-o://3efca6e4ec1a1f6cd2e9d42eb57fcd9f2b65e0de31c7b0d2d451c3c7c4d5f9f1" gracePeriod=30 Nov 03 01:02:49.447645 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:49.447615 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-07f5f6-5b6bcc7f-bdvrj"] Nov 03 01:02:49.447863 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:49.447827 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-07f5f6-5b6bcc7f-bdvrj" podUID="f8d52857-85e2-4eb2-a223-33177d94ccfb" containerName="payload-pre-processing" containerID="cri-o://3c1456e60721a6a7aaa70dc8debc699858a7eed15c77bf65a1cfaebde5b1f12b" gracePeriod=30 Nov 03 01:02:49.997316 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:49.997259 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-07f5f6-6448498d8b-k8grc" Nov 03 01:02:50.066894 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:50.066857 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c232835-2f2e-4ee2-b646-9c56f413919c-kube-api-access-mjpnw\" (UniqueName: \"kubernetes.io/projected/4c232835-2f2e-4ee2-b646-9c56f413919c-kube-api-access-mjpnw\") pod \"4c232835-2f2e-4ee2-b646-9c56f413919c\" (UID: \"4c232835-2f2e-4ee2-b646-9c56f413919c\") " Nov 03 01:02:50.067050 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:50.066973 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c232835-2f2e-4ee2-b646-9c56f413919c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c232835-2f2e-4ee2-b646-9c56f413919c-plugins-config-volume\") pod \"4c232835-2f2e-4ee2-b646-9c56f413919c\" (UID: \"4c232835-2f2e-4ee2-b646-9c56f413919c\") " Nov 03 01:02:50.067410 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:50.067384 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c232835-2f2e-4ee2-b646-9c56f413919c-plugins-config-volume" pod "4c232835-2f2e-4ee2-b646-9c56f413919c" (UID: "4c232835-2f2e-4ee2-b646-9c56f413919c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:02:50.068719 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:50.068694 4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c232835-2f2e-4ee2-b646-9c56f413919c-plugins-config-volume\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:02:50.071433 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:50.071405 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c232835-2f2e-4ee2-b646-9c56f413919c-kube-api-access-mjpnw" pod "4c232835-2f2e-4ee2-b646-9c56f413919c" (UID: "4c232835-2f2e-4ee2-b646-9c56f413919c"). InnerVolumeSpecName "kube-api-access-mjpnw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:02:50.093965 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:50.093929 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-07f5f6-5b6bcc7f-bdvrj" Nov 03 01:02:50.171069 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:50.169784 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8d52857-85e2-4eb2-a223-33177d94ccfb-kube-api-access-h5csc\" (UniqueName: \"kubernetes.io/projected/f8d52857-85e2-4eb2-a223-33177d94ccfb-kube-api-access-h5csc\") pod \"f8d52857-85e2-4eb2-a223-33177d94ccfb\" (UID: \"f8d52857-85e2-4eb2-a223-33177d94ccfb\") " Nov 03 01:02:50.171069 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:50.169857 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8d52857-85e2-4eb2-a223-33177d94ccfb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8d52857-85e2-4eb2-a223-33177d94ccfb-plugins-config-volume\") pod \"f8d52857-85e2-4eb2-a223-33177d94ccfb\" (UID: \"f8d52857-85e2-4eb2-a223-33177d94ccfb\") " Nov 03 01:02:50.171069 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:50.170781 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mjpnw\" (UniqueName: \"kubernetes.io/projected/4c232835-2f2e-4ee2-b646-9c56f413919c-kube-api-access-mjpnw\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:02:50.171686 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:50.171633 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8d52857-85e2-4eb2-a223-33177d94ccfb-plugins-config-volume" pod "f8d52857-85e2-4eb2-a223-33177d94ccfb" (UID: "f8d52857-85e2-4eb2-a223-33177d94ccfb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:02:50.174581 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:50.174542 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8d52857-85e2-4eb2-a223-33177d94ccfb-kube-api-access-h5csc" pod "f8d52857-85e2-4eb2-a223-33177d94ccfb" (UID: "f8d52857-85e2-4eb2-a223-33177d94ccfb"). InnerVolumeSpecName "kube-api-access-h5csc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:02:50.272715 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:50.272676 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5csc\" (UniqueName: \"kubernetes.io/projected/f8d52857-85e2-4eb2-a223-33177d94ccfb-kube-api-access-h5csc\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:02:50.272715 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:50.272711 4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8d52857-85e2-4eb2-a223-33177d94ccfb-plugins-config-volume\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:02:50.325715 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:50.325621 4421 generic.go:356] "Generic (PLEG): container finished" podID="f8d52857-85e2-4eb2-a223-33177d94ccfb" containerID="3c1456e60721a6a7aaa70dc8debc699858a7eed15c77bf65a1cfaebde5b1f12b" exitCode=0 Nov 03 01:02:50.325715 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:50.325679 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-07f5f6-5b6bcc7f-bdvrj" Nov 03 01:02:50.325967 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:50.325720 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-07f5f6-5b6bcc7f-bdvrj" event={"ID":"f8d52857-85e2-4eb2-a223-33177d94ccfb","Type":"ContainerDied","Data":"3c1456e60721a6a7aaa70dc8debc699858a7eed15c77bf65a1cfaebde5b1f12b"} Nov 03 01:02:50.325967 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:50.325797 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-07f5f6-5b6bcc7f-bdvrj" event={"ID":"f8d52857-85e2-4eb2-a223-33177d94ccfb","Type":"ContainerDied","Data":"069c4d852f926644b561feb48114aa5dbc33d8449f0891035f9ab44c8a4c5c5a"} Nov 03 01:02:50.325967 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:50.325851 4421 scope.go:122] "RemoveContainer" containerID="3c1456e60721a6a7aaa70dc8debc699858a7eed15c77bf65a1cfaebde5b1f12b" Nov 03 01:02:50.327655 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:50.327629 4421 generic.go:356] "Generic (PLEG): container finished" podID="4c232835-2f2e-4ee2-b646-9c56f413919c" containerID="3efca6e4ec1a1f6cd2e9d42eb57fcd9f2b65e0de31c7b0d2d451c3c7c4d5f9f1" exitCode=0 Nov 03 01:02:50.327844 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:50.327824 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-07f5f6-6448498d8b-k8grc" event={"ID":"4c232835-2f2e-4ee2-b646-9c56f413919c","Type":"ContainerDied","Data":"3efca6e4ec1a1f6cd2e9d42eb57fcd9f2b65e0de31c7b0d2d451c3c7c4d5f9f1"} Nov 03 01:02:50.327934 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:50.327867 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-07f5f6-6448498d8b-k8grc" event={"ID":"4c232835-2f2e-4ee2-b646-9c56f413919c","Type":"ContainerDied","Data":"b7fec3215156b8bf855044d1d34440f38f61c0ce57db21f2bf9bcb552d5d42e6"} Nov 03 01:02:50.328003 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:50.327935 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-07f5f6-6448498d8b-k8grc" Nov 03 01:02:50.338480 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:50.338444 4421 scope.go:122] "RemoveContainer" containerID="3c1456e60721a6a7aaa70dc8debc699858a7eed15c77bf65a1cfaebde5b1f12b" Nov 03 01:02:50.338838 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:02:50.338811 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c1456e60721a6a7aaa70dc8debc699858a7eed15c77bf65a1cfaebde5b1f12b\": container with ID starting with 3c1456e60721a6a7aaa70dc8debc699858a7eed15c77bf65a1cfaebde5b1f12b not found: ID does not exist" containerID="3c1456e60721a6a7aaa70dc8debc699858a7eed15c77bf65a1cfaebde5b1f12b" Nov 03 01:02:50.338935 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:50.338849 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c1456e60721a6a7aaa70dc8debc699858a7eed15c77bf65a1cfaebde5b1f12b"} err="failed to get container status \"3c1456e60721a6a7aaa70dc8debc699858a7eed15c77bf65a1cfaebde5b1f12b\": rpc error: code = NotFound desc = could not find container \"3c1456e60721a6a7aaa70dc8debc699858a7eed15c77bf65a1cfaebde5b1f12b\": container with ID starting with 3c1456e60721a6a7aaa70dc8debc699858a7eed15c77bf65a1cfaebde5b1f12b not found: ID does not exist" Nov 03 01:02:50.338935 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:50.338873 4421 scope.go:122] "RemoveContainer" containerID="3efca6e4ec1a1f6cd2e9d42eb57fcd9f2b65e0de31c7b0d2d451c3c7c4d5f9f1" Nov 03 01:02:50.348757 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:50.348738 4421 scope.go:122] "RemoveContainer" containerID="3efca6e4ec1a1f6cd2e9d42eb57fcd9f2b65e0de31c7b0d2d451c3c7c4d5f9f1" Nov 03 01:02:50.349085 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:02:50.349064 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3efca6e4ec1a1f6cd2e9d42eb57fcd9f2b65e0de31c7b0d2d451c3c7c4d5f9f1\": container with ID starting with 3efca6e4ec1a1f6cd2e9d42eb57fcd9f2b65e0de31c7b0d2d451c3c7c4d5f9f1 not found: ID does not exist" containerID="3efca6e4ec1a1f6cd2e9d42eb57fcd9f2b65e0de31c7b0d2d451c3c7c4d5f9f1" Nov 03 01:02:50.349134 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:50.349092 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3efca6e4ec1a1f6cd2e9d42eb57fcd9f2b65e0de31c7b0d2d451c3c7c4d5f9f1"} err="failed to get container status \"3efca6e4ec1a1f6cd2e9d42eb57fcd9f2b65e0de31c7b0d2d451c3c7c4d5f9f1\": rpc error: code = NotFound desc = could not find container \"3efca6e4ec1a1f6cd2e9d42eb57fcd9f2b65e0de31c7b0d2d451c3c7c4d5f9f1\": container with ID starting with 3efca6e4ec1a1f6cd2e9d42eb57fcd9f2b65e0de31c7b0d2d451c3c7c4d5f9f1 not found: ID does not exist" Nov 03 01:02:50.356903 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:50.356880 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-07f5f6-5b6bcc7f-bdvrj"] Nov 03 01:02:50.362049 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:50.362025 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-07f5f6-5b6bcc7f-bdvrj"] Nov 03 01:02:50.373817 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:50.373795 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-07f5f6-6448498d8b-k8grc"] Nov 03 01:02:50.379348 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:50.379323 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-07f5f6-6448498d8b-k8grc"] Nov 03 01:02:51.228615 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:51.228523 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-07f5f6-openshift-default-98746c89f-mr8c7"] Nov 03 01:02:51.229076 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:51.228798 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-07f5f6-openshift-default-98746c89f-mr8c7" podUID="648ce4f9-456f-48ff-a857-6d9bc7f61dc5" containerName="istio-proxy" containerID="cri-o://e8d36fcabdc7131a5fe2cd29b7f0be64d86820085a902770d892bccf5421627e" gracePeriod=30 Nov 03 01:02:51.980215 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:51.980180 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c232835-2f2e-4ee2-b646-9c56f413919c" path="/var/lib/kubelet/pods/4c232835-2f2e-4ee2-b646-9c56f413919c/volumes" Nov 03 01:02:51.980674 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:51.980659 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8d52857-85e2-4eb2-a223-33177d94ccfb" path="/var/lib/kubelet/pods/f8d52857-85e2-4eb2-a223-33177d94ccfb/volumes" Nov 03 01:02:53.276640 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:53.276606 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-ff4113-679c7cdbdb-cgkkl" Nov 03 01:02:55.045364 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:55.045328 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9918c79-jtszs"] Nov 03 01:02:55.101299 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:55.082362 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9918c79-jtszs"] Nov 03 01:02:55.981546 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:55.980554 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74cb1387-3dbb-4a02-8102-c0d86c5a9738" path="/var/lib/kubelet/pods/74cb1387-3dbb-4a02-8102-c0d86c5a9738/volumes" Nov 03 01:02:56.578853 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:56.578817 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-07f5f6-openshift-default-98746c89f-mr8c7" Nov 03 01:02:56.598625 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:56.598591 4421 generic.go:356] "Generic (PLEG): container finished" podID="648ce4f9-456f-48ff-a857-6d9bc7f61dc5" containerID="e8d36fcabdc7131a5fe2cd29b7f0be64d86820085a902770d892bccf5421627e" exitCode=0 Nov 03 01:02:56.598812 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:56.598644 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-07f5f6-openshift-default-98746c89f-mr8c7" Nov 03 01:02:56.598812 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:56.598681 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-07f5f6-openshift-default-98746c89f-mr8c7" event={"ID":"648ce4f9-456f-48ff-a857-6d9bc7f61dc5","Type":"ContainerDied","Data":"e8d36fcabdc7131a5fe2cd29b7f0be64d86820085a902770d892bccf5421627e"} Nov 03 01:02:56.598812 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:56.598752 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-07f5f6-openshift-default-98746c89f-mr8c7" event={"ID":"648ce4f9-456f-48ff-a857-6d9bc7f61dc5","Type":"ContainerDied","Data":"befd15367093e14b6f83c51a9fd3f73503d178d8dbffa64d662a654767714d3f"} Nov 03 01:02:56.598812 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:56.598781 4421 scope.go:122] "RemoveContainer" containerID="e8d36fcabdc7131a5fe2cd29b7f0be64d86820085a902770d892bccf5421627e" Nov 03 01:02:56.609444 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:56.609411 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/648ce4f9-456f-48ff-a857-6d9bc7f61dc5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/648ce4f9-456f-48ff-a857-6d9bc7f61dc5-workload-socket\") pod \"648ce4f9-456f-48ff-a857-6d9bc7f61dc5\" (UID: \"648ce4f9-456f-48ff-a857-6d9bc7f61dc5\") " Nov 03 01:02:56.609726 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:56.609537 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/648ce4f9-456f-48ff-a857-6d9bc7f61dc5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/648ce4f9-456f-48ff-a857-6d9bc7f61dc5-istio-podinfo\") pod \"648ce4f9-456f-48ff-a857-6d9bc7f61dc5\" (UID: \"648ce4f9-456f-48ff-a857-6d9bc7f61dc5\") " Nov 03 01:02:56.609726 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:56.609564 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/648ce4f9-456f-48ff-a857-6d9bc7f61dc5-kube-api-access-qqq2m\" (UniqueName: \"kubernetes.io/projected/648ce4f9-456f-48ff-a857-6d9bc7f61dc5-kube-api-access-qqq2m\") pod \"648ce4f9-456f-48ff-a857-6d9bc7f61dc5\" (UID: \"648ce4f9-456f-48ff-a857-6d9bc7f61dc5\") " Nov 03 01:02:56.609726 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:56.609591 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/648ce4f9-456f-48ff-a857-6d9bc7f61dc5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/648ce4f9-456f-48ff-a857-6d9bc7f61dc5-istio-envoy\") pod \"648ce4f9-456f-48ff-a857-6d9bc7f61dc5\" (UID: \"648ce4f9-456f-48ff-a857-6d9bc7f61dc5\") " Nov 03 01:02:56.609726 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:56.609651 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/648ce4f9-456f-48ff-a857-6d9bc7f61dc5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/648ce4f9-456f-48ff-a857-6d9bc7f61dc5-istio-data\") pod \"648ce4f9-456f-48ff-a857-6d9bc7f61dc5\" (UID: \"648ce4f9-456f-48ff-a857-6d9bc7f61dc5\") " Nov 03 01:02:56.610019 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:56.609827 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/648ce4f9-456f-48ff-a857-6d9bc7f61dc5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/648ce4f9-456f-48ff-a857-6d9bc7f61dc5-istiod-ca-cert\") pod \"648ce4f9-456f-48ff-a857-6d9bc7f61dc5\" (UID: \"648ce4f9-456f-48ff-a857-6d9bc7f61dc5\") " Nov 03 01:02:56.610019 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:56.609864 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/648ce4f9-456f-48ff-a857-6d9bc7f61dc5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/648ce4f9-456f-48ff-a857-6d9bc7f61dc5-credential-socket\") pod \"648ce4f9-456f-48ff-a857-6d9bc7f61dc5\" (UID: \"648ce4f9-456f-48ff-a857-6d9bc7f61dc5\") " Nov 03 01:02:56.610111 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:56.610019 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/648ce4f9-456f-48ff-a857-6d9bc7f61dc5-istio-token\" (UniqueName: \"kubernetes.io/projected/648ce4f9-456f-48ff-a857-6d9bc7f61dc5-istio-token\") pod \"648ce4f9-456f-48ff-a857-6d9bc7f61dc5\" (UID: \"648ce4f9-456f-48ff-a857-6d9bc7f61dc5\") " Nov 03 01:02:56.610111 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:56.610062 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/648ce4f9-456f-48ff-a857-6d9bc7f61dc5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/648ce4f9-456f-48ff-a857-6d9bc7f61dc5-workload-certs\") pod \"648ce4f9-456f-48ff-a857-6d9bc7f61dc5\" (UID: \"648ce4f9-456f-48ff-a857-6d9bc7f61dc5\") " Nov 03 01:02:56.611081 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:56.611040 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/648ce4f9-456f-48ff-a857-6d9bc7f61dc5-istiod-ca-cert" pod "648ce4f9-456f-48ff-a857-6d9bc7f61dc5" (UID: "648ce4f9-456f-48ff-a857-6d9bc7f61dc5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:02:56.611493 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:56.611472 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/648ce4f9-456f-48ff-a857-6d9bc7f61dc5-istio-data" pod "648ce4f9-456f-48ff-a857-6d9bc7f61dc5" (UID: "648ce4f9-456f-48ff-a857-6d9bc7f61dc5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:02:56.611665 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:56.611640 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/648ce4f9-456f-48ff-a857-6d9bc7f61dc5-credential-socket" pod "648ce4f9-456f-48ff-a857-6d9bc7f61dc5" (UID: "648ce4f9-456f-48ff-a857-6d9bc7f61dc5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:02:56.611937 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:56.611914 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/648ce4f9-456f-48ff-a857-6d9bc7f61dc5-workload-socket" pod "648ce4f9-456f-48ff-a857-6d9bc7f61dc5" (UID: "648ce4f9-456f-48ff-a857-6d9bc7f61dc5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:02:56.613853 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:56.613811 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/648ce4f9-456f-48ff-a857-6d9bc7f61dc5-workload-certs" pod "648ce4f9-456f-48ff-a857-6d9bc7f61dc5" (UID: "648ce4f9-456f-48ff-a857-6d9bc7f61dc5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:02:56.614906 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:56.614863 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/648ce4f9-456f-48ff-a857-6d9bc7f61dc5-istio-podinfo" pod "648ce4f9-456f-48ff-a857-6d9bc7f61dc5" (UID: "648ce4f9-456f-48ff-a857-6d9bc7f61dc5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:02:56.615006 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:56.614927 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/648ce4f9-456f-48ff-a857-6d9bc7f61dc5-kube-api-access-qqq2m" pod "648ce4f9-456f-48ff-a857-6d9bc7f61dc5" (UID: "648ce4f9-456f-48ff-a857-6d9bc7f61dc5"). InnerVolumeSpecName "kube-api-access-qqq2m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:02:56.615147 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:56.615123 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/648ce4f9-456f-48ff-a857-6d9bc7f61dc5-istio-envoy" pod "648ce4f9-456f-48ff-a857-6d9bc7f61dc5" (UID: "648ce4f9-456f-48ff-a857-6d9bc7f61dc5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:02:56.616985 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:56.616956 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/648ce4f9-456f-48ff-a857-6d9bc7f61dc5-istio-token" pod "648ce4f9-456f-48ff-a857-6d9bc7f61dc5" (UID: "648ce4f9-456f-48ff-a857-6d9bc7f61dc5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:02:56.622463 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:56.622429 4421 scope.go:122] "RemoveContainer" containerID="e8d36fcabdc7131a5fe2cd29b7f0be64d86820085a902770d892bccf5421627e" Nov 03 01:02:56.622912 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:02:56.622885 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8d36fcabdc7131a5fe2cd29b7f0be64d86820085a902770d892bccf5421627e\": container with ID starting with e8d36fcabdc7131a5fe2cd29b7f0be64d86820085a902770d892bccf5421627e not found: ID does not exist" containerID="e8d36fcabdc7131a5fe2cd29b7f0be64d86820085a902770d892bccf5421627e" Nov 03 01:02:56.623062 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:56.622924 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8d36fcabdc7131a5fe2cd29b7f0be64d86820085a902770d892bccf5421627e"} err="failed to get container status \"e8d36fcabdc7131a5fe2cd29b7f0be64d86820085a902770d892bccf5421627e\": rpc error: code = NotFound desc = could not find container \"e8d36fcabdc7131a5fe2cd29b7f0be64d86820085a902770d892bccf5421627e\": container with ID starting with e8d36fcabdc7131a5fe2cd29b7f0be64d86820085a902770d892bccf5421627e not found: ID does not exist" Nov 03 01:02:56.712669 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:56.712624 4421 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/648ce4f9-456f-48ff-a857-6d9bc7f61dc5-istiod-ca-cert\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:02:56.712669 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:56.712658 4421 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/648ce4f9-456f-48ff-a857-6d9bc7f61dc5-credential-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:02:56.712669 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:56.712675 4421 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/648ce4f9-456f-48ff-a857-6d9bc7f61dc5-istio-token\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:02:56.712915 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:56.712687 4421 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/648ce4f9-456f-48ff-a857-6d9bc7f61dc5-workload-certs\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:02:56.712915 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:56.712699 4421 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/648ce4f9-456f-48ff-a857-6d9bc7f61dc5-workload-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:02:56.712915 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:56.712710 4421 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/648ce4f9-456f-48ff-a857-6d9bc7f61dc5-istio-podinfo\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:02:56.712915 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:56.712722 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqq2m\" (UniqueName: \"kubernetes.io/projected/648ce4f9-456f-48ff-a857-6d9bc7f61dc5-kube-api-access-qqq2m\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:02:56.712915 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:56.712735 4421 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/648ce4f9-456f-48ff-a857-6d9bc7f61dc5-istio-envoy\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:02:56.712915 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:56.712747 4421 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/648ce4f9-456f-48ff-a857-6d9bc7f61dc5-istio-data\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:02:56.924469 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:56.924436 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-07f5f6-openshift-default-98746c89f-mr8c7"] Nov 03 01:02:56.928719 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:56.928687 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-07f5f6-openshift-default-98746c89f-mr8c7"] Nov 03 01:02:57.980440 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:02:57.980404 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="648ce4f9-456f-48ff-a857-6d9bc7f61dc5" path="/var/lib/kubelet/pods/648ce4f9-456f-48ff-a857-6d9bc7f61dc5/volumes" Nov 03 01:03:00.012336 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:00.010350 4421 scope.go:122] "RemoveContainer" containerID="560b1cdc0b3ba6d352f3d7e95f5a67721eb23e89536263fb28b4b8ab5e58ebcc" Nov 03 01:03:04.837208 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:04.837173 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-db5422-8464b57b74-pgngf"] Nov 03 01:03:04.837762 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:04.837513 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-db5422-8464b57b74-pgngf" podUID="82a61b7a-409c-4939-8715-1114dab89b24" containerName="payload-processing" containerID="cri-o://fdba2e2f24937aa6033c40b03dbe4b800edf82fea08052bc7b1c6aca540e93df" gracePeriod=30 Nov 03 01:03:04.866667 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:04.866630 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-db5422-7f6b57dbd7-b5nvk"] Nov 03 01:03:04.867195 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:04.867161 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-db5422-7f6b57dbd7-b5nvk" podUID="7a3242e9-467f-4c9f-b145-0ae70423d5cb" containerName="payload-pre-processing" containerID="cri-o://d69307c467ec7d304c28ce8fcae8dd0dcdea9e7fadfb4ffbaa49079ffe68d44f" gracePeriod=30 Nov 03 01:03:05.347545 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:05.347514 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-db5422-8464b57b74-pgngf" Nov 03 01:03:05.411254 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:05.407631 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82a61b7a-409c-4939-8715-1114dab89b24-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82a61b7a-409c-4939-8715-1114dab89b24-plugins-config-volume\") pod \"82a61b7a-409c-4939-8715-1114dab89b24\" (UID: \"82a61b7a-409c-4939-8715-1114dab89b24\") " Nov 03 01:03:05.411254 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:05.407867 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82a61b7a-409c-4939-8715-1114dab89b24-kube-api-access-z9cc2\" (UniqueName: \"kubernetes.io/projected/82a61b7a-409c-4939-8715-1114dab89b24-kube-api-access-z9cc2\") pod \"82a61b7a-409c-4939-8715-1114dab89b24\" (UID: \"82a61b7a-409c-4939-8715-1114dab89b24\") " Nov 03 01:03:05.411254 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:05.409132 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82a61b7a-409c-4939-8715-1114dab89b24-plugins-config-volume" pod "82a61b7a-409c-4939-8715-1114dab89b24" (UID: "82a61b7a-409c-4939-8715-1114dab89b24"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:03:05.412108 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:05.412074 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82a61b7a-409c-4939-8715-1114dab89b24-kube-api-access-z9cc2" pod "82a61b7a-409c-4939-8715-1114dab89b24" (UID: "82a61b7a-409c-4939-8715-1114dab89b24"). InnerVolumeSpecName "kube-api-access-z9cc2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:03:05.440745 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:05.440712 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-db5422-7f6b57dbd7-b5nvk" Nov 03 01:03:05.509203 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:05.509162 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a3242e9-467f-4c9f-b145-0ae70423d5cb-kube-api-access-6m9ht\" (UniqueName: \"kubernetes.io/projected/7a3242e9-467f-4c9f-b145-0ae70423d5cb-kube-api-access-6m9ht\") pod \"7a3242e9-467f-4c9f-b145-0ae70423d5cb\" (UID: \"7a3242e9-467f-4c9f-b145-0ae70423d5cb\") " Nov 03 01:03:05.509423 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:05.509256 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a3242e9-467f-4c9f-b145-0ae70423d5cb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a3242e9-467f-4c9f-b145-0ae70423d5cb-plugins-config-volume\") pod \"7a3242e9-467f-4c9f-b145-0ae70423d5cb\" (UID: \"7a3242e9-467f-4c9f-b145-0ae70423d5cb\") " Nov 03 01:03:05.509689 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:05.509655 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a3242e9-467f-4c9f-b145-0ae70423d5cb-plugins-config-volume" pod "7a3242e9-467f-4c9f-b145-0ae70423d5cb" (UID: "7a3242e9-467f-4c9f-b145-0ae70423d5cb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:03:05.510096 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:05.510066 4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a3242e9-467f-4c9f-b145-0ae70423d5cb-plugins-config-volume\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:03:05.510186 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:05.510101 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9cc2\" (UniqueName: \"kubernetes.io/projected/82a61b7a-409c-4939-8715-1114dab89b24-kube-api-access-z9cc2\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:03:05.510186 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:05.510116 4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82a61b7a-409c-4939-8715-1114dab89b24-plugins-config-volume\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:03:05.511613 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:05.511580 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a3242e9-467f-4c9f-b145-0ae70423d5cb-kube-api-access-6m9ht" pod "7a3242e9-467f-4c9f-b145-0ae70423d5cb" (UID: "7a3242e9-467f-4c9f-b145-0ae70423d5cb"). InnerVolumeSpecName "kube-api-access-6m9ht". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:03:05.620300 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:05.615405 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6m9ht\" (UniqueName: \"kubernetes.io/projected/7a3242e9-467f-4c9f-b145-0ae70423d5cb-kube-api-access-6m9ht\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:03:05.705366 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:05.696950 4421 generic.go:356] "Generic (PLEG): container finished" podID="82a61b7a-409c-4939-8715-1114dab89b24" containerID="fdba2e2f24937aa6033c40b03dbe4b800edf82fea08052bc7b1c6aca540e93df" exitCode=0 Nov 03 01:03:05.705366 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:05.697065 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-db5422-8464b57b74-pgngf" event={"ID":"82a61b7a-409c-4939-8715-1114dab89b24","Type":"ContainerDied","Data":"fdba2e2f24937aa6033c40b03dbe4b800edf82fea08052bc7b1c6aca540e93df"} Nov 03 01:03:05.705366 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:05.697115 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-db5422-8464b57b74-pgngf" event={"ID":"82a61b7a-409c-4939-8715-1114dab89b24","Type":"ContainerDied","Data":"c3d689f812180bd501a0a874f52833d6e81290c39cc5bce23e079920c616b227"} Nov 03 01:03:05.705366 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:05.697142 4421 scope.go:122] "RemoveContainer" containerID="fdba2e2f24937aa6033c40b03dbe4b800edf82fea08052bc7b1c6aca540e93df" Nov 03 01:03:05.705366 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:05.697315 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-db5422-8464b57b74-pgngf" Nov 03 01:03:05.705366 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:05.704432 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-db5422-7f6b57dbd7-b5nvk" Nov 03 01:03:05.705366 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:05.704812 4421 generic.go:356] "Generic (PLEG): container finished" podID="7a3242e9-467f-4c9f-b145-0ae70423d5cb" containerID="d69307c467ec7d304c28ce8fcae8dd0dcdea9e7fadfb4ffbaa49079ffe68d44f" exitCode=0 Nov 03 01:03:05.705366 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:05.704870 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-db5422-7f6b57dbd7-b5nvk" event={"ID":"7a3242e9-467f-4c9f-b145-0ae70423d5cb","Type":"ContainerDied","Data":"d69307c467ec7d304c28ce8fcae8dd0dcdea9e7fadfb4ffbaa49079ffe68d44f"} Nov 03 01:03:05.705366 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:05.704919 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-db5422-7f6b57dbd7-b5nvk" event={"ID":"7a3242e9-467f-4c9f-b145-0ae70423d5cb","Type":"ContainerDied","Data":"83e0c006fcc1888418b2c60b9691c8b5d3b6e82a1226c97c58cb46cf1dc4e166"} Nov 03 01:03:05.735166 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:05.733066 4421 scope.go:122] "RemoveContainer" containerID="fdba2e2f24937aa6033c40b03dbe4b800edf82fea08052bc7b1c6aca540e93df" Nov 03 01:03:05.735166 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:03:05.734908 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fdba2e2f24937aa6033c40b03dbe4b800edf82fea08052bc7b1c6aca540e93df\": container with ID starting with fdba2e2f24937aa6033c40b03dbe4b800edf82fea08052bc7b1c6aca540e93df not found: ID does not exist" containerID="fdba2e2f24937aa6033c40b03dbe4b800edf82fea08052bc7b1c6aca540e93df" Nov 03 01:03:05.735166 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:05.734951 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdba2e2f24937aa6033c40b03dbe4b800edf82fea08052bc7b1c6aca540e93df"} err="failed to get container status \"fdba2e2f24937aa6033c40b03dbe4b800edf82fea08052bc7b1c6aca540e93df\": rpc error: code = NotFound desc = could not find container \"fdba2e2f24937aa6033c40b03dbe4b800edf82fea08052bc7b1c6aca540e93df\": container with ID starting with fdba2e2f24937aa6033c40b03dbe4b800edf82fea08052bc7b1c6aca540e93df not found: ID does not exist" Nov 03 01:03:05.735166 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:05.734979 4421 scope.go:122] "RemoveContainer" containerID="d69307c467ec7d304c28ce8fcae8dd0dcdea9e7fadfb4ffbaa49079ffe68d44f" Nov 03 01:03:05.763846 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:05.763815 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-db5422-8464b57b74-pgngf"] Nov 03 01:03:05.768874 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:05.768850 4421 scope.go:122] "RemoveContainer" containerID="d69307c467ec7d304c28ce8fcae8dd0dcdea9e7fadfb4ffbaa49079ffe68d44f" Nov 03 01:03:05.769348 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:03:05.769295 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d69307c467ec7d304c28ce8fcae8dd0dcdea9e7fadfb4ffbaa49079ffe68d44f\": container with ID starting with d69307c467ec7d304c28ce8fcae8dd0dcdea9e7fadfb4ffbaa49079ffe68d44f not found: ID does not exist" containerID="d69307c467ec7d304c28ce8fcae8dd0dcdea9e7fadfb4ffbaa49079ffe68d44f" Nov 03 01:03:05.769348 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:05.769330 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d69307c467ec7d304c28ce8fcae8dd0dcdea9e7fadfb4ffbaa49079ffe68d44f"} err="failed to get container status \"d69307c467ec7d304c28ce8fcae8dd0dcdea9e7fadfb4ffbaa49079ffe68d44f\": rpc error: code = NotFound desc = could not find container \"d69307c467ec7d304c28ce8fcae8dd0dcdea9e7fadfb4ffbaa49079ffe68d44f\": container with ID starting with d69307c467ec7d304c28ce8fcae8dd0dcdea9e7fadfb4ffbaa49079ffe68d44f not found: ID does not exist" Nov 03 01:03:05.770347 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:05.770324 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-db5422-8464b57b74-pgngf"] Nov 03 01:03:05.778385 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:05.778113 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-db5422-7f6b57dbd7-b5nvk"] Nov 03 01:03:05.782704 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:05.782679 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-db5422-7f6b57dbd7-b5nvk"] Nov 03 01:03:05.980005 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:05.979929 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a3242e9-467f-4c9f-b145-0ae70423d5cb" path="/var/lib/kubelet/pods/7a3242e9-467f-4c9f-b145-0ae70423d5cb/volumes" Nov 03 01:03:05.980448 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:05.980434 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82a61b7a-409c-4939-8715-1114dab89b24" path="/var/lib/kubelet/pods/82a61b7a-409c-4939-8715-1114dab89b24/volumes" Nov 03 01:03:06.789368 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:06.789319 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-f14bda-5bf9699c87-z52pg"] Nov 03 01:03:06.790014 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:06.789997 4421 cpu_manager.go:414] "containerMap: removing container" podUID="7a3242e9-467f-4c9f-b145-0ae70423d5cb" containerName="payload-pre-processing" Nov 03 01:03:06.790068 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:06.790016 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a3242e9-467f-4c9f-b145-0ae70423d5cb" containerName="payload-pre-processing" Nov 03 01:03:06.790068 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:06.790030 4421 cpu_manager.go:414] "containerMap: removing container" podUID="e13eb472-c996-46dd-b804-774416975943" containerName="revoke-keys" Nov 03 01:03:06.790068 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:06.790036 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e13eb472-c996-46dd-b804-774416975943" containerName="revoke-keys" Nov 03 01:03:06.790068 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:06.790047 4421 cpu_manager.go:414] "containerMap: removing container" podUID="f8d52857-85e2-4eb2-a223-33177d94ccfb" containerName="payload-pre-processing" Nov 03 01:03:06.790068 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:06.790052 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8d52857-85e2-4eb2-a223-33177d94ccfb" containerName="payload-pre-processing" Nov 03 01:03:06.790068 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:06.790061 4421 cpu_manager.go:414] "containerMap: removing container" podUID="4c232835-2f2e-4ee2-b646-9c56f413919c" containerName="payload-processing" Nov 03 01:03:06.790068 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:06.790065 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c232835-2f2e-4ee2-b646-9c56f413919c" containerName="payload-processing" Nov 03 01:03:06.790324 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:06.790079 4421 cpu_manager.go:414] "containerMap: removing container" podUID="648ce4f9-456f-48ff-a857-6d9bc7f61dc5" containerName="istio-proxy" Nov 03 01:03:06.790324 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:06.790083 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="648ce4f9-456f-48ff-a857-6d9bc7f61dc5" containerName="istio-proxy" Nov 03 01:03:06.790324 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:06.790094 4421 cpu_manager.go:414] "containerMap: removing container" podUID="82a61b7a-409c-4939-8715-1114dab89b24" containerName="payload-processing" Nov 03 01:03:06.790324 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:06.790098 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82a61b7a-409c-4939-8715-1114dab89b24" containerName="payload-processing" Nov 03 01:03:06.790324 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:06.790209 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c232835-2f2e-4ee2-b646-9c56f413919c" containerName="payload-processing" Nov 03 01:03:06.790324 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:06.790223 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="648ce4f9-456f-48ff-a857-6d9bc7f61dc5" containerName="istio-proxy" Nov 03 01:03:06.790324 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:06.790231 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="82a61b7a-409c-4939-8715-1114dab89b24" containerName="payload-processing" Nov 03 01:03:06.790324 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:06.790242 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a3242e9-467f-4c9f-b145-0ae70423d5cb" containerName="payload-pre-processing" Nov 03 01:03:06.790324 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:06.790249 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="e13eb472-c996-46dd-b804-774416975943" containerName="revoke-keys" Nov 03 01:03:06.790324 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:06.790256 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8d52857-85e2-4eb2-a223-33177d94ccfb" containerName="payload-pre-processing" Nov 03 01:03:06.794686 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:06.794663 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f14bda-5bf9699c87-z52pg" Nov 03 01:03:06.800180 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:06.800157 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-f14bda\"" Nov 03 01:03:06.801378 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:06.800861 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-f14bda-dockercfg-mkrgn\"" Nov 03 01:03:06.802561 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:06.802538 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-f14bda-5bf9699c87-z52pg"] Nov 03 01:03:06.827595 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:06.827562 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-f14bda-7f498d85cc-v2z67"] Nov 03 01:03:06.831321 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:06.831301 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-f14bda-7f498d85cc-v2z67" Nov 03 01:03:06.834398 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:06.834375 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4777\" (UniqueName: \"kubernetes.io/projected/5175e636-02af-4ba5-a711-226c87746bca-kube-api-access-k4777\") pod \"payload-pre-processing-e2e-praxis-b-f14bda-5bf9699c87-z52pg\" (UID: \"5175e636-02af-4ba5-a711-226c87746bca\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f14bda-5bf9699c87-z52pg" Nov 03 01:03:06.834671 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:06.834642 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5175e636-02af-4ba5-a711-226c87746bca-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-f14bda-5bf9699c87-z52pg\" (UID: \"5175e636-02af-4ba5-a711-226c87746bca\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f14bda-5bf9699c87-z52pg" Nov 03 01:03:06.856442 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:06.849967 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-f14bda-7f498d85cc-v2z67"] Nov 03 01:03:06.951513 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:06.951471 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5175e636-02af-4ba5-a711-226c87746bca-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-f14bda-5bf9699c87-z52pg\" (UID: \"5175e636-02af-4ba5-a711-226c87746bca\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f14bda-5bf9699c87-z52pg" Nov 03 01:03:06.955978 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:06.955943 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b384fb21-bb39-46b8-8b5a-40a74ce711bf-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-f14bda-7f498d85cc-v2z67\" (UID: \"b384fb21-bb39-46b8-8b5a-40a74ce711bf\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-f14bda-7f498d85cc-v2z67" Nov 03 01:03:06.956267 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:06.956247 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbjqv\" (UniqueName: \"kubernetes.io/projected/b384fb21-bb39-46b8-8b5a-40a74ce711bf-kube-api-access-vbjqv\") pod \"payload-processing-e2e-praxis-b-f14bda-7f498d85cc-v2z67\" (UID: \"b384fb21-bb39-46b8-8b5a-40a74ce711bf\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-f14bda-7f498d85cc-v2z67" Nov 03 01:03:06.956557 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:06.956534 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4777\" (UniqueName: \"kubernetes.io/projected/5175e636-02af-4ba5-a711-226c87746bca-kube-api-access-k4777\") pod \"payload-pre-processing-e2e-praxis-b-f14bda-5bf9699c87-z52pg\" (UID: \"5175e636-02af-4ba5-a711-226c87746bca\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f14bda-5bf9699c87-z52pg" Nov 03 01:03:06.958969 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:06.953781 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5175e636-02af-4ba5-a711-226c87746bca-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-f14bda-5bf9699c87-z52pg\" (UID: \"5175e636-02af-4ba5-a711-226c87746bca\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f14bda-5bf9699c87-z52pg" Nov 03 01:03:06.983296 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:06.978423 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4777\" (UniqueName: \"kubernetes.io/projected/5175e636-02af-4ba5-a711-226c87746bca-kube-api-access-k4777\") pod \"payload-pre-processing-e2e-praxis-b-f14bda-5bf9699c87-z52pg\" (UID: \"5175e636-02af-4ba5-a711-226c87746bca\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f14bda-5bf9699c87-z52pg" Nov 03 01:03:07.062338 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:07.062228 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b384fb21-bb39-46b8-8b5a-40a74ce711bf-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-f14bda-7f498d85cc-v2z67\" (UID: \"b384fb21-bb39-46b8-8b5a-40a74ce711bf\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-f14bda-7f498d85cc-v2z67" Nov 03 01:03:07.062338 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:07.062325 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbjqv\" (UniqueName: \"kubernetes.io/projected/b384fb21-bb39-46b8-8b5a-40a74ce711bf-kube-api-access-vbjqv\") pod \"payload-processing-e2e-praxis-b-f14bda-7f498d85cc-v2z67\" (UID: \"b384fb21-bb39-46b8-8b5a-40a74ce711bf\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-f14bda-7f498d85cc-v2z67" Nov 03 01:03:07.063228 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:07.063196 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b384fb21-bb39-46b8-8b5a-40a74ce711bf-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-f14bda-7f498d85cc-v2z67\" (UID: \"b384fb21-bb39-46b8-8b5a-40a74ce711bf\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-f14bda-7f498d85cc-v2z67" Nov 03 01:03:07.071383 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:07.070590 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbjqv\" (UniqueName: \"kubernetes.io/projected/b384fb21-bb39-46b8-8b5a-40a74ce711bf-kube-api-access-vbjqv\") pod \"payload-processing-e2e-praxis-b-f14bda-7f498d85cc-v2z67\" (UID: \"b384fb21-bb39-46b8-8b5a-40a74ce711bf\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-f14bda-7f498d85cc-v2z67" Nov 03 01:03:07.117983 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:07.117949 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f14bda-5bf9699c87-z52pg" Nov 03 01:03:07.152719 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:07.152679 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-f14bda-7f498d85cc-v2z67" Nov 03 01:03:07.412975 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:07.412940 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-f14bda-5bf9699c87-z52pg"] Nov 03 01:03:07.413836 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:03:07.413809 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5175e636_02af_4ba5_a711_226c87746bca.slice/crio-4bb0977018345131a35273b528d70b9563508e44263cf14db2deed5bbc04a9a1 WatchSource:0}: Error finding container 4bb0977018345131a35273b528d70b9563508e44263cf14db2deed5bbc04a9a1: Status 404 returned error can't find the container with id 4bb0977018345131a35273b528d70b9563508e44263cf14db2deed5bbc04a9a1 Nov 03 01:03:07.456826 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:07.456794 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-f14bda-7f498d85cc-v2z67"] Nov 03 01:03:07.457422 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:03:07.457391 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb384fb21_bb39_46b8_8b5a_40a74ce711bf.slice/crio-5ffe9d2cc6bf1cd797cd70b7d45dff6c032d5d23e935905cc1519903a930c545 WatchSource:0}: Error finding container 5ffe9d2cc6bf1cd797cd70b7d45dff6c032d5d23e935905cc1519903a930c545: Status 404 returned error can't find the container with id 5ffe9d2cc6bf1cd797cd70b7d45dff6c032d5d23e935905cc1519903a930c545 Nov 03 01:03:07.725823 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:07.725740 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-f14bda-7f498d85cc-v2z67" event={"ID":"b384fb21-bb39-46b8-8b5a-40a74ce711bf","Type":"ContainerStarted","Data":"f5422178a896d94db564137c39036a121f46f06a32076523d06477ceb1a3a204"} Nov 03 01:03:07.725823 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:07.725794 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-f14bda-7f498d85cc-v2z67" event={"ID":"b384fb21-bb39-46b8-8b5a-40a74ce711bf","Type":"ContainerStarted","Data":"5ffe9d2cc6bf1cd797cd70b7d45dff6c032d5d23e935905cc1519903a930c545"} Nov 03 01:03:07.726053 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:07.725969 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-f14bda-7f498d85cc-v2z67" Nov 03 01:03:07.727541 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:07.727519 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f14bda-5bf9699c87-z52pg" event={"ID":"5175e636-02af-4ba5-a711-226c87746bca","Type":"ContainerStarted","Data":"4d35d62e738bb3ca9a18086f3b671355bf4277096874d9c2ecd16ddac755621d"} Nov 03 01:03:07.727671 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:07.727554 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f14bda-5bf9699c87-z52pg" event={"ID":"5175e636-02af-4ba5-a711-226c87746bca","Type":"ContainerStarted","Data":"4bb0977018345131a35273b528d70b9563508e44263cf14db2deed5bbc04a9a1"} Nov 03 01:03:07.727671 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:07.727627 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f14bda-5bf9699c87-z52pg" Nov 03 01:03:07.779315 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:07.779228 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-f14bda-7f498d85cc-v2z67" podStartSLOduration=1.779204818 podStartE2EDuration="1.779204818s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:03:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:03:07.766559653 +0000 UTC m=+1329.937562756" watchObservedRunningTime="2026-11-03 01:03:07.779204818 +0000 UTC m=+1329.950207915" Nov 03 01:03:07.800296 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:07.795395 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f14bda-5bf9699c87-z52pg" podStartSLOduration=1.79537338 podStartE2EDuration="1.79537338s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:03:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:03:07.784857386 +0000 UTC m=+1329.955860484" watchObservedRunningTime="2026-11-03 01:03:07.79537338 +0000 UTC m=+1329.966376477" Nov 03 01:03:09.304173 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:09.304139 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56b638a8-8pvz8"] Nov 03 01:03:09.308124 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:09.308101 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56b638a8-8pvz8" Nov 03 01:03:09.311734 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:09.311710 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 01:03:09.311734 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:09.311711 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 01:03:09.313830 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:09.313802 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:03:09.313987 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:09.313838 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6q4xp\"" Nov 03 01:03:09.318031 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:09.317999 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56b638a8-8pvz8"] Nov 03 01:03:09.408415 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:09.408370 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fb996be-f95b-4a6f-9884-5e25ca6e19a3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-56b638a8-8pvz8\" (UID: \"8fb996be-f95b-4a6f-9884-5e25ca6e19a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56b638a8-8pvz8" Nov 03 01:03:09.408628 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:09.408594 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fb996be-f95b-4a6f-9884-5e25ca6e19a3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-56b638a8-8pvz8\" (UID: \"8fb996be-f95b-4a6f-9884-5e25ca6e19a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56b638a8-8pvz8" Nov 03 01:03:09.408726 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:09.408710 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cg7tb\" (UniqueName: \"kubernetes.io/projected/8fb996be-f95b-4a6f-9884-5e25ca6e19a3-kube-api-access-cg7tb\") pod \"maas-api-revoke-sub-56b638a8-8pvz8\" (UID: \"8fb996be-f95b-4a6f-9884-5e25ca6e19a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56b638a8-8pvz8" Nov 03 01:03:09.509730 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:09.509686 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fb996be-f95b-4a6f-9884-5e25ca6e19a3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-56b638a8-8pvz8\" (UID: \"8fb996be-f95b-4a6f-9884-5e25ca6e19a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56b638a8-8pvz8" Nov 03 01:03:09.509968 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:09.509816 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cg7tb\" (UniqueName: \"kubernetes.io/projected/8fb996be-f95b-4a6f-9884-5e25ca6e19a3-kube-api-access-cg7tb\") pod \"maas-api-revoke-sub-56b638a8-8pvz8\" (UID: \"8fb996be-f95b-4a6f-9884-5e25ca6e19a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56b638a8-8pvz8" Nov 03 01:03:09.509968 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:09.509878 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fb996be-f95b-4a6f-9884-5e25ca6e19a3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-56b638a8-8pvz8\" (UID: \"8fb996be-f95b-4a6f-9884-5e25ca6e19a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56b638a8-8pvz8" Nov 03 01:03:09.510623 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:09.510594 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fb996be-f95b-4a6f-9884-5e25ca6e19a3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-56b638a8-8pvz8\" (UID: \"8fb996be-f95b-4a6f-9884-5e25ca6e19a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56b638a8-8pvz8" Nov 03 01:03:09.512125 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:09.512101 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fb996be-f95b-4a6f-9884-5e25ca6e19a3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-56b638a8-8pvz8\" (UID: \"8fb996be-f95b-4a6f-9884-5e25ca6e19a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56b638a8-8pvz8" Nov 03 01:03:09.520622 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:09.520593 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cg7tb\" (UniqueName: \"kubernetes.io/projected/8fb996be-f95b-4a6f-9884-5e25ca6e19a3-kube-api-access-cg7tb\") pod \"maas-api-revoke-sub-56b638a8-8pvz8\" (UID: \"8fb996be-f95b-4a6f-9884-5e25ca6e19a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56b638a8-8pvz8" Nov 03 01:03:09.626431 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:09.626340 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56b638a8-8pvz8" Nov 03 01:03:09.921874 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:09.921844 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56b638a8-8pvz8"] Nov 03 01:03:09.922350 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:03:09.922321 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fb996be_f95b_4a6f_9884_5e25ca6e19a3.slice/crio-05ad26f4f15169664c5b8f94b72fee7f6ba21fc23c4cfe8915c21d91659d0aba WatchSource:0}: Error finding container 05ad26f4f15169664c5b8f94b72fee7f6ba21fc23c4cfe8915c21d91659d0aba: Status 404 returned error can't find the container with id 05ad26f4f15169664c5b8f94b72fee7f6ba21fc23c4cfe8915c21d91659d0aba Nov 03 01:03:10.470745 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:10.470706 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-db5422-openshift-default-696cf4dc99-75p47"] Nov 03 01:03:10.471240 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:10.470962 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-db5422-openshift-default-696cf4dc99-75p47" podUID="33f4acce-9a60-4f9c-811d-81e8abfcabe8" containerName="istio-proxy" containerID="cri-o://d2a15723d7b6701815942f97a5762ea7f6fa989487c86467b5ff61c932816e5e" gracePeriod=30 Nov 03 01:03:10.771709 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:10.771658 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56b638a8-8pvz8" event={"ID":"8fb996be-f95b-4a6f-9884-5e25ca6e19a3","Type":"ContainerStarted","Data":"36b6d60e166b775d8583f375d29aac8b2f853d0f320d40e410178f0e047607fb"} Nov 03 01:03:10.772013 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:10.771736 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56b638a8-8pvz8" event={"ID":"8fb996be-f95b-4a6f-9884-5e25ca6e19a3","Type":"ContainerStarted","Data":"05ad26f4f15169664c5b8f94b72fee7f6ba21fc23c4cfe8915c21d91659d0aba"} Nov 03 01:03:10.792802 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:10.792745 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56b638a8-8pvz8" podStartSLOduration=1.792731817 podStartE2EDuration="1.792731817s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:03:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:03:10.79119927 +0000 UTC m=+1332.962202371" watchObservedRunningTime="2026-11-03 01:03:10.792731817 +0000 UTC m=+1332.963734913" Nov 03 01:03:11.781199 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:11.781166 4421 generic.go:356] "Generic (PLEG): container finished" podID="8fb996be-f95b-4a6f-9884-5e25ca6e19a3" containerID="36b6d60e166b775d8583f375d29aac8b2f853d0f320d40e410178f0e047607fb" exitCode=0 Nov 03 01:03:11.781629 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:11.781248 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56b638a8-8pvz8" event={"ID":"8fb996be-f95b-4a6f-9884-5e25ca6e19a3","Type":"ContainerDied","Data":"36b6d60e166b775d8583f375d29aac8b2f853d0f320d40e410178f0e047607fb"} Nov 03 01:03:12.545050 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:12.544864 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-dc0f3f-7b78d8bfdb-4s767"] Nov 03 01:03:12.545304 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:12.545216 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-dc0f3f-7b78d8bfdb-4s767" podUID="f894ad12-9d4e-4133-860c-b053bbbaf136" containerName="payload-processing" containerID="cri-o://d63dccf3bb816de8905099f315280f1fd57d15720dbcec16e154fb0bb410f98b" gracePeriod=30 Nov 03 01:03:12.554366 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:12.554335 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-dc0f3f-59446658cf-n6msk"] Nov 03 01:03:12.554599 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:12.554561 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dc0f3f-59446658cf-n6msk" podUID="9ffa6158-c563-4d0e-91db-d5b6665bebb3" containerName="payload-pre-processing" containerID="cri-o://98198872aec313e9d9129ebc5355a7602365a4de9803af2abcf7eed8e86247ce" gracePeriod=30 Nov 03 01:03:12.895521 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:12.894934 4421 generic.go:356] "Generic (PLEG): container finished" podID="9ffa6158-c563-4d0e-91db-d5b6665bebb3" containerID="98198872aec313e9d9129ebc5355a7602365a4de9803af2abcf7eed8e86247ce" exitCode=0 Nov 03 01:03:12.895521 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:12.895042 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dc0f3f-59446658cf-n6msk" event={"ID":"9ffa6158-c563-4d0e-91db-d5b6665bebb3","Type":"ContainerDied","Data":"98198872aec313e9d9129ebc5355a7602365a4de9803af2abcf7eed8e86247ce"} Nov 03 01:03:12.897514 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:12.897056 4421 generic.go:356] "Generic (PLEG): container finished" podID="f894ad12-9d4e-4133-860c-b053bbbaf136" containerID="d63dccf3bb816de8905099f315280f1fd57d15720dbcec16e154fb0bb410f98b" exitCode=0 Nov 03 01:03:12.903815 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:12.901499 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-dc0f3f-7b78d8bfdb-4s767" event={"ID":"f894ad12-9d4e-4133-860c-b053bbbaf136","Type":"ContainerDied","Data":"d63dccf3bb816de8905099f315280f1fd57d15720dbcec16e154fb0bb410f98b"} Nov 03 01:03:13.012523 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:13.012369 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-dc0f3f-7b78d8bfdb-4s767" Nov 03 01:03:13.069834 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:13.069801 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f894ad12-9d4e-4133-860c-b053bbbaf136-kube-api-access-xqrrr\" (UniqueName: \"kubernetes.io/projected/f894ad12-9d4e-4133-860c-b053bbbaf136-kube-api-access-xqrrr\") pod \"f894ad12-9d4e-4133-860c-b053bbbaf136\" (UID: \"f894ad12-9d4e-4133-860c-b053bbbaf136\") " Nov 03 01:03:13.070497 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:13.070475 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f894ad12-9d4e-4133-860c-b053bbbaf136-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f894ad12-9d4e-4133-860c-b053bbbaf136-plugins-config-volume\") pod \"f894ad12-9d4e-4133-860c-b053bbbaf136\" (UID: \"f894ad12-9d4e-4133-860c-b053bbbaf136\") " Nov 03 01:03:13.071399 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:13.071368 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f894ad12-9d4e-4133-860c-b053bbbaf136-plugins-config-volume" pod "f894ad12-9d4e-4133-860c-b053bbbaf136" (UID: "f894ad12-9d4e-4133-860c-b053bbbaf136"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:03:13.072269 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:13.072252 4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f894ad12-9d4e-4133-860c-b053bbbaf136-plugins-config-volume\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:03:13.093329 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:13.087909 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f894ad12-9d4e-4133-860c-b053bbbaf136-kube-api-access-xqrrr" pod "f894ad12-9d4e-4133-860c-b053bbbaf136" (UID: "f894ad12-9d4e-4133-860c-b053bbbaf136"). InnerVolumeSpecName "kube-api-access-xqrrr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:03:13.174221 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:13.174117 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xqrrr\" (UniqueName: \"kubernetes.io/projected/f894ad12-9d4e-4133-860c-b053bbbaf136-kube-api-access-xqrrr\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:03:13.179455 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:13.179412 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56b638a8-8pvz8" Nov 03 01:03:13.184299 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:13.184260 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dc0f3f-59446658cf-n6msk" Nov 03 01:03:13.275328 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:13.275264 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fb996be-f95b-4a6f-9884-5e25ca6e19a3-kube-api-access-cg7tb\" (UniqueName: \"kubernetes.io/projected/8fb996be-f95b-4a6f-9884-5e25ca6e19a3-kube-api-access-cg7tb\") pod \"8fb996be-f95b-4a6f-9884-5e25ca6e19a3\" (UID: \"8fb996be-f95b-4a6f-9884-5e25ca6e19a3\") " Nov 03 01:03:13.275588 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:13.275355 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ffa6158-c563-4d0e-91db-d5b6665bebb3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ffa6158-c563-4d0e-91db-d5b6665bebb3-plugins-config-volume\") pod \"9ffa6158-c563-4d0e-91db-d5b6665bebb3\" (UID: \"9ffa6158-c563-4d0e-91db-d5b6665bebb3\") " Nov 03 01:03:13.275588 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:13.275459 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8fb996be-f95b-4a6f-9884-5e25ca6e19a3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fb996be-f95b-4a6f-9884-5e25ca6e19a3-maas-api-xks-ca\") pod \"8fb996be-f95b-4a6f-9884-5e25ca6e19a3\" (UID: \"8fb996be-f95b-4a6f-9884-5e25ca6e19a3\") " Nov 03 01:03:13.275588 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:13.275510 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ffa6158-c563-4d0e-91db-d5b6665bebb3-kube-api-access-sjgsk\" (UniqueName: \"kubernetes.io/projected/9ffa6158-c563-4d0e-91db-d5b6665bebb3-kube-api-access-sjgsk\") pod \"9ffa6158-c563-4d0e-91db-d5b6665bebb3\" (UID: \"9ffa6158-c563-4d0e-91db-d5b6665bebb3\") " Nov 03 01:03:13.275757 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:13.275688 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8fb996be-f95b-4a6f-9884-5e25ca6e19a3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fb996be-f95b-4a6f-9884-5e25ca6e19a3-maas-api-openshift-ca\") pod \"8fb996be-f95b-4a6f-9884-5e25ca6e19a3\" (UID: \"8fb996be-f95b-4a6f-9884-5e25ca6e19a3\") " Nov 03 01:03:13.275815 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:13.275785 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ffa6158-c563-4d0e-91db-d5b6665bebb3-plugins-config-volume" pod "9ffa6158-c563-4d0e-91db-d5b6665bebb3" (UID: "9ffa6158-c563-4d0e-91db-d5b6665bebb3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:03:13.276123 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:13.276093 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fb996be-f95b-4a6f-9884-5e25ca6e19a3-maas-api-openshift-ca" pod "8fb996be-f95b-4a6f-9884-5e25ca6e19a3" (UID: "8fb996be-f95b-4a6f-9884-5e25ca6e19a3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:03:13.276391 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:13.276365 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fb996be-f95b-4a6f-9884-5e25ca6e19a3-maas-api-openshift-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:03:13.276551 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:13.276528 4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ffa6158-c563-4d0e-91db-d5b6665bebb3-plugins-config-volume\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:03:13.277634 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:13.277606 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ffa6158-c563-4d0e-91db-d5b6665bebb3-kube-api-access-sjgsk" pod "9ffa6158-c563-4d0e-91db-d5b6665bebb3" (UID: "9ffa6158-c563-4d0e-91db-d5b6665bebb3"). InnerVolumeSpecName "kube-api-access-sjgsk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:03:13.277715 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:13.277644 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8fb996be-f95b-4a6f-9884-5e25ca6e19a3-maas-api-xks-ca" pod "8fb996be-f95b-4a6f-9884-5e25ca6e19a3" (UID: "8fb996be-f95b-4a6f-9884-5e25ca6e19a3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:03:13.277715 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:13.277651 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fb996be-f95b-4a6f-9884-5e25ca6e19a3-kube-api-access-cg7tb" pod "8fb996be-f95b-4a6f-9884-5e25ca6e19a3" (UID: "8fb996be-f95b-4a6f-9884-5e25ca6e19a3"). InnerVolumeSpecName "kube-api-access-cg7tb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:03:13.377846 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:13.377805 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fb996be-f95b-4a6f-9884-5e25ca6e19a3-maas-api-xks-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:03:13.377846 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:13.377836 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjgsk\" (UniqueName: \"kubernetes.io/projected/9ffa6158-c563-4d0e-91db-d5b6665bebb3-kube-api-access-sjgsk\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:03:13.377846 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:13.377848 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cg7tb\" (UniqueName: \"kubernetes.io/projected/8fb996be-f95b-4a6f-9884-5e25ca6e19a3-kube-api-access-cg7tb\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:03:13.913746 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:13.913653 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56b638a8-8pvz8" Nov 03 01:03:13.913746 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:13.913669 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56b638a8-8pvz8" event={"ID":"8fb996be-f95b-4a6f-9884-5e25ca6e19a3","Type":"ContainerDied","Data":"05ad26f4f15169664c5b8f94b72fee7f6ba21fc23c4cfe8915c21d91659d0aba"} Nov 03 01:03:13.914318 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:13.913760 4421 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="05ad26f4f15169664c5b8f94b72fee7f6ba21fc23c4cfe8915c21d91659d0aba" Nov 03 01:03:13.917558 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:13.917534 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dc0f3f-59446658cf-n6msk" Nov 03 01:03:13.917704 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:13.917556 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dc0f3f-59446658cf-n6msk" event={"ID":"9ffa6158-c563-4d0e-91db-d5b6665bebb3","Type":"ContainerDied","Data":"c4a973da8f02b50af97187ff36fac67f37541e832c8a5f7b1e5ed3c5d2182a6c"} Nov 03 01:03:13.917704 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:13.917623 4421 scope.go:122] "RemoveContainer" containerID="98198872aec313e9d9129ebc5355a7602365a4de9803af2abcf7eed8e86247ce" Nov 03 01:03:13.919773 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:13.919664 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-dc0f3f-7b78d8bfdb-4s767" event={"ID":"f894ad12-9d4e-4133-860c-b053bbbaf136","Type":"ContainerDied","Data":"f98d231c2bab6183dba1b9d2a0f56dcd83d34a1d5c9d2d51a06d58ba39cf4f28"} Nov 03 01:03:13.919773 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:13.919732 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-dc0f3f-7b78d8bfdb-4s767" Nov 03 01:03:13.937153 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:13.937127 4421 scope.go:122] "RemoveContainer" containerID="d63dccf3bb816de8905099f315280f1fd57d15720dbcec16e154fb0bb410f98b" Nov 03 01:03:13.950747 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:13.950720 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-dc0f3f-59446658cf-n6msk"] Nov 03 01:03:13.957927 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:13.957897 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-dc0f3f-59446658cf-n6msk"] Nov 03 01:03:13.986977 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:13.986932 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ffa6158-c563-4d0e-91db-d5b6665bebb3" path="/var/lib/kubelet/pods/9ffa6158-c563-4d0e-91db-d5b6665bebb3/volumes" Nov 03 01:03:13.987367 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:13.987348 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-dc0f3f-7b78d8bfdb-4s767"] Nov 03 01:03:13.987460 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:13.987388 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-dc0f3f-7b78d8bfdb-4s767"] Nov 03 01:03:14.670103 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:03:14.670070 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 01:03:15.936754 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:15.936659 4421 generic.go:356] "Generic (PLEG): container finished" podID="33f4acce-9a60-4f9c-811d-81e8abfcabe8" containerID="d2a15723d7b6701815942f97a5762ea7f6fa989487c86467b5ff61c932816e5e" exitCode=0 Nov 03 01:03:15.936754 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:15.936739 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-db5422-openshift-default-696cf4dc99-75p47" event={"ID":"33f4acce-9a60-4f9c-811d-81e8abfcabe8","Type":"ContainerDied","Data":"d2a15723d7b6701815942f97a5762ea7f6fa989487c86467b5ff61c932816e5e"} Nov 03 01:03:15.937189 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:15.936806 4421 scope.go:122] "RemoveContainer" containerID="88750a243ff2f3dd38bac3810635e35888160a711beb60db021fceed8d99aeb1" Nov 03 01:03:15.979227 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:15.979193 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f894ad12-9d4e-4133-860c-b053bbbaf136" path="/var/lib/kubelet/pods/f894ad12-9d4e-4133-860c-b053bbbaf136/volumes" Nov 03 01:03:16.339693 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:16.339662 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-db5422-openshift-default-696cf4dc99-75p47" Nov 03 01:03:16.469415 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:16.469385 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33f4acce-9a60-4f9c-811d-81e8abfcabe8-kube-api-access-rvj9w\" (UniqueName: \"kubernetes.io/projected/33f4acce-9a60-4f9c-811d-81e8abfcabe8-kube-api-access-rvj9w\") pod \"33f4acce-9a60-4f9c-811d-81e8abfcabe8\" (UID: \"33f4acce-9a60-4f9c-811d-81e8abfcabe8\") " Nov 03 01:03:16.469568 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:16.469445 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/33f4acce-9a60-4f9c-811d-81e8abfcabe8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33f4acce-9a60-4f9c-811d-81e8abfcabe8-istio-podinfo\") pod \"33f4acce-9a60-4f9c-811d-81e8abfcabe8\" (UID: \"33f4acce-9a60-4f9c-811d-81e8abfcabe8\") " Nov 03 01:03:16.469568 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:16.469467 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33f4acce-9a60-4f9c-811d-81e8abfcabe8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33f4acce-9a60-4f9c-811d-81e8abfcabe8-workload-socket\") pod \"33f4acce-9a60-4f9c-811d-81e8abfcabe8\" (UID: \"33f4acce-9a60-4f9c-811d-81e8abfcabe8\") " Nov 03 01:03:16.469568 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:16.469526 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33f4acce-9a60-4f9c-811d-81e8abfcabe8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33f4acce-9a60-4f9c-811d-81e8abfcabe8-istio-envoy\") pod \"33f4acce-9a60-4f9c-811d-81e8abfcabe8\" (UID: \"33f4acce-9a60-4f9c-811d-81e8abfcabe8\") " Nov 03 01:03:16.469698 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:16.469650 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33f4acce-9a60-4f9c-811d-81e8abfcabe8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33f4acce-9a60-4f9c-811d-81e8abfcabe8-credential-socket\") pod \"33f4acce-9a60-4f9c-811d-81e8abfcabe8\" (UID: \"33f4acce-9a60-4f9c-811d-81e8abfcabe8\") " Nov 03 01:03:16.469797 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:16.469763 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33f4acce-9a60-4f9c-811d-81e8abfcabe8-istio-token\" (UniqueName: \"kubernetes.io/projected/33f4acce-9a60-4f9c-811d-81e8abfcabe8-istio-token\") pod \"33f4acce-9a60-4f9c-811d-81e8abfcabe8\" (UID: \"33f4acce-9a60-4f9c-811d-81e8abfcabe8\") " Nov 03 01:03:16.469864 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:16.469832 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33f4acce-9a60-4f9c-811d-81e8abfcabe8-workload-socket" pod "33f4acce-9a60-4f9c-811d-81e8abfcabe8" (UID: "33f4acce-9a60-4f9c-811d-81e8abfcabe8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:03:16.469941 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:16.469920 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33f4acce-9a60-4f9c-811d-81e8abfcabe8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33f4acce-9a60-4f9c-811d-81e8abfcabe8-istiod-ca-cert\") pod \"33f4acce-9a60-4f9c-811d-81e8abfcabe8\" (UID: \"33f4acce-9a60-4f9c-811d-81e8abfcabe8\") " Nov 03 01:03:16.470014 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:16.469998 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33f4acce-9a60-4f9c-811d-81e8abfcabe8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33f4acce-9a60-4f9c-811d-81e8abfcabe8-workload-certs\") pod \"33f4acce-9a60-4f9c-811d-81e8abfcabe8\" (UID: \"33f4acce-9a60-4f9c-811d-81e8abfcabe8\") " Nov 03 01:03:16.470078 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:16.469921 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33f4acce-9a60-4f9c-811d-81e8abfcabe8-credential-socket" pod "33f4acce-9a60-4f9c-811d-81e8abfcabe8" (UID: "33f4acce-9a60-4f9c-811d-81e8abfcabe8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:03:16.470149 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:16.470096 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33f4acce-9a60-4f9c-811d-81e8abfcabe8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33f4acce-9a60-4f9c-811d-81e8abfcabe8-istio-data\") pod \"33f4acce-9a60-4f9c-811d-81e8abfcabe8\" (UID: \"33f4acce-9a60-4f9c-811d-81e8abfcabe8\") " Nov 03 01:03:16.470300 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:16.470251 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33f4acce-9a60-4f9c-811d-81e8abfcabe8-istiod-ca-cert" pod "33f4acce-9a60-4f9c-811d-81e8abfcabe8" (UID: "33f4acce-9a60-4f9c-811d-81e8abfcabe8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:03:16.470454 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:16.470354 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33f4acce-9a60-4f9c-811d-81e8abfcabe8-workload-certs" pod "33f4acce-9a60-4f9c-811d-81e8abfcabe8" (UID: "33f4acce-9a60-4f9c-811d-81e8abfcabe8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:03:16.470599 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:16.470574 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33f4acce-9a60-4f9c-811d-81e8abfcabe8-istio-data" pod "33f4acce-9a60-4f9c-811d-81e8abfcabe8" (UID: "33f4acce-9a60-4f9c-811d-81e8abfcabe8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:03:16.471335 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:16.471310 4421 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33f4acce-9a60-4f9c-811d-81e8abfcabe8-workload-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:03:16.471441 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:16.471339 4421 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33f4acce-9a60-4f9c-811d-81e8abfcabe8-credential-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:03:16.471441 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:16.471358 4421 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33f4acce-9a60-4f9c-811d-81e8abfcabe8-istiod-ca-cert\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:03:16.471441 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:16.471377 4421 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33f4acce-9a60-4f9c-811d-81e8abfcabe8-workload-certs\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:03:16.471441 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:16.471393 4421 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33f4acce-9a60-4f9c-811d-81e8abfcabe8-istio-data\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:03:16.471850 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:16.471821 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/33f4acce-9a60-4f9c-811d-81e8abfcabe8-istio-podinfo" pod "33f4acce-9a60-4f9c-811d-81e8abfcabe8" (UID: "33f4acce-9a60-4f9c-811d-81e8abfcabe8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:03:16.471958 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:16.471938 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33f4acce-9a60-4f9c-811d-81e8abfcabe8-kube-api-access-rvj9w" pod "33f4acce-9a60-4f9c-811d-81e8abfcabe8" (UID: "33f4acce-9a60-4f9c-811d-81e8abfcabe8"). InnerVolumeSpecName "kube-api-access-rvj9w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:03:16.472188 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:16.472161 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33f4acce-9a60-4f9c-811d-81e8abfcabe8-istio-envoy" pod "33f4acce-9a60-4f9c-811d-81e8abfcabe8" (UID: "33f4acce-9a60-4f9c-811d-81e8abfcabe8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:03:16.472325 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:16.472232 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33f4acce-9a60-4f9c-811d-81e8abfcabe8-istio-token" pod "33f4acce-9a60-4f9c-811d-81e8abfcabe8" (UID: "33f4acce-9a60-4f9c-811d-81e8abfcabe8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:03:16.573167 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:16.573127 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvj9w\" (UniqueName: \"kubernetes.io/projected/33f4acce-9a60-4f9c-811d-81e8abfcabe8-kube-api-access-rvj9w\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:03:16.573167 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:16.573162 4421 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33f4acce-9a60-4f9c-811d-81e8abfcabe8-istio-podinfo\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:03:16.573167 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:16.573173 4421 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33f4acce-9a60-4f9c-811d-81e8abfcabe8-istio-envoy\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:03:16.573430 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:16.573182 4421 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33f4acce-9a60-4f9c-811d-81e8abfcabe8-istio-token\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:03:16.945183 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:16.945146 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-db5422-openshift-default-696cf4dc99-75p47" event={"ID":"33f4acce-9a60-4f9c-811d-81e8abfcabe8","Type":"ContainerDied","Data":"a55d07aed794ed8c392803da3d41f0277377785c34923f3086cf8839617cafae"} Nov 03 01:03:16.945690 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:16.945200 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-db5422-openshift-default-696cf4dc99-75p47" Nov 03 01:03:16.945690 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:16.945226 4421 scope.go:122] "RemoveContainer" containerID="d2a15723d7b6701815942f97a5762ea7f6fa989487c86467b5ff61c932816e5e" Nov 03 01:03:16.970897 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:16.970865 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-db5422-openshift-default-696cf4dc99-75p47"] Nov 03 01:03:16.977796 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:16.977767 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-db5422-openshift-default-696cf4dc99-75p47"] Nov 03 01:03:17.979890 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:17.979856 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33f4acce-9a60-4f9c-811d-81e8abfcabe8" path="/var/lib/kubelet/pods/33f4acce-9a60-4f9c-811d-81e8abfcabe8/volumes" Nov 03 01:03:18.739151 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:18.739121 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f14bda-5bf9699c87-z52pg" Nov 03 01:03:18.739359 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:18.739195 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-f14bda-7f498d85cc-v2z67" Nov 03 01:03:19.225091 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:19.225055 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77fa57c7-dsk6x"] Nov 03 01:03:19.225687 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:19.225673 4421 cpu_manager.go:414] "containerMap: removing container" podUID="33f4acce-9a60-4f9c-811d-81e8abfcabe8" containerName="istio-proxy" Nov 03 01:03:19.225737 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:19.225688 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33f4acce-9a60-4f9c-811d-81e8abfcabe8" containerName="istio-proxy" Nov 03 01:03:19.225737 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:19.225715 4421 cpu_manager.go:414] "containerMap: removing container" podUID="8fb996be-f95b-4a6f-9884-5e25ca6e19a3" containerName="revoke-keys" Nov 03 01:03:19.225737 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:19.225722 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fb996be-f95b-4a6f-9884-5e25ca6e19a3" containerName="revoke-keys" Nov 03 01:03:19.225737 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:19.225735 4421 cpu_manager.go:414] "containerMap: removing container" podUID="9ffa6158-c563-4d0e-91db-d5b6665bebb3" containerName="payload-pre-processing" Nov 03 01:03:19.225855 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:19.225740 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ffa6158-c563-4d0e-91db-d5b6665bebb3" containerName="payload-pre-processing" Nov 03 01:03:19.225855 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:19.225760 4421 cpu_manager.go:414] "containerMap: removing container" podUID="f894ad12-9d4e-4133-860c-b053bbbaf136" containerName="payload-processing" Nov 03 01:03:19.225855 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:19.225765 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f894ad12-9d4e-4133-860c-b053bbbaf136" containerName="payload-processing" Nov 03 01:03:19.225855 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:19.225842 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="f894ad12-9d4e-4133-860c-b053bbbaf136" containerName="payload-processing" Nov 03 01:03:19.225855 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:19.225853 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="33f4acce-9a60-4f9c-811d-81e8abfcabe8" containerName="istio-proxy" Nov 03 01:03:19.226009 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:19.225863 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fb996be-f95b-4a6f-9884-5e25ca6e19a3" containerName="revoke-keys" Nov 03 01:03:19.226009 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:19.225871 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ffa6158-c563-4d0e-91db-d5b6665bebb3" containerName="payload-pre-processing" Nov 03 01:03:19.229587 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:19.229570 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77fa57c7-dsk6x" Nov 03 01:03:19.235501 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:19.234590 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 01:03:19.235501 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:19.235159 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 01:03:19.237168 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:19.237130 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6q4xp\"" Nov 03 01:03:19.237338 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:19.237192 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:03:19.243685 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:19.243661 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77fa57c7-dsk6x"] Nov 03 01:03:19.318336 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:19.318254 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/075b622b-9932-49cf-8c79-87ebcdc7e4bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-77fa57c7-dsk6x\" (UID: \"075b622b-9932-49cf-8c79-87ebcdc7e4bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77fa57c7-dsk6x" Nov 03 01:03:19.318531 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:19.318392 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdfhc\" (UniqueName: \"kubernetes.io/projected/075b622b-9932-49cf-8c79-87ebcdc7e4bb-kube-api-access-qdfhc\") pod \"maas-api-revoke-sub-77fa57c7-dsk6x\" (UID: \"075b622b-9932-49cf-8c79-87ebcdc7e4bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77fa57c7-dsk6x" Nov 03 01:03:19.318593 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:19.318574 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/075b622b-9932-49cf-8c79-87ebcdc7e4bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-77fa57c7-dsk6x\" (UID: \"075b622b-9932-49cf-8c79-87ebcdc7e4bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77fa57c7-dsk6x" Nov 03 01:03:19.419955 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:19.419919 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/075b622b-9932-49cf-8c79-87ebcdc7e4bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-77fa57c7-dsk6x\" (UID: \"075b622b-9932-49cf-8c79-87ebcdc7e4bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77fa57c7-dsk6x" Nov 03 01:03:19.419955 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:19.419957 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdfhc\" (UniqueName: \"kubernetes.io/projected/075b622b-9932-49cf-8c79-87ebcdc7e4bb-kube-api-access-qdfhc\") pod \"maas-api-revoke-sub-77fa57c7-dsk6x\" (UID: \"075b622b-9932-49cf-8c79-87ebcdc7e4bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77fa57c7-dsk6x" Nov 03 01:03:19.420175 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:19.420077 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/075b622b-9932-49cf-8c79-87ebcdc7e4bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-77fa57c7-dsk6x\" (UID: \"075b622b-9932-49cf-8c79-87ebcdc7e4bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77fa57c7-dsk6x" Nov 03 01:03:19.420728 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:19.420704 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/075b622b-9932-49cf-8c79-87ebcdc7e4bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-77fa57c7-dsk6x\" (UID: \"075b622b-9932-49cf-8c79-87ebcdc7e4bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77fa57c7-dsk6x" Nov 03 01:03:19.422576 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:19.422545 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/075b622b-9932-49cf-8c79-87ebcdc7e4bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-77fa57c7-dsk6x\" (UID: \"075b622b-9932-49cf-8c79-87ebcdc7e4bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77fa57c7-dsk6x" Nov 03 01:03:19.427887 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:19.427856 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdfhc\" (UniqueName: \"kubernetes.io/projected/075b622b-9932-49cf-8c79-87ebcdc7e4bb-kube-api-access-qdfhc\") pod \"maas-api-revoke-sub-77fa57c7-dsk6x\" (UID: \"075b622b-9932-49cf-8c79-87ebcdc7e4bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77fa57c7-dsk6x" Nov 03 01:03:19.556789 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:19.556750 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77fa57c7-dsk6x" Nov 03 01:03:19.839197 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:19.838776 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77fa57c7-dsk6x"] Nov 03 01:03:19.982416 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:19.982376 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77fa57c7-dsk6x" event={"ID":"075b622b-9932-49cf-8c79-87ebcdc7e4bb","Type":"ContainerStarted","Data":"5a65b3b45d0b679e713309abe9197e4f92cf065992cb5f706e11bdbd7e5f213e"} Nov 03 01:03:20.995709 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:20.995672 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77fa57c7-dsk6x" event={"ID":"075b622b-9932-49cf-8c79-87ebcdc7e4bb","Type":"ContainerStarted","Data":"c85acad2d042be5cbaeae365de5cf3f9d3418bc577a72985dd1e53dc7fd85259"} Nov 03 01:03:22.044459 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:22.043844 4421 generic.go:356] "Generic (PLEG): container finished" podID="075b622b-9932-49cf-8c79-87ebcdc7e4bb" containerID="c85acad2d042be5cbaeae365de5cf3f9d3418bc577a72985dd1e53dc7fd85259" exitCode=0 Nov 03 01:03:22.044459 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:22.044064 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77fa57c7-dsk6x" event={"ID":"075b622b-9932-49cf-8c79-87ebcdc7e4bb","Type":"ContainerDied","Data":"c85acad2d042be5cbaeae365de5cf3f9d3418bc577a72985dd1e53dc7fd85259"} Nov 03 01:03:22.874736 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:22.874700 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a440037f-z754x"] Nov 03 01:03:22.875387 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:22.875371 4421 cpu_manager.go:414] "containerMap: removing container" podUID="33f4acce-9a60-4f9c-811d-81e8abfcabe8" containerName="istio-proxy" Nov 03 01:03:22.875451 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:22.875387 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33f4acce-9a60-4f9c-811d-81e8abfcabe8" containerName="istio-proxy" Nov 03 01:03:22.875515 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:22.875505 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="33f4acce-9a60-4f9c-811d-81e8abfcabe8" containerName="istio-proxy" Nov 03 01:03:22.878564 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:22.878549 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a440037f-z754x" Nov 03 01:03:22.885509 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:22.885480 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a440037f-z754x"] Nov 03 01:03:22.887846 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:22.887819 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/082dc2ba-188b-4d58-8d73-b53344fcd088-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a440037f-z754x\" (UID: \"082dc2ba-188b-4d58-8d73-b53344fcd088\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a440037f-z754x" Nov 03 01:03:22.887972 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:22.887863 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/082dc2ba-188b-4d58-8d73-b53344fcd088-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a440037f-z754x\" (UID: \"082dc2ba-188b-4d58-8d73-b53344fcd088\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a440037f-z754x" Nov 03 01:03:22.888057 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:22.888032 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68ggq\" (UniqueName: \"kubernetes.io/projected/082dc2ba-188b-4d58-8d73-b53344fcd088-kube-api-access-68ggq\") pod \"maas-api-revoke-sub-a440037f-z754x\" (UID: \"082dc2ba-188b-4d58-8d73-b53344fcd088\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a440037f-z754x" Nov 03 01:03:22.989750 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:22.989712 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/082dc2ba-188b-4d58-8d73-b53344fcd088-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a440037f-z754x\" (UID: \"082dc2ba-188b-4d58-8d73-b53344fcd088\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a440037f-z754x" Nov 03 01:03:22.990156 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:22.989773 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/082dc2ba-188b-4d58-8d73-b53344fcd088-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a440037f-z754x\" (UID: \"082dc2ba-188b-4d58-8d73-b53344fcd088\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a440037f-z754x" Nov 03 01:03:22.990156 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:22.989846 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68ggq\" (UniqueName: \"kubernetes.io/projected/082dc2ba-188b-4d58-8d73-b53344fcd088-kube-api-access-68ggq\") pod \"maas-api-revoke-sub-a440037f-z754x\" (UID: \"082dc2ba-188b-4d58-8d73-b53344fcd088\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a440037f-z754x" Nov 03 01:03:22.990446 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:22.990420 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/082dc2ba-188b-4d58-8d73-b53344fcd088-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a440037f-z754x\" (UID: \"082dc2ba-188b-4d58-8d73-b53344fcd088\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a440037f-z754x" Nov 03 01:03:22.992411 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:22.992387 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/082dc2ba-188b-4d58-8d73-b53344fcd088-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a440037f-z754x\" (UID: \"082dc2ba-188b-4d58-8d73-b53344fcd088\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a440037f-z754x" Nov 03 01:03:22.997961 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:22.997932 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68ggq\" (UniqueName: \"kubernetes.io/projected/082dc2ba-188b-4d58-8d73-b53344fcd088-kube-api-access-68ggq\") pod \"maas-api-revoke-sub-a440037f-z754x\" (UID: \"082dc2ba-188b-4d58-8d73-b53344fcd088\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a440037f-z754x" Nov 03 01:03:23.203880 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:23.203840 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a440037f-z754x" Nov 03 01:03:23.240964 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:23.240932 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77fa57c7-dsk6x" Nov 03 01:03:23.293692 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:23.293662 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/075b622b-9932-49cf-8c79-87ebcdc7e4bb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/075b622b-9932-49cf-8c79-87ebcdc7e4bb-maas-api-openshift-ca\") pod \"075b622b-9932-49cf-8c79-87ebcdc7e4bb\" (UID: \"075b622b-9932-49cf-8c79-87ebcdc7e4bb\") " Nov 03 01:03:23.293952 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:23.293932 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/075b622b-9932-49cf-8c79-87ebcdc7e4bb-kube-api-access-qdfhc\" (UniqueName: \"kubernetes.io/projected/075b622b-9932-49cf-8c79-87ebcdc7e4bb-kube-api-access-qdfhc\") pod \"075b622b-9932-49cf-8c79-87ebcdc7e4bb\" (UID: \"075b622b-9932-49cf-8c79-87ebcdc7e4bb\") " Nov 03 01:03:23.294061 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:23.294045 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/075b622b-9932-49cf-8c79-87ebcdc7e4bb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/075b622b-9932-49cf-8c79-87ebcdc7e4bb-maas-api-xks-ca\") pod \"075b622b-9932-49cf-8c79-87ebcdc7e4bb\" (UID: \"075b622b-9932-49cf-8c79-87ebcdc7e4bb\") " Nov 03 01:03:23.299294 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:23.294993 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/075b622b-9932-49cf-8c79-87ebcdc7e4bb-maas-api-openshift-ca" pod "075b622b-9932-49cf-8c79-87ebcdc7e4bb" (UID: "075b622b-9932-49cf-8c79-87ebcdc7e4bb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:03:23.299294 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:23.298852 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/075b622b-9932-49cf-8c79-87ebcdc7e4bb-maas-api-xks-ca" pod "075b622b-9932-49cf-8c79-87ebcdc7e4bb" (UID: "075b622b-9932-49cf-8c79-87ebcdc7e4bb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:03:23.299294 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:23.298949 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/075b622b-9932-49cf-8c79-87ebcdc7e4bb-kube-api-access-qdfhc" pod "075b622b-9932-49cf-8c79-87ebcdc7e4bb" (UID: "075b622b-9932-49cf-8c79-87ebcdc7e4bb"). InnerVolumeSpecName "kube-api-access-qdfhc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:03:23.395993 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:23.395959 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdfhc\" (UniqueName: \"kubernetes.io/projected/075b622b-9932-49cf-8c79-87ebcdc7e4bb-kube-api-access-qdfhc\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:03:23.395993 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:23.395992 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/075b622b-9932-49cf-8c79-87ebcdc7e4bb-maas-api-xks-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:03:23.395993 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:23.396003 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/075b622b-9932-49cf-8c79-87ebcdc7e4bb-maas-api-openshift-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:03:23.498339 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:23.498311 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a440037f-z754x"] Nov 03 01:03:23.499663 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:03:23.499634 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod082dc2ba_188b_4d58_8d73_b53344fcd088.slice/crio-6a46591df7690e448a8d702e0dd5a6157d7c225c7ec3d5c4adf1a7a1d27ce49f WatchSource:0}: Error finding container 6a46591df7690e448a8d702e0dd5a6157d7c225c7ec3d5c4adf1a7a1d27ce49f: Status 404 returned error can't find the container with id 6a46591df7690e448a8d702e0dd5a6157d7c225c7ec3d5c4adf1a7a1d27ce49f Nov 03 01:03:24.060072 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:24.060035 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77fa57c7-dsk6x" Nov 03 01:03:24.060263 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:24.060034 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77fa57c7-dsk6x" event={"ID":"075b622b-9932-49cf-8c79-87ebcdc7e4bb","Type":"ContainerDied","Data":"5a65b3b45d0b679e713309abe9197e4f92cf065992cb5f706e11bdbd7e5f213e"} Nov 03 01:03:24.060263 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:24.060159 4421 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5a65b3b45d0b679e713309abe9197e4f92cf065992cb5f706e11bdbd7e5f213e" Nov 03 01:03:24.062265 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:24.062232 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a440037f-z754x" event={"ID":"082dc2ba-188b-4d58-8d73-b53344fcd088","Type":"ContainerStarted","Data":"e396a3c0ced74d47e6d14ff45cc2d83b5863a36b517cd702248c2a11431c4e9b"} Nov 03 01:03:24.062265 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:24.062294 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a440037f-z754x" event={"ID":"082dc2ba-188b-4d58-8d73-b53344fcd088","Type":"ContainerStarted","Data":"6a46591df7690e448a8d702e0dd5a6157d7c225c7ec3d5c4adf1a7a1d27ce49f"} Nov 03 01:03:24.077983 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:24.077889 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a440037f-z754x" podStartSLOduration=2.077874892 podStartE2EDuration="2.077874892s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:03:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:03:24.077623028 +0000 UTC m=+1346.248626124" watchObservedRunningTime="2026-11-03 01:03:24.077874892 +0000 UTC m=+1346.248877990" Nov 03 01:03:25.129514 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:25.129474 4421 generic.go:356] "Generic (PLEG): container finished" podID="082dc2ba-188b-4d58-8d73-b53344fcd088" containerID="e396a3c0ced74d47e6d14ff45cc2d83b5863a36b517cd702248c2a11431c4e9b" exitCode=0 Nov 03 01:03:25.130017 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:25.129627 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a440037f-z754x" event={"ID":"082dc2ba-188b-4d58-8d73-b53344fcd088","Type":"ContainerDied","Data":"e396a3c0ced74d47e6d14ff45cc2d83b5863a36b517cd702248c2a11431c4e9b"} Nov 03 01:03:26.320980 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:26.320953 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a440037f-z754x" Nov 03 01:03:26.357032 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:26.357003 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/082dc2ba-188b-4d58-8d73-b53344fcd088-kube-api-access-68ggq\" (UniqueName: \"kubernetes.io/projected/082dc2ba-188b-4d58-8d73-b53344fcd088-kube-api-access-68ggq\") pod \"082dc2ba-188b-4d58-8d73-b53344fcd088\" (UID: \"082dc2ba-188b-4d58-8d73-b53344fcd088\") " Nov 03 01:03:26.357182 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:26.357078 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/082dc2ba-188b-4d58-8d73-b53344fcd088-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/082dc2ba-188b-4d58-8d73-b53344fcd088-maas-api-openshift-ca\") pod \"082dc2ba-188b-4d58-8d73-b53344fcd088\" (UID: \"082dc2ba-188b-4d58-8d73-b53344fcd088\") " Nov 03 01:03:26.357182 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:26.357130 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/082dc2ba-188b-4d58-8d73-b53344fcd088-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/082dc2ba-188b-4d58-8d73-b53344fcd088-maas-api-xks-ca\") pod \"082dc2ba-188b-4d58-8d73-b53344fcd088\" (UID: \"082dc2ba-188b-4d58-8d73-b53344fcd088\") " Nov 03 01:03:26.357549 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:26.357516 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/082dc2ba-188b-4d58-8d73-b53344fcd088-maas-api-openshift-ca" pod "082dc2ba-188b-4d58-8d73-b53344fcd088" (UID: "082dc2ba-188b-4d58-8d73-b53344fcd088"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:03:26.358014 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:26.357987 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/082dc2ba-188b-4d58-8d73-b53344fcd088-maas-api-openshift-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:03:26.359423 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:26.359392 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/082dc2ba-188b-4d58-8d73-b53344fcd088-maas-api-xks-ca" pod "082dc2ba-188b-4d58-8d73-b53344fcd088" (UID: "082dc2ba-188b-4d58-8d73-b53344fcd088"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:03:26.359509 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:26.359428 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/082dc2ba-188b-4d58-8d73-b53344fcd088-kube-api-access-68ggq" pod "082dc2ba-188b-4d58-8d73-b53344fcd088" (UID: "082dc2ba-188b-4d58-8d73-b53344fcd088"). InnerVolumeSpecName "kube-api-access-68ggq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:03:26.459394 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:26.459299 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68ggq\" (UniqueName: \"kubernetes.io/projected/082dc2ba-188b-4d58-8d73-b53344fcd088-kube-api-access-68ggq\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:03:26.459394 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:26.459337 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/082dc2ba-188b-4d58-8d73-b53344fcd088-maas-api-xks-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:03:27.168509 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:27.168476 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a440037f-z754x" Nov 03 01:03:27.168509 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:27.168499 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a440037f-z754x" event={"ID":"082dc2ba-188b-4d58-8d73-b53344fcd088","Type":"ContainerDied","Data":"6a46591df7690e448a8d702e0dd5a6157d7c225c7ec3d5c4adf1a7a1d27ce49f"} Nov 03 01:03:27.168806 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:27.168557 4421 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6a46591df7690e448a8d702e0dd5a6157d7c225c7ec3d5c4adf1a7a1d27ce49f" Nov 03 01:03:31.727312 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:31.727231 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ff4113-679c7cdbdb-cgkkl"] Nov 03 01:03:31.727815 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:31.727520 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-ff4113-679c7cdbdb-cgkkl" podUID="e35bd864-89b4-4a36-b59f-b623da7a86ac" containerName="payload-processing" containerID="cri-o://6222b760dcc5b9f9fa1adadb50368f14bd2193512c749f3c7dfe3bc326eb6c6f" gracePeriod=30 Nov 03 01:03:32.149413 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:32.149383 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e6556d35-vr5qh"] Nov 03 01:03:32.152191 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:32.152160 4421 cpu_manager.go:414] "containerMap: removing container" podUID="082dc2ba-188b-4d58-8d73-b53344fcd088" containerName="revoke-keys" Nov 03 01:03:32.152418 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:32.152401 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="082dc2ba-188b-4d58-8d73-b53344fcd088" containerName="revoke-keys" Nov 03 01:03:32.152529 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:32.152518 4421 cpu_manager.go:414] "containerMap: removing container" podUID="075b622b-9932-49cf-8c79-87ebcdc7e4bb" containerName="revoke-keys" Nov 03 01:03:32.152614 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:32.152602 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="075b622b-9932-49cf-8c79-87ebcdc7e4bb" containerName="revoke-keys" Nov 03 01:03:32.152936 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:32.152915 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="075b622b-9932-49cf-8c79-87ebcdc7e4bb" containerName="revoke-keys" Nov 03 01:03:32.153076 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:32.153065 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="082dc2ba-188b-4d58-8d73-b53344fcd088" containerName="revoke-keys" Nov 03 01:03:32.178636 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:32.178604 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e6556d35-vr5qh"] Nov 03 01:03:32.178787 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:32.178727 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6556d35-vr5qh" Nov 03 01:03:32.183724 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:32.183695 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 01:03:32.184139 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:32.184116 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 01:03:32.214838 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:32.214798 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ff4113-679c7cdbdb-cgkkl" Nov 03 01:03:32.220347 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:32.220316 4421 generic.go:356] "Generic (PLEG): container finished" podID="e35bd864-89b4-4a36-b59f-b623da7a86ac" containerID="6222b760dcc5b9f9fa1adadb50368f14bd2193512c749f3c7dfe3bc326eb6c6f" exitCode=0 Nov 03 01:03:32.220655 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:32.220634 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ff4113-679c7cdbdb-cgkkl" event={"ID":"e35bd864-89b4-4a36-b59f-b623da7a86ac","Type":"ContainerDied","Data":"6222b760dcc5b9f9fa1adadb50368f14bd2193512c749f3c7dfe3bc326eb6c6f"} Nov 03 01:03:32.220787 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:32.220773 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ff4113-679c7cdbdb-cgkkl" event={"ID":"e35bd864-89b4-4a36-b59f-b623da7a86ac","Type":"ContainerDied","Data":"7a0e6c3a4d4eac6bc523b05fc22135da5e8629bf192782ea3b6b437a7981c7a9"} Nov 03 01:03:32.220914 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:32.220902 4421 scope.go:122] "RemoveContainer" containerID="6222b760dcc5b9f9fa1adadb50368f14bd2193512c749f3c7dfe3bc326eb6c6f" Nov 03 01:03:32.221151 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:32.221131 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ff4113-679c7cdbdb-cgkkl" Nov 03 01:03:32.242005 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:32.241981 4421 scope.go:122] "RemoveContainer" containerID="6222b760dcc5b9f9fa1adadb50368f14bd2193512c749f3c7dfe3bc326eb6c6f" Nov 03 01:03:32.248400 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:32.247035 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:03:32.248541 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:32.247132 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6q4xp\"" Nov 03 01:03:32.248933 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:03:32.248909 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6222b760dcc5b9f9fa1adadb50368f14bd2193512c749f3c7dfe3bc326eb6c6f\": container with ID starting with 6222b760dcc5b9f9fa1adadb50368f14bd2193512c749f3c7dfe3bc326eb6c6f not found: ID does not exist" containerID="6222b760dcc5b9f9fa1adadb50368f14bd2193512c749f3c7dfe3bc326eb6c6f" Nov 03 01:03:32.249109 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:32.249083 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6222b760dcc5b9f9fa1adadb50368f14bd2193512c749f3c7dfe3bc326eb6c6f"} err="failed to get container status \"6222b760dcc5b9f9fa1adadb50368f14bd2193512c749f3c7dfe3bc326eb6c6f\": rpc error: code = NotFound desc = could not find container \"6222b760dcc5b9f9fa1adadb50368f14bd2193512c749f3c7dfe3bc326eb6c6f\": container with ID starting with 6222b760dcc5b9f9fa1adadb50368f14bd2193512c749f3c7dfe3bc326eb6c6f not found: ID does not exist" Nov 03 01:03:32.277879 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:32.277835 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e35bd864-89b4-4a36-b59f-b623da7a86ac-kube-api-access-2t55v\" (UniqueName: \"kubernetes.io/projected/e35bd864-89b4-4a36-b59f-b623da7a86ac-kube-api-access-2t55v\") pod \"e35bd864-89b4-4a36-b59f-b623da7a86ac\" (UID: \"e35bd864-89b4-4a36-b59f-b623da7a86ac\") " Nov 03 01:03:32.281041 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:32.280997 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e35bd864-89b4-4a36-b59f-b623da7a86ac-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e35bd864-89b4-4a36-b59f-b623da7a86ac-plugins-config-volume\") pod \"e35bd864-89b4-4a36-b59f-b623da7a86ac\" (UID: \"e35bd864-89b4-4a36-b59f-b623da7a86ac\") " Nov 03 01:03:32.281670 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:32.281646 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e35bd864-89b4-4a36-b59f-b623da7a86ac-plugins-config-volume" pod "e35bd864-89b4-4a36-b59f-b623da7a86ac" (UID: "e35bd864-89b4-4a36-b59f-b623da7a86ac"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:03:32.282019 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:32.281995 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7106c2fc-62bf-42cf-8faa-11f561707d0d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e6556d35-vr5qh\" (UID: \"7106c2fc-62bf-42cf-8faa-11f561707d0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6556d35-vr5qh" Nov 03 01:03:32.282266 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:32.282249 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bd8bx\" (UniqueName: \"kubernetes.io/projected/7106c2fc-62bf-42cf-8faa-11f561707d0d-kube-api-access-bd8bx\") pod \"maas-api-revoke-sub-e6556d35-vr5qh\" (UID: \"7106c2fc-62bf-42cf-8faa-11f561707d0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6556d35-vr5qh" Nov 03 01:03:32.282481 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:32.282460 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7106c2fc-62bf-42cf-8faa-11f561707d0d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e6556d35-vr5qh\" (UID: \"7106c2fc-62bf-42cf-8faa-11f561707d0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6556d35-vr5qh" Nov 03 01:03:32.282669 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:32.282642 4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e35bd864-89b4-4a36-b59f-b623da7a86ac-plugins-config-volume\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:03:32.284828 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:32.284771 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e35bd864-89b4-4a36-b59f-b623da7a86ac-kube-api-access-2t55v" pod "e35bd864-89b4-4a36-b59f-b623da7a86ac" (UID: "e35bd864-89b4-4a36-b59f-b623da7a86ac"). InnerVolumeSpecName "kube-api-access-2t55v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:03:32.384037 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:32.383999 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bd8bx\" (UniqueName: \"kubernetes.io/projected/7106c2fc-62bf-42cf-8faa-11f561707d0d-kube-api-access-bd8bx\") pod \"maas-api-revoke-sub-e6556d35-vr5qh\" (UID: \"7106c2fc-62bf-42cf-8faa-11f561707d0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6556d35-vr5qh" Nov 03 01:03:32.384267 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:32.384112 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7106c2fc-62bf-42cf-8faa-11f561707d0d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e6556d35-vr5qh\" (UID: \"7106c2fc-62bf-42cf-8faa-11f561707d0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6556d35-vr5qh" Nov 03 01:03:32.384267 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:32.384166 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7106c2fc-62bf-42cf-8faa-11f561707d0d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e6556d35-vr5qh\" (UID: \"7106c2fc-62bf-42cf-8faa-11f561707d0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6556d35-vr5qh" Nov 03 01:03:32.384450 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:32.384310 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2t55v\" (UniqueName: \"kubernetes.io/projected/e35bd864-89b4-4a36-b59f-b623da7a86ac-kube-api-access-2t55v\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:03:32.384859 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:32.384831 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7106c2fc-62bf-42cf-8faa-11f561707d0d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e6556d35-vr5qh\" (UID: \"7106c2fc-62bf-42cf-8faa-11f561707d0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6556d35-vr5qh" Nov 03 01:03:32.386719 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:32.386689 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7106c2fc-62bf-42cf-8faa-11f561707d0d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e6556d35-vr5qh\" (UID: \"7106c2fc-62bf-42cf-8faa-11f561707d0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6556d35-vr5qh" Nov 03 01:03:32.392344 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:32.392320 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bd8bx\" (UniqueName: \"kubernetes.io/projected/7106c2fc-62bf-42cf-8faa-11f561707d0d-kube-api-access-bd8bx\") pod \"maas-api-revoke-sub-e6556d35-vr5qh\" (UID: \"7106c2fc-62bf-42cf-8faa-11f561707d0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6556d35-vr5qh" Nov 03 01:03:32.529804 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:32.529767 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6556d35-vr5qh" Nov 03 01:03:32.548322 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:32.548253 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ff4113-679c7cdbdb-cgkkl"] Nov 03 01:03:32.552106 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:32.552079 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ff4113-679c7cdbdb-cgkkl"] Nov 03 01:03:32.810891 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:32.810862 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e6556d35-vr5qh"] Nov 03 01:03:32.812354 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:03:32.812326 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7106c2fc_62bf_42cf_8faa_11f561707d0d.slice/crio-af254db24a537b2354a5db4edf00de6e1792bc1d7ccd00140a5762725470487a WatchSource:0}: Error finding container af254db24a537b2354a5db4edf00de6e1792bc1d7ccd00140a5762725470487a: Status 404 returned error can't find the container with id af254db24a537b2354a5db4edf00de6e1792bc1d7ccd00140a5762725470487a Nov 03 01:03:33.234456 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:33.234399 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6556d35-vr5qh" event={"ID":"7106c2fc-62bf-42cf-8faa-11f561707d0d","Type":"ContainerStarted","Data":"36e5aecd991fdc78ce856619b8d0b93118d230fad7ca6c5eaa63ed0bd80204e0"} Nov 03 01:03:33.234634 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:33.234482 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6556d35-vr5qh" event={"ID":"7106c2fc-62bf-42cf-8faa-11f561707d0d","Type":"ContainerStarted","Data":"af254db24a537b2354a5db4edf00de6e1792bc1d7ccd00140a5762725470487a"} Nov 03 01:03:33.256769 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:33.256701 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6556d35-vr5qh" podStartSLOduration=2.256682507 podStartE2EDuration="2.256682507s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:03:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:03:33.249113634 +0000 UTC m=+1355.420116731" watchObservedRunningTime="2026-11-03 01:03:33.256682507 +0000 UTC m=+1355.427685604" Nov 03 01:03:33.984162 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:33.984069 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e35bd864-89b4-4a36-b59f-b623da7a86ac" path="/var/lib/kubelet/pods/e35bd864-89b4-4a36-b59f-b623da7a86ac/volumes" Nov 03 01:03:33.990722 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:33.990693 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-ff4113-openshift-default-6b94f7db5f-4p87g"] Nov 03 01:03:33.990998 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:33.990972 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-ff4113-openshift-default-6b94f7db5f-4p87g" podUID="6ab8d9c7-c645-4569-81a8-c5b1edaa7857" containerName="istio-proxy" containerID="cri-o://666f452f17f45ddc14cc8cde9b4809fc6a78eb9255dcc158e2021173ba609ffa" gracePeriod=30 Nov 03 01:03:34.257108 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:34.257069 4421 generic.go:356] "Generic (PLEG): container finished" podID="7106c2fc-62bf-42cf-8faa-11f561707d0d" containerID="36e5aecd991fdc78ce856619b8d0b93118d230fad7ca6c5eaa63ed0bd80204e0" exitCode=0 Nov 03 01:03:34.257443 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:34.257401 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6556d35-vr5qh" event={"ID":"7106c2fc-62bf-42cf-8faa-11f561707d0d","Type":"ContainerDied","Data":"36e5aecd991fdc78ce856619b8d0b93118d230fad7ca6c5eaa63ed0bd80204e0"} Nov 03 01:03:35.451464 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:35.451435 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6556d35-vr5qh" Nov 03 01:03:35.567709 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:35.567618 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7106c2fc-62bf-42cf-8faa-11f561707d0d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7106c2fc-62bf-42cf-8faa-11f561707d0d-maas-api-openshift-ca\") pod \"7106c2fc-62bf-42cf-8faa-11f561707d0d\" (UID: \"7106c2fc-62bf-42cf-8faa-11f561707d0d\") " Nov 03 01:03:35.567877 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:35.567809 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7106c2fc-62bf-42cf-8faa-11f561707d0d-kube-api-access-bd8bx\" (UniqueName: \"kubernetes.io/projected/7106c2fc-62bf-42cf-8faa-11f561707d0d-kube-api-access-bd8bx\") pod \"7106c2fc-62bf-42cf-8faa-11f561707d0d\" (UID: \"7106c2fc-62bf-42cf-8faa-11f561707d0d\") " Nov 03 01:03:35.567920 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:35.567910 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7106c2fc-62bf-42cf-8faa-11f561707d0d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7106c2fc-62bf-42cf-8faa-11f561707d0d-maas-api-xks-ca\") pod \"7106c2fc-62bf-42cf-8faa-11f561707d0d\" (UID: \"7106c2fc-62bf-42cf-8faa-11f561707d0d\") " Nov 03 01:03:35.567976 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:35.567949 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7106c2fc-62bf-42cf-8faa-11f561707d0d-maas-api-openshift-ca" pod "7106c2fc-62bf-42cf-8faa-11f561707d0d" (UID: "7106c2fc-62bf-42cf-8faa-11f561707d0d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:03:35.568317 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:35.568294 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7106c2fc-62bf-42cf-8faa-11f561707d0d-maas-api-openshift-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:03:35.570036 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:35.570010 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7106c2fc-62bf-42cf-8faa-11f561707d0d-maas-api-xks-ca" pod "7106c2fc-62bf-42cf-8faa-11f561707d0d" (UID: "7106c2fc-62bf-42cf-8faa-11f561707d0d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:03:35.570142 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:35.570096 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7106c2fc-62bf-42cf-8faa-11f561707d0d-kube-api-access-bd8bx" pod "7106c2fc-62bf-42cf-8faa-11f561707d0d" (UID: "7106c2fc-62bf-42cf-8faa-11f561707d0d"). InnerVolumeSpecName "kube-api-access-bd8bx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:03:35.669193 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:35.669158 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7106c2fc-62bf-42cf-8faa-11f561707d0d-maas-api-xks-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:03:35.669193 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:35.669187 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bd8bx\" (UniqueName: \"kubernetes.io/projected/7106c2fc-62bf-42cf-8faa-11f561707d0d-kube-api-access-bd8bx\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:03:36.005053 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:36.005017 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf2d166c-q8psv"] Nov 03 01:03:36.005673 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:36.005657 4421 cpu_manager.go:414] "containerMap: removing container" podUID="7106c2fc-62bf-42cf-8faa-11f561707d0d" containerName="revoke-keys" Nov 03 01:03:36.005749 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:36.005674 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7106c2fc-62bf-42cf-8faa-11f561707d0d" containerName="revoke-keys" Nov 03 01:03:36.005749 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:36.005701 4421 cpu_manager.go:414] "containerMap: removing container" podUID="e35bd864-89b4-4a36-b59f-b623da7a86ac" containerName="payload-processing" Nov 03 01:03:36.005749 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:36.005706 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e35bd864-89b4-4a36-b59f-b623da7a86ac" containerName="payload-processing" Nov 03 01:03:36.005847 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:36.005810 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="7106c2fc-62bf-42cf-8faa-11f561707d0d" containerName="revoke-keys" Nov 03 01:03:36.005847 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:36.005821 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="e35bd864-89b4-4a36-b59f-b623da7a86ac" containerName="payload-processing" Nov 03 01:03:36.009626 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:36.009605 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf2d166c-q8psv" Nov 03 01:03:36.017026 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:36.016991 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf2d166c-q8psv"] Nov 03 01:03:36.175858 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:36.175812 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/965c6f6a-6363-4ee1-8c1c-e5252502790c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cf2d166c-q8psv\" (UID: \"965c6f6a-6363-4ee1-8c1c-e5252502790c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf2d166c-q8psv" Nov 03 01:03:36.176011 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:36.175878 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/965c6f6a-6363-4ee1-8c1c-e5252502790c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cf2d166c-q8psv\" (UID: \"965c6f6a-6363-4ee1-8c1c-e5252502790c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf2d166c-q8psv" Nov 03 01:03:36.176076 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:36.176049 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kbfj\" (UniqueName: \"kubernetes.io/projected/965c6f6a-6363-4ee1-8c1c-e5252502790c-kube-api-access-6kbfj\") pod \"maas-api-revoke-sub-cf2d166c-q8psv\" (UID: \"965c6f6a-6363-4ee1-8c1c-e5252502790c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf2d166c-q8psv" Nov 03 01:03:36.272181 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:36.272149 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6556d35-vr5qh" Nov 03 01:03:36.272399 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:36.272179 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6556d35-vr5qh" event={"ID":"7106c2fc-62bf-42cf-8faa-11f561707d0d","Type":"ContainerDied","Data":"af254db24a537b2354a5db4edf00de6e1792bc1d7ccd00140a5762725470487a"} Nov 03 01:03:36.272399 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:36.272234 4421 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="af254db24a537b2354a5db4edf00de6e1792bc1d7ccd00140a5762725470487a" Nov 03 01:03:36.277214 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:36.277187 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/965c6f6a-6363-4ee1-8c1c-e5252502790c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cf2d166c-q8psv\" (UID: \"965c6f6a-6363-4ee1-8c1c-e5252502790c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf2d166c-q8psv" Nov 03 01:03:36.277382 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:36.277233 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/965c6f6a-6363-4ee1-8c1c-e5252502790c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cf2d166c-q8psv\" (UID: \"965c6f6a-6363-4ee1-8c1c-e5252502790c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf2d166c-q8psv" Nov 03 01:03:36.277382 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:36.277335 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kbfj\" (UniqueName: \"kubernetes.io/projected/965c6f6a-6363-4ee1-8c1c-e5252502790c-kube-api-access-6kbfj\") pod \"maas-api-revoke-sub-cf2d166c-q8psv\" (UID: \"965c6f6a-6363-4ee1-8c1c-e5252502790c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf2d166c-q8psv" Nov 03 01:03:36.277923 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:36.277903 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/965c6f6a-6363-4ee1-8c1c-e5252502790c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cf2d166c-q8psv\" (UID: \"965c6f6a-6363-4ee1-8c1c-e5252502790c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf2d166c-q8psv" Nov 03 01:03:36.279696 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:36.279673 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/965c6f6a-6363-4ee1-8c1c-e5252502790c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cf2d166c-q8psv\" (UID: \"965c6f6a-6363-4ee1-8c1c-e5252502790c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf2d166c-q8psv" Nov 03 01:03:36.285164 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:36.285140 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kbfj\" (UniqueName: \"kubernetes.io/projected/965c6f6a-6363-4ee1-8c1c-e5252502790c-kube-api-access-6kbfj\") pod \"maas-api-revoke-sub-cf2d166c-q8psv\" (UID: \"965c6f6a-6363-4ee1-8c1c-e5252502790c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf2d166c-q8psv" Nov 03 01:03:36.326433 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:36.326394 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf2d166c-q8psv" Nov 03 01:03:36.653300 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:03:36.653114 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod965c6f6a_6363_4ee1_8c1c_e5252502790c.slice/crio-0f0378b0126dcd21ed05ee1721004541088730b4ab294b1fa3beab958a0601e2 WatchSource:0}: Error finding container 0f0378b0126dcd21ed05ee1721004541088730b4ab294b1fa3beab958a0601e2: Status 404 returned error can't find the container with id 0f0378b0126dcd21ed05ee1721004541088730b4ab294b1fa3beab958a0601e2 Nov 03 01:03:36.653300 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:36.653147 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf2d166c-q8psv"] Nov 03 01:03:37.281595 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:37.281553 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf2d166c-q8psv" event={"ID":"965c6f6a-6363-4ee1-8c1c-e5252502790c","Type":"ContainerStarted","Data":"58e09291833fd3be000d16983f800524f4af36d51ae49c7b61c5389695631d79"} Nov 03 01:03:37.281773 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:37.281626 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf2d166c-q8psv" event={"ID":"965c6f6a-6363-4ee1-8c1c-e5252502790c","Type":"ContainerStarted","Data":"0f0378b0126dcd21ed05ee1721004541088730b4ab294b1fa3beab958a0601e2"} Nov 03 01:03:37.300773 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:37.300719 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf2d166c-q8psv" podStartSLOduration=2.300703161 podStartE2EDuration="2.300703161s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:03:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:03:37.296679634 +0000 UTC m=+1359.467682731" watchObservedRunningTime="2026-11-03 01:03:37.300703161 +0000 UTC m=+1359.471706257" Nov 03 01:03:38.289776 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:38.289740 4421 generic.go:356] "Generic (PLEG): container finished" podID="965c6f6a-6363-4ee1-8c1c-e5252502790c" containerID="58e09291833fd3be000d16983f800524f4af36d51ae49c7b61c5389695631d79" exitCode=0 Nov 03 01:03:38.290159 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:38.289835 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf2d166c-q8psv" event={"ID":"965c6f6a-6363-4ee1-8c1c-e5252502790c","Type":"ContainerDied","Data":"58e09291833fd3be000d16983f800524f4af36d51ae49c7b61c5389695631d79"} Nov 03 01:03:39.292491 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:39.292463 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-ff4113-openshift-default-6b94f7db5f-4p87g" Nov 03 01:03:39.298147 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:39.298119 4421 generic.go:356] "Generic (PLEG): container finished" podID="6ab8d9c7-c645-4569-81a8-c5b1edaa7857" containerID="666f452f17f45ddc14cc8cde9b4809fc6a78eb9255dcc158e2021173ba609ffa" exitCode=0 Nov 03 01:03:39.298358 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:39.298171 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-ff4113-openshift-default-6b94f7db5f-4p87g" Nov 03 01:03:39.298358 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:39.298198 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-ff4113-openshift-default-6b94f7db5f-4p87g" event={"ID":"6ab8d9c7-c645-4569-81a8-c5b1edaa7857","Type":"ContainerDied","Data":"666f452f17f45ddc14cc8cde9b4809fc6a78eb9255dcc158e2021173ba609ffa"} Nov 03 01:03:39.298358 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:39.298257 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-ff4113-openshift-default-6b94f7db5f-4p87g" event={"ID":"6ab8d9c7-c645-4569-81a8-c5b1edaa7857","Type":"ContainerDied","Data":"1db82aab5dc5f2f49bb7aa828097dc67f390672ba74e9e1cda4a78594fb0b472"} Nov 03 01:03:39.298358 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:39.298302 4421 scope.go:122] "RemoveContainer" containerID="666f452f17f45ddc14cc8cde9b4809fc6a78eb9255dcc158e2021173ba609ffa" Nov 03 01:03:39.309401 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:39.309387 4421 scope.go:122] "RemoveContainer" containerID="b478cd4bdd25d7b2802f37b37f38112c582f06cbd4121a2c311eb0d585a60df9" Nov 03 01:03:39.325876 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:39.325783 4421 scope.go:122] "RemoveContainer" containerID="666f452f17f45ddc14cc8cde9b4809fc6a78eb9255dcc158e2021173ba609ffa" Nov 03 01:03:39.326208 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:03:39.326108 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"666f452f17f45ddc14cc8cde9b4809fc6a78eb9255dcc158e2021173ba609ffa\": container with ID starting with 666f452f17f45ddc14cc8cde9b4809fc6a78eb9255dcc158e2021173ba609ffa not found: ID does not exist" containerID="666f452f17f45ddc14cc8cde9b4809fc6a78eb9255dcc158e2021173ba609ffa" Nov 03 01:03:39.326208 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:39.326138 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"666f452f17f45ddc14cc8cde9b4809fc6a78eb9255dcc158e2021173ba609ffa"} err="failed to get container status \"666f452f17f45ddc14cc8cde9b4809fc6a78eb9255dcc158e2021173ba609ffa\": rpc error: code = NotFound desc = could not find container \"666f452f17f45ddc14cc8cde9b4809fc6a78eb9255dcc158e2021173ba609ffa\": container with ID starting with 666f452f17f45ddc14cc8cde9b4809fc6a78eb9255dcc158e2021173ba609ffa not found: ID does not exist" Nov 03 01:03:39.326208 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:39.326154 4421 scope.go:122] "RemoveContainer" containerID="b478cd4bdd25d7b2802f37b37f38112c582f06cbd4121a2c311eb0d585a60df9" Nov 03 01:03:39.326451 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:03:39.326429 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b478cd4bdd25d7b2802f37b37f38112c582f06cbd4121a2c311eb0d585a60df9\": container with ID starting with b478cd4bdd25d7b2802f37b37f38112c582f06cbd4121a2c311eb0d585a60df9 not found: ID does not exist" containerID="b478cd4bdd25d7b2802f37b37f38112c582f06cbd4121a2c311eb0d585a60df9" Nov 03 01:03:39.326498 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:39.326456 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b478cd4bdd25d7b2802f37b37f38112c582f06cbd4121a2c311eb0d585a60df9"} err="failed to get container status \"b478cd4bdd25d7b2802f37b37f38112c582f06cbd4121a2c311eb0d585a60df9\": rpc error: code = NotFound desc = could not find container \"b478cd4bdd25d7b2802f37b37f38112c582f06cbd4121a2c311eb0d585a60df9\": container with ID starting with b478cd4bdd25d7b2802f37b37f38112c582f06cbd4121a2c311eb0d585a60df9 not found: ID does not exist" Nov 03 01:03:39.452793 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:39.442962 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ab8d9c7-c645-4569-81a8-c5b1edaa7857-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ab8d9c7-c645-4569-81a8-c5b1edaa7857-istiod-ca-cert\") pod \"6ab8d9c7-c645-4569-81a8-c5b1edaa7857\" (UID: \"6ab8d9c7-c645-4569-81a8-c5b1edaa7857\") " Nov 03 01:03:39.454700 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:39.447970 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ab8d9c7-c645-4569-81a8-c5b1edaa7857-istiod-ca-cert" pod "6ab8d9c7-c645-4569-81a8-c5b1edaa7857" (UID: "6ab8d9c7-c645-4569-81a8-c5b1edaa7857"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:03:39.454882 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:39.454738 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6ab8d9c7-c645-4569-81a8-c5b1edaa7857-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ab8d9c7-c645-4569-81a8-c5b1edaa7857-istio-podinfo\") pod \"6ab8d9c7-c645-4569-81a8-c5b1edaa7857\" (UID: \"6ab8d9c7-c645-4569-81a8-c5b1edaa7857\") " Nov 03 01:03:39.455001 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:39.454937 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ab8d9c7-c645-4569-81a8-c5b1edaa7857-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ab8d9c7-c645-4569-81a8-c5b1edaa7857-istio-envoy\") pod \"6ab8d9c7-c645-4569-81a8-c5b1edaa7857\" (UID: \"6ab8d9c7-c645-4569-81a8-c5b1edaa7857\") " Nov 03 01:03:39.455001 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:39.454990 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ab8d9c7-c645-4569-81a8-c5b1edaa7857-istio-token\" (UniqueName: \"kubernetes.io/projected/6ab8d9c7-c645-4569-81a8-c5b1edaa7857-istio-token\") pod \"6ab8d9c7-c645-4569-81a8-c5b1edaa7857\" (UID: \"6ab8d9c7-c645-4569-81a8-c5b1edaa7857\") " Nov 03 01:03:39.455096 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:39.455029 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ab8d9c7-c645-4569-81a8-c5b1edaa7857-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ab8d9c7-c645-4569-81a8-c5b1edaa7857-istio-data\") pod \"6ab8d9c7-c645-4569-81a8-c5b1edaa7857\" (UID: \"6ab8d9c7-c645-4569-81a8-c5b1edaa7857\") " Nov 03 01:03:39.455134 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:39.455103 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ab8d9c7-c645-4569-81a8-c5b1edaa7857-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ab8d9c7-c645-4569-81a8-c5b1edaa7857-credential-socket\") pod \"6ab8d9c7-c645-4569-81a8-c5b1edaa7857\" (UID: \"6ab8d9c7-c645-4569-81a8-c5b1edaa7857\") " Nov 03 01:03:39.455193 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:39.455146 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ab8d9c7-c645-4569-81a8-c5b1edaa7857-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ab8d9c7-c645-4569-81a8-c5b1edaa7857-workload-socket\") pod \"6ab8d9c7-c645-4569-81a8-c5b1edaa7857\" (UID: \"6ab8d9c7-c645-4569-81a8-c5b1edaa7857\") " Nov 03 01:03:39.455251 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:39.455203 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ab8d9c7-c645-4569-81a8-c5b1edaa7857-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ab8d9c7-c645-4569-81a8-c5b1edaa7857-workload-certs\") pod \"6ab8d9c7-c645-4569-81a8-c5b1edaa7857\" (UID: \"6ab8d9c7-c645-4569-81a8-c5b1edaa7857\") " Nov 03 01:03:39.455379 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:39.455362 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ab8d9c7-c645-4569-81a8-c5b1edaa7857-kube-api-access-k6z7g\" (UniqueName: \"kubernetes.io/projected/6ab8d9c7-c645-4569-81a8-c5b1edaa7857-kube-api-access-k6z7g\") pod \"6ab8d9c7-c645-4569-81a8-c5b1edaa7857\" (UID: \"6ab8d9c7-c645-4569-81a8-c5b1edaa7857\") " Nov 03 01:03:39.455446 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:39.455402 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ab8d9c7-c645-4569-81a8-c5b1edaa7857-istio-data" pod "6ab8d9c7-c645-4569-81a8-c5b1edaa7857" (UID: "6ab8d9c7-c645-4569-81a8-c5b1edaa7857"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:03:39.455647 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:39.455620 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ab8d9c7-c645-4569-81a8-c5b1edaa7857-workload-socket" pod "6ab8d9c7-c645-4569-81a8-c5b1edaa7857" (UID: "6ab8d9c7-c645-4569-81a8-c5b1edaa7857"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:03:39.455743 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:39.455724 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ab8d9c7-c645-4569-81a8-c5b1edaa7857-credential-socket" pod "6ab8d9c7-c645-4569-81a8-c5b1edaa7857" (UID: "6ab8d9c7-c645-4569-81a8-c5b1edaa7857"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:03:39.455824 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:39.455746 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ab8d9c7-c645-4569-81a8-c5b1edaa7857-workload-certs" pod "6ab8d9c7-c645-4569-81a8-c5b1edaa7857" (UID: "6ab8d9c7-c645-4569-81a8-c5b1edaa7857"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:03:39.456410 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:39.456387 4421 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ab8d9c7-c645-4569-81a8-c5b1edaa7857-istiod-ca-cert\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:03:39.456517 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:39.456417 4421 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ab8d9c7-c645-4569-81a8-c5b1edaa7857-istio-data\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:03:39.456517 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:39.456432 4421 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ab8d9c7-c645-4569-81a8-c5b1edaa7857-credential-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:03:39.456517 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:39.456456 4421 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ab8d9c7-c645-4569-81a8-c5b1edaa7857-workload-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:03:39.456517 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:39.456473 4421 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ab8d9c7-c645-4569-81a8-c5b1edaa7857-workload-certs\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:03:39.461241 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:39.461216 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ab8d9c7-c645-4569-81a8-c5b1edaa7857-istio-envoy" pod "6ab8d9c7-c645-4569-81a8-c5b1edaa7857" (UID: "6ab8d9c7-c645-4569-81a8-c5b1edaa7857"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:03:39.462767 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:39.462736 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6ab8d9c7-c645-4569-81a8-c5b1edaa7857-istio-podinfo" pod "6ab8d9c7-c645-4569-81a8-c5b1edaa7857" (UID: "6ab8d9c7-c645-4569-81a8-c5b1edaa7857"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:03:39.464471 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:39.464429 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ab8d9c7-c645-4569-81a8-c5b1edaa7857-istio-token" pod "6ab8d9c7-c645-4569-81a8-c5b1edaa7857" (UID: "6ab8d9c7-c645-4569-81a8-c5b1edaa7857"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:03:39.467506 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:39.467478 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ab8d9c7-c645-4569-81a8-c5b1edaa7857-kube-api-access-k6z7g" pod "6ab8d9c7-c645-4569-81a8-c5b1edaa7857" (UID: "6ab8d9c7-c645-4569-81a8-c5b1edaa7857"). InnerVolumeSpecName "kube-api-access-k6z7g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:03:39.515148 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:39.515085 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf2d166c-q8psv" Nov 03 01:03:39.557737 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:39.557698 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/965c6f6a-6363-4ee1-8c1c-e5252502790c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/965c6f6a-6363-4ee1-8c1c-e5252502790c-maas-api-xks-ca\") pod \"965c6f6a-6363-4ee1-8c1c-e5252502790c\" (UID: \"965c6f6a-6363-4ee1-8c1c-e5252502790c\") " Nov 03 01:03:39.557968 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:39.557838 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/965c6f6a-6363-4ee1-8c1c-e5252502790c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/965c6f6a-6363-4ee1-8c1c-e5252502790c-maas-api-openshift-ca\") pod \"965c6f6a-6363-4ee1-8c1c-e5252502790c\" (UID: \"965c6f6a-6363-4ee1-8c1c-e5252502790c\") " Nov 03 01:03:39.558054 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:39.558035 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/965c6f6a-6363-4ee1-8c1c-e5252502790c-kube-api-access-6kbfj\" (UniqueName: \"kubernetes.io/projected/965c6f6a-6363-4ee1-8c1c-e5252502790c-kube-api-access-6kbfj\") pod \"965c6f6a-6363-4ee1-8c1c-e5252502790c\" (UID: \"965c6f6a-6363-4ee1-8c1c-e5252502790c\") " Nov 03 01:03:39.558318 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:39.558287 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/965c6f6a-6363-4ee1-8c1c-e5252502790c-maas-api-openshift-ca" pod "965c6f6a-6363-4ee1-8c1c-e5252502790c" (UID: "965c6f6a-6363-4ee1-8c1c-e5252502790c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:03:39.558723 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:39.558697 4421 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ab8d9c7-c645-4569-81a8-c5b1edaa7857-istio-token\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:03:39.558829 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:39.558728 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/965c6f6a-6363-4ee1-8c1c-e5252502790c-maas-api-openshift-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:03:39.558829 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:39.558747 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6z7g\" (UniqueName: \"kubernetes.io/projected/6ab8d9c7-c645-4569-81a8-c5b1edaa7857-kube-api-access-k6z7g\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:03:39.558829 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:39.558762 4421 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ab8d9c7-c645-4569-81a8-c5b1edaa7857-istio-podinfo\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:03:39.558829 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:39.558775 4421 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ab8d9c7-c645-4569-81a8-c5b1edaa7857-istio-envoy\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:03:39.560263 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:39.560232 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/965c6f6a-6363-4ee1-8c1c-e5252502790c-kube-api-access-6kbfj" pod "965c6f6a-6363-4ee1-8c1c-e5252502790c" (UID: "965c6f6a-6363-4ee1-8c1c-e5252502790c"). InnerVolumeSpecName "kube-api-access-6kbfj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:03:39.560908 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:39.560886 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/965c6f6a-6363-4ee1-8c1c-e5252502790c-maas-api-xks-ca" pod "965c6f6a-6363-4ee1-8c1c-e5252502790c" (UID: "965c6f6a-6363-4ee1-8c1c-e5252502790c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:03:39.623207 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:39.623177 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-ff4113-openshift-default-6b94f7db5f-4p87g"] Nov 03 01:03:39.627449 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:39.627424 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-ff4113-openshift-default-6b94f7db5f-4p87g"] Nov 03 01:03:39.660295 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:39.660244 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6kbfj\" (UniqueName: \"kubernetes.io/projected/965c6f6a-6363-4ee1-8c1c-e5252502790c-kube-api-access-6kbfj\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:03:39.660436 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:39.660309 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/965c6f6a-6363-4ee1-8c1c-e5252502790c-maas-api-xks-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:03:39.978952 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:39.978868 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ab8d9c7-c645-4569-81a8-c5b1edaa7857" path="/var/lib/kubelet/pods/6ab8d9c7-c645-4569-81a8-c5b1edaa7857/volumes" Nov 03 01:03:40.308308 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:40.308262 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf2d166c-q8psv" Nov 03 01:03:40.308751 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:40.308262 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf2d166c-q8psv" event={"ID":"965c6f6a-6363-4ee1-8c1c-e5252502790c","Type":"ContainerDied","Data":"0f0378b0126dcd21ed05ee1721004541088730b4ab294b1fa3beab958a0601e2"} Nov 03 01:03:40.308751 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:40.308442 4421 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0f0378b0126dcd21ed05ee1721004541088730b4ab294b1fa3beab958a0601e2" Nov 03 01:03:43.034298 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:43.026360 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9lp9t"] Nov 03 01:03:43.034298 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:43.027325 4421 cpu_manager.go:414] "containerMap: removing container" podUID="6ab8d9c7-c645-4569-81a8-c5b1edaa7857" containerName="istio-proxy" Nov 03 01:03:43.034298 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:43.027346 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ab8d9c7-c645-4569-81a8-c5b1edaa7857" containerName="istio-proxy" Nov 03 01:03:43.034298 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:43.027397 4421 cpu_manager.go:414] "containerMap: removing container" podUID="965c6f6a-6363-4ee1-8c1c-e5252502790c" containerName="revoke-keys" Nov 03 01:03:43.034298 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:43.027404 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="965c6f6a-6363-4ee1-8c1c-e5252502790c" containerName="revoke-keys" Nov 03 01:03:43.034298 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:43.027565 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ab8d9c7-c645-4569-81a8-c5b1edaa7857" containerName="istio-proxy" Nov 03 01:03:43.034298 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:43.027586 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="965c6f6a-6363-4ee1-8c1c-e5252502790c" containerName="revoke-keys" Nov 03 01:03:43.105301 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:43.102087 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9lp9t"] Nov 03 01:03:43.105301 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:43.102237 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9lp9t" Nov 03 01:03:43.163327 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:43.163296 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vcq7\" (UniqueName: \"kubernetes.io/projected/d2a5f2d2-7356-496f-84ed-dfd3b2eea8e8-kube-api-access-6vcq7\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9lp9t\" (UID: \"d2a5f2d2-7356-496f-84ed-dfd3b2eea8e8\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9lp9t" Nov 03 01:03:43.163532 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:43.163335 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2a5f2d2-7356-496f-84ed-dfd3b2eea8e8-workload-certs\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9lp9t\" (UID: \"d2a5f2d2-7356-496f-84ed-dfd3b2eea8e8\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9lp9t" Nov 03 01:03:43.163532 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:43.163357 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2a5f2d2-7356-496f-84ed-dfd3b2eea8e8-istiod-ca-cert\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9lp9t\" (UID: \"d2a5f2d2-7356-496f-84ed-dfd3b2eea8e8\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9lp9t" Nov 03 01:03:43.163532 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:43.163488 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2a5f2d2-7356-496f-84ed-dfd3b2eea8e8-workload-socket\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9lp9t\" (UID: \"d2a5f2d2-7356-496f-84ed-dfd3b2eea8e8\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9lp9t" Nov 03 01:03:43.163532 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:43.163524 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2a5f2d2-7356-496f-84ed-dfd3b2eea8e8-istio-podinfo\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9lp9t\" (UID: \"d2a5f2d2-7356-496f-84ed-dfd3b2eea8e8\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9lp9t" Nov 03 01:03:43.163766 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:43.163702 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2a5f2d2-7356-496f-84ed-dfd3b2eea8e8-istio-token\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9lp9t\" (UID: \"d2a5f2d2-7356-496f-84ed-dfd3b2eea8e8\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9lp9t" Nov 03 01:03:43.163829 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:43.163763 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2a5f2d2-7356-496f-84ed-dfd3b2eea8e8-istio-data\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9lp9t\" (UID: \"d2a5f2d2-7356-496f-84ed-dfd3b2eea8e8\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9lp9t" Nov 03 01:03:43.163829 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:43.163809 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2a5f2d2-7356-496f-84ed-dfd3b2eea8e8-istio-envoy\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9lp9t\" (UID: \"d2a5f2d2-7356-496f-84ed-dfd3b2eea8e8\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9lp9t" Nov 03 01:03:43.163931 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:43.163888 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2a5f2d2-7356-496f-84ed-dfd3b2eea8e8-credential-socket\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9lp9t\" (UID: \"d2a5f2d2-7356-496f-84ed-dfd3b2eea8e8\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9lp9t" Nov 03 01:03:43.265294 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:43.265097 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vcq7\" (UniqueName: \"kubernetes.io/projected/d2a5f2d2-7356-496f-84ed-dfd3b2eea8e8-kube-api-access-6vcq7\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9lp9t\" (UID: \"d2a5f2d2-7356-496f-84ed-dfd3b2eea8e8\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9lp9t" Nov 03 01:03:43.265294 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:43.265155 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2a5f2d2-7356-496f-84ed-dfd3b2eea8e8-workload-certs\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9lp9t\" (UID: \"d2a5f2d2-7356-496f-84ed-dfd3b2eea8e8\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9lp9t" Nov 03 01:03:43.265294 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:43.265184 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2a5f2d2-7356-496f-84ed-dfd3b2eea8e8-istiod-ca-cert\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9lp9t\" (UID: \"d2a5f2d2-7356-496f-84ed-dfd3b2eea8e8\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9lp9t" Nov 03 01:03:43.265294 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:43.265224 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2a5f2d2-7356-496f-84ed-dfd3b2eea8e8-workload-socket\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9lp9t\" (UID: \"d2a5f2d2-7356-496f-84ed-dfd3b2eea8e8\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9lp9t" Nov 03 01:03:43.265294 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:43.265248 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2a5f2d2-7356-496f-84ed-dfd3b2eea8e8-istio-podinfo\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9lp9t\" (UID: \"d2a5f2d2-7356-496f-84ed-dfd3b2eea8e8\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9lp9t" Nov 03 01:03:43.265854 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:43.265351 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2a5f2d2-7356-496f-84ed-dfd3b2eea8e8-istio-token\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9lp9t\" (UID: \"d2a5f2d2-7356-496f-84ed-dfd3b2eea8e8\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9lp9t" Nov 03 01:03:43.268285 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:43.267829 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2a5f2d2-7356-496f-84ed-dfd3b2eea8e8-workload-certs\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9lp9t\" (UID: \"d2a5f2d2-7356-496f-84ed-dfd3b2eea8e8\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9lp9t" Nov 03 01:03:43.268285 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:43.267971 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2a5f2d2-7356-496f-84ed-dfd3b2eea8e8-istio-data\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9lp9t\" (UID: \"d2a5f2d2-7356-496f-84ed-dfd3b2eea8e8\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9lp9t" Nov 03 01:03:43.268285 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:43.268034 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2a5f2d2-7356-496f-84ed-dfd3b2eea8e8-istio-envoy\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9lp9t\" (UID: \"d2a5f2d2-7356-496f-84ed-dfd3b2eea8e8\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9lp9t" Nov 03 01:03:43.268285 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:43.268114 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2a5f2d2-7356-496f-84ed-dfd3b2eea8e8-credential-socket\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9lp9t\" (UID: \"d2a5f2d2-7356-496f-84ed-dfd3b2eea8e8\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9lp9t" Nov 03 01:03:43.268640 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:43.268591 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2a5f2d2-7356-496f-84ed-dfd3b2eea8e8-credential-socket\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9lp9t\" (UID: \"d2a5f2d2-7356-496f-84ed-dfd3b2eea8e8\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9lp9t" Nov 03 01:03:43.269246 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:43.269216 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2a5f2d2-7356-496f-84ed-dfd3b2eea8e8-istiod-ca-cert\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9lp9t\" (UID: \"d2a5f2d2-7356-496f-84ed-dfd3b2eea8e8\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9lp9t" Nov 03 01:03:43.269841 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:43.269583 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2a5f2d2-7356-496f-84ed-dfd3b2eea8e8-istio-data\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9lp9t\" (UID: \"d2a5f2d2-7356-496f-84ed-dfd3b2eea8e8\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9lp9t" Nov 03 01:03:43.270018 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:43.269997 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2a5f2d2-7356-496f-84ed-dfd3b2eea8e8-workload-socket\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9lp9t\" (UID: \"d2a5f2d2-7356-496f-84ed-dfd3b2eea8e8\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9lp9t" Nov 03 01:03:43.272239 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:43.272216 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2a5f2d2-7356-496f-84ed-dfd3b2eea8e8-istio-envoy\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9lp9t\" (UID: \"d2a5f2d2-7356-496f-84ed-dfd3b2eea8e8\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9lp9t" Nov 03 01:03:43.272666 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:43.272646 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2a5f2d2-7356-496f-84ed-dfd3b2eea8e8-istio-podinfo\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9lp9t\" (UID: \"d2a5f2d2-7356-496f-84ed-dfd3b2eea8e8\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9lp9t" Nov 03 01:03:43.278317 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:43.277086 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vcq7\" (UniqueName: \"kubernetes.io/projected/d2a5f2d2-7356-496f-84ed-dfd3b2eea8e8-kube-api-access-6vcq7\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9lp9t\" (UID: \"d2a5f2d2-7356-496f-84ed-dfd3b2eea8e8\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9lp9t" Nov 03 01:03:43.280517 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:43.280489 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2a5f2d2-7356-496f-84ed-dfd3b2eea8e8-istio-token\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9lp9t\" (UID: \"d2a5f2d2-7356-496f-84ed-dfd3b2eea8e8\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9lp9t" Nov 03 01:03:43.442350 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:43.441893 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9lp9t" Nov 03 01:03:43.856672 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:43.856632 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9lp9t"] Nov 03 01:03:43.857863 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:03:43.857833 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2a5f2d2_7356_496f_84ed_dfd3b2eea8e8.slice/crio-bdf2f0c3cdfe0ccc797d7a8504b592b02a72f6ff572163d1d22d8b5d7c8cf5fc WatchSource:0}: Error finding container bdf2f0c3cdfe0ccc797d7a8504b592b02a72f6ff572163d1d22d8b5d7c8cf5fc: Status 404 returned error can't find the container with id bdf2f0c3cdfe0ccc797d7a8504b592b02a72f6ff572163d1d22d8b5d7c8cf5fc Nov 03 01:03:43.861642 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:43.861599 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:03:43.861759 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:43.861697 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:03:43.861759 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:43.861743 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:03:44.377842 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:44.376988 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9lp9t" event={"ID":"d2a5f2d2-7356-496f-84ed-dfd3b2eea8e8","Type":"ContainerStarted","Data":"cc758e54f3e51c9d05b8ed7ec4dbb39fa8f4ee499be71add295e0cb4706cab9d"} Nov 03 01:03:44.377842 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:44.377094 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9lp9t" event={"ID":"d2a5f2d2-7356-496f-84ed-dfd3b2eea8e8","Type":"ContainerStarted","Data":"bdf2f0c3cdfe0ccc797d7a8504b592b02a72f6ff572163d1d22d8b5d7c8cf5fc"} Nov 03 01:03:44.403114 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:44.403051 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9lp9t" podStartSLOduration=2.403035206 podStartE2EDuration="2.403035206s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:03:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:03:44.400540517 +0000 UTC m=+1366.571543613" watchObservedRunningTime="2026-11-03 01:03:44.403035206 +0000 UTC m=+1366.574038301" Nov 03 01:03:44.443299 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:44.442842 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9lp9t" Nov 03 01:03:44.447067 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:44.447021 4421 patch_prober.go:29] interesting pod/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9lp9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" start-of-body= Nov 03 01:03:44.447347 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:44.447318 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9lp9t" podUID="d2a5f2d2-7356-496f-84ed-dfd3b2eea8e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" Nov 03 01:03:44.698062 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:03:44.697974 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 01:03:45.443413 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:45.443377 4421 patch_prober.go:29] interesting pod/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9lp9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" start-of-body= Nov 03 01:03:45.443837 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:45.443436 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9lp9t" podUID="d2a5f2d2-7356-496f-84ed-dfd3b2eea8e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" Nov 03 01:03:46.277241 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:46.277207 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-141f3283-5r4gf"] Nov 03 01:03:46.277875 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:46.277857 4421 cpu_manager.go:414] "containerMap: removing container" podUID="6ab8d9c7-c645-4569-81a8-c5b1edaa7857" containerName="istio-proxy" Nov 03 01:03:46.277928 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:46.277876 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ab8d9c7-c645-4569-81a8-c5b1edaa7857" containerName="istio-proxy" Nov 03 01:03:46.277986 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:46.277977 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ab8d9c7-c645-4569-81a8-c5b1edaa7857" containerName="istio-proxy" Nov 03 01:03:46.281657 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:46.281640 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-141f3283-5r4gf" Nov 03 01:03:46.287151 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:46.287124 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 01:03:46.287151 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:46.287125 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 01:03:46.287382 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:46.287160 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:03:46.287443 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:46.287431 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6q4xp\"" Nov 03 01:03:46.290601 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:46.290579 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-141f3283-5r4gf"] Nov 03 01:03:46.443366 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:46.443328 4421 patch_prober.go:29] interesting pod/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9lp9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" start-of-body= Nov 03 01:03:46.443586 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:46.443392 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9lp9t" podUID="d2a5f2d2-7356-496f-84ed-dfd3b2eea8e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" Nov 03 01:03:46.447833 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:46.447805 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0ae6246-255c-42fa-9210-adfaabe6f36f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-141f3283-5r4gf\" (UID: \"d0ae6246-255c-42fa-9210-adfaabe6f36f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-141f3283-5r4gf" Nov 03 01:03:46.448043 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:46.448021 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0ae6246-255c-42fa-9210-adfaabe6f36f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-141f3283-5r4gf\" (UID: \"d0ae6246-255c-42fa-9210-adfaabe6f36f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-141f3283-5r4gf" Nov 03 01:03:46.448143 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:46.448126 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhfcj\" (UniqueName: \"kubernetes.io/projected/d0ae6246-255c-42fa-9210-adfaabe6f36f-kube-api-access-nhfcj\") pod \"maas-api-revoke-sub-141f3283-5r4gf\" (UID: \"d0ae6246-255c-42fa-9210-adfaabe6f36f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-141f3283-5r4gf" Nov 03 01:03:46.549595 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:46.549503 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0ae6246-255c-42fa-9210-adfaabe6f36f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-141f3283-5r4gf\" (UID: \"d0ae6246-255c-42fa-9210-adfaabe6f36f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-141f3283-5r4gf" Nov 03 01:03:46.549595 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:46.549585 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhfcj\" (UniqueName: \"kubernetes.io/projected/d0ae6246-255c-42fa-9210-adfaabe6f36f-kube-api-access-nhfcj\") pod \"maas-api-revoke-sub-141f3283-5r4gf\" (UID: \"d0ae6246-255c-42fa-9210-adfaabe6f36f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-141f3283-5r4gf" Nov 03 01:03:46.549841 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:46.549740 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0ae6246-255c-42fa-9210-adfaabe6f36f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-141f3283-5r4gf\" (UID: \"d0ae6246-255c-42fa-9210-adfaabe6f36f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-141f3283-5r4gf" Nov 03 01:03:46.550692 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:46.550659 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0ae6246-255c-42fa-9210-adfaabe6f36f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-141f3283-5r4gf\" (UID: \"d0ae6246-255c-42fa-9210-adfaabe6f36f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-141f3283-5r4gf" Nov 03 01:03:46.552091 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:46.552067 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0ae6246-255c-42fa-9210-adfaabe6f36f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-141f3283-5r4gf\" (UID: \"d0ae6246-255c-42fa-9210-adfaabe6f36f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-141f3283-5r4gf" Nov 03 01:03:46.558825 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:46.558774 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhfcj\" (UniqueName: \"kubernetes.io/projected/d0ae6246-255c-42fa-9210-adfaabe6f36f-kube-api-access-nhfcj\") pod \"maas-api-revoke-sub-141f3283-5r4gf\" (UID: \"d0ae6246-255c-42fa-9210-adfaabe6f36f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-141f3283-5r4gf" Nov 03 01:03:46.599231 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:46.599198 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-141f3283-5r4gf" Nov 03 01:03:46.642394 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:46.642351 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24ca0955-cf9nt"] Nov 03 01:03:46.658591 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:46.658554 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24ca0955-cf9nt"] Nov 03 01:03:46.658776 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:46.658710 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24ca0955-cf9nt" Nov 03 01:03:46.754881 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:46.754728 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8m64m\" (UniqueName: \"kubernetes.io/projected/8e1f5804-5e9e-4cf3-b024-ec6d4cd9d739-kube-api-access-8m64m\") pod \"maas-api-revoke-sub-24ca0955-cf9nt\" (UID: \"8e1f5804-5e9e-4cf3-b024-ec6d4cd9d739\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24ca0955-cf9nt" Nov 03 01:03:46.754881 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:46.754842 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e1f5804-5e9e-4cf3-b024-ec6d4cd9d739-maas-api-xks-ca\") pod \"maas-api-revoke-sub-24ca0955-cf9nt\" (UID: \"8e1f5804-5e9e-4cf3-b024-ec6d4cd9d739\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24ca0955-cf9nt" Nov 03 01:03:46.755138 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:46.754906 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e1f5804-5e9e-4cf3-b024-ec6d4cd9d739-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-24ca0955-cf9nt\" (UID: \"8e1f5804-5e9e-4cf3-b024-ec6d4cd9d739\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24ca0955-cf9nt" Nov 03 01:03:46.768917 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:46.760482 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35618553-t49t6"] Nov 03 01:03:46.770336 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:46.770306 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35618553-t49t6" Nov 03 01:03:46.824343 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:46.801998 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35618553-t49t6"] Nov 03 01:03:46.856598 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:46.856571 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8m64m\" (UniqueName: \"kubernetes.io/projected/8e1f5804-5e9e-4cf3-b024-ec6d4cd9d739-kube-api-access-8m64m\") pod \"maas-api-revoke-sub-24ca0955-cf9nt\" (UID: \"8e1f5804-5e9e-4cf3-b024-ec6d4cd9d739\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24ca0955-cf9nt" Nov 03 01:03:46.856749 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:46.856608 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e1f5804-5e9e-4cf3-b024-ec6d4cd9d739-maas-api-xks-ca\") pod \"maas-api-revoke-sub-24ca0955-cf9nt\" (UID: \"8e1f5804-5e9e-4cf3-b024-ec6d4cd9d739\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24ca0955-cf9nt" Nov 03 01:03:46.856749 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:46.856640 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e1f5804-5e9e-4cf3-b024-ec6d4cd9d739-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-24ca0955-cf9nt\" (UID: \"8e1f5804-5e9e-4cf3-b024-ec6d4cd9d739\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24ca0955-cf9nt" Nov 03 01:03:46.856749 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:46.856716 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1df22d7-a6c9-41d0-a379-ab5e852fbc0a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-35618553-t49t6\" (UID: \"d1df22d7-a6c9-41d0-a379-ab5e852fbc0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35618553-t49t6" Nov 03 01:03:46.856863 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:46.856774 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1df22d7-a6c9-41d0-a379-ab5e852fbc0a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-35618553-t49t6\" (UID: \"d1df22d7-a6c9-41d0-a379-ab5e852fbc0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35618553-t49t6" Nov 03 01:03:46.856975 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:46.856953 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twfpd\" (UniqueName: \"kubernetes.io/projected/d1df22d7-a6c9-41d0-a379-ab5e852fbc0a-kube-api-access-twfpd\") pod \"maas-api-revoke-sub-35618553-t49t6\" (UID: \"d1df22d7-a6c9-41d0-a379-ab5e852fbc0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35618553-t49t6" Nov 03 01:03:46.857471 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:46.857445 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e1f5804-5e9e-4cf3-b024-ec6d4cd9d739-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-24ca0955-cf9nt\" (UID: \"8e1f5804-5e9e-4cf3-b024-ec6d4cd9d739\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24ca0955-cf9nt" Nov 03 01:03:46.859177 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:46.859155 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e1f5804-5e9e-4cf3-b024-ec6d4cd9d739-maas-api-xks-ca\") pod \"maas-api-revoke-sub-24ca0955-cf9nt\" (UID: \"8e1f5804-5e9e-4cf3-b024-ec6d4cd9d739\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24ca0955-cf9nt" Nov 03 01:03:46.865605 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:46.865578 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8m64m\" (UniqueName: \"kubernetes.io/projected/8e1f5804-5e9e-4cf3-b024-ec6d4cd9d739-kube-api-access-8m64m\") pod \"maas-api-revoke-sub-24ca0955-cf9nt\" (UID: \"8e1f5804-5e9e-4cf3-b024-ec6d4cd9d739\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24ca0955-cf9nt" Nov 03 01:03:46.958197 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:46.958161 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twfpd\" (UniqueName: \"kubernetes.io/projected/d1df22d7-a6c9-41d0-a379-ab5e852fbc0a-kube-api-access-twfpd\") pod \"maas-api-revoke-sub-35618553-t49t6\" (UID: \"d1df22d7-a6c9-41d0-a379-ab5e852fbc0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35618553-t49t6" Nov 03 01:03:46.958427 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:46.958320 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1df22d7-a6c9-41d0-a379-ab5e852fbc0a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-35618553-t49t6\" (UID: \"d1df22d7-a6c9-41d0-a379-ab5e852fbc0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35618553-t49t6" Nov 03 01:03:46.958427 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:46.958380 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1df22d7-a6c9-41d0-a379-ab5e852fbc0a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-35618553-t49t6\" (UID: \"d1df22d7-a6c9-41d0-a379-ab5e852fbc0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35618553-t49t6" Nov 03 01:03:46.959262 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:46.959228 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1df22d7-a6c9-41d0-a379-ab5e852fbc0a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-35618553-t49t6\" (UID: \"d1df22d7-a6c9-41d0-a379-ab5e852fbc0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35618553-t49t6" Nov 03 01:03:46.960644 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:46.960625 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1df22d7-a6c9-41d0-a379-ab5e852fbc0a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-35618553-t49t6\" (UID: \"d1df22d7-a6c9-41d0-a379-ab5e852fbc0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35618553-t49t6" Nov 03 01:03:46.984767 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:46.984743 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-twfpd\" (UniqueName: \"kubernetes.io/projected/d1df22d7-a6c9-41d0-a379-ab5e852fbc0a-kube-api-access-twfpd\") pod \"maas-api-revoke-sub-35618553-t49t6\" (UID: \"d1df22d7-a6c9-41d0-a379-ab5e852fbc0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35618553-t49t6" Nov 03 01:03:47.031358 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:47.029639 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-141f3283-5r4gf"] Nov 03 01:03:47.032160 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:03:47.032125 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0ae6246_255c_42fa_9210_adfaabe6f36f.slice/crio-0f5c953bca55eec6180c51c90a6c1d3028dc6bd7365c37fc2f8d7c1bf8653419 WatchSource:0}: Error finding container 0f5c953bca55eec6180c51c90a6c1d3028dc6bd7365c37fc2f8d7c1bf8653419: Status 404 returned error can't find the container with id 0f5c953bca55eec6180c51c90a6c1d3028dc6bd7365c37fc2f8d7c1bf8653419 Nov 03 01:03:47.062551 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:47.062467 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24ca0955-cf9nt" Nov 03 01:03:47.155298 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:47.154745 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35618553-t49t6" Nov 03 01:03:47.404833 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:47.404791 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-141f3283-5r4gf" event={"ID":"d0ae6246-255c-42fa-9210-adfaabe6f36f","Type":"ContainerStarted","Data":"0f5c953bca55eec6180c51c90a6c1d3028dc6bd7365c37fc2f8d7c1bf8653419"} Nov 03 01:03:47.440609 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:47.440570 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24ca0955-cf9nt"] Nov 03 01:03:47.447290 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:47.445831 4421 patch_prober.go:29] interesting pod/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9lp9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" start-of-body= Nov 03 01:03:47.447290 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:47.445883 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9lp9t" podUID="d2a5f2d2-7356-496f-84ed-dfd3b2eea8e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" Nov 03 01:03:47.448710 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:03:47.448685 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e1f5804_5e9e_4cf3_b024_ec6d4cd9d739.slice/crio-8fc96371ee53423b4568c6841793eb8ad45fd2d1fd6078d4a253987c7be14c86 WatchSource:0}: Error finding container 8fc96371ee53423b4568c6841793eb8ad45fd2d1fd6078d4a253987c7be14c86: Status 404 returned error can't find the container with id 8fc96371ee53423b4568c6841793eb8ad45fd2d1fd6078d4a253987c7be14c86 Nov 03 01:03:47.539487 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:47.539459 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35618553-t49t6"] Nov 03 01:03:47.638993 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:03:47.638889 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1df22d7_a6c9_41d0_a379_ab5e852fbc0a.slice/crio-c114bba484ad484cbbdfa9f3bb7a2a3053f1f5f2b2cfa1a6995f1464e14bbcb3 WatchSource:0}: Error finding container c114bba484ad484cbbdfa9f3bb7a2a3053f1f5f2b2cfa1a6995f1464e14bbcb3: Status 404 returned error can't find the container with id c114bba484ad484cbbdfa9f3bb7a2a3053f1f5f2b2cfa1a6995f1464e14bbcb3 Nov 03 01:03:48.443096 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:48.443054 4421 patch_prober.go:29] interesting pod/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9lp9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" start-of-body= Nov 03 01:03:48.443393 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:48.443362 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9lp9t" podUID="d2a5f2d2-7356-496f-84ed-dfd3b2eea8e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" Nov 03 01:03:48.444822 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:48.444793 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35618553-t49t6" event={"ID":"d1df22d7-a6c9-41d0-a379-ab5e852fbc0a","Type":"ContainerStarted","Data":"acb81fddd8a1546c0524b7590ee8369a268d40b88a5c0fa8abc1a4f80a1d53e9"} Nov 03 01:03:48.445017 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:48.445001 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35618553-t49t6" event={"ID":"d1df22d7-a6c9-41d0-a379-ab5e852fbc0a","Type":"ContainerStarted","Data":"c114bba484ad484cbbdfa9f3bb7a2a3053f1f5f2b2cfa1a6995f1464e14bbcb3"} Nov 03 01:03:48.461858 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:48.461827 4421 generic.go:356] "Generic (PLEG): container finished" podID="d0ae6246-255c-42fa-9210-adfaabe6f36f" containerID="6c9441f81a99800521b0bea31448f770fe1ef97866c8149dbaab8ff7b06e64f5" exitCode=0 Nov 03 01:03:48.462556 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:48.462517 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-141f3283-5r4gf" event={"ID":"d0ae6246-255c-42fa-9210-adfaabe6f36f","Type":"ContainerDied","Data":"6c9441f81a99800521b0bea31448f770fe1ef97866c8149dbaab8ff7b06e64f5"} Nov 03 01:03:48.469453 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:48.469417 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24ca0955-cf9nt" event={"ID":"8e1f5804-5e9e-4cf3-b024-ec6d4cd9d739","Type":"ContainerStarted","Data":"40beb931d095cbd2da03c06c770f6ee0ecf0c04ed98c516087c46962b09fc06c"} Nov 03 01:03:48.469673 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:48.469654 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24ca0955-cf9nt" event={"ID":"8e1f5804-5e9e-4cf3-b024-ec6d4cd9d739","Type":"ContainerStarted","Data":"8fc96371ee53423b4568c6841793eb8ad45fd2d1fd6078d4a253987c7be14c86"} Nov 03 01:03:48.502378 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:48.496794 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35618553-t49t6" podStartSLOduration=2.496775037 podStartE2EDuration="2.496775037s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:03:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:03:48.47086607 +0000 UTC m=+1370.641869167" watchObservedRunningTime="2026-11-03 01:03:48.496775037 +0000 UTC m=+1370.667778133" Nov 03 01:03:48.502378 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:48.501388 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24ca0955-cf9nt" podStartSLOduration=2.5013723839999997 podStartE2EDuration="2.501372384s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:03:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:03:48.492845117 +0000 UTC m=+1370.663848213" watchObservedRunningTime="2026-11-03 01:03:48.501372384 +0000 UTC m=+1370.672375480" Nov 03 01:03:49.443103 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:49.443059 4421 patch_prober.go:29] interesting pod/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9lp9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" start-of-body= Nov 03 01:03:49.443322 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:49.443137 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9lp9t" podUID="d2a5f2d2-7356-496f-84ed-dfd3b2eea8e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" Nov 03 01:03:49.478908 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:49.478871 4421 generic.go:356] "Generic (PLEG): container finished" podID="d1df22d7-a6c9-41d0-a379-ab5e852fbc0a" containerID="acb81fddd8a1546c0524b7590ee8369a268d40b88a5c0fa8abc1a4f80a1d53e9" exitCode=0 Nov 03 01:03:49.479395 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:49.478963 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35618553-t49t6" event={"ID":"d1df22d7-a6c9-41d0-a379-ab5e852fbc0a","Type":"ContainerDied","Data":"acb81fddd8a1546c0524b7590ee8369a268d40b88a5c0fa8abc1a4f80a1d53e9"} Nov 03 01:03:49.480875 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:49.480853 4421 generic.go:356] "Generic (PLEG): container finished" podID="8e1f5804-5e9e-4cf3-b024-ec6d4cd9d739" containerID="40beb931d095cbd2da03c06c770f6ee0ecf0c04ed98c516087c46962b09fc06c" exitCode=0 Nov 03 01:03:49.481028 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:49.480922 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24ca0955-cf9nt" event={"ID":"8e1f5804-5e9e-4cf3-b024-ec6d4cd9d739","Type":"ContainerDied","Data":"40beb931d095cbd2da03c06c770f6ee0ecf0c04ed98c516087c46962b09fc06c"} Nov 03 01:03:49.731630 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:49.731547 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-141f3283-5r4gf" Nov 03 01:03:49.746426 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:49.746396 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0ae6246-255c-42fa-9210-adfaabe6f36f-kube-api-access-nhfcj\" (UniqueName: \"kubernetes.io/projected/d0ae6246-255c-42fa-9210-adfaabe6f36f-kube-api-access-nhfcj\") pod \"d0ae6246-255c-42fa-9210-adfaabe6f36f\" (UID: \"d0ae6246-255c-42fa-9210-adfaabe6f36f\") " Nov 03 01:03:49.746601 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:49.746575 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0ae6246-255c-42fa-9210-adfaabe6f36f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0ae6246-255c-42fa-9210-adfaabe6f36f-maas-api-openshift-ca\") pod \"d0ae6246-255c-42fa-9210-adfaabe6f36f\" (UID: \"d0ae6246-255c-42fa-9210-adfaabe6f36f\") " Nov 03 01:03:49.746779 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:49.746742 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d0ae6246-255c-42fa-9210-adfaabe6f36f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0ae6246-255c-42fa-9210-adfaabe6f36f-maas-api-xks-ca\") pod \"d0ae6246-255c-42fa-9210-adfaabe6f36f\" (UID: \"d0ae6246-255c-42fa-9210-adfaabe6f36f\") " Nov 03 01:03:49.747101 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:49.747068 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0ae6246-255c-42fa-9210-adfaabe6f36f-maas-api-openshift-ca" pod "d0ae6246-255c-42fa-9210-adfaabe6f36f" (UID: "d0ae6246-255c-42fa-9210-adfaabe6f36f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:03:49.747533 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:49.747501 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0ae6246-255c-42fa-9210-adfaabe6f36f-maas-api-openshift-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:03:49.749241 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:49.749215 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0ae6246-255c-42fa-9210-adfaabe6f36f-kube-api-access-nhfcj" pod "d0ae6246-255c-42fa-9210-adfaabe6f36f" (UID: "d0ae6246-255c-42fa-9210-adfaabe6f36f"). InnerVolumeSpecName "kube-api-access-nhfcj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:03:49.749840 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:49.749814 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0ae6246-255c-42fa-9210-adfaabe6f36f-maas-api-xks-ca" pod "d0ae6246-255c-42fa-9210-adfaabe6f36f" (UID: "d0ae6246-255c-42fa-9210-adfaabe6f36f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:03:49.848780 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:49.848740 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0ae6246-255c-42fa-9210-adfaabe6f36f-maas-api-xks-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:03:49.848780 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:49.848778 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhfcj\" (UniqueName: \"kubernetes.io/projected/d0ae6246-255c-42fa-9210-adfaabe6f36f-kube-api-access-nhfcj\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:03:50.443328 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:50.443289 4421 patch_prober.go:29] interesting pod/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9lp9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" start-of-body= Nov 03 01:03:50.443542 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:50.443360 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9lp9t" podUID="d2a5f2d2-7356-496f-84ed-dfd3b2eea8e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" Nov 03 01:03:50.489797 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:50.489770 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-141f3283-5r4gf" Nov 03 01:03:50.490195 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:50.489798 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-141f3283-5r4gf" event={"ID":"d0ae6246-255c-42fa-9210-adfaabe6f36f","Type":"ContainerDied","Data":"0f5c953bca55eec6180c51c90a6c1d3028dc6bd7365c37fc2f8d7c1bf8653419"} Nov 03 01:03:50.490195 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:50.489852 4421 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0f5c953bca55eec6180c51c90a6c1d3028dc6bd7365c37fc2f8d7c1bf8653419" Nov 03 01:03:50.651055 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:50.651030 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24ca0955-cf9nt" Nov 03 01:03:50.662162 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:50.662122 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e1f5804-5e9e-4cf3-b024-ec6d4cd9d739-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e1f5804-5e9e-4cf3-b024-ec6d4cd9d739-maas-api-openshift-ca\") pod \"8e1f5804-5e9e-4cf3-b024-ec6d4cd9d739\" (UID: \"8e1f5804-5e9e-4cf3-b024-ec6d4cd9d739\") " Nov 03 01:03:50.662357 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:50.662181 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e1f5804-5e9e-4cf3-b024-ec6d4cd9d739-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e1f5804-5e9e-4cf3-b024-ec6d4cd9d739-maas-api-xks-ca\") pod \"8e1f5804-5e9e-4cf3-b024-ec6d4cd9d739\" (UID: \"8e1f5804-5e9e-4cf3-b024-ec6d4cd9d739\") " Nov 03 01:03:50.662357 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:50.662251 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e1f5804-5e9e-4cf3-b024-ec6d4cd9d739-kube-api-access-8m64m\" (UniqueName: \"kubernetes.io/projected/8e1f5804-5e9e-4cf3-b024-ec6d4cd9d739-kube-api-access-8m64m\") pod \"8e1f5804-5e9e-4cf3-b024-ec6d4cd9d739\" (UID: \"8e1f5804-5e9e-4cf3-b024-ec6d4cd9d739\") " Nov 03 01:03:50.663104 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:50.663057 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e1f5804-5e9e-4cf3-b024-ec6d4cd9d739-maas-api-openshift-ca" pod "8e1f5804-5e9e-4cf3-b024-ec6d4cd9d739" (UID: "8e1f5804-5e9e-4cf3-b024-ec6d4cd9d739"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:03:50.665593 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:50.665549 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e1f5804-5e9e-4cf3-b024-ec6d4cd9d739-kube-api-access-8m64m" pod "8e1f5804-5e9e-4cf3-b024-ec6d4cd9d739" (UID: "8e1f5804-5e9e-4cf3-b024-ec6d4cd9d739"). InnerVolumeSpecName "kube-api-access-8m64m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:03:50.667499 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:50.667472 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e1f5804-5e9e-4cf3-b024-ec6d4cd9d739-maas-api-xks-ca" pod "8e1f5804-5e9e-4cf3-b024-ec6d4cd9d739" (UID: "8e1f5804-5e9e-4cf3-b024-ec6d4cd9d739"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:03:50.778352 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:50.765945 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e1f5804-5e9e-4cf3-b024-ec6d4cd9d739-maas-api-openshift-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:03:50.778352 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:50.765984 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e1f5804-5e9e-4cf3-b024-ec6d4cd9d739-maas-api-xks-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:03:50.778352 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:50.766001 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8m64m\" (UniqueName: \"kubernetes.io/projected/8e1f5804-5e9e-4cf3-b024-ec6d4cd9d739-kube-api-access-8m64m\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:03:50.778693 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:50.778477 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35618553-t49t6" Nov 03 01:03:50.866644 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:50.866602 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1df22d7-a6c9-41d0-a379-ab5e852fbc0a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1df22d7-a6c9-41d0-a379-ab5e852fbc0a-maas-api-openshift-ca\") pod \"d1df22d7-a6c9-41d0-a379-ab5e852fbc0a\" (UID: \"d1df22d7-a6c9-41d0-a379-ab5e852fbc0a\") " Nov 03 01:03:50.866836 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:50.866751 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d1df22d7-a6c9-41d0-a379-ab5e852fbc0a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1df22d7-a6c9-41d0-a379-ab5e852fbc0a-maas-api-xks-ca\") pod \"d1df22d7-a6c9-41d0-a379-ab5e852fbc0a\" (UID: \"d1df22d7-a6c9-41d0-a379-ab5e852fbc0a\") " Nov 03 01:03:50.866930 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:50.866906 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1df22d7-a6c9-41d0-a379-ab5e852fbc0a-kube-api-access-twfpd\" (UniqueName: \"kubernetes.io/projected/d1df22d7-a6c9-41d0-a379-ab5e852fbc0a-kube-api-access-twfpd\") pod \"d1df22d7-a6c9-41d0-a379-ab5e852fbc0a\" (UID: \"d1df22d7-a6c9-41d0-a379-ab5e852fbc0a\") " Nov 03 01:03:50.867032 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:50.867011 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1df22d7-a6c9-41d0-a379-ab5e852fbc0a-maas-api-openshift-ca" pod "d1df22d7-a6c9-41d0-a379-ab5e852fbc0a" (UID: "d1df22d7-a6c9-41d0-a379-ab5e852fbc0a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:03:50.867401 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:50.867382 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1df22d7-a6c9-41d0-a379-ab5e852fbc0a-maas-api-openshift-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:03:50.869034 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:50.869003 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1df22d7-a6c9-41d0-a379-ab5e852fbc0a-kube-api-access-twfpd" pod "d1df22d7-a6c9-41d0-a379-ab5e852fbc0a" (UID: "d1df22d7-a6c9-41d0-a379-ab5e852fbc0a"). InnerVolumeSpecName "kube-api-access-twfpd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:03:50.869162 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:50.869068 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1df22d7-a6c9-41d0-a379-ab5e852fbc0a-maas-api-xks-ca" pod "d1df22d7-a6c9-41d0-a379-ab5e852fbc0a" (UID: "d1df22d7-a6c9-41d0-a379-ab5e852fbc0a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:03:50.969073 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:50.968979 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1df22d7-a6c9-41d0-a379-ab5e852fbc0a-maas-api-xks-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:03:50.969073 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:50.969015 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twfpd\" (UniqueName: \"kubernetes.io/projected/d1df22d7-a6c9-41d0-a379-ab5e852fbc0a-kube-api-access-twfpd\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:03:51.503528 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:51.503495 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35618553-t49t6" Nov 03 01:03:51.504024 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:51.503531 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35618553-t49t6" event={"ID":"d1df22d7-a6c9-41d0-a379-ab5e852fbc0a","Type":"ContainerDied","Data":"c114bba484ad484cbbdfa9f3bb7a2a3053f1f5f2b2cfa1a6995f1464e14bbcb3"} Nov 03 01:03:51.504024 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:51.503590 4421 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c114bba484ad484cbbdfa9f3bb7a2a3053f1f5f2b2cfa1a6995f1464e14bbcb3" Nov 03 01:03:51.505643 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:51.505448 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24ca0955-cf9nt" event={"ID":"8e1f5804-5e9e-4cf3-b024-ec6d4cd9d739","Type":"ContainerDied","Data":"8fc96371ee53423b4568c6841793eb8ad45fd2d1fd6078d4a253987c7be14c86"} Nov 03 01:03:51.505643 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:51.505488 4421 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8fc96371ee53423b4568c6841793eb8ad45fd2d1fd6078d4a253987c7be14c86" Nov 03 01:03:51.505643 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:51.505492 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24ca0955-cf9nt" Nov 03 01:03:51.521159 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:51.521022 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9lp9t" Nov 03 01:03:51.522127 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:51.522047 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9lp9t" Nov 03 01:03:51.523348 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:51.523324 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9lp9t" Nov 03 01:03:54.687993 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:54.687957 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-f14bda-7f498d85cc-v2z67"] Nov 03 01:03:54.700806 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:54.700757 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-f14bda-5bf9699c87-z52pg"] Nov 03 01:03:58.519102 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:58.519062 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39016e27-ldfgl"] Nov 03 01:03:58.519713 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:58.519697 4421 cpu_manager.go:414] "containerMap: removing container" podUID="d0ae6246-255c-42fa-9210-adfaabe6f36f" containerName="revoke-keys" Nov 03 01:03:58.519713 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:58.519713 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0ae6246-255c-42fa-9210-adfaabe6f36f" containerName="revoke-keys" Nov 03 01:03:58.519798 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:58.519726 4421 cpu_manager.go:414] "containerMap: removing container" podUID="d1df22d7-a6c9-41d0-a379-ab5e852fbc0a" containerName="revoke-keys" Nov 03 01:03:58.519798 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:58.519731 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1df22d7-a6c9-41d0-a379-ab5e852fbc0a" containerName="revoke-keys" Nov 03 01:03:58.519798 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:58.519744 4421 cpu_manager.go:414] "containerMap: removing container" podUID="8e1f5804-5e9e-4cf3-b024-ec6d4cd9d739" containerName="revoke-keys" Nov 03 01:03:58.519798 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:58.519750 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e1f5804-5e9e-4cf3-b024-ec6d4cd9d739" containerName="revoke-keys" Nov 03 01:03:58.519972 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:58.519862 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0ae6246-255c-42fa-9210-adfaabe6f36f" containerName="revoke-keys" Nov 03 01:03:58.519972 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:58.519870 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1df22d7-a6c9-41d0-a379-ab5e852fbc0a" containerName="revoke-keys" Nov 03 01:03:58.519972 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:58.519879 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e1f5804-5e9e-4cf3-b024-ec6d4cd9d739" containerName="revoke-keys" Nov 03 01:03:58.524561 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:58.524544 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39016e27-ldfgl" Nov 03 01:03:58.529777 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:58.529750 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:03:58.529961 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:58.529779 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 01:03:58.529961 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:58.529832 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39016e27-ldfgl"] Nov 03 01:03:58.530106 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:58.529883 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6q4xp\"" Nov 03 01:03:58.530106 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:58.529911 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 01:03:58.584850 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:58.584816 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/529f8d33-82a7-49d9-acdb-83e638d2e2a4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-39016e27-ldfgl\" (UID: \"529f8d33-82a7-49d9-acdb-83e638d2e2a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39016e27-ldfgl" Nov 03 01:03:58.585041 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:58.584872 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nfc9\" (UniqueName: \"kubernetes.io/projected/529f8d33-82a7-49d9-acdb-83e638d2e2a4-kube-api-access-8nfc9\") pod \"maas-api-revoke-sub-39016e27-ldfgl\" (UID: \"529f8d33-82a7-49d9-acdb-83e638d2e2a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39016e27-ldfgl" Nov 03 01:03:58.585297 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:58.585255 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/529f8d33-82a7-49d9-acdb-83e638d2e2a4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-39016e27-ldfgl\" (UID: \"529f8d33-82a7-49d9-acdb-83e638d2e2a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39016e27-ldfgl" Nov 03 01:03:58.686358 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:58.686320 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/529f8d33-82a7-49d9-acdb-83e638d2e2a4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-39016e27-ldfgl\" (UID: \"529f8d33-82a7-49d9-acdb-83e638d2e2a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39016e27-ldfgl" Nov 03 01:03:58.686516 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:58.686384 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/529f8d33-82a7-49d9-acdb-83e638d2e2a4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-39016e27-ldfgl\" (UID: \"529f8d33-82a7-49d9-acdb-83e638d2e2a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39016e27-ldfgl" Nov 03 01:03:58.686516 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:58.686433 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nfc9\" (UniqueName: \"kubernetes.io/projected/529f8d33-82a7-49d9-acdb-83e638d2e2a4-kube-api-access-8nfc9\") pod \"maas-api-revoke-sub-39016e27-ldfgl\" (UID: \"529f8d33-82a7-49d9-acdb-83e638d2e2a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39016e27-ldfgl" Nov 03 01:03:58.687024 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:58.687004 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/529f8d33-82a7-49d9-acdb-83e638d2e2a4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-39016e27-ldfgl\" (UID: \"529f8d33-82a7-49d9-acdb-83e638d2e2a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39016e27-ldfgl" Nov 03 01:03:58.688779 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:58.688756 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/529f8d33-82a7-49d9-acdb-83e638d2e2a4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-39016e27-ldfgl\" (UID: \"529f8d33-82a7-49d9-acdb-83e638d2e2a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39016e27-ldfgl" Nov 03 01:03:58.695940 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:58.695913 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nfc9\" (UniqueName: \"kubernetes.io/projected/529f8d33-82a7-49d9-acdb-83e638d2e2a4-kube-api-access-8nfc9\") pod \"maas-api-revoke-sub-39016e27-ldfgl\" (UID: \"529f8d33-82a7-49d9-acdb-83e638d2e2a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39016e27-ldfgl" Nov 03 01:03:58.823267 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:58.823185 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a125e38-pmmf4"] Nov 03 01:03:58.827677 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:58.827661 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a125e38-pmmf4" Nov 03 01:03:58.835265 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:58.835234 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a125e38-pmmf4"] Nov 03 01:03:58.841541 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:58.841514 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39016e27-ldfgl" Nov 03 01:03:58.888937 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:58.888899 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/990696f8-a427-49f3-bc32-326d521e6fa8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a125e38-pmmf4\" (UID: \"990696f8-a427-49f3-bc32-326d521e6fa8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a125e38-pmmf4" Nov 03 01:03:58.889146 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:58.888949 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgkmj\" (UniqueName: \"kubernetes.io/projected/990696f8-a427-49f3-bc32-326d521e6fa8-kube-api-access-pgkmj\") pod \"maas-api-revoke-sub-5a125e38-pmmf4\" (UID: \"990696f8-a427-49f3-bc32-326d521e6fa8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a125e38-pmmf4" Nov 03 01:03:58.889146 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:58.889067 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/990696f8-a427-49f3-bc32-326d521e6fa8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a125e38-pmmf4\" (UID: \"990696f8-a427-49f3-bc32-326d521e6fa8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a125e38-pmmf4" Nov 03 01:03:58.990079 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:58.990041 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/990696f8-a427-49f3-bc32-326d521e6fa8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a125e38-pmmf4\" (UID: \"990696f8-a427-49f3-bc32-326d521e6fa8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a125e38-pmmf4" Nov 03 01:03:58.990262 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:58.990088 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgkmj\" (UniqueName: \"kubernetes.io/projected/990696f8-a427-49f3-bc32-326d521e6fa8-kube-api-access-pgkmj\") pod \"maas-api-revoke-sub-5a125e38-pmmf4\" (UID: \"990696f8-a427-49f3-bc32-326d521e6fa8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a125e38-pmmf4" Nov 03 01:03:58.990262 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:58.990134 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/990696f8-a427-49f3-bc32-326d521e6fa8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a125e38-pmmf4\" (UID: \"990696f8-a427-49f3-bc32-326d521e6fa8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a125e38-pmmf4" Nov 03 01:03:58.990797 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:58.990774 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/990696f8-a427-49f3-bc32-326d521e6fa8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a125e38-pmmf4\" (UID: \"990696f8-a427-49f3-bc32-326d521e6fa8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a125e38-pmmf4" Nov 03 01:03:58.992569 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:58.992539 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/990696f8-a427-49f3-bc32-326d521e6fa8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a125e38-pmmf4\" (UID: \"990696f8-a427-49f3-bc32-326d521e6fa8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a125e38-pmmf4" Nov 03 01:03:58.998132 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:58.998106 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgkmj\" (UniqueName: \"kubernetes.io/projected/990696f8-a427-49f3-bc32-326d521e6fa8-kube-api-access-pgkmj\") pod \"maas-api-revoke-sub-5a125e38-pmmf4\" (UID: \"990696f8-a427-49f3-bc32-326d521e6fa8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a125e38-pmmf4" Nov 03 01:03:59.160356 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:59.159593 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a125e38-pmmf4" Nov 03 01:03:59.170906 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:59.169859 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39016e27-ldfgl"] Nov 03 01:03:59.457337 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:59.457307 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a125e38-pmmf4"] Nov 03 01:03:59.507879 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:03:59.507846 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod990696f8_a427_49f3_bc32_326d521e6fa8.slice/crio-0f8e29249183f9c27e9cc1c32c90a099fd1a926dbfb3d7c6c5ad086a0149ccff WatchSource:0}: Error finding container 0f8e29249183f9c27e9cc1c32c90a099fd1a926dbfb3d7c6c5ad086a0149ccff: Status 404 returned error can't find the container with id 0f8e29249183f9c27e9cc1c32c90a099fd1a926dbfb3d7c6c5ad086a0149ccff Nov 03 01:03:59.573714 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:59.573679 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39016e27-ldfgl" event={"ID":"529f8d33-82a7-49d9-acdb-83e638d2e2a4","Type":"ContainerStarted","Data":"685455d820d4891f21dc257ab41b7d22befe708357b99ac1d5f0536e34617f43"} Nov 03 01:03:59.574217 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:59.573736 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39016e27-ldfgl" event={"ID":"529f8d33-82a7-49d9-acdb-83e638d2e2a4","Type":"ContainerStarted","Data":"235d3c3459067255e029ecdcbc1dc9ea6c2106332b4cdf8b24483b26939db4e3"} Nov 03 01:03:59.575035 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:59.575011 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a125e38-pmmf4" event={"ID":"990696f8-a427-49f3-bc32-326d521e6fa8","Type":"ContainerStarted","Data":"0f8e29249183f9c27e9cc1c32c90a099fd1a926dbfb3d7c6c5ad086a0149ccff"} Nov 03 01:03:59.594398 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:59.594345 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39016e27-ldfgl" podStartSLOduration=1.5943311549999999 podStartE2EDuration="1.594331155s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:03:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:03:59.593747403 +0000 UTC m=+1381.764750499" watchObservedRunningTime="2026-11-03 01:03:59.594331155 +0000 UTC m=+1381.765334250" Nov 03 01:03:59.688505 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:59.688390 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-f14bda-7f498d85cc-v2z67" podUID="b384fb21-bb39-46b8-8b5a-40a74ce711bf" containerName="payload-processing" containerID="cri-o://f5422178a896d94db564137c39036a121f46f06a32076523d06477ceb1a3a204" gracePeriod=25 Nov 03 01:03:59.701686 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:03:59.701391 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f14bda-5bf9699c87-z52pg" podUID="5175e636-02af-4ba5-a711-226c87746bca" containerName="payload-pre-processing" containerID="cri-o://4d35d62e738bb3ca9a18086f3b671355bf4277096874d9c2ecd16ddac755621d" gracePeriod=25 Nov 03 01:04:00.068401 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:00.068374 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-f14bda-7f498d85cc-v2z67" Nov 03 01:04:00.077435 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:00.077403 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f14bda-5bf9699c87-z52pg" Nov 03 01:04:00.097645 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:00.097614 4421 scope.go:122] "RemoveContainer" containerID="f50ecb708219daf73c5693972dc90431e6192f29d538724d08f5b21b041748df" Nov 03 01:04:00.117703 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:00.115893 4421 scope.go:122] "RemoveContainer" containerID="b7580aacf21666a9deb4495ed376a7e4d913cb231095d3fefe7e6d89e0c7387e" Nov 03 01:04:00.117703 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:00.116164 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5175e636-02af-4ba5-a711-226c87746bca-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5175e636-02af-4ba5-a711-226c87746bca-plugins-config-volume\") pod \"5175e636-02af-4ba5-a711-226c87746bca\" (UID: \"5175e636-02af-4ba5-a711-226c87746bca\") " Nov 03 01:04:00.117703 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:00.116257 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b384fb21-bb39-46b8-8b5a-40a74ce711bf-kube-api-access-vbjqv\" (UniqueName: \"kubernetes.io/projected/b384fb21-bb39-46b8-8b5a-40a74ce711bf-kube-api-access-vbjqv\") pod \"b384fb21-bb39-46b8-8b5a-40a74ce711bf\" (UID: \"b384fb21-bb39-46b8-8b5a-40a74ce711bf\") " Nov 03 01:04:00.117703 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:00.116352 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5175e636-02af-4ba5-a711-226c87746bca-kube-api-access-k4777\" (UniqueName: \"kubernetes.io/projected/5175e636-02af-4ba5-a711-226c87746bca-kube-api-access-k4777\") pod \"5175e636-02af-4ba5-a711-226c87746bca\" (UID: \"5175e636-02af-4ba5-a711-226c87746bca\") " Nov 03 01:04:00.117703 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:00.116430 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b384fb21-bb39-46b8-8b5a-40a74ce711bf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b384fb21-bb39-46b8-8b5a-40a74ce711bf-plugins-config-volume\") pod \"b384fb21-bb39-46b8-8b5a-40a74ce711bf\" (UID: \"b384fb21-bb39-46b8-8b5a-40a74ce711bf\") " Nov 03 01:04:00.118194 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:00.118098 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5175e636-02af-4ba5-a711-226c87746bca-plugins-config-volume" pod "5175e636-02af-4ba5-a711-226c87746bca" (UID: "5175e636-02af-4ba5-a711-226c87746bca"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:04:00.118250 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:00.118188 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b384fb21-bb39-46b8-8b5a-40a74ce711bf-plugins-config-volume" pod "b384fb21-bb39-46b8-8b5a-40a74ce711bf" (UID: "b384fb21-bb39-46b8-8b5a-40a74ce711bf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:04:00.138933 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:00.138902 4421 scope.go:122] "RemoveContainer" containerID="9fac9d730ba79ac5c42913ff4ab87de7c119bc8ee5d510d3eb235d898c0585a5" Nov 03 01:04:00.148994 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:00.148974 4421 scope.go:122] "RemoveContainer" containerID="313f78e952d8a50e4f99f8c9af1526aa531bbadb03b67d5cd4744b498422217a" Nov 03 01:04:00.158534 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:00.158506 4421 scope.go:122] "RemoveContainer" containerID="bee8bf0fd1c08956484a5c09269610a4279268366e4a1ee82cfd9e93747c6531" Nov 03 01:04:00.205942 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:00.205867 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5175e636-02af-4ba5-a711-226c87746bca-kube-api-access-k4777" pod "5175e636-02af-4ba5-a711-226c87746bca" (UID: "5175e636-02af-4ba5-a711-226c87746bca"). InnerVolumeSpecName "kube-api-access-k4777". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:04:00.205942 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:00.205920 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b384fb21-bb39-46b8-8b5a-40a74ce711bf-kube-api-access-vbjqv" pod "b384fb21-bb39-46b8-8b5a-40a74ce711bf" (UID: "b384fb21-bb39-46b8-8b5a-40a74ce711bf"). InnerVolumeSpecName "kube-api-access-vbjqv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:04:00.219213 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:00.219185 4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5175e636-02af-4ba5-a711-226c87746bca-plugins-config-volume\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:04:00.219318 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:00.219216 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbjqv\" (UniqueName: \"kubernetes.io/projected/b384fb21-bb39-46b8-8b5a-40a74ce711bf-kube-api-access-vbjqv\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:04:00.219318 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:00.219227 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k4777\" (UniqueName: \"kubernetes.io/projected/5175e636-02af-4ba5-a711-226c87746bca-kube-api-access-k4777\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:04:00.219318 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:00.219241 4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b384fb21-bb39-46b8-8b5a-40a74ce711bf-plugins-config-volume\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:04:00.583904 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:00.583868 4421 generic.go:356] "Generic (PLEG): container finished" podID="b384fb21-bb39-46b8-8b5a-40a74ce711bf" containerID="f5422178a896d94db564137c39036a121f46f06a32076523d06477ceb1a3a204" exitCode=0 Nov 03 01:04:00.584449 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:00.583920 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-f14bda-7f498d85cc-v2z67" Nov 03 01:04:00.584449 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:00.583955 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-f14bda-7f498d85cc-v2z67" event={"ID":"b384fb21-bb39-46b8-8b5a-40a74ce711bf","Type":"ContainerDied","Data":"f5422178a896d94db564137c39036a121f46f06a32076523d06477ceb1a3a204"} Nov 03 01:04:00.584449 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:00.584025 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-f14bda-7f498d85cc-v2z67" event={"ID":"b384fb21-bb39-46b8-8b5a-40a74ce711bf","Type":"ContainerDied","Data":"5ffe9d2cc6bf1cd797cd70b7d45dff6c032d5d23e935905cc1519903a930c545"} Nov 03 01:04:00.584449 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:00.584058 4421 scope.go:122] "RemoveContainer" containerID="f5422178a896d94db564137c39036a121f46f06a32076523d06477ceb1a3a204" Nov 03 01:04:00.585477 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:00.585453 4421 generic.go:356] "Generic (PLEG): container finished" podID="5175e636-02af-4ba5-a711-226c87746bca" containerID="4d35d62e738bb3ca9a18086f3b671355bf4277096874d9c2ecd16ddac755621d" exitCode=0 Nov 03 01:04:00.585601 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:00.585502 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f14bda-5bf9699c87-z52pg" Nov 03 01:04:00.585661 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:00.585603 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f14bda-5bf9699c87-z52pg" event={"ID":"5175e636-02af-4ba5-a711-226c87746bca","Type":"ContainerDied","Data":"4d35d62e738bb3ca9a18086f3b671355bf4277096874d9c2ecd16ddac755621d"} Nov 03 01:04:00.585661 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:00.585647 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f14bda-5bf9699c87-z52pg" event={"ID":"5175e636-02af-4ba5-a711-226c87746bca","Type":"ContainerDied","Data":"4bb0977018345131a35273b528d70b9563508e44263cf14db2deed5bbc04a9a1"} Nov 03 01:04:00.587710 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:00.587685 4421 generic.go:356] "Generic (PLEG): container finished" podID="529f8d33-82a7-49d9-acdb-83e638d2e2a4" containerID="685455d820d4891f21dc257ab41b7d22befe708357b99ac1d5f0536e34617f43" exitCode=0 Nov 03 01:04:00.587845 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:00.587732 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39016e27-ldfgl" event={"ID":"529f8d33-82a7-49d9-acdb-83e638d2e2a4","Type":"ContainerDied","Data":"685455d820d4891f21dc257ab41b7d22befe708357b99ac1d5f0536e34617f43"} Nov 03 01:04:00.589883 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:00.589857 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a125e38-pmmf4" event={"ID":"990696f8-a427-49f3-bc32-326d521e6fa8","Type":"ContainerStarted","Data":"4de6da265253ba73223a3bbf7ab5220d340592ac13ed66e28f4ff9e624e32bb1"} Nov 03 01:04:00.595815 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:00.595789 4421 scope.go:122] "RemoveContainer" containerID="f5422178a896d94db564137c39036a121f46f06a32076523d06477ceb1a3a204" Nov 03 01:04:00.596118 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:04:00.596097 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5422178a896d94db564137c39036a121f46f06a32076523d06477ceb1a3a204\": container with ID starting with f5422178a896d94db564137c39036a121f46f06a32076523d06477ceb1a3a204 not found: ID does not exist" containerID="f5422178a896d94db564137c39036a121f46f06a32076523d06477ceb1a3a204" Nov 03 01:04:00.596192 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:00.596127 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5422178a896d94db564137c39036a121f46f06a32076523d06477ceb1a3a204"} err="failed to get container status \"f5422178a896d94db564137c39036a121f46f06a32076523d06477ceb1a3a204\": rpc error: code = NotFound desc = could not find container \"f5422178a896d94db564137c39036a121f46f06a32076523d06477ceb1a3a204\": container with ID starting with f5422178a896d94db564137c39036a121f46f06a32076523d06477ceb1a3a204 not found: ID does not exist" Nov 03 01:04:00.596192 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:00.596143 4421 scope.go:122] "RemoveContainer" containerID="4d35d62e738bb3ca9a18086f3b671355bf4277096874d9c2ecd16ddac755621d" Nov 03 01:04:00.605958 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:00.605935 4421 scope.go:122] "RemoveContainer" containerID="4d35d62e738bb3ca9a18086f3b671355bf4277096874d9c2ecd16ddac755621d" Nov 03 01:04:00.606312 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:04:00.606291 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d35d62e738bb3ca9a18086f3b671355bf4277096874d9c2ecd16ddac755621d\": container with ID starting with 4d35d62e738bb3ca9a18086f3b671355bf4277096874d9c2ecd16ddac755621d not found: ID does not exist" containerID="4d35d62e738bb3ca9a18086f3b671355bf4277096874d9c2ecd16ddac755621d" Nov 03 01:04:00.606375 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:00.606321 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d35d62e738bb3ca9a18086f3b671355bf4277096874d9c2ecd16ddac755621d"} err="failed to get container status \"4d35d62e738bb3ca9a18086f3b671355bf4277096874d9c2ecd16ddac755621d\": rpc error: code = NotFound desc = could not find container \"4d35d62e738bb3ca9a18086f3b671355bf4277096874d9c2ecd16ddac755621d\": container with ID starting with 4d35d62e738bb3ca9a18086f3b671355bf4277096874d9c2ecd16ddac755621d not found: ID does not exist" Nov 03 01:04:00.632498 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:00.632469 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-f14bda-5bf9699c87-z52pg"] Nov 03 01:04:00.638481 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:00.638450 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-f14bda-5bf9699c87-z52pg"] Nov 03 01:04:00.645186 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:00.645157 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-f14bda-7f498d85cc-v2z67"] Nov 03 01:04:00.651468 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:00.651441 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-f14bda-7f498d85cc-v2z67"] Nov 03 01:04:00.669487 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:00.669437 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a125e38-pmmf4" podStartSLOduration=2.669423246 podStartE2EDuration="2.669423246s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:03:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:04:00.665774767 +0000 UTC m=+1382.836777866" watchObservedRunningTime="2026-11-03 01:04:00.669423246 +0000 UTC m=+1382.840426373" Nov 03 01:04:01.598429 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:01.598395 4421 generic.go:356] "Generic (PLEG): container finished" podID="990696f8-a427-49f3-bc32-326d521e6fa8" containerID="4de6da265253ba73223a3bbf7ab5220d340592ac13ed66e28f4ff9e624e32bb1" exitCode=0 Nov 03 01:04:01.598879 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:01.598480 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a125e38-pmmf4" event={"ID":"990696f8-a427-49f3-bc32-326d521e6fa8","Type":"ContainerDied","Data":"4de6da265253ba73223a3bbf7ab5220d340592ac13ed66e28f4ff9e624e32bb1"} Nov 03 01:04:01.780195 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:01.780171 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39016e27-ldfgl" Nov 03 01:04:01.857521 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:01.857428 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/529f8d33-82a7-49d9-acdb-83e638d2e2a4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/529f8d33-82a7-49d9-acdb-83e638d2e2a4-maas-api-openshift-ca\") pod \"529f8d33-82a7-49d9-acdb-83e638d2e2a4\" (UID: \"529f8d33-82a7-49d9-acdb-83e638d2e2a4\") " Nov 03 01:04:01.857521 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:01.857481 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/529f8d33-82a7-49d9-acdb-83e638d2e2a4-kube-api-access-8nfc9\" (UniqueName: \"kubernetes.io/projected/529f8d33-82a7-49d9-acdb-83e638d2e2a4-kube-api-access-8nfc9\") pod \"529f8d33-82a7-49d9-acdb-83e638d2e2a4\" (UID: \"529f8d33-82a7-49d9-acdb-83e638d2e2a4\") " Nov 03 01:04:01.857772 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:01.857594 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/529f8d33-82a7-49d9-acdb-83e638d2e2a4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/529f8d33-82a7-49d9-acdb-83e638d2e2a4-maas-api-xks-ca\") pod \"529f8d33-82a7-49d9-acdb-83e638d2e2a4\" (UID: \"529f8d33-82a7-49d9-acdb-83e638d2e2a4\") " Nov 03 01:04:01.857839 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:01.857811 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/529f8d33-82a7-49d9-acdb-83e638d2e2a4-maas-api-openshift-ca" pod "529f8d33-82a7-49d9-acdb-83e638d2e2a4" (UID: "529f8d33-82a7-49d9-acdb-83e638d2e2a4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:04:01.858058 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:01.858041 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/529f8d33-82a7-49d9-acdb-83e638d2e2a4-maas-api-openshift-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:04:01.859731 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:01.859712 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/529f8d33-82a7-49d9-acdb-83e638d2e2a4-maas-api-xks-ca" pod "529f8d33-82a7-49d9-acdb-83e638d2e2a4" (UID: "529f8d33-82a7-49d9-acdb-83e638d2e2a4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:04:01.859820 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:01.859799 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/529f8d33-82a7-49d9-acdb-83e638d2e2a4-kube-api-access-8nfc9" pod "529f8d33-82a7-49d9-acdb-83e638d2e2a4" (UID: "529f8d33-82a7-49d9-acdb-83e638d2e2a4"). InnerVolumeSpecName "kube-api-access-8nfc9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:04:01.959186 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:01.959144 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/529f8d33-82a7-49d9-acdb-83e638d2e2a4-maas-api-xks-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:04:01.959186 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:01.959180 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8nfc9\" (UniqueName: \"kubernetes.io/projected/529f8d33-82a7-49d9-acdb-83e638d2e2a4-kube-api-access-8nfc9\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:04:01.980601 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:01.980566 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5175e636-02af-4ba5-a711-226c87746bca" path="/var/lib/kubelet/pods/5175e636-02af-4ba5-a711-226c87746bca/volumes" Nov 03 01:04:01.981006 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:01.980993 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b384fb21-bb39-46b8-8b5a-40a74ce711bf" path="/var/lib/kubelet/pods/b384fb21-bb39-46b8-8b5a-40a74ce711bf/volumes" Nov 03 01:04:02.609290 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:02.609250 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39016e27-ldfgl" Nov 03 01:04:02.609791 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:02.609256 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39016e27-ldfgl" event={"ID":"529f8d33-82a7-49d9-acdb-83e638d2e2a4","Type":"ContainerDied","Data":"235d3c3459067255e029ecdcbc1dc9ea6c2106332b4cdf8b24483b26939db4e3"} Nov 03 01:04:02.609791 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:02.609436 4421 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="235d3c3459067255e029ecdcbc1dc9ea6c2106332b4cdf8b24483b26939db4e3" Nov 03 01:04:02.827150 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:02.827122 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a125e38-pmmf4" Nov 03 01:04:02.988020 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:02.987932 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/990696f8-a427-49f3-bc32-326d521e6fa8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/990696f8-a427-49f3-bc32-326d521e6fa8-maas-api-openshift-ca\") pod \"990696f8-a427-49f3-bc32-326d521e6fa8\" (UID: \"990696f8-a427-49f3-bc32-326d521e6fa8\") " Nov 03 01:04:02.988020 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:02.988007 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/990696f8-a427-49f3-bc32-326d521e6fa8-kube-api-access-pgkmj\" (UniqueName: \"kubernetes.io/projected/990696f8-a427-49f3-bc32-326d521e6fa8-kube-api-access-pgkmj\") pod \"990696f8-a427-49f3-bc32-326d521e6fa8\" (UID: \"990696f8-a427-49f3-bc32-326d521e6fa8\") " Nov 03 01:04:02.988221 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:02.988150 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/990696f8-a427-49f3-bc32-326d521e6fa8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/990696f8-a427-49f3-bc32-326d521e6fa8-maas-api-xks-ca\") pod \"990696f8-a427-49f3-bc32-326d521e6fa8\" (UID: \"990696f8-a427-49f3-bc32-326d521e6fa8\") " Nov 03 01:04:02.988405 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:02.988382 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/990696f8-a427-49f3-bc32-326d521e6fa8-maas-api-openshift-ca" pod "990696f8-a427-49f3-bc32-326d521e6fa8" (UID: "990696f8-a427-49f3-bc32-326d521e6fa8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:04:02.988783 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:02.988757 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/990696f8-a427-49f3-bc32-326d521e6fa8-maas-api-openshift-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:04:02.990213 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:02.990186 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/990696f8-a427-49f3-bc32-326d521e6fa8-maas-api-xks-ca" pod "990696f8-a427-49f3-bc32-326d521e6fa8" (UID: "990696f8-a427-49f3-bc32-326d521e6fa8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:04:02.990328 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:02.990294 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/990696f8-a427-49f3-bc32-326d521e6fa8-kube-api-access-pgkmj" pod "990696f8-a427-49f3-bc32-326d521e6fa8" (UID: "990696f8-a427-49f3-bc32-326d521e6fa8"). InnerVolumeSpecName "kube-api-access-pgkmj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:04:03.100340 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:03.092659 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pgkmj\" (UniqueName: \"kubernetes.io/projected/990696f8-a427-49f3-bc32-326d521e6fa8-kube-api-access-pgkmj\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:04:03.100340 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:03.092706 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/990696f8-a427-49f3-bc32-326d521e6fa8-maas-api-xks-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:04:03.617102 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:03.617070 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a125e38-pmmf4" Nov 03 01:04:03.617102 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:03.617084 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a125e38-pmmf4" event={"ID":"990696f8-a427-49f3-bc32-326d521e6fa8","Type":"ContainerDied","Data":"0f8e29249183f9c27e9cc1c32c90a099fd1a926dbfb3d7c6c5ad086a0149ccff"} Nov 03 01:04:03.617556 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:03.617139 4421 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0f8e29249183f9c27e9cc1c32c90a099fd1a926dbfb3d7c6c5ad086a0149ccff" Nov 03 01:04:03.801493 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:03.801444 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15014ab9-hq8gm"] Nov 03 01:04:03.802030 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:03.802015 4421 cpu_manager.go:414] "containerMap: removing container" podUID="990696f8-a427-49f3-bc32-326d521e6fa8" containerName="revoke-keys" Nov 03 01:04:03.802030 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:03.802030 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="990696f8-a427-49f3-bc32-326d521e6fa8" containerName="revoke-keys" Nov 03 01:04:03.802157 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:03.802065 4421 cpu_manager.go:414] "containerMap: removing container" podUID="529f8d33-82a7-49d9-acdb-83e638d2e2a4" containerName="revoke-keys" Nov 03 01:04:03.802157 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:03.802071 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="529f8d33-82a7-49d9-acdb-83e638d2e2a4" containerName="revoke-keys" Nov 03 01:04:03.802157 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:03.802083 4421 cpu_manager.go:414] "containerMap: removing container" podUID="b384fb21-bb39-46b8-8b5a-40a74ce711bf" containerName="payload-processing" Nov 03 01:04:03.802157 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:03.802088 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b384fb21-bb39-46b8-8b5a-40a74ce711bf" containerName="payload-processing" Nov 03 01:04:03.802157 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:03.802097 4421 cpu_manager.go:414] "containerMap: removing container" podUID="5175e636-02af-4ba5-a711-226c87746bca" containerName="payload-pre-processing" Nov 03 01:04:03.802157 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:03.802101 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5175e636-02af-4ba5-a711-226c87746bca" containerName="payload-pre-processing" Nov 03 01:04:03.802471 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:03.802192 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="990696f8-a427-49f3-bc32-326d521e6fa8" containerName="revoke-keys" Nov 03 01:04:03.802471 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:03.802205 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="529f8d33-82a7-49d9-acdb-83e638d2e2a4" containerName="revoke-keys" Nov 03 01:04:03.802471 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:03.802212 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="b384fb21-bb39-46b8-8b5a-40a74ce711bf" containerName="payload-processing" Nov 03 01:04:03.802471 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:03.802218 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="5175e636-02af-4ba5-a711-226c87746bca" containerName="payload-pre-processing" Nov 03 01:04:03.805941 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:03.805920 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15014ab9-hq8gm" Nov 03 01:04:03.816314 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:03.815346 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6q4xp\"" Nov 03 01:04:03.817345 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:03.817324 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:04:03.818796 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:03.818775 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15014ab9-hq8gm"] Nov 03 01:04:03.845485 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:03.845450 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 01:04:03.845485 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:03.845466 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 01:04:03.911746 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:03.911640 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40a29166-9958-4232-bdae-16802b39f316-maas-api-xks-ca\") pod \"maas-api-revoke-sub-15014ab9-hq8gm\" (UID: \"40a29166-9958-4232-bdae-16802b39f316\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15014ab9-hq8gm" Nov 03 01:04:03.911746 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:03.911717 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4d8d\" (UniqueName: \"kubernetes.io/projected/40a29166-9958-4232-bdae-16802b39f316-kube-api-access-l4d8d\") pod \"maas-api-revoke-sub-15014ab9-hq8gm\" (UID: \"40a29166-9958-4232-bdae-16802b39f316\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15014ab9-hq8gm" Nov 03 01:04:03.911970 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:03.911886 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40a29166-9958-4232-bdae-16802b39f316-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-15014ab9-hq8gm\" (UID: \"40a29166-9958-4232-bdae-16802b39f316\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15014ab9-hq8gm" Nov 03 01:04:04.013493 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:04.013457 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40a29166-9958-4232-bdae-16802b39f316-maas-api-xks-ca\") pod \"maas-api-revoke-sub-15014ab9-hq8gm\" (UID: \"40a29166-9958-4232-bdae-16802b39f316\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15014ab9-hq8gm" Nov 03 01:04:04.013646 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:04.013519 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4d8d\" (UniqueName: \"kubernetes.io/projected/40a29166-9958-4232-bdae-16802b39f316-kube-api-access-l4d8d\") pod \"maas-api-revoke-sub-15014ab9-hq8gm\" (UID: \"40a29166-9958-4232-bdae-16802b39f316\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15014ab9-hq8gm" Nov 03 01:04:04.013646 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:04.013636 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40a29166-9958-4232-bdae-16802b39f316-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-15014ab9-hq8gm\" (UID: \"40a29166-9958-4232-bdae-16802b39f316\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15014ab9-hq8gm" Nov 03 01:04:04.014264 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:04.014239 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40a29166-9958-4232-bdae-16802b39f316-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-15014ab9-hq8gm\" (UID: \"40a29166-9958-4232-bdae-16802b39f316\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15014ab9-hq8gm" Nov 03 01:04:04.015972 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:04.015950 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40a29166-9958-4232-bdae-16802b39f316-maas-api-xks-ca\") pod \"maas-api-revoke-sub-15014ab9-hq8gm\" (UID: \"40a29166-9958-4232-bdae-16802b39f316\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15014ab9-hq8gm" Nov 03 01:04:04.023114 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:04.023091 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4d8d\" (UniqueName: \"kubernetes.io/projected/40a29166-9958-4232-bdae-16802b39f316-kube-api-access-l4d8d\") pod \"maas-api-revoke-sub-15014ab9-hq8gm\" (UID: \"40a29166-9958-4232-bdae-16802b39f316\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15014ab9-hq8gm" Nov 03 01:04:04.122051 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:04.122019 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15014ab9-hq8gm" Nov 03 01:04:04.407561 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:04.407532 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15014ab9-hq8gm"] Nov 03 01:04:04.408530 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:04:04.408499 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40a29166_9958_4232_bdae_16802b39f316.slice/crio-748aee8ae6d1465b02bf66492529857763a775b4a7b5554d067703b09190b3fa WatchSource:0}: Error finding container 748aee8ae6d1465b02bf66492529857763a775b4a7b5554d067703b09190b3fa: Status 404 returned error can't find the container with id 748aee8ae6d1465b02bf66492529857763a775b4a7b5554d067703b09190b3fa Nov 03 01:04:04.625871 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:04.625834 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15014ab9-hq8gm" event={"ID":"40a29166-9958-4232-bdae-16802b39f316","Type":"ContainerStarted","Data":"748aee8ae6d1465b02bf66492529857763a775b4a7b5554d067703b09190b3fa"} Nov 03 01:04:05.651327 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:05.649898 4421 generic.go:356] "Generic (PLEG): container finished" podID="40a29166-9958-4232-bdae-16802b39f316" containerID="d776777727ce1f660b9f6772b72313f206ac51e3d494fa3234e07f02318a1463" exitCode=0 Nov 03 01:04:05.651327 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:05.649981 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15014ab9-hq8gm" event={"ID":"40a29166-9958-4232-bdae-16802b39f316","Type":"ContainerDied","Data":"d776777727ce1f660b9f6772b72313f206ac51e3d494fa3234e07f02318a1463"} Nov 03 01:04:06.836154 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:06.836117 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15014ab9-hq8gm" Nov 03 01:04:06.885929 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:06.885880 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40a29166-9958-4232-bdae-16802b39f316-kube-api-access-l4d8d\" (UniqueName: \"kubernetes.io/projected/40a29166-9958-4232-bdae-16802b39f316-kube-api-access-l4d8d\") pod \"40a29166-9958-4232-bdae-16802b39f316\" (UID: \"40a29166-9958-4232-bdae-16802b39f316\") " Nov 03 01:04:06.886111 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:06.885983 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40a29166-9958-4232-bdae-16802b39f316-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40a29166-9958-4232-bdae-16802b39f316-maas-api-openshift-ca\") pod \"40a29166-9958-4232-bdae-16802b39f316\" (UID: \"40a29166-9958-4232-bdae-16802b39f316\") " Nov 03 01:04:06.886111 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:06.886099 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/40a29166-9958-4232-bdae-16802b39f316-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40a29166-9958-4232-bdae-16802b39f316-maas-api-xks-ca\") pod \"40a29166-9958-4232-bdae-16802b39f316\" (UID: \"40a29166-9958-4232-bdae-16802b39f316\") " Nov 03 01:04:06.886642 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:06.886397 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40a29166-9958-4232-bdae-16802b39f316-maas-api-openshift-ca" pod "40a29166-9958-4232-bdae-16802b39f316" (UID: "40a29166-9958-4232-bdae-16802b39f316"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:04:06.886775 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:06.886739 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40a29166-9958-4232-bdae-16802b39f316-maas-api-openshift-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:04:06.888258 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:06.888229 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40a29166-9958-4232-bdae-16802b39f316-kube-api-access-l4d8d" pod "40a29166-9958-4232-bdae-16802b39f316" (UID: "40a29166-9958-4232-bdae-16802b39f316"). InnerVolumeSpecName "kube-api-access-l4d8d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:04:06.888986 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:06.888959 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40a29166-9958-4232-bdae-16802b39f316-maas-api-xks-ca" pod "40a29166-9958-4232-bdae-16802b39f316" (UID: "40a29166-9958-4232-bdae-16802b39f316"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:04:06.988339 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:06.988204 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4d8d\" (UniqueName: \"kubernetes.io/projected/40a29166-9958-4232-bdae-16802b39f316-kube-api-access-l4d8d\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:04:06.988339 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:06.988239 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40a29166-9958-4232-bdae-16802b39f316-maas-api-xks-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:04:07.630949 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:07.630915 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1e2ac75-c9cfl"] Nov 03 01:04:07.631658 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:07.631637 4421 cpu_manager.go:414] "containerMap: removing container" podUID="40a29166-9958-4232-bdae-16802b39f316" containerName="revoke-keys" Nov 03 01:04:07.631658 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:07.631656 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40a29166-9958-4232-bdae-16802b39f316" containerName="revoke-keys" Nov 03 01:04:07.631804 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:07.631777 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="40a29166-9958-4232-bdae-16802b39f316" containerName="revoke-keys" Nov 03 01:04:07.635410 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:07.635390 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1e2ac75-c9cfl" Nov 03 01:04:07.643864 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:07.643834 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1e2ac75-c9cfl"] Nov 03 01:04:07.672521 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:07.668742 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15014ab9-hq8gm" event={"ID":"40a29166-9958-4232-bdae-16802b39f316","Type":"ContainerDied","Data":"748aee8ae6d1465b02bf66492529857763a775b4a7b5554d067703b09190b3fa"} Nov 03 01:04:07.672521 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:07.668815 4421 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="748aee8ae6d1465b02bf66492529857763a775b4a7b5554d067703b09190b3fa" Nov 03 01:04:07.672521 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:07.668863 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15014ab9-hq8gm" Nov 03 01:04:07.722688 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:07.722637 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d053a07c-ed76-4510-98ea-1097312fcf38-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f1e2ac75-c9cfl\" (UID: \"d053a07c-ed76-4510-98ea-1097312fcf38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1e2ac75-c9cfl" Nov 03 01:04:07.725571 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:07.722997 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwxqb\" (UniqueName: \"kubernetes.io/projected/d053a07c-ed76-4510-98ea-1097312fcf38-kube-api-access-bwxqb\") pod \"maas-api-revoke-sub-f1e2ac75-c9cfl\" (UID: \"d053a07c-ed76-4510-98ea-1097312fcf38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1e2ac75-c9cfl" Nov 03 01:04:07.725571 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:07.723083 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d053a07c-ed76-4510-98ea-1097312fcf38-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f1e2ac75-c9cfl\" (UID: \"d053a07c-ed76-4510-98ea-1097312fcf38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1e2ac75-c9cfl" Nov 03 01:04:07.827440 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:07.827389 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwxqb\" (UniqueName: \"kubernetes.io/projected/d053a07c-ed76-4510-98ea-1097312fcf38-kube-api-access-bwxqb\") pod \"maas-api-revoke-sub-f1e2ac75-c9cfl\" (UID: \"d053a07c-ed76-4510-98ea-1097312fcf38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1e2ac75-c9cfl" Nov 03 01:04:07.827618 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:07.827473 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d053a07c-ed76-4510-98ea-1097312fcf38-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f1e2ac75-c9cfl\" (UID: \"d053a07c-ed76-4510-98ea-1097312fcf38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1e2ac75-c9cfl" Nov 03 01:04:07.827618 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:07.827566 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d053a07c-ed76-4510-98ea-1097312fcf38-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f1e2ac75-c9cfl\" (UID: \"d053a07c-ed76-4510-98ea-1097312fcf38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1e2ac75-c9cfl" Nov 03 01:04:07.828642 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:07.828615 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d053a07c-ed76-4510-98ea-1097312fcf38-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f1e2ac75-c9cfl\" (UID: \"d053a07c-ed76-4510-98ea-1097312fcf38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1e2ac75-c9cfl" Nov 03 01:04:07.861417 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:07.861380 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwxqb\" (UniqueName: \"kubernetes.io/projected/d053a07c-ed76-4510-98ea-1097312fcf38-kube-api-access-bwxqb\") pod \"maas-api-revoke-sub-f1e2ac75-c9cfl\" (UID: \"d053a07c-ed76-4510-98ea-1097312fcf38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1e2ac75-c9cfl" Nov 03 01:04:07.868194 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:07.868156 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d053a07c-ed76-4510-98ea-1097312fcf38-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f1e2ac75-c9cfl\" (UID: \"d053a07c-ed76-4510-98ea-1097312fcf38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1e2ac75-c9cfl" Nov 03 01:04:07.955145 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:07.955057 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1e2ac75-c9cfl" Nov 03 01:04:08.255941 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:08.254045 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1e2ac75-c9cfl"] Nov 03 01:04:08.258514 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:04:08.258485 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd053a07c_ed76_4510_98ea_1097312fcf38.slice/crio-a94bf39044e8580dab6581b46fc0f9ab04ef2286aa2707ec6c08a3a11688736f WatchSource:0}: Error finding container a94bf39044e8580dab6581b46fc0f9ab04ef2286aa2707ec6c08a3a11688736f: Status 404 returned error can't find the container with id a94bf39044e8580dab6581b46fc0f9ab04ef2286aa2707ec6c08a3a11688736f Nov 03 01:04:08.677350 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:08.677309 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1e2ac75-c9cfl" event={"ID":"d053a07c-ed76-4510-98ea-1097312fcf38","Type":"ContainerStarted","Data":"18068f51d0b3f50cab57a90bf91c5aac8331e19017464b56be3536ddcfec62cf"} Nov 03 01:04:08.677500 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:08.677365 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1e2ac75-c9cfl" event={"ID":"d053a07c-ed76-4510-98ea-1097312fcf38","Type":"ContainerStarted","Data":"a94bf39044e8580dab6581b46fc0f9ab04ef2286aa2707ec6c08a3a11688736f"} Nov 03 01:04:08.695773 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:08.695717 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1e2ac75-c9cfl" podStartSLOduration=1.6957034439999998 podStartE2EDuration="1.695703444s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:04:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:04:08.692755104 +0000 UTC m=+1390.863758195" watchObservedRunningTime="2026-11-03 01:04:08.695703444 +0000 UTC m=+1390.866706557" Nov 03 01:04:09.639337 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:09.639298 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-g4cld"] Nov 03 01:04:09.643636 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:09.643591 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-g4cld" Nov 03 01:04:09.656349 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:09.656323 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-g4cld"] Nov 03 01:04:09.689432 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:09.689398 4421 generic.go:356] "Generic (PLEG): container finished" podID="d053a07c-ed76-4510-98ea-1097312fcf38" containerID="18068f51d0b3f50cab57a90bf91c5aac8331e19017464b56be3536ddcfec62cf" exitCode=0 Nov 03 01:04:09.689639 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:09.689490 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1e2ac75-c9cfl" event={"ID":"d053a07c-ed76-4510-98ea-1097312fcf38","Type":"ContainerDied","Data":"18068f51d0b3f50cab57a90bf91c5aac8331e19017464b56be3536ddcfec62cf"} Nov 03 01:04:09.776084 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:09.776048 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cef07ce-2ada-4d6d-93ba-0241dd7e95a4-credential-socket\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-g4cld\" (UID: \"2cef07ce-2ada-4d6d-93ba-0241dd7e95a4\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-g4cld" Nov 03 01:04:09.776249 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:09.776153 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2cef07ce-2ada-4d6d-93ba-0241dd7e95a4-istio-data\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-g4cld\" (UID: \"2cef07ce-2ada-4d6d-93ba-0241dd7e95a4\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-g4cld" Nov 03 01:04:09.776249 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:09.776226 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2cef07ce-2ada-4d6d-93ba-0241dd7e95a4-istio-token\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-g4cld\" (UID: \"2cef07ce-2ada-4d6d-93ba-0241dd7e95a4\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-g4cld" Nov 03 01:04:09.776249 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:09.776245 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2cef07ce-2ada-4d6d-93ba-0241dd7e95a4-workload-certs\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-g4cld\" (UID: \"2cef07ce-2ada-4d6d-93ba-0241dd7e95a4\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-g4cld" Nov 03 01:04:09.776380 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:09.776321 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2cef07ce-2ada-4d6d-93ba-0241dd7e95a4-istio-envoy\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-g4cld\" (UID: \"2cef07ce-2ada-4d6d-93ba-0241dd7e95a4\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-g4cld" Nov 03 01:04:09.776380 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:09.776369 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cef07ce-2ada-4d6d-93ba-0241dd7e95a4-workload-socket\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-g4cld\" (UID: \"2cef07ce-2ada-4d6d-93ba-0241dd7e95a4\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-g4cld" Nov 03 01:04:09.776456 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:09.776388 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2cef07ce-2ada-4d6d-93ba-0241dd7e95a4-istiod-ca-cert\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-g4cld\" (UID: \"2cef07ce-2ada-4d6d-93ba-0241dd7e95a4\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-g4cld" Nov 03 01:04:09.776456 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:09.776427 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2cef07ce-2ada-4d6d-93ba-0241dd7e95a4-istio-podinfo\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-g4cld\" (UID: \"2cef07ce-2ada-4d6d-93ba-0241dd7e95a4\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-g4cld" Nov 03 01:04:09.776611 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:09.776586 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-np9gf\" (UniqueName: \"kubernetes.io/projected/2cef07ce-2ada-4d6d-93ba-0241dd7e95a4-kube-api-access-np9gf\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-g4cld\" (UID: \"2cef07ce-2ada-4d6d-93ba-0241dd7e95a4\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-g4cld" Nov 03 01:04:09.877903 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:09.877871 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-np9gf\" (UniqueName: \"kubernetes.io/projected/2cef07ce-2ada-4d6d-93ba-0241dd7e95a4-kube-api-access-np9gf\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-g4cld\" (UID: \"2cef07ce-2ada-4d6d-93ba-0241dd7e95a4\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-g4cld" Nov 03 01:04:09.878111 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:09.877939 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cef07ce-2ada-4d6d-93ba-0241dd7e95a4-credential-socket\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-g4cld\" (UID: \"2cef07ce-2ada-4d6d-93ba-0241dd7e95a4\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-g4cld" Nov 03 01:04:09.878111 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:09.877984 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2cef07ce-2ada-4d6d-93ba-0241dd7e95a4-istio-data\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-g4cld\" (UID: \"2cef07ce-2ada-4d6d-93ba-0241dd7e95a4\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-g4cld" Nov 03 01:04:09.878111 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:09.878020 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2cef07ce-2ada-4d6d-93ba-0241dd7e95a4-istio-token\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-g4cld\" (UID: \"2cef07ce-2ada-4d6d-93ba-0241dd7e95a4\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-g4cld" Nov 03 01:04:09.878111 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:09.878048 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2cef07ce-2ada-4d6d-93ba-0241dd7e95a4-workload-certs\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-g4cld\" (UID: \"2cef07ce-2ada-4d6d-93ba-0241dd7e95a4\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-g4cld" Nov 03 01:04:09.878367 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:09.878114 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2cef07ce-2ada-4d6d-93ba-0241dd7e95a4-istio-envoy\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-g4cld\" (UID: \"2cef07ce-2ada-4d6d-93ba-0241dd7e95a4\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-g4cld" Nov 03 01:04:09.878367 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:09.878180 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cef07ce-2ada-4d6d-93ba-0241dd7e95a4-workload-socket\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-g4cld\" (UID: \"2cef07ce-2ada-4d6d-93ba-0241dd7e95a4\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-g4cld" Nov 03 01:04:09.878367 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:09.878205 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2cef07ce-2ada-4d6d-93ba-0241dd7e95a4-istiod-ca-cert\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-g4cld\" (UID: \"2cef07ce-2ada-4d6d-93ba-0241dd7e95a4\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-g4cld" Nov 03 01:04:09.878367 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:09.878256 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2cef07ce-2ada-4d6d-93ba-0241dd7e95a4-istio-podinfo\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-g4cld\" (UID: \"2cef07ce-2ada-4d6d-93ba-0241dd7e95a4\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-g4cld" Nov 03 01:04:09.878885 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:09.878849 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2cef07ce-2ada-4d6d-93ba-0241dd7e95a4-workload-certs\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-g4cld\" (UID: \"2cef07ce-2ada-4d6d-93ba-0241dd7e95a4\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-g4cld" Nov 03 01:04:09.879020 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:09.878425 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cef07ce-2ada-4d6d-93ba-0241dd7e95a4-credential-socket\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-g4cld\" (UID: \"2cef07ce-2ada-4d6d-93ba-0241dd7e95a4\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-g4cld" Nov 03 01:04:09.879079 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:09.879011 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2cef07ce-2ada-4d6d-93ba-0241dd7e95a4-istio-data\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-g4cld\" (UID: \"2cef07ce-2ada-4d6d-93ba-0241dd7e95a4\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-g4cld" Nov 03 01:04:09.880852 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:09.880826 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2cef07ce-2ada-4d6d-93ba-0241dd7e95a4-istiod-ca-cert\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-g4cld\" (UID: \"2cef07ce-2ada-4d6d-93ba-0241dd7e95a4\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-g4cld" Nov 03 01:04:09.881029 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:09.881012 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cef07ce-2ada-4d6d-93ba-0241dd7e95a4-workload-socket\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-g4cld\" (UID: \"2cef07ce-2ada-4d6d-93ba-0241dd7e95a4\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-g4cld" Nov 03 01:04:09.885199 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:09.881538 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2cef07ce-2ada-4d6d-93ba-0241dd7e95a4-istio-envoy\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-g4cld\" (UID: \"2cef07ce-2ada-4d6d-93ba-0241dd7e95a4\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-g4cld" Nov 03 01:04:09.885199 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:09.883730 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2cef07ce-2ada-4d6d-93ba-0241dd7e95a4-istio-podinfo\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-g4cld\" (UID: \"2cef07ce-2ada-4d6d-93ba-0241dd7e95a4\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-g4cld" Nov 03 01:04:09.887477 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:09.887450 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2cef07ce-2ada-4d6d-93ba-0241dd7e95a4-istio-token\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-g4cld\" (UID: \"2cef07ce-2ada-4d6d-93ba-0241dd7e95a4\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-g4cld" Nov 03 01:04:09.890152 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:09.890100 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-np9gf\" (UniqueName: \"kubernetes.io/projected/2cef07ce-2ada-4d6d-93ba-0241dd7e95a4-kube-api-access-np9gf\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-g4cld\" (UID: \"2cef07ce-2ada-4d6d-93ba-0241dd7e95a4\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-g4cld" Nov 03 01:04:09.959792 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:09.959757 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-g4cld" Nov 03 01:04:10.250415 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:10.250391 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-g4cld"] Nov 03 01:04:10.250904 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:04:10.250875 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cef07ce_2ada_4d6d_93ba_0241dd7e95a4.slice/crio-aa568a85dd45684564dd799cc85d05c975ea11110e85054785514682f3e7d619 WatchSource:0}: Error finding container aa568a85dd45684564dd799cc85d05c975ea11110e85054785514682f3e7d619: Status 404 returned error can't find the container with id aa568a85dd45684564dd799cc85d05c975ea11110e85054785514682f3e7d619 Nov 03 01:04:10.255215 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:10.255177 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:04:10.255322 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:10.255261 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:04:10.255383 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:10.255360 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:04:10.701870 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:10.701770 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-g4cld" event={"ID":"2cef07ce-2ada-4d6d-93ba-0241dd7e95a4","Type":"ContainerStarted","Data":"9741c9096d91d0e1cdaba0e6c7ce91be5e329a229144cda49463d44f4a65f862"} Nov 03 01:04:10.701870 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:10.701838 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-g4cld" event={"ID":"2cef07ce-2ada-4d6d-93ba-0241dd7e95a4","Type":"ContainerStarted","Data":"aa568a85dd45684564dd799cc85d05c975ea11110e85054785514682f3e7d619"} Nov 03 01:04:10.723492 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:10.723427 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-g4cld" podStartSLOduration=1.723408689 podStartE2EDuration="1.723408689s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:04:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:04:10.720914337 +0000 UTC m=+1392.891917435" watchObservedRunningTime="2026-11-03 01:04:10.723408689 +0000 UTC m=+1392.894411786" Nov 03 01:04:10.853502 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:10.853474 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1e2ac75-c9cfl" Nov 03 01:04:10.951340 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:10.927110 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff1448af-9knz8"] Nov 03 01:04:10.951340 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:10.928109 4421 cpu_manager.go:414] "containerMap: removing container" podUID="d053a07c-ed76-4510-98ea-1097312fcf38" containerName="revoke-keys" Nov 03 01:04:10.951340 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:10.928139 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d053a07c-ed76-4510-98ea-1097312fcf38" containerName="revoke-keys" Nov 03 01:04:10.951340 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:10.928399 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="d053a07c-ed76-4510-98ea-1097312fcf38" containerName="revoke-keys" Nov 03 01:04:10.951340 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:10.944317 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff1448af-9knz8"] Nov 03 01:04:10.951340 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:10.944448 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff1448af-9knz8" Nov 03 01:04:10.960915 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:10.960789 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-g4cld" Nov 03 01:04:11.039301 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:11.037808 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d053a07c-ed76-4510-98ea-1097312fcf38-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d053a07c-ed76-4510-98ea-1097312fcf38-maas-api-openshift-ca\") pod \"d053a07c-ed76-4510-98ea-1097312fcf38\" (UID: \"d053a07c-ed76-4510-98ea-1097312fcf38\") " Nov 03 01:04:11.039301 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:11.037942 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d053a07c-ed76-4510-98ea-1097312fcf38-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d053a07c-ed76-4510-98ea-1097312fcf38-maas-api-xks-ca\") pod \"d053a07c-ed76-4510-98ea-1097312fcf38\" (UID: \"d053a07c-ed76-4510-98ea-1097312fcf38\") " Nov 03 01:04:11.039301 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:11.038170 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d053a07c-ed76-4510-98ea-1097312fcf38-kube-api-access-bwxqb\" (UniqueName: \"kubernetes.io/projected/d053a07c-ed76-4510-98ea-1097312fcf38-kube-api-access-bwxqb\") pod \"d053a07c-ed76-4510-98ea-1097312fcf38\" (UID: \"d053a07c-ed76-4510-98ea-1097312fcf38\") " Nov 03 01:04:11.049875 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:11.049836 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d053a07c-ed76-4510-98ea-1097312fcf38-maas-api-openshift-ca" pod "d053a07c-ed76-4510-98ea-1097312fcf38" (UID: "d053a07c-ed76-4510-98ea-1097312fcf38"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:04:11.060340 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:11.054304 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d053a07c-ed76-4510-98ea-1097312fcf38-kube-api-access-bwxqb" pod "d053a07c-ed76-4510-98ea-1097312fcf38" (UID: "d053a07c-ed76-4510-98ea-1097312fcf38"). InnerVolumeSpecName "kube-api-access-bwxqb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:04:11.060340 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:11.060232 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d053a07c-ed76-4510-98ea-1097312fcf38-maas-api-xks-ca" pod "d053a07c-ed76-4510-98ea-1097312fcf38" (UID: "d053a07c-ed76-4510-98ea-1097312fcf38"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:04:11.140584 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:11.140545 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4zrx\" (UniqueName: \"kubernetes.io/projected/4367a12b-c4d5-47ef-a064-8f2c1930d7c3-kube-api-access-k4zrx\") pod \"maas-api-revoke-sub-ff1448af-9knz8\" (UID: \"4367a12b-c4d5-47ef-a064-8f2c1930d7c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff1448af-9knz8" Nov 03 01:04:11.140804 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:11.140703 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4367a12b-c4d5-47ef-a064-8f2c1930d7c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ff1448af-9knz8\" (UID: \"4367a12b-c4d5-47ef-a064-8f2c1930d7c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff1448af-9knz8" Nov 03 01:04:11.140804 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:11.140779 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4367a12b-c4d5-47ef-a064-8f2c1930d7c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ff1448af-9knz8\" (UID: \"4367a12b-c4d5-47ef-a064-8f2c1930d7c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff1448af-9knz8" Nov 03 01:04:11.140954 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:11.140941 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d053a07c-ed76-4510-98ea-1097312fcf38-maas-api-openshift-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:04:11.141007 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:11.140957 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d053a07c-ed76-4510-98ea-1097312fcf38-maas-api-xks-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:04:11.141007 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:11.140966 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bwxqb\" (UniqueName: \"kubernetes.io/projected/d053a07c-ed76-4510-98ea-1097312fcf38-kube-api-access-bwxqb\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:04:11.242236 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:11.242150 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4367a12b-c4d5-47ef-a064-8f2c1930d7c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ff1448af-9knz8\" (UID: \"4367a12b-c4d5-47ef-a064-8f2c1930d7c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff1448af-9knz8" Nov 03 01:04:11.242422 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:11.242241 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4367a12b-c4d5-47ef-a064-8f2c1930d7c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ff1448af-9knz8\" (UID: \"4367a12b-c4d5-47ef-a064-8f2c1930d7c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff1448af-9knz8" Nov 03 01:04:11.242482 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:11.242423 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4zrx\" (UniqueName: \"kubernetes.io/projected/4367a12b-c4d5-47ef-a064-8f2c1930d7c3-kube-api-access-k4zrx\") pod \"maas-api-revoke-sub-ff1448af-9knz8\" (UID: \"4367a12b-c4d5-47ef-a064-8f2c1930d7c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff1448af-9knz8" Nov 03 01:04:11.243086 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:11.243061 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4367a12b-c4d5-47ef-a064-8f2c1930d7c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ff1448af-9knz8\" (UID: \"4367a12b-c4d5-47ef-a064-8f2c1930d7c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff1448af-9knz8" Nov 03 01:04:11.244594 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:11.244572 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4367a12b-c4d5-47ef-a064-8f2c1930d7c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ff1448af-9knz8\" (UID: \"4367a12b-c4d5-47ef-a064-8f2c1930d7c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff1448af-9knz8" Nov 03 01:04:11.250922 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:11.250902 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4zrx\" (UniqueName: \"kubernetes.io/projected/4367a12b-c4d5-47ef-a064-8f2c1930d7c3-kube-api-access-k4zrx\") pod \"maas-api-revoke-sub-ff1448af-9knz8\" (UID: \"4367a12b-c4d5-47ef-a064-8f2c1930d7c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff1448af-9knz8" Nov 03 01:04:11.294444 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:11.294415 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff1448af-9knz8" Nov 03 01:04:11.336322 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:11.335609 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8308a673-mzz2s"] Nov 03 01:04:11.346664 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:11.346630 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8308a673-mzz2s"] Nov 03 01:04:11.346820 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:11.346759 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8308a673-mzz2s" Nov 03 01:04:11.445992 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:11.445836 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68f88e39-b2ce-4ba5-b235-caf9bdaff12f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8308a673-mzz2s\" (UID: \"68f88e39-b2ce-4ba5-b235-caf9bdaff12f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8308a673-mzz2s" Nov 03 01:04:11.446495 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:11.446471 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68f88e39-b2ce-4ba5-b235-caf9bdaff12f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8308a673-mzz2s\" (UID: \"68f88e39-b2ce-4ba5-b235-caf9bdaff12f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8308a673-mzz2s" Nov 03 01:04:11.446808 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:11.446787 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nq6j7\" (UniqueName: \"kubernetes.io/projected/68f88e39-b2ce-4ba5-b235-caf9bdaff12f-kube-api-access-nq6j7\") pod \"maas-api-revoke-sub-8308a673-mzz2s\" (UID: \"68f88e39-b2ce-4ba5-b235-caf9bdaff12f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8308a673-mzz2s" Nov 03 01:04:11.547756 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:11.547724 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nq6j7\" (UniqueName: \"kubernetes.io/projected/68f88e39-b2ce-4ba5-b235-caf9bdaff12f-kube-api-access-nq6j7\") pod \"maas-api-revoke-sub-8308a673-mzz2s\" (UID: \"68f88e39-b2ce-4ba5-b235-caf9bdaff12f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8308a673-mzz2s" Nov 03 01:04:11.547961 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:11.547805 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68f88e39-b2ce-4ba5-b235-caf9bdaff12f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8308a673-mzz2s\" (UID: \"68f88e39-b2ce-4ba5-b235-caf9bdaff12f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8308a673-mzz2s" Nov 03 01:04:11.548043 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:11.548014 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68f88e39-b2ce-4ba5-b235-caf9bdaff12f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8308a673-mzz2s\" (UID: \"68f88e39-b2ce-4ba5-b235-caf9bdaff12f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8308a673-mzz2s" Nov 03 01:04:11.548660 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:11.548637 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68f88e39-b2ce-4ba5-b235-caf9bdaff12f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8308a673-mzz2s\" (UID: \"68f88e39-b2ce-4ba5-b235-caf9bdaff12f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8308a673-mzz2s" Nov 03 01:04:11.550198 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:11.550178 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68f88e39-b2ce-4ba5-b235-caf9bdaff12f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8308a673-mzz2s\" (UID: \"68f88e39-b2ce-4ba5-b235-caf9bdaff12f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8308a673-mzz2s" Nov 03 01:04:11.559319 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:11.559295 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nq6j7\" (UniqueName: \"kubernetes.io/projected/68f88e39-b2ce-4ba5-b235-caf9bdaff12f-kube-api-access-nq6j7\") pod \"maas-api-revoke-sub-8308a673-mzz2s\" (UID: \"68f88e39-b2ce-4ba5-b235-caf9bdaff12f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8308a673-mzz2s" Nov 03 01:04:11.636002 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:04:11.635960 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4367a12b_c4d5_47ef_a064_8f2c1930d7c3.slice/crio-41143db10fe8a2906c84f60737f1b1ca0f53eb86f5d9dedaa0819e2fff941001 WatchSource:0}: Error finding container 41143db10fe8a2906c84f60737f1b1ca0f53eb86f5d9dedaa0819e2fff941001: Status 404 returned error can't find the container with id 41143db10fe8a2906c84f60737f1b1ca0f53eb86f5d9dedaa0819e2fff941001 Nov 03 01:04:11.641081 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:11.641057 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff1448af-9knz8"] Nov 03 01:04:11.665491 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:11.665462 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8308a673-mzz2s" Nov 03 01:04:11.737812 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:11.737774 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff1448af-9knz8" event={"ID":"4367a12b-c4d5-47ef-a064-8f2c1930d7c3","Type":"ContainerStarted","Data":"41143db10fe8a2906c84f60737f1b1ca0f53eb86f5d9dedaa0819e2fff941001"} Nov 03 01:04:11.740609 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:11.740579 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1e2ac75-c9cfl" Nov 03 01:04:11.744936 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:11.744900 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1e2ac75-c9cfl" event={"ID":"d053a07c-ed76-4510-98ea-1097312fcf38","Type":"ContainerDied","Data":"a94bf39044e8580dab6581b46fc0f9ab04ef2286aa2707ec6c08a3a11688736f"} Nov 03 01:04:11.745028 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:11.744973 4421 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a94bf39044e8580dab6581b46fc0f9ab04ef2286aa2707ec6c08a3a11688736f" Nov 03 01:04:11.967832 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:11.965758 4421 patch_prober.go:29] interesting pod/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-g4cld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" start-of-body= Nov 03 01:04:11.967832 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:11.965803 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-g4cld" podUID="2cef07ce-2ada-4d6d-93ba-0241dd7e95a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" Nov 03 01:04:11.989238 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:11.989202 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8308a673-mzz2s"] Nov 03 01:04:11.994177 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:04:11.994149 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68f88e39_b2ce_4ba5_b235_caf9bdaff12f.slice/crio-c03cfbb33f2a4ff262c2210136e5e3b10436dc955529abf0d5359d36aed33a5d WatchSource:0}: Error finding container c03cfbb33f2a4ff262c2210136e5e3b10436dc955529abf0d5359d36aed33a5d: Status 404 returned error can't find the container with id c03cfbb33f2a4ff262c2210136e5e3b10436dc955529abf0d5359d36aed33a5d Nov 03 01:04:12.748631 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:12.748599 4421 generic.go:356] "Generic (PLEG): container finished" podID="4367a12b-c4d5-47ef-a064-8f2c1930d7c3" containerID="3c9a423bdb5d8887e7ad9e46d5be762a8a11de2cebd9620a1f4e5b3c7076291d" exitCode=0 Nov 03 01:04:12.749058 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:12.748682 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff1448af-9knz8" event={"ID":"4367a12b-c4d5-47ef-a064-8f2c1930d7c3","Type":"ContainerDied","Data":"3c9a423bdb5d8887e7ad9e46d5be762a8a11de2cebd9620a1f4e5b3c7076291d"} Nov 03 01:04:12.750490 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:12.750465 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8308a673-mzz2s" event={"ID":"68f88e39-b2ce-4ba5-b235-caf9bdaff12f","Type":"ContainerStarted","Data":"08328ecc38835b3ce804edab79c0bc73f422eab44865c2b2408bf4112800c917"} Nov 03 01:04:12.750607 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:12.750522 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8308a673-mzz2s" event={"ID":"68f88e39-b2ce-4ba5-b235-caf9bdaff12f","Type":"ContainerStarted","Data":"c03cfbb33f2a4ff262c2210136e5e3b10436dc955529abf0d5359d36aed33a5d"} Nov 03 01:04:12.783723 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:12.783668 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8308a673-mzz2s" podStartSLOduration=1.783653353 podStartE2EDuration="1.783653353s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:04:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:04:12.779421744 +0000 UTC m=+1394.950424841" watchObservedRunningTime="2026-11-03 01:04:12.783653353 +0000 UTC m=+1394.954656535" Nov 03 01:04:12.960192 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:12.960160 4421 patch_prober.go:29] interesting pod/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-g4cld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" start-of-body= Nov 03 01:04:12.960403 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:12.960229 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-g4cld" podUID="2cef07ce-2ada-4d6d-93ba-0241dd7e95a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" Nov 03 01:04:13.759865 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:13.759830 4421 generic.go:356] "Generic (PLEG): container finished" podID="68f88e39-b2ce-4ba5-b235-caf9bdaff12f" containerID="08328ecc38835b3ce804edab79c0bc73f422eab44865c2b2408bf4112800c917" exitCode=0 Nov 03 01:04:13.760322 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:13.759917 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8308a673-mzz2s" event={"ID":"68f88e39-b2ce-4ba5-b235-caf9bdaff12f","Type":"ContainerDied","Data":"08328ecc38835b3ce804edab79c0bc73f422eab44865c2b2408bf4112800c917"} Nov 03 01:04:13.960600 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:13.960568 4421 patch_prober.go:29] interesting pod/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-g4cld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" start-of-body= Nov 03 01:04:13.960810 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:13.960786 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-g4cld" podUID="2cef07ce-2ada-4d6d-93ba-0241dd7e95a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" Nov 03 01:04:14.003316 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:14.003263 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff1448af-9knz8" Nov 03 01:04:14.102336 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:14.102221 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4367a12b-c4d5-47ef-a064-8f2c1930d7c3-kube-api-access-k4zrx\" (UniqueName: \"kubernetes.io/projected/4367a12b-c4d5-47ef-a064-8f2c1930d7c3-kube-api-access-k4zrx\") pod \"4367a12b-c4d5-47ef-a064-8f2c1930d7c3\" (UID: \"4367a12b-c4d5-47ef-a064-8f2c1930d7c3\") " Nov 03 01:04:14.102336 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:14.102298 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4367a12b-c4d5-47ef-a064-8f2c1930d7c3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4367a12b-c4d5-47ef-a064-8f2c1930d7c3-maas-api-openshift-ca\") pod \"4367a12b-c4d5-47ef-a064-8f2c1930d7c3\" (UID: \"4367a12b-c4d5-47ef-a064-8f2c1930d7c3\") " Nov 03 01:04:14.102336 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:14.102335 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4367a12b-c4d5-47ef-a064-8f2c1930d7c3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4367a12b-c4d5-47ef-a064-8f2c1930d7c3-maas-api-xks-ca\") pod \"4367a12b-c4d5-47ef-a064-8f2c1930d7c3\" (UID: \"4367a12b-c4d5-47ef-a064-8f2c1930d7c3\") " Nov 03 01:04:14.102758 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:14.102729 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4367a12b-c4d5-47ef-a064-8f2c1930d7c3-maas-api-openshift-ca" pod "4367a12b-c4d5-47ef-a064-8f2c1930d7c3" (UID: "4367a12b-c4d5-47ef-a064-8f2c1930d7c3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:04:14.102986 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:14.102968 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4367a12b-c4d5-47ef-a064-8f2c1930d7c3-maas-api-openshift-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:04:14.104969 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:14.104939 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4367a12b-c4d5-47ef-a064-8f2c1930d7c3-maas-api-xks-ca" pod "4367a12b-c4d5-47ef-a064-8f2c1930d7c3" (UID: "4367a12b-c4d5-47ef-a064-8f2c1930d7c3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:04:14.127298 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:14.105368 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4367a12b-c4d5-47ef-a064-8f2c1930d7c3-kube-api-access-k4zrx" pod "4367a12b-c4d5-47ef-a064-8f2c1930d7c3" (UID: "4367a12b-c4d5-47ef-a064-8f2c1930d7c3"). InnerVolumeSpecName "kube-api-access-k4zrx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:04:14.203931 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:14.203898 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k4zrx\" (UniqueName: \"kubernetes.io/projected/4367a12b-c4d5-47ef-a064-8f2c1930d7c3-kube-api-access-k4zrx\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:04:14.203931 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:14.203929 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4367a12b-c4d5-47ef-a064-8f2c1930d7c3-maas-api-xks-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:04:14.675307 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:04:14.675238 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 01:04:14.768650 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:14.768622 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff1448af-9knz8" Nov 03 01:04:14.769044 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:14.768647 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff1448af-9knz8" event={"ID":"4367a12b-c4d5-47ef-a064-8f2c1930d7c3","Type":"ContainerDied","Data":"41143db10fe8a2906c84f60737f1b1ca0f53eb86f5d9dedaa0819e2fff941001"} Nov 03 01:04:14.769044 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:14.768697 4421 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="41143db10fe8a2906c84f60737f1b1ca0f53eb86f5d9dedaa0819e2fff941001" Nov 03 01:04:14.960939 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:14.960907 4421 patch_prober.go:29] interesting pod/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-g4cld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" start-of-body= Nov 03 01:04:14.961082 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:14.960963 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-g4cld" podUID="2cef07ce-2ada-4d6d-93ba-0241dd7e95a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" Nov 03 01:04:14.971079 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:14.971052 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8308a673-mzz2s" Nov 03 01:04:15.038156 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:15.038126 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/68f88e39-b2ce-4ba5-b235-caf9bdaff12f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68f88e39-b2ce-4ba5-b235-caf9bdaff12f-maas-api-xks-ca\") pod \"68f88e39-b2ce-4ba5-b235-caf9bdaff12f\" (UID: \"68f88e39-b2ce-4ba5-b235-caf9bdaff12f\") " Nov 03 01:04:15.038416 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:15.038397 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68f88e39-b2ce-4ba5-b235-caf9bdaff12f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68f88e39-b2ce-4ba5-b235-caf9bdaff12f-maas-api-openshift-ca\") pod \"68f88e39-b2ce-4ba5-b235-caf9bdaff12f\" (UID: \"68f88e39-b2ce-4ba5-b235-caf9bdaff12f\") " Nov 03 01:04:15.038515 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:15.038502 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68f88e39-b2ce-4ba5-b235-caf9bdaff12f-kube-api-access-nq6j7\" (UniqueName: \"kubernetes.io/projected/68f88e39-b2ce-4ba5-b235-caf9bdaff12f-kube-api-access-nq6j7\") pod \"68f88e39-b2ce-4ba5-b235-caf9bdaff12f\" (UID: \"68f88e39-b2ce-4ba5-b235-caf9bdaff12f\") " Nov 03 01:04:15.039954 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:15.039925 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68f88e39-b2ce-4ba5-b235-caf9bdaff12f-maas-api-openshift-ca" pod "68f88e39-b2ce-4ba5-b235-caf9bdaff12f" (UID: "68f88e39-b2ce-4ba5-b235-caf9bdaff12f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:04:15.124421 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:15.103056 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68f88e39-b2ce-4ba5-b235-caf9bdaff12f-maas-api-xks-ca" pod "68f88e39-b2ce-4ba5-b235-caf9bdaff12f" (UID: "68f88e39-b2ce-4ba5-b235-caf9bdaff12f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:04:15.158118 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:15.134136 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68f88e39-b2ce-4ba5-b235-caf9bdaff12f-kube-api-access-nq6j7" pod "68f88e39-b2ce-4ba5-b235-caf9bdaff12f" (UID: "68f88e39-b2ce-4ba5-b235-caf9bdaff12f"). InnerVolumeSpecName "kube-api-access-nq6j7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:04:15.158118 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:15.156234 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68f88e39-b2ce-4ba5-b235-caf9bdaff12f-maas-api-openshift-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:04:15.158118 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:15.156264 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nq6j7\" (UniqueName: \"kubernetes.io/projected/68f88e39-b2ce-4ba5-b235-caf9bdaff12f-kube-api-access-nq6j7\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:04:15.158118 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:15.156300 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68f88e39-b2ce-4ba5-b235-caf9bdaff12f-maas-api-xks-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:04:15.776878 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:15.776845 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8308a673-mzz2s" Nov 03 01:04:15.777388 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:15.776888 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8308a673-mzz2s" event={"ID":"68f88e39-b2ce-4ba5-b235-caf9bdaff12f","Type":"ContainerDied","Data":"c03cfbb33f2a4ff262c2210136e5e3b10436dc955529abf0d5359d36aed33a5d"} Nov 03 01:04:15.777388 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:15.776933 4421 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c03cfbb33f2a4ff262c2210136e5e3b10436dc955529abf0d5359d36aed33a5d" Nov 03 01:04:15.962299 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:15.960969 4421 patch_prober.go:29] interesting pod/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-g4cld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" start-of-body= Nov 03 01:04:15.962299 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:15.961029 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-g4cld" podUID="2cef07ce-2ada-4d6d-93ba-0241dd7e95a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" Nov 03 01:04:16.961073 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:16.961025 4421 patch_prober.go:29] interesting pod/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-g4cld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" start-of-body= Nov 03 01:04:16.961522 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:16.961104 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-g4cld" podUID="2cef07ce-2ada-4d6d-93ba-0241dd7e95a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" Nov 03 01:04:17.965381 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:17.965347 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-g4cld" Nov 03 01:04:17.965877 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:17.965719 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-g4cld" Nov 03 01:04:17.966373 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:17.966351 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-g4cld" Nov 03 01:04:22.320674 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:22.320627 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52aa8264-p5dtw"] Nov 03 01:04:22.322390 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:22.322352 4421 cpu_manager.go:414] "containerMap: removing container" podUID="4367a12b-c4d5-47ef-a064-8f2c1930d7c3" containerName="revoke-keys" Nov 03 01:04:22.322663 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:22.322636 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4367a12b-c4d5-47ef-a064-8f2c1930d7c3" containerName="revoke-keys" Nov 03 01:04:22.322782 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:22.322714 4421 cpu_manager.go:414] "containerMap: removing container" podUID="68f88e39-b2ce-4ba5-b235-caf9bdaff12f" containerName="revoke-keys" Nov 03 01:04:22.322782 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:22.322725 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68f88e39-b2ce-4ba5-b235-caf9bdaff12f" containerName="revoke-keys" Nov 03 01:04:22.323017 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:22.322997 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="4367a12b-c4d5-47ef-a064-8f2c1930d7c3" containerName="revoke-keys" Nov 03 01:04:22.323082 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:22.323035 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="68f88e39-b2ce-4ba5-b235-caf9bdaff12f" containerName="revoke-keys" Nov 03 01:04:22.338041 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:22.337993 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52aa8264-p5dtw"] Nov 03 01:04:22.338301 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:22.338162 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52aa8264-p5dtw" Nov 03 01:04:22.343213 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:22.343164 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:04:22.343439 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:22.343395 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6q4xp\"" Nov 03 01:04:22.343618 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:22.343590 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 01:04:22.343746 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:22.343721 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 01:04:22.492697 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:22.492659 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6586d098-8183-4430-99b7-1b5e0973b0ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-52aa8264-p5dtw\" (UID: \"6586d098-8183-4430-99b7-1b5e0973b0ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52aa8264-p5dtw" Nov 03 01:04:22.492884 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:22.492808 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6586d098-8183-4430-99b7-1b5e0973b0ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-52aa8264-p5dtw\" (UID: \"6586d098-8183-4430-99b7-1b5e0973b0ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52aa8264-p5dtw" Nov 03 01:04:22.492884 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:22.492855 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4chmr\" (UniqueName: \"kubernetes.io/projected/6586d098-8183-4430-99b7-1b5e0973b0ab-kube-api-access-4chmr\") pod \"maas-api-revoke-sub-52aa8264-p5dtw\" (UID: \"6586d098-8183-4430-99b7-1b5e0973b0ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52aa8264-p5dtw" Nov 03 01:04:22.594832 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:22.594751 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6586d098-8183-4430-99b7-1b5e0973b0ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-52aa8264-p5dtw\" (UID: \"6586d098-8183-4430-99b7-1b5e0973b0ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52aa8264-p5dtw" Nov 03 01:04:22.594979 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:22.594857 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6586d098-8183-4430-99b7-1b5e0973b0ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-52aa8264-p5dtw\" (UID: \"6586d098-8183-4430-99b7-1b5e0973b0ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52aa8264-p5dtw" Nov 03 01:04:22.594979 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:22.594894 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4chmr\" (UniqueName: \"kubernetes.io/projected/6586d098-8183-4430-99b7-1b5e0973b0ab-kube-api-access-4chmr\") pod \"maas-api-revoke-sub-52aa8264-p5dtw\" (UID: \"6586d098-8183-4430-99b7-1b5e0973b0ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52aa8264-p5dtw" Nov 03 01:04:22.595608 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:22.595556 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6586d098-8183-4430-99b7-1b5e0973b0ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-52aa8264-p5dtw\" (UID: \"6586d098-8183-4430-99b7-1b5e0973b0ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52aa8264-p5dtw" Nov 03 01:04:22.597228 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:22.597203 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6586d098-8183-4430-99b7-1b5e0973b0ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-52aa8264-p5dtw\" (UID: \"6586d098-8183-4430-99b7-1b5e0973b0ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52aa8264-p5dtw" Nov 03 01:04:22.602706 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:22.602682 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4chmr\" (UniqueName: \"kubernetes.io/projected/6586d098-8183-4430-99b7-1b5e0973b0ab-kube-api-access-4chmr\") pod \"maas-api-revoke-sub-52aa8264-p5dtw\" (UID: \"6586d098-8183-4430-99b7-1b5e0973b0ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52aa8264-p5dtw" Nov 03 01:04:22.673044 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:22.673007 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52aa8264-p5dtw" Nov 03 01:04:22.784464 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:22.784428 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17a596cf-nb48g"] Nov 03 01:04:22.789496 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:22.789468 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17a596cf-nb48g" Nov 03 01:04:22.796478 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:22.796447 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17a596cf-nb48g"] Nov 03 01:04:22.898610 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:22.898517 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q54pl\" (UniqueName: \"kubernetes.io/projected/50854d7f-ce7a-47eb-9eef-358b178c0dd9-kube-api-access-q54pl\") pod \"maas-api-revoke-sub-17a596cf-nb48g\" (UID: \"50854d7f-ce7a-47eb-9eef-358b178c0dd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17a596cf-nb48g" Nov 03 01:04:22.898767 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:22.898721 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50854d7f-ce7a-47eb-9eef-358b178c0dd9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-17a596cf-nb48g\" (UID: \"50854d7f-ce7a-47eb-9eef-358b178c0dd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17a596cf-nb48g" Nov 03 01:04:22.898767 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:22.898750 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50854d7f-ce7a-47eb-9eef-358b178c0dd9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-17a596cf-nb48g\" (UID: \"50854d7f-ce7a-47eb-9eef-358b178c0dd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17a596cf-nb48g" Nov 03 01:04:22.977327 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:22.977290 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52aa8264-p5dtw"] Nov 03 01:04:22.983933 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:04:22.983890 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6586d098_8183_4430_99b7_1b5e0973b0ab.slice/crio-19ff96f80c3c2b4de9ed06c11f9c02f170db75156ca5d9830a1deea0380de931 WatchSource:0}: Error finding container 19ff96f80c3c2b4de9ed06c11f9c02f170db75156ca5d9830a1deea0380de931: Status 404 returned error can't find the container with id 19ff96f80c3c2b4de9ed06c11f9c02f170db75156ca5d9830a1deea0380de931 Nov 03 01:04:23.000304 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:23.000264 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q54pl\" (UniqueName: \"kubernetes.io/projected/50854d7f-ce7a-47eb-9eef-358b178c0dd9-kube-api-access-q54pl\") pod \"maas-api-revoke-sub-17a596cf-nb48g\" (UID: \"50854d7f-ce7a-47eb-9eef-358b178c0dd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17a596cf-nb48g" Nov 03 01:04:23.000439 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:23.000400 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50854d7f-ce7a-47eb-9eef-358b178c0dd9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-17a596cf-nb48g\" (UID: \"50854d7f-ce7a-47eb-9eef-358b178c0dd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17a596cf-nb48g" Nov 03 01:04:23.000439 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:23.000427 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50854d7f-ce7a-47eb-9eef-358b178c0dd9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-17a596cf-nb48g\" (UID: \"50854d7f-ce7a-47eb-9eef-358b178c0dd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17a596cf-nb48g" Nov 03 01:04:23.001105 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:23.001081 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50854d7f-ce7a-47eb-9eef-358b178c0dd9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-17a596cf-nb48g\" (UID: \"50854d7f-ce7a-47eb-9eef-358b178c0dd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17a596cf-nb48g" Nov 03 01:04:23.002920 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:23.002897 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50854d7f-ce7a-47eb-9eef-358b178c0dd9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-17a596cf-nb48g\" (UID: \"50854d7f-ce7a-47eb-9eef-358b178c0dd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17a596cf-nb48g" Nov 03 01:04:23.014560 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:23.014525 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q54pl\" (UniqueName: \"kubernetes.io/projected/50854d7f-ce7a-47eb-9eef-358b178c0dd9-kube-api-access-q54pl\") pod \"maas-api-revoke-sub-17a596cf-nb48g\" (UID: \"50854d7f-ce7a-47eb-9eef-358b178c0dd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17a596cf-nb48g" Nov 03 01:04:23.112129 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:23.112096 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17a596cf-nb48g" Nov 03 01:04:23.549367 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:23.549325 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17a596cf-nb48g"] Nov 03 01:04:23.868499 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:23.868397 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17a596cf-nb48g" event={"ID":"50854d7f-ce7a-47eb-9eef-358b178c0dd9","Type":"ContainerStarted","Data":"f8b1ff6a962a02925c12d2a9ea1eb0f84d828112a18dc0951246e6e31fbb0674"} Nov 03 01:04:23.871074 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:23.871023 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52aa8264-p5dtw" event={"ID":"6586d098-8183-4430-99b7-1b5e0973b0ab","Type":"ContainerStarted","Data":"5a900ba3e20c0dd7ad4283e577eac7f3a750d775012fb27cb02f11d746bf83d0"} Nov 03 01:04:23.871359 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:23.871334 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52aa8264-p5dtw" event={"ID":"6586d098-8183-4430-99b7-1b5e0973b0ab","Type":"ContainerStarted","Data":"19ff96f80c3c2b4de9ed06c11f9c02f170db75156ca5d9830a1deea0380de931"} Nov 03 01:04:23.906315 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:23.894629 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52aa8264-p5dtw" podStartSLOduration=1.894609772 podStartE2EDuration="1.894609772s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:04:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:04:23.889685371 +0000 UTC m=+1406.060688480" watchObservedRunningTime="2026-11-03 01:04:23.894609772 +0000 UTC m=+1406.065612872" Nov 03 01:04:25.280390 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:25.271862 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-650be0/simulator-w4-650be0-kserve-7bc588774f-gvz2k" podUID="a827fa88-f078-4312-8f12-2c799c11b461" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Nov 03 01:04:25.280390 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:25.274702 4421 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8gm8n" podUID="2d367e12-e46f-4a68-9825-9b1c934d4d12" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:04:25.351677 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:25.351624 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-650be0/premium-w4-650be0-kserve-7bf8555dfb-z97n5" podUID="40601dc5-808f-456f-826f-6e0c35966ada" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:04:25.468400 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:25.443899 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8gm8n" podUID="2d367e12-e46f-4a68-9825-9b1c934d4d12" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:04:25.468400 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:25.444221 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/simulator-w3-83755a-kserve-77d6664559-psqtg" podUID="b0a4c234-af58-4da0-a1a4-e1f167953f6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:04:25.535348 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:25.480693 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/simulator-w2-f7680d-kserve-85bdc86cd6-8gxrg" podUID="2bdcf521-cebd-4d6b-8554-1b60db8b0f3c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:04:25.569621 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:25.568923 4421 generic.go:356] "Generic (PLEG): container finished" podID="6586d098-8183-4430-99b7-1b5e0973b0ab" containerID="5a900ba3e20c0dd7ad4283e577eac7f3a750d775012fb27cb02f11d746bf83d0" exitCode=0 Nov 03 01:04:25.569621 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:25.569148 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52aa8264-p5dtw" event={"ID":"6586d098-8183-4430-99b7-1b5e0973b0ab","Type":"ContainerDied","Data":"5a900ba3e20c0dd7ad4283e577eac7f3a750d775012fb27cb02f11d746bf83d0"} Nov 03 01:04:25.601463 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:25.594880 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17a596cf-nb48g" event={"ID":"50854d7f-ce7a-47eb-9eef-358b178c0dd9","Type":"ContainerStarted","Data":"6ed0341f3cd8b54982948ca9648afdb1c68aab15a02b6fbbe2a35cfa0a0a23b5"} Nov 03 01:04:25.867836 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:25.867736 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/premium-w3-83755a-kserve-7c9d66988b-lt4xd" podUID="c0c6b02e-2ecb-428e-a13d-be6d7cd1ba39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 01:04:26.296510 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:26.277863 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/distinct-w3-83755a-kserve-86bdbbdd85-2cnnl" podUID="f4ae9624-d127-4a55-837d-7c9ff0267b37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:04:26.298135 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:26.298102 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-650be0/simulator-w4-650be0-kserve-7bc588774f-gvz2k" podUID="a827fa88-f078-4312-8f12-2c799c11b461" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 01:04:26.365390 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:26.354810 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-650be0/premium-w4-650be0-kserve-7bf8555dfb-z97n5" podUID="40601dc5-808f-456f-826f-6e0c35966ada" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Nov 03 01:04:26.366808 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:26.366773 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-650be0/simulator-w4-650be0-kserve-7bc588774f-gvz2k" Nov 03 01:04:26.367090 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:26.367061 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-650be0/premium-w4-650be0-kserve-7bf8555dfb-z97n5" Nov 03 01:04:26.513299 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:26.444189 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8gm8n" podUID="2d367e12-e46f-4a68-9825-9b1c934d4d12" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:04:26.513299 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:26.453487 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8gm8n" Nov 03 01:04:26.513299 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:26.454478 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/simulator-w3-83755a-kserve-77d6664559-psqtg" podUID="b0a4c234-af58-4da0-a1a4-e1f167953f6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Nov 03 01:04:26.513299 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:26.455012 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-83755a/simulator-w3-83755a-kserve-77d6664559-psqtg" Nov 03 01:04:26.545451 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:26.545403 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/simulator-w2-f7680d-kserve-85bdc86cd6-8gxrg" podUID="2bdcf521-cebd-4d6b-8554-1b60db8b0f3c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:04:26.587351 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:26.551407 4421 patch_prober.go:29] interesting pod/e2e-worker-w3-83755a-openshift-default-c86f9565c-pm4c4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 01:04:26.639346 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:26.613072 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-pm4c4" podUID="af8d7fc5-bed0-4ebf-be1c-44805efec25a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 01:04:26.639346 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:26.608213 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f7680d/simulator-w2-f7680d-kserve-85bdc86cd6-8gxrg" Nov 03 01:04:26.639346 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:26.613013 4421 patch_prober.go:29] interesting pod/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-bktsz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 01:04:26.639346 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:26.613508 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-bktsz" podUID="85cec606-e8b7-4ce1-8af2-4afb303268f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 01:04:26.869471 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:26.869376 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/premium-w3-83755a-kserve-7c9d66988b-lt4xd" podUID="c0c6b02e-2ecb-428e-a13d-be6d7cd1ba39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Nov 03 01:04:26.870229 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:26.870203 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-83755a/premium-w3-83755a-kserve-7c9d66988b-lt4xd" Nov 03 01:04:26.955389 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:26.955358 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-83755a/simulator-w3-83755a-kserve-77d6664559-psqtg" Nov 03 01:04:26.959164 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:26.959140 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8gm8n" Nov 03 01:04:26.977380 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:26.977354 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-650be0/simulator-w4-650be0-kserve-7bc588774f-gvz2k" Nov 03 01:04:26.993078 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:26.990715 4421 generic.go:356] "Generic (PLEG): container finished" podID="50854d7f-ce7a-47eb-9eef-358b178c0dd9" containerID="6ed0341f3cd8b54982948ca9648afdb1c68aab15a02b6fbbe2a35cfa0a0a23b5" exitCode=0 Nov 03 01:04:26.993078 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:26.991475 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17a596cf-nb48g" event={"ID":"50854d7f-ce7a-47eb-9eef-358b178c0dd9","Type":"ContainerDied","Data":"6ed0341f3cd8b54982948ca9648afdb1c68aab15a02b6fbbe2a35cfa0a0a23b5"} Nov 03 01:04:27.027296 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:27.017285 4421 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-83755a/distinct-w3-83755a-kserve-86bdbbdd85-2cnnl" podUID="f4ae9624-d127-4a55-837d-7c9ff0267b37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/health\": context deadline exceeded" Nov 03 01:04:27.052961 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:27.052924 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-83755a/premium-w3-83755a-kserve-7c9d66988b-lt4xd" Nov 03 01:04:27.414651 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:27.414617 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f7680d/simulator-w2-f7680d-kserve-85bdc86cd6-8gxrg" Nov 03 01:04:27.438698 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:27.438634 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-650be0/premium-w4-650be0-kserve-7bf8555dfb-z97n5" podUID="40601dc5-808f-456f-826f-6e0c35966ada" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 01:04:27.525380 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:27.525340 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-650be0/premium-w4-650be0-kserve-7bf8555dfb-z97n5" Nov 03 01:04:27.819186 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:27.818941 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17a596cf-nb48g" Nov 03 01:04:27.822764 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:27.822734 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52aa8264-p5dtw" Nov 03 01:04:27.927730 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:27.927692 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50854d7f-ce7a-47eb-9eef-358b178c0dd9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50854d7f-ce7a-47eb-9eef-358b178c0dd9-maas-api-openshift-ca\") pod \"50854d7f-ce7a-47eb-9eef-358b178c0dd9\" (UID: \"50854d7f-ce7a-47eb-9eef-358b178c0dd9\") " Nov 03 01:04:27.927909 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:27.927753 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6586d098-8183-4430-99b7-1b5e0973b0ab-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6586d098-8183-4430-99b7-1b5e0973b0ab-maas-api-openshift-ca\") pod \"6586d098-8183-4430-99b7-1b5e0973b0ab\" (UID: \"6586d098-8183-4430-99b7-1b5e0973b0ab\") " Nov 03 01:04:27.927909 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:27.927793 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50854d7f-ce7a-47eb-9eef-358b178c0dd9-kube-api-access-q54pl\" (UniqueName: \"kubernetes.io/projected/50854d7f-ce7a-47eb-9eef-358b178c0dd9-kube-api-access-q54pl\") pod \"50854d7f-ce7a-47eb-9eef-358b178c0dd9\" (UID: \"50854d7f-ce7a-47eb-9eef-358b178c0dd9\") " Nov 03 01:04:27.928054 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:27.927959 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/50854d7f-ce7a-47eb-9eef-358b178c0dd9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50854d7f-ce7a-47eb-9eef-358b178c0dd9-maas-api-xks-ca\") pod \"50854d7f-ce7a-47eb-9eef-358b178c0dd9\" (UID: \"50854d7f-ce7a-47eb-9eef-358b178c0dd9\") " Nov 03 01:04:27.928054 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:27.928023 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6586d098-8183-4430-99b7-1b5e0973b0ab-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6586d098-8183-4430-99b7-1b5e0973b0ab-maas-api-xks-ca\") pod \"6586d098-8183-4430-99b7-1b5e0973b0ab\" (UID: \"6586d098-8183-4430-99b7-1b5e0973b0ab\") " Nov 03 01:04:27.928176 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:27.928123 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6586d098-8183-4430-99b7-1b5e0973b0ab-kube-api-access-4chmr\" (UniqueName: \"kubernetes.io/projected/6586d098-8183-4430-99b7-1b5e0973b0ab-kube-api-access-4chmr\") pod \"6586d098-8183-4430-99b7-1b5e0973b0ab\" (UID: \"6586d098-8183-4430-99b7-1b5e0973b0ab\") " Nov 03 01:04:27.928238 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:27.928223 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6586d098-8183-4430-99b7-1b5e0973b0ab-maas-api-openshift-ca" pod "6586d098-8183-4430-99b7-1b5e0973b0ab" (UID: "6586d098-8183-4430-99b7-1b5e0973b0ab"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:04:27.928702 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:27.928638 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50854d7f-ce7a-47eb-9eef-358b178c0dd9-maas-api-openshift-ca" pod "50854d7f-ce7a-47eb-9eef-358b178c0dd9" (UID: "50854d7f-ce7a-47eb-9eef-358b178c0dd9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:04:27.928872 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:27.928849 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6586d098-8183-4430-99b7-1b5e0973b0ab-maas-api-openshift-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:04:27.928944 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:27.928878 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50854d7f-ce7a-47eb-9eef-358b178c0dd9-maas-api-openshift-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:04:27.932191 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:27.932155 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6586d098-8183-4430-99b7-1b5e0973b0ab-maas-api-xks-ca" pod "6586d098-8183-4430-99b7-1b5e0973b0ab" (UID: "6586d098-8183-4430-99b7-1b5e0973b0ab"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:04:27.932353 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:27.932226 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50854d7f-ce7a-47eb-9eef-358b178c0dd9-maas-api-xks-ca" pod "50854d7f-ce7a-47eb-9eef-358b178c0dd9" (UID: "50854d7f-ce7a-47eb-9eef-358b178c0dd9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:04:27.933427 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:27.933404 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50854d7f-ce7a-47eb-9eef-358b178c0dd9-kube-api-access-q54pl" pod "50854d7f-ce7a-47eb-9eef-358b178c0dd9" (UID: "50854d7f-ce7a-47eb-9eef-358b178c0dd9"). InnerVolumeSpecName "kube-api-access-q54pl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:04:27.934522 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:27.934499 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6586d098-8183-4430-99b7-1b5e0973b0ab-kube-api-access-4chmr" pod "6586d098-8183-4430-99b7-1b5e0973b0ab" (UID: "6586d098-8183-4430-99b7-1b5e0973b0ab"). InnerVolumeSpecName "kube-api-access-4chmr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:04:28.000806 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:28.000744 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17a596cf-nb48g" Nov 03 01:04:28.000806 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:28.000782 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17a596cf-nb48g" event={"ID":"50854d7f-ce7a-47eb-9eef-358b178c0dd9","Type":"ContainerDied","Data":"f8b1ff6a962a02925c12d2a9ea1eb0f84d828112a18dc0951246e6e31fbb0674"} Nov 03 01:04:28.001024 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:28.000834 4421 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f8b1ff6a962a02925c12d2a9ea1eb0f84d828112a18dc0951246e6e31fbb0674" Nov 03 01:04:28.002376 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:28.002356 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52aa8264-p5dtw" Nov 03 01:04:28.002501 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:28.002398 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52aa8264-p5dtw" event={"ID":"6586d098-8183-4430-99b7-1b5e0973b0ab","Type":"ContainerDied","Data":"19ff96f80c3c2b4de9ed06c11f9c02f170db75156ca5d9830a1deea0380de931"} Nov 03 01:04:28.002501 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:28.002444 4421 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="19ff96f80c3c2b4de9ed06c11f9c02f170db75156ca5d9830a1deea0380de931" Nov 03 01:04:28.036369 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:28.036338 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6586d098-8183-4430-99b7-1b5e0973b0ab-maas-api-xks-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:04:28.036662 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:28.036611 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4chmr\" (UniqueName: \"kubernetes.io/projected/6586d098-8183-4430-99b7-1b5e0973b0ab-kube-api-access-4chmr\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:04:28.036662 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:28.036655 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q54pl\" (UniqueName: \"kubernetes.io/projected/50854d7f-ce7a-47eb-9eef-358b178c0dd9-kube-api-access-q54pl\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:04:28.037171 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:28.037136 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50854d7f-ce7a-47eb-9eef-358b178c0dd9-maas-api-xks-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:04:33.811428 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:33.811394 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a025f30-zpcwt"] Nov 03 01:04:33.812019 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:33.812004 4421 cpu_manager.go:414] "containerMap: removing container" podUID="6586d098-8183-4430-99b7-1b5e0973b0ab" containerName="revoke-keys" Nov 03 01:04:33.812062 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:33.812021 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6586d098-8183-4430-99b7-1b5e0973b0ab" containerName="revoke-keys" Nov 03 01:04:33.812062 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:33.812044 4421 cpu_manager.go:414] "containerMap: removing container" podUID="50854d7f-ce7a-47eb-9eef-358b178c0dd9" containerName="revoke-keys" Nov 03 01:04:33.812062 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:33.812049 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50854d7f-ce7a-47eb-9eef-358b178c0dd9" containerName="revoke-keys" Nov 03 01:04:33.812181 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:33.812170 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="6586d098-8183-4430-99b7-1b5e0973b0ab" containerName="revoke-keys" Nov 03 01:04:33.812218 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:33.812184 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="50854d7f-ce7a-47eb-9eef-358b178c0dd9" containerName="revoke-keys" Nov 03 01:04:33.817303 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:33.817251 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a025f30-zpcwt" Nov 03 01:04:33.821002 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:33.820973 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 01:04:33.821002 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:33.820981 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:04:33.822123 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:33.822107 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 01:04:33.822226 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:33.822207 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6q4xp\"" Nov 03 01:04:33.822687 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:33.822665 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a025f30-zpcwt"] Nov 03 01:04:33.961723 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:33.961689 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6e8ed68-3a01-46cf-874e-61ec9ef563c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1a025f30-zpcwt\" (UID: \"c6e8ed68-3a01-46cf-874e-61ec9ef563c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a025f30-zpcwt" Nov 03 01:04:33.961906 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:33.961730 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxj2q\" (UniqueName: \"kubernetes.io/projected/c6e8ed68-3a01-46cf-874e-61ec9ef563c1-kube-api-access-cxj2q\") pod \"maas-api-revoke-sub-1a025f30-zpcwt\" (UID: \"c6e8ed68-3a01-46cf-874e-61ec9ef563c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a025f30-zpcwt" Nov 03 01:04:33.961960 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:33.961920 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6e8ed68-3a01-46cf-874e-61ec9ef563c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1a025f30-zpcwt\" (UID: \"c6e8ed68-3a01-46cf-874e-61ec9ef563c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a025f30-zpcwt" Nov 03 01:04:34.063144 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:34.063057 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6e8ed68-3a01-46cf-874e-61ec9ef563c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1a025f30-zpcwt\" (UID: \"c6e8ed68-3a01-46cf-874e-61ec9ef563c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a025f30-zpcwt" Nov 03 01:04:34.063424 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:34.063244 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6e8ed68-3a01-46cf-874e-61ec9ef563c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1a025f30-zpcwt\" (UID: \"c6e8ed68-3a01-46cf-874e-61ec9ef563c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a025f30-zpcwt" Nov 03 01:04:34.063424 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:34.063264 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxj2q\" (UniqueName: \"kubernetes.io/projected/c6e8ed68-3a01-46cf-874e-61ec9ef563c1-kube-api-access-cxj2q\") pod \"maas-api-revoke-sub-1a025f30-zpcwt\" (UID: \"c6e8ed68-3a01-46cf-874e-61ec9ef563c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a025f30-zpcwt" Nov 03 01:04:34.063824 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:34.063799 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6e8ed68-3a01-46cf-874e-61ec9ef563c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1a025f30-zpcwt\" (UID: \"c6e8ed68-3a01-46cf-874e-61ec9ef563c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a025f30-zpcwt" Nov 03 01:04:34.065774 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:34.065752 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6e8ed68-3a01-46cf-874e-61ec9ef563c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1a025f30-zpcwt\" (UID: \"c6e8ed68-3a01-46cf-874e-61ec9ef563c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a025f30-zpcwt" Nov 03 01:04:34.071092 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:34.071070 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxj2q\" (UniqueName: \"kubernetes.io/projected/c6e8ed68-3a01-46cf-874e-61ec9ef563c1-kube-api-access-cxj2q\") pod \"maas-api-revoke-sub-1a025f30-zpcwt\" (UID: \"c6e8ed68-3a01-46cf-874e-61ec9ef563c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a025f30-zpcwt" Nov 03 01:04:34.143303 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:34.142518 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a025f30-zpcwt" Nov 03 01:04:34.448967 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:04:34.448921 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6e8ed68_3a01_46cf_874e_61ec9ef563c1.slice/crio-149f9226611947e36bae5778950a417a39a443abd7e8622db14e50969710bf18 WatchSource:0}: Error finding container 149f9226611947e36bae5778950a417a39a443abd7e8622db14e50969710bf18: Status 404 returned error can't find the container with id 149f9226611947e36bae5778950a417a39a443abd7e8622db14e50969710bf18 Nov 03 01:04:34.449115 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:34.448983 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a025f30-zpcwt"] Nov 03 01:04:34.482240 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:34.482212 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a23337fa-mj5q4"] Nov 03 01:04:34.487918 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:34.487896 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a23337fa-mj5q4" Nov 03 01:04:34.493743 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:34.493717 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a23337fa-mj5q4"] Nov 03 01:04:34.573992 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:34.573959 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-php4d\" (UniqueName: \"kubernetes.io/projected/490ffe57-6ed5-434f-9253-3cbc4b6a1a18-kube-api-access-php4d\") pod \"maas-api-revoke-sub-a23337fa-mj5q4\" (UID: \"490ffe57-6ed5-434f-9253-3cbc4b6a1a18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a23337fa-mj5q4" Nov 03 01:04:34.574165 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:34.574024 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/490ffe57-6ed5-434f-9253-3cbc4b6a1a18-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a23337fa-mj5q4\" (UID: \"490ffe57-6ed5-434f-9253-3cbc4b6a1a18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a23337fa-mj5q4" Nov 03 01:04:34.574165 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:34.574121 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/490ffe57-6ed5-434f-9253-3cbc4b6a1a18-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a23337fa-mj5q4\" (UID: \"490ffe57-6ed5-434f-9253-3cbc4b6a1a18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a23337fa-mj5q4" Nov 03 01:04:34.675549 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:34.675512 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-php4d\" (UniqueName: \"kubernetes.io/projected/490ffe57-6ed5-434f-9253-3cbc4b6a1a18-kube-api-access-php4d\") pod \"maas-api-revoke-sub-a23337fa-mj5q4\" (UID: \"490ffe57-6ed5-434f-9253-3cbc4b6a1a18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a23337fa-mj5q4" Nov 03 01:04:34.675729 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:34.675607 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/490ffe57-6ed5-434f-9253-3cbc4b6a1a18-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a23337fa-mj5q4\" (UID: \"490ffe57-6ed5-434f-9253-3cbc4b6a1a18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a23337fa-mj5q4" Nov 03 01:04:34.675729 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:34.675654 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/490ffe57-6ed5-434f-9253-3cbc4b6a1a18-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a23337fa-mj5q4\" (UID: \"490ffe57-6ed5-434f-9253-3cbc4b6a1a18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a23337fa-mj5q4" Nov 03 01:04:34.676296 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:34.676258 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/490ffe57-6ed5-434f-9253-3cbc4b6a1a18-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a23337fa-mj5q4\" (UID: \"490ffe57-6ed5-434f-9253-3cbc4b6a1a18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a23337fa-mj5q4" Nov 03 01:04:34.678062 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:34.678030 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/490ffe57-6ed5-434f-9253-3cbc4b6a1a18-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a23337fa-mj5q4\" (UID: \"490ffe57-6ed5-434f-9253-3cbc4b6a1a18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a23337fa-mj5q4" Nov 03 01:04:34.683587 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:34.683566 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-php4d\" (UniqueName: \"kubernetes.io/projected/490ffe57-6ed5-434f-9253-3cbc4b6a1a18-kube-api-access-php4d\") pod \"maas-api-revoke-sub-a23337fa-mj5q4\" (UID: \"490ffe57-6ed5-434f-9253-3cbc4b6a1a18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a23337fa-mj5q4" Nov 03 01:04:34.816953 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:34.816916 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a23337fa-mj5q4" Nov 03 01:04:35.054445 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:35.054407 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a025f30-zpcwt" event={"ID":"c6e8ed68-3a01-46cf-874e-61ec9ef563c1","Type":"ContainerStarted","Data":"0a1a9c233f7ed8f9c8448eb3d0fedbb4aea5b3c9891f201dc86e35fc7fd50be5"} Nov 03 01:04:35.054663 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:35.054465 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a025f30-zpcwt" event={"ID":"c6e8ed68-3a01-46cf-874e-61ec9ef563c1","Type":"ContainerStarted","Data":"149f9226611947e36bae5778950a417a39a443abd7e8622db14e50969710bf18"} Nov 03 01:04:35.071075 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:35.070970 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a025f30-zpcwt" podStartSLOduration=2.070956227 podStartE2EDuration="2.070956227s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:04:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:04:35.069638375 +0000 UTC m=+1417.240641473" watchObservedRunningTime="2026-11-03 01:04:35.070956227 +0000 UTC m=+1417.241959373" Nov 03 01:04:35.121730 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:04:35.121698 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod490ffe57_6ed5_434f_9253_3cbc4b6a1a18.slice/crio-5288897e76ba48d194d5aa36ab534cde152acd2886574725548369780aebf5db WatchSource:0}: Error finding container 5288897e76ba48d194d5aa36ab534cde152acd2886574725548369780aebf5db: Status 404 returned error can't find the container with id 5288897e76ba48d194d5aa36ab534cde152acd2886574725548369780aebf5db Nov 03 01:04:35.122453 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:35.122428 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a23337fa-mj5q4"] Nov 03 01:04:36.062782 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:36.062751 4421 generic.go:356] "Generic (PLEG): container finished" podID="c6e8ed68-3a01-46cf-874e-61ec9ef563c1" containerID="0a1a9c233f7ed8f9c8448eb3d0fedbb4aea5b3c9891f201dc86e35fc7fd50be5" exitCode=0 Nov 03 01:04:36.063183 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:36.062839 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a025f30-zpcwt" event={"ID":"c6e8ed68-3a01-46cf-874e-61ec9ef563c1","Type":"ContainerDied","Data":"0a1a9c233f7ed8f9c8448eb3d0fedbb4aea5b3c9891f201dc86e35fc7fd50be5"} Nov 03 01:04:36.064531 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:36.064512 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a23337fa-mj5q4" event={"ID":"490ffe57-6ed5-434f-9253-3cbc4b6a1a18","Type":"ContainerStarted","Data":"1f80ee5dda79c13cca4b34aef98dfd231fa34358bd15e8034dfd87f915d152ca"} Nov 03 01:04:36.064609 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:36.064553 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a23337fa-mj5q4" event={"ID":"490ffe57-6ed5-434f-9253-3cbc4b6a1a18","Type":"ContainerStarted","Data":"5288897e76ba48d194d5aa36ab534cde152acd2886574725548369780aebf5db"} Nov 03 01:04:36.101614 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:36.101560 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a23337fa-mj5q4" podStartSLOduration=2.101545329 podStartE2EDuration="2.101545329s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:04:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:04:36.096083657 +0000 UTC m=+1418.267086766" watchObservedRunningTime="2026-11-03 01:04:36.101545329 +0000 UTC m=+1418.272548452" Nov 03 01:04:37.074926 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:37.074896 4421 generic.go:356] "Generic (PLEG): container finished" podID="490ffe57-6ed5-434f-9253-3cbc4b6a1a18" containerID="1f80ee5dda79c13cca4b34aef98dfd231fa34358bd15e8034dfd87f915d152ca" exitCode=0 Nov 03 01:04:37.075363 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:37.074980 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a23337fa-mj5q4" event={"ID":"490ffe57-6ed5-434f-9253-3cbc4b6a1a18","Type":"ContainerDied","Data":"1f80ee5dda79c13cca4b34aef98dfd231fa34358bd15e8034dfd87f915d152ca"} Nov 03 01:04:37.260468 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:37.260439 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a025f30-zpcwt" Nov 03 01:04:37.322584 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:37.322539 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c6e8ed68-3a01-46cf-874e-61ec9ef563c1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6e8ed68-3a01-46cf-874e-61ec9ef563c1-maas-api-xks-ca\") pod \"c6e8ed68-3a01-46cf-874e-61ec9ef563c1\" (UID: \"c6e8ed68-3a01-46cf-874e-61ec9ef563c1\") " Nov 03 01:04:37.322826 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:37.322645 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6e8ed68-3a01-46cf-874e-61ec9ef563c1-kube-api-access-cxj2q\" (UniqueName: \"kubernetes.io/projected/c6e8ed68-3a01-46cf-874e-61ec9ef563c1-kube-api-access-cxj2q\") pod \"c6e8ed68-3a01-46cf-874e-61ec9ef563c1\" (UID: \"c6e8ed68-3a01-46cf-874e-61ec9ef563c1\") " Nov 03 01:04:37.322826 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:37.322682 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6e8ed68-3a01-46cf-874e-61ec9ef563c1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6e8ed68-3a01-46cf-874e-61ec9ef563c1-maas-api-openshift-ca\") pod \"c6e8ed68-3a01-46cf-874e-61ec9ef563c1\" (UID: \"c6e8ed68-3a01-46cf-874e-61ec9ef563c1\") " Nov 03 01:04:37.323492 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:37.323459 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6e8ed68-3a01-46cf-874e-61ec9ef563c1-maas-api-openshift-ca" pod "c6e8ed68-3a01-46cf-874e-61ec9ef563c1" (UID: "c6e8ed68-3a01-46cf-874e-61ec9ef563c1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:04:37.326491 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:37.326405 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6e8ed68-3a01-46cf-874e-61ec9ef563c1-maas-api-xks-ca" pod "c6e8ed68-3a01-46cf-874e-61ec9ef563c1" (UID: "c6e8ed68-3a01-46cf-874e-61ec9ef563c1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:04:37.326491 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:37.326454 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6e8ed68-3a01-46cf-874e-61ec9ef563c1-kube-api-access-cxj2q" pod "c6e8ed68-3a01-46cf-874e-61ec9ef563c1" (UID: "c6e8ed68-3a01-46cf-874e-61ec9ef563c1"). InnerVolumeSpecName "kube-api-access-cxj2q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:04:37.426936 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:37.424422 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6e8ed68-3a01-46cf-874e-61ec9ef563c1-maas-api-xks-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:04:37.426936 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:37.424467 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxj2q\" (UniqueName: \"kubernetes.io/projected/c6e8ed68-3a01-46cf-874e-61ec9ef563c1-kube-api-access-cxj2q\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:04:37.426936 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:37.424483 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6e8ed68-3a01-46cf-874e-61ec9ef563c1-maas-api-openshift-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:04:38.093708 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:38.093657 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a025f30-zpcwt" Nov 03 01:04:38.094189 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:38.094102 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a025f30-zpcwt" event={"ID":"c6e8ed68-3a01-46cf-874e-61ec9ef563c1","Type":"ContainerDied","Data":"149f9226611947e36bae5778950a417a39a443abd7e8622db14e50969710bf18"} Nov 03 01:04:38.094189 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:38.094156 4421 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="149f9226611947e36bae5778950a417a39a443abd7e8622db14e50969710bf18" Nov 03 01:04:38.269753 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:38.269724 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a23337fa-mj5q4" Nov 03 01:04:38.442007 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:38.441974 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/490ffe57-6ed5-434f-9253-3cbc4b6a1a18-kube-api-access-php4d\" (UniqueName: \"kubernetes.io/projected/490ffe57-6ed5-434f-9253-3cbc4b6a1a18-kube-api-access-php4d\") pod \"490ffe57-6ed5-434f-9253-3cbc4b6a1a18\" (UID: \"490ffe57-6ed5-434f-9253-3cbc4b6a1a18\") " Nov 03 01:04:38.442162 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:38.442104 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/490ffe57-6ed5-434f-9253-3cbc4b6a1a18-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/490ffe57-6ed5-434f-9253-3cbc4b6a1a18-maas-api-openshift-ca\") pod \"490ffe57-6ed5-434f-9253-3cbc4b6a1a18\" (UID: \"490ffe57-6ed5-434f-9253-3cbc4b6a1a18\") " Nov 03 01:04:38.442248 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:38.442230 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/490ffe57-6ed5-434f-9253-3cbc4b6a1a18-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/490ffe57-6ed5-434f-9253-3cbc4b6a1a18-maas-api-xks-ca\") pod \"490ffe57-6ed5-434f-9253-3cbc4b6a1a18\" (UID: \"490ffe57-6ed5-434f-9253-3cbc4b6a1a18\") " Nov 03 01:04:38.442570 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:38.442534 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/490ffe57-6ed5-434f-9253-3cbc4b6a1a18-maas-api-openshift-ca" pod "490ffe57-6ed5-434f-9253-3cbc4b6a1a18" (UID: "490ffe57-6ed5-434f-9253-3cbc4b6a1a18"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:04:38.442851 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:38.442833 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/490ffe57-6ed5-434f-9253-3cbc4b6a1a18-maas-api-openshift-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:04:38.444180 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:38.444153 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/490ffe57-6ed5-434f-9253-3cbc4b6a1a18-maas-api-xks-ca" pod "490ffe57-6ed5-434f-9253-3cbc4b6a1a18" (UID: "490ffe57-6ed5-434f-9253-3cbc4b6a1a18"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:04:38.444180 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:38.444168 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/490ffe57-6ed5-434f-9253-3cbc4b6a1a18-kube-api-access-php4d" pod "490ffe57-6ed5-434f-9253-3cbc4b6a1a18" (UID: "490ffe57-6ed5-434f-9253-3cbc4b6a1a18"). InnerVolumeSpecName "kube-api-access-php4d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:04:38.543816 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:38.543731 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-php4d\" (UniqueName: \"kubernetes.io/projected/490ffe57-6ed5-434f-9253-3cbc4b6a1a18-kube-api-access-php4d\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:04:38.543816 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:38.543763 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/490ffe57-6ed5-434f-9253-3cbc4b6a1a18-maas-api-xks-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:04:39.103917 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:39.103879 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a23337fa-mj5q4" event={"ID":"490ffe57-6ed5-434f-9253-3cbc4b6a1a18","Type":"ContainerDied","Data":"5288897e76ba48d194d5aa36ab534cde152acd2886574725548369780aebf5db"} Nov 03 01:04:39.104387 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:39.103930 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a23337fa-mj5q4" Nov 03 01:04:39.104387 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:39.103935 4421 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5288897e76ba48d194d5aa36ab534cde152acd2886574725548369780aebf5db" Nov 03 01:04:39.343511 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:39.343477 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3b8268a-985qs"] Nov 03 01:04:39.344089 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:39.344074 4421 cpu_manager.go:414] "containerMap: removing container" podUID="c6e8ed68-3a01-46cf-874e-61ec9ef563c1" containerName="revoke-keys" Nov 03 01:04:39.344140 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:39.344090 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6e8ed68-3a01-46cf-874e-61ec9ef563c1" containerName="revoke-keys" Nov 03 01:04:39.344140 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:39.344116 4421 cpu_manager.go:414] "containerMap: removing container" podUID="490ffe57-6ed5-434f-9253-3cbc4b6a1a18" containerName="revoke-keys" Nov 03 01:04:39.344140 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:39.344121 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="490ffe57-6ed5-434f-9253-3cbc4b6a1a18" containerName="revoke-keys" Nov 03 01:04:39.344236 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:39.344221 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="490ffe57-6ed5-434f-9253-3cbc4b6a1a18" containerName="revoke-keys" Nov 03 01:04:39.344267 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:39.344237 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6e8ed68-3a01-46cf-874e-61ec9ef563c1" containerName="revoke-keys" Nov 03 01:04:39.347975 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:39.347957 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3b8268a-985qs" Nov 03 01:04:39.351628 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:39.351601 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 01:04:39.351802 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:39.351653 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 01:04:39.355851 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:39.355781 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3b8268a-985qs"] Nov 03 01:04:39.360372 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:39.360322 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b6b9bed-b022-4614-923a-bcb39e9f5073-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b3b8268a-985qs\" (UID: \"3b6b9bed-b022-4614-923a-bcb39e9f5073\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3b8268a-985qs" Nov 03 01:04:39.360518 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:39.360480 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdw9l\" (UniqueName: \"kubernetes.io/projected/3b6b9bed-b022-4614-923a-bcb39e9f5073-kube-api-access-kdw9l\") pod \"maas-api-revoke-sub-b3b8268a-985qs\" (UID: \"3b6b9bed-b022-4614-923a-bcb39e9f5073\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3b8268a-985qs" Nov 03 01:04:39.360594 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:39.360580 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b6b9bed-b022-4614-923a-bcb39e9f5073-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b3b8268a-985qs\" (UID: \"3b6b9bed-b022-4614-923a-bcb39e9f5073\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3b8268a-985qs" Nov 03 01:04:39.428058 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:39.428025 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:04:39.428224 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:39.428068 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6q4xp\"" Nov 03 01:04:39.461944 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:39.461909 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b6b9bed-b022-4614-923a-bcb39e9f5073-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b3b8268a-985qs\" (UID: \"3b6b9bed-b022-4614-923a-bcb39e9f5073\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3b8268a-985qs" Nov 03 01:04:39.462111 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:39.462001 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdw9l\" (UniqueName: \"kubernetes.io/projected/3b6b9bed-b022-4614-923a-bcb39e9f5073-kube-api-access-kdw9l\") pod \"maas-api-revoke-sub-b3b8268a-985qs\" (UID: \"3b6b9bed-b022-4614-923a-bcb39e9f5073\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3b8268a-985qs" Nov 03 01:04:39.462111 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:39.462095 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b6b9bed-b022-4614-923a-bcb39e9f5073-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b3b8268a-985qs\" (UID: \"3b6b9bed-b022-4614-923a-bcb39e9f5073\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3b8268a-985qs" Nov 03 01:04:39.462773 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:39.462753 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b6b9bed-b022-4614-923a-bcb39e9f5073-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b3b8268a-985qs\" (UID: \"3b6b9bed-b022-4614-923a-bcb39e9f5073\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3b8268a-985qs" Nov 03 01:04:39.464393 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:39.464375 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b6b9bed-b022-4614-923a-bcb39e9f5073-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b3b8268a-985qs\" (UID: \"3b6b9bed-b022-4614-923a-bcb39e9f5073\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3b8268a-985qs" Nov 03 01:04:39.469264 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:39.469241 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdw9l\" (UniqueName: \"kubernetes.io/projected/3b6b9bed-b022-4614-923a-bcb39e9f5073-kube-api-access-kdw9l\") pod \"maas-api-revoke-sub-b3b8268a-985qs\" (UID: \"3b6b9bed-b022-4614-923a-bcb39e9f5073\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3b8268a-985qs" Nov 03 01:04:39.667819 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:39.667739 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3b8268a-985qs" Nov 03 01:04:40.026296 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:40.025505 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3b8268a-985qs"] Nov 03 01:04:40.050294 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:04:40.037371 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b6b9bed_b022_4614_923a_bcb39e9f5073.slice/crio-8bdbbc85bfb86849b0ecacdbcbab8565bdc8da6c3c0923325800ee4166b769fe WatchSource:0}: Error finding container 8bdbbc85bfb86849b0ecacdbcbab8565bdc8da6c3c0923325800ee4166b769fe: Status 404 returned error can't find the container with id 8bdbbc85bfb86849b0ecacdbcbab8565bdc8da6c3c0923325800ee4166b769fe Nov 03 01:04:40.119946 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:40.118943 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3b8268a-985qs" event={"ID":"3b6b9bed-b022-4614-923a-bcb39e9f5073","Type":"ContainerStarted","Data":"8bdbbc85bfb86849b0ecacdbcbab8565bdc8da6c3c0923325800ee4166b769fe"} Nov 03 01:04:41.128795 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:41.128755 4421 generic.go:356] "Generic (PLEG): container finished" podID="3b6b9bed-b022-4614-923a-bcb39e9f5073" containerID="b86453dd86bfec9ca29cf35a4487ea0cbc414cca36e81aedc56a0bc1a27bf079" exitCode=0 Nov 03 01:04:41.129177 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:41.128844 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3b8268a-985qs" event={"ID":"3b6b9bed-b022-4614-923a-bcb39e9f5073","Type":"ContainerDied","Data":"b86453dd86bfec9ca29cf35a4487ea0cbc414cca36e81aedc56a0bc1a27bf079"} Nov 03 01:04:42.425660 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:42.425612 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3b8268a-985qs" Nov 03 01:04:42.996317 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:42.993033 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b6b9bed-b022-4614-923a-bcb39e9f5073-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b6b9bed-b022-4614-923a-bcb39e9f5073-maas-api-openshift-ca\") pod \"3b6b9bed-b022-4614-923a-bcb39e9f5073\" (UID: \"3b6b9bed-b022-4614-923a-bcb39e9f5073\") " Nov 03 01:04:42.996317 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:42.993306 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b6b9bed-b022-4614-923a-bcb39e9f5073-kube-api-access-kdw9l\" (UniqueName: \"kubernetes.io/projected/3b6b9bed-b022-4614-923a-bcb39e9f5073-kube-api-access-kdw9l\") pod \"3b6b9bed-b022-4614-923a-bcb39e9f5073\" (UID: \"3b6b9bed-b022-4614-923a-bcb39e9f5073\") " Nov 03 01:04:42.996317 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:42.993369 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b6b9bed-b022-4614-923a-bcb39e9f5073-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b6b9bed-b022-4614-923a-bcb39e9f5073-maas-api-xks-ca\") pod \"3b6b9bed-b022-4614-923a-bcb39e9f5073\" (UID: \"3b6b9bed-b022-4614-923a-bcb39e9f5073\") " Nov 03 01:04:43.004466 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:43.004420 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b6b9bed-b022-4614-923a-bcb39e9f5073-maas-api-xks-ca" pod "3b6b9bed-b022-4614-923a-bcb39e9f5073" (UID: "3b6b9bed-b022-4614-923a-bcb39e9f5073"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:04:43.030919 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:43.025514 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b6b9bed-b022-4614-923a-bcb39e9f5073-maas-api-openshift-ca" pod "3b6b9bed-b022-4614-923a-bcb39e9f5073" (UID: "3b6b9bed-b022-4614-923a-bcb39e9f5073"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:04:43.030919 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:43.026339 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b6b9bed-b022-4614-923a-bcb39e9f5073-kube-api-access-kdw9l" pod "3b6b9bed-b022-4614-923a-bcb39e9f5073" (UID: "3b6b9bed-b022-4614-923a-bcb39e9f5073"). InnerVolumeSpecName "kube-api-access-kdw9l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:04:43.099306 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:43.097936 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdw9l\" (UniqueName: \"kubernetes.io/projected/3b6b9bed-b022-4614-923a-bcb39e9f5073-kube-api-access-kdw9l\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:04:43.099306 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:43.097970 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b6b9bed-b022-4614-923a-bcb39e9f5073-maas-api-xks-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:04:43.099306 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:43.097989 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b6b9bed-b022-4614-923a-bcb39e9f5073-maas-api-openshift-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:04:43.160347 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:43.158878 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3b8268a-985qs" event={"ID":"3b6b9bed-b022-4614-923a-bcb39e9f5073","Type":"ContainerDied","Data":"8bdbbc85bfb86849b0ecacdbcbab8565bdc8da6c3c0923325800ee4166b769fe"} Nov 03 01:04:43.160347 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:43.158939 4421 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8bdbbc85bfb86849b0ecacdbcbab8565bdc8da6c3c0923325800ee4166b769fe" Nov 03 01:04:43.160347 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:43.159007 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3b8268a-985qs" Nov 03 01:04:44.413300 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:44.412383 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/simulator-w2-f7680d-kserve-85bdc86cd6-8gxrg" podUID="2bdcf521-cebd-4d6b-8554-1b60db8b0f3c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:04:44.863297 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:04:44.862866 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 01:04:44.967532 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:44.966430 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8gm8n" podUID="2d367e12-e46f-4a68-9825-9b1c934d4d12" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:04:44.968365 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:44.967830 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/simulator-w3-83755a-kserve-77d6664559-psqtg" podUID="b0a4c234-af58-4da0-a1a4-e1f167953f6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:04:45.044340 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:45.018387 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-650be0/simulator-w4-650be0-kserve-7bc588774f-gvz2k" podUID="a827fa88-f078-4312-8f12-2c799c11b461" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:04:45.265717 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:45.238376 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/premium-w3-83755a-kserve-7c9d66988b-lt4xd" podUID="c0c6b02e-2ecb-428e-a13d-be6d7cd1ba39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Nov 03 01:04:45.265717 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:45.264388 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/distinct-w3-83755a-kserve-86bdbbdd85-2cnnl" podUID="f4ae9624-d127-4a55-837d-7c9ff0267b37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded" Nov 03 01:04:45.294406 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:45.269106 4421 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8gm8n" podUID="2d367e12-e46f-4a68-9825-9b1c934d4d12" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/health\": context deadline exceeded" Nov 03 01:04:45.413918 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:45.413863 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/simulator-w2-f7680d-kserve-85bdc86cd6-8gxrg" podUID="2bdcf521-cebd-4d6b-8554-1b60db8b0f3c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:04:45.442300 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:45.417623 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f7680d/simulator-w2-f7680d-kserve-85bdc86cd6-8gxrg" Nov 03 01:04:46.016302 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:46.015404 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/simulator-w3-83755a-kserve-77d6664559-psqtg" podUID="b0a4c234-af58-4da0-a1a4-e1f167953f6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:04:46.043302 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:46.041923 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-650be0/simulator-w4-650be0-kserve-7bc588774f-gvz2k" podUID="a827fa88-f078-4312-8f12-2c799c11b461" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Nov 03 01:04:46.085300 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:46.079177 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8gm8n" podUID="2d367e12-e46f-4a68-9825-9b1c934d4d12" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": context deadline exceeded" Nov 03 01:04:46.163318 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:46.153927 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-83755a/simulator-w3-83755a-kserve-77d6664559-psqtg" Nov 03 01:04:46.163318 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:46.154004 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-650be0/simulator-w4-650be0-kserve-7bc588774f-gvz2k" Nov 03 01:04:46.163318 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:46.154041 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8gm8n" Nov 03 01:04:46.229346 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:46.224882 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-83755a/simulator-w3-83755a-kserve-77d6664559-psqtg" Nov 03 01:04:46.254889 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:46.241912 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/premium-w3-83755a-kserve-7c9d66988b-lt4xd" podUID="c0c6b02e-2ecb-428e-a13d-be6d7cd1ba39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:04:46.254889 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:46.243071 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-83755a/premium-w3-83755a-kserve-7c9d66988b-lt4xd" Nov 03 01:04:46.271173 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:46.266640 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/distinct-w3-83755a-kserve-86bdbbdd85-2cnnl" podUID="f4ae9624-d127-4a55-837d-7c9ff0267b37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:04:46.271173 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:46.267225 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-83755a/distinct-w3-83755a-kserve-86bdbbdd85-2cnnl" Nov 03 01:04:46.302098 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:46.302065 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-83755a/premium-w3-83755a-kserve-7c9d66988b-lt4xd" Nov 03 01:04:46.348434 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:46.348407 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-83755a/distinct-w3-83755a-kserve-86bdbbdd85-2cnnl" Nov 03 01:04:46.411300 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:46.411229 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f7680d/simulator-w2-f7680d-kserve-85bdc86cd6-8gxrg" Nov 03 01:04:46.478101 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:46.454310 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8gm8n" Nov 03 01:04:46.480592 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:46.480514 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-650be0/simulator-w4-650be0-kserve-7bc588774f-gvz2k" Nov 03 01:04:46.509118 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:46.506340 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-tt9jg"] Nov 03 01:04:46.509118 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:46.507351 4421 cpu_manager.go:414] "containerMap: removing container" podUID="3b6b9bed-b022-4614-923a-bcb39e9f5073" containerName="revoke-keys" Nov 03 01:04:46.509118 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:46.507371 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b6b9bed-b022-4614-923a-bcb39e9f5073" containerName="revoke-keys" Nov 03 01:04:46.509118 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:46.507599 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b6b9bed-b022-4614-923a-bcb39e9f5073" containerName="revoke-keys" Nov 03 01:04:46.519867 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:46.515568 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-tt9jg"] Nov 03 01:04:46.519867 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:46.515668 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-tt9jg" Nov 03 01:04:46.690592 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:46.690542 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67dd8eb6-fd6b-4a86-a8e4-b393e62044bf-workload-socket\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-tt9jg\" (UID: \"67dd8eb6-fd6b-4a86-a8e4-b393e62044bf\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-tt9jg" Nov 03 01:04:46.690788 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:46.690677 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67dd8eb6-fd6b-4a86-a8e4-b393e62044bf-istio-podinfo\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-tt9jg\" (UID: \"67dd8eb6-fd6b-4a86-a8e4-b393e62044bf\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-tt9jg" Nov 03 01:04:46.690788 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:46.690708 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67dd8eb6-fd6b-4a86-a8e4-b393e62044bf-istio-envoy\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-tt9jg\" (UID: \"67dd8eb6-fd6b-4a86-a8e4-b393e62044bf\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-tt9jg" Nov 03 01:04:46.690788 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:46.690734 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jt5xh\" (UniqueName: \"kubernetes.io/projected/67dd8eb6-fd6b-4a86-a8e4-b393e62044bf-kube-api-access-jt5xh\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-tt9jg\" (UID: \"67dd8eb6-fd6b-4a86-a8e4-b393e62044bf\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-tt9jg" Nov 03 01:04:46.690897 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:46.690869 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67dd8eb6-fd6b-4a86-a8e4-b393e62044bf-istio-data\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-tt9jg\" (UID: \"67dd8eb6-fd6b-4a86-a8e4-b393e62044bf\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-tt9jg" Nov 03 01:04:46.690946 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:46.690925 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67dd8eb6-fd6b-4a86-a8e4-b393e62044bf-istio-token\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-tt9jg\" (UID: \"67dd8eb6-fd6b-4a86-a8e4-b393e62044bf\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-tt9jg" Nov 03 01:04:46.690986 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:46.690961 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67dd8eb6-fd6b-4a86-a8e4-b393e62044bf-istiod-ca-cert\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-tt9jg\" (UID: \"67dd8eb6-fd6b-4a86-a8e4-b393e62044bf\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-tt9jg" Nov 03 01:04:46.691025 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:46.691001 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67dd8eb6-fd6b-4a86-a8e4-b393e62044bf-credential-socket\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-tt9jg\" (UID: \"67dd8eb6-fd6b-4a86-a8e4-b393e62044bf\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-tt9jg" Nov 03 01:04:46.691186 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:46.691169 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67dd8eb6-fd6b-4a86-a8e4-b393e62044bf-workload-certs\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-tt9jg\" (UID: \"67dd8eb6-fd6b-4a86-a8e4-b393e62044bf\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-tt9jg" Nov 03 01:04:46.792078 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:46.791979 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67dd8eb6-fd6b-4a86-a8e4-b393e62044bf-workload-socket\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-tt9jg\" (UID: \"67dd8eb6-fd6b-4a86-a8e4-b393e62044bf\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-tt9jg" Nov 03 01:04:46.792078 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:46.792065 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67dd8eb6-fd6b-4a86-a8e4-b393e62044bf-istio-podinfo\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-tt9jg\" (UID: \"67dd8eb6-fd6b-4a86-a8e4-b393e62044bf\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-tt9jg" Nov 03 01:04:46.792344 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:46.792094 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67dd8eb6-fd6b-4a86-a8e4-b393e62044bf-istio-envoy\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-tt9jg\" (UID: \"67dd8eb6-fd6b-4a86-a8e4-b393e62044bf\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-tt9jg" Nov 03 01:04:46.792344 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:46.792117 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jt5xh\" (UniqueName: \"kubernetes.io/projected/67dd8eb6-fd6b-4a86-a8e4-b393e62044bf-kube-api-access-jt5xh\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-tt9jg\" (UID: \"67dd8eb6-fd6b-4a86-a8e4-b393e62044bf\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-tt9jg" Nov 03 01:04:46.792344 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:46.792209 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67dd8eb6-fd6b-4a86-a8e4-b393e62044bf-istio-data\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-tt9jg\" (UID: \"67dd8eb6-fd6b-4a86-a8e4-b393e62044bf\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-tt9jg" Nov 03 01:04:46.792344 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:46.792232 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67dd8eb6-fd6b-4a86-a8e4-b393e62044bf-istio-token\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-tt9jg\" (UID: \"67dd8eb6-fd6b-4a86-a8e4-b393e62044bf\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-tt9jg" Nov 03 01:04:46.792344 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:46.792257 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67dd8eb6-fd6b-4a86-a8e4-b393e62044bf-istiod-ca-cert\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-tt9jg\" (UID: \"67dd8eb6-fd6b-4a86-a8e4-b393e62044bf\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-tt9jg" Nov 03 01:04:46.792625 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:46.792357 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67dd8eb6-fd6b-4a86-a8e4-b393e62044bf-credential-socket\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-tt9jg\" (UID: \"67dd8eb6-fd6b-4a86-a8e4-b393e62044bf\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-tt9jg" Nov 03 01:04:46.792625 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:46.792463 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67dd8eb6-fd6b-4a86-a8e4-b393e62044bf-workload-socket\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-tt9jg\" (UID: \"67dd8eb6-fd6b-4a86-a8e4-b393e62044bf\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-tt9jg" Nov 03 01:04:46.792738 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:46.792713 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67dd8eb6-fd6b-4a86-a8e4-b393e62044bf-workload-certs\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-tt9jg\" (UID: \"67dd8eb6-fd6b-4a86-a8e4-b393e62044bf\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-tt9jg" Nov 03 01:04:46.792809 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:46.792732 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67dd8eb6-fd6b-4a86-a8e4-b393e62044bf-istio-data\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-tt9jg\" (UID: \"67dd8eb6-fd6b-4a86-a8e4-b393e62044bf\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-tt9jg" Nov 03 01:04:46.792809 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:46.792776 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67dd8eb6-fd6b-4a86-a8e4-b393e62044bf-credential-socket\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-tt9jg\" (UID: \"67dd8eb6-fd6b-4a86-a8e4-b393e62044bf\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-tt9jg" Nov 03 01:04:46.793131 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:46.793107 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67dd8eb6-fd6b-4a86-a8e4-b393e62044bf-workload-certs\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-tt9jg\" (UID: \"67dd8eb6-fd6b-4a86-a8e4-b393e62044bf\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-tt9jg" Nov 03 01:04:46.793352 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:46.793333 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67dd8eb6-fd6b-4a86-a8e4-b393e62044bf-istiod-ca-cert\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-tt9jg\" (UID: \"67dd8eb6-fd6b-4a86-a8e4-b393e62044bf\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-tt9jg" Nov 03 01:04:46.795044 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:46.795012 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67dd8eb6-fd6b-4a86-a8e4-b393e62044bf-istio-envoy\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-tt9jg\" (UID: \"67dd8eb6-fd6b-4a86-a8e4-b393e62044bf\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-tt9jg" Nov 03 01:04:46.795514 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:46.795491 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67dd8eb6-fd6b-4a86-a8e4-b393e62044bf-istio-podinfo\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-tt9jg\" (UID: \"67dd8eb6-fd6b-4a86-a8e4-b393e62044bf\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-tt9jg" Nov 03 01:04:46.805032 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:46.804992 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67dd8eb6-fd6b-4a86-a8e4-b393e62044bf-istio-token\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-tt9jg\" (UID: \"67dd8eb6-fd6b-4a86-a8e4-b393e62044bf\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-tt9jg" Nov 03 01:04:46.805161 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:46.805142 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jt5xh\" (UniqueName: \"kubernetes.io/projected/67dd8eb6-fd6b-4a86-a8e4-b393e62044bf-kube-api-access-jt5xh\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-tt9jg\" (UID: \"67dd8eb6-fd6b-4a86-a8e4-b393e62044bf\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-tt9jg" Nov 03 01:04:46.843004 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:46.842965 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-tt9jg" Nov 03 01:04:47.380417 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:04:47.380374 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67dd8eb6_fd6b_4a86_a8e4_b393e62044bf.slice/crio-b37b51d30e6995b3e8a27acef67d3d5f3f973637afe83d9e4b7b600d1f185423 WatchSource:0}: Error finding container b37b51d30e6995b3e8a27acef67d3d5f3f973637afe83d9e4b7b600d1f185423: Status 404 returned error can't find the container with id b37b51d30e6995b3e8a27acef67d3d5f3f973637afe83d9e4b7b600d1f185423 Nov 03 01:04:47.383751 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:47.383712 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:04:47.383872 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:47.383801 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:04:47.383939 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:47.383873 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:04:47.384475 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:47.384424 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-tt9jg"] Nov 03 01:04:47.500518 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:47.500471 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-tt9jg" event={"ID":"67dd8eb6-fd6b-4a86-a8e4-b393e62044bf","Type":"ContainerStarted","Data":"91e7a11e0d31b71fc299a4faa5c90530dbcef8e9a2a767fd2524b061064134b6"} Nov 03 01:04:47.500983 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:47.500558 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-tt9jg" event={"ID":"67dd8eb6-fd6b-4a86-a8e4-b393e62044bf","Type":"ContainerStarted","Data":"b37b51d30e6995b3e8a27acef67d3d5f3f973637afe83d9e4b7b600d1f185423"} Nov 03 01:04:47.533558 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:47.533495 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-tt9jg" podStartSLOduration=1.533477445 podStartE2EDuration="1.533477445s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:04:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:04:47.528009379 +0000 UTC m=+1429.699012480" watchObservedRunningTime="2026-11-03 01:04:47.533477445 +0000 UTC m=+1429.704480545" Nov 03 01:04:47.843776 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:47.843737 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-tt9jg" Nov 03 01:04:48.264903 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:48.264414 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-208a0d-openshift-default-67457fb65c-ksfzw"] Nov 03 01:04:48.265375 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:48.265332 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-208a0d-openshift-default-67457fb65c-ksfzw" podUID="a47649ab-48b3-4de0-b057-5445c923c427" containerName="istio-proxy" containerID="cri-o://646434756c4cd65ff57a10037ec12e3886ac199876cf56718af7ea6ea311f17e" gracePeriod=30 Nov 03 01:04:48.576531 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:48.576424 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-9jh88"] Nov 03 01:04:48.577004 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:48.576740 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-9jh88" podUID="72f72bc8-0d67-4200-ab31-c167fe8b877e" containerName="istio-proxy" containerID="cri-o://aff03fc28fb084070ff8360f656037f46775f47711662b3bf3b6b035a848967e" gracePeriod=30 Nov 03 01:04:48.837849 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:48.837719 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b050a313-fz7x5"] Nov 03 01:04:48.842852 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:48.842815 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b050a313-fz7x5"] Nov 03 01:04:48.842996 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:48.842945 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b050a313-fz7x5" Nov 03 01:04:48.844780 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:48.844749 4421 patch_prober.go:29] interesting pod/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-tt9jg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" start-of-body= Nov 03 01:04:48.844927 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:48.844803 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-tt9jg" podUID="67dd8eb6-fd6b-4a86-a8e4-b393e62044bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" Nov 03 01:04:48.862749 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:48.862724 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6q4xp\"" Nov 03 01:04:48.868712 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:48.868391 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 01:04:48.872155 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:48.872128 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 01:04:48.902463 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:48.902429 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf46dbdd-d501-400c-b82a-aaf689363bab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b050a313-fz7x5\" (UID: \"bf46dbdd-d501-400c-b82a-aaf689363bab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b050a313-fz7x5" Nov 03 01:04:48.902644 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:48.902510 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf46dbdd-d501-400c-b82a-aaf689363bab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b050a313-fz7x5\" (UID: \"bf46dbdd-d501-400c-b82a-aaf689363bab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b050a313-fz7x5" Nov 03 01:04:48.902765 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:48.902743 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4j9m\" (UniqueName: \"kubernetes.io/projected/bf46dbdd-d501-400c-b82a-aaf689363bab-kube-api-access-t4j9m\") pod \"maas-api-revoke-sub-b050a313-fz7x5\" (UID: \"bf46dbdd-d501-400c-b82a-aaf689363bab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b050a313-fz7x5" Nov 03 01:04:48.959224 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:48.959188 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:04:49.004376 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:49.004330 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf46dbdd-d501-400c-b82a-aaf689363bab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b050a313-fz7x5\" (UID: \"bf46dbdd-d501-400c-b82a-aaf689363bab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b050a313-fz7x5" Nov 03 01:04:49.004586 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:49.004465 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf46dbdd-d501-400c-b82a-aaf689363bab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b050a313-fz7x5\" (UID: \"bf46dbdd-d501-400c-b82a-aaf689363bab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b050a313-fz7x5" Nov 03 01:04:49.004720 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:49.004703 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4j9m\" (UniqueName: \"kubernetes.io/projected/bf46dbdd-d501-400c-b82a-aaf689363bab-kube-api-access-t4j9m\") pod \"maas-api-revoke-sub-b050a313-fz7x5\" (UID: \"bf46dbdd-d501-400c-b82a-aaf689363bab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b050a313-fz7x5" Nov 03 01:04:49.013492 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:49.005947 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf46dbdd-d501-400c-b82a-aaf689363bab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b050a313-fz7x5\" (UID: \"bf46dbdd-d501-400c-b82a-aaf689363bab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b050a313-fz7x5" Nov 03 01:04:49.055323 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:49.050034 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf46dbdd-d501-400c-b82a-aaf689363bab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b050a313-fz7x5\" (UID: \"bf46dbdd-d501-400c-b82a-aaf689363bab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b050a313-fz7x5" Nov 03 01:04:49.059152 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:49.059119 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4j9m\" (UniqueName: \"kubernetes.io/projected/bf46dbdd-d501-400c-b82a-aaf689363bab-kube-api-access-t4j9m\") pod \"maas-api-revoke-sub-b050a313-fz7x5\" (UID: \"bf46dbdd-d501-400c-b82a-aaf689363bab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b050a313-fz7x5" Nov 03 01:04:49.163000 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:49.162912 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b050a313-fz7x5" Nov 03 01:04:49.477079 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:49.476998 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b050a313-fz7x5"] Nov 03 01:04:49.480947 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:04:49.480915 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf46dbdd_d501_400c_b82a_aaf689363bab.slice/crio-bfa5245da232c38db64eb76d38d136b2b90b8c1fbe7204a9405207a24f00ac1d WatchSource:0}: Error finding container bfa5245da232c38db64eb76d38d136b2b90b8c1fbe7204a9405207a24f00ac1d: Status 404 returned error can't find the container with id bfa5245da232c38db64eb76d38d136b2b90b8c1fbe7204a9405207a24f00ac1d Nov 03 01:04:49.572091 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:49.572048 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b050a313-fz7x5" event={"ID":"bf46dbdd-d501-400c-b82a-aaf689363bab","Type":"ContainerStarted","Data":"bfa5245da232c38db64eb76d38d136b2b90b8c1fbe7204a9405207a24f00ac1d"} Nov 03 01:04:49.799191 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:49.799156 4421 patch_prober.go:29] interesting pod/e2e-praxis-b-208a0d-openshift-default-67457fb65c-ksfzw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 01:04:49.799599 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:49.799214 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-208a0d-openshift-default-67457fb65c-ksfzw" podUID="a47649ab-48b3-4de0-b057-5445c923c427" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 01:04:49.843885 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:49.843853 4421 patch_prober.go:29] interesting pod/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-tt9jg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" start-of-body= Nov 03 01:04:49.844058 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:49.843909 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-tt9jg" podUID="67dd8eb6-fd6b-4a86-a8e4-b393e62044bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" Nov 03 01:04:49.862347 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:49.862321 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-97d101cc-vf5cc"] Nov 03 01:04:49.867826 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:49.867806 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97d101cc-vf5cc" Nov 03 01:04:49.875756 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:49.875732 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-97d101cc-vf5cc"] Nov 03 01:04:49.941882 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:49.941850 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32b8c1b0-8e81-4430-840b-ca5d03a7195d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-97d101cc-vf5cc\" (UID: \"32b8c1b0-8e81-4430-840b-ca5d03a7195d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97d101cc-vf5cc" Nov 03 01:04:49.942044 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:49.942018 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zctn\" (UniqueName: \"kubernetes.io/projected/32b8c1b0-8e81-4430-840b-ca5d03a7195d-kube-api-access-7zctn\") pod \"maas-api-revoke-sub-97d101cc-vf5cc\" (UID: \"32b8c1b0-8e81-4430-840b-ca5d03a7195d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97d101cc-vf5cc" Nov 03 01:04:49.942199 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:49.942176 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32b8c1b0-8e81-4430-840b-ca5d03a7195d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-97d101cc-vf5cc\" (UID: \"32b8c1b0-8e81-4430-840b-ca5d03a7195d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97d101cc-vf5cc" Nov 03 01:04:50.043300 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:50.043241 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32b8c1b0-8e81-4430-840b-ca5d03a7195d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-97d101cc-vf5cc\" (UID: \"32b8c1b0-8e81-4430-840b-ca5d03a7195d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97d101cc-vf5cc" Nov 03 01:04:50.043474 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:50.043367 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zctn\" (UniqueName: \"kubernetes.io/projected/32b8c1b0-8e81-4430-840b-ca5d03a7195d-kube-api-access-7zctn\") pod \"maas-api-revoke-sub-97d101cc-vf5cc\" (UID: \"32b8c1b0-8e81-4430-840b-ca5d03a7195d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97d101cc-vf5cc" Nov 03 01:04:50.043583 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:50.043508 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32b8c1b0-8e81-4430-840b-ca5d03a7195d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-97d101cc-vf5cc\" (UID: \"32b8c1b0-8e81-4430-840b-ca5d03a7195d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97d101cc-vf5cc" Nov 03 01:04:50.044124 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:50.044096 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32b8c1b0-8e81-4430-840b-ca5d03a7195d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-97d101cc-vf5cc\" (UID: \"32b8c1b0-8e81-4430-840b-ca5d03a7195d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97d101cc-vf5cc" Nov 03 01:04:50.046808 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:50.046767 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32b8c1b0-8e81-4430-840b-ca5d03a7195d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-97d101cc-vf5cc\" (UID: \"32b8c1b0-8e81-4430-840b-ca5d03a7195d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97d101cc-vf5cc" Nov 03 01:04:50.052554 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:50.052533 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zctn\" (UniqueName: \"kubernetes.io/projected/32b8c1b0-8e81-4430-840b-ca5d03a7195d-kube-api-access-7zctn\") pod \"maas-api-revoke-sub-97d101cc-vf5cc\" (UID: \"32b8c1b0-8e81-4430-840b-ca5d03a7195d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97d101cc-vf5cc" Nov 03 01:04:50.189408 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:50.189376 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97d101cc-vf5cc" Nov 03 01:04:50.537992 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:50.537958 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-97d101cc-vf5cc"] Nov 03 01:04:50.538468 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:04:50.538436 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32b8c1b0_8e81_4430_840b_ca5d03a7195d.slice/crio-915d8218785565736f11733a6ee729817ba7c49411241838a4110a11f8062d95 WatchSource:0}: Error finding container 915d8218785565736f11733a6ee729817ba7c49411241838a4110a11f8062d95: Status 404 returned error can't find the container with id 915d8218785565736f11733a6ee729817ba7c49411241838a4110a11f8062d95 Nov 03 01:04:50.582160 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:50.582127 4421 generic.go:356] "Generic (PLEG): container finished" podID="bf46dbdd-d501-400c-b82a-aaf689363bab" containerID="2876ca3c7ee93ffe453542b4021df38c53516ad611837f6d53c3b99e90a2eb6b" exitCode=0 Nov 03 01:04:50.582374 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:50.582217 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b050a313-fz7x5" event={"ID":"bf46dbdd-d501-400c-b82a-aaf689363bab","Type":"ContainerDied","Data":"2876ca3c7ee93ffe453542b4021df38c53516ad611837f6d53c3b99e90a2eb6b"} Nov 03 01:04:50.583744 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:50.583714 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97d101cc-vf5cc" event={"ID":"32b8c1b0-8e81-4430-840b-ca5d03a7195d","Type":"ContainerStarted","Data":"915d8218785565736f11733a6ee729817ba7c49411241838a4110a11f8062d95"} Nov 03 01:04:50.844301 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:50.844196 4421 patch_prober.go:29] interesting pod/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-tt9jg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" start-of-body= Nov 03 01:04:50.844301 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:50.844258 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-tt9jg" podUID="67dd8eb6-fd6b-4a86-a8e4-b393e62044bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" Nov 03 01:04:51.596399 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:51.596355 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97d101cc-vf5cc" event={"ID":"32b8c1b0-8e81-4430-840b-ca5d03a7195d","Type":"ContainerStarted","Data":"d44aabc392f84beba7ee975c0d56a4b0321d698f695880cdeb9d9d01eb133d67"} Nov 03 01:04:51.639428 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:51.639337 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97d101cc-vf5cc" podStartSLOduration=2.6393206879999997 podStartE2EDuration="2.639320688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:04:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:04:51.632468152 +0000 UTC m=+1433.803471249" watchObservedRunningTime="2026-11-03 01:04:51.639320688 +0000 UTC m=+1433.810323783" Nov 03 01:04:51.831211 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:51.830834 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b050a313-fz7x5" Nov 03 01:04:51.844220 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:51.844190 4421 patch_prober.go:29] interesting pod/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-tt9jg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" start-of-body= Nov 03 01:04:51.844360 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:51.844255 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-tt9jg" podUID="67dd8eb6-fd6b-4a86-a8e4-b393e62044bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" Nov 03 01:04:51.901085 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:51.900998 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf46dbdd-d501-400c-b82a-aaf689363bab-kube-api-access-t4j9m\" (UniqueName: \"kubernetes.io/projected/bf46dbdd-d501-400c-b82a-aaf689363bab-kube-api-access-t4j9m\") pod \"bf46dbdd-d501-400c-b82a-aaf689363bab\" (UID: \"bf46dbdd-d501-400c-b82a-aaf689363bab\") " Nov 03 01:04:51.901085 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:51.901066 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf46dbdd-d501-400c-b82a-aaf689363bab-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf46dbdd-d501-400c-b82a-aaf689363bab-maas-api-openshift-ca\") pod \"bf46dbdd-d501-400c-b82a-aaf689363bab\" (UID: \"bf46dbdd-d501-400c-b82a-aaf689363bab\") " Nov 03 01:04:51.901473 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:51.901128 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bf46dbdd-d501-400c-b82a-aaf689363bab-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf46dbdd-d501-400c-b82a-aaf689363bab-maas-api-xks-ca\") pod \"bf46dbdd-d501-400c-b82a-aaf689363bab\" (UID: \"bf46dbdd-d501-400c-b82a-aaf689363bab\") " Nov 03 01:04:51.901547 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:51.901514 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf46dbdd-d501-400c-b82a-aaf689363bab-maas-api-openshift-ca" pod "bf46dbdd-d501-400c-b82a-aaf689363bab" (UID: "bf46dbdd-d501-400c-b82a-aaf689363bab"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:04:51.901828 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:51.901802 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf46dbdd-d501-400c-b82a-aaf689363bab-maas-api-openshift-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:04:51.903322 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:51.903295 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf46dbdd-d501-400c-b82a-aaf689363bab-kube-api-access-t4j9m" pod "bf46dbdd-d501-400c-b82a-aaf689363bab" (UID: "bf46dbdd-d501-400c-b82a-aaf689363bab"). InnerVolumeSpecName "kube-api-access-t4j9m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:04:51.903400 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:51.903378 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf46dbdd-d501-400c-b82a-aaf689363bab-maas-api-xks-ca" pod "bf46dbdd-d501-400c-b82a-aaf689363bab" (UID: "bf46dbdd-d501-400c-b82a-aaf689363bab"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:04:52.002875 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:52.002837 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf46dbdd-d501-400c-b82a-aaf689363bab-maas-api-xks-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:04:52.002875 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:52.002868 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4j9m\" (UniqueName: \"kubernetes.io/projected/bf46dbdd-d501-400c-b82a-aaf689363bab-kube-api-access-t4j9m\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:04:52.651418 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:52.650461 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b050a313-fz7x5" Nov 03 01:04:52.651418 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:52.650524 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b050a313-fz7x5" event={"ID":"bf46dbdd-d501-400c-b82a-aaf689363bab","Type":"ContainerDied","Data":"bfa5245da232c38db64eb76d38d136b2b90b8c1fbe7204a9405207a24f00ac1d"} Nov 03 01:04:52.651418 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:52.650602 4421 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bfa5245da232c38db64eb76d38d136b2b90b8c1fbe7204a9405207a24f00ac1d" Nov 03 01:04:52.657727 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:52.653082 4421 generic.go:356] "Generic (PLEG): container finished" podID="32b8c1b0-8e81-4430-840b-ca5d03a7195d" containerID="d44aabc392f84beba7ee975c0d56a4b0321d698f695880cdeb9d9d01eb133d67" exitCode=0 Nov 03 01:04:52.657727 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:52.653346 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97d101cc-vf5cc" event={"ID":"32b8c1b0-8e81-4430-840b-ca5d03a7195d","Type":"ContainerDied","Data":"d44aabc392f84beba7ee975c0d56a4b0321d698f695880cdeb9d9d01eb133d67"} Nov 03 01:04:52.843520 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:52.843484 4421 patch_prober.go:29] interesting pod/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-tt9jg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" start-of-body= Nov 03 01:04:52.843685 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:52.843553 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-tt9jg" podUID="67dd8eb6-fd6b-4a86-a8e4-b393e62044bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" Nov 03 01:04:53.596375 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:53.596342 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-208a0d-openshift-default-67457fb65c-ksfzw" Nov 03 01:04:53.663897 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:53.663861 4421 generic.go:356] "Generic (PLEG): container finished" podID="a47649ab-48b3-4de0-b057-5445c923c427" containerID="646434756c4cd65ff57a10037ec12e3886ac199876cf56718af7ea6ea311f17e" exitCode=0 Nov 03 01:04:53.664074 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:53.663917 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-208a0d-openshift-default-67457fb65c-ksfzw" Nov 03 01:04:53.664074 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:53.663947 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-208a0d-openshift-default-67457fb65c-ksfzw" event={"ID":"a47649ab-48b3-4de0-b057-5445c923c427","Type":"ContainerDied","Data":"646434756c4cd65ff57a10037ec12e3886ac199876cf56718af7ea6ea311f17e"} Nov 03 01:04:53.664074 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:53.664007 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-208a0d-openshift-default-67457fb65c-ksfzw" event={"ID":"a47649ab-48b3-4de0-b057-5445c923c427","Type":"ContainerDied","Data":"fbd3340cda97898e99d665c9b51ed26e349f1775c95a8a507ff0b9d9eec97852"} Nov 03 01:04:53.664074 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:53.664033 4421 scope.go:122] "RemoveContainer" containerID="646434756c4cd65ff57a10037ec12e3886ac199876cf56718af7ea6ea311f17e" Nov 03 01:04:53.666461 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:53.666436 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a47649ab-48b3-4de0-b057-5445c923c427-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a47649ab-48b3-4de0-b057-5445c923c427-credential-socket\") pod \"a47649ab-48b3-4de0-b057-5445c923c427\" (UID: \"a47649ab-48b3-4de0-b057-5445c923c427\") " Nov 03 01:04:53.666783 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:53.666753 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a47649ab-48b3-4de0-b057-5445c923c427-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a47649ab-48b3-4de0-b057-5445c923c427-istiod-ca-cert\") pod \"a47649ab-48b3-4de0-b057-5445c923c427\" (UID: \"a47649ab-48b3-4de0-b057-5445c923c427\") " Nov 03 01:04:53.666911 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:53.666826 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a47649ab-48b3-4de0-b057-5445c923c427-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a47649ab-48b3-4de0-b057-5445c923c427-istio-podinfo\") pod \"a47649ab-48b3-4de0-b057-5445c923c427\" (UID: \"a47649ab-48b3-4de0-b057-5445c923c427\") " Nov 03 01:04:53.666911 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:53.666875 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a47649ab-48b3-4de0-b057-5445c923c427-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a47649ab-48b3-4de0-b057-5445c923c427-istio-data\") pod \"a47649ab-48b3-4de0-b057-5445c923c427\" (UID: \"a47649ab-48b3-4de0-b057-5445c923c427\") " Nov 03 01:04:53.667038 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:53.666932 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a47649ab-48b3-4de0-b057-5445c923c427-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a47649ab-48b3-4de0-b057-5445c923c427-workload-certs\") pod \"a47649ab-48b3-4de0-b057-5445c923c427\" (UID: \"a47649ab-48b3-4de0-b057-5445c923c427\") " Nov 03 01:04:53.667038 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:53.666971 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a47649ab-48b3-4de0-b057-5445c923c427-istio-token\" (UniqueName: \"kubernetes.io/projected/a47649ab-48b3-4de0-b057-5445c923c427-istio-token\") pod \"a47649ab-48b3-4de0-b057-5445c923c427\" (UID: \"a47649ab-48b3-4de0-b057-5445c923c427\") " Nov 03 01:04:53.667038 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:53.667002 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a47649ab-48b3-4de0-b057-5445c923c427-kube-api-access-rm65l\" (UniqueName: \"kubernetes.io/projected/a47649ab-48b3-4de0-b057-5445c923c427-kube-api-access-rm65l\") pod \"a47649ab-48b3-4de0-b057-5445c923c427\" (UID: \"a47649ab-48b3-4de0-b057-5445c923c427\") " Nov 03 01:04:53.667193 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:53.667039 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a47649ab-48b3-4de0-b057-5445c923c427-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a47649ab-48b3-4de0-b057-5445c923c427-workload-socket\") pod \"a47649ab-48b3-4de0-b057-5445c923c427\" (UID: \"a47649ab-48b3-4de0-b057-5445c923c427\") " Nov 03 01:04:53.667193 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:53.667114 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a47649ab-48b3-4de0-b057-5445c923c427-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a47649ab-48b3-4de0-b057-5445c923c427-istio-envoy\") pod \"a47649ab-48b3-4de0-b057-5445c923c427\" (UID: \"a47649ab-48b3-4de0-b057-5445c923c427\") " Nov 03 01:04:53.667921 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:53.667833 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a47649ab-48b3-4de0-b057-5445c923c427-istio-data" pod "a47649ab-48b3-4de0-b057-5445c923c427" (UID: "a47649ab-48b3-4de0-b057-5445c923c427"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:04:53.668126 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:53.668102 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a47649ab-48b3-4de0-b057-5445c923c427-credential-socket" pod "a47649ab-48b3-4de0-b057-5445c923c427" (UID: "a47649ab-48b3-4de0-b057-5445c923c427"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:04:53.668493 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:53.668466 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a47649ab-48b3-4de0-b057-5445c923c427-istiod-ca-cert" pod "a47649ab-48b3-4de0-b057-5445c923c427" (UID: "a47649ab-48b3-4de0-b057-5445c923c427"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:04:53.669469 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:53.669443 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a47649ab-48b3-4de0-b057-5445c923c427-workload-certs" pod "a47649ab-48b3-4de0-b057-5445c923c427" (UID: "a47649ab-48b3-4de0-b057-5445c923c427"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:04:53.669937 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:53.669912 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a47649ab-48b3-4de0-b057-5445c923c427-workload-socket" pod "a47649ab-48b3-4de0-b057-5445c923c427" (UID: "a47649ab-48b3-4de0-b057-5445c923c427"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:04:53.671693 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:53.671655 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a47649ab-48b3-4de0-b057-5445c923c427-istio-podinfo" pod "a47649ab-48b3-4de0-b057-5445c923c427" (UID: "a47649ab-48b3-4de0-b057-5445c923c427"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:04:53.672371 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:53.672290 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a47649ab-48b3-4de0-b057-5445c923c427-kube-api-access-rm65l" pod "a47649ab-48b3-4de0-b057-5445c923c427" (UID: "a47649ab-48b3-4de0-b057-5445c923c427"). InnerVolumeSpecName "kube-api-access-rm65l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:04:53.673869 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:53.673842 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a47649ab-48b3-4de0-b057-5445c923c427-istio-token" pod "a47649ab-48b3-4de0-b057-5445c923c427" (UID: "a47649ab-48b3-4de0-b057-5445c923c427"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:04:53.674889 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:53.674856 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a47649ab-48b3-4de0-b057-5445c923c427-istio-envoy" pod "a47649ab-48b3-4de0-b057-5445c923c427" (UID: "a47649ab-48b3-4de0-b057-5445c923c427"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:04:53.697648 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:53.697619 4421 scope.go:122] "RemoveContainer" containerID="e68dbf918c3fed3dce2f889f1e30293534ea755b4403b621185c3638abd9173a" Nov 03 01:04:53.717156 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:53.716951 4421 scope.go:122] "RemoveContainer" containerID="646434756c4cd65ff57a10037ec12e3886ac199876cf56718af7ea6ea311f17e" Nov 03 01:04:53.717585 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:04:53.717525 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"646434756c4cd65ff57a10037ec12e3886ac199876cf56718af7ea6ea311f17e\": container with ID starting with 646434756c4cd65ff57a10037ec12e3886ac199876cf56718af7ea6ea311f17e not found: ID does not exist" containerID="646434756c4cd65ff57a10037ec12e3886ac199876cf56718af7ea6ea311f17e" Nov 03 01:04:53.717585 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:53.717557 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"646434756c4cd65ff57a10037ec12e3886ac199876cf56718af7ea6ea311f17e"} err="failed to get container status \"646434756c4cd65ff57a10037ec12e3886ac199876cf56718af7ea6ea311f17e\": rpc error: code = NotFound desc = could not find container \"646434756c4cd65ff57a10037ec12e3886ac199876cf56718af7ea6ea311f17e\": container with ID starting with 646434756c4cd65ff57a10037ec12e3886ac199876cf56718af7ea6ea311f17e not found: ID does not exist" Nov 03 01:04:53.717585 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:53.717581 4421 scope.go:122] "RemoveContainer" containerID="e68dbf918c3fed3dce2f889f1e30293534ea755b4403b621185c3638abd9173a" Nov 03 01:04:53.718151 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:04:53.718101 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e68dbf918c3fed3dce2f889f1e30293534ea755b4403b621185c3638abd9173a\": container with ID starting with e68dbf918c3fed3dce2f889f1e30293534ea755b4403b621185c3638abd9173a not found: ID does not exist" containerID="e68dbf918c3fed3dce2f889f1e30293534ea755b4403b621185c3638abd9173a" Nov 03 01:04:53.718151 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:53.718133 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e68dbf918c3fed3dce2f889f1e30293534ea755b4403b621185c3638abd9173a"} err="failed to get container status \"e68dbf918c3fed3dce2f889f1e30293534ea755b4403b621185c3638abd9173a\": rpc error: code = NotFound desc = could not find container \"e68dbf918c3fed3dce2f889f1e30293534ea755b4403b621185c3638abd9173a\": container with ID starting with e68dbf918c3fed3dce2f889f1e30293534ea755b4403b621185c3638abd9173a not found: ID does not exist" Nov 03 01:04:53.779297 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:53.771006 4421 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a47649ab-48b3-4de0-b057-5445c923c427-istiod-ca-cert\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:04:53.779297 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:53.771047 4421 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a47649ab-48b3-4de0-b057-5445c923c427-istio-podinfo\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:04:53.779297 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:53.771080 4421 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a47649ab-48b3-4de0-b057-5445c923c427-istio-data\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:04:53.779297 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:53.771096 4421 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a47649ab-48b3-4de0-b057-5445c923c427-workload-certs\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:04:53.779297 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:53.771112 4421 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a47649ab-48b3-4de0-b057-5445c923c427-istio-token\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:04:53.779297 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:53.771126 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rm65l\" (UniqueName: \"kubernetes.io/projected/a47649ab-48b3-4de0-b057-5445c923c427-kube-api-access-rm65l\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:04:53.779297 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:53.771142 4421 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a47649ab-48b3-4de0-b057-5445c923c427-workload-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:04:53.779297 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:53.771155 4421 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a47649ab-48b3-4de0-b057-5445c923c427-istio-envoy\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:04:53.779297 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:53.771169 4421 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a47649ab-48b3-4de0-b057-5445c923c427-credential-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:04:53.843888 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:53.843485 4421 patch_prober.go:29] interesting pod/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-tt9jg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" start-of-body= Nov 03 01:04:53.843888 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:53.843547 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-tt9jg" podUID="67dd8eb6-fd6b-4a86-a8e4-b393e62044bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" Nov 03 01:04:53.852257 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:53.852226 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97d101cc-vf5cc" Nov 03 01:04:53.872353 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:53.872316 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32b8c1b0-8e81-4430-840b-ca5d03a7195d-kube-api-access-7zctn\" (UniqueName: \"kubernetes.io/projected/32b8c1b0-8e81-4430-840b-ca5d03a7195d-kube-api-access-7zctn\") pod \"32b8c1b0-8e81-4430-840b-ca5d03a7195d\" (UID: \"32b8c1b0-8e81-4430-840b-ca5d03a7195d\") " Nov 03 01:04:53.872562 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:53.872379 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/32b8c1b0-8e81-4430-840b-ca5d03a7195d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32b8c1b0-8e81-4430-840b-ca5d03a7195d-maas-api-xks-ca\") pod \"32b8c1b0-8e81-4430-840b-ca5d03a7195d\" (UID: \"32b8c1b0-8e81-4430-840b-ca5d03a7195d\") " Nov 03 01:04:53.872562 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:53.872470 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32b8c1b0-8e81-4430-840b-ca5d03a7195d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32b8c1b0-8e81-4430-840b-ca5d03a7195d-maas-api-openshift-ca\") pod \"32b8c1b0-8e81-4430-840b-ca5d03a7195d\" (UID: \"32b8c1b0-8e81-4430-840b-ca5d03a7195d\") " Nov 03 01:04:53.873183 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:53.873142 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32b8c1b0-8e81-4430-840b-ca5d03a7195d-maas-api-openshift-ca" pod "32b8c1b0-8e81-4430-840b-ca5d03a7195d" (UID: "32b8c1b0-8e81-4430-840b-ca5d03a7195d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:04:53.875488 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:53.875446 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32b8c1b0-8e81-4430-840b-ca5d03a7195d-maas-api-xks-ca" pod "32b8c1b0-8e81-4430-840b-ca5d03a7195d" (UID: "32b8c1b0-8e81-4430-840b-ca5d03a7195d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:04:53.875488 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:53.875478 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32b8c1b0-8e81-4430-840b-ca5d03a7195d-kube-api-access-7zctn" pod "32b8c1b0-8e81-4430-840b-ca5d03a7195d" (UID: "32b8c1b0-8e81-4430-840b-ca5d03a7195d"). InnerVolumeSpecName "kube-api-access-7zctn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:04:53.913324 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:53.913288 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-9jh88" Nov 03 01:04:53.973600 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:53.973563 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72f72bc8-0d67-4200-ab31-c167fe8b877e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72f72bc8-0d67-4200-ab31-c167fe8b877e-istio-envoy\") pod \"72f72bc8-0d67-4200-ab31-c167fe8b877e\" (UID: \"72f72bc8-0d67-4200-ab31-c167fe8b877e\") " Nov 03 01:04:53.973814 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:53.973654 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72f72bc8-0d67-4200-ab31-c167fe8b877e-istio-token\" (UniqueName: \"kubernetes.io/projected/72f72bc8-0d67-4200-ab31-c167fe8b877e-istio-token\") pod \"72f72bc8-0d67-4200-ab31-c167fe8b877e\" (UID: \"72f72bc8-0d67-4200-ab31-c167fe8b877e\") " Nov 03 01:04:53.973814 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:53.973705 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72f72bc8-0d67-4200-ab31-c167fe8b877e-kube-api-access-rgzdg\" (UniqueName: \"kubernetes.io/projected/72f72bc8-0d67-4200-ab31-c167fe8b877e-kube-api-access-rgzdg\") pod \"72f72bc8-0d67-4200-ab31-c167fe8b877e\" (UID: \"72f72bc8-0d67-4200-ab31-c167fe8b877e\") " Nov 03 01:04:53.973814 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:53.973759 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/72f72bc8-0d67-4200-ab31-c167fe8b877e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72f72bc8-0d67-4200-ab31-c167fe8b877e-istio-podinfo\") pod \"72f72bc8-0d67-4200-ab31-c167fe8b877e\" (UID: \"72f72bc8-0d67-4200-ab31-c167fe8b877e\") " Nov 03 01:04:53.973991 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:53.973879 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72f72bc8-0d67-4200-ab31-c167fe8b877e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72f72bc8-0d67-4200-ab31-c167fe8b877e-workload-socket\") pod \"72f72bc8-0d67-4200-ab31-c167fe8b877e\" (UID: \"72f72bc8-0d67-4200-ab31-c167fe8b877e\") " Nov 03 01:04:53.973991 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:53.973925 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72f72bc8-0d67-4200-ab31-c167fe8b877e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72f72bc8-0d67-4200-ab31-c167fe8b877e-istiod-ca-cert\") pod \"72f72bc8-0d67-4200-ab31-c167fe8b877e\" (UID: \"72f72bc8-0d67-4200-ab31-c167fe8b877e\") " Nov 03 01:04:53.973991 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:53.973962 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72f72bc8-0d67-4200-ab31-c167fe8b877e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72f72bc8-0d67-4200-ab31-c167fe8b877e-istio-data\") pod \"72f72bc8-0d67-4200-ab31-c167fe8b877e\" (UID: \"72f72bc8-0d67-4200-ab31-c167fe8b877e\") " Nov 03 01:04:53.975996 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:53.974361 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72f72bc8-0d67-4200-ab31-c167fe8b877e-istio-data" pod "72f72bc8-0d67-4200-ab31-c167fe8b877e" (UID: "72f72bc8-0d67-4200-ab31-c167fe8b877e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:04:53.975996 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:53.974695 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72f72bc8-0d67-4200-ab31-c167fe8b877e-istiod-ca-cert" pod "72f72bc8-0d67-4200-ab31-c167fe8b877e" (UID: "72f72bc8-0d67-4200-ab31-c167fe8b877e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:04:53.975996 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:53.974982 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72f72bc8-0d67-4200-ab31-c167fe8b877e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72f72bc8-0d67-4200-ab31-c167fe8b877e-workload-certs\") pod \"72f72bc8-0d67-4200-ab31-c167fe8b877e\" (UID: \"72f72bc8-0d67-4200-ab31-c167fe8b877e\") " Nov 03 01:04:53.975996 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:53.975095 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72f72bc8-0d67-4200-ab31-c167fe8b877e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72f72bc8-0d67-4200-ab31-c167fe8b877e-credential-socket\") pod \"72f72bc8-0d67-4200-ab31-c167fe8b877e\" (UID: \"72f72bc8-0d67-4200-ab31-c167fe8b877e\") " Nov 03 01:04:53.975996 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:53.975717 4421 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72f72bc8-0d67-4200-ab31-c167fe8b877e-istiod-ca-cert\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:04:53.975996 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:53.975742 4421 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72f72bc8-0d67-4200-ab31-c167fe8b877e-istio-data\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:04:53.975996 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:53.975758 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zctn\" (UniqueName: \"kubernetes.io/projected/32b8c1b0-8e81-4430-840b-ca5d03a7195d-kube-api-access-7zctn\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:04:53.975996 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:53.975774 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32b8c1b0-8e81-4430-840b-ca5d03a7195d-maas-api-xks-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:04:53.975996 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:53.975788 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32b8c1b0-8e81-4430-840b-ca5d03a7195d-maas-api-openshift-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:04:53.976568 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:53.976244 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72f72bc8-0d67-4200-ab31-c167fe8b877e-workload-certs" pod "72f72bc8-0d67-4200-ab31-c167fe8b877e" (UID: "72f72bc8-0d67-4200-ab31-c167fe8b877e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:04:53.976769 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:53.976680 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72f72bc8-0d67-4200-ab31-c167fe8b877e-credential-socket" pod "72f72bc8-0d67-4200-ab31-c167fe8b877e" (UID: "72f72bc8-0d67-4200-ab31-c167fe8b877e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:04:53.976769 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:53.976744 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72f72bc8-0d67-4200-ab31-c167fe8b877e-workload-socket" pod "72f72bc8-0d67-4200-ab31-c167fe8b877e" (UID: "72f72bc8-0d67-4200-ab31-c167fe8b877e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:04:53.978473 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:53.978439 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/72f72bc8-0d67-4200-ab31-c167fe8b877e-istio-podinfo" pod "72f72bc8-0d67-4200-ab31-c167fe8b877e" (UID: "72f72bc8-0d67-4200-ab31-c167fe8b877e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:04:53.979213 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:53.979183 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72f72bc8-0d67-4200-ab31-c167fe8b877e-kube-api-access-rgzdg" pod "72f72bc8-0d67-4200-ab31-c167fe8b877e" (UID: "72f72bc8-0d67-4200-ab31-c167fe8b877e"). InnerVolumeSpecName "kube-api-access-rgzdg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:04:53.980562 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:53.980539 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72f72bc8-0d67-4200-ab31-c167fe8b877e-istio-envoy" pod "72f72bc8-0d67-4200-ab31-c167fe8b877e" (UID: "72f72bc8-0d67-4200-ab31-c167fe8b877e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:04:53.980851 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:53.980830 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72f72bc8-0d67-4200-ab31-c167fe8b877e-istio-token" pod "72f72bc8-0d67-4200-ab31-c167fe8b877e" (UID: "72f72bc8-0d67-4200-ab31-c167fe8b877e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:04:53.999476 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:53.999437 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-208a0d-openshift-default-67457fb65c-ksfzw"] Nov 03 01:04:54.005331 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:54.005305 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-208a0d-openshift-default-67457fb65c-ksfzw"] Nov 03 01:04:54.077701 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:54.077606 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rgzdg\" (UniqueName: \"kubernetes.io/projected/72f72bc8-0d67-4200-ab31-c167fe8b877e-kube-api-access-rgzdg\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:04:54.077701 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:54.077641 4421 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72f72bc8-0d67-4200-ab31-c167fe8b877e-istio-podinfo\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:04:54.077701 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:54.077671 4421 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72f72bc8-0d67-4200-ab31-c167fe8b877e-workload-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:04:54.077701 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:54.077691 4421 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72f72bc8-0d67-4200-ab31-c167fe8b877e-workload-certs\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:04:54.078175 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:54.077707 4421 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72f72bc8-0d67-4200-ab31-c167fe8b877e-credential-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:04:54.078175 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:54.077722 4421 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72f72bc8-0d67-4200-ab31-c167fe8b877e-istio-envoy\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:04:54.078175 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:54.077739 4421 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72f72bc8-0d67-4200-ab31-c167fe8b877e-istio-token\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:04:54.678212 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:54.678184 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97d101cc-vf5cc" Nov 03 01:04:54.678848 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:54.678195 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97d101cc-vf5cc" event={"ID":"32b8c1b0-8e81-4430-840b-ca5d03a7195d","Type":"ContainerDied","Data":"915d8218785565736f11733a6ee729817ba7c49411241838a4110a11f8062d95"} Nov 03 01:04:54.679645 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:54.678877 4421 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="915d8218785565736f11733a6ee729817ba7c49411241838a4110a11f8062d95" Nov 03 01:04:54.681507 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:54.681474 4421 generic.go:356] "Generic (PLEG): container finished" podID="72f72bc8-0d67-4200-ab31-c167fe8b877e" containerID="aff03fc28fb084070ff8360f656037f46775f47711662b3bf3b6b035a848967e" exitCode=0 Nov 03 01:04:54.681710 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:54.681685 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-9jh88" event={"ID":"72f72bc8-0d67-4200-ab31-c167fe8b877e","Type":"ContainerDied","Data":"aff03fc28fb084070ff8360f656037f46775f47711662b3bf3b6b035a848967e"} Nov 03 01:04:54.681902 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:54.681880 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-9jh88" Nov 03 01:04:54.681990 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:54.681970 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-9jh88" event={"ID":"72f72bc8-0d67-4200-ab31-c167fe8b877e","Type":"ContainerDied","Data":"66cadfb8389ee7be6f698bebe30d5d456745f43c5deaed3e5854d6e25e1e32c2"} Nov 03 01:04:54.682044 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:54.682014 4421 scope.go:122] "RemoveContainer" containerID="aff03fc28fb084070ff8360f656037f46775f47711662b3bf3b6b035a848967e" Nov 03 01:04:54.694311 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:54.694260 4421 scope.go:122] "RemoveContainer" containerID="aff03fc28fb084070ff8360f656037f46775f47711662b3bf3b6b035a848967e" Nov 03 01:04:54.694661 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:04:54.694638 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aff03fc28fb084070ff8360f656037f46775f47711662b3bf3b6b035a848967e\": container with ID starting with aff03fc28fb084070ff8360f656037f46775f47711662b3bf3b6b035a848967e not found: ID does not exist" containerID="aff03fc28fb084070ff8360f656037f46775f47711662b3bf3b6b035a848967e" Nov 03 01:04:54.694774 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:54.694680 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aff03fc28fb084070ff8360f656037f46775f47711662b3bf3b6b035a848967e"} err="failed to get container status \"aff03fc28fb084070ff8360f656037f46775f47711662b3bf3b6b035a848967e\": rpc error: code = NotFound desc = could not find container \"aff03fc28fb084070ff8360f656037f46775f47711662b3bf3b6b035a848967e\": container with ID starting with aff03fc28fb084070ff8360f656037f46775f47711662b3bf3b6b035a848967e not found: ID does not exist" Nov 03 01:04:54.711967 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:54.711935 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-9jh88"] Nov 03 01:04:54.718378 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:54.718349 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-9jh88"] Nov 03 01:04:55.162267 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:55.162229 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-tt9jg" Nov 03 01:04:55.163815 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:55.163792 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-tt9jg" Nov 03 01:04:55.164678 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:55.164654 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-tt9jg" Nov 03 01:04:55.369575 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:55.369535 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c50969f-s5cbm"] Nov 03 01:04:55.370501 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:55.370480 4421 cpu_manager.go:414] "containerMap: removing container" podUID="72f72bc8-0d67-4200-ab31-c167fe8b877e" containerName="istio-proxy" Nov 03 01:04:55.370612 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:55.370504 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72f72bc8-0d67-4200-ab31-c167fe8b877e" containerName="istio-proxy" Nov 03 01:04:55.370612 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:55.370531 4421 cpu_manager.go:414] "containerMap: removing container" podUID="bf46dbdd-d501-400c-b82a-aaf689363bab" containerName="revoke-keys" Nov 03 01:04:55.370612 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:55.370538 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf46dbdd-d501-400c-b82a-aaf689363bab" containerName="revoke-keys" Nov 03 01:04:55.370612 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:55.370572 4421 cpu_manager.go:414] "containerMap: removing container" podUID="a47649ab-48b3-4de0-b057-5445c923c427" containerName="istio-proxy" Nov 03 01:04:55.370612 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:55.370579 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a47649ab-48b3-4de0-b057-5445c923c427" containerName="istio-proxy" Nov 03 01:04:55.370612 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:55.370589 4421 cpu_manager.go:414] "containerMap: removing container" podUID="32b8c1b0-8e81-4430-840b-ca5d03a7195d" containerName="revoke-keys" Nov 03 01:04:55.370612 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:55.370595 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32b8c1b0-8e81-4430-840b-ca5d03a7195d" containerName="revoke-keys" Nov 03 01:04:55.370829 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:55.370626 4421 cpu_manager.go:414] "containerMap: removing container" podUID="a47649ab-48b3-4de0-b057-5445c923c427" containerName="istio-proxy" Nov 03 01:04:55.370829 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:55.370633 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a47649ab-48b3-4de0-b057-5445c923c427" containerName="istio-proxy" Nov 03 01:04:55.370829 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:55.370790 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="a47649ab-48b3-4de0-b057-5445c923c427" containerName="istio-proxy" Nov 03 01:04:55.370829 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:55.370807 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="72f72bc8-0d67-4200-ab31-c167fe8b877e" containerName="istio-proxy" Nov 03 01:04:55.370829 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:55.370817 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf46dbdd-d501-400c-b82a-aaf689363bab" containerName="revoke-keys" Nov 03 01:04:55.370829 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:55.370828 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="32b8c1b0-8e81-4430-840b-ca5d03a7195d" containerName="revoke-keys" Nov 03 01:04:55.376843 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:55.376820 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c50969f-s5cbm" Nov 03 01:04:55.382385 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:55.382337 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c50969f-s5cbm"] Nov 03 01:04:55.382719 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:55.382702 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6q4xp\"" Nov 03 01:04:55.382839 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:55.382808 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 01:04:55.382960 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:55.382857 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:04:55.382960 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:55.382939 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 01:04:55.417156 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:55.415755 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8a50bdf-26dd-49d9-a899-a9dfec083da4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9c50969f-s5cbm\" (UID: \"f8a50bdf-26dd-49d9-a899-a9dfec083da4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c50969f-s5cbm" Nov 03 01:04:55.417156 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:55.415947 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8a50bdf-26dd-49d9-a899-a9dfec083da4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9c50969f-s5cbm\" (UID: \"f8a50bdf-26dd-49d9-a899-a9dfec083da4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c50969f-s5cbm" Nov 03 01:04:55.417156 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:55.416103 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47mqj\" (UniqueName: \"kubernetes.io/projected/f8a50bdf-26dd-49d9-a899-a9dfec083da4-kube-api-access-47mqj\") pod \"maas-api-revoke-sub-9c50969f-s5cbm\" (UID: \"f8a50bdf-26dd-49d9-a899-a9dfec083da4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c50969f-s5cbm" Nov 03 01:04:55.517402 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:55.517361 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8a50bdf-26dd-49d9-a899-a9dfec083da4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9c50969f-s5cbm\" (UID: \"f8a50bdf-26dd-49d9-a899-a9dfec083da4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c50969f-s5cbm" Nov 03 01:04:55.518596 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:55.518562 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8a50bdf-26dd-49d9-a899-a9dfec083da4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9c50969f-s5cbm\" (UID: \"f8a50bdf-26dd-49d9-a899-a9dfec083da4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c50969f-s5cbm" Nov 03 01:04:55.518742 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:55.518699 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47mqj\" (UniqueName: \"kubernetes.io/projected/f8a50bdf-26dd-49d9-a899-a9dfec083da4-kube-api-access-47mqj\") pod \"maas-api-revoke-sub-9c50969f-s5cbm\" (UID: \"f8a50bdf-26dd-49d9-a899-a9dfec083da4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c50969f-s5cbm" Nov 03 01:04:55.519433 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:55.519410 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8a50bdf-26dd-49d9-a899-a9dfec083da4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9c50969f-s5cbm\" (UID: \"f8a50bdf-26dd-49d9-a899-a9dfec083da4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c50969f-s5cbm" Nov 03 01:04:55.523185 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:55.523157 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8a50bdf-26dd-49d9-a899-a9dfec083da4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9c50969f-s5cbm\" (UID: \"f8a50bdf-26dd-49d9-a899-a9dfec083da4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c50969f-s5cbm" Nov 03 01:04:55.528771 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:55.528742 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47mqj\" (UniqueName: \"kubernetes.io/projected/f8a50bdf-26dd-49d9-a899-a9dfec083da4-kube-api-access-47mqj\") pod \"maas-api-revoke-sub-9c50969f-s5cbm\" (UID: \"f8a50bdf-26dd-49d9-a899-a9dfec083da4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c50969f-s5cbm" Nov 03 01:04:55.770651 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:55.770613 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c50969f-s5cbm" Nov 03 01:04:55.982315 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:55.982259 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72f72bc8-0d67-4200-ab31-c167fe8b877e" path="/var/lib/kubelet/pods/72f72bc8-0d67-4200-ab31-c167fe8b877e/volumes" Nov 03 01:04:55.983057 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:55.983037 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a47649ab-48b3-4de0-b057-5445c923c427" path="/var/lib/kubelet/pods/a47649ab-48b3-4de0-b057-5445c923c427/volumes" Nov 03 01:04:56.054448 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:56.054415 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c50969f-s5cbm"] Nov 03 01:04:56.055200 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:04:56.055174 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8a50bdf_26dd_49d9_a899_a9dfec083da4.slice/crio-f397f875897ec8752079edef541d30b88b309c0da0d038704b0023de258f01e5 WatchSource:0}: Error finding container f397f875897ec8752079edef541d30b88b309c0da0d038704b0023de258f01e5: Status 404 returned error can't find the container with id f397f875897ec8752079edef541d30b88b309c0da0d038704b0023de258f01e5 Nov 03 01:04:56.702746 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:56.702638 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c50969f-s5cbm" event={"ID":"f8a50bdf-26dd-49d9-a899-a9dfec083da4","Type":"ContainerStarted","Data":"82e2680ea46199a1dc1751a9f8eb3f983b8297632bf5a133923e9474e90f101f"} Nov 03 01:04:56.702746 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:56.702708 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c50969f-s5cbm" event={"ID":"f8a50bdf-26dd-49d9-a899-a9dfec083da4","Type":"ContainerStarted","Data":"f397f875897ec8752079edef541d30b88b309c0da0d038704b0023de258f01e5"} Nov 03 01:04:56.720598 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:56.720545 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c50969f-s5cbm" podStartSLOduration=1.720530427 podStartE2EDuration="1.720530427s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:04:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:04:56.719824644 +0000 UTC m=+1438.890827741" watchObservedRunningTime="2026-11-03 01:04:56.720530427 +0000 UTC m=+1438.891533523" Nov 03 01:04:57.712999 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:57.712966 4421 generic.go:356] "Generic (PLEG): container finished" podID="f8a50bdf-26dd-49d9-a899-a9dfec083da4" containerID="82e2680ea46199a1dc1751a9f8eb3f983b8297632bf5a133923e9474e90f101f" exitCode=0 Nov 03 01:04:57.713472 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:57.713059 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c50969f-s5cbm" event={"ID":"f8a50bdf-26dd-49d9-a899-a9dfec083da4","Type":"ContainerDied","Data":"82e2680ea46199a1dc1751a9f8eb3f983b8297632bf5a133923e9474e90f101f"} Nov 03 01:04:58.872497 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:58.872467 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c50969f-s5cbm" Nov 03 01:04:58.997790 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:58.997753 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f8a50bdf-26dd-49d9-a899-a9dfec083da4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8a50bdf-26dd-49d9-a899-a9dfec083da4-maas-api-xks-ca\") pod \"f8a50bdf-26dd-49d9-a899-a9dfec083da4\" (UID: \"f8a50bdf-26dd-49d9-a899-a9dfec083da4\") " Nov 03 01:04:58.998012 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:58.997868 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8a50bdf-26dd-49d9-a899-a9dfec083da4-kube-api-access-47mqj\" (UniqueName: \"kubernetes.io/projected/f8a50bdf-26dd-49d9-a899-a9dfec083da4-kube-api-access-47mqj\") pod \"f8a50bdf-26dd-49d9-a899-a9dfec083da4\" (UID: \"f8a50bdf-26dd-49d9-a899-a9dfec083da4\") " Nov 03 01:04:58.998012 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:58.997960 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8a50bdf-26dd-49d9-a899-a9dfec083da4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8a50bdf-26dd-49d9-a899-a9dfec083da4-maas-api-openshift-ca\") pod \"f8a50bdf-26dd-49d9-a899-a9dfec083da4\" (UID: \"f8a50bdf-26dd-49d9-a899-a9dfec083da4\") " Nov 03 01:04:58.998783 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:58.998750 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8a50bdf-26dd-49d9-a899-a9dfec083da4-maas-api-openshift-ca" pod "f8a50bdf-26dd-49d9-a899-a9dfec083da4" (UID: "f8a50bdf-26dd-49d9-a899-a9dfec083da4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:04:59.000637 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:59.000565 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8a50bdf-26dd-49d9-a899-a9dfec083da4-maas-api-xks-ca" pod "f8a50bdf-26dd-49d9-a899-a9dfec083da4" (UID: "f8a50bdf-26dd-49d9-a899-a9dfec083da4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:04:59.001056 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:59.001031 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8a50bdf-26dd-49d9-a899-a9dfec083da4-kube-api-access-47mqj" pod "f8a50bdf-26dd-49d9-a899-a9dfec083da4" (UID: "f8a50bdf-26dd-49d9-a899-a9dfec083da4"). InnerVolumeSpecName "kube-api-access-47mqj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:04:59.099799 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:59.099759 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8a50bdf-26dd-49d9-a899-a9dfec083da4-maas-api-xks-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:04:59.099799 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:59.099794 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47mqj\" (UniqueName: \"kubernetes.io/projected/f8a50bdf-26dd-49d9-a899-a9dfec083da4-kube-api-access-47mqj\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:04:59.100022 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:59.099810 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8a50bdf-26dd-49d9-a899-a9dfec083da4-maas-api-openshift-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:04:59.761583 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:59.761544 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c50969f-s5cbm" event={"ID":"f8a50bdf-26dd-49d9-a899-a9dfec083da4","Type":"ContainerDied","Data":"f397f875897ec8752079edef541d30b88b309c0da0d038704b0023de258f01e5"} Nov 03 01:04:59.761802 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:59.761606 4421 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f397f875897ec8752079edef541d30b88b309c0da0d038704b0023de258f01e5" Nov 03 01:04:59.761802 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:04:59.761675 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c50969f-s5cbm" Nov 03 01:05:00.010333 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:00.010297 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-gwl6k"] Nov 03 01:05:00.011240 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:00.011220 4421 cpu_manager.go:414] "containerMap: removing container" podUID="f8a50bdf-26dd-49d9-a899-a9dfec083da4" containerName="revoke-keys" Nov 03 01:05:00.011463 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:00.011241 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8a50bdf-26dd-49d9-a899-a9dfec083da4" containerName="revoke-keys" Nov 03 01:05:00.011463 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:00.011457 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8a50bdf-26dd-49d9-a899-a9dfec083da4" containerName="revoke-keys" Nov 03 01:05:00.011680 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:00.011479 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="a47649ab-48b3-4de0-b057-5445c923c427" containerName="istio-proxy" Nov 03 01:05:00.016127 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:00.015718 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-gwl6k" Nov 03 01:05:00.024646 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:00.024605 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-717217-openshift-default-dockercfg-4kr2v\"" Nov 03 01:05:00.029443 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:00.029415 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-gwl6k"] Nov 03 01:05:00.117063 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:00.117025 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6c36e3a-89e3-4f54-8266-c7dcfc0ca19b-credential-socket\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-gwl6k\" (UID: \"e6c36e3a-89e3-4f54-8266-c7dcfc0ca19b\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-gwl6k" Nov 03 01:05:00.117213 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:00.117082 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhlzz\" (UniqueName: \"kubernetes.io/projected/e6c36e3a-89e3-4f54-8266-c7dcfc0ca19b-kube-api-access-bhlzz\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-gwl6k\" (UID: \"e6c36e3a-89e3-4f54-8266-c7dcfc0ca19b\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-gwl6k" Nov 03 01:05:00.117213 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:00.117178 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6c36e3a-89e3-4f54-8266-c7dcfc0ca19b-istio-envoy\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-gwl6k\" (UID: \"e6c36e3a-89e3-4f54-8266-c7dcfc0ca19b\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-gwl6k" Nov 03 01:05:00.117213 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:00.117206 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6c36e3a-89e3-4f54-8266-c7dcfc0ca19b-istio-token\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-gwl6k\" (UID: \"e6c36e3a-89e3-4f54-8266-c7dcfc0ca19b\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-gwl6k" Nov 03 01:05:00.117349 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:00.117234 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6c36e3a-89e3-4f54-8266-c7dcfc0ca19b-istio-podinfo\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-gwl6k\" (UID: \"e6c36e3a-89e3-4f54-8266-c7dcfc0ca19b\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-gwl6k" Nov 03 01:05:00.117349 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:00.117324 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6c36e3a-89e3-4f54-8266-c7dcfc0ca19b-workload-socket\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-gwl6k\" (UID: \"e6c36e3a-89e3-4f54-8266-c7dcfc0ca19b\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-gwl6k" Nov 03 01:05:00.117410 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:00.117392 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6c36e3a-89e3-4f54-8266-c7dcfc0ca19b-workload-certs\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-gwl6k\" (UID: \"e6c36e3a-89e3-4f54-8266-c7dcfc0ca19b\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-gwl6k" Nov 03 01:05:00.117452 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:00.117415 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6c36e3a-89e3-4f54-8266-c7dcfc0ca19b-istio-data\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-gwl6k\" (UID: \"e6c36e3a-89e3-4f54-8266-c7dcfc0ca19b\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-gwl6k" Nov 03 01:05:00.117497 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:00.117484 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6c36e3a-89e3-4f54-8266-c7dcfc0ca19b-istiod-ca-cert\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-gwl6k\" (UID: \"e6c36e3a-89e3-4f54-8266-c7dcfc0ca19b\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-gwl6k" Nov 03 01:05:00.225463 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:00.221169 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6c36e3a-89e3-4f54-8266-c7dcfc0ca19b-workload-certs\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-gwl6k\" (UID: \"e6c36e3a-89e3-4f54-8266-c7dcfc0ca19b\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-gwl6k" Nov 03 01:05:00.225463 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:00.221226 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6c36e3a-89e3-4f54-8266-c7dcfc0ca19b-istio-data\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-gwl6k\" (UID: \"e6c36e3a-89e3-4f54-8266-c7dcfc0ca19b\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-gwl6k" Nov 03 01:05:00.225463 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:00.221337 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6c36e3a-89e3-4f54-8266-c7dcfc0ca19b-istiod-ca-cert\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-gwl6k\" (UID: \"e6c36e3a-89e3-4f54-8266-c7dcfc0ca19b\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-gwl6k" Nov 03 01:05:00.225463 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:00.221503 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6c36e3a-89e3-4f54-8266-c7dcfc0ca19b-credential-socket\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-gwl6k\" (UID: \"e6c36e3a-89e3-4f54-8266-c7dcfc0ca19b\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-gwl6k" Nov 03 01:05:00.225463 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:00.221550 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhlzz\" (UniqueName: \"kubernetes.io/projected/e6c36e3a-89e3-4f54-8266-c7dcfc0ca19b-kube-api-access-bhlzz\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-gwl6k\" (UID: \"e6c36e3a-89e3-4f54-8266-c7dcfc0ca19b\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-gwl6k" Nov 03 01:05:00.225463 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:00.221620 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6c36e3a-89e3-4f54-8266-c7dcfc0ca19b-istio-envoy\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-gwl6k\" (UID: \"e6c36e3a-89e3-4f54-8266-c7dcfc0ca19b\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-gwl6k" Nov 03 01:05:00.225463 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:00.221670 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6c36e3a-89e3-4f54-8266-c7dcfc0ca19b-istio-token\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-gwl6k\" (UID: \"e6c36e3a-89e3-4f54-8266-c7dcfc0ca19b\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-gwl6k" Nov 03 01:05:00.225463 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:00.221715 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6c36e3a-89e3-4f54-8266-c7dcfc0ca19b-istio-podinfo\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-gwl6k\" (UID: \"e6c36e3a-89e3-4f54-8266-c7dcfc0ca19b\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-gwl6k" Nov 03 01:05:00.225463 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:00.221745 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6c36e3a-89e3-4f54-8266-c7dcfc0ca19b-workload-socket\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-gwl6k\" (UID: \"e6c36e3a-89e3-4f54-8266-c7dcfc0ca19b\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-gwl6k" Nov 03 01:05:00.225463 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:00.222200 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6c36e3a-89e3-4f54-8266-c7dcfc0ca19b-workload-socket\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-gwl6k\" (UID: \"e6c36e3a-89e3-4f54-8266-c7dcfc0ca19b\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-gwl6k" Nov 03 01:05:00.225463 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:00.222492 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6c36e3a-89e3-4f54-8266-c7dcfc0ca19b-workload-certs\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-gwl6k\" (UID: \"e6c36e3a-89e3-4f54-8266-c7dcfc0ca19b\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-gwl6k" Nov 03 01:05:00.225463 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:00.222743 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6c36e3a-89e3-4f54-8266-c7dcfc0ca19b-istio-data\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-gwl6k\" (UID: \"e6c36e3a-89e3-4f54-8266-c7dcfc0ca19b\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-gwl6k" Nov 03 01:05:00.225463 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:00.223449 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6c36e3a-89e3-4f54-8266-c7dcfc0ca19b-istiod-ca-cert\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-gwl6k\" (UID: \"e6c36e3a-89e3-4f54-8266-c7dcfc0ca19b\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-gwl6k" Nov 03 01:05:00.225463 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:00.223703 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6c36e3a-89e3-4f54-8266-c7dcfc0ca19b-credential-socket\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-gwl6k\" (UID: \"e6c36e3a-89e3-4f54-8266-c7dcfc0ca19b\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-gwl6k" Nov 03 01:05:00.246328 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:00.246251 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6c36e3a-89e3-4f54-8266-c7dcfc0ca19b-istio-token\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-gwl6k\" (UID: \"e6c36e3a-89e3-4f54-8266-c7dcfc0ca19b\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-gwl6k" Nov 03 01:05:00.260918 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:00.260846 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6c36e3a-89e3-4f54-8266-c7dcfc0ca19b-istio-envoy\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-gwl6k\" (UID: \"e6c36e3a-89e3-4f54-8266-c7dcfc0ca19b\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-gwl6k" Nov 03 01:05:00.275593 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:00.275514 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhlzz\" (UniqueName: \"kubernetes.io/projected/e6c36e3a-89e3-4f54-8266-c7dcfc0ca19b-kube-api-access-bhlzz\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-gwl6k\" (UID: \"e6c36e3a-89e3-4f54-8266-c7dcfc0ca19b\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-gwl6k" Nov 03 01:05:00.283390 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:00.282739 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6c36e3a-89e3-4f54-8266-c7dcfc0ca19b-istio-podinfo\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-gwl6k\" (UID: \"e6c36e3a-89e3-4f54-8266-c7dcfc0ca19b\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-gwl6k" Nov 03 01:05:00.339637 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:00.339376 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-gwl6k" Nov 03 01:05:00.643560 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:05:00.643533 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6c36e3a_89e3_4f54_8266_c7dcfc0ca19b.slice/crio-b1e181a1e651c1a44f62794edcbab0f86277704334d2ba45f7789760491d7b87 WatchSource:0}: Error finding container b1e181a1e651c1a44f62794edcbab0f86277704334d2ba45f7789760491d7b87: Status 404 returned error can't find the container with id b1e181a1e651c1a44f62794edcbab0f86277704334d2ba45f7789760491d7b87 Nov 03 01:05:00.643940 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:00.643916 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-gwl6k"] Nov 03 01:05:00.646260 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:00.646223 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:05:00.646448 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:00.646337 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:05:00.646448 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:00.646370 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:05:00.774444 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:00.774396 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-gwl6k" event={"ID":"e6c36e3a-89e3-4f54-8266-c7dcfc0ca19b","Type":"ContainerStarted","Data":"35d6c0da9bf4ae0002ba6cde6d7fcb78cc529f65db86aca46c4b5fe17840a493"} Nov 03 01:05:00.774610 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:00.774458 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-gwl6k" event={"ID":"e6c36e3a-89e3-4f54-8266-c7dcfc0ca19b","Type":"ContainerStarted","Data":"b1e181a1e651c1a44f62794edcbab0f86277704334d2ba45f7789760491d7b87"} Nov 03 01:05:00.795289 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:00.795217 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-gwl6k" podStartSLOduration=1.795199077 podStartE2EDuration="1.795199077s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:04:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:05:00.791681466 +0000 UTC m=+1442.962684563" watchObservedRunningTime="2026-11-03 01:05:00.795199077 +0000 UTC m=+1442.966202173" Nov 03 01:05:01.340389 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:01.340355 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-gwl6k" Nov 03 01:05:01.346651 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:01.346623 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-gwl6k" Nov 03 01:05:01.782120 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:01.782089 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-gwl6k" Nov 03 01:05:01.783051 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:01.783024 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-gwl6k" Nov 03 01:05:03.254472 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:03.254399 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-4248ae-openshift-default-94bc8c67-b5pgk"] Nov 03 01:05:03.260612 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:03.260580 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-4248ae-openshift-default-94bc8c67-b5pgk" Nov 03 01:05:03.286744 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:03.286569 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-4248ae-openshift-default-dockercfg-2tk48\"" Nov 03 01:05:03.330313 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:03.303602 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-4248ae-openshift-default-94bc8c67-b5pgk"] Nov 03 01:05:03.446442 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:03.411472 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2afb12c9-7c7a-4527-95c4-0b2febd22e90-istiod-ca-cert\") pod \"e2e-autores-4248ae-openshift-default-94bc8c67-b5pgk\" (UID: \"2afb12c9-7c7a-4527-95c4-0b2febd22e90\") " pod="openshift-ingress/e2e-autores-4248ae-openshift-default-94bc8c67-b5pgk" Nov 03 01:05:03.446442 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:03.411640 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2afb12c9-7c7a-4527-95c4-0b2febd22e90-istio-token\") pod \"e2e-autores-4248ae-openshift-default-94bc8c67-b5pgk\" (UID: \"2afb12c9-7c7a-4527-95c4-0b2febd22e90\") " pod="openshift-ingress/e2e-autores-4248ae-openshift-default-94bc8c67-b5pgk" Nov 03 01:05:03.446442 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:03.411709 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwz2h\" (UniqueName: \"kubernetes.io/projected/2afb12c9-7c7a-4527-95c4-0b2febd22e90-kube-api-access-vwz2h\") pod \"e2e-autores-4248ae-openshift-default-94bc8c67-b5pgk\" (UID: \"2afb12c9-7c7a-4527-95c4-0b2febd22e90\") " pod="openshift-ingress/e2e-autores-4248ae-openshift-default-94bc8c67-b5pgk" Nov 03 01:05:03.446442 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:03.411823 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2afb12c9-7c7a-4527-95c4-0b2febd22e90-credential-socket\") pod \"e2e-autores-4248ae-openshift-default-94bc8c67-b5pgk\" (UID: \"2afb12c9-7c7a-4527-95c4-0b2febd22e90\") " pod="openshift-ingress/e2e-autores-4248ae-openshift-default-94bc8c67-b5pgk" Nov 03 01:05:03.446442 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:03.411848 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2afb12c9-7c7a-4527-95c4-0b2febd22e90-workload-certs\") pod \"e2e-autores-4248ae-openshift-default-94bc8c67-b5pgk\" (UID: \"2afb12c9-7c7a-4527-95c4-0b2febd22e90\") " pod="openshift-ingress/e2e-autores-4248ae-openshift-default-94bc8c67-b5pgk" Nov 03 01:05:03.446442 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:03.411885 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2afb12c9-7c7a-4527-95c4-0b2febd22e90-istio-podinfo\") pod \"e2e-autores-4248ae-openshift-default-94bc8c67-b5pgk\" (UID: \"2afb12c9-7c7a-4527-95c4-0b2febd22e90\") " pod="openshift-ingress/e2e-autores-4248ae-openshift-default-94bc8c67-b5pgk" Nov 03 01:05:03.446442 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:03.411913 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2afb12c9-7c7a-4527-95c4-0b2febd22e90-istio-envoy\") pod \"e2e-autores-4248ae-openshift-default-94bc8c67-b5pgk\" (UID: \"2afb12c9-7c7a-4527-95c4-0b2febd22e90\") " pod="openshift-ingress/e2e-autores-4248ae-openshift-default-94bc8c67-b5pgk" Nov 03 01:05:03.446442 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:03.411978 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2afb12c9-7c7a-4527-95c4-0b2febd22e90-workload-socket\") pod \"e2e-autores-4248ae-openshift-default-94bc8c67-b5pgk\" (UID: \"2afb12c9-7c7a-4527-95c4-0b2febd22e90\") " pod="openshift-ingress/e2e-autores-4248ae-openshift-default-94bc8c67-b5pgk" Nov 03 01:05:03.446442 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:03.412114 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2afb12c9-7c7a-4527-95c4-0b2febd22e90-istio-data\") pod \"e2e-autores-4248ae-openshift-default-94bc8c67-b5pgk\" (UID: \"2afb12c9-7c7a-4527-95c4-0b2febd22e90\") " pod="openshift-ingress/e2e-autores-4248ae-openshift-default-94bc8c67-b5pgk" Nov 03 01:05:03.591361 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:03.587711 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2afb12c9-7c7a-4527-95c4-0b2febd22e90-workload-socket\") pod \"e2e-autores-4248ae-openshift-default-94bc8c67-b5pgk\" (UID: \"2afb12c9-7c7a-4527-95c4-0b2febd22e90\") " pod="openshift-ingress/e2e-autores-4248ae-openshift-default-94bc8c67-b5pgk" Nov 03 01:05:03.591361 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:03.587841 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2afb12c9-7c7a-4527-95c4-0b2febd22e90-istio-data\") pod \"e2e-autores-4248ae-openshift-default-94bc8c67-b5pgk\" (UID: \"2afb12c9-7c7a-4527-95c4-0b2febd22e90\") " pod="openshift-ingress/e2e-autores-4248ae-openshift-default-94bc8c67-b5pgk" Nov 03 01:05:03.591361 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:03.587925 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2afb12c9-7c7a-4527-95c4-0b2febd22e90-istiod-ca-cert\") pod \"e2e-autores-4248ae-openshift-default-94bc8c67-b5pgk\" (UID: \"2afb12c9-7c7a-4527-95c4-0b2febd22e90\") " pod="openshift-ingress/e2e-autores-4248ae-openshift-default-94bc8c67-b5pgk" Nov 03 01:05:03.591361 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:03.587964 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2afb12c9-7c7a-4527-95c4-0b2febd22e90-istio-token\") pod \"e2e-autores-4248ae-openshift-default-94bc8c67-b5pgk\" (UID: \"2afb12c9-7c7a-4527-95c4-0b2febd22e90\") " pod="openshift-ingress/e2e-autores-4248ae-openshift-default-94bc8c67-b5pgk" Nov 03 01:05:03.591361 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:03.588011 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwz2h\" (UniqueName: \"kubernetes.io/projected/2afb12c9-7c7a-4527-95c4-0b2febd22e90-kube-api-access-vwz2h\") pod \"e2e-autores-4248ae-openshift-default-94bc8c67-b5pgk\" (UID: \"2afb12c9-7c7a-4527-95c4-0b2febd22e90\") " pod="openshift-ingress/e2e-autores-4248ae-openshift-default-94bc8c67-b5pgk" Nov 03 01:05:03.591361 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:03.588102 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2afb12c9-7c7a-4527-95c4-0b2febd22e90-credential-socket\") pod \"e2e-autores-4248ae-openshift-default-94bc8c67-b5pgk\" (UID: \"2afb12c9-7c7a-4527-95c4-0b2febd22e90\") " pod="openshift-ingress/e2e-autores-4248ae-openshift-default-94bc8c67-b5pgk" Nov 03 01:05:03.591361 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:03.588121 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2afb12c9-7c7a-4527-95c4-0b2febd22e90-workload-certs\") pod \"e2e-autores-4248ae-openshift-default-94bc8c67-b5pgk\" (UID: \"2afb12c9-7c7a-4527-95c4-0b2febd22e90\") " pod="openshift-ingress/e2e-autores-4248ae-openshift-default-94bc8c67-b5pgk" Nov 03 01:05:03.591361 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:03.588158 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2afb12c9-7c7a-4527-95c4-0b2febd22e90-istio-podinfo\") pod \"e2e-autores-4248ae-openshift-default-94bc8c67-b5pgk\" (UID: \"2afb12c9-7c7a-4527-95c4-0b2febd22e90\") " pod="openshift-ingress/e2e-autores-4248ae-openshift-default-94bc8c67-b5pgk" Nov 03 01:05:03.591361 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:03.588182 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2afb12c9-7c7a-4527-95c4-0b2febd22e90-istio-envoy\") pod \"e2e-autores-4248ae-openshift-default-94bc8c67-b5pgk\" (UID: \"2afb12c9-7c7a-4527-95c4-0b2febd22e90\") " pod="openshift-ingress/e2e-autores-4248ae-openshift-default-94bc8c67-b5pgk" Nov 03 01:05:03.658551 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:03.594444 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2afb12c9-7c7a-4527-95c4-0b2febd22e90-istio-data\") pod \"e2e-autores-4248ae-openshift-default-94bc8c67-b5pgk\" (UID: \"2afb12c9-7c7a-4527-95c4-0b2febd22e90\") " pod="openshift-ingress/e2e-autores-4248ae-openshift-default-94bc8c67-b5pgk" Nov 03 01:05:03.658551 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:03.595139 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2afb12c9-7c7a-4527-95c4-0b2febd22e90-istiod-ca-cert\") pod \"e2e-autores-4248ae-openshift-default-94bc8c67-b5pgk\" (UID: \"2afb12c9-7c7a-4527-95c4-0b2febd22e90\") " pod="openshift-ingress/e2e-autores-4248ae-openshift-default-94bc8c67-b5pgk" Nov 03 01:05:03.658551 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:03.595930 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2afb12c9-7c7a-4527-95c4-0b2febd22e90-credential-socket\") pod \"e2e-autores-4248ae-openshift-default-94bc8c67-b5pgk\" (UID: \"2afb12c9-7c7a-4527-95c4-0b2febd22e90\") " pod="openshift-ingress/e2e-autores-4248ae-openshift-default-94bc8c67-b5pgk" Nov 03 01:05:03.658551 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:03.596204 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2afb12c9-7c7a-4527-95c4-0b2febd22e90-workload-certs\") pod \"e2e-autores-4248ae-openshift-default-94bc8c67-b5pgk\" (UID: \"2afb12c9-7c7a-4527-95c4-0b2febd22e90\") " pod="openshift-ingress/e2e-autores-4248ae-openshift-default-94bc8c67-b5pgk" Nov 03 01:05:03.658551 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:03.617616 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2afb12c9-7c7a-4527-95c4-0b2febd22e90-istio-envoy\") pod \"e2e-autores-4248ae-openshift-default-94bc8c67-b5pgk\" (UID: \"2afb12c9-7c7a-4527-95c4-0b2febd22e90\") " pod="openshift-ingress/e2e-autores-4248ae-openshift-default-94bc8c67-b5pgk" Nov 03 01:05:03.658551 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:03.652108 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2afb12c9-7c7a-4527-95c4-0b2febd22e90-workload-socket\") pod \"e2e-autores-4248ae-openshift-default-94bc8c67-b5pgk\" (UID: \"2afb12c9-7c7a-4527-95c4-0b2febd22e90\") " pod="openshift-ingress/e2e-autores-4248ae-openshift-default-94bc8c67-b5pgk" Nov 03 01:05:03.739896 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:03.728770 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2afb12c9-7c7a-4527-95c4-0b2febd22e90-istio-podinfo\") pod \"e2e-autores-4248ae-openshift-default-94bc8c67-b5pgk\" (UID: \"2afb12c9-7c7a-4527-95c4-0b2febd22e90\") " pod="openshift-ingress/e2e-autores-4248ae-openshift-default-94bc8c67-b5pgk" Nov 03 01:05:03.809902 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:03.794078 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwz2h\" (UniqueName: \"kubernetes.io/projected/2afb12c9-7c7a-4527-95c4-0b2febd22e90-kube-api-access-vwz2h\") pod \"e2e-autores-4248ae-openshift-default-94bc8c67-b5pgk\" (UID: \"2afb12c9-7c7a-4527-95c4-0b2febd22e90\") " pod="openshift-ingress/e2e-autores-4248ae-openshift-default-94bc8c67-b5pgk" Nov 03 01:05:03.826315 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:03.818809 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2afb12c9-7c7a-4527-95c4-0b2febd22e90-istio-token\") pod \"e2e-autores-4248ae-openshift-default-94bc8c67-b5pgk\" (UID: \"2afb12c9-7c7a-4527-95c4-0b2febd22e90\") " pod="openshift-ingress/e2e-autores-4248ae-openshift-default-94bc8c67-b5pgk" Nov 03 01:05:04.025908 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:04.025599 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-4248ae-openshift-default-94bc8c67-b5pgk" Nov 03 01:05:04.542314 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:04.539773 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8gm8n" podUID="2d367e12-e46f-4a68-9825-9b1c934d4d12" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 01:05:05.492330 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:05.482461 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-650be0/simulator-w4-650be0-kserve-7bc588774f-gvz2k" podUID="a827fa88-f078-4312-8f12-2c799c11b461" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:05:05.538300 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:05.513098 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/simulator-w3-83755a-kserve-77d6664559-psqtg" podUID="b0a4c234-af58-4da0-a1a4-e1f167953f6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:05:05.538300 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:05.520578 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/simulator-w2-f7680d-kserve-85bdc86cd6-8gxrg" podUID="2bdcf521-cebd-4d6b-8554-1b60db8b0f3c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 03 01:05:05.739355 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:05.668681 4421 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8gm8n" podUID="2d367e12-e46f-4a68-9825-9b1c934d4d12" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/health\": context deadline exceeded" Nov 03 01:05:06.036357 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:05.674067 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/distinct-w3-83755a-kserve-86bdbbdd85-2cnnl" podUID="f4ae9624-d127-4a55-837d-7c9ff0267b37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded" Nov 03 01:05:06.114809 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:05.674837 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-650be0/premium-w4-650be0-kserve-7bf8555dfb-z97n5" podUID="40601dc5-808f-456f-826f-6e0c35966ada" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:05:06.115369 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:05.694520 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/premium-w3-83755a-kserve-7c9d66988b-lt4xd" podUID="c0c6b02e-2ecb-428e-a13d-be6d7cd1ba39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Nov 03 01:05:06.277750 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:05.834930 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:05:06.302355 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:06.296047 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:05:06.302355 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:06.296454 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:05:06.302355 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:05.961723 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8gm8n" podUID="2d367e12-e46f-4a68-9825-9b1c934d4d12" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:05:06.711359 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:06.710584 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-650be0/simulator-w4-650be0-kserve-7bc588774f-gvz2k" podUID="a827fa88-f078-4312-8f12-2c799c11b461" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Nov 03 01:05:06.736300 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:06.722493 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/simulator-w3-83755a-kserve-77d6664559-psqtg" podUID="b0a4c234-af58-4da0-a1a4-e1f167953f6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Nov 03 01:05:06.777406 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:06.776236 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/simulator-w2-f7680d-kserve-85bdc86cd6-8gxrg" podUID="2bdcf521-cebd-4d6b-8554-1b60db8b0f3c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:05:07.047677 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:07.047631 4421 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-83755a/distinct-w3-83755a-kserve-86bdbbdd85-2cnnl" podUID="f4ae9624-d127-4a55-837d-7c9ff0267b37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/health\": context deadline exceeded" Nov 03 01:05:07.138659 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:07.138603 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/premium-w3-83755a-kserve-7c9d66988b-lt4xd" podUID="c0c6b02e-2ecb-428e-a13d-be6d7cd1ba39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 01:05:07.140647 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:07.140612 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/distinct-w3-83755a-kserve-86bdbbdd85-2cnnl" podUID="f4ae9624-d127-4a55-837d-7c9ff0267b37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded" Nov 03 01:05:07.142957 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:07.142924 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-650be0/premium-w4-650be0-kserve-7bf8555dfb-z97n5" podUID="40601dc5-808f-456f-826f-6e0c35966ada" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 01:05:07.253726 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:07.253599 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tmxmj" podUID="1b62c15b-bbbf-4926-b4c3-27d12277743b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:05:07.291613 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:05:07.291555 4421 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.296s" Nov 03 01:05:07.291801 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:07.291676 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-4248ae-openshift-default-94bc8c67-b5pgk"] Nov 03 01:05:07.291801 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:07.291722 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-846d385a-rz8xm"] Nov 03 01:05:07.523991 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:07.462896 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-4248ae-openshift-default-94bc8c67-b5pgk" event={"ID":"2afb12c9-7c7a-4527-95c4-0b2febd22e90","Type":"ContainerStarted","Data":"97f6cd7a1f4bb5a0fc1795e160b671084b1dc197eb9cb2a75b67b507d1751906"} Nov 03 01:05:07.523991 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:07.462984 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8gm8n" Nov 03 01:05:07.523991 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:07.463009 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-846d385a-rz8xm"] Nov 03 01:05:07.523991 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:07.463040 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-717217-5f86cd4989-6685p"] Nov 03 01:05:07.523991 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:07.479329 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-717217-5f86cd4989-6685p"] Nov 03 01:05:07.523991 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:07.479404 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-717217-54d8c4bcf-v8j62"] Nov 03 01:05:07.523991 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:07.484722 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-846d385a-rz8xm" Nov 03 01:05:07.523991 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:07.489324 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:05:07.523991 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:07.489412 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-717217-5f86cd4989-6685p" Nov 03 01:05:07.523991 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:07.492656 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 01:05:07.523991 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:07.493058 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6q4xp\"" Nov 03 01:05:07.523991 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:07.493136 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 01:05:07.523991 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:07.496630 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-650be0/simulator-w4-650be0-kserve-7bc588774f-gvz2k" Nov 03 01:05:07.523991 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:07.502152 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-717217-dockercfg-cjjsl\"" Nov 03 01:05:07.523991 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:07.502233 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-717217\"" Nov 03 01:05:07.523991 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:07.503366 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-717217-54d8c4bcf-v8j62" Nov 03 01:05:07.523991 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:07.504831 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-717217-54d8c4bcf-v8j62"] Nov 03 01:05:07.523991 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:07.504947 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-83755a/simulator-w3-83755a-kserve-77d6664559-psqtg" Nov 03 01:05:07.523991 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:07.504983 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8gm8n" Nov 03 01:05:07.523991 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:07.505018 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f7680d/simulator-w2-f7680d-kserve-85bdc86cd6-8gxrg" Nov 03 01:05:07.523991 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:07.505064 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-650be0/simulator-w4-650be0-kserve-7bc588774f-gvz2k" Nov 03 01:05:07.523991 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:07.505098 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-83755a/premium-w3-83755a-kserve-7c9d66988b-lt4xd" Nov 03 01:05:07.523991 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:07.505145 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-83755a/distinct-w3-83755a-kserve-86bdbbdd85-2cnnl" Nov 03 01:05:07.523991 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:07.505180 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-650be0/premium-w4-650be0-kserve-7bf8555dfb-z97n5" Nov 03 01:05:07.523991 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:07.505220 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f7680d/simulator-w2-f7680d-kserve-85bdc86cd6-8gxrg" Nov 03 01:05:07.523991 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:07.505247 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-83755a/simulator-w3-83755a-kserve-77d6664559-psqtg" Nov 03 01:05:07.523991 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:07.505333 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-83755a/distinct-w3-83755a-kserve-86bdbbdd85-2cnnl" Nov 03 01:05:07.523991 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:07.505364 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-650be0/premium-w4-650be0-kserve-7bf8555dfb-z97n5" Nov 03 01:05:07.523991 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:07.505389 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-83755a/premium-w3-83755a-kserve-7c9d66988b-lt4xd" Nov 03 01:05:07.690169 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:07.688031 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76467cc3-c3fc-4372-b6c7-af22e0af51e8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-846d385a-rz8xm\" (UID: \"76467cc3-c3fc-4372-b6c7-af22e0af51e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-846d385a-rz8xm" Nov 03 01:05:07.690169 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:07.688182 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0f55109-ee26-4eab-b6ca-b610c783906d-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-717217-5f86cd4989-6685p\" (UID: \"f0f55109-ee26-4eab-b6ca-b610c783906d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-717217-5f86cd4989-6685p" Nov 03 01:05:07.690169 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:07.688581 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4kth\" (UniqueName: \"kubernetes.io/projected/38f1d8ba-2f55-4d82-b5e6-0491c36a8009-kube-api-access-r4kth\") pod \"payload-pre-processing-e2e-shared-a-w3-717217-54d8c4bcf-v8j62\" (UID: \"38f1d8ba-2f55-4d82-b5e6-0491c36a8009\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-717217-54d8c4bcf-v8j62" Nov 03 01:05:07.690169 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:07.689328 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38f1d8ba-2f55-4d82-b5e6-0491c36a8009-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-717217-54d8c4bcf-v8j62\" (UID: \"38f1d8ba-2f55-4d82-b5e6-0491c36a8009\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-717217-54d8c4bcf-v8j62" Nov 03 01:05:07.690169 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:07.689702 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gv8mp\" (UniqueName: \"kubernetes.io/projected/f0f55109-ee26-4eab-b6ca-b610c783906d-kube-api-access-gv8mp\") pod \"payload-processing-e2e-shared-a-w3-717217-5f86cd4989-6685p\" (UID: \"f0f55109-ee26-4eab-b6ca-b610c783906d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-717217-5f86cd4989-6685p" Nov 03 01:05:07.690169 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:07.689752 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9n4rn\" (UniqueName: \"kubernetes.io/projected/76467cc3-c3fc-4372-b6c7-af22e0af51e8-kube-api-access-9n4rn\") pod \"maas-api-revoke-sub-846d385a-rz8xm\" (UID: \"76467cc3-c3fc-4372-b6c7-af22e0af51e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-846d385a-rz8xm" Nov 03 01:05:07.690169 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:07.689883 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76467cc3-c3fc-4372-b6c7-af22e0af51e8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-846d385a-rz8xm\" (UID: \"76467cc3-c3fc-4372-b6c7-af22e0af51e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-846d385a-rz8xm" Nov 03 01:05:07.799267 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:07.799201 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38f1d8ba-2f55-4d82-b5e6-0491c36a8009-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-717217-54d8c4bcf-v8j62\" (UID: \"38f1d8ba-2f55-4d82-b5e6-0491c36a8009\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-717217-54d8c4bcf-v8j62" Nov 03 01:05:07.800585 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:07.799371 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gv8mp\" (UniqueName: \"kubernetes.io/projected/f0f55109-ee26-4eab-b6ca-b610c783906d-kube-api-access-gv8mp\") pod \"payload-processing-e2e-shared-a-w3-717217-5f86cd4989-6685p\" (UID: \"f0f55109-ee26-4eab-b6ca-b610c783906d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-717217-5f86cd4989-6685p" Nov 03 01:05:07.800585 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:07.799416 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9n4rn\" (UniqueName: \"kubernetes.io/projected/76467cc3-c3fc-4372-b6c7-af22e0af51e8-kube-api-access-9n4rn\") pod \"maas-api-revoke-sub-846d385a-rz8xm\" (UID: \"76467cc3-c3fc-4372-b6c7-af22e0af51e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-846d385a-rz8xm" Nov 03 01:05:07.800585 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:07.799573 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76467cc3-c3fc-4372-b6c7-af22e0af51e8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-846d385a-rz8xm\" (UID: \"76467cc3-c3fc-4372-b6c7-af22e0af51e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-846d385a-rz8xm" Nov 03 01:05:07.800585 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:07.799649 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76467cc3-c3fc-4372-b6c7-af22e0af51e8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-846d385a-rz8xm\" (UID: \"76467cc3-c3fc-4372-b6c7-af22e0af51e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-846d385a-rz8xm" Nov 03 01:05:07.800585 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:07.799726 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0f55109-ee26-4eab-b6ca-b610c783906d-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-717217-5f86cd4989-6685p\" (UID: \"f0f55109-ee26-4eab-b6ca-b610c783906d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-717217-5f86cd4989-6685p" Nov 03 01:05:07.800585 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:07.799800 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4kth\" (UniqueName: \"kubernetes.io/projected/38f1d8ba-2f55-4d82-b5e6-0491c36a8009-kube-api-access-r4kth\") pod \"payload-pre-processing-e2e-shared-a-w3-717217-54d8c4bcf-v8j62\" (UID: \"38f1d8ba-2f55-4d82-b5e6-0491c36a8009\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-717217-54d8c4bcf-v8j62" Nov 03 01:05:07.801577 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:07.801232 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38f1d8ba-2f55-4d82-b5e6-0491c36a8009-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-717217-54d8c4bcf-v8j62\" (UID: \"38f1d8ba-2f55-4d82-b5e6-0491c36a8009\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-717217-54d8c4bcf-v8j62" Nov 03 01:05:07.802453 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:07.802407 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0f55109-ee26-4eab-b6ca-b610c783906d-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-717217-5f86cd4989-6685p\" (UID: \"f0f55109-ee26-4eab-b6ca-b610c783906d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-717217-5f86cd4989-6685p" Nov 03 01:05:07.803131 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:07.803105 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76467cc3-c3fc-4372-b6c7-af22e0af51e8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-846d385a-rz8xm\" (UID: \"76467cc3-c3fc-4372-b6c7-af22e0af51e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-846d385a-rz8xm" Nov 03 01:05:07.813639 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:07.813595 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4kth\" (UniqueName: \"kubernetes.io/projected/38f1d8ba-2f55-4d82-b5e6-0491c36a8009-kube-api-access-r4kth\") pod \"payload-pre-processing-e2e-shared-a-w3-717217-54d8c4bcf-v8j62\" (UID: \"38f1d8ba-2f55-4d82-b5e6-0491c36a8009\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-717217-54d8c4bcf-v8j62" Nov 03 01:05:07.815065 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:07.815033 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76467cc3-c3fc-4372-b6c7-af22e0af51e8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-846d385a-rz8xm\" (UID: \"76467cc3-c3fc-4372-b6c7-af22e0af51e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-846d385a-rz8xm" Nov 03 01:05:07.815263 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:07.815107 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gv8mp\" (UniqueName: \"kubernetes.io/projected/f0f55109-ee26-4eab-b6ca-b610c783906d-kube-api-access-gv8mp\") pod \"payload-processing-e2e-shared-a-w3-717217-5f86cd4989-6685p\" (UID: \"f0f55109-ee26-4eab-b6ca-b610c783906d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-717217-5f86cd4989-6685p" Nov 03 01:05:07.816957 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:07.816926 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9n4rn\" (UniqueName: \"kubernetes.io/projected/76467cc3-c3fc-4372-b6c7-af22e0af51e8-kube-api-access-9n4rn\") pod \"maas-api-revoke-sub-846d385a-rz8xm\" (UID: \"76467cc3-c3fc-4372-b6c7-af22e0af51e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-846d385a-rz8xm" Nov 03 01:05:07.885642 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:07.885563 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-846d385a-rz8xm" Nov 03 01:05:07.944290 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:07.944236 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-717217-5f86cd4989-6685p" Nov 03 01:05:07.978505 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:07.978468 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-717217-54d8c4bcf-v8j62" Nov 03 01:05:08.185163 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:05:08.185129 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76467cc3_c3fc_4372_b6c7_af22e0af51e8.slice/crio-80db18472a90010307c9cc96f4ff9c07bb4d7cdcaa7f07acbcb75180adc3dd7a WatchSource:0}: Error finding container 80db18472a90010307c9cc96f4ff9c07bb4d7cdcaa7f07acbcb75180adc3dd7a: Status 404 returned error can't find the container with id 80db18472a90010307c9cc96f4ff9c07bb4d7cdcaa7f07acbcb75180adc3dd7a Nov 03 01:05:08.186440 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:08.186415 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-846d385a-rz8xm"] Nov 03 01:05:08.261980 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:05:08.261937 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0f55109_ee26_4eab_b6ca_b610c783906d.slice/crio-2685c07cc40073e5f18c8da383c403cab84c214becfb0282c6a842e68d27ebf4 WatchSource:0}: Error finding container 2685c07cc40073e5f18c8da383c403cab84c214becfb0282c6a842e68d27ebf4: Status 404 returned error can't find the container with id 2685c07cc40073e5f18c8da383c403cab84c214becfb0282c6a842e68d27ebf4 Nov 03 01:05:08.262778 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:08.262752 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-717217-5f86cd4989-6685p"] Nov 03 01:05:08.277238 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:05:08.277206 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38f1d8ba_2f55_4d82_b5e6_0491c36a8009.slice/crio-df214b099585ec039e612fd2f531c4b79dc0a88e0c52e742e21dead93d7edad9 WatchSource:0}: Error finding container df214b099585ec039e612fd2f531c4b79dc0a88e0c52e742e21dead93d7edad9: Status 404 returned error can't find the container with id df214b099585ec039e612fd2f531c4b79dc0a88e0c52e742e21dead93d7edad9 Nov 03 01:05:08.278917 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:08.278885 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-717217-54d8c4bcf-v8j62"] Nov 03 01:05:08.369424 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:08.369387 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-717217-5f86cd4989-6685p" event={"ID":"f0f55109-ee26-4eab-b6ca-b610c783906d","Type":"ContainerStarted","Data":"2685c07cc40073e5f18c8da383c403cab84c214becfb0282c6a842e68d27ebf4"} Nov 03 01:05:08.371238 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:08.371198 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-4248ae-openshift-default-94bc8c67-b5pgk" event={"ID":"2afb12c9-7c7a-4527-95c4-0b2febd22e90","Type":"ContainerStarted","Data":"20a772bbd0dc8daf02ef3acf699274dd1c38946168746fc7b03613467719f1b5"} Nov 03 01:05:08.373339 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:08.373305 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-717217-54d8c4bcf-v8j62" event={"ID":"38f1d8ba-2f55-4d82-b5e6-0491c36a8009","Type":"ContainerStarted","Data":"045a5208c7c98ef9a29aaebb16150c29be541504e6561b1ae71b8a056031c398"} Nov 03 01:05:08.373472 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:08.373363 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-717217-54d8c4bcf-v8j62" Nov 03 01:05:08.373472 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:08.373388 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-717217-54d8c4bcf-v8j62" event={"ID":"38f1d8ba-2f55-4d82-b5e6-0491c36a8009","Type":"ContainerStarted","Data":"df214b099585ec039e612fd2f531c4b79dc0a88e0c52e742e21dead93d7edad9"} Nov 03 01:05:08.374817 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:08.374771 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-846d385a-rz8xm" event={"ID":"76467cc3-c3fc-4372-b6c7-af22e0af51e8","Type":"ContainerStarted","Data":"80db18472a90010307c9cc96f4ff9c07bb4d7cdcaa7f07acbcb75180adc3dd7a"} Nov 03 01:05:08.395565 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:08.395515 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-4248ae-openshift-default-94bc8c67-b5pgk" podStartSLOduration=5.395499052 podStartE2EDuration="5.395499052s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:05:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:05:08.391137961 +0000 UTC m=+1450.562141080" watchObservedRunningTime="2026-11-03 01:05:08.395499052 +0000 UTC m=+1450.566502150" Nov 03 01:05:08.416323 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:08.416233 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-717217-54d8c4bcf-v8j62" podStartSLOduration=1.416215644 podStartE2EDuration="1.416215644s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:05:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:05:08.413502582 +0000 UTC m=+1450.584505886" watchObservedRunningTime="2026-11-03 01:05:08.416215644 +0000 UTC m=+1450.587218741" Nov 03 01:05:09.041463 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:09.041426 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-4248ae-openshift-default-94bc8c67-b5pgk" Nov 03 01:05:09.047702 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:09.047661 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-4248ae-openshift-default-94bc8c67-b5pgk" Nov 03 01:05:09.387403 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:09.387317 4421 generic.go:356] "Generic (PLEG): container finished" podID="76467cc3-c3fc-4372-b6c7-af22e0af51e8" containerID="9387a663d5d0703d7e6862c17a2abc112b8d3451fdd0dccf563a954e5e955bb4" exitCode=0 Nov 03 01:05:09.387403 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:09.387358 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-846d385a-rz8xm" event={"ID":"76467cc3-c3fc-4372-b6c7-af22e0af51e8","Type":"ContainerDied","Data":"9387a663d5d0703d7e6862c17a2abc112b8d3451fdd0dccf563a954e5e955bb4"} Nov 03 01:05:09.389529 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:09.389494 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-717217-5f86cd4989-6685p" event={"ID":"f0f55109-ee26-4eab-b6ca-b610c783906d","Type":"ContainerStarted","Data":"53f45c8b8b8508e4304cfb541ef35380dfdc7dcf9d19189c7b0173207a1fa436"} Nov 03 01:05:09.389746 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:09.389730 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-717217-5f86cd4989-6685p" Nov 03 01:05:09.389897 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:09.389867 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-4248ae-openshift-default-94bc8c67-b5pgk" Nov 03 01:05:09.390952 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:09.390933 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-4248ae-openshift-default-94bc8c67-b5pgk" Nov 03 01:05:09.456243 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:09.456181 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-717217-5f86cd4989-6685p" podStartSLOduration=2.456165877 podStartE2EDuration="2.456165877s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:05:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:05:09.452218361 +0000 UTC m=+1451.623221460" watchObservedRunningTime="2026-11-03 01:05:09.456165877 +0000 UTC m=+1451.627168972" Nov 03 01:05:10.591812 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:10.591785 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-846d385a-rz8xm" Nov 03 01:05:10.648296 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:10.644493 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/76467cc3-c3fc-4372-b6c7-af22e0af51e8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76467cc3-c3fc-4372-b6c7-af22e0af51e8-maas-api-xks-ca\") pod \"76467cc3-c3fc-4372-b6c7-af22e0af51e8\" (UID: \"76467cc3-c3fc-4372-b6c7-af22e0af51e8\") " Nov 03 01:05:10.648296 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:10.644614 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76467cc3-c3fc-4372-b6c7-af22e0af51e8-kube-api-access-9n4rn\" (UniqueName: \"kubernetes.io/projected/76467cc3-c3fc-4372-b6c7-af22e0af51e8-kube-api-access-9n4rn\") pod \"76467cc3-c3fc-4372-b6c7-af22e0af51e8\" (UID: \"76467cc3-c3fc-4372-b6c7-af22e0af51e8\") " Nov 03 01:05:10.648296 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:10.644925 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76467cc3-c3fc-4372-b6c7-af22e0af51e8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76467cc3-c3fc-4372-b6c7-af22e0af51e8-maas-api-openshift-ca\") pod \"76467cc3-c3fc-4372-b6c7-af22e0af51e8\" (UID: \"76467cc3-c3fc-4372-b6c7-af22e0af51e8\") " Nov 03 01:05:10.648296 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:10.645854 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76467cc3-c3fc-4372-b6c7-af22e0af51e8-maas-api-openshift-ca" pod "76467cc3-c3fc-4372-b6c7-af22e0af51e8" (UID: "76467cc3-c3fc-4372-b6c7-af22e0af51e8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:05:10.736566 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:10.721511 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76467cc3-c3fc-4372-b6c7-af22e0af51e8-maas-api-xks-ca" pod "76467cc3-c3fc-4372-b6c7-af22e0af51e8" (UID: "76467cc3-c3fc-4372-b6c7-af22e0af51e8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:05:10.739747 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:10.739682 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76467cc3-c3fc-4372-b6c7-af22e0af51e8-kube-api-access-9n4rn" pod "76467cc3-c3fc-4372-b6c7-af22e0af51e8" (UID: "76467cc3-c3fc-4372-b6c7-af22e0af51e8"). InnerVolumeSpecName "kube-api-access-9n4rn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:05:10.748453 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:10.747562 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76467cc3-c3fc-4372-b6c7-af22e0af51e8-maas-api-openshift-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:05:10.748453 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:10.747602 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76467cc3-c3fc-4372-b6c7-af22e0af51e8-maas-api-xks-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:05:10.748453 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:10.747617 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9n4rn\" (UniqueName: \"kubernetes.io/projected/76467cc3-c3fc-4372-b6c7-af22e0af51e8-kube-api-access-9n4rn\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:05:11.408342 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:11.408311 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-846d385a-rz8xm" Nov 03 01:05:11.408342 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:11.408330 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-846d385a-rz8xm" event={"ID":"76467cc3-c3fc-4372-b6c7-af22e0af51e8","Type":"ContainerDied","Data":"80db18472a90010307c9cc96f4ff9c07bb4d7cdcaa7f07acbcb75180adc3dd7a"} Nov 03 01:05:11.408530 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:11.408378 4421 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="80db18472a90010307c9cc96f4ff9c07bb4d7cdcaa7f07acbcb75180adc3dd7a" Nov 03 01:05:12.523704 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:12.523669 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-4248ae-85686d5965-jxtv4"] Nov 03 01:05:12.524362 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:12.524343 4421 cpu_manager.go:414] "containerMap: removing container" podUID="76467cc3-c3fc-4372-b6c7-af22e0af51e8" containerName="revoke-keys" Nov 03 01:05:12.524417 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:12.524365 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76467cc3-c3fc-4372-b6c7-af22e0af51e8" containerName="revoke-keys" Nov 03 01:05:12.524542 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:12.524532 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="76467cc3-c3fc-4372-b6c7-af22e0af51e8" containerName="revoke-keys" Nov 03 01:05:12.528329 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:12.528308 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-4248ae-85686d5965-jxtv4" Nov 03 01:05:12.538927 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:12.538901 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-4248ae\"" Nov 03 01:05:12.549003 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:12.548964 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-4248ae-85686d5965-jxtv4"] Nov 03 01:05:12.574267 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:12.574228 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-4248ae-69f65cb7f5-d6njn"] Nov 03 01:05:12.581734 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:12.580346 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-4248ae-69f65cb7f5-d6njn" Nov 03 01:05:12.581734 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:12.581612 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hn545\" (UniqueName: \"kubernetes.io/projected/48cda01c-0810-4fea-9459-9f8f6e6f1e0c-kube-api-access-hn545\") pod \"payload-processing-e2e-autores-4248ae-85686d5965-jxtv4\" (UID: \"48cda01c-0810-4fea-9459-9f8f6e6f1e0c\") " pod="openshift-ingress/payload-processing-e2e-autores-4248ae-85686d5965-jxtv4" Nov 03 01:05:12.581981 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:12.581951 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48cda01c-0810-4fea-9459-9f8f6e6f1e0c-plugins-config-volume\") pod \"payload-processing-e2e-autores-4248ae-85686d5965-jxtv4\" (UID: \"48cda01c-0810-4fea-9459-9f8f6e6f1e0c\") " pod="openshift-ingress/payload-processing-e2e-autores-4248ae-85686d5965-jxtv4" Nov 03 01:05:12.588990 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:12.588948 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-4248ae-69f65cb7f5-d6njn"] Nov 03 01:05:12.590697 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:12.590671 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-4248ae-dockercfg-w2hv2\"" Nov 03 01:05:12.696303 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:12.692164 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d0a4c35-92c7-4abc-b563-60b3b1d406a8-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-4248ae-69f65cb7f5-d6njn\" (UID: \"7d0a4c35-92c7-4abc-b563-60b3b1d406a8\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-4248ae-69f65cb7f5-d6njn" Nov 03 01:05:12.696303 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:12.692222 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nztxj\" (UniqueName: \"kubernetes.io/projected/7d0a4c35-92c7-4abc-b563-60b3b1d406a8-kube-api-access-nztxj\") pod \"payload-pre-processing-e2e-autores-4248ae-69f65cb7f5-d6njn\" (UID: \"7d0a4c35-92c7-4abc-b563-60b3b1d406a8\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-4248ae-69f65cb7f5-d6njn" Nov 03 01:05:12.696303 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:12.692557 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48cda01c-0810-4fea-9459-9f8f6e6f1e0c-plugins-config-volume\") pod \"payload-processing-e2e-autores-4248ae-85686d5965-jxtv4\" (UID: \"48cda01c-0810-4fea-9459-9f8f6e6f1e0c\") " pod="openshift-ingress/payload-processing-e2e-autores-4248ae-85686d5965-jxtv4" Nov 03 01:05:12.696303 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:12.692649 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hn545\" (UniqueName: \"kubernetes.io/projected/48cda01c-0810-4fea-9459-9f8f6e6f1e0c-kube-api-access-hn545\") pod \"payload-processing-e2e-autores-4248ae-85686d5965-jxtv4\" (UID: \"48cda01c-0810-4fea-9459-9f8f6e6f1e0c\") " pod="openshift-ingress/payload-processing-e2e-autores-4248ae-85686d5965-jxtv4" Nov 03 01:05:12.696303 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:12.693994 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48cda01c-0810-4fea-9459-9f8f6e6f1e0c-plugins-config-volume\") pod \"payload-processing-e2e-autores-4248ae-85686d5965-jxtv4\" (UID: \"48cda01c-0810-4fea-9459-9f8f6e6f1e0c\") " pod="openshift-ingress/payload-processing-e2e-autores-4248ae-85686d5965-jxtv4" Nov 03 01:05:12.771300 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:12.744609 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hn545\" (UniqueName: \"kubernetes.io/projected/48cda01c-0810-4fea-9459-9f8f6e6f1e0c-kube-api-access-hn545\") pod \"payload-processing-e2e-autores-4248ae-85686d5965-jxtv4\" (UID: \"48cda01c-0810-4fea-9459-9f8f6e6f1e0c\") " pod="openshift-ingress/payload-processing-e2e-autores-4248ae-85686d5965-jxtv4" Nov 03 01:05:12.805356 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:12.797398 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d0a4c35-92c7-4abc-b563-60b3b1d406a8-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-4248ae-69f65cb7f5-d6njn\" (UID: \"7d0a4c35-92c7-4abc-b563-60b3b1d406a8\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-4248ae-69f65cb7f5-d6njn" Nov 03 01:05:12.805356 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:12.797449 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nztxj\" (UniqueName: \"kubernetes.io/projected/7d0a4c35-92c7-4abc-b563-60b3b1d406a8-kube-api-access-nztxj\") pod \"payload-pre-processing-e2e-autores-4248ae-69f65cb7f5-d6njn\" (UID: \"7d0a4c35-92c7-4abc-b563-60b3b1d406a8\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-4248ae-69f65cb7f5-d6njn" Nov 03 01:05:12.805356 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:12.799033 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d0a4c35-92c7-4abc-b563-60b3b1d406a8-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-4248ae-69f65cb7f5-d6njn\" (UID: \"7d0a4c35-92c7-4abc-b563-60b3b1d406a8\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-4248ae-69f65cb7f5-d6njn" Nov 03 01:05:12.812680 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:12.812608 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nztxj\" (UniqueName: \"kubernetes.io/projected/7d0a4c35-92c7-4abc-b563-60b3b1d406a8-kube-api-access-nztxj\") pod \"payload-pre-processing-e2e-autores-4248ae-69f65cb7f5-d6njn\" (UID: \"7d0a4c35-92c7-4abc-b563-60b3b1d406a8\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-4248ae-69f65cb7f5-d6njn" Nov 03 01:05:12.866303 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:12.865586 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-4248ae-85686d5965-jxtv4" Nov 03 01:05:12.902549 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:12.902506 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-4248ae-69f65cb7f5-d6njn" Nov 03 01:05:13.172901 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:05:13.172863 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48cda01c_0810_4fea_9459_9f8f6e6f1e0c.slice/crio-1ff0d2ee6e8f76eaaca06bd5ff6c5bb007e2b0bbd880514031f18df8601184de WatchSource:0}: Error finding container 1ff0d2ee6e8f76eaaca06bd5ff6c5bb007e2b0bbd880514031f18df8601184de: Status 404 returned error can't find the container with id 1ff0d2ee6e8f76eaaca06bd5ff6c5bb007e2b0bbd880514031f18df8601184de Nov 03 01:05:13.174782 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:13.174758 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-4248ae-85686d5965-jxtv4"] Nov 03 01:05:13.214464 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:05:13.214435 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d0a4c35_92c7_4abc_b563_60b3b1d406a8.slice/crio-c8d495df7ed82b0674c64b2b62303d1aa078d7debc1a7beddcc9a02c468b6615 WatchSource:0}: Error finding container c8d495df7ed82b0674c64b2b62303d1aa078d7debc1a7beddcc9a02c468b6615: Status 404 returned error can't find the container with id c8d495df7ed82b0674c64b2b62303d1aa078d7debc1a7beddcc9a02c468b6615 Nov 03 01:05:13.215530 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:13.215422 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-4248ae-69f65cb7f5-d6njn"] Nov 03 01:05:13.435985 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:13.435876 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-4248ae-69f65cb7f5-d6njn" event={"ID":"7d0a4c35-92c7-4abc-b563-60b3b1d406a8","Type":"ContainerStarted","Data":"21faadb15cde6551b0c639a2c528338fab0a2f7e8eb45b22f3017262b499ae97"} Nov 03 01:05:13.435985 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:13.435940 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-4248ae-69f65cb7f5-d6njn" event={"ID":"7d0a4c35-92c7-4abc-b563-60b3b1d406a8","Type":"ContainerStarted","Data":"c8d495df7ed82b0674c64b2b62303d1aa078d7debc1a7beddcc9a02c468b6615"} Nov 03 01:05:13.436219 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:13.435994 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-4248ae-69f65cb7f5-d6njn" Nov 03 01:05:13.438309 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:13.438266 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-4248ae-85686d5965-jxtv4" event={"ID":"48cda01c-0810-4fea-9459-9f8f6e6f1e0c","Type":"ContainerStarted","Data":"3ad4ebe3c9bdcf0cab9ad730f903c4f726deb3e4f22c68eeb007d8342b0b215c"} Nov 03 01:05:13.438453 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:13.438321 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-4248ae-85686d5965-jxtv4" event={"ID":"48cda01c-0810-4fea-9459-9f8f6e6f1e0c","Type":"ContainerStarted","Data":"1ff0d2ee6e8f76eaaca06bd5ff6c5bb007e2b0bbd880514031f18df8601184de"} Nov 03 01:05:13.438453 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:13.438346 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-4248ae-85686d5965-jxtv4" Nov 03 01:05:13.466492 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:13.466435 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-4248ae-69f65cb7f5-d6njn" podStartSLOduration=1.466419388 podStartE2EDuration="1.466419388s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:05:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:05:13.463014203 +0000 UTC m=+1455.634017299" watchObservedRunningTime="2026-11-03 01:05:13.466419388 +0000 UTC m=+1455.637422484" Nov 03 01:05:13.493755 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:13.493695 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-4248ae-85686d5965-jxtv4" podStartSLOduration=1.49367578 podStartE2EDuration="1.49367578s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:05:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:05:13.492513782 +0000 UTC m=+1455.663516878" watchObservedRunningTime="2026-11-03 01:05:13.49367578 +0000 UTC m=+1455.664678877" Nov 03 01:05:14.670242 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:05:14.670203 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 01:05:15.184171 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:15.184133 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-sjhcb"] Nov 03 01:05:15.189861 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:15.189829 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-sjhcb" Nov 03 01:05:15.198216 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:15.198189 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-4af1f7-openshift-default-dockercfg-dmrp9\"" Nov 03 01:05:15.211070 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:15.211032 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-sjhcb"] Nov 03 01:05:15.264041 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:15.264003 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1d81d36-60e4-4f12-81e6-7c5be483af12-istio-podinfo\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-sjhcb\" (UID: \"d1d81d36-60e4-4f12-81e6-7c5be483af12\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-sjhcb" Nov 03 01:05:15.264220 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:15.264083 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1d81d36-60e4-4f12-81e6-7c5be483af12-credential-socket\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-sjhcb\" (UID: \"d1d81d36-60e4-4f12-81e6-7c5be483af12\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-sjhcb" Nov 03 01:05:15.264220 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:15.264129 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1d81d36-60e4-4f12-81e6-7c5be483af12-istio-envoy\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-sjhcb\" (UID: \"d1d81d36-60e4-4f12-81e6-7c5be483af12\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-sjhcb" Nov 03 01:05:15.264375 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:15.264343 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1d81d36-60e4-4f12-81e6-7c5be483af12-workload-certs\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-sjhcb\" (UID: \"d1d81d36-60e4-4f12-81e6-7c5be483af12\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-sjhcb" Nov 03 01:05:15.264440 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:15.264418 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1d81d36-60e4-4f12-81e6-7c5be483af12-istio-data\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-sjhcb\" (UID: \"d1d81d36-60e4-4f12-81e6-7c5be483af12\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-sjhcb" Nov 03 01:05:15.264578 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:15.264552 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9vv6\" (UniqueName: \"kubernetes.io/projected/d1d81d36-60e4-4f12-81e6-7c5be483af12-kube-api-access-c9vv6\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-sjhcb\" (UID: \"d1d81d36-60e4-4f12-81e6-7c5be483af12\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-sjhcb" Nov 03 01:05:15.264628 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:15.264596 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1d81d36-60e4-4f12-81e6-7c5be483af12-istio-token\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-sjhcb\" (UID: \"d1d81d36-60e4-4f12-81e6-7c5be483af12\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-sjhcb" Nov 03 01:05:15.264690 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:15.264676 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1d81d36-60e4-4f12-81e6-7c5be483af12-workload-socket\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-sjhcb\" (UID: \"d1d81d36-60e4-4f12-81e6-7c5be483af12\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-sjhcb" Nov 03 01:05:15.264732 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:15.264723 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1d81d36-60e4-4f12-81e6-7c5be483af12-istiod-ca-cert\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-sjhcb\" (UID: \"d1d81d36-60e4-4f12-81e6-7c5be483af12\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-sjhcb" Nov 03 01:05:15.376296 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:15.367301 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9vv6\" (UniqueName: \"kubernetes.io/projected/d1d81d36-60e4-4f12-81e6-7c5be483af12-kube-api-access-c9vv6\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-sjhcb\" (UID: \"d1d81d36-60e4-4f12-81e6-7c5be483af12\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-sjhcb" Nov 03 01:05:15.376296 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:15.367377 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1d81d36-60e4-4f12-81e6-7c5be483af12-istio-token\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-sjhcb\" (UID: \"d1d81d36-60e4-4f12-81e6-7c5be483af12\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-sjhcb" Nov 03 01:05:15.376296 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:15.367413 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1d81d36-60e4-4f12-81e6-7c5be483af12-workload-socket\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-sjhcb\" (UID: \"d1d81d36-60e4-4f12-81e6-7c5be483af12\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-sjhcb" Nov 03 01:05:15.376296 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:15.367477 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1d81d36-60e4-4f12-81e6-7c5be483af12-istiod-ca-cert\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-sjhcb\" (UID: \"d1d81d36-60e4-4f12-81e6-7c5be483af12\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-sjhcb" Nov 03 01:05:15.376296 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:15.367560 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1d81d36-60e4-4f12-81e6-7c5be483af12-istio-podinfo\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-sjhcb\" (UID: \"d1d81d36-60e4-4f12-81e6-7c5be483af12\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-sjhcb" Nov 03 01:05:15.376296 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:15.367621 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1d81d36-60e4-4f12-81e6-7c5be483af12-credential-socket\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-sjhcb\" (UID: \"d1d81d36-60e4-4f12-81e6-7c5be483af12\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-sjhcb" Nov 03 01:05:15.376296 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:15.367668 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1d81d36-60e4-4f12-81e6-7c5be483af12-istio-envoy\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-sjhcb\" (UID: \"d1d81d36-60e4-4f12-81e6-7c5be483af12\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-sjhcb" Nov 03 01:05:15.376296 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:15.367792 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1d81d36-60e4-4f12-81e6-7c5be483af12-workload-certs\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-sjhcb\" (UID: \"d1d81d36-60e4-4f12-81e6-7c5be483af12\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-sjhcb" Nov 03 01:05:15.376296 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:15.367865 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1d81d36-60e4-4f12-81e6-7c5be483af12-istio-data\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-sjhcb\" (UID: \"d1d81d36-60e4-4f12-81e6-7c5be483af12\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-sjhcb" Nov 03 01:05:15.376296 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:15.368453 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1d81d36-60e4-4f12-81e6-7c5be483af12-istio-data\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-sjhcb\" (UID: \"d1d81d36-60e4-4f12-81e6-7c5be483af12\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-sjhcb" Nov 03 01:05:15.376296 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:15.369631 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1d81d36-60e4-4f12-81e6-7c5be483af12-workload-socket\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-sjhcb\" (UID: \"d1d81d36-60e4-4f12-81e6-7c5be483af12\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-sjhcb" Nov 03 01:05:15.376296 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:15.370395 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1d81d36-60e4-4f12-81e6-7c5be483af12-istiod-ca-cert\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-sjhcb\" (UID: \"d1d81d36-60e4-4f12-81e6-7c5be483af12\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-sjhcb" Nov 03 01:05:15.376296 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:15.371464 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1d81d36-60e4-4f12-81e6-7c5be483af12-credential-socket\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-sjhcb\" (UID: \"d1d81d36-60e4-4f12-81e6-7c5be483af12\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-sjhcb" Nov 03 01:05:15.376296 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:15.372989 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1d81d36-60e4-4f12-81e6-7c5be483af12-workload-certs\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-sjhcb\" (UID: \"d1d81d36-60e4-4f12-81e6-7c5be483af12\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-sjhcb" Nov 03 01:05:15.385311 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:15.382871 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1d81d36-60e4-4f12-81e6-7c5be483af12-istio-envoy\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-sjhcb\" (UID: \"d1d81d36-60e4-4f12-81e6-7c5be483af12\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-sjhcb" Nov 03 01:05:15.385311 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:15.383691 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1d81d36-60e4-4f12-81e6-7c5be483af12-istio-token\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-sjhcb\" (UID: \"d1d81d36-60e4-4f12-81e6-7c5be483af12\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-sjhcb" Nov 03 01:05:15.385311 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:15.385150 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1d81d36-60e4-4f12-81e6-7c5be483af12-istio-podinfo\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-sjhcb\" (UID: \"d1d81d36-60e4-4f12-81e6-7c5be483af12\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-sjhcb" Nov 03 01:05:15.387342 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:15.386375 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9vv6\" (UniqueName: \"kubernetes.io/projected/d1d81d36-60e4-4f12-81e6-7c5be483af12-kube-api-access-c9vv6\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-sjhcb\" (UID: \"d1d81d36-60e4-4f12-81e6-7c5be483af12\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-sjhcb" Nov 03 01:05:15.508201 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:15.506911 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-sjhcb" Nov 03 01:05:15.862483 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:05:15.862447 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1d81d36_60e4_4f12_81e6_7c5be483af12.slice/crio-80e56c89512261a0f15e115b325ab065c89f61494d6f8072bc29ec82e0e7a4e8 WatchSource:0}: Error finding container 80e56c89512261a0f15e115b325ab065c89f61494d6f8072bc29ec82e0e7a4e8: Status 404 returned error can't find the container with id 80e56c89512261a0f15e115b325ab065c89f61494d6f8072bc29ec82e0e7a4e8 Nov 03 01:05:15.862984 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:15.862917 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-sjhcb"] Nov 03 01:05:15.865560 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:15.865523 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:05:15.865650 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:15.865613 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:05:15.865690 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:15.865658 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:05:16.494193 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:16.494150 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-sjhcb" event={"ID":"d1d81d36-60e4-4f12-81e6-7c5be483af12","Type":"ContainerStarted","Data":"88843295f0b313641db6de1e5c6e5f67887fa1739aa9a8995bcf4d96a892cfeb"} Nov 03 01:05:16.494449 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:16.494207 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-sjhcb" event={"ID":"d1d81d36-60e4-4f12-81e6-7c5be483af12","Type":"ContainerStarted","Data":"80e56c89512261a0f15e115b325ab065c89f61494d6f8072bc29ec82e0e7a4e8"} Nov 03 01:05:16.507619 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:16.507583 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-sjhcb" Nov 03 01:05:16.513790 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:16.513766 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-sjhcb" Nov 03 01:05:16.572906 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:16.572838 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-sjhcb" podStartSLOduration=1.572818076 podStartE2EDuration="1.572818076s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:05:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:05:16.519756213 +0000 UTC m=+1458.690759309" watchObservedRunningTime="2026-11-03 01:05:16.572818076 +0000 UTC m=+1458.743821172" Nov 03 01:05:17.503328 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:17.503261 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-sjhcb" Nov 03 01:05:17.504415 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:17.504392 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-sjhcb" Nov 03 01:05:18.352551 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:18.352512 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-v8jpj"] Nov 03 01:05:18.362419 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:18.362382 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-v8jpj" Nov 03 01:05:18.371318 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:18.371235 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-v8jpj"] Nov 03 01:05:18.385923 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:18.385886 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4x8n4\" (UniqueName: \"kubernetes.io/projected/9ca37a4e-e37b-453c-a937-a35fedc2af8c-kube-api-access-4x8n4\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-v8jpj\" (UID: \"9ca37a4e-e37b-453c-a937-a35fedc2af8c\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-v8jpj" Nov 03 01:05:18.386111 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:18.385951 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ca37a4e-e37b-453c-a937-a35fedc2af8c-istio-envoy\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-v8jpj\" (UID: \"9ca37a4e-e37b-453c-a937-a35fedc2af8c\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-v8jpj" Nov 03 01:05:18.386111 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:18.386056 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ca37a4e-e37b-453c-a937-a35fedc2af8c-istio-data\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-v8jpj\" (UID: \"9ca37a4e-e37b-453c-a937-a35fedc2af8c\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-v8jpj" Nov 03 01:05:18.386111 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:18.386093 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ca37a4e-e37b-453c-a937-a35fedc2af8c-workload-certs\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-v8jpj\" (UID: \"9ca37a4e-e37b-453c-a937-a35fedc2af8c\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-v8jpj" Nov 03 01:05:18.386361 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:18.386130 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ca37a4e-e37b-453c-a937-a35fedc2af8c-istio-token\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-v8jpj\" (UID: \"9ca37a4e-e37b-453c-a937-a35fedc2af8c\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-v8jpj" Nov 03 01:05:18.386361 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:18.386314 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ca37a4e-e37b-453c-a937-a35fedc2af8c-istio-podinfo\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-v8jpj\" (UID: \"9ca37a4e-e37b-453c-a937-a35fedc2af8c\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-v8jpj" Nov 03 01:05:18.386488 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:18.386379 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ca37a4e-e37b-453c-a937-a35fedc2af8c-istiod-ca-cert\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-v8jpj\" (UID: \"9ca37a4e-e37b-453c-a937-a35fedc2af8c\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-v8jpj" Nov 03 01:05:18.386488 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:18.386459 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ca37a4e-e37b-453c-a937-a35fedc2af8c-credential-socket\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-v8jpj\" (UID: \"9ca37a4e-e37b-453c-a937-a35fedc2af8c\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-v8jpj" Nov 03 01:05:18.386554 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:18.386535 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ca37a4e-e37b-453c-a937-a35fedc2af8c-workload-socket\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-v8jpj\" (UID: \"9ca37a4e-e37b-453c-a937-a35fedc2af8c\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-v8jpj" Nov 03 01:05:18.413820 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:18.413782 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-50e623-openshift-default-dockercfg-ngqv2\"" Nov 03 01:05:18.591299 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:18.589098 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ca37a4e-e37b-453c-a937-a35fedc2af8c-credential-socket\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-v8jpj\" (UID: \"9ca37a4e-e37b-453c-a937-a35fedc2af8c\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-v8jpj" Nov 03 01:05:18.591299 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:18.589226 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ca37a4e-e37b-453c-a937-a35fedc2af8c-workload-socket\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-v8jpj\" (UID: \"9ca37a4e-e37b-453c-a937-a35fedc2af8c\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-v8jpj" Nov 03 01:05:18.591299 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:18.589316 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4x8n4\" (UniqueName: \"kubernetes.io/projected/9ca37a4e-e37b-453c-a937-a35fedc2af8c-kube-api-access-4x8n4\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-v8jpj\" (UID: \"9ca37a4e-e37b-453c-a937-a35fedc2af8c\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-v8jpj" Nov 03 01:05:18.591299 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:18.589406 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ca37a4e-e37b-453c-a937-a35fedc2af8c-istio-envoy\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-v8jpj\" (UID: \"9ca37a4e-e37b-453c-a937-a35fedc2af8c\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-v8jpj" Nov 03 01:05:18.591299 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:18.589486 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ca37a4e-e37b-453c-a937-a35fedc2af8c-istio-data\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-v8jpj\" (UID: \"9ca37a4e-e37b-453c-a937-a35fedc2af8c\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-v8jpj" Nov 03 01:05:18.591299 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:18.589539 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ca37a4e-e37b-453c-a937-a35fedc2af8c-workload-certs\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-v8jpj\" (UID: \"9ca37a4e-e37b-453c-a937-a35fedc2af8c\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-v8jpj" Nov 03 01:05:18.591299 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:18.589578 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ca37a4e-e37b-453c-a937-a35fedc2af8c-istio-token\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-v8jpj\" (UID: \"9ca37a4e-e37b-453c-a937-a35fedc2af8c\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-v8jpj" Nov 03 01:05:18.591299 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:18.589716 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ca37a4e-e37b-453c-a937-a35fedc2af8c-istio-podinfo\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-v8jpj\" (UID: \"9ca37a4e-e37b-453c-a937-a35fedc2af8c\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-v8jpj" Nov 03 01:05:18.591299 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:18.589781 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ca37a4e-e37b-453c-a937-a35fedc2af8c-istiod-ca-cert\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-v8jpj\" (UID: \"9ca37a4e-e37b-453c-a937-a35fedc2af8c\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-v8jpj" Nov 03 01:05:18.618349 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:18.617900 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ca37a4e-e37b-453c-a937-a35fedc2af8c-workload-socket\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-v8jpj\" (UID: \"9ca37a4e-e37b-453c-a937-a35fedc2af8c\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-v8jpj" Nov 03 01:05:18.618349 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:18.618222 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ca37a4e-e37b-453c-a937-a35fedc2af8c-credential-socket\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-v8jpj\" (UID: \"9ca37a4e-e37b-453c-a937-a35fedc2af8c\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-v8jpj" Nov 03 01:05:18.668302 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:18.663748 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ca37a4e-e37b-453c-a937-a35fedc2af8c-workload-certs\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-v8jpj\" (UID: \"9ca37a4e-e37b-453c-a937-a35fedc2af8c\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-v8jpj" Nov 03 01:05:18.668302 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:18.664189 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ca37a4e-e37b-453c-a937-a35fedc2af8c-istio-data\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-v8jpj\" (UID: \"9ca37a4e-e37b-453c-a937-a35fedc2af8c\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-v8jpj" Nov 03 01:05:18.692331 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:18.691405 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ca37a4e-e37b-453c-a937-a35fedc2af8c-istio-podinfo\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-v8jpj\" (UID: \"9ca37a4e-e37b-453c-a937-a35fedc2af8c\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-v8jpj" Nov 03 01:05:18.733300 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:18.730954 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ca37a4e-e37b-453c-a937-a35fedc2af8c-istio-token\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-v8jpj\" (UID: \"9ca37a4e-e37b-453c-a937-a35fedc2af8c\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-v8jpj" Nov 03 01:05:18.740860 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:18.740826 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ca37a4e-e37b-453c-a937-a35fedc2af8c-istio-envoy\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-v8jpj\" (UID: \"9ca37a4e-e37b-453c-a937-a35fedc2af8c\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-v8jpj" Nov 03 01:05:18.741836 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:18.741808 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4x8n4\" (UniqueName: \"kubernetes.io/projected/9ca37a4e-e37b-453c-a937-a35fedc2af8c-kube-api-access-4x8n4\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-v8jpj\" (UID: \"9ca37a4e-e37b-453c-a937-a35fedc2af8c\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-v8jpj" Nov 03 01:05:18.751249 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:18.751195 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ca37a4e-e37b-453c-a937-a35fedc2af8c-istiod-ca-cert\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-v8jpj\" (UID: \"9ca37a4e-e37b-453c-a937-a35fedc2af8c\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-v8jpj" Nov 03 01:05:18.983979 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:18.983881 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-v8jpj" Nov 03 01:05:19.376302 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:19.375679 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-v8jpj"] Nov 03 01:05:19.394612 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:19.394542 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:05:19.398319 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:19.394654 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:05:19.398319 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:19.394701 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:05:19.400576 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:19.400068 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-717217-54d8c4bcf-v8j62" Nov 03 01:05:19.709073 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:19.708970 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-v8jpj" event={"ID":"9ca37a4e-e37b-453c-a937-a35fedc2af8c","Type":"ContainerStarted","Data":"2f7c8781a132b4ccff1df6db612942c2b474bda21f3232903c72e69ab3c02219"} Nov 03 01:05:19.709073 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:19.709038 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-v8jpj" event={"ID":"9ca37a4e-e37b-453c-a937-a35fedc2af8c","Type":"ContainerStarted","Data":"99f44fa0a7b48f58111b3c30a7b722fba3d410269bb9249b7a5ee9c1f76f5c1a"} Nov 03 01:05:19.732326 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:19.732260 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-v8jpj" podStartSLOduration=1.732244584 podStartE2EDuration="1.732244584s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:05:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:05:19.728762432 +0000 UTC m=+1461.899765528" watchObservedRunningTime="2026-11-03 01:05:19.732244584 +0000 UTC m=+1461.903247681" Nov 03 01:05:20.009743 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:20.009658 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-v8jpj" Nov 03 01:05:20.356652 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:20.356550 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-4af1f7-7fc8d84d47-mp2bh"] Nov 03 01:05:20.361705 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:20.361680 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-4af1f7-7cc74c488c-tsnxd"] Nov 03 01:05:20.361838 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:20.361793 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-4af1f7-7fc8d84d47-mp2bh" Nov 03 01:05:20.369115 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:20.369090 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-4af1f7-7cc74c488c-tsnxd" Nov 03 01:05:20.370255 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:20.370229 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-4af1f7\"" Nov 03 01:05:20.375067 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:20.374840 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-4af1f7-7cc74c488c-tsnxd"] Nov 03 01:05:20.382997 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:20.382957 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-4af1f7-7fc8d84d47-mp2bh"] Nov 03 01:05:20.403154 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:20.403125 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-717217-5f86cd4989-6685p" Nov 03 01:05:20.548866 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:20.548804 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af3bcba4-1005-480a-859f-8fc73f0cbe53-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-4af1f7-7fc8d84d47-mp2bh\" (UID: \"af3bcba4-1005-480a-859f-8fc73f0cbe53\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-4af1f7-7fc8d84d47-mp2bh" Nov 03 01:05:20.549070 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:20.549019 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqxp5\" (UniqueName: \"kubernetes.io/projected/6ff46e7d-ec61-4e17-ae76-0dc25edd95b4-kube-api-access-qqxp5\") pod \"payload-pre-processing-e2e-shared-b-w1-4af1f7-7cc74c488c-tsnxd\" (UID: \"6ff46e7d-ec61-4e17-ae76-0dc25edd95b4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-4af1f7-7cc74c488c-tsnxd" Nov 03 01:05:20.549070 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:20.549067 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rqtq\" (UniqueName: \"kubernetes.io/projected/af3bcba4-1005-480a-859f-8fc73f0cbe53-kube-api-access-5rqtq\") pod \"payload-processing-e2e-shared-b-w1-4af1f7-7fc8d84d47-mp2bh\" (UID: \"af3bcba4-1005-480a-859f-8fc73f0cbe53\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-4af1f7-7fc8d84d47-mp2bh" Nov 03 01:05:20.549188 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:20.549163 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ff46e7d-ec61-4e17-ae76-0dc25edd95b4-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-4af1f7-7cc74c488c-tsnxd\" (UID: \"6ff46e7d-ec61-4e17-ae76-0dc25edd95b4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-4af1f7-7cc74c488c-tsnxd" Nov 03 01:05:20.650252 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:20.650154 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqxp5\" (UniqueName: \"kubernetes.io/projected/6ff46e7d-ec61-4e17-ae76-0dc25edd95b4-kube-api-access-qqxp5\") pod \"payload-pre-processing-e2e-shared-b-w1-4af1f7-7cc74c488c-tsnxd\" (UID: \"6ff46e7d-ec61-4e17-ae76-0dc25edd95b4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-4af1f7-7cc74c488c-tsnxd" Nov 03 01:05:20.650436 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:20.650255 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rqtq\" (UniqueName: \"kubernetes.io/projected/af3bcba4-1005-480a-859f-8fc73f0cbe53-kube-api-access-5rqtq\") pod \"payload-processing-e2e-shared-b-w1-4af1f7-7fc8d84d47-mp2bh\" (UID: \"af3bcba4-1005-480a-859f-8fc73f0cbe53\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-4af1f7-7fc8d84d47-mp2bh" Nov 03 01:05:20.650436 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:20.650342 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ff46e7d-ec61-4e17-ae76-0dc25edd95b4-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-4af1f7-7cc74c488c-tsnxd\" (UID: \"6ff46e7d-ec61-4e17-ae76-0dc25edd95b4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-4af1f7-7cc74c488c-tsnxd" Nov 03 01:05:20.650572 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:20.650509 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af3bcba4-1005-480a-859f-8fc73f0cbe53-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-4af1f7-7fc8d84d47-mp2bh\" (UID: \"af3bcba4-1005-480a-859f-8fc73f0cbe53\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-4af1f7-7fc8d84d47-mp2bh" Nov 03 01:05:20.651102 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:20.651080 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ff46e7d-ec61-4e17-ae76-0dc25edd95b4-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-4af1f7-7cc74c488c-tsnxd\" (UID: \"6ff46e7d-ec61-4e17-ae76-0dc25edd95b4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-4af1f7-7cc74c488c-tsnxd" Nov 03 01:05:20.651225 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:20.651207 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af3bcba4-1005-480a-859f-8fc73f0cbe53-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-4af1f7-7fc8d84d47-mp2bh\" (UID: \"af3bcba4-1005-480a-859f-8fc73f0cbe53\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-4af1f7-7fc8d84d47-mp2bh" Nov 03 01:05:20.658928 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:20.658894 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqxp5\" (UniqueName: \"kubernetes.io/projected/6ff46e7d-ec61-4e17-ae76-0dc25edd95b4-kube-api-access-qqxp5\") pod \"payload-pre-processing-e2e-shared-b-w1-4af1f7-7cc74c488c-tsnxd\" (UID: \"6ff46e7d-ec61-4e17-ae76-0dc25edd95b4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-4af1f7-7cc74c488c-tsnxd" Nov 03 01:05:20.661657 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:20.661631 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rqtq\" (UniqueName: \"kubernetes.io/projected/af3bcba4-1005-480a-859f-8fc73f0cbe53-kube-api-access-5rqtq\") pod \"payload-processing-e2e-shared-b-w1-4af1f7-7fc8d84d47-mp2bh\" (UID: \"af3bcba4-1005-480a-859f-8fc73f0cbe53\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-4af1f7-7fc8d84d47-mp2bh" Nov 03 01:05:20.690422 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:20.690395 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-4af1f7-7fc8d84d47-mp2bh" Nov 03 01:05:20.700870 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:20.700847 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-4af1f7-7cc74c488c-tsnxd" Nov 03 01:05:20.992030 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:20.991996 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-v8jpj" Nov 03 01:05:21.174533 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:05:21.174493 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf3bcba4_1005_480a_859f_8fc73f0cbe53.slice/crio-168ea17e1d3f76f3d1ef595095e6f820a94ad533f5971be7dacd45d1a31f42b6 WatchSource:0}: Error finding container 168ea17e1d3f76f3d1ef595095e6f820a94ad533f5971be7dacd45d1a31f42b6: Status 404 returned error can't find the container with id 168ea17e1d3f76f3d1ef595095e6f820a94ad533f5971be7dacd45d1a31f42b6 Nov 03 01:05:21.175316 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:21.175287 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-4af1f7-7fc8d84d47-mp2bh"] Nov 03 01:05:21.232184 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:21.232153 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-4af1f7-7cc74c488c-tsnxd"] Nov 03 01:05:21.756857 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:21.756819 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-4af1f7-7fc8d84d47-mp2bh" event={"ID":"af3bcba4-1005-480a-859f-8fc73f0cbe53","Type":"ContainerStarted","Data":"74547c4d0d974eb572b4856b9a95a03223bdfbaa4f775e5000f3306df8c788be"} Nov 03 01:05:21.757101 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:21.756883 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-4af1f7-7fc8d84d47-mp2bh" Nov 03 01:05:21.757101 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:21.756907 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-4af1f7-7fc8d84d47-mp2bh" event={"ID":"af3bcba4-1005-480a-859f-8fc73f0cbe53","Type":"ContainerStarted","Data":"168ea17e1d3f76f3d1ef595095e6f820a94ad533f5971be7dacd45d1a31f42b6"} Nov 03 01:05:21.758805 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:21.758772 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-4af1f7-7cc74c488c-tsnxd" event={"ID":"6ff46e7d-ec61-4e17-ae76-0dc25edd95b4","Type":"ContainerStarted","Data":"64e2268ac33a34235b706162fd388297fc7e09136e6c290a7eb4b036e78812dc"} Nov 03 01:05:21.758955 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:21.758822 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-4af1f7-7cc74c488c-tsnxd" event={"ID":"6ff46e7d-ec61-4e17-ae76-0dc25edd95b4","Type":"ContainerStarted","Data":"50dcd91fbeec832bd266e47e22c842128bcde4aaca98a3dae4d1f10b1e0b74d0"} Nov 03 01:05:21.759042 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:21.759016 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-v8jpj" Nov 03 01:05:21.759042 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:21.759042 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-4af1f7-7cc74c488c-tsnxd" Nov 03 01:05:21.760127 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:21.760106 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-v8jpj" Nov 03 01:05:21.782266 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:21.782212 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-4af1f7-7fc8d84d47-mp2bh" podStartSLOduration=1.782198991 podStartE2EDuration="1.782198991s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:05:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:05:21.780107449 +0000 UTC m=+1463.951110547" watchObservedRunningTime="2026-11-03 01:05:21.782198991 +0000 UTC m=+1463.953202090" Nov 03 01:05:21.841619 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:21.841539 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-4af1f7-7cc74c488c-tsnxd" podStartSLOduration=1.841492679 podStartE2EDuration="1.841492679s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:05:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:05:21.835395109 +0000 UTC m=+1464.006398210" watchObservedRunningTime="2026-11-03 01:05:21.841492679 +0000 UTC m=+1464.012495776" Nov 03 01:05:23.845851 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:23.845814 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-50e623-6d6668784c-kp496"] Nov 03 01:05:23.850814 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:23.850788 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-50e623-6d6668784c-kp496" Nov 03 01:05:23.856155 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:23.856108 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w0-50e623\"" Nov 03 01:05:23.861237 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:23.861208 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-50e623-6d6668784c-kp496"] Nov 03 01:05:23.920389 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:23.920331 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d12f07a3-b300-4368-b119-f860d1d62b5a-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-50e623-6d6668784c-kp496\" (UID: \"d12f07a3-b300-4368-b119-f860d1d62b5a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-50e623-6d6668784c-kp496" Nov 03 01:05:23.920596 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:23.920572 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2kz7\" (UniqueName: \"kubernetes.io/projected/d12f07a3-b300-4368-b119-f860d1d62b5a-kube-api-access-d2kz7\") pod \"payload-processing-e2e-shared-a-w0-50e623-6d6668784c-kp496\" (UID: \"d12f07a3-b300-4368-b119-f860d1d62b5a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-50e623-6d6668784c-kp496" Nov 03 01:05:23.953911 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:23.953855 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w0-50e623-dockercfg-hwxhx\"" Nov 03 01:05:24.043297 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:24.031931 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d12f07a3-b300-4368-b119-f860d1d62b5a-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-50e623-6d6668784c-kp496\" (UID: \"d12f07a3-b300-4368-b119-f860d1d62b5a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-50e623-6d6668784c-kp496" Nov 03 01:05:24.043297 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:24.032192 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2kz7\" (UniqueName: \"kubernetes.io/projected/d12f07a3-b300-4368-b119-f860d1d62b5a-kube-api-access-d2kz7\") pod \"payload-processing-e2e-shared-a-w0-50e623-6d6668784c-kp496\" (UID: \"d12f07a3-b300-4368-b119-f860d1d62b5a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-50e623-6d6668784c-kp496" Nov 03 01:05:24.043297 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:24.033632 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d12f07a3-b300-4368-b119-f860d1d62b5a-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-50e623-6d6668784c-kp496\" (UID: \"d12f07a3-b300-4368-b119-f860d1d62b5a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-50e623-6d6668784c-kp496" Nov 03 01:05:24.059353 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:24.056435 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2kz7\" (UniqueName: \"kubernetes.io/projected/d12f07a3-b300-4368-b119-f860d1d62b5a-kube-api-access-d2kz7\") pod \"payload-processing-e2e-shared-a-w0-50e623-6d6668784c-kp496\" (UID: \"d12f07a3-b300-4368-b119-f860d1d62b5a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-50e623-6d6668784c-kp496" Nov 03 01:05:24.178911 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:24.175093 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-50e623-6d6668784c-kp496" Nov 03 01:05:24.452052 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:24.451967 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-4248ae-85686d5965-jxtv4" Nov 03 01:05:24.452052 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:24.452035 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-4248ae-69f65cb7f5-d6njn" Nov 03 01:05:24.487346 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:24.487182 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-50e623-6d6668784c-kp496"] Nov 03 01:05:24.787870 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:24.787833 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-50e623-6d6668784c-kp496" event={"ID":"d12f07a3-b300-4368-b119-f860d1d62b5a","Type":"ContainerStarted","Data":"f50315af9d472cb97c2fa507fa868fca5b42f5f694097606ab0f8ab7082e780b"} Nov 03 01:05:24.788093 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:24.787897 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-50e623-6d6668784c-kp496" Nov 03 01:05:24.788093 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:24.787915 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-50e623-6d6668784c-kp496" event={"ID":"d12f07a3-b300-4368-b119-f860d1d62b5a","Type":"ContainerStarted","Data":"e2a278ca1e83884d4368a6d3d080d2a839c8ebb06638e2df5129a107ddd9f793"} Nov 03 01:05:24.811440 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:24.811383 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-50e623-6d6668784c-kp496" podStartSLOduration=1.811366349 podStartE2EDuration="1.811366349s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:05:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:05:24.807030369 +0000 UTC m=+1466.978033470" watchObservedRunningTime="2026-11-03 01:05:24.811366349 +0000 UTC m=+1466.982369444" Nov 03 01:05:32.771071 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:32.771035 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-4af1f7-7fc8d84d47-mp2bh" Nov 03 01:05:32.771610 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:32.771135 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-4af1f7-7cc74c488c-tsnxd" Nov 03 01:05:33.650986 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:33.650929 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xj4dr"] Nov 03 01:05:33.677552 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:33.677517 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xj4dr"] Nov 03 01:05:33.677747 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:33.677637 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xj4dr" Nov 03 01:05:33.765803 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:33.765753 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5td7\" (UniqueName: \"kubernetes.io/projected/88bdc98e-5c06-4a49-9c50-013ca50e932d-kube-api-access-k5td7\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xj4dr\" (UID: \"88bdc98e-5c06-4a49-9c50-013ca50e932d\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xj4dr" Nov 03 01:05:33.766028 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:33.765820 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88bdc98e-5c06-4a49-9c50-013ca50e932d-istio-data\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xj4dr\" (UID: \"88bdc98e-5c06-4a49-9c50-013ca50e932d\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xj4dr" Nov 03 01:05:33.766102 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:33.766040 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88bdc98e-5c06-4a49-9c50-013ca50e932d-workload-certs\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xj4dr\" (UID: \"88bdc98e-5c06-4a49-9c50-013ca50e932d\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xj4dr" Nov 03 01:05:33.766172 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:33.766131 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88bdc98e-5c06-4a49-9c50-013ca50e932d-istiod-ca-cert\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xj4dr\" (UID: \"88bdc98e-5c06-4a49-9c50-013ca50e932d\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xj4dr" Nov 03 01:05:33.766233 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:33.766187 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88bdc98e-5c06-4a49-9c50-013ca50e932d-istio-envoy\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xj4dr\" (UID: \"88bdc98e-5c06-4a49-9c50-013ca50e932d\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xj4dr" Nov 03 01:05:33.766358 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:33.766240 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88bdc98e-5c06-4a49-9c50-013ca50e932d-workload-socket\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xj4dr\" (UID: \"88bdc98e-5c06-4a49-9c50-013ca50e932d\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xj4dr" Nov 03 01:05:33.766431 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:33.766390 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88bdc98e-5c06-4a49-9c50-013ca50e932d-credential-socket\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xj4dr\" (UID: \"88bdc98e-5c06-4a49-9c50-013ca50e932d\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xj4dr" Nov 03 01:05:33.766498 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:33.766444 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88bdc98e-5c06-4a49-9c50-013ca50e932d-istio-podinfo\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xj4dr\" (UID: \"88bdc98e-5c06-4a49-9c50-013ca50e932d\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xj4dr" Nov 03 01:05:33.766590 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:33.766565 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88bdc98e-5c06-4a49-9c50-013ca50e932d-istio-token\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xj4dr\" (UID: \"88bdc98e-5c06-4a49-9c50-013ca50e932d\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xj4dr" Nov 03 01:05:33.867957 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:33.867916 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5td7\" (UniqueName: \"kubernetes.io/projected/88bdc98e-5c06-4a49-9c50-013ca50e932d-kube-api-access-k5td7\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xj4dr\" (UID: \"88bdc98e-5c06-4a49-9c50-013ca50e932d\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xj4dr" Nov 03 01:05:33.868495 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:33.867972 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88bdc98e-5c06-4a49-9c50-013ca50e932d-istio-data\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xj4dr\" (UID: \"88bdc98e-5c06-4a49-9c50-013ca50e932d\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xj4dr" Nov 03 01:05:33.868495 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:33.868043 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88bdc98e-5c06-4a49-9c50-013ca50e932d-workload-certs\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xj4dr\" (UID: \"88bdc98e-5c06-4a49-9c50-013ca50e932d\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xj4dr" Nov 03 01:05:33.868495 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:33.868090 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88bdc98e-5c06-4a49-9c50-013ca50e932d-istiod-ca-cert\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xj4dr\" (UID: \"88bdc98e-5c06-4a49-9c50-013ca50e932d\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xj4dr" Nov 03 01:05:33.868495 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:33.868106 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88bdc98e-5c06-4a49-9c50-013ca50e932d-istio-envoy\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xj4dr\" (UID: \"88bdc98e-5c06-4a49-9c50-013ca50e932d\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xj4dr" Nov 03 01:05:33.868495 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:33.868134 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88bdc98e-5c06-4a49-9c50-013ca50e932d-workload-socket\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xj4dr\" (UID: \"88bdc98e-5c06-4a49-9c50-013ca50e932d\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xj4dr" Nov 03 01:05:33.868495 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:33.868180 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88bdc98e-5c06-4a49-9c50-013ca50e932d-credential-socket\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xj4dr\" (UID: \"88bdc98e-5c06-4a49-9c50-013ca50e932d\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xj4dr" Nov 03 01:05:33.868495 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:33.868229 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88bdc98e-5c06-4a49-9c50-013ca50e932d-istio-podinfo\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xj4dr\" (UID: \"88bdc98e-5c06-4a49-9c50-013ca50e932d\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xj4dr" Nov 03 01:05:33.868495 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:33.868305 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88bdc98e-5c06-4a49-9c50-013ca50e932d-istio-token\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xj4dr\" (UID: \"88bdc98e-5c06-4a49-9c50-013ca50e932d\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xj4dr" Nov 03 01:05:33.868923 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:33.868557 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88bdc98e-5c06-4a49-9c50-013ca50e932d-istio-data\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xj4dr\" (UID: \"88bdc98e-5c06-4a49-9c50-013ca50e932d\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xj4dr" Nov 03 01:05:33.868923 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:33.868684 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88bdc98e-5c06-4a49-9c50-013ca50e932d-credential-socket\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xj4dr\" (UID: \"88bdc98e-5c06-4a49-9c50-013ca50e932d\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xj4dr" Nov 03 01:05:33.868923 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:33.868776 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88bdc98e-5c06-4a49-9c50-013ca50e932d-workload-socket\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xj4dr\" (UID: \"88bdc98e-5c06-4a49-9c50-013ca50e932d\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xj4dr" Nov 03 01:05:33.868923 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:33.868820 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88bdc98e-5c06-4a49-9c50-013ca50e932d-workload-certs\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xj4dr\" (UID: \"88bdc98e-5c06-4a49-9c50-013ca50e932d\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xj4dr" Nov 03 01:05:33.869075 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:33.868977 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88bdc98e-5c06-4a49-9c50-013ca50e932d-istiod-ca-cert\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xj4dr\" (UID: \"88bdc98e-5c06-4a49-9c50-013ca50e932d\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xj4dr" Nov 03 01:05:33.870426 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:33.870409 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88bdc98e-5c06-4a49-9c50-013ca50e932d-istio-envoy\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xj4dr\" (UID: \"88bdc98e-5c06-4a49-9c50-013ca50e932d\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xj4dr" Nov 03 01:05:33.870944 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:33.870928 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88bdc98e-5c06-4a49-9c50-013ca50e932d-istio-podinfo\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xj4dr\" (UID: \"88bdc98e-5c06-4a49-9c50-013ca50e932d\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xj4dr" Nov 03 01:05:33.876894 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:33.876866 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5td7\" (UniqueName: \"kubernetes.io/projected/88bdc98e-5c06-4a49-9c50-013ca50e932d-kube-api-access-k5td7\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xj4dr\" (UID: \"88bdc98e-5c06-4a49-9c50-013ca50e932d\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xj4dr" Nov 03 01:05:33.876984 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:33.876938 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88bdc98e-5c06-4a49-9c50-013ca50e932d-istio-token\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xj4dr\" (UID: \"88bdc98e-5c06-4a49-9c50-013ca50e932d\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xj4dr" Nov 03 01:05:33.993149 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:33.993068 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xj4dr" Nov 03 01:05:34.191886 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:34.191849 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-x67zh"] Nov 03 01:05:34.198760 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:34.197968 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-x67zh" Nov 03 01:05:34.203422 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:34.203396 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-556c14-openshift-default-dockercfg-bpvll\"" Nov 03 01:05:34.205996 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:34.205960 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-x67zh"] Nov 03 01:05:34.274875 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:34.274832 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/894c4be7-9b51-47c4-87d7-05eee1e8bcbc-workload-certs\") pod \"e2e-shared-b-w3-556c14-openshift-default-754b5bc799-x67zh\" (UID: \"894c4be7-9b51-47c4-87d7-05eee1e8bcbc\") " pod="openshift-ingress/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-x67zh" Nov 03 01:05:34.274875 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:34.274882 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/894c4be7-9b51-47c4-87d7-05eee1e8bcbc-workload-socket\") pod \"e2e-shared-b-w3-556c14-openshift-default-754b5bc799-x67zh\" (UID: \"894c4be7-9b51-47c4-87d7-05eee1e8bcbc\") " pod="openshift-ingress/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-x67zh" Nov 03 01:05:34.275190 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:34.274911 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/894c4be7-9b51-47c4-87d7-05eee1e8bcbc-istio-podinfo\") pod \"e2e-shared-b-w3-556c14-openshift-default-754b5bc799-x67zh\" (UID: \"894c4be7-9b51-47c4-87d7-05eee1e8bcbc\") " pod="openshift-ingress/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-x67zh" Nov 03 01:05:34.275190 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:34.274954 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/894c4be7-9b51-47c4-87d7-05eee1e8bcbc-istio-token\") pod \"e2e-shared-b-w3-556c14-openshift-default-754b5bc799-x67zh\" (UID: \"894c4be7-9b51-47c4-87d7-05eee1e8bcbc\") " pod="openshift-ingress/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-x67zh" Nov 03 01:05:34.275190 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:34.275131 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/894c4be7-9b51-47c4-87d7-05eee1e8bcbc-istio-data\") pod \"e2e-shared-b-w3-556c14-openshift-default-754b5bc799-x67zh\" (UID: \"894c4be7-9b51-47c4-87d7-05eee1e8bcbc\") " pod="openshift-ingress/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-x67zh" Nov 03 01:05:34.275418 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:34.275195 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/894c4be7-9b51-47c4-87d7-05eee1e8bcbc-istiod-ca-cert\") pod \"e2e-shared-b-w3-556c14-openshift-default-754b5bc799-x67zh\" (UID: \"894c4be7-9b51-47c4-87d7-05eee1e8bcbc\") " pod="openshift-ingress/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-x67zh" Nov 03 01:05:34.275418 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:34.275311 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wbpc\" (UniqueName: \"kubernetes.io/projected/894c4be7-9b51-47c4-87d7-05eee1e8bcbc-kube-api-access-7wbpc\") pod \"e2e-shared-b-w3-556c14-openshift-default-754b5bc799-x67zh\" (UID: \"894c4be7-9b51-47c4-87d7-05eee1e8bcbc\") " pod="openshift-ingress/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-x67zh" Nov 03 01:05:34.275533 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:34.275466 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/894c4be7-9b51-47c4-87d7-05eee1e8bcbc-credential-socket\") pod \"e2e-shared-b-w3-556c14-openshift-default-754b5bc799-x67zh\" (UID: \"894c4be7-9b51-47c4-87d7-05eee1e8bcbc\") " pod="openshift-ingress/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-x67zh" Nov 03 01:05:34.275533 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:34.275525 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/894c4be7-9b51-47c4-87d7-05eee1e8bcbc-istio-envoy\") pod \"e2e-shared-b-w3-556c14-openshift-default-754b5bc799-x67zh\" (UID: \"894c4be7-9b51-47c4-87d7-05eee1e8bcbc\") " pod="openshift-ingress/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-x67zh" Nov 03 01:05:34.399687 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:34.393390 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/894c4be7-9b51-47c4-87d7-05eee1e8bcbc-istio-data\") pod \"e2e-shared-b-w3-556c14-openshift-default-754b5bc799-x67zh\" (UID: \"894c4be7-9b51-47c4-87d7-05eee1e8bcbc\") " pod="openshift-ingress/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-x67zh" Nov 03 01:05:34.399966 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:34.399943 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/894c4be7-9b51-47c4-87d7-05eee1e8bcbc-istiod-ca-cert\") pod \"e2e-shared-b-w3-556c14-openshift-default-754b5bc799-x67zh\" (UID: \"894c4be7-9b51-47c4-87d7-05eee1e8bcbc\") " pod="openshift-ingress/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-x67zh" Nov 03 01:05:34.400160 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:34.400144 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wbpc\" (UniqueName: \"kubernetes.io/projected/894c4be7-9b51-47c4-87d7-05eee1e8bcbc-kube-api-access-7wbpc\") pod \"e2e-shared-b-w3-556c14-openshift-default-754b5bc799-x67zh\" (UID: \"894c4be7-9b51-47c4-87d7-05eee1e8bcbc\") " pod="openshift-ingress/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-x67zh" Nov 03 01:05:34.400385 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:34.400367 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/894c4be7-9b51-47c4-87d7-05eee1e8bcbc-credential-socket\") pod \"e2e-shared-b-w3-556c14-openshift-default-754b5bc799-x67zh\" (UID: \"894c4be7-9b51-47c4-87d7-05eee1e8bcbc\") " pod="openshift-ingress/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-x67zh" Nov 03 01:05:34.400540 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:34.400523 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/894c4be7-9b51-47c4-87d7-05eee1e8bcbc-istio-envoy\") pod \"e2e-shared-b-w3-556c14-openshift-default-754b5bc799-x67zh\" (UID: \"894c4be7-9b51-47c4-87d7-05eee1e8bcbc\") " pod="openshift-ingress/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-x67zh" Nov 03 01:05:34.400828 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:34.400808 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/894c4be7-9b51-47c4-87d7-05eee1e8bcbc-workload-certs\") pod \"e2e-shared-b-w3-556c14-openshift-default-754b5bc799-x67zh\" (UID: \"894c4be7-9b51-47c4-87d7-05eee1e8bcbc\") " pod="openshift-ingress/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-x67zh" Nov 03 01:05:34.400958 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:34.400939 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/894c4be7-9b51-47c4-87d7-05eee1e8bcbc-workload-socket\") pod \"e2e-shared-b-w3-556c14-openshift-default-754b5bc799-x67zh\" (UID: \"894c4be7-9b51-47c4-87d7-05eee1e8bcbc\") " pod="openshift-ingress/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-x67zh" Nov 03 01:05:34.401085 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:34.401066 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/894c4be7-9b51-47c4-87d7-05eee1e8bcbc-istio-podinfo\") pod \"e2e-shared-b-w3-556c14-openshift-default-754b5bc799-x67zh\" (UID: \"894c4be7-9b51-47c4-87d7-05eee1e8bcbc\") " pod="openshift-ingress/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-x67zh" Nov 03 01:05:34.401195 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:34.401179 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/894c4be7-9b51-47c4-87d7-05eee1e8bcbc-istio-token\") pod \"e2e-shared-b-w3-556c14-openshift-default-754b5bc799-x67zh\" (UID: \"894c4be7-9b51-47c4-87d7-05eee1e8bcbc\") " pod="openshift-ingress/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-x67zh" Nov 03 01:05:34.401468 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:34.401292 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/894c4be7-9b51-47c4-87d7-05eee1e8bcbc-credential-socket\") pod \"e2e-shared-b-w3-556c14-openshift-default-754b5bc799-x67zh\" (UID: \"894c4be7-9b51-47c4-87d7-05eee1e8bcbc\") " pod="openshift-ingress/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-x67zh" Nov 03 01:05:34.401583 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:34.399151 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:05:34.401760 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:34.401731 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:05:34.401903 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:34.401885 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:05:34.406939 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:34.405354 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/894c4be7-9b51-47c4-87d7-05eee1e8bcbc-istio-envoy\") pod \"e2e-shared-b-w3-556c14-openshift-default-754b5bc799-x67zh\" (UID: \"894c4be7-9b51-47c4-87d7-05eee1e8bcbc\") " pod="openshift-ingress/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-x67zh" Nov 03 01:05:34.406939 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:34.405689 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/894c4be7-9b51-47c4-87d7-05eee1e8bcbc-workload-certs\") pod \"e2e-shared-b-w3-556c14-openshift-default-754b5bc799-x67zh\" (UID: \"894c4be7-9b51-47c4-87d7-05eee1e8bcbc\") " pod="openshift-ingress/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-x67zh" Nov 03 01:05:34.406939 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:34.405936 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/894c4be7-9b51-47c4-87d7-05eee1e8bcbc-workload-socket\") pod \"e2e-shared-b-w3-556c14-openshift-default-754b5bc799-x67zh\" (UID: \"894c4be7-9b51-47c4-87d7-05eee1e8bcbc\") " pod="openshift-ingress/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-x67zh" Nov 03 01:05:34.406939 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:34.401024 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/894c4be7-9b51-47c4-87d7-05eee1e8bcbc-istiod-ca-cert\") pod \"e2e-shared-b-w3-556c14-openshift-default-754b5bc799-x67zh\" (UID: \"894c4be7-9b51-47c4-87d7-05eee1e8bcbc\") " pod="openshift-ingress/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-x67zh" Nov 03 01:05:34.411974 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:34.394441 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/894c4be7-9b51-47c4-87d7-05eee1e8bcbc-istio-data\") pod \"e2e-shared-b-w3-556c14-openshift-default-754b5bc799-x67zh\" (UID: \"894c4be7-9b51-47c4-87d7-05eee1e8bcbc\") " pod="openshift-ingress/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-x67zh" Nov 03 01:05:34.477217 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:34.454897 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xj4dr"] Nov 03 01:05:34.501357 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:34.501319 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wbpc\" (UniqueName: \"kubernetes.io/projected/894c4be7-9b51-47c4-87d7-05eee1e8bcbc-kube-api-access-7wbpc\") pod \"e2e-shared-b-w3-556c14-openshift-default-754b5bc799-x67zh\" (UID: \"894c4be7-9b51-47c4-87d7-05eee1e8bcbc\") " pod="openshift-ingress/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-x67zh" Nov 03 01:05:34.513816 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:34.513752 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/894c4be7-9b51-47c4-87d7-05eee1e8bcbc-istio-podinfo\") pod \"e2e-shared-b-w3-556c14-openshift-default-754b5bc799-x67zh\" (UID: \"894c4be7-9b51-47c4-87d7-05eee1e8bcbc\") " pod="openshift-ingress/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-x67zh" Nov 03 01:05:34.543347 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:34.540840 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/894c4be7-9b51-47c4-87d7-05eee1e8bcbc-istio-token\") pod \"e2e-shared-b-w3-556c14-openshift-default-754b5bc799-x67zh\" (UID: \"894c4be7-9b51-47c4-87d7-05eee1e8bcbc\") " pod="openshift-ingress/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-x67zh" Nov 03 01:05:34.826353 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:34.823628 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-x67zh" Nov 03 01:05:34.951592 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:34.951557 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xj4dr" event={"ID":"88bdc98e-5c06-4a49-9c50-013ca50e932d","Type":"ContainerStarted","Data":"c081f5813e845276d7dd8cc9dbebadb59bd176953e9d5ba8a95457e539da8d93"} Nov 03 01:05:34.952079 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:34.951622 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xj4dr" event={"ID":"88bdc98e-5c06-4a49-9c50-013ca50e932d","Type":"ContainerStarted","Data":"32674b1c68d04f489cd6d37f3a84c9b9bd9294957c2f54c16e58f1cd1a2f9ec1"} Nov 03 01:05:34.981025 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:34.980956 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xj4dr" podStartSLOduration=1.98093369 podStartE2EDuration="1.98093369s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:05:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:05:34.972364675 +0000 UTC m=+1477.143367771" watchObservedRunningTime="2026-11-03 01:05:34.98093369 +0000 UTC m=+1477.151936788" Nov 03 01:05:34.993770 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:34.993718 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xj4dr" Nov 03 01:05:35.144975 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:35.144934 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-x67zh"] Nov 03 01:05:35.146104 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:05:35.146076 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod894c4be7_9b51_47c4_87d7_05eee1e8bcbc.slice/crio-fdd2613d60209122cf80912ea65b7a571f6b0c50b2f5889a0664177f7f022bb3 WatchSource:0}: Error finding container fdd2613d60209122cf80912ea65b7a571f6b0c50b2f5889a0664177f7f022bb3: Status 404 returned error can't find the container with id fdd2613d60209122cf80912ea65b7a571f6b0c50b2f5889a0664177f7f022bb3 Nov 03 01:05:35.149351 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:35.149307 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:05:35.149450 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:35.149400 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:05:35.149506 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:35.149444 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:05:35.839717 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:35.839687 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-50e623-6d6668784c-kp496" Nov 03 01:05:35.961309 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:35.961261 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-x67zh" event={"ID":"894c4be7-9b51-47c4-87d7-05eee1e8bcbc","Type":"ContainerStarted","Data":"222e52cd77cb070a4e69d96d60d2a4c50bebd5e667232db677487b8cf0b86476"} Nov 03 01:05:35.961802 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:35.961329 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-x67zh" event={"ID":"894c4be7-9b51-47c4-87d7-05eee1e8bcbc","Type":"ContainerStarted","Data":"fdd2613d60209122cf80912ea65b7a571f6b0c50b2f5889a0664177f7f022bb3"} Nov 03 01:05:35.987795 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:35.987742 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-x67zh" podStartSLOduration=1.987729302 podStartE2EDuration="1.987729302s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:05:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:05:35.98197968 +0000 UTC m=+1478.152982778" watchObservedRunningTime="2026-11-03 01:05:35.987729302 +0000 UTC m=+1478.158732397" Nov 03 01:05:35.995314 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:35.995282 4421 patch_prober.go:29] interesting pod/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xj4dr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" start-of-body= Nov 03 01:05:35.995472 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:35.995349 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xj4dr" podUID="88bdc98e-5c06-4a49-9c50-013ca50e932d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" Nov 03 01:05:36.843301 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:36.830587 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-x67zh" Nov 03 01:05:36.852964 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:36.852931 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-x67zh" Nov 03 01:05:36.968823 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:36.968784 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-x67zh" Nov 03 01:05:36.969594 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:36.969573 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-x67zh" Nov 03 01:05:36.994564 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:36.994522 4421 patch_prober.go:29] interesting pod/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xj4dr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" start-of-body= Nov 03 01:05:36.994740 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:36.994593 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xj4dr" podUID="88bdc98e-5c06-4a49-9c50-013ca50e932d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" Nov 03 01:05:38.073341 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:38.065514 4421 patch_prober.go:29] interesting pod/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xj4dr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" start-of-body= Nov 03 01:05:38.073341 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:38.065579 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xj4dr" podUID="88bdc98e-5c06-4a49-9c50-013ca50e932d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" Nov 03 01:05:38.993874 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:38.993842 4421 patch_prober.go:29] interesting pod/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xj4dr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" start-of-body= Nov 03 01:05:38.994039 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:38.993897 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xj4dr" podUID="88bdc98e-5c06-4a49-9c50-013ca50e932d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" Nov 03 01:05:40.032297 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:40.018512 4421 patch_prober.go:29] interesting pod/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xj4dr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" start-of-body= Nov 03 01:05:40.032297 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:40.018582 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xj4dr" podUID="88bdc98e-5c06-4a49-9c50-013ca50e932d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" Nov 03 01:05:40.994460 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:40.994421 4421 patch_prober.go:29] interesting pod/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xj4dr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" start-of-body= Nov 03 01:05:40.994638 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:40.994482 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xj4dr" podUID="88bdc98e-5c06-4a49-9c50-013ca50e932d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" Nov 03 01:05:42.365679 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:42.365629 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xj4dr" Nov 03 01:05:42.366214 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:42.366179 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xj4dr" Nov 03 01:05:42.367446 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:42.367418 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xj4dr" Nov 03 01:05:44.671308 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:05:44.671233 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 01:05:45.080294 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:45.080047 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-wzfts"] Nov 03 01:05:45.109004 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:45.106309 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-wzfts" Nov 03 01:05:45.123630 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:45.123596 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j52qt"] Nov 03 01:05:45.143426 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:45.131129 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j52qt" Nov 03 01:05:45.143426 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:45.131985 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-wzfts"] Nov 03 01:05:45.143426 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:45.143375 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j52qt"] Nov 03 01:05:45.204917 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:45.204884 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-2729f2-openshift-default-dockercfg-9nk2m\"" Nov 03 01:05:45.208573 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:45.208543 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5d698b3-22ca-43c5-9d37-37e38d418444-istio-podinfo\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-wzfts\" (UID: \"a5d698b3-22ca-43c5-9d37-37e38d418444\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-wzfts" Nov 03 01:05:45.208707 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:45.208581 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9417c149-c243-4ba7-9f66-7ad0688a9f78-istio-data\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j52qt\" (UID: \"9417c149-c243-4ba7-9f66-7ad0688a9f78\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j52qt" Nov 03 01:05:45.208707 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:45.208605 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9417c149-c243-4ba7-9f66-7ad0688a9f78-istio-token\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j52qt\" (UID: \"9417c149-c243-4ba7-9f66-7ad0688a9f78\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j52qt" Nov 03 01:05:45.208802 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:45.208755 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9417c149-c243-4ba7-9f66-7ad0688a9f78-istiod-ca-cert\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j52qt\" (UID: \"9417c149-c243-4ba7-9f66-7ad0688a9f78\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j52qt" Nov 03 01:05:45.208802 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:45.208787 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9417c149-c243-4ba7-9f66-7ad0688a9f78-credential-socket\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j52qt\" (UID: \"9417c149-c243-4ba7-9f66-7ad0688a9f78\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j52qt" Nov 03 01:05:45.208878 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:45.208859 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5d698b3-22ca-43c5-9d37-37e38d418444-credential-socket\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-wzfts\" (UID: \"a5d698b3-22ca-43c5-9d37-37e38d418444\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-wzfts" Nov 03 01:05:45.208963 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:45.208950 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5d698b3-22ca-43c5-9d37-37e38d418444-istio-data\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-wzfts\" (UID: \"a5d698b3-22ca-43c5-9d37-37e38d418444\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-wzfts" Nov 03 01:05:45.209016 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:45.209004 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9417c149-c243-4ba7-9f66-7ad0688a9f78-workload-certs\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j52qt\" (UID: \"9417c149-c243-4ba7-9f66-7ad0688a9f78\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j52qt" Nov 03 01:05:45.209076 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:45.209064 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9417c149-c243-4ba7-9f66-7ad0688a9f78-istio-podinfo\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j52qt\" (UID: \"9417c149-c243-4ba7-9f66-7ad0688a9f78\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j52qt" Nov 03 01:05:45.209109 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:45.209099 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5d698b3-22ca-43c5-9d37-37e38d418444-workload-certs\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-wzfts\" (UID: \"a5d698b3-22ca-43c5-9d37-37e38d418444\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-wzfts" Nov 03 01:05:45.209200 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:45.209188 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9417c149-c243-4ba7-9f66-7ad0688a9f78-workload-socket\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j52qt\" (UID: \"9417c149-c243-4ba7-9f66-7ad0688a9f78\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j52qt" Nov 03 01:05:45.209237 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:45.209213 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5d698b3-22ca-43c5-9d37-37e38d418444-istio-token\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-wzfts\" (UID: \"a5d698b3-22ca-43c5-9d37-37e38d418444\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-wzfts" Nov 03 01:05:45.209319 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:45.209304 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5d698b3-22ca-43c5-9d37-37e38d418444-workload-socket\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-wzfts\" (UID: \"a5d698b3-22ca-43c5-9d37-37e38d418444\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-wzfts" Nov 03 01:05:45.209359 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:45.209346 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9417c149-c243-4ba7-9f66-7ad0688a9f78-istio-envoy\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j52qt\" (UID: \"9417c149-c243-4ba7-9f66-7ad0688a9f78\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j52qt" Nov 03 01:05:45.209395 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:45.209370 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjsw9\" (UniqueName: \"kubernetes.io/projected/9417c149-c243-4ba7-9f66-7ad0688a9f78-kube-api-access-hjsw9\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j52qt\" (UID: \"9417c149-c243-4ba7-9f66-7ad0688a9f78\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j52qt" Nov 03 01:05:45.209447 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:45.209431 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5d698b3-22ca-43c5-9d37-37e38d418444-istio-envoy\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-wzfts\" (UID: \"a5d698b3-22ca-43c5-9d37-37e38d418444\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-wzfts" Nov 03 01:05:45.209486 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:45.209467 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfl7p\" (UniqueName: \"kubernetes.io/projected/a5d698b3-22ca-43c5-9d37-37e38d418444-kube-api-access-gfl7p\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-wzfts\" (UID: \"a5d698b3-22ca-43c5-9d37-37e38d418444\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-wzfts" Nov 03 01:05:45.209521 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:45.209484 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5d698b3-22ca-43c5-9d37-37e38d418444-istiod-ca-cert\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-wzfts\" (UID: \"a5d698b3-22ca-43c5-9d37-37e38d418444\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-wzfts" Nov 03 01:05:45.310632 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:45.310591 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5d698b3-22ca-43c5-9d37-37e38d418444-workload-socket\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-wzfts\" (UID: \"a5d698b3-22ca-43c5-9d37-37e38d418444\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-wzfts" Nov 03 01:05:45.310817 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:45.310663 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9417c149-c243-4ba7-9f66-7ad0688a9f78-istio-envoy\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j52qt\" (UID: \"9417c149-c243-4ba7-9f66-7ad0688a9f78\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j52qt" Nov 03 01:05:45.310817 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:45.310699 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjsw9\" (UniqueName: \"kubernetes.io/projected/9417c149-c243-4ba7-9f66-7ad0688a9f78-kube-api-access-hjsw9\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j52qt\" (UID: \"9417c149-c243-4ba7-9f66-7ad0688a9f78\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j52qt" Nov 03 01:05:45.310817 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:45.310729 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5d698b3-22ca-43c5-9d37-37e38d418444-istio-envoy\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-wzfts\" (UID: \"a5d698b3-22ca-43c5-9d37-37e38d418444\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-wzfts" Nov 03 01:05:45.310817 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:45.310773 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfl7p\" (UniqueName: \"kubernetes.io/projected/a5d698b3-22ca-43c5-9d37-37e38d418444-kube-api-access-gfl7p\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-wzfts\" (UID: \"a5d698b3-22ca-43c5-9d37-37e38d418444\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-wzfts" Nov 03 01:05:45.310817 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:45.310800 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5d698b3-22ca-43c5-9d37-37e38d418444-istiod-ca-cert\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-wzfts\" (UID: \"a5d698b3-22ca-43c5-9d37-37e38d418444\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-wzfts" Nov 03 01:05:45.311180 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:45.310824 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5d698b3-22ca-43c5-9d37-37e38d418444-istio-podinfo\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-wzfts\" (UID: \"a5d698b3-22ca-43c5-9d37-37e38d418444\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-wzfts" Nov 03 01:05:45.311180 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:45.310849 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9417c149-c243-4ba7-9f66-7ad0688a9f78-istio-data\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j52qt\" (UID: \"9417c149-c243-4ba7-9f66-7ad0688a9f78\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j52qt" Nov 03 01:05:45.311180 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:45.310881 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9417c149-c243-4ba7-9f66-7ad0688a9f78-istio-token\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j52qt\" (UID: \"9417c149-c243-4ba7-9f66-7ad0688a9f78\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j52qt" Nov 03 01:05:45.311180 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:45.310980 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9417c149-c243-4ba7-9f66-7ad0688a9f78-istiod-ca-cert\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j52qt\" (UID: \"9417c149-c243-4ba7-9f66-7ad0688a9f78\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j52qt" Nov 03 01:05:45.311180 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:45.311010 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9417c149-c243-4ba7-9f66-7ad0688a9f78-credential-socket\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j52qt\" (UID: \"9417c149-c243-4ba7-9f66-7ad0688a9f78\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j52qt" Nov 03 01:05:45.311180 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:45.311063 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5d698b3-22ca-43c5-9d37-37e38d418444-credential-socket\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-wzfts\" (UID: \"a5d698b3-22ca-43c5-9d37-37e38d418444\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-wzfts" Nov 03 01:05:45.311180 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:45.311085 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5d698b3-22ca-43c5-9d37-37e38d418444-workload-socket\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-wzfts\" (UID: \"a5d698b3-22ca-43c5-9d37-37e38d418444\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-wzfts" Nov 03 01:05:45.311578 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:45.311191 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5d698b3-22ca-43c5-9d37-37e38d418444-istio-data\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-wzfts\" (UID: \"a5d698b3-22ca-43c5-9d37-37e38d418444\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-wzfts" Nov 03 01:05:45.311578 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:45.311256 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9417c149-c243-4ba7-9f66-7ad0688a9f78-workload-certs\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j52qt\" (UID: \"9417c149-c243-4ba7-9f66-7ad0688a9f78\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j52qt" Nov 03 01:05:45.311578 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:45.311354 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9417c149-c243-4ba7-9f66-7ad0688a9f78-istio-data\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j52qt\" (UID: \"9417c149-c243-4ba7-9f66-7ad0688a9f78\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j52qt" Nov 03 01:05:45.311578 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:45.311366 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9417c149-c243-4ba7-9f66-7ad0688a9f78-istio-podinfo\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j52qt\" (UID: \"9417c149-c243-4ba7-9f66-7ad0688a9f78\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j52qt" Nov 03 01:05:45.311578 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:45.311425 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5d698b3-22ca-43c5-9d37-37e38d418444-credential-socket\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-wzfts\" (UID: \"a5d698b3-22ca-43c5-9d37-37e38d418444\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-wzfts" Nov 03 01:05:45.311578 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:45.311449 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5d698b3-22ca-43c5-9d37-37e38d418444-workload-certs\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-wzfts\" (UID: \"a5d698b3-22ca-43c5-9d37-37e38d418444\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-wzfts" Nov 03 01:05:45.311578 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:45.311524 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9417c149-c243-4ba7-9f66-7ad0688a9f78-workload-socket\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j52qt\" (UID: \"9417c149-c243-4ba7-9f66-7ad0688a9f78\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j52qt" Nov 03 01:05:45.311578 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:45.311571 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5d698b3-22ca-43c5-9d37-37e38d418444-istio-token\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-wzfts\" (UID: \"a5d698b3-22ca-43c5-9d37-37e38d418444\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-wzfts" Nov 03 01:05:45.312018 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:45.311819 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5d698b3-22ca-43c5-9d37-37e38d418444-istiod-ca-cert\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-wzfts\" (UID: \"a5d698b3-22ca-43c5-9d37-37e38d418444\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-wzfts" Nov 03 01:05:45.312162 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:45.312121 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9417c149-c243-4ba7-9f66-7ad0688a9f78-credential-socket\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j52qt\" (UID: \"9417c149-c243-4ba7-9f66-7ad0688a9f78\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j52qt" Nov 03 01:05:45.312247 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:45.312210 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5d698b3-22ca-43c5-9d37-37e38d418444-workload-certs\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-wzfts\" (UID: \"a5d698b3-22ca-43c5-9d37-37e38d418444\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-wzfts" Nov 03 01:05:45.312456 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:45.312435 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5d698b3-22ca-43c5-9d37-37e38d418444-istio-data\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-wzfts\" (UID: \"a5d698b3-22ca-43c5-9d37-37e38d418444\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-wzfts" Nov 03 01:05:45.312512 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:45.312435 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9417c149-c243-4ba7-9f66-7ad0688a9f78-istiod-ca-cert\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j52qt\" (UID: \"9417c149-c243-4ba7-9f66-7ad0688a9f78\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j52qt" Nov 03 01:05:45.312575 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:45.312528 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9417c149-c243-4ba7-9f66-7ad0688a9f78-workload-socket\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j52qt\" (UID: \"9417c149-c243-4ba7-9f66-7ad0688a9f78\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j52qt" Nov 03 01:05:45.312711 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:45.312687 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9417c149-c243-4ba7-9f66-7ad0688a9f78-workload-certs\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j52qt\" (UID: \"9417c149-c243-4ba7-9f66-7ad0688a9f78\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j52qt" Nov 03 01:05:45.313656 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:45.313626 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5d698b3-22ca-43c5-9d37-37e38d418444-istio-envoy\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-wzfts\" (UID: \"a5d698b3-22ca-43c5-9d37-37e38d418444\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-wzfts" Nov 03 01:05:45.313950 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:45.313924 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9417c149-c243-4ba7-9f66-7ad0688a9f78-istio-envoy\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j52qt\" (UID: \"9417c149-c243-4ba7-9f66-7ad0688a9f78\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j52qt" Nov 03 01:05:45.314053 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:45.314009 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9417c149-c243-4ba7-9f66-7ad0688a9f78-istio-podinfo\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j52qt\" (UID: \"9417c149-c243-4ba7-9f66-7ad0688a9f78\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j52qt" Nov 03 01:05:45.314576 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:45.314540 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5d698b3-22ca-43c5-9d37-37e38d418444-istio-podinfo\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-wzfts\" (UID: \"a5d698b3-22ca-43c5-9d37-37e38d418444\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-wzfts" Nov 03 01:05:45.317888 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:45.317865 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfl7p\" (UniqueName: \"kubernetes.io/projected/a5d698b3-22ca-43c5-9d37-37e38d418444-kube-api-access-gfl7p\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-wzfts\" (UID: \"a5d698b3-22ca-43c5-9d37-37e38d418444\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-wzfts" Nov 03 01:05:45.318144 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:45.318126 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjsw9\" (UniqueName: \"kubernetes.io/projected/9417c149-c243-4ba7-9f66-7ad0688a9f78-kube-api-access-hjsw9\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j52qt\" (UID: \"9417c149-c243-4ba7-9f66-7ad0688a9f78\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j52qt" Nov 03 01:05:45.320617 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:45.320592 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9417c149-c243-4ba7-9f66-7ad0688a9f78-istio-token\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j52qt\" (UID: \"9417c149-c243-4ba7-9f66-7ad0688a9f78\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j52qt" Nov 03 01:05:45.320942 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:45.320923 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5d698b3-22ca-43c5-9d37-37e38d418444-istio-token\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-wzfts\" (UID: \"a5d698b3-22ca-43c5-9d37-37e38d418444\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-wzfts" Nov 03 01:05:45.452938 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:45.452850 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-wzfts" Nov 03 01:05:45.465650 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:45.465626 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j52qt" Nov 03 01:05:45.751133 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:45.751092 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-wzfts"] Nov 03 01:05:45.755398 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:45.755361 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:05:45.755507 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:45.755433 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:05:45.755507 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:45.755466 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:05:45.771348 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:05:45.771257 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9417c149_c243_4ba7_9f66_7ad0688a9f78.slice/crio-659390c7b7b414dd071cc1dd1d4293746d608cd2a0a6ebed598b119342b9a7af WatchSource:0}: Error finding container 659390c7b7b414dd071cc1dd1d4293746d608cd2a0a6ebed598b119342b9a7af: Status 404 returned error can't find the container with id 659390c7b7b414dd071cc1dd1d4293746d608cd2a0a6ebed598b119342b9a7af Nov 03 01:05:45.773501 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:45.773477 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j52qt"] Nov 03 01:05:45.774853 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:45.774822 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:05:45.774952 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:45.774889 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:05:45.774952 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:45.774925 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:05:46.213226 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:46.213126 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-wzfts" event={"ID":"a5d698b3-22ca-43c5-9d37-37e38d418444","Type":"ContainerStarted","Data":"37a2ddb3407dde0c226d00c827faa3b7e19edc83e8b14a160e1bec20cdccceb3"} Nov 03 01:05:46.213226 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:46.213192 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-wzfts" event={"ID":"a5d698b3-22ca-43c5-9d37-37e38d418444","Type":"ContainerStarted","Data":"2119ba706eebb10d5fd0f9fe2f2325fad0cfa26f325d4c00ae4ae6814ff0d1ec"} Nov 03 01:05:46.215828 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:46.215785 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j52qt" event={"ID":"9417c149-c243-4ba7-9f66-7ad0688a9f78","Type":"ContainerStarted","Data":"c39c010b7c84d7a414eb844b763fa6e4134edc385464d389a6b7425498aca746"} Nov 03 01:05:46.215971 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:46.215845 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j52qt" event={"ID":"9417c149-c243-4ba7-9f66-7ad0688a9f78","Type":"ContainerStarted","Data":"659390c7b7b414dd071cc1dd1d4293746d608cd2a0a6ebed598b119342b9a7af"} Nov 03 01:05:46.236502 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:46.236444 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-wzfts" podStartSLOduration=1.236429537 podStartE2EDuration="1.236429537s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:05:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:05:46.232038181 +0000 UTC m=+1488.403041278" watchObservedRunningTime="2026-11-03 01:05:46.236429537 +0000 UTC m=+1488.407432633" Nov 03 01:05:46.260025 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:46.259961 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j52qt" podStartSLOduration=1.259942275 podStartE2EDuration="1.259942275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:05:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:05:46.254502067 +0000 UTC m=+1488.425505164" watchObservedRunningTime="2026-11-03 01:05:46.259942275 +0000 UTC m=+1488.430945373" Nov 03 01:05:46.453925 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:46.453893 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-wzfts" Nov 03 01:05:46.455618 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:46.455590 4421 patch_prober.go:29] interesting pod/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-wzfts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused" start-of-body= Nov 03 01:05:46.455729 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:46.455644 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-wzfts" podUID="a5d698b3-22ca-43c5-9d37-37e38d418444" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused" Nov 03 01:05:46.465921 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:46.465852 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j52qt" Nov 03 01:05:46.467226 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:46.467206 4421 patch_prober.go:29] interesting pod/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j52qt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" start-of-body= Nov 03 01:05:46.467330 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:46.467259 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j52qt" podUID="9417c149-c243-4ba7-9f66-7ad0688a9f78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" Nov 03 01:05:47.467956 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:47.467916 4421 patch_prober.go:29] interesting pod/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j52qt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" start-of-body= Nov 03 01:05:47.468607 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:47.468572 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j52qt" podUID="9417c149-c243-4ba7-9f66-7ad0688a9f78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" Nov 03 01:05:47.469459 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:47.469434 4421 patch_prober.go:29] interesting pod/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-wzfts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused" start-of-body= Nov 03 01:05:47.469643 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:47.469623 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-wzfts" podUID="a5d698b3-22ca-43c5-9d37-37e38d418444" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused" Nov 03 01:05:48.561308 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:48.551972 4421 patch_prober.go:29] interesting pod/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-wzfts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused" start-of-body= Nov 03 01:05:48.561308 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:48.552052 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-wzfts" podUID="a5d698b3-22ca-43c5-9d37-37e38d418444" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused" Nov 03 01:05:48.561308 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:48.558501 4421 patch_prober.go:29] interesting pod/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j52qt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" start-of-body= Nov 03 01:05:48.561308 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:48.558942 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j52qt" podUID="9417c149-c243-4ba7-9f66-7ad0688a9f78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" Nov 03 01:05:49.455570 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:49.455533 4421 patch_prober.go:29] interesting pod/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-wzfts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused" start-of-body= Nov 03 01:05:49.455888 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:49.455859 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-wzfts" podUID="a5d698b3-22ca-43c5-9d37-37e38d418444" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused" Nov 03 01:05:49.468296 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:49.466062 4421 patch_prober.go:29] interesting pod/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j52qt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" start-of-body= Nov 03 01:05:49.468296 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:49.466100 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j52qt" podUID="9417c149-c243-4ba7-9f66-7ad0688a9f78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" Nov 03 01:05:50.461300 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:50.457822 4421 patch_prober.go:29] interesting pod/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-wzfts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused" start-of-body= Nov 03 01:05:50.461300 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:50.458416 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-wzfts" podUID="a5d698b3-22ca-43c5-9d37-37e38d418444" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused" Nov 03 01:05:50.484654 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:50.484616 4421 patch_prober.go:29] interesting pod/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j52qt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" start-of-body= Nov 03 01:05:50.484984 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:50.484951 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j52qt" podUID="9417c149-c243-4ba7-9f66-7ad0688a9f78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" Nov 03 01:05:51.473299 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:51.472482 4421 patch_prober.go:29] interesting pod/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j52qt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" start-of-body= Nov 03 01:05:51.473299 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:51.472547 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j52qt" podUID="9417c149-c243-4ba7-9f66-7ad0688a9f78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" Nov 03 01:05:51.498296 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:51.491722 4421 patch_prober.go:29] interesting pod/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-wzfts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused" start-of-body= Nov 03 01:05:51.498296 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:51.491781 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-wzfts" podUID="a5d698b3-22ca-43c5-9d37-37e38d418444" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused" Nov 03 01:05:52.235301 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:52.234367 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/simulator-w3-83755a-kserve-77d6664559-psqtg" podUID="b0a4c234-af58-4da0-a1a4-e1f167953f6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:05:52.320404 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:52.320355 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/premium-w3-83755a-kserve-7c9d66988b-lt4xd" podUID="c0c6b02e-2ecb-428e-a13d-be6d7cd1ba39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:05:52.342419 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:52.342374 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/distinct-w3-83755a-kserve-86bdbbdd85-2cnnl" podUID="f4ae9624-d127-4a55-837d-7c9ff0267b37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded" Nov 03 01:05:52.415461 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:52.415412 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/simulator-w2-f7680d-kserve-85bdc86cd6-8gxrg" podUID="2bdcf521-cebd-4d6b-8554-1b60db8b0f3c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:05:52.458746 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:52.458710 4421 patch_prober.go:29] interesting pod/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-wzfts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused" start-of-body= Nov 03 01:05:52.459058 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:52.459028 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-wzfts" podUID="a5d698b3-22ca-43c5-9d37-37e38d418444" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused" Nov 03 01:05:52.471490 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:52.471446 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8gm8n" podUID="2d367e12-e46f-4a68-9825-9b1c934d4d12" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:05:52.473315 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:52.473265 4421 patch_prober.go:29] interesting pod/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j52qt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" start-of-body= Nov 03 01:05:52.473873 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:52.473846 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j52qt" podUID="9417c149-c243-4ba7-9f66-7ad0688a9f78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" Nov 03 01:05:52.492987 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:52.492889 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-650be0/simulator-w4-650be0-kserve-7bc588774f-gvz2k" podUID="a827fa88-f078-4312-8f12-2c799c11b461" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:05:52.574029 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:52.573984 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-650be0/premium-w4-650be0-kserve-7bf8555dfb-z97n5" podUID="40601dc5-808f-456f-826f-6e0c35966ada" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:05:52.691674 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:52.691388 4421 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-83755a/simulator-w3-83755a-kserve-77d6664559-psqtg" podUID="b0a4c234-af58-4da0-a1a4-e1f167953f6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:05:53.066999 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:53.066958 4421 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-650be0/simulator-w4-650be0-kserve-7bc588774f-gvz2k" podUID="a827fa88-f078-4312-8f12-2c799c11b461" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": context deadline exceeded" Nov 03 01:05:53.242298 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:53.239386 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/simulator-w3-83755a-kserve-77d6664559-psqtg" podUID="b0a4c234-af58-4da0-a1a4-e1f167953f6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:05:53.242298 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:53.239869 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-83755a/simulator-w3-83755a-kserve-77d6664559-psqtg" Nov 03 01:05:53.327775 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:53.327657 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/premium-w3-83755a-kserve-7c9d66988b-lt4xd" podUID="c0c6b02e-2ecb-428e-a13d-be6d7cd1ba39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:05:53.328778 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:53.328755 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-83755a/premium-w3-83755a-kserve-7c9d66988b-lt4xd" Nov 03 01:05:53.352587 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:53.352543 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/distinct-w3-83755a-kserve-86bdbbdd85-2cnnl" podUID="f4ae9624-d127-4a55-837d-7c9ff0267b37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:05:53.354010 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:53.353985 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-83755a/distinct-w3-83755a-kserve-86bdbbdd85-2cnnl" Nov 03 01:05:53.454742 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:53.454433 4421 patch_prober.go:29] interesting pod/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-wzfts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused" start-of-body= Nov 03 01:05:53.454742 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:53.454488 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-wzfts" podUID="a5d698b3-22ca-43c5-9d37-37e38d418444" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused" Nov 03 01:05:53.456296 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:53.455208 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/simulator-w2-f7680d-kserve-85bdc86cd6-8gxrg" podUID="2bdcf521-cebd-4d6b-8554-1b60db8b0f3c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:05:53.456296 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:53.455617 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f7680d/simulator-w2-f7680d-kserve-85bdc86cd6-8gxrg" Nov 03 01:05:53.494412 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:53.494355 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-650be0/simulator-w4-650be0-kserve-7bc588774f-gvz2k" podUID="a827fa88-f078-4312-8f12-2c799c11b461" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:05:53.494927 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:53.494805 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8gm8n" podUID="2d367e12-e46f-4a68-9825-9b1c934d4d12" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": context deadline exceeded" Nov 03 01:05:53.498294 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:53.495200 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-650be0/simulator-w4-650be0-kserve-7bc588774f-gvz2k" Nov 03 01:05:53.498294 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:53.495262 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8gm8n" Nov 03 01:05:53.551032 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:53.549110 4421 patch_prober.go:29] interesting pod/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j52qt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" start-of-body= Nov 03 01:05:53.551032 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:53.549173 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j52qt" podUID="9417c149-c243-4ba7-9f66-7ad0688a9f78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" Nov 03 01:05:53.617596 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:53.617494 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-650be0/premium-w4-650be0-kserve-7bf8555dfb-z97n5" podUID="40601dc5-808f-456f-826f-6e0c35966ada" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:05:53.674307 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:53.673194 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-650be0/premium-w4-650be0-kserve-7bf8555dfb-z97n5" Nov 03 01:05:54.250301 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:54.239546 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/simulator-w3-83755a-kserve-77d6664559-psqtg" podUID="b0a4c234-af58-4da0-a1a4-e1f167953f6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Nov 03 01:05:54.342845 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:54.342801 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/premium-w3-83755a-kserve-7c9d66988b-lt4xd" podUID="c0c6b02e-2ecb-428e-a13d-be6d7cd1ba39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:05:54.398253 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:54.398208 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/distinct-w3-83755a-kserve-86bdbbdd85-2cnnl" podUID="f4ae9624-d127-4a55-837d-7c9ff0267b37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded" Nov 03 01:05:54.515604 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:54.515488 4421 patch_prober.go:29] interesting pod/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-wzfts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused" start-of-body= Nov 03 01:05:54.516543 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:54.516215 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-wzfts" podUID="a5d698b3-22ca-43c5-9d37-37e38d418444" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused" Nov 03 01:05:54.540407 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:54.540353 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-650be0/simulator-w4-650be0-kserve-7bc588774f-gvz2k" podUID="a827fa88-f078-4312-8f12-2c799c11b461" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Nov 03 01:05:54.541383 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:54.541338 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/simulator-w2-f7680d-kserve-85bdc86cd6-8gxrg" podUID="2bdcf521-cebd-4d6b-8554-1b60db8b0f3c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 01:05:54.643324 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:54.609614 4421 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-650be0/premium-w4-650be0-kserve-7bf8555dfb-z97n5" podUID="40601dc5-808f-456f-826f-6e0c35966ada" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": context deadline exceeded" Nov 03 01:05:54.684877 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:54.684829 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-650be0/premium-w4-650be0-kserve-7bf8555dfb-z97n5" podUID="40601dc5-808f-456f-826f-6e0c35966ada" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Nov 03 01:05:54.727304 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:54.715215 4421 patch_prober.go:29] interesting pod/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j52qt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" start-of-body= Nov 03 01:05:54.785585 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:54.785475 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j52qt" podUID="9417c149-c243-4ba7-9f66-7ad0688a9f78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" Nov 03 01:05:54.885977 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:54.831361 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8gm8n" podUID="2d367e12-e46f-4a68-9825-9b1c934d4d12" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:05:55.243371 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:55.239756 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/simulator-w3-83755a-kserve-77d6664559-psqtg" podUID="b0a4c234-af58-4da0-a1a4-e1f167953f6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:05:55.336302 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:55.335687 4421 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8gm8n" podUID="2d367e12-e46f-4a68-9825-9b1c934d4d12" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:05:55.364302 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:55.349413 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/premium-w3-83755a-kserve-7c9d66988b-lt4xd" podUID="c0c6b02e-2ecb-428e-a13d-be6d7cd1ba39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:05:55.427489 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:55.427385 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/distinct-w3-83755a-kserve-86bdbbdd85-2cnnl" podUID="f4ae9624-d127-4a55-837d-7c9ff0267b37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:05:55.546708 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:55.546652 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-650be0/simulator-w4-650be0-kserve-7bc588774f-gvz2k" podUID="a827fa88-f078-4312-8f12-2c799c11b461" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Nov 03 01:05:55.550528 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:55.550488 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/simulator-w2-f7680d-kserve-85bdc86cd6-8gxrg" podUID="2bdcf521-cebd-4d6b-8554-1b60db8b0f3c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:05:55.686356 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:55.685319 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-650be0/premium-w4-650be0-kserve-7bf8555dfb-z97n5" podUID="40601dc5-808f-456f-826f-6e0c35966ada" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:05:55.901527 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:55.900528 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8gm8n" podUID="2d367e12-e46f-4a68-9825-9b1c934d4d12" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:05:56.070987 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:56.070952 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8gm8n" Nov 03 01:05:56.123469 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:56.123426 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j52qt" Nov 03 01:05:56.131952 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:56.131499 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j52qt" Nov 03 01:05:56.138185 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:56.138157 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j52qt" Nov 03 01:05:56.297298 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:56.285696 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-wzfts" Nov 03 01:05:56.316303 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:56.309335 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/simulator-w3-83755a-kserve-77d6664559-psqtg" podUID="b0a4c234-af58-4da0-a1a4-e1f167953f6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Nov 03 01:05:56.362297 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:56.352559 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/premium-w3-83755a-kserve-7c9d66988b-lt4xd" podUID="c0c6b02e-2ecb-428e-a13d-be6d7cd1ba39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:05:56.362297 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:56.361174 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-wzfts" Nov 03 01:05:56.378330 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:56.378009 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-wzfts" Nov 03 01:05:56.472298 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:56.465931 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/distinct-w3-83755a-kserve-86bdbbdd85-2cnnl" podUID="f4ae9624-d127-4a55-837d-7c9ff0267b37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:05:56.549369 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:56.547593 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-650be0/simulator-w4-650be0-kserve-7bc588774f-gvz2k" podUID="a827fa88-f078-4312-8f12-2c799c11b461" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Nov 03 01:05:56.583338 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:56.568425 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/simulator-w2-f7680d-kserve-85bdc86cd6-8gxrg" podUID="2bdcf521-cebd-4d6b-8554-1b60db8b0f3c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:05:56.701301 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:56.694437 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-650be0/premium-w4-650be0-kserve-7bf8555dfb-z97n5" podUID="40601dc5-808f-456f-826f-6e0c35966ada" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:05:56.994363 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:56.987384 4421 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-83755a/distinct-w3-83755a-kserve-86bdbbdd85-2cnnl" podUID="f4ae9624-d127-4a55-837d-7c9ff0267b37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:05:57.313524 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:57.313459 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/simulator-w3-83755a-kserve-77d6664559-psqtg" podUID="b0a4c234-af58-4da0-a1a4-e1f167953f6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Nov 03 01:05:57.355134 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:57.355083 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/premium-w3-83755a-kserve-7c9d66988b-lt4xd" podUID="c0c6b02e-2ecb-428e-a13d-be6d7cd1ba39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:05:57.467416 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:57.467368 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/distinct-w3-83755a-kserve-86bdbbdd85-2cnnl" podUID="f4ae9624-d127-4a55-837d-7c9ff0267b37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded" Nov 03 01:05:57.558450 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:57.558398 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-650be0/simulator-w4-650be0-kserve-7bc588774f-gvz2k" podUID="a827fa88-f078-4312-8f12-2c799c11b461" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:05:57.591150 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:57.569348 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/simulator-w2-f7680d-kserve-85bdc86cd6-8gxrg" podUID="2bdcf521-cebd-4d6b-8554-1b60db8b0f3c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 03 01:05:57.703653 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:57.703611 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-650be0/premium-w4-650be0-kserve-7bf8555dfb-z97n5" podUID="40601dc5-808f-456f-826f-6e0c35966ada" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Nov 03 01:05:57.908446 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:57.905634 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8gm8n" podUID="2d367e12-e46f-4a68-9825-9b1c934d4d12" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:05:58.320005 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:58.319954 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/simulator-w3-83755a-kserve-77d6664559-psqtg" podUID="b0a4c234-af58-4da0-a1a4-e1f167953f6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Nov 03 01:05:58.427603 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:58.427548 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/premium-w3-83755a-kserve-7c9d66988b-lt4xd" podUID="c0c6b02e-2ecb-428e-a13d-be6d7cd1ba39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 01:05:58.533367 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:58.532892 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/distinct-w3-83755a-kserve-86bdbbdd85-2cnnl" podUID="f4ae9624-d127-4a55-837d-7c9ff0267b37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded" Nov 03 01:05:58.596738 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:58.596634 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-650be0/simulator-w4-650be0-kserve-7bc588774f-gvz2k" podUID="a827fa88-f078-4312-8f12-2c799c11b461" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:05:58.621028 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:58.620374 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/simulator-w2-f7680d-kserve-85bdc86cd6-8gxrg" podUID="2bdcf521-cebd-4d6b-8554-1b60db8b0f3c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:05:58.725448 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:58.722426 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-650be0/premium-w4-650be0-kserve-7bf8555dfb-z97n5" podUID="40601dc5-808f-456f-826f-6e0c35966ada" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:05:58.909693 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:58.909572 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8gm8n" podUID="2d367e12-e46f-4a68-9825-9b1c934d4d12" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:05:58.910663 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:58.910633 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8gm8n" Nov 03 01:05:59.347299 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:59.346047 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/simulator-w3-83755a-kserve-77d6664559-psqtg" podUID="b0a4c234-af58-4da0-a1a4-e1f167953f6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Nov 03 01:05:59.471656 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:59.471595 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/premium-w3-83755a-kserve-7c9d66988b-lt4xd" podUID="c0c6b02e-2ecb-428e-a13d-be6d7cd1ba39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:05:59.538186 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:59.538143 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/distinct-w3-83755a-kserve-86bdbbdd85-2cnnl" podUID="f4ae9624-d127-4a55-837d-7c9ff0267b37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:05:59.628504 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:59.628374 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-650be0/simulator-w4-650be0-kserve-7bc588774f-gvz2k" podUID="a827fa88-f078-4312-8f12-2c799c11b461" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:05:59.629774 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:59.629735 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/simulator-w2-f7680d-kserve-85bdc86cd6-8gxrg" podUID="2bdcf521-cebd-4d6b-8554-1b60db8b0f3c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:05:59.735043 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:59.734995 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-650be0/premium-w4-650be0-kserve-7bf8555dfb-z97n5" podUID="40601dc5-808f-456f-826f-6e0c35966ada" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:05:59.911533 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:05:59.911414 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8gm8n" podUID="2d367e12-e46f-4a68-9825-9b1c934d4d12" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:06:00.371916 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:00.371868 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/simulator-w3-83755a-kserve-77d6664559-psqtg" podUID="b0a4c234-af58-4da0-a1a4-e1f167953f6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Nov 03 01:06:00.488344 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:00.476846 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/premium-w3-83755a-kserve-7c9d66988b-lt4xd" podUID="c0c6b02e-2ecb-428e-a13d-be6d7cd1ba39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:06:00.540478 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:00.540376 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/distinct-w3-83755a-kserve-86bdbbdd85-2cnnl" podUID="f4ae9624-d127-4a55-837d-7c9ff0267b37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:06:00.637492 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:00.637392 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/simulator-w2-f7680d-kserve-85bdc86cd6-8gxrg" podUID="2bdcf521-cebd-4d6b-8554-1b60db8b0f3c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 03 01:06:00.638417 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:00.638376 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-650be0/simulator-w4-650be0-kserve-7bc588774f-gvz2k" podUID="a827fa88-f078-4312-8f12-2c799c11b461" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Nov 03 01:06:00.738641 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:00.738594 4421 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-f7680d/simulator-w2-f7680d-kserve-85bdc86cd6-8gxrg" podUID="2bdcf521-cebd-4d6b-8554-1b60db8b0f3c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:06:00.739119 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:00.739088 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-650be0/premium-w4-650be0-kserve-7bf8555dfb-z97n5" podUID="40601dc5-808f-456f-826f-6e0c35966ada" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:06:00.861672 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:00.861634 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-83755a/premium-w3-83755a-kserve-7c9d66988b-lt4xd" Nov 03 01:06:00.915369 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:00.912106 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8gm8n" podUID="2d367e12-e46f-4a68-9825-9b1c934d4d12" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:06:01.088450 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:01.086915 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-650be0/simulator-w4-650be0-kserve-7bc588774f-gvz2k" Nov 03 01:06:01.097747 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:01.096755 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-650be0/premium-w4-650be0-kserve-7bf8555dfb-z97n5" Nov 03 01:06:01.259791 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:01.259199 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-83755a/distinct-w3-83755a-kserve-86bdbbdd85-2cnnl" Nov 03 01:06:01.282900 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:01.272137 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8gm8n" Nov 03 01:06:01.291986 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:01.289711 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f7680d/simulator-w2-f7680d-kserve-85bdc86cd6-8gxrg" Nov 03 01:06:01.299171 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:01.299145 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-83755a/simulator-w3-83755a-kserve-77d6664559-psqtg" Nov 03 01:06:02.831300 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:02.830656 4421 patch_prober.go:29] interesting pod/e2e-shared-a-w3-717217-openshift-default-578c69554c-gwl6k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.201:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 01:06:02.831300 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:02.830732 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-gwl6k" podUID="e6c36e3a-89e3-4f54-8266-c7dcfc0ca19b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.201:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 01:06:03.557717 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:03.557430 4421 patch_prober.go:29] interesting pod/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-sjhcb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.208:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 01:06:03.557717 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:03.557486 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-sjhcb" podUID="d1d81d36-60e4-4f12-81e6-7c5be483af12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.208:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 01:06:07.760388 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:07.760337 4421 patch_prober.go:29] interesting pod/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-v8jpj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.209:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 01:06:07.760908 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:07.760417 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-v8jpj" podUID="9ca37a4e-e37b-453c-a937-a35fedc2af8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.209:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 01:06:08.825577 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:08.825541 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-020af9-7fcf787fd7-9fgx6"] Nov 03 01:06:08.893489 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:08.893455 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-020af9-7fcf787fd7-9fgx6"] Nov 03 01:06:08.893683 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:08.893512 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-020af9-66449c479f-bmnx4"] Nov 03 01:06:08.893984 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:08.893908 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-020af9-7fcf787fd7-9fgx6" Nov 03 01:06:09.036307 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:08.926167 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-020af9\"" Nov 03 01:06:09.036307 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:08.926910 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-020af9-dockercfg-hw9tl\"" Nov 03 01:06:09.036307 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:09.019651 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-020af9-66449c479f-bmnx4"] Nov 03 01:06:09.036307 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:09.034517 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-020af9-66449c479f-bmnx4" Nov 03 01:06:09.102240 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:09.102134 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcfbn\" (UniqueName: \"kubernetes.io/projected/7f5f8dc1-5619-4ffb-b968-06869fa59c39-kube-api-access-tcfbn\") pod \"payload-pre-processing-e2e-shared-b-w0-020af9-7fcf787fd7-9fgx6\" (UID: \"7f5f8dc1-5619-4ffb-b968-06869fa59c39\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-020af9-7fcf787fd7-9fgx6" Nov 03 01:06:09.102726 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:09.102684 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f5f8dc1-5619-4ffb-b968-06869fa59c39-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-020af9-7fcf787fd7-9fgx6\" (UID: \"7f5f8dc1-5619-4ffb-b968-06869fa59c39\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-020af9-7fcf787fd7-9fgx6" Nov 03 01:06:09.208257 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:09.208220 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lk9wg\" (UniqueName: \"kubernetes.io/projected/c671f325-44a0-497f-afb5-24f3871cc496-kube-api-access-lk9wg\") pod \"payload-processing-e2e-shared-b-w0-020af9-66449c479f-bmnx4\" (UID: \"c671f325-44a0-497f-afb5-24f3871cc496\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-020af9-66449c479f-bmnx4" Nov 03 01:06:09.208486 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:09.208318 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f5f8dc1-5619-4ffb-b968-06869fa59c39-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-020af9-7fcf787fd7-9fgx6\" (UID: \"7f5f8dc1-5619-4ffb-b968-06869fa59c39\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-020af9-7fcf787fd7-9fgx6" Nov 03 01:06:09.208573 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:09.208549 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c671f325-44a0-497f-afb5-24f3871cc496-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-020af9-66449c479f-bmnx4\" (UID: \"c671f325-44a0-497f-afb5-24f3871cc496\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-020af9-66449c479f-bmnx4" Nov 03 01:06:09.209033 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:09.209010 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f5f8dc1-5619-4ffb-b968-06869fa59c39-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-020af9-7fcf787fd7-9fgx6\" (UID: \"7f5f8dc1-5619-4ffb-b968-06869fa59c39\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-020af9-7fcf787fd7-9fgx6" Nov 03 01:06:09.209338 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:09.209299 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcfbn\" (UniqueName: \"kubernetes.io/projected/7f5f8dc1-5619-4ffb-b968-06869fa59c39-kube-api-access-tcfbn\") pod \"payload-pre-processing-e2e-shared-b-w0-020af9-7fcf787fd7-9fgx6\" (UID: \"7f5f8dc1-5619-4ffb-b968-06869fa59c39\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-020af9-7fcf787fd7-9fgx6" Nov 03 01:06:09.217930 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:09.217876 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcfbn\" (UniqueName: \"kubernetes.io/projected/7f5f8dc1-5619-4ffb-b968-06869fa59c39-kube-api-access-tcfbn\") pod \"payload-pre-processing-e2e-shared-b-w0-020af9-7fcf787fd7-9fgx6\" (UID: \"7f5f8dc1-5619-4ffb-b968-06869fa59c39\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-020af9-7fcf787fd7-9fgx6" Nov 03 01:06:09.311045 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:09.311000 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lk9wg\" (UniqueName: \"kubernetes.io/projected/c671f325-44a0-497f-afb5-24f3871cc496-kube-api-access-lk9wg\") pod \"payload-processing-e2e-shared-b-w0-020af9-66449c479f-bmnx4\" (UID: \"c671f325-44a0-497f-afb5-24f3871cc496\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-020af9-66449c479f-bmnx4" Nov 03 01:06:09.311359 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:09.311176 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c671f325-44a0-497f-afb5-24f3871cc496-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-020af9-66449c479f-bmnx4\" (UID: \"c671f325-44a0-497f-afb5-24f3871cc496\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-020af9-66449c479f-bmnx4" Nov 03 01:06:09.312183 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:09.312153 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c671f325-44a0-497f-afb5-24f3871cc496-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-020af9-66449c479f-bmnx4\" (UID: \"c671f325-44a0-497f-afb5-24f3871cc496\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-020af9-66449c479f-bmnx4" Nov 03 01:06:09.321397 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:09.321363 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lk9wg\" (UniqueName: \"kubernetes.io/projected/c671f325-44a0-497f-afb5-24f3871cc496-kube-api-access-lk9wg\") pod \"payload-processing-e2e-shared-b-w0-020af9-66449c479f-bmnx4\" (UID: \"c671f325-44a0-497f-afb5-24f3871cc496\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-020af9-66449c479f-bmnx4" Nov 03 01:06:09.492374 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:09.492289 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-020af9-7fcf787fd7-9fgx6" Nov 03 01:06:09.578181 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:09.578139 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-020af9-66449c479f-bmnx4" Nov 03 01:06:09.929300 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:09.925961 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-020af9-7fcf787fd7-9fgx6"] Nov 03 01:06:10.044330 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:10.044197 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-020af9-66449c479f-bmnx4"] Nov 03 01:06:10.044786 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:06:10.044744 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc671f325_44a0_497f_afb5_24f3871cc496.slice/crio-111cd64b5758df0e0a7bf53bab3376b7e06e099f1a09cf0cfa867f59939f5370 WatchSource:0}: Error finding container 111cd64b5758df0e0a7bf53bab3376b7e06e099f1a09cf0cfa867f59939f5370: Status 404 returned error can't find the container with id 111cd64b5758df0e0a7bf53bab3376b7e06e099f1a09cf0cfa867f59939f5370 Nov 03 01:06:10.432786 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:10.432448 4421 patch_prober.go:29] interesting pod/e2e-autores-4248ae-openshift-default-94bc8c67-b5pgk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.202:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 01:06:10.432786 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:10.432512 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-autores-4248ae-openshift-default-94bc8c67-b5pgk" podUID="2afb12c9-7c7a-4527-95c4-0b2febd22e90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.202:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 01:06:10.794292 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:10.794194 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-020af9-66449c479f-bmnx4" event={"ID":"c671f325-44a0-497f-afb5-24f3871cc496","Type":"ContainerStarted","Data":"e445366f56173baa16d4e190841a3e0dff3eaffcbe18c6d11775c966f22c74fe"} Nov 03 01:06:10.798337 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:10.794312 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-020af9-66449c479f-bmnx4" event={"ID":"c671f325-44a0-497f-afb5-24f3871cc496","Type":"ContainerStarted","Data":"111cd64b5758df0e0a7bf53bab3376b7e06e099f1a09cf0cfa867f59939f5370"} Nov 03 01:06:10.799847 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:10.799448 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-020af9-66449c479f-bmnx4" Nov 03 01:06:10.803774 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:10.801908 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-020af9-7fcf787fd7-9fgx6" event={"ID":"7f5f8dc1-5619-4ffb-b968-06869fa59c39","Type":"ContainerStarted","Data":"6630ccebf1f1ad6d0a9d1180bf83ad096d3e8150dcaea1b51fe21615822b338c"} Nov 03 01:06:10.803774 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:10.801966 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-020af9-7fcf787fd7-9fgx6" event={"ID":"7f5f8dc1-5619-4ffb-b968-06869fa59c39","Type":"ContainerStarted","Data":"7f8f9a835cd8a74893a1af9a7a937a0dd44210308b0e84343b8b2b57d7037cd2"} Nov 03 01:06:10.803774 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:10.802896 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-020af9-7fcf787fd7-9fgx6" Nov 03 01:06:10.828298 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:10.826054 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-020af9-66449c479f-bmnx4" podStartSLOduration=2.826038315 podStartE2EDuration="2.826038315s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:06:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:06:10.81634779 +0000 UTC m=+1512.987350886" watchObservedRunningTime="2026-11-03 01:06:10.826038315 +0000 UTC m=+1512.997041411" Nov 03 01:06:10.847841 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:10.847773 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-020af9-7fcf787fd7-9fgx6" podStartSLOduration=2.847751628 podStartE2EDuration="2.847751628s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:06:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:06:10.842161005 +0000 UTC m=+1513.013164101" watchObservedRunningTime="2026-11-03 01:06:10.847751628 +0000 UTC m=+1513.018754724" Nov 03 01:06:14.745921 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:06:14.745561 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 01:06:14.951770 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:14.951736 4421 generic.go:356] "Generic (PLEG): container finished" podID="2afb12c9-7c7a-4527-95c4-0b2febd22e90" containerID="20a772bbd0dc8daf02ef3acf699274dd1c38946168746fc7b03613467719f1b5" exitCode=0 Nov 03 01:06:14.951977 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:14.951802 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-4248ae-openshift-default-94bc8c67-b5pgk" event={"ID":"2afb12c9-7c7a-4527-95c4-0b2febd22e90","Type":"ContainerDied","Data":"20a772bbd0dc8daf02ef3acf699274dd1c38946168746fc7b03613467719f1b5"} Nov 03 01:06:14.952283 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:14.952251 4421 scope.go:122] "RemoveContainer" containerID="20a772bbd0dc8daf02ef3acf699274dd1c38946168746fc7b03613467719f1b5" Nov 03 01:06:14.954311 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:14.953951 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:06:14.954311 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:14.954034 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:06:14.954311 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:14.954070 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:06:15.287048 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:15.286921 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-27vg6"] Nov 03 01:06:15.344606 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:15.339546 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-27vg6"] Nov 03 01:06:15.344606 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:15.339603 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-pvzvs"] Nov 03 01:06:15.344606 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:15.341584 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-27vg6" Nov 03 01:06:15.456436 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:15.363151 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-pvzvs"] Nov 03 01:06:15.456436 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:15.363302 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-pvzvs" Nov 03 01:06:15.521232 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:15.489412 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae4f838b-7b65-477f-8e03-56a00c7635b1-workload-certs\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-pvzvs\" (UID: \"ae4f838b-7b65-477f-8e03-56a00c7635b1\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-pvzvs" Nov 03 01:06:15.521232 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:15.489474 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae4f838b-7b65-477f-8e03-56a00c7635b1-workload-socket\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-pvzvs\" (UID: \"ae4f838b-7b65-477f-8e03-56a00c7635b1\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-pvzvs" Nov 03 01:06:15.521232 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:15.489520 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/793a3a3a-a1fa-4174-a94a-a8382bc6f412-workload-socket\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-27vg6\" (UID: \"793a3a3a-a1fa-4174-a94a-a8382bc6f412\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-27vg6" Nov 03 01:06:15.521232 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:15.489555 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae4f838b-7b65-477f-8e03-56a00c7635b1-istio-token\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-pvzvs\" (UID: \"ae4f838b-7b65-477f-8e03-56a00c7635b1\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-pvzvs" Nov 03 01:06:15.521232 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:15.489678 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/793a3a3a-a1fa-4174-a94a-a8382bc6f412-istio-podinfo\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-27vg6\" (UID: \"793a3a3a-a1fa-4174-a94a-a8382bc6f412\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-27vg6" Nov 03 01:06:15.521232 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:15.489718 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/793a3a3a-a1fa-4174-a94a-a8382bc6f412-istio-data\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-27vg6\" (UID: \"793a3a3a-a1fa-4174-a94a-a8382bc6f412\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-27vg6" Nov 03 01:06:15.521232 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:15.489782 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/793a3a3a-a1fa-4174-a94a-a8382bc6f412-credential-socket\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-27vg6\" (UID: \"793a3a3a-a1fa-4174-a94a-a8382bc6f412\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-27vg6" Nov 03 01:06:15.521232 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:15.489860 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/793a3a3a-a1fa-4174-a94a-a8382bc6f412-istiod-ca-cert\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-27vg6\" (UID: \"793a3a3a-a1fa-4174-a94a-a8382bc6f412\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-27vg6" Nov 03 01:06:15.521232 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:15.489933 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/793a3a3a-a1fa-4174-a94a-a8382bc6f412-istio-envoy\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-27vg6\" (UID: \"793a3a3a-a1fa-4174-a94a-a8382bc6f412\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-27vg6" Nov 03 01:06:15.521232 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:15.490001 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae4f838b-7b65-477f-8e03-56a00c7635b1-istio-data\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-pvzvs\" (UID: \"ae4f838b-7b65-477f-8e03-56a00c7635b1\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-pvzvs" Nov 03 01:06:15.521232 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:15.490030 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae4f838b-7b65-477f-8e03-56a00c7635b1-istiod-ca-cert\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-pvzvs\" (UID: \"ae4f838b-7b65-477f-8e03-56a00c7635b1\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-pvzvs" Nov 03 01:06:15.521232 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:15.490163 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/793a3a3a-a1fa-4174-a94a-a8382bc6f412-istio-token\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-27vg6\" (UID: \"793a3a3a-a1fa-4174-a94a-a8382bc6f412\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-27vg6" Nov 03 01:06:15.521232 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:15.490242 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae4f838b-7b65-477f-8e03-56a00c7635b1-istio-envoy\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-pvzvs\" (UID: \"ae4f838b-7b65-477f-8e03-56a00c7635b1\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-pvzvs" Nov 03 01:06:15.521232 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:15.490294 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae4f838b-7b65-477f-8e03-56a00c7635b1-credential-socket\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-pvzvs\" (UID: \"ae4f838b-7b65-477f-8e03-56a00c7635b1\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-pvzvs" Nov 03 01:06:15.521232 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:15.490346 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae4f838b-7b65-477f-8e03-56a00c7635b1-istio-podinfo\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-pvzvs\" (UID: \"ae4f838b-7b65-477f-8e03-56a00c7635b1\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-pvzvs" Nov 03 01:06:15.521232 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:15.490417 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/793a3a3a-a1fa-4174-a94a-a8382bc6f412-workload-certs\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-27vg6\" (UID: \"793a3a3a-a1fa-4174-a94a-a8382bc6f412\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-27vg6" Nov 03 01:06:15.521232 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:15.490460 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ll7v2\" (UniqueName: \"kubernetes.io/projected/793a3a3a-a1fa-4174-a94a-a8382bc6f412-kube-api-access-ll7v2\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-27vg6\" (UID: \"793a3a3a-a1fa-4174-a94a-a8382bc6f412\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-27vg6" Nov 03 01:06:15.521232 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:15.490488 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fc6h7\" (UniqueName: \"kubernetes.io/projected/ae4f838b-7b65-477f-8e03-56a00c7635b1-kube-api-access-fc6h7\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-pvzvs\" (UID: \"ae4f838b-7b65-477f-8e03-56a00c7635b1\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-pvzvs" Nov 03 01:06:15.606667 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:15.592334 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/793a3a3a-a1fa-4174-a94a-a8382bc6f412-workload-certs\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-27vg6\" (UID: \"793a3a3a-a1fa-4174-a94a-a8382bc6f412\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-27vg6" Nov 03 01:06:15.606667 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:15.592409 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ll7v2\" (UniqueName: \"kubernetes.io/projected/793a3a3a-a1fa-4174-a94a-a8382bc6f412-kube-api-access-ll7v2\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-27vg6\" (UID: \"793a3a3a-a1fa-4174-a94a-a8382bc6f412\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-27vg6" Nov 03 01:06:15.606667 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:15.592447 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fc6h7\" (UniqueName: \"kubernetes.io/projected/ae4f838b-7b65-477f-8e03-56a00c7635b1-kube-api-access-fc6h7\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-pvzvs\" (UID: \"ae4f838b-7b65-477f-8e03-56a00c7635b1\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-pvzvs" Nov 03 01:06:15.606667 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:15.592553 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae4f838b-7b65-477f-8e03-56a00c7635b1-workload-certs\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-pvzvs\" (UID: \"ae4f838b-7b65-477f-8e03-56a00c7635b1\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-pvzvs" Nov 03 01:06:15.606667 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:15.592593 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae4f838b-7b65-477f-8e03-56a00c7635b1-workload-socket\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-pvzvs\" (UID: \"ae4f838b-7b65-477f-8e03-56a00c7635b1\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-pvzvs" Nov 03 01:06:15.606667 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:15.592637 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/793a3a3a-a1fa-4174-a94a-a8382bc6f412-workload-socket\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-27vg6\" (UID: \"793a3a3a-a1fa-4174-a94a-a8382bc6f412\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-27vg6" Nov 03 01:06:15.606667 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:15.592674 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae4f838b-7b65-477f-8e03-56a00c7635b1-istio-token\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-pvzvs\" (UID: \"ae4f838b-7b65-477f-8e03-56a00c7635b1\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-pvzvs" Nov 03 01:06:15.606667 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:15.592799 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/793a3a3a-a1fa-4174-a94a-a8382bc6f412-istio-podinfo\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-27vg6\" (UID: \"793a3a3a-a1fa-4174-a94a-a8382bc6f412\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-27vg6" Nov 03 01:06:15.606667 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:15.592846 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/793a3a3a-a1fa-4174-a94a-a8382bc6f412-istio-data\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-27vg6\" (UID: \"793a3a3a-a1fa-4174-a94a-a8382bc6f412\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-27vg6" Nov 03 01:06:15.606667 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:15.592887 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/793a3a3a-a1fa-4174-a94a-a8382bc6f412-credential-socket\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-27vg6\" (UID: \"793a3a3a-a1fa-4174-a94a-a8382bc6f412\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-27vg6" Nov 03 01:06:15.606667 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:15.592967 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/793a3a3a-a1fa-4174-a94a-a8382bc6f412-istiod-ca-cert\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-27vg6\" (UID: \"793a3a3a-a1fa-4174-a94a-a8382bc6f412\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-27vg6" Nov 03 01:06:15.606667 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:15.593046 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/793a3a3a-a1fa-4174-a94a-a8382bc6f412-istio-envoy\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-27vg6\" (UID: \"793a3a3a-a1fa-4174-a94a-a8382bc6f412\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-27vg6" Nov 03 01:06:15.606667 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:15.593112 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae4f838b-7b65-477f-8e03-56a00c7635b1-istio-data\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-pvzvs\" (UID: \"ae4f838b-7b65-477f-8e03-56a00c7635b1\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-pvzvs" Nov 03 01:06:15.606667 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:15.593141 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae4f838b-7b65-477f-8e03-56a00c7635b1-istiod-ca-cert\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-pvzvs\" (UID: \"ae4f838b-7b65-477f-8e03-56a00c7635b1\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-pvzvs" Nov 03 01:06:15.606667 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:15.593304 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/793a3a3a-a1fa-4174-a94a-a8382bc6f412-istio-token\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-27vg6\" (UID: \"793a3a3a-a1fa-4174-a94a-a8382bc6f412\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-27vg6" Nov 03 01:06:15.606667 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:15.593391 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae4f838b-7b65-477f-8e03-56a00c7635b1-istio-envoy\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-pvzvs\" (UID: \"ae4f838b-7b65-477f-8e03-56a00c7635b1\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-pvzvs" Nov 03 01:06:15.606667 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:15.593427 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae4f838b-7b65-477f-8e03-56a00c7635b1-credential-socket\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-pvzvs\" (UID: \"ae4f838b-7b65-477f-8e03-56a00c7635b1\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-pvzvs" Nov 03 01:06:15.606667 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:15.593479 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae4f838b-7b65-477f-8e03-56a00c7635b1-istio-podinfo\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-pvzvs\" (UID: \"ae4f838b-7b65-477f-8e03-56a00c7635b1\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-pvzvs" Nov 03 01:06:15.606667 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:15.604784 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/793a3a3a-a1fa-4174-a94a-a8382bc6f412-workload-certs\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-27vg6\" (UID: \"793a3a3a-a1fa-4174-a94a-a8382bc6f412\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-27vg6" Nov 03 01:06:15.606667 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:15.605608 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae4f838b-7b65-477f-8e03-56a00c7635b1-workload-certs\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-pvzvs\" (UID: \"ae4f838b-7b65-477f-8e03-56a00c7635b1\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-pvzvs" Nov 03 01:06:15.606667 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:15.605832 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae4f838b-7b65-477f-8e03-56a00c7635b1-workload-socket\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-pvzvs\" (UID: \"ae4f838b-7b65-477f-8e03-56a00c7635b1\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-pvzvs" Nov 03 01:06:15.606667 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:15.606038 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/793a3a3a-a1fa-4174-a94a-a8382bc6f412-workload-socket\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-27vg6\" (UID: \"793a3a3a-a1fa-4174-a94a-a8382bc6f412\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-27vg6" Nov 03 01:06:15.646095 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:15.620094 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae4f838b-7b65-477f-8e03-56a00c7635b1-istio-podinfo\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-pvzvs\" (UID: \"ae4f838b-7b65-477f-8e03-56a00c7635b1\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-pvzvs" Nov 03 01:06:15.646095 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:15.624706 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae4f838b-7b65-477f-8e03-56a00c7635b1-istio-data\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-pvzvs\" (UID: \"ae4f838b-7b65-477f-8e03-56a00c7635b1\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-pvzvs" Nov 03 01:06:15.646095 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:15.624990 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/793a3a3a-a1fa-4174-a94a-a8382bc6f412-istio-data\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-27vg6\" (UID: \"793a3a3a-a1fa-4174-a94a-a8382bc6f412\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-27vg6" Nov 03 01:06:15.646095 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:15.625248 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/793a3a3a-a1fa-4174-a94a-a8382bc6f412-credential-socket\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-27vg6\" (UID: \"793a3a3a-a1fa-4174-a94a-a8382bc6f412\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-27vg6" Nov 03 01:06:15.646095 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:15.625475 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae4f838b-7b65-477f-8e03-56a00c7635b1-istiod-ca-cert\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-pvzvs\" (UID: \"ae4f838b-7b65-477f-8e03-56a00c7635b1\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-pvzvs" Nov 03 01:06:15.646095 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:15.633569 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/793a3a3a-a1fa-4174-a94a-a8382bc6f412-istiod-ca-cert\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-27vg6\" (UID: \"793a3a3a-a1fa-4174-a94a-a8382bc6f412\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-27vg6" Nov 03 01:06:15.669739 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:15.667564 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae4f838b-7b65-477f-8e03-56a00c7635b1-credential-socket\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-pvzvs\" (UID: \"ae4f838b-7b65-477f-8e03-56a00c7635b1\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-pvzvs" Nov 03 01:06:15.669739 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:15.667987 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae4f838b-7b65-477f-8e03-56a00c7635b1-istio-token\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-pvzvs\" (UID: \"ae4f838b-7b65-477f-8e03-56a00c7635b1\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-pvzvs" Nov 03 01:06:15.680841 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:15.679920 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ll7v2\" (UniqueName: \"kubernetes.io/projected/793a3a3a-a1fa-4174-a94a-a8382bc6f412-kube-api-access-ll7v2\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-27vg6\" (UID: \"793a3a3a-a1fa-4174-a94a-a8382bc6f412\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-27vg6" Nov 03 01:06:15.681495 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:15.681459 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/793a3a3a-a1fa-4174-a94a-a8382bc6f412-istio-envoy\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-27vg6\" (UID: \"793a3a3a-a1fa-4174-a94a-a8382bc6f412\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-27vg6" Nov 03 01:06:15.686831 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:15.685235 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae4f838b-7b65-477f-8e03-56a00c7635b1-istio-envoy\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-pvzvs\" (UID: \"ae4f838b-7b65-477f-8e03-56a00c7635b1\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-pvzvs" Nov 03 01:06:15.686831 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:15.685487 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/793a3a3a-a1fa-4174-a94a-a8382bc6f412-istio-token\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-27vg6\" (UID: \"793a3a3a-a1fa-4174-a94a-a8382bc6f412\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-27vg6" Nov 03 01:06:15.686831 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:15.685730 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fc6h7\" (UniqueName: \"kubernetes.io/projected/ae4f838b-7b65-477f-8e03-56a00c7635b1-kube-api-access-fc6h7\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-pvzvs\" (UID: \"ae4f838b-7b65-477f-8e03-56a00c7635b1\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-pvzvs" Nov 03 01:06:15.686831 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:15.685943 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/793a3a3a-a1fa-4174-a94a-a8382bc6f412-istio-podinfo\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-27vg6\" (UID: \"793a3a3a-a1fa-4174-a94a-a8382bc6f412\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-27vg6" Nov 03 01:06:15.719736 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:15.718344 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-pvzvs" Nov 03 01:06:16.352852 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:15.936060 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-27vg6" Nov 03 01:06:16.352852 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:16.013536 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-4248ae-openshift-default-94bc8c67-b5pgk" event={"ID":"2afb12c9-7c7a-4527-95c4-0b2febd22e90","Type":"ContainerStarted","Data":"e80a07df9efede7a14c7c16e47016e346dcb62910ad6c53860f41ccecc0f403e"} Nov 03 01:06:16.352852 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:16.041299 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-4248ae-openshift-default-94bc8c67-b5pgk" Nov 03 01:06:16.352852 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:16.041418 4421 patch_prober.go:29] interesting pod/e2e-autores-4248ae-openshift-default-94bc8c67-b5pgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused" start-of-body= Nov 03 01:06:16.352852 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:16.041461 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-4248ae-openshift-default-94bc8c67-b5pgk" podUID="2afb12c9-7c7a-4527-95c4-0b2febd22e90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused" Nov 03 01:06:16.352852 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:16.142889 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-pvzvs"] Nov 03 01:06:16.352852 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:16.158227 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:06:16.352852 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:16.158362 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:06:16.352852 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:16.158423 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:06:16.456396 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:16.456253 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-27vg6"] Nov 03 01:06:16.485640 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:06:16.478226 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod793a3a3a_a1fa_4174_a94a_a8382bc6f412.slice/crio-c6a254689969bbf542bb53903b6b77b023f3429d5db3884073ab3d8b9e80d89c WatchSource:0}: Error finding container c6a254689969bbf542bb53903b6b77b023f3429d5db3884073ab3d8b9e80d89c: Status 404 returned error can't find the container with id c6a254689969bbf542bb53903b6b77b023f3429d5db3884073ab3d8b9e80d89c Nov 03 01:06:16.492456 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:16.492239 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:06:16.492661 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:16.492629 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:06:16.492830 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:16.492798 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:06:16.796297 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:16.786444 4421 patch_prober.go:29] interesting pod/e2e-shared-a-w3-717217-openshift-default-578c69554c-gwl6k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" start-of-body= Nov 03 01:06:16.796297 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:16.786559 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-gwl6k" podUID="e6c36e3a-89e3-4f54-8266-c7dcfc0ca19b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" Nov 03 01:06:17.100217 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:17.071630 4421 patch_prober.go:29] interesting pod/e2e-autores-4248ae-openshift-default-94bc8c67-b5pgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused" start-of-body= Nov 03 01:06:17.100217 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:17.071729 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-4248ae-openshift-default-94bc8c67-b5pgk" podUID="2afb12c9-7c7a-4527-95c4-0b2febd22e90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused" Nov 03 01:06:17.100217 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:17.075798 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-pvzvs" event={"ID":"ae4f838b-7b65-477f-8e03-56a00c7635b1","Type":"ContainerStarted","Data":"e9f4aa83b36003440bc9b7c255040bae3b1312e0b5c93dee8db857bd60d70fc6"} Nov 03 01:06:17.100217 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:17.075868 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-pvzvs" event={"ID":"ae4f838b-7b65-477f-8e03-56a00c7635b1","Type":"ContainerStarted","Data":"fdc471991de0be67dc118d68cd9ba04b57a365a332a9a47750f8f86827bf765b"} Nov 03 01:06:17.100217 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:17.079407 4421 generic.go:356] "Generic (PLEG): container finished" podID="e6c36e3a-89e3-4f54-8266-c7dcfc0ca19b" containerID="35d6c0da9bf4ae0002ba6cde6d7fcb78cc529f65db86aca46c4b5fe17840a493" exitCode=0 Nov 03 01:06:17.100217 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:17.079570 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-gwl6k" event={"ID":"e6c36e3a-89e3-4f54-8266-c7dcfc0ca19b","Type":"ContainerDied","Data":"35d6c0da9bf4ae0002ba6cde6d7fcb78cc529f65db86aca46c4b5fe17840a493"} Nov 03 01:06:17.100217 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:17.079997 4421 scope.go:122] "RemoveContainer" containerID="35d6c0da9bf4ae0002ba6cde6d7fcb78cc529f65db86aca46c4b5fe17840a493" Nov 03 01:06:17.100217 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:17.085191 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-27vg6" event={"ID":"793a3a3a-a1fa-4174-a94a-a8382bc6f412","Type":"ContainerStarted","Data":"c6a254689969bbf542bb53903b6b77b023f3429d5db3884073ab3d8b9e80d89c"} Nov 03 01:06:17.128352 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:17.113178 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:06:17.128352 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:17.113407 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:06:17.128352 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:17.113476 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:06:17.740300 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:17.734913 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-pvzvs" Nov 03 01:06:17.835307 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:17.834541 4421 patch_prober.go:29] interesting pod/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-pvzvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" start-of-body= Nov 03 01:06:17.835307 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:17.834711 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-pvzvs" podUID="ae4f838b-7b65-477f-8e03-56a00c7635b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" Nov 03 01:06:17.891298 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:17.891204 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-pvzvs" podStartSLOduration=2.891177966 podStartE2EDuration="2.891177966s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:06:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:06:17.192055179 +0000 UTC m=+1519.363058277" watchObservedRunningTime="2026-11-03 01:06:17.891177966 +0000 UTC m=+1520.062181063" Nov 03 01:06:18.059757 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:18.057357 4421 patch_prober.go:29] interesting pod/e2e-autores-4248ae-openshift-default-94bc8c67-b5pgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused" start-of-body= Nov 03 01:06:18.059757 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:18.057417 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-4248ae-openshift-default-94bc8c67-b5pgk" podUID="2afb12c9-7c7a-4527-95c4-0b2febd22e90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused" Nov 03 01:06:18.320350 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:18.303892 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-27vg6" event={"ID":"793a3a3a-a1fa-4174-a94a-a8382bc6f412","Type":"ContainerStarted","Data":"bffed1cf3083920f03a0c871e1ec52375da877d82b7ffdc202146f6b8b9749a3"} Nov 03 01:06:18.334297 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:18.330006 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-27vg6" podStartSLOduration=3.329982984 podStartE2EDuration="3.329982984s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:06:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:06:18.304198661 +0000 UTC m=+1520.475201758" watchObservedRunningTime="2026-11-03 01:06:18.329982984 +0000 UTC m=+1520.500986079" Nov 03 01:06:18.489146 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:18.489077 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-c5x7c"] Nov 03 01:06:18.610389 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:18.602609 4421 patch_prober.go:29] interesting pod/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-sjhcb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.208:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 01:06:18.610389 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:18.602675 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-sjhcb" podUID="d1d81d36-60e4-4f12-81e6-7c5be483af12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.208:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 01:06:18.743098 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:18.743063 4421 patch_prober.go:29] interesting pod/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-pvzvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" start-of-body= Nov 03 01:06:18.756513 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:18.756475 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-pvzvs" podUID="ae4f838b-7b65-477f-8e03-56a00c7635b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" Nov 03 01:06:18.938715 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:18.938300 4421 patch_prober.go:29] interesting pod/e2e-shared-a-w3-717217-openshift-default-578c69554c-27vg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" start-of-body= Nov 03 01:06:18.938715 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:18.938380 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-27vg6" podUID="793a3a3a-a1fa-4174-a94a-a8382bc6f412" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" Nov 03 01:06:19.060444 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:19.059587 4421 patch_prober.go:29] interesting pod/e2e-autores-4248ae-openshift-default-94bc8c67-b5pgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused" start-of-body= Nov 03 01:06:19.060444 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:19.059656 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-4248ae-openshift-default-94bc8c67-b5pgk" podUID="2afb12c9-7c7a-4527-95c4-0b2febd22e90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused" Nov 03 01:06:19.202526 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:19.202436 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-27vg6" Nov 03 01:06:19.202526 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:19.202486 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-c5x7c"] Nov 03 01:06:19.202780 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:19.202594 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-c5x7c" Nov 03 01:06:19.384323 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:19.360669 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/652b215e-11c7-4343-86ec-bf27918c1c35-istiod-ca-cert\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-c5x7c\" (UID: \"652b215e-11c7-4343-86ec-bf27918c1c35\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-c5x7c" Nov 03 01:06:19.384323 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:19.360770 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/652b215e-11c7-4343-86ec-bf27918c1c35-istio-envoy\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-c5x7c\" (UID: \"652b215e-11c7-4343-86ec-bf27918c1c35\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-c5x7c" Nov 03 01:06:19.384323 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:19.360813 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/652b215e-11c7-4343-86ec-bf27918c1c35-workload-certs\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-c5x7c\" (UID: \"652b215e-11c7-4343-86ec-bf27918c1c35\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-c5x7c" Nov 03 01:06:19.384323 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:19.360852 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmwf5\" (UniqueName: \"kubernetes.io/projected/652b215e-11c7-4343-86ec-bf27918c1c35-kube-api-access-qmwf5\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-c5x7c\" (UID: \"652b215e-11c7-4343-86ec-bf27918c1c35\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-c5x7c" Nov 03 01:06:19.384323 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:19.360911 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/652b215e-11c7-4343-86ec-bf27918c1c35-credential-socket\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-c5x7c\" (UID: \"652b215e-11c7-4343-86ec-bf27918c1c35\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-c5x7c" Nov 03 01:06:19.384323 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:19.360956 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/652b215e-11c7-4343-86ec-bf27918c1c35-istio-token\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-c5x7c\" (UID: \"652b215e-11c7-4343-86ec-bf27918c1c35\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-c5x7c" Nov 03 01:06:19.384323 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:19.361069 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/652b215e-11c7-4343-86ec-bf27918c1c35-istio-podinfo\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-c5x7c\" (UID: \"652b215e-11c7-4343-86ec-bf27918c1c35\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-c5x7c" Nov 03 01:06:19.384323 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:19.361119 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/652b215e-11c7-4343-86ec-bf27918c1c35-workload-socket\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-c5x7c\" (UID: \"652b215e-11c7-4343-86ec-bf27918c1c35\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-c5x7c" Nov 03 01:06:19.384323 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:19.361175 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/652b215e-11c7-4343-86ec-bf27918c1c35-istio-data\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-c5x7c\" (UID: \"652b215e-11c7-4343-86ec-bf27918c1c35\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-c5x7c" Nov 03 01:06:19.400299 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:19.399870 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-gwl6k" event={"ID":"e6c36e3a-89e3-4f54-8266-c7dcfc0ca19b","Type":"ContainerStarted","Data":"25c5f74ca2414a88339beef2035ed59bc07f8a1ba6fdb15614b0c100590ec788"} Nov 03 01:06:19.468362 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:19.466565 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/652b215e-11c7-4343-86ec-bf27918c1c35-istio-envoy\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-c5x7c\" (UID: \"652b215e-11c7-4343-86ec-bf27918c1c35\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-c5x7c" Nov 03 01:06:19.468362 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:19.466659 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/652b215e-11c7-4343-86ec-bf27918c1c35-workload-certs\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-c5x7c\" (UID: \"652b215e-11c7-4343-86ec-bf27918c1c35\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-c5x7c" Nov 03 01:06:19.468362 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:19.466701 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmwf5\" (UniqueName: \"kubernetes.io/projected/652b215e-11c7-4343-86ec-bf27918c1c35-kube-api-access-qmwf5\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-c5x7c\" (UID: \"652b215e-11c7-4343-86ec-bf27918c1c35\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-c5x7c" Nov 03 01:06:19.468362 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:19.466767 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/652b215e-11c7-4343-86ec-bf27918c1c35-credential-socket\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-c5x7c\" (UID: \"652b215e-11c7-4343-86ec-bf27918c1c35\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-c5x7c" Nov 03 01:06:19.468362 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:19.466830 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/652b215e-11c7-4343-86ec-bf27918c1c35-istio-token\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-c5x7c\" (UID: \"652b215e-11c7-4343-86ec-bf27918c1c35\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-c5x7c" Nov 03 01:06:19.468362 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:19.466986 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/652b215e-11c7-4343-86ec-bf27918c1c35-istio-podinfo\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-c5x7c\" (UID: \"652b215e-11c7-4343-86ec-bf27918c1c35\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-c5x7c" Nov 03 01:06:19.468362 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:19.467061 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/652b215e-11c7-4343-86ec-bf27918c1c35-workload-socket\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-c5x7c\" (UID: \"652b215e-11c7-4343-86ec-bf27918c1c35\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-c5x7c" Nov 03 01:06:19.468362 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:19.467117 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/652b215e-11c7-4343-86ec-bf27918c1c35-istio-data\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-c5x7c\" (UID: \"652b215e-11c7-4343-86ec-bf27918c1c35\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-c5x7c" Nov 03 01:06:19.468362 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:19.467469 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/652b215e-11c7-4343-86ec-bf27918c1c35-istiod-ca-cert\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-c5x7c\" (UID: \"652b215e-11c7-4343-86ec-bf27918c1c35\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-c5x7c" Nov 03 01:06:19.470314 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:19.470263 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/652b215e-11c7-4343-86ec-bf27918c1c35-workload-socket\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-c5x7c\" (UID: \"652b215e-11c7-4343-86ec-bf27918c1c35\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-c5x7c" Nov 03 01:06:19.492316 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:19.471952 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/652b215e-11c7-4343-86ec-bf27918c1c35-workload-certs\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-c5x7c\" (UID: \"652b215e-11c7-4343-86ec-bf27918c1c35\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-c5x7c" Nov 03 01:06:19.492562 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:19.478251 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/652b215e-11c7-4343-86ec-bf27918c1c35-istio-data\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-c5x7c\" (UID: \"652b215e-11c7-4343-86ec-bf27918c1c35\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-c5x7c" Nov 03 01:06:19.492667 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:19.483992 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/652b215e-11c7-4343-86ec-bf27918c1c35-istiod-ca-cert\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-c5x7c\" (UID: \"652b215e-11c7-4343-86ec-bf27918c1c35\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-c5x7c" Nov 03 01:06:19.492767 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:19.492216 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/652b215e-11c7-4343-86ec-bf27918c1c35-credential-socket\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-c5x7c\" (UID: \"652b215e-11c7-4343-86ec-bf27918c1c35\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-c5x7c" Nov 03 01:06:19.704298 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:19.695329 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/652b215e-11c7-4343-86ec-bf27918c1c35-istio-token\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-c5x7c\" (UID: \"652b215e-11c7-4343-86ec-bf27918c1c35\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-c5x7c" Nov 03 01:06:19.704298 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:19.695564 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/652b215e-11c7-4343-86ec-bf27918c1c35-istio-envoy\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-c5x7c\" (UID: \"652b215e-11c7-4343-86ec-bf27918c1c35\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-c5x7c" Nov 03 01:06:19.754360 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:19.730077 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmwf5\" (UniqueName: \"kubernetes.io/projected/652b215e-11c7-4343-86ec-bf27918c1c35-kube-api-access-qmwf5\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-c5x7c\" (UID: \"652b215e-11c7-4343-86ec-bf27918c1c35\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-c5x7c" Nov 03 01:06:19.754360 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:19.730703 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/652b215e-11c7-4343-86ec-bf27918c1c35-istio-podinfo\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-c5x7c\" (UID: \"652b215e-11c7-4343-86ec-bf27918c1c35\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-c5x7c" Nov 03 01:06:19.784299 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:19.783956 4421 patch_prober.go:29] interesting pod/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-pvzvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" start-of-body= Nov 03 01:06:19.784299 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:19.784021 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-pvzvs" podUID="ae4f838b-7b65-477f-8e03-56a00c7635b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" Nov 03 01:06:19.947296 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:19.947238 4421 patch_prober.go:29] interesting pod/e2e-shared-a-w3-717217-openshift-default-578c69554c-27vg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" start-of-body= Nov 03 01:06:20.308077 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:19.947512 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-27vg6" podUID="793a3a3a-a1fa-4174-a94a-a8382bc6f412" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" Nov 03 01:06:20.308077 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:19.948685 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-c5x7c" Nov 03 01:06:20.308077 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:20.101254 4421 patch_prober.go:29] interesting pod/e2e-autores-4248ae-openshift-default-94bc8c67-b5pgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused" start-of-body= Nov 03 01:06:20.308077 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:20.101335 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-4248ae-openshift-default-94bc8c67-b5pgk" podUID="2afb12c9-7c7a-4527-95c4-0b2febd22e90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused" Nov 03 01:06:20.366507 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:20.366464 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-gwl6k" Nov 03 01:06:20.366771 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:20.366647 4421 patch_prober.go:29] interesting pod/e2e-shared-a-w3-717217-openshift-default-578c69554c-gwl6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" start-of-body= Nov 03 01:06:20.366771 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:20.366692 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-gwl6k" podUID="e6c36e3a-89e3-4f54-8266-c7dcfc0ca19b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" Nov 03 01:06:20.534638 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:20.534029 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/distinct-w3-83755a-kserve-86bdbbdd85-2cnnl" podUID="f4ae9624-d127-4a55-837d-7c9ff0267b37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 01:06:20.729761 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:20.728758 4421 patch_prober.go:29] interesting pod/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-pvzvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" start-of-body= Nov 03 01:06:20.729761 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:20.728831 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-pvzvs" podUID="ae4f838b-7b65-477f-8e03-56a00c7635b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" Nov 03 01:06:20.959451 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:20.955982 4421 patch_prober.go:29] interesting pod/e2e-shared-a-w3-717217-openshift-default-578c69554c-27vg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" start-of-body= Nov 03 01:06:20.959451 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:20.956060 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-27vg6" podUID="793a3a3a-a1fa-4174-a94a-a8382bc6f412" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" Nov 03 01:06:21.026298 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:21.024034 4421 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-83755a/premium-w3-83755a-kserve-7c9d66988b-lt4xd" podUID="c0c6b02e-2ecb-428e-a13d-be6d7cd1ba39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:06:21.175307 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:21.069543 4421 patch_prober.go:29] interesting pod/e2e-autores-4248ae-openshift-default-94bc8c67-b5pgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused" start-of-body= Nov 03 01:06:21.175307 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:21.069698 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-4248ae-openshift-default-94bc8c67-b5pgk" podUID="2afb12c9-7c7a-4527-95c4-0b2febd22e90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused" Nov 03 01:06:21.264876 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:21.264758 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/simulator-w3-83755a-kserve-77d6664559-psqtg" podUID="b0a4c234-af58-4da0-a1a4-e1f167953f6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:06:21.386951 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:21.367079 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/premium-w3-83755a-kserve-7c9d66988b-lt4xd" podUID="c0c6b02e-2ecb-428e-a13d-be6d7cd1ba39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Nov 03 01:06:21.499305 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:21.498061 4421 patch_prober.go:29] interesting pod/e2e-shared-a-w3-717217-openshift-default-578c69554c-gwl6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" start-of-body= Nov 03 01:06:21.499305 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:21.498147 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-gwl6k" podUID="e6c36e3a-89e3-4f54-8266-c7dcfc0ca19b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" Nov 03 01:06:21.556373 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:21.554240 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-650be0/simulator-w4-650be0-kserve-7bc588774f-gvz2k" podUID="a827fa88-f078-4312-8f12-2c799c11b461" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:06:21.646484 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:21.560331 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/distinct-w3-83755a-kserve-86bdbbdd85-2cnnl" podUID="f4ae9624-d127-4a55-837d-7c9ff0267b37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:06:21.646484 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:21.575113 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-83755a/distinct-w3-83755a-kserve-86bdbbdd85-2cnnl" Nov 03 01:06:21.646484 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:21.576318 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/simulator-w2-f7680d-kserve-85bdc86cd6-8gxrg" podUID="2bdcf521-cebd-4d6b-8554-1b60db8b0f3c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:06:21.776300 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:21.740152 4421 patch_prober.go:29] interesting pod/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-pvzvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" start-of-body= Nov 03 01:06:21.776300 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:21.740212 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-pvzvs" podUID="ae4f838b-7b65-477f-8e03-56a00c7635b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" Nov 03 01:06:21.806481 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:06:21.806376 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod652b215e_11c7_4343_86ec_bf27918c1c35.slice/crio-e7a1205c7936651f3f6a8ebf6a2717e903be8147833542e7c096d4e91d86e3ba WatchSource:0}: Error finding container e7a1205c7936651f3f6a8ebf6a2717e903be8147833542e7c096d4e91d86e3ba: Status 404 returned error can't find the container with id e7a1205c7936651f3f6a8ebf6a2717e903be8147833542e7c096d4e91d86e3ba Nov 03 01:06:22.024345 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:22.018373 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8gm8n" podUID="2d367e12-e46f-4a68-9825-9b1c934d4d12" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:06:22.148358 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:22.132734 4421 patch_prober.go:29] interesting pod/e2e-autores-4248ae-openshift-default-94bc8c67-b5pgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused" start-of-body= Nov 03 01:06:22.148358 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:22.132796 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-4248ae-openshift-default-94bc8c67-b5pgk" podUID="2afb12c9-7c7a-4527-95c4-0b2febd22e90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused" Nov 03 01:06:22.148358 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:22.134875 4421 patch_prober.go:29] interesting pod/e2e-shared-a-w3-717217-openshift-default-578c69554c-27vg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" start-of-body= Nov 03 01:06:22.148358 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:22.134925 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-27vg6" podUID="793a3a3a-a1fa-4174-a94a-a8382bc6f412" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" Nov 03 01:06:22.177898 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:22.150156 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:06:22.177898 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:22.150250 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:06:22.177898 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:22.150309 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:06:22.298086 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:22.298043 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/simulator-w3-83755a-kserve-77d6664559-psqtg" podUID="b0a4c234-af58-4da0-a1a4-e1f167953f6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:06:22.386295 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:22.385076 4421 patch_prober.go:29] interesting pod/e2e-shared-a-w3-717217-openshift-default-578c69554c-gwl6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" start-of-body= Nov 03 01:06:22.386295 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:22.385139 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-gwl6k" podUID="e6c36e3a-89e3-4f54-8266-c7dcfc0ca19b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" Nov 03 01:06:22.570713 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:22.570669 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/distinct-w3-83755a-kserve-86bdbbdd85-2cnnl" podUID="f4ae9624-d127-4a55-837d-7c9ff0267b37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded" Nov 03 01:06:22.626835 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:22.624367 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-650be0/simulator-w4-650be0-kserve-7bc588774f-gvz2k" podUID="a827fa88-f078-4312-8f12-2c799c11b461" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:06:22.628541 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:22.628510 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/simulator-w2-f7680d-kserve-85bdc86cd6-8gxrg" podUID="2bdcf521-cebd-4d6b-8554-1b60db8b0f3c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 03 01:06:22.681348 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:22.677516 4421 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-83755a/simulator-w3-83755a-kserve-77d6664559-psqtg" podUID="b0a4c234-af58-4da0-a1a4-e1f167953f6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:06:22.702589 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:22.698390 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-650be0/premium-w4-650be0-kserve-7bf8555dfb-z97n5" podUID="40601dc5-808f-456f-826f-6e0c35966ada" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:06:22.933522 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:22.918976 4421 patch_prober.go:29] interesting pod/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-pvzvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" start-of-body= Nov 03 01:06:22.933522 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:22.919048 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-pvzvs" podUID="ae4f838b-7b65-477f-8e03-56a00c7635b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" Nov 03 01:06:22.979298 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:22.972719 4421 patch_prober.go:29] interesting pod/e2e-shared-a-w3-717217-openshift-default-578c69554c-27vg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" start-of-body= Nov 03 01:06:22.979298 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:22.972788 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-27vg6" podUID="793a3a3a-a1fa-4174-a94a-a8382bc6f412" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" Nov 03 01:06:23.009629 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:23.009589 4421 patch_prober.go:29] interesting pod/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-v8jpj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.209:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 01:06:23.009928 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:23.009896 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-v8jpj" podUID="9ca37a4e-e37b-453c-a937-a35fedc2af8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.209:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 01:06:23.041292 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:23.041229 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8gm8n" podUID="2d367e12-e46f-4a68-9825-9b1c934d4d12" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:06:23.059987 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:23.058415 4421 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-650be0/simulator-w4-650be0-kserve-7bc588774f-gvz2k" podUID="a827fa88-f078-4312-8f12-2c799c11b461" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:06:23.158603 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:23.158554 4421 patch_prober.go:29] interesting pod/e2e-autores-4248ae-openshift-default-94bc8c67-b5pgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused" start-of-body= Nov 03 01:06:23.158941 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:23.158886 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-4248ae-openshift-default-94bc8c67-b5pgk" podUID="2afb12c9-7c7a-4527-95c4-0b2febd22e90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused" Nov 03 01:06:23.335354 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:23.310433 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/simulator-w3-83755a-kserve-77d6664559-psqtg" podUID="b0a4c234-af58-4da0-a1a4-e1f167953f6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:06:23.357820 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:23.350560 4421 patch_prober.go:29] interesting pod/e2e-shared-a-w3-717217-openshift-default-578c69554c-gwl6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" start-of-body= Nov 03 01:06:23.357820 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:23.350630 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-gwl6k" podUID="e6c36e3a-89e3-4f54-8266-c7dcfc0ca19b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" Nov 03 01:06:23.511493 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:06:23.500289 4421 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.511s" Nov 03 01:06:23.526505 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:23.519687 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-c5x7c"] Nov 03 01:06:23.526505 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:23.519804 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-83755a/simulator-w3-83755a-kserve-77d6664559-psqtg" Nov 03 01:06:23.526505 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:23.519846 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-650be0/simulator-w4-650be0-kserve-7bc588774f-gvz2k" Nov 03 01:06:23.526505 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:23.519880 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f7680d/simulator-w2-f7680d-kserve-85bdc86cd6-8gxrg" Nov 03 01:06:23.526505 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:23.519913 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-020af9-66449c479f-bmnx4" Nov 03 01:06:23.526505 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:23.519958 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8gm8n" Nov 03 01:06:23.526505 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:23.519996 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-020af9-7fcf787fd7-9fgx6" Nov 03 01:06:23.526505 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:23.520034 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f7680d/simulator-w2-f7680d-kserve-85bdc86cd6-8gxrg" Nov 03 01:06:23.634357 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:23.584416 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/distinct-w3-83755a-kserve-86bdbbdd85-2cnnl" podUID="f4ae9624-d127-4a55-837d-7c9ff0267b37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:06:23.637289 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:23.635724 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-650be0/simulator-w4-650be0-kserve-7bc588774f-gvz2k" podUID="a827fa88-f078-4312-8f12-2c799c11b461" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:06:23.699365 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:23.698878 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-650be0/premium-w4-650be0-kserve-7bf8555dfb-z97n5" podUID="40601dc5-808f-456f-826f-6e0c35966ada" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:06:23.699563 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:23.699458 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-650be0/premium-w4-650be0-kserve-7bf8555dfb-z97n5" Nov 03 01:06:23.745470 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:23.726610 4421 patch_prober.go:29] interesting pod/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-pvzvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" start-of-body= Nov 03 01:06:23.745470 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:23.726701 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-pvzvs" podUID="ae4f838b-7b65-477f-8e03-56a00c7635b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" Nov 03 01:06:23.927515 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:23.927423 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-650be0/premium-w4-650be0-kserve-7bf8555dfb-z97n5" Nov 03 01:06:23.953531 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:23.938766 4421 patch_prober.go:29] interesting pod/e2e-shared-a-w3-717217-openshift-default-578c69554c-27vg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" start-of-body= Nov 03 01:06:23.953531 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:23.953130 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-27vg6" podUID="793a3a3a-a1fa-4174-a94a-a8382bc6f412" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" Nov 03 01:06:24.178377 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:24.169250 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8gm8n" podUID="2d367e12-e46f-4a68-9825-9b1c934d4d12" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:06:24.471943 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:24.463404 4421 patch_prober.go:29] interesting pod/e2e-shared-a-w3-717217-openshift-default-578c69554c-gwl6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" start-of-body= Nov 03 01:06:24.471943 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:24.463470 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-gwl6k" podUID="e6c36e3a-89e3-4f54-8266-c7dcfc0ca19b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" Nov 03 01:06:24.492330 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:24.482840 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/simulator-w3-83755a-kserve-77d6664559-psqtg" podUID="b0a4c234-af58-4da0-a1a4-e1f167953f6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Nov 03 01:06:24.530411 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:24.523040 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-4248ae-openshift-default-94bc8c67-b5pgk" Nov 03 01:06:24.587917 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:24.587422 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/distinct-w3-83755a-kserve-86bdbbdd85-2cnnl" podUID="f4ae9624-d127-4a55-837d-7c9ff0267b37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded" Nov 03 01:06:24.587917 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:24.587692 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/simulator-w2-f7680d-kserve-85bdc86cd6-8gxrg" podUID="2bdcf521-cebd-4d6b-8554-1b60db8b0f3c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:06:24.693381 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:24.693334 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-650be0/simulator-w4-650be0-kserve-7bc588774f-gvz2k" podUID="a827fa88-f078-4312-8f12-2c799c11b461" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 01:06:24.703399 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:24.703344 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-c5x7c" event={"ID":"652b215e-11c7-4343-86ec-bf27918c1c35","Type":"ContainerStarted","Data":"e7a1205c7936651f3f6a8ebf6a2717e903be8147833542e7c096d4e91d86e3ba"} Nov 03 01:06:24.730418 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:24.722754 4421 patch_prober.go:29] interesting pod/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-pvzvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" start-of-body= Nov 03 01:06:24.730750 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:24.730712 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-pvzvs" podUID="ae4f838b-7b65-477f-8e03-56a00c7635b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" Nov 03 01:06:24.951301 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:24.942487 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-650be0/premium-w4-650be0-kserve-7bf8555dfb-z97n5" podUID="40601dc5-808f-456f-826f-6e0c35966ada" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:06:24.951301 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:24.944844 4421 patch_prober.go:29] interesting pod/e2e-shared-a-w3-717217-openshift-default-578c69554c-27vg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" start-of-body= Nov 03 01:06:24.951301 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:24.944897 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-27vg6" podUID="793a3a3a-a1fa-4174-a94a-a8382bc6f412" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" Nov 03 01:06:25.442297 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:25.426502 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8gm8n" podUID="2d367e12-e46f-4a68-9825-9b1c934d4d12" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": context deadline exceeded" Nov 03 01:06:25.442297 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:25.427489 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-4248ae-openshift-default-94bc8c67-b5pgk" Nov 03 01:06:25.537413 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:25.500387 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/premium-w3-83755a-kserve-7c9d66988b-lt4xd" podUID="c0c6b02e-2ecb-428e-a13d-be6d7cd1ba39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:06:25.537413 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:25.500685 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/simulator-w3-83755a-kserve-77d6664559-psqtg" podUID="b0a4c234-af58-4da0-a1a4-e1f167953f6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:06:25.537413 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:25.503264 4421 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8gm8n" podUID="2d367e12-e46f-4a68-9825-9b1c934d4d12" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:06:25.537413 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:25.503833 4421 patch_prober.go:29] interesting pod/e2e-shared-a-w3-717217-openshift-default-578c69554c-gwl6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" start-of-body= Nov 03 01:06:25.537413 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:25.503873 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-gwl6k" podUID="e6c36e3a-89e3-4f54-8266-c7dcfc0ca19b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" Nov 03 01:06:25.540120 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:25.538429 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-ktz44" podUID="ee2112c8-97e1-487d-aba9-66cf6ff002e7" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.134.0.20:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 01:06:25.596566 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:25.596441 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/simulator-w2-f7680d-kserve-85bdc86cd6-8gxrg" podUID="2bdcf521-cebd-4d6b-8554-1b60db8b0f3c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:06:25.597605 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:25.597505 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f7680d/simulator-w2-f7680d-kserve-85bdc86cd6-8gxrg" Nov 03 01:06:25.639470 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:25.600346 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/distinct-w3-83755a-kserve-86bdbbdd85-2cnnl" podUID="f4ae9624-d127-4a55-837d-7c9ff0267b37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:06:25.781342 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:25.756665 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-650be0/simulator-w4-650be0-kserve-7bc588774f-gvz2k" podUID="a827fa88-f078-4312-8f12-2c799c11b461" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:06:25.904031 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:25.890984 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-pvzvs" Nov 03 01:06:25.961374 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:25.957685 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-650be0/premium-w4-650be0-kserve-7bf8555dfb-z97n5" podUID="40601dc5-808f-456f-826f-6e0c35966ada" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:06:25.961374 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:25.958133 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-650be0/premium-w4-650be0-kserve-7bf8555dfb-z97n5" Nov 03 01:06:26.061456 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:26.057592 4421 patch_prober.go:29] interesting pod/e2e-shared-a-w3-717217-openshift-default-578c69554c-27vg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" start-of-body= Nov 03 01:06:26.061456 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:26.057659 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-27vg6" podUID="793a3a3a-a1fa-4174-a94a-a8382bc6f412" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" Nov 03 01:06:26.237315 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:26.229939 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-pvzvs" Nov 03 01:06:26.237315 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:26.230005 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-c5x7c" event={"ID":"652b215e-11c7-4343-86ec-bf27918c1c35","Type":"ContainerStarted","Data":"6a3743d0f74bb4c4289533a869fa36f0b7a71f52e3823905476b32e05870bf03"} Nov 03 01:06:26.237315 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:26.230076 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-pvzvs" Nov 03 01:06:26.237315 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:26.230131 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-4248ae-openshift-default-94bc8c67-b5pgk" Nov 03 01:06:26.237315 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:26.230174 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-650be0/simulator-w4-650be0-kserve-7bc588774f-gvz2k" Nov 03 01:06:26.313388 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:26.306336 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8gm8n" Nov 03 01:06:26.331883 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:26.331854 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-650be0/premium-w4-650be0-kserve-7bf8555dfb-z97n5" Nov 03 01:06:26.338301 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:26.333534 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-83755a/distinct-w3-83755a-kserve-86bdbbdd85-2cnnl" Nov 03 01:06:26.367298 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:26.340861 4421 patch_prober.go:29] interesting pod/e2e-shared-a-w3-717217-openshift-default-578c69554c-gwl6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" start-of-body= Nov 03 01:06:26.367588 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:26.367553 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-gwl6k" podUID="e6c36e3a-89e3-4f54-8266-c7dcfc0ca19b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" Nov 03 01:06:26.447318 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:26.436218 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-83755a/simulator-w3-83755a-kserve-77d6664559-psqtg" Nov 03 01:06:26.678926 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:26.664255 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/simulator-w2-f7680d-kserve-85bdc86cd6-8gxrg" podUID="2bdcf521-cebd-4d6b-8554-1b60db8b0f3c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 03 01:06:26.835405 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:26.835337 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-c5x7c" podStartSLOduration=8.835314994 podStartE2EDuration="8.835314994s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:06:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:06:26.168899217 +0000 UTC m=+1528.339902324" watchObservedRunningTime="2026-11-03 01:06:26.835314994 +0000 UTC m=+1529.006318090" Nov 03 01:06:26.990493 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:26.972022 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-27vg6" Nov 03 01:06:26.990493 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:26.984716 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-27vg6" Nov 03 01:06:27.062353 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:27.036303 4421 patch_prober.go:29] interesting pod/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-c5x7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.221:15021/healthz/ready\": dial tcp 10.134.0.221:15021: connect: connection refused" start-of-body= Nov 03 01:06:27.062353 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:27.036366 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-c5x7c" podUID="652b215e-11c7-4343-86ec-bf27918c1c35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.221:15021/healthz/ready\": dial tcp 10.134.0.221:15021: connect: connection refused" Nov 03 01:06:27.062353 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:27.037132 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-c5x7c" Nov 03 01:06:27.211296 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:27.160065 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f7680d/simulator-w2-f7680d-kserve-85bdc86cd6-8gxrg" Nov 03 01:06:27.211296 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:06:27.176860 4421 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ca37a4e_e37b_453c_a937_a35fedc2af8c.slice/crio-2f7c8781a132b4ccff1df6db612942c2b474bda21f3232903c72e69ab3c02219.scope\": RecentStats: unable to find data in memory cache]" Nov 03 01:06:27.287717 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:27.284549 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-27vg6" Nov 03 01:06:27.328691 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:27.328655 4421 generic.go:356] "Generic (PLEG): container finished" podID="9ca37a4e-e37b-453c-a937-a35fedc2af8c" containerID="2f7c8781a132b4ccff1df6db612942c2b474bda21f3232903c72e69ab3c02219" exitCode=0 Nov 03 01:06:27.375312 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:27.367219 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-v8jpj" event={"ID":"9ca37a4e-e37b-453c-a937-a35fedc2af8c","Type":"ContainerDied","Data":"2f7c8781a132b4ccff1df6db612942c2b474bda21f3232903c72e69ab3c02219"} Nov 03 01:06:27.375312 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:27.372547 4421 scope.go:122] "RemoveContainer" containerID="2f7c8781a132b4ccff1df6db612942c2b474bda21f3232903c72e69ab3c02219" Nov 03 01:06:27.375312 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:27.374732 4421 patch_prober.go:29] interesting pod/e2e-shared-a-w3-717217-openshift-default-578c69554c-gwl6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" start-of-body= Nov 03 01:06:27.375312 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:27.374775 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-gwl6k" podUID="e6c36e3a-89e3-4f54-8266-c7dcfc0ca19b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" Nov 03 01:06:27.394035 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:27.379830 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:06:27.394035 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:27.379945 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:06:27.394035 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:27.379990 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:06:27.995825 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:27.995327 4421 patch_prober.go:29] interesting pod/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-c5x7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.221:15021/healthz/ready\": dial tcp 10.134.0.221:15021: connect: connection refused" start-of-body= Nov 03 01:06:27.995825 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:27.995385 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-c5x7c" podUID="652b215e-11c7-4343-86ec-bf27918c1c35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.221:15021/healthz/ready\": dial tcp 10.134.0.221:15021: connect: connection refused" Nov 03 01:06:28.365032 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:28.363249 4421 generic.go:356] "Generic (PLEG): container finished" podID="d1d81d36-60e4-4f12-81e6-7c5be483af12" containerID="88843295f0b313641db6de1e5c6e5f67887fa1739aa9a8995bcf4d96a892cfeb" exitCode=0 Nov 03 01:06:28.365032 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:28.363495 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-sjhcb" event={"ID":"d1d81d36-60e4-4f12-81e6-7c5be483af12","Type":"ContainerDied","Data":"88843295f0b313641db6de1e5c6e5f67887fa1739aa9a8995bcf4d96a892cfeb"} Nov 03 01:06:28.365032 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:28.364079 4421 scope.go:122] "RemoveContainer" containerID="88843295f0b313641db6de1e5c6e5f67887fa1739aa9a8995bcf4d96a892cfeb" Nov 03 01:06:28.367381 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:28.367340 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:06:28.367695 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:28.367669 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:06:28.367866 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:28.367846 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:06:28.377353 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:28.377317 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-v8jpj" event={"ID":"9ca37a4e-e37b-453c-a937-a35fedc2af8c","Type":"ContainerStarted","Data":"be6f5ed2a61a6b545363fe16cb2e24b3dc7a069500d992a17cfcddcd50d0da54"} Nov 03 01:06:28.380169 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:28.380140 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-gwl6k" Nov 03 01:06:28.381596 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:28.381569 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-gwl6k" Nov 03 01:06:28.382352 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:28.382335 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-gwl6k" Nov 03 01:06:28.982602 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:28.982494 4421 patch_prober.go:29] interesting pod/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-c5x7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.221:15021/healthz/ready\": dial tcp 10.134.0.221:15021: connect: connection refused" start-of-body= Nov 03 01:06:28.982602 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:28.982569 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-c5x7c" podUID="652b215e-11c7-4343-86ec-bf27918c1c35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.221:15021/healthz/ready\": dial tcp 10.134.0.221:15021: connect: connection refused" Nov 03 01:06:28.984762 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:28.984741 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-v8jpj" Nov 03 01:06:28.985394 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:28.985370 4421 patch_prober.go:29] interesting pod/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-v8jpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" start-of-body= Nov 03 01:06:28.985459 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:28.985410 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-v8jpj" podUID="9ca37a4e-e37b-453c-a937-a35fedc2af8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" Nov 03 01:06:29.392364 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:29.392322 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-sjhcb" event={"ID":"d1d81d36-60e4-4f12-81e6-7c5be483af12","Type":"ContainerStarted","Data":"10b36843f4f9c99389d43f4eeb582894ba750cd8d4209716a59be05217067d09"} Nov 03 01:06:29.522296 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:29.514891 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-sjhcb" Nov 03 01:06:29.522296 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:29.515058 4421 patch_prober.go:29] interesting pod/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-sjhcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" start-of-body= Nov 03 01:06:29.522296 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:29.515109 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-sjhcb" podUID="d1d81d36-60e4-4f12-81e6-7c5be483af12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" Nov 03 01:06:29.991299 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:29.990738 4421 patch_prober.go:29] interesting pod/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-c5x7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.221:15021/healthz/ready\": dial tcp 10.134.0.221:15021: connect: connection refused" start-of-body= Nov 03 01:06:29.991299 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:29.990802 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-c5x7c" podUID="652b215e-11c7-4343-86ec-bf27918c1c35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.221:15021/healthz/ready\": dial tcp 10.134.0.221:15021: connect: connection refused" Nov 03 01:06:29.991590 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:29.991438 4421 patch_prober.go:29] interesting pod/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-v8jpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" start-of-body= Nov 03 01:06:29.991590 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:29.991478 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-v8jpj" podUID="9ca37a4e-e37b-453c-a937-a35fedc2af8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" Nov 03 01:06:30.273593 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:30.201152 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dqlwj"] Nov 03 01:06:30.383301 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:30.379300 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dqlwj"] Nov 03 01:06:30.383301 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:30.379442 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dqlwj" Nov 03 01:06:30.429372 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:30.403694 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52eafa3f-6965-4775-9394-013e1bf9c157-istio-podinfo\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dqlwj\" (UID: \"52eafa3f-6965-4775-9394-013e1bf9c157\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dqlwj" Nov 03 01:06:30.429372 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:30.403750 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52eafa3f-6965-4775-9394-013e1bf9c157-workload-socket\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dqlwj\" (UID: \"52eafa3f-6965-4775-9394-013e1bf9c157\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dqlwj" Nov 03 01:06:30.429372 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:30.403876 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52eafa3f-6965-4775-9394-013e1bf9c157-istio-data\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dqlwj\" (UID: \"52eafa3f-6965-4775-9394-013e1bf9c157\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dqlwj" Nov 03 01:06:30.429372 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:30.403925 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52eafa3f-6965-4775-9394-013e1bf9c157-credential-socket\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dqlwj\" (UID: \"52eafa3f-6965-4775-9394-013e1bf9c157\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dqlwj" Nov 03 01:06:30.429372 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:30.403985 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlxqf\" (UniqueName: \"kubernetes.io/projected/52eafa3f-6965-4775-9394-013e1bf9c157-kube-api-access-qlxqf\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dqlwj\" (UID: \"52eafa3f-6965-4775-9394-013e1bf9c157\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dqlwj" Nov 03 01:06:30.429372 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:30.404076 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52eafa3f-6965-4775-9394-013e1bf9c157-istio-envoy\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dqlwj\" (UID: \"52eafa3f-6965-4775-9394-013e1bf9c157\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dqlwj" Nov 03 01:06:30.429372 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:30.404116 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52eafa3f-6965-4775-9394-013e1bf9c157-istiod-ca-cert\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dqlwj\" (UID: \"52eafa3f-6965-4775-9394-013e1bf9c157\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dqlwj" Nov 03 01:06:30.429372 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:30.404160 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52eafa3f-6965-4775-9394-013e1bf9c157-workload-certs\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dqlwj\" (UID: \"52eafa3f-6965-4775-9394-013e1bf9c157\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dqlwj" Nov 03 01:06:30.429372 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:30.404218 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52eafa3f-6965-4775-9394-013e1bf9c157-istio-token\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dqlwj\" (UID: \"52eafa3f-6965-4775-9394-013e1bf9c157\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dqlwj" Nov 03 01:06:30.531363 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:30.508128 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52eafa3f-6965-4775-9394-013e1bf9c157-istio-envoy\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dqlwj\" (UID: \"52eafa3f-6965-4775-9394-013e1bf9c157\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dqlwj" Nov 03 01:06:30.531363 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:30.508225 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52eafa3f-6965-4775-9394-013e1bf9c157-istiod-ca-cert\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dqlwj\" (UID: \"52eafa3f-6965-4775-9394-013e1bf9c157\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dqlwj" Nov 03 01:06:30.599800 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:30.583839 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52eafa3f-6965-4775-9394-013e1bf9c157-workload-certs\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dqlwj\" (UID: \"52eafa3f-6965-4775-9394-013e1bf9c157\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dqlwj" Nov 03 01:06:30.599800 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:30.584001 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52eafa3f-6965-4775-9394-013e1bf9c157-istio-token\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dqlwj\" (UID: \"52eafa3f-6965-4775-9394-013e1bf9c157\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dqlwj" Nov 03 01:06:30.599800 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:30.586698 4421 patch_prober.go:29] interesting pod/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-sjhcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" start-of-body= Nov 03 01:06:30.599800 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:30.586767 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-sjhcb" podUID="d1d81d36-60e4-4f12-81e6-7c5be483af12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" Nov 03 01:06:30.599800 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:30.587551 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52eafa3f-6965-4775-9394-013e1bf9c157-istio-podinfo\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dqlwj\" (UID: \"52eafa3f-6965-4775-9394-013e1bf9c157\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dqlwj" Nov 03 01:06:30.599800 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:30.587629 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52eafa3f-6965-4775-9394-013e1bf9c157-workload-socket\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dqlwj\" (UID: \"52eafa3f-6965-4775-9394-013e1bf9c157\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dqlwj" Nov 03 01:06:30.599800 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:30.587739 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52eafa3f-6965-4775-9394-013e1bf9c157-istio-data\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dqlwj\" (UID: \"52eafa3f-6965-4775-9394-013e1bf9c157\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dqlwj" Nov 03 01:06:30.599800 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:30.587790 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52eafa3f-6965-4775-9394-013e1bf9c157-credential-socket\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dqlwj\" (UID: \"52eafa3f-6965-4775-9394-013e1bf9c157\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dqlwj" Nov 03 01:06:30.599800 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:30.587894 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlxqf\" (UniqueName: \"kubernetes.io/projected/52eafa3f-6965-4775-9394-013e1bf9c157-kube-api-access-qlxqf\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dqlwj\" (UID: \"52eafa3f-6965-4775-9394-013e1bf9c157\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dqlwj" Nov 03 01:06:30.599800 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:30.588646 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52eafa3f-6965-4775-9394-013e1bf9c157-istiod-ca-cert\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dqlwj\" (UID: \"52eafa3f-6965-4775-9394-013e1bf9c157\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dqlwj" Nov 03 01:06:30.599800 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:30.588816 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52eafa3f-6965-4775-9394-013e1bf9c157-workload-certs\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dqlwj\" (UID: \"52eafa3f-6965-4775-9394-013e1bf9c157\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dqlwj" Nov 03 01:06:30.599800 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:30.588936 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52eafa3f-6965-4775-9394-013e1bf9c157-istio-envoy\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dqlwj\" (UID: \"52eafa3f-6965-4775-9394-013e1bf9c157\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dqlwj" Nov 03 01:06:30.599800 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:30.589383 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52eafa3f-6965-4775-9394-013e1bf9c157-workload-socket\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dqlwj\" (UID: \"52eafa3f-6965-4775-9394-013e1bf9c157\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dqlwj" Nov 03 01:06:30.599800 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:30.589620 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52eafa3f-6965-4775-9394-013e1bf9c157-credential-socket\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dqlwj\" (UID: \"52eafa3f-6965-4775-9394-013e1bf9c157\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dqlwj" Nov 03 01:06:30.599800 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:30.589828 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52eafa3f-6965-4775-9394-013e1bf9c157-istio-data\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dqlwj\" (UID: \"52eafa3f-6965-4775-9394-013e1bf9c157\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dqlwj" Nov 03 01:06:30.599800 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:30.595620 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52eafa3f-6965-4775-9394-013e1bf9c157-istio-podinfo\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dqlwj\" (UID: \"52eafa3f-6965-4775-9394-013e1bf9c157\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dqlwj" Nov 03 01:06:30.599800 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:30.599059 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlxqf\" (UniqueName: \"kubernetes.io/projected/52eafa3f-6965-4775-9394-013e1bf9c157-kube-api-access-qlxqf\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dqlwj\" (UID: \"52eafa3f-6965-4775-9394-013e1bf9c157\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dqlwj" Nov 03 01:06:30.603333 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:30.603304 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52eafa3f-6965-4775-9394-013e1bf9c157-istio-token\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dqlwj\" (UID: \"52eafa3f-6965-4775-9394-013e1bf9c157\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dqlwj" Nov 03 01:06:30.944359 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:30.922460 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dqlwj" Nov 03 01:06:31.124301 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:31.108203 4421 patch_prober.go:29] interesting pod/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-v8jpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" start-of-body= Nov 03 01:06:31.124301 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:31.108261 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-v8jpj" podUID="9ca37a4e-e37b-453c-a937-a35fedc2af8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" Nov 03 01:06:31.162300 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:31.159145 4421 patch_prober.go:29] interesting pod/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-c5x7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.221:15021/healthz/ready\": dial tcp 10.134.0.221:15021: connect: connection refused" start-of-body= Nov 03 01:06:31.162300 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:31.159212 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-c5x7c" podUID="652b215e-11c7-4343-86ec-bf27918c1c35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.221:15021/healthz/ready\": dial tcp 10.134.0.221:15021: connect: connection refused" Nov 03 01:06:31.591090 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:31.590533 4421 patch_prober.go:29] interesting pod/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-sjhcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" start-of-body= Nov 03 01:06:31.591090 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:31.590601 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-sjhcb" podUID="d1d81d36-60e4-4f12-81e6-7c5be483af12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" Nov 03 01:06:31.701263 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:31.695212 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dqlwj"] Nov 03 01:06:31.702700 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:31.702655 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:06:31.702805 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:31.702742 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:06:31.702805 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:31.702776 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:06:31.982153 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:31.981735 4421 patch_prober.go:29] interesting pod/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-c5x7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.221:15021/healthz/ready\": dial tcp 10.134.0.221:15021: connect: connection refused" start-of-body= Nov 03 01:06:31.982153 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:31.981795 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-c5x7c" podUID="652b215e-11c7-4343-86ec-bf27918c1c35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.221:15021/healthz/ready\": dial tcp 10.134.0.221:15021: connect: connection refused" Nov 03 01:06:31.985695 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:31.985667 4421 patch_prober.go:29] interesting pod/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-v8jpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" start-of-body= Nov 03 01:06:31.985892 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:31.985872 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-v8jpj" podUID="9ca37a4e-e37b-453c-a937-a35fedc2af8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" Nov 03 01:06:32.503960 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:32.503917 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-sjhcb" Nov 03 01:06:32.523301 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:32.516698 4421 patch_prober.go:29] interesting pod/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-sjhcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" start-of-body= Nov 03 01:06:32.523301 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:32.516762 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-sjhcb" podUID="d1d81d36-60e4-4f12-81e6-7c5be483af12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" Nov 03 01:06:32.585990 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:32.585946 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dqlwj" event={"ID":"52eafa3f-6965-4775-9394-013e1bf9c157","Type":"ContainerStarted","Data":"a971182eccbea82a0f2ffb674b6846ca3497101e4b4aeee3015b0c0d36ff706b"} Nov 03 01:06:32.586302 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:32.586246 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dqlwj" event={"ID":"52eafa3f-6965-4775-9394-013e1bf9c157","Type":"ContainerStarted","Data":"7245bcd7ef71be82b5a5561cd7004ed9ff60b70a5289a66effe689a870883a82"} Nov 03 01:06:32.626407 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:32.625841 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dqlwj" podStartSLOduration=2.625818471 podStartE2EDuration="2.625818471s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:06:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:06:32.613799842 +0000 UTC m=+1534.784802953" watchObservedRunningTime="2026-11-03 01:06:32.625818471 +0000 UTC m=+1534.796821568" Nov 03 01:06:32.986366 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:32.985334 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dqlwj" Nov 03 01:06:32.989502 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:32.987250 4421 patch_prober.go:29] interesting pod/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-v8jpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" start-of-body= Nov 03 01:06:32.989502 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:32.987322 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-v8jpj" podUID="9ca37a4e-e37b-453c-a937-a35fedc2af8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" Nov 03 01:06:32.989502 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:32.989112 4421 patch_prober.go:29] interesting pod/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dqlwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" start-of-body= Nov 03 01:06:32.989502 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:32.989159 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dqlwj" podUID="52eafa3f-6965-4775-9394-013e1bf9c157" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" Nov 03 01:06:32.995922 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:32.995870 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-c5x7c" Nov 03 01:06:32.997167 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:32.997144 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-c5x7c" Nov 03 01:06:32.998126 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:32.998076 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-c5x7c" Nov 03 01:06:33.551334 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:33.549818 4421 patch_prober.go:29] interesting pod/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-sjhcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" start-of-body= Nov 03 01:06:33.551334 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:33.549878 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-sjhcb" podUID="d1d81d36-60e4-4f12-81e6-7c5be483af12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" Nov 03 01:06:33.960358 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:33.957680 4421 patch_prober.go:29] interesting pod/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dqlwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" start-of-body= Nov 03 01:06:33.960358 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:33.957748 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dqlwj" podUID="52eafa3f-6965-4775-9394-013e1bf9c157" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" Nov 03 01:06:33.984848 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:33.984803 4421 patch_prober.go:29] interesting pod/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-v8jpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" start-of-body= Nov 03 01:06:33.985016 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:33.984881 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-v8jpj" podUID="9ca37a4e-e37b-453c-a937-a35fedc2af8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" Nov 03 01:06:34.522954 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:34.522892 4421 patch_prober.go:29] interesting pod/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-sjhcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" start-of-body= Nov 03 01:06:34.523313 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:34.523256 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-sjhcb" podUID="d1d81d36-60e4-4f12-81e6-7c5be483af12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" Nov 03 01:06:34.955362 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:34.954176 4421 patch_prober.go:29] interesting pod/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dqlwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" start-of-body= Nov 03 01:06:34.955362 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:34.954251 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dqlwj" podUID="52eafa3f-6965-4775-9394-013e1bf9c157" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" Nov 03 01:06:34.990821 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:34.989829 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-v8jpj" Nov 03 01:06:34.990821 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:34.990499 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-v8jpj" Nov 03 01:06:34.991381 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:34.991338 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-v8jpj" Nov 03 01:06:35.522329 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:35.516811 4421 patch_prober.go:29] interesting pod/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-sjhcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" start-of-body= Nov 03 01:06:35.522329 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:35.516881 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-sjhcb" podUID="d1d81d36-60e4-4f12-81e6-7c5be483af12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" Nov 03 01:06:35.948757 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:35.948665 4421 patch_prober.go:29] interesting pod/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dqlwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" start-of-body= Nov 03 01:06:35.948757 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:35.948738 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dqlwj" podUID="52eafa3f-6965-4775-9394-013e1bf9c157" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" Nov 03 01:06:36.535669 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:36.535634 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-sjhcb" Nov 03 01:06:36.562309 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:36.561530 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-sjhcb" Nov 03 01:06:36.949130 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:36.949027 4421 patch_prober.go:29] interesting pod/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dqlwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" start-of-body= Nov 03 01:06:36.949130 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:36.949083 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dqlwj" podUID="52eafa3f-6965-4775-9394-013e1bf9c157" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" Nov 03 01:06:37.949413 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:37.949383 4421 patch_prober.go:29] interesting pod/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dqlwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" start-of-body= Nov 03 01:06:37.949829 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:37.949436 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dqlwj" podUID="52eafa3f-6965-4775-9394-013e1bf9c157" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" Nov 03 01:06:39.028701 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:39.028668 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dqlwj" Nov 03 01:06:39.029372 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:39.029157 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dqlwj" Nov 03 01:06:39.030781 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:39.030756 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dqlwj" Nov 03 01:06:44.274767 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:44.272093 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w0-50e623/auth-test-model-50e623-kserve-7db6664d89-zdq97"] Nov 03 01:06:44.296403 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:44.296368 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-50e623/auth-test-model-50e623-kserve-7db6664d89-zdq97"] Nov 03 01:06:44.296741 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:44.296720 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-50e623/auth-test-model-50e623-kserve-7db6664d89-zdq97" Nov 03 01:06:44.305097 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:44.305003 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-50e623\"/\"kube-root-ca.crt\"" Nov 03 01:06:44.312001 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:44.311857 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-50e623\"/\"default-dockercfg-vn4vg\"" Nov 03 01:06:44.312001 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:44.311955 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-50e623\"/\"auth-test-model-50e623-kserve-self-signed-certs\"" Nov 03 01:06:44.384187 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:44.384145 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e387182e-8e68-421f-b3ec-143283631b5e-model-cache\") pod \"auth-test-model-50e623-kserve-7db6664d89-zdq97\" (UID: \"e387182e-8e68-421f-b3ec-143283631b5e\") " pod="ai-tenant-e2e-shared-a-w0-50e623/auth-test-model-50e623-kserve-7db6664d89-zdq97" Nov 03 01:06:44.384438 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:44.384203 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vn7kf\" (UniqueName: \"kubernetes.io/projected/e387182e-8e68-421f-b3ec-143283631b5e-kube-api-access-vn7kf\") pod \"auth-test-model-50e623-kserve-7db6664d89-zdq97\" (UID: \"e387182e-8e68-421f-b3ec-143283631b5e\") " pod="ai-tenant-e2e-shared-a-w0-50e623/auth-test-model-50e623-kserve-7db6664d89-zdq97" Nov 03 01:06:44.384518 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:44.384440 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e387182e-8e68-421f-b3ec-143283631b5e-home\") pod \"auth-test-model-50e623-kserve-7db6664d89-zdq97\" (UID: \"e387182e-8e68-421f-b3ec-143283631b5e\") " pod="ai-tenant-e2e-shared-a-w0-50e623/auth-test-model-50e623-kserve-7db6664d89-zdq97" Nov 03 01:06:44.384710 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:44.384688 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e387182e-8e68-421f-b3ec-143283631b5e-tmp-dir\") pod \"auth-test-model-50e623-kserve-7db6664d89-zdq97\" (UID: \"e387182e-8e68-421f-b3ec-143283631b5e\") " pod="ai-tenant-e2e-shared-a-w0-50e623/auth-test-model-50e623-kserve-7db6664d89-zdq97" Nov 03 01:06:44.384762 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:44.384752 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e387182e-8e68-421f-b3ec-143283631b5e-tls-certs\") pod \"auth-test-model-50e623-kserve-7db6664d89-zdq97\" (UID: \"e387182e-8e68-421f-b3ec-143283631b5e\") " pod="ai-tenant-e2e-shared-a-w0-50e623/auth-test-model-50e623-kserve-7db6664d89-zdq97" Nov 03 01:06:44.384826 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:44.384809 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e387182e-8e68-421f-b3ec-143283631b5e-dshm\") pod \"auth-test-model-50e623-kserve-7db6664d89-zdq97\" (UID: \"e387182e-8e68-421f-b3ec-143283631b5e\") " pod="ai-tenant-e2e-shared-a-w0-50e623/auth-test-model-50e623-kserve-7db6664d89-zdq97" Nov 03 01:06:44.405137 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:44.404931 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-50e623\"/\"openshift-service-ca.crt\"" Nov 03 01:06:44.487220 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:44.487180 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e387182e-8e68-421f-b3ec-143283631b5e-home\") pod \"auth-test-model-50e623-kserve-7db6664d89-zdq97\" (UID: \"e387182e-8e68-421f-b3ec-143283631b5e\") " pod="ai-tenant-e2e-shared-a-w0-50e623/auth-test-model-50e623-kserve-7db6664d89-zdq97" Nov 03 01:06:44.487496 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:44.487430 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e387182e-8e68-421f-b3ec-143283631b5e-tmp-dir\") pod \"auth-test-model-50e623-kserve-7db6664d89-zdq97\" (UID: \"e387182e-8e68-421f-b3ec-143283631b5e\") " pod="ai-tenant-e2e-shared-a-w0-50e623/auth-test-model-50e623-kserve-7db6664d89-zdq97" Nov 03 01:06:44.487558 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:44.487494 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e387182e-8e68-421f-b3ec-143283631b5e-tls-certs\") pod \"auth-test-model-50e623-kserve-7db6664d89-zdq97\" (UID: \"e387182e-8e68-421f-b3ec-143283631b5e\") " pod="ai-tenant-e2e-shared-a-w0-50e623/auth-test-model-50e623-kserve-7db6664d89-zdq97" Nov 03 01:06:44.487558 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:44.487551 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e387182e-8e68-421f-b3ec-143283631b5e-dshm\") pod \"auth-test-model-50e623-kserve-7db6664d89-zdq97\" (UID: \"e387182e-8e68-421f-b3ec-143283631b5e\") " pod="ai-tenant-e2e-shared-a-w0-50e623/auth-test-model-50e623-kserve-7db6664d89-zdq97" Nov 03 01:06:44.487753 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:44.487734 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e387182e-8e68-421f-b3ec-143283631b5e-model-cache\") pod \"auth-test-model-50e623-kserve-7db6664d89-zdq97\" (UID: \"e387182e-8e68-421f-b3ec-143283631b5e\") " pod="ai-tenant-e2e-shared-a-w0-50e623/auth-test-model-50e623-kserve-7db6664d89-zdq97" Nov 03 01:06:44.487817 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:44.487784 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vn7kf\" (UniqueName: \"kubernetes.io/projected/e387182e-8e68-421f-b3ec-143283631b5e-kube-api-access-vn7kf\") pod \"auth-test-model-50e623-kserve-7db6664d89-zdq97\" (UID: \"e387182e-8e68-421f-b3ec-143283631b5e\") " pod="ai-tenant-e2e-shared-a-w0-50e623/auth-test-model-50e623-kserve-7db6664d89-zdq97" Nov 03 01:06:44.488155 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:44.487738 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e387182e-8e68-421f-b3ec-143283631b5e-home\") pod \"auth-test-model-50e623-kserve-7db6664d89-zdq97\" (UID: \"e387182e-8e68-421f-b3ec-143283631b5e\") " pod="ai-tenant-e2e-shared-a-w0-50e623/auth-test-model-50e623-kserve-7db6664d89-zdq97" Nov 03 01:06:44.488521 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:44.488498 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e387182e-8e68-421f-b3ec-143283631b5e-model-cache\") pod \"auth-test-model-50e623-kserve-7db6664d89-zdq97\" (UID: \"e387182e-8e68-421f-b3ec-143283631b5e\") " pod="ai-tenant-e2e-shared-a-w0-50e623/auth-test-model-50e623-kserve-7db6664d89-zdq97" Nov 03 01:06:44.488887 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:44.488867 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e387182e-8e68-421f-b3ec-143283631b5e-tmp-dir\") pod \"auth-test-model-50e623-kserve-7db6664d89-zdq97\" (UID: \"e387182e-8e68-421f-b3ec-143283631b5e\") " pod="ai-tenant-e2e-shared-a-w0-50e623/auth-test-model-50e623-kserve-7db6664d89-zdq97" Nov 03 01:06:44.491332 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:44.491179 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e387182e-8e68-421f-b3ec-143283631b5e-tls-certs\") pod \"auth-test-model-50e623-kserve-7db6664d89-zdq97\" (UID: \"e387182e-8e68-421f-b3ec-143283631b5e\") " pod="ai-tenant-e2e-shared-a-w0-50e623/auth-test-model-50e623-kserve-7db6664d89-zdq97" Nov 03 01:06:44.491547 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:44.491527 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e387182e-8e68-421f-b3ec-143283631b5e-dshm\") pod \"auth-test-model-50e623-kserve-7db6664d89-zdq97\" (UID: \"e387182e-8e68-421f-b3ec-143283631b5e\") " pod="ai-tenant-e2e-shared-a-w0-50e623/auth-test-model-50e623-kserve-7db6664d89-zdq97" Nov 03 01:06:44.501677 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:44.501649 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vn7kf\" (UniqueName: \"kubernetes.io/projected/e387182e-8e68-421f-b3ec-143283631b5e-kube-api-access-vn7kf\") pod \"auth-test-model-50e623-kserve-7db6664d89-zdq97\" (UID: \"e387182e-8e68-421f-b3ec-143283631b5e\") " pod="ai-tenant-e2e-shared-a-w0-50e623/auth-test-model-50e623-kserve-7db6664d89-zdq97" Nov 03 01:06:44.631350 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:44.630553 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-50e623/auth-test-model-50e623-kserve-7db6664d89-zdq97" Nov 03 01:06:44.739475 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:06:44.732933 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 01:06:45.317301 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:45.316417 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-50e623/auth-test-model-50e623-kserve-7db6664d89-zdq97"] Nov 03 01:06:45.367922 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:06:45.367889 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode387182e_8e68_421f_b3ec_143283631b5e.slice/crio-831cd8e349e5b7e201b1153d06936c2d3d8d98e91365dfd91d37005a319be9a8 WatchSource:0}: Error finding container 831cd8e349e5b7e201b1153d06936c2d3d8d98e91365dfd91d37005a319be9a8: Status 404 returned error can't find the container with id 831cd8e349e5b7e201b1153d06936c2d3d8d98e91365dfd91d37005a319be9a8 Nov 03 01:06:45.951596 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:45.932464 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-50e623/auth-test-model-50e623-kserve-7db6664d89-zdq97" event={"ID":"e387182e-8e68-421f-b3ec-143283631b5e","Type":"ContainerStarted","Data":"2663ce90c2fb9130b1280134a6f7b3f3b5f613972bf017716d1a7e7c48d6e23d"} Nov 03 01:06:45.951596 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:45.932517 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-50e623/auth-test-model-50e623-kserve-7db6664d89-zdq97" event={"ID":"e387182e-8e68-421f-b3ec-143283631b5e","Type":"ContainerStarted","Data":"831cd8e349e5b7e201b1153d06936c2d3d8d98e91365dfd91d37005a319be9a8"} Nov 03 01:06:45.968335 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:45.955048 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w0-50e623/auth-test-model-50e623-kserve-7db6664d89-zdq97" podStartSLOduration=1.955025945 podStartE2EDuration="1.955025945s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:06:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:06:45.950873012 +0000 UTC m=+1548.121876112" watchObservedRunningTime="2026-11-03 01:06:45.955025945 +0000 UTC m=+1548.126029043" Nov 03 01:06:46.633817 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:46.633772 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w0-50e623/auth-test-model-50e623-kserve-7db6664d89-zdq97" Nov 03 01:06:51.619102 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:51.619049 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-4248ae-85686d5965-jxtv4"] Nov 03 01:06:51.632727 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:51.632694 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-4248ae-69f65cb7f5-d6njn"] Nov 03 01:06:54.647299 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:54.642660 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w0-50e623/auth-test-model-50e623-kserve-7db6664d89-zdq97" Nov 03 01:06:54.680399 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:54.680359 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w0-50e623/auth-test-model-50e623-kserve-7db6664d89-zdq97" Nov 03 01:06:55.081349 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:55.081322 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w0-50e623/auth-test-model-50e623-kserve-7db6664d89-zdq97" Nov 03 01:06:56.629298 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:56.625356 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-4248ae-85686d5965-jxtv4" podUID="48cda01c-0810-4fea-9459-9f8f6e6f1e0c" containerName="payload-processing" containerID="cri-o://3ad4ebe3c9bdcf0cab9ad730f903c4f726deb3e4f22c68eeb007d8342b0b215c" gracePeriod=25 Nov 03 01:06:56.634648 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:56.634499 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-4248ae-69f65cb7f5-d6njn" podUID="7d0a4c35-92c7-4abc-b563-60b3b1d406a8" containerName="payload-pre-processing" containerID="cri-o://21faadb15cde6551b0c639a2c528338fab0a2f7e8eb45b22f3017262b499ae97" gracePeriod=25 Nov 03 01:06:56.985483 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:56.985453 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-4248ae-85686d5965-jxtv4" Nov 03 01:06:57.003171 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:57.003033 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48cda01c-0810-4fea-9459-9f8f6e6f1e0c-kube-api-access-hn545\" (UniqueName: \"kubernetes.io/projected/48cda01c-0810-4fea-9459-9f8f6e6f1e0c-kube-api-access-hn545\") pod \"48cda01c-0810-4fea-9459-9f8f6e6f1e0c\" (UID: \"48cda01c-0810-4fea-9459-9f8f6e6f1e0c\") " Nov 03 01:06:57.003171 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:57.003115 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48cda01c-0810-4fea-9459-9f8f6e6f1e0c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48cda01c-0810-4fea-9459-9f8f6e6f1e0c-plugins-config-volume\") pod \"48cda01c-0810-4fea-9459-9f8f6e6f1e0c\" (UID: \"48cda01c-0810-4fea-9459-9f8f6e6f1e0c\") " Nov 03 01:06:57.010613 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:57.008973 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48cda01c-0810-4fea-9459-9f8f6e6f1e0c-plugins-config-volume" pod "48cda01c-0810-4fea-9459-9f8f6e6f1e0c" (UID: "48cda01c-0810-4fea-9459-9f8f6e6f1e0c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:06:57.022338 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:57.022217 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48cda01c-0810-4fea-9459-9f8f6e6f1e0c-kube-api-access-hn545" pod "48cda01c-0810-4fea-9459-9f8f6e6f1e0c" (UID: "48cda01c-0810-4fea-9459-9f8f6e6f1e0c"). InnerVolumeSpecName "kube-api-access-hn545". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:06:57.067185 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:57.067159 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-4248ae-69f65cb7f5-d6njn" Nov 03 01:06:57.097854 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:57.097817 4421 generic.go:356] "Generic (PLEG): container finished" podID="7d0a4c35-92c7-4abc-b563-60b3b1d406a8" containerID="21faadb15cde6551b0c639a2c528338fab0a2f7e8eb45b22f3017262b499ae97" exitCode=0 Nov 03 01:06:57.098033 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:57.097873 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-4248ae-69f65cb7f5-d6njn" Nov 03 01:06:57.098033 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:57.097889 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-4248ae-69f65cb7f5-d6njn" event={"ID":"7d0a4c35-92c7-4abc-b563-60b3b1d406a8","Type":"ContainerDied","Data":"21faadb15cde6551b0c639a2c528338fab0a2f7e8eb45b22f3017262b499ae97"} Nov 03 01:06:57.098033 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:57.097943 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-4248ae-69f65cb7f5-d6njn" event={"ID":"7d0a4c35-92c7-4abc-b563-60b3b1d406a8","Type":"ContainerDied","Data":"c8d495df7ed82b0674c64b2b62303d1aa078d7debc1a7beddcc9a02c468b6615"} Nov 03 01:06:57.098033 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:57.097970 4421 scope.go:122] "RemoveContainer" containerID="21faadb15cde6551b0c639a2c528338fab0a2f7e8eb45b22f3017262b499ae97" Nov 03 01:06:57.099996 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:57.099962 4421 generic.go:356] "Generic (PLEG): container finished" podID="48cda01c-0810-4fea-9459-9f8f6e6f1e0c" containerID="3ad4ebe3c9bdcf0cab9ad730f903c4f726deb3e4f22c68eeb007d8342b0b215c" exitCode=0 Nov 03 01:06:57.100151 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:57.100039 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-4248ae-85686d5965-jxtv4" event={"ID":"48cda01c-0810-4fea-9459-9f8f6e6f1e0c","Type":"ContainerDied","Data":"3ad4ebe3c9bdcf0cab9ad730f903c4f726deb3e4f22c68eeb007d8342b0b215c"} Nov 03 01:06:57.100151 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:57.100075 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-4248ae-85686d5965-jxtv4" event={"ID":"48cda01c-0810-4fea-9459-9f8f6e6f1e0c","Type":"ContainerDied","Data":"1ff0d2ee6e8f76eaaca06bd5ff6c5bb007e2b0bbd880514031f18df8601184de"} Nov 03 01:06:57.100151 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:57.100129 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-4248ae-85686d5965-jxtv4" Nov 03 01:06:57.110057 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:57.110007 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d0a4c35-92c7-4abc-b563-60b3b1d406a8-kube-api-access-nztxj\" (UniqueName: \"kubernetes.io/projected/7d0a4c35-92c7-4abc-b563-60b3b1d406a8-kube-api-access-nztxj\") pod \"7d0a4c35-92c7-4abc-b563-60b3b1d406a8\" (UID: \"7d0a4c35-92c7-4abc-b563-60b3b1d406a8\") " Nov 03 01:06:57.111007 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:57.110981 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d0a4c35-92c7-4abc-b563-60b3b1d406a8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d0a4c35-92c7-4abc-b563-60b3b1d406a8-plugins-config-volume\") pod \"7d0a4c35-92c7-4abc-b563-60b3b1d406a8\" (UID: \"7d0a4c35-92c7-4abc-b563-60b3b1d406a8\") " Nov 03 01:06:57.115571 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:57.113115 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hn545\" (UniqueName: \"kubernetes.io/projected/48cda01c-0810-4fea-9459-9f8f6e6f1e0c-kube-api-access-hn545\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:06:57.115571 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:57.113142 4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48cda01c-0810-4fea-9459-9f8f6e6f1e0c-plugins-config-volume\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:06:57.115571 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:57.113861 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d0a4c35-92c7-4abc-b563-60b3b1d406a8-plugins-config-volume" pod "7d0a4c35-92c7-4abc-b563-60b3b1d406a8" (UID: "7d0a4c35-92c7-4abc-b563-60b3b1d406a8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:06:57.117140 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:57.116953 4421 scope.go:122] "RemoveContainer" containerID="21faadb15cde6551b0c639a2c528338fab0a2f7e8eb45b22f3017262b499ae97" Nov 03 01:06:57.189999 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:06:57.189824 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21faadb15cde6551b0c639a2c528338fab0a2f7e8eb45b22f3017262b499ae97\": container with ID starting with 21faadb15cde6551b0c639a2c528338fab0a2f7e8eb45b22f3017262b499ae97 not found: ID does not exist" containerID="21faadb15cde6551b0c639a2c528338fab0a2f7e8eb45b22f3017262b499ae97" Nov 03 01:06:57.190295 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:57.190243 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21faadb15cde6551b0c639a2c528338fab0a2f7e8eb45b22f3017262b499ae97"} err="failed to get container status \"21faadb15cde6551b0c639a2c528338fab0a2f7e8eb45b22f3017262b499ae97\": rpc error: code = NotFound desc = could not find container \"21faadb15cde6551b0c639a2c528338fab0a2f7e8eb45b22f3017262b499ae97\": container with ID starting with 21faadb15cde6551b0c639a2c528338fab0a2f7e8eb45b22f3017262b499ae97 not found: ID does not exist" Nov 03 01:06:57.190432 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:57.190419 4421 scope.go:122] "RemoveContainer" containerID="3ad4ebe3c9bdcf0cab9ad730f903c4f726deb3e4f22c68eeb007d8342b0b215c" Nov 03 01:06:57.190756 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:57.190706 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-4248ae-85686d5965-jxtv4"] Nov 03 01:06:57.216653 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:57.216473 4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d0a4c35-92c7-4abc-b563-60b3b1d406a8-plugins-config-volume\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:06:57.216898 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:57.216733 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d0a4c35-92c7-4abc-b563-60b3b1d406a8-kube-api-access-nztxj" pod "7d0a4c35-92c7-4abc-b563-60b3b1d406a8" (UID: "7d0a4c35-92c7-4abc-b563-60b3b1d406a8"). InnerVolumeSpecName "kube-api-access-nztxj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:06:57.219067 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:57.219043 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-4248ae-85686d5965-jxtv4"] Nov 03 01:06:57.228283 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:57.228198 4421 scope.go:122] "RemoveContainer" containerID="3ad4ebe3c9bdcf0cab9ad730f903c4f726deb3e4f22c68eeb007d8342b0b215c" Nov 03 01:06:57.228617 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:06:57.228595 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ad4ebe3c9bdcf0cab9ad730f903c4f726deb3e4f22c68eeb007d8342b0b215c\": container with ID starting with 3ad4ebe3c9bdcf0cab9ad730f903c4f726deb3e4f22c68eeb007d8342b0b215c not found: ID does not exist" containerID="3ad4ebe3c9bdcf0cab9ad730f903c4f726deb3e4f22c68eeb007d8342b0b215c" Nov 03 01:06:57.228693 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:57.228645 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ad4ebe3c9bdcf0cab9ad730f903c4f726deb3e4f22c68eeb007d8342b0b215c"} err="failed to get container status \"3ad4ebe3c9bdcf0cab9ad730f903c4f726deb3e4f22c68eeb007d8342b0b215c\": rpc error: code = NotFound desc = could not find container \"3ad4ebe3c9bdcf0cab9ad730f903c4f726deb3e4f22c68eeb007d8342b0b215c\": container with ID starting with 3ad4ebe3c9bdcf0cab9ad730f903c4f726deb3e4f22c68eeb007d8342b0b215c not found: ID does not exist" Nov 03 01:06:57.318623 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:57.318590 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nztxj\" (UniqueName: \"kubernetes.io/projected/7d0a4c35-92c7-4abc-b563-60b3b1d406a8-kube-api-access-nztxj\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:06:57.424762 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:57.424723 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-4248ae-69f65cb7f5-d6njn"] Nov 03 01:06:57.430566 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:57.430530 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-4248ae-69f65cb7f5-d6njn"] Nov 03 01:06:57.981720 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:57.981629 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48cda01c-0810-4fea-9459-9f8f6e6f1e0c" path="/var/lib/kubelet/pods/48cda01c-0810-4fea-9459-9f8f6e6f1e0c/volumes" Nov 03 01:06:57.982352 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:06:57.982328 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d0a4c35-92c7-4abc-b563-60b3b1d406a8" path="/var/lib/kubelet/pods/7d0a4c35-92c7-4abc-b563-60b3b1d406a8/volumes" Nov 03 01:07:00.371435 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:00.370060 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-d55st"] Nov 03 01:07:00.378481 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:00.371513 4421 cpu_manager.go:414] "containerMap: removing container" podUID="7d0a4c35-92c7-4abc-b563-60b3b1d406a8" containerName="payload-pre-processing" Nov 03 01:07:00.378481 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:00.371536 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d0a4c35-92c7-4abc-b563-60b3b1d406a8" containerName="payload-pre-processing" Nov 03 01:07:00.378481 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:00.371560 4421 cpu_manager.go:414] "containerMap: removing container" podUID="48cda01c-0810-4fea-9459-9f8f6e6f1e0c" containerName="payload-processing" Nov 03 01:07:00.378481 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:00.371569 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48cda01c-0810-4fea-9459-9f8f6e6f1e0c" containerName="payload-processing" Nov 03 01:07:00.378481 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:00.371769 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d0a4c35-92c7-4abc-b563-60b3b1d406a8" containerName="payload-pre-processing" Nov 03 01:07:00.378481 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:00.371784 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="48cda01c-0810-4fea-9459-9f8f6e6f1e0c" containerName="payload-processing" Nov 03 01:07:00.381864 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:00.381831 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-d55st" Nov 03 01:07:00.391466 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:00.391442 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-d55st"] Nov 03 01:07:00.428735 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:00.428691 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1548cadc-4d40-4310-89bd-0b2021af6faf-istio-data\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-d55st\" (UID: \"1548cadc-4d40-4310-89bd-0b2021af6faf\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-d55st" Nov 03 01:07:00.428735 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:00.428747 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1548cadc-4d40-4310-89bd-0b2021af6faf-workload-certs\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-d55st\" (UID: \"1548cadc-4d40-4310-89bd-0b2021af6faf\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-d55st" Nov 03 01:07:00.429005 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:00.428785 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1548cadc-4d40-4310-89bd-0b2021af6faf-istio-envoy\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-d55st\" (UID: \"1548cadc-4d40-4310-89bd-0b2021af6faf\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-d55st" Nov 03 01:07:00.429005 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:00.428819 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1548cadc-4d40-4310-89bd-0b2021af6faf-workload-socket\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-d55st\" (UID: \"1548cadc-4d40-4310-89bd-0b2021af6faf\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-d55st" Nov 03 01:07:00.429127 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:00.429099 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1548cadc-4d40-4310-89bd-0b2021af6faf-credential-socket\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-d55st\" (UID: \"1548cadc-4d40-4310-89bd-0b2021af6faf\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-d55st" Nov 03 01:07:00.429186 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:00.429134 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1548cadc-4d40-4310-89bd-0b2021af6faf-istiod-ca-cert\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-d55st\" (UID: \"1548cadc-4d40-4310-89bd-0b2021af6faf\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-d55st" Nov 03 01:07:00.429248 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:00.429210 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1548cadc-4d40-4310-89bd-0b2021af6faf-istio-token\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-d55st\" (UID: \"1548cadc-4d40-4310-89bd-0b2021af6faf\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-d55st" Nov 03 01:07:00.429332 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:00.429302 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1548cadc-4d40-4310-89bd-0b2021af6faf-istio-podinfo\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-d55st\" (UID: \"1548cadc-4d40-4310-89bd-0b2021af6faf\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-d55st" Nov 03 01:07:00.429411 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:00.429390 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbswp\" (UniqueName: \"kubernetes.io/projected/1548cadc-4d40-4310-89bd-0b2021af6faf-kube-api-access-zbswp\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-d55st\" (UID: \"1548cadc-4d40-4310-89bd-0b2021af6faf\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-d55st" Nov 03 01:07:00.530589 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:00.530553 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1548cadc-4d40-4310-89bd-0b2021af6faf-credential-socket\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-d55st\" (UID: \"1548cadc-4d40-4310-89bd-0b2021af6faf\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-d55st" Nov 03 01:07:00.530824 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:00.530601 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1548cadc-4d40-4310-89bd-0b2021af6faf-istiod-ca-cert\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-d55st\" (UID: \"1548cadc-4d40-4310-89bd-0b2021af6faf\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-d55st" Nov 03 01:07:00.530824 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:00.530652 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1548cadc-4d40-4310-89bd-0b2021af6faf-istio-token\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-d55st\" (UID: \"1548cadc-4d40-4310-89bd-0b2021af6faf\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-d55st" Nov 03 01:07:00.530824 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:00.530700 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1548cadc-4d40-4310-89bd-0b2021af6faf-istio-podinfo\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-d55st\" (UID: \"1548cadc-4d40-4310-89bd-0b2021af6faf\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-d55st" Nov 03 01:07:00.530824 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:00.530758 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbswp\" (UniqueName: \"kubernetes.io/projected/1548cadc-4d40-4310-89bd-0b2021af6faf-kube-api-access-zbswp\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-d55st\" (UID: \"1548cadc-4d40-4310-89bd-0b2021af6faf\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-d55st" Nov 03 01:07:00.530824 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:00.530806 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1548cadc-4d40-4310-89bd-0b2021af6faf-istio-data\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-d55st\" (UID: \"1548cadc-4d40-4310-89bd-0b2021af6faf\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-d55st" Nov 03 01:07:00.531326 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:00.530836 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1548cadc-4d40-4310-89bd-0b2021af6faf-workload-certs\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-d55st\" (UID: \"1548cadc-4d40-4310-89bd-0b2021af6faf\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-d55st" Nov 03 01:07:00.531326 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:00.530871 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1548cadc-4d40-4310-89bd-0b2021af6faf-istio-envoy\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-d55st\" (UID: \"1548cadc-4d40-4310-89bd-0b2021af6faf\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-d55st" Nov 03 01:07:00.531326 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:00.530903 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1548cadc-4d40-4310-89bd-0b2021af6faf-workload-socket\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-d55st\" (UID: \"1548cadc-4d40-4310-89bd-0b2021af6faf\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-d55st" Nov 03 01:07:00.531326 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:00.530965 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1548cadc-4d40-4310-89bd-0b2021af6faf-credential-socket\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-d55st\" (UID: \"1548cadc-4d40-4310-89bd-0b2021af6faf\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-d55st" Nov 03 01:07:00.531578 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:00.531449 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1548cadc-4d40-4310-89bd-0b2021af6faf-istiod-ca-cert\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-d55st\" (UID: \"1548cadc-4d40-4310-89bd-0b2021af6faf\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-d55st" Nov 03 01:07:00.532223 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:00.532109 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1548cadc-4d40-4310-89bd-0b2021af6faf-workload-certs\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-d55st\" (UID: \"1548cadc-4d40-4310-89bd-0b2021af6faf\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-d55st" Nov 03 01:07:00.532495 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:00.532469 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1548cadc-4d40-4310-89bd-0b2021af6faf-workload-socket\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-d55st\" (UID: \"1548cadc-4d40-4310-89bd-0b2021af6faf\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-d55st" Nov 03 01:07:00.532793 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:00.532774 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1548cadc-4d40-4310-89bd-0b2021af6faf-istio-data\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-d55st\" (UID: \"1548cadc-4d40-4310-89bd-0b2021af6faf\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-d55st" Nov 03 01:07:00.534551 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:00.534528 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1548cadc-4d40-4310-89bd-0b2021af6faf-istio-envoy\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-d55st\" (UID: \"1548cadc-4d40-4310-89bd-0b2021af6faf\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-d55st" Nov 03 01:07:00.534708 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:00.534685 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1548cadc-4d40-4310-89bd-0b2021af6faf-istio-podinfo\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-d55st\" (UID: \"1548cadc-4d40-4310-89bd-0b2021af6faf\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-d55st" Nov 03 01:07:00.540340 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:00.540319 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1548cadc-4d40-4310-89bd-0b2021af6faf-istio-token\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-d55st\" (UID: \"1548cadc-4d40-4310-89bd-0b2021af6faf\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-d55st" Nov 03 01:07:00.540438 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:00.540416 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbswp\" (UniqueName: \"kubernetes.io/projected/1548cadc-4d40-4310-89bd-0b2021af6faf-kube-api-access-zbswp\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-d55st\" (UID: \"1548cadc-4d40-4310-89bd-0b2021af6faf\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-d55st" Nov 03 01:07:00.703819 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:00.703729 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-d55st" Nov 03 01:07:01.161558 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:01.157587 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-d55st"] Nov 03 01:07:01.187629 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:01.181599 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:07:01.187629 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:01.182017 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:07:01.210778 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:01.188663 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:07:02.090651 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:02.089107 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-gh5gs"] Nov 03 01:07:02.099601 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:02.099570 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-gh5gs" Nov 03 01:07:02.105107 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:02.104868 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-gh5gs"] Nov 03 01:07:02.111110 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:02.111080 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-6c0ec0-openshift-default-dockercfg-dgkf8\"" Nov 03 01:07:02.194168 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:02.194124 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-d55st" event={"ID":"1548cadc-4d40-4310-89bd-0b2021af6faf","Type":"ContainerStarted","Data":"3d718dedeb1f8cbb8158b0c7caec274cceb7566dc5f44a6623c5e5e618eaa192"} Nov 03 01:07:02.194405 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:02.194202 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-d55st" event={"ID":"1548cadc-4d40-4310-89bd-0b2021af6faf","Type":"ContainerStarted","Data":"1875d298785835728da841941970bdded2454e057d20c3f9562acbe2706029bb"} Nov 03 01:07:02.248806 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:02.248741 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-d55st" podStartSLOduration=2.248724185 podStartE2EDuration="2.248724185s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:07:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:07:02.243854276 +0000 UTC m=+1564.414857371" watchObservedRunningTime="2026-11-03 01:07:02.248724185 +0000 UTC m=+1564.419727277" Nov 03 01:07:02.269936 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:02.269891 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c6e134c-a6fe-4794-9c80-b52ebb11abb2-credential-socket\") pod \"e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-gh5gs\" (UID: \"5c6e134c-a6fe-4794-9c80-b52ebb11abb2\") " pod="openshift-ingress/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-gh5gs" Nov 03 01:07:02.269936 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:02.269942 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c6e134c-a6fe-4794-9c80-b52ebb11abb2-istio-token\") pod \"e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-gh5gs\" (UID: \"5c6e134c-a6fe-4794-9c80-b52ebb11abb2\") " pod="openshift-ingress/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-gh5gs" Nov 03 01:07:02.270217 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:02.270043 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c6e134c-a6fe-4794-9c80-b52ebb11abb2-workload-certs\") pod \"e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-gh5gs\" (UID: \"5c6e134c-a6fe-4794-9c80-b52ebb11abb2\") " pod="openshift-ingress/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-gh5gs" Nov 03 01:07:02.270217 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:02.270079 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c6e134c-a6fe-4794-9c80-b52ebb11abb2-istiod-ca-cert\") pod \"e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-gh5gs\" (UID: \"5c6e134c-a6fe-4794-9c80-b52ebb11abb2\") " pod="openshift-ingress/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-gh5gs" Nov 03 01:07:02.270217 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:02.270128 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j48wr\" (UniqueName: \"kubernetes.io/projected/5c6e134c-a6fe-4794-9c80-b52ebb11abb2-kube-api-access-j48wr\") pod \"e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-gh5gs\" (UID: \"5c6e134c-a6fe-4794-9c80-b52ebb11abb2\") " pod="openshift-ingress/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-gh5gs" Nov 03 01:07:02.270451 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:02.270306 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c6e134c-a6fe-4794-9c80-b52ebb11abb2-workload-socket\") pod \"e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-gh5gs\" (UID: \"5c6e134c-a6fe-4794-9c80-b52ebb11abb2\") " pod="openshift-ingress/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-gh5gs" Nov 03 01:07:02.270520 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:02.270455 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c6e134c-a6fe-4794-9c80-b52ebb11abb2-istio-data\") pod \"e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-gh5gs\" (UID: \"5c6e134c-a6fe-4794-9c80-b52ebb11abb2\") " pod="openshift-ingress/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-gh5gs" Nov 03 01:07:02.270520 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:02.270498 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c6e134c-a6fe-4794-9c80-b52ebb11abb2-istio-podinfo\") pod \"e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-gh5gs\" (UID: \"5c6e134c-a6fe-4794-9c80-b52ebb11abb2\") " pod="openshift-ingress/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-gh5gs" Nov 03 01:07:02.270647 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:02.270554 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c6e134c-a6fe-4794-9c80-b52ebb11abb2-istio-envoy\") pod \"e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-gh5gs\" (UID: \"5c6e134c-a6fe-4794-9c80-b52ebb11abb2\") " pod="openshift-ingress/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-gh5gs" Nov 03 01:07:02.372225 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:02.372130 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c6e134c-a6fe-4794-9c80-b52ebb11abb2-credential-socket\") pod \"e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-gh5gs\" (UID: \"5c6e134c-a6fe-4794-9c80-b52ebb11abb2\") " pod="openshift-ingress/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-gh5gs" Nov 03 01:07:02.372422 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:02.372228 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c6e134c-a6fe-4794-9c80-b52ebb11abb2-istio-token\") pod \"e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-gh5gs\" (UID: \"5c6e134c-a6fe-4794-9c80-b52ebb11abb2\") " pod="openshift-ingress/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-gh5gs" Nov 03 01:07:02.372422 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:02.372335 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c6e134c-a6fe-4794-9c80-b52ebb11abb2-workload-certs\") pod \"e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-gh5gs\" (UID: \"5c6e134c-a6fe-4794-9c80-b52ebb11abb2\") " pod="openshift-ingress/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-gh5gs" Nov 03 01:07:02.372422 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:02.372387 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c6e134c-a6fe-4794-9c80-b52ebb11abb2-istiod-ca-cert\") pod \"e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-gh5gs\" (UID: \"5c6e134c-a6fe-4794-9c80-b52ebb11abb2\") " pod="openshift-ingress/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-gh5gs" Nov 03 01:07:02.372923 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:02.372466 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j48wr\" (UniqueName: \"kubernetes.io/projected/5c6e134c-a6fe-4794-9c80-b52ebb11abb2-kube-api-access-j48wr\") pod \"e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-gh5gs\" (UID: \"5c6e134c-a6fe-4794-9c80-b52ebb11abb2\") " pod="openshift-ingress/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-gh5gs" Nov 03 01:07:02.372923 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:02.372608 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c6e134c-a6fe-4794-9c80-b52ebb11abb2-workload-socket\") pod \"e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-gh5gs\" (UID: \"5c6e134c-a6fe-4794-9c80-b52ebb11abb2\") " pod="openshift-ingress/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-gh5gs" Nov 03 01:07:02.372923 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:02.372627 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c6e134c-a6fe-4794-9c80-b52ebb11abb2-credential-socket\") pod \"e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-gh5gs\" (UID: \"5c6e134c-a6fe-4794-9c80-b52ebb11abb2\") " pod="openshift-ingress/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-gh5gs" Nov 03 01:07:02.372923 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:02.372665 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c6e134c-a6fe-4794-9c80-b52ebb11abb2-istio-data\") pod \"e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-gh5gs\" (UID: \"5c6e134c-a6fe-4794-9c80-b52ebb11abb2\") " pod="openshift-ingress/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-gh5gs" Nov 03 01:07:02.372923 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:02.372748 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c6e134c-a6fe-4794-9c80-b52ebb11abb2-istio-podinfo\") pod \"e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-gh5gs\" (UID: \"5c6e134c-a6fe-4794-9c80-b52ebb11abb2\") " pod="openshift-ingress/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-gh5gs" Nov 03 01:07:02.372923 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:02.372762 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c6e134c-a6fe-4794-9c80-b52ebb11abb2-workload-certs\") pod \"e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-gh5gs\" (UID: \"5c6e134c-a6fe-4794-9c80-b52ebb11abb2\") " pod="openshift-ingress/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-gh5gs" Nov 03 01:07:02.372923 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:02.372812 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c6e134c-a6fe-4794-9c80-b52ebb11abb2-istio-envoy\") pod \"e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-gh5gs\" (UID: \"5c6e134c-a6fe-4794-9c80-b52ebb11abb2\") " pod="openshift-ingress/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-gh5gs" Nov 03 01:07:02.373671 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:02.373010 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c6e134c-a6fe-4794-9c80-b52ebb11abb2-workload-socket\") pod \"e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-gh5gs\" (UID: \"5c6e134c-a6fe-4794-9c80-b52ebb11abb2\") " pod="openshift-ingress/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-gh5gs" Nov 03 01:07:02.373671 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:02.373088 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c6e134c-a6fe-4794-9c80-b52ebb11abb2-istio-data\") pod \"e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-gh5gs\" (UID: \"5c6e134c-a6fe-4794-9c80-b52ebb11abb2\") " pod="openshift-ingress/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-gh5gs" Nov 03 01:07:02.373671 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:02.373507 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c6e134c-a6fe-4794-9c80-b52ebb11abb2-istiod-ca-cert\") pod \"e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-gh5gs\" (UID: \"5c6e134c-a6fe-4794-9c80-b52ebb11abb2\") " pod="openshift-ingress/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-gh5gs" Nov 03 01:07:02.375964 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:02.375935 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c6e134c-a6fe-4794-9c80-b52ebb11abb2-istio-envoy\") pod \"e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-gh5gs\" (UID: \"5c6e134c-a6fe-4794-9c80-b52ebb11abb2\") " pod="openshift-ingress/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-gh5gs" Nov 03 01:07:02.376098 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:02.376078 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c6e134c-a6fe-4794-9c80-b52ebb11abb2-istio-podinfo\") pod \"e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-gh5gs\" (UID: \"5c6e134c-a6fe-4794-9c80-b52ebb11abb2\") " pod="openshift-ingress/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-gh5gs" Nov 03 01:07:02.383152 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:02.383126 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c6e134c-a6fe-4794-9c80-b52ebb11abb2-istio-token\") pod \"e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-gh5gs\" (UID: \"5c6e134c-a6fe-4794-9c80-b52ebb11abb2\") " pod="openshift-ingress/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-gh5gs" Nov 03 01:07:02.388096 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:02.388071 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j48wr\" (UniqueName: \"kubernetes.io/projected/5c6e134c-a6fe-4794-9c80-b52ebb11abb2-kube-api-access-j48wr\") pod \"e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-gh5gs\" (UID: \"5c6e134c-a6fe-4794-9c80-b52ebb11abb2\") " pod="openshift-ingress/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-gh5gs" Nov 03 01:07:02.449448 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:02.449409 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-gh5gs" Nov 03 01:07:02.705025 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:02.704944 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-d55st" Nov 03 01:07:02.721295 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:02.718427 4421 patch_prober.go:29] interesting pod/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-d55st container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" start-of-body= Nov 03 01:07:02.721295 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:02.718500 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-d55st" podUID="1548cadc-4d40-4310-89bd-0b2021af6faf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" Nov 03 01:07:02.769541 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:07:02.769211 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c6e134c_a6fe_4794_9c80_b52ebb11abb2.slice/crio-47d0f45950f762349d90ea749b03a069d22a14dbaf0312fdedade3f0f0168ad3 WatchSource:0}: Error finding container 47d0f45950f762349d90ea749b03a069d22a14dbaf0312fdedade3f0f0168ad3: Status 404 returned error can't find the container with id 47d0f45950f762349d90ea749b03a069d22a14dbaf0312fdedade3f0f0168ad3 Nov 03 01:07:02.772315 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:02.771931 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-gh5gs"] Nov 03 01:07:02.772800 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:02.772769 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:07:02.772908 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:02.772846 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:07:02.772908 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:02.772894 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:07:03.220300 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:03.208920 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-gh5gs" event={"ID":"5c6e134c-a6fe-4794-9c80-b52ebb11abb2","Type":"ContainerStarted","Data":"16a01b50082bb8e9803e2536274a12f92a4e69e3e8742e9fc97aad2d2fbe60ed"} Nov 03 01:07:03.221095 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:03.221052 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-gh5gs" event={"ID":"5c6e134c-a6fe-4794-9c80-b52ebb11abb2","Type":"ContainerStarted","Data":"47d0f45950f762349d90ea749b03a069d22a14dbaf0312fdedade3f0f0168ad3"} Nov 03 01:07:03.267299 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:03.263994 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-gh5gs" podStartSLOduration=1.263969915 podStartE2EDuration="1.263969915s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:07:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:07:03.259103233 +0000 UTC m=+1565.430106320" watchObservedRunningTime="2026-11-03 01:07:03.263969915 +0000 UTC m=+1565.434973005" Nov 03 01:07:03.450651 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:03.450618 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-gh5gs" Nov 03 01:07:03.453641 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:03.453357 4421 patch_prober.go:29] interesting pod/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-gh5gs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" start-of-body= Nov 03 01:07:03.453641 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:03.453446 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-gh5gs" podUID="5c6e134c-a6fe-4794-9c80-b52ebb11abb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" Nov 03 01:07:03.570082 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:03.567887 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-96hvv"] Nov 03 01:07:03.579037 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:03.578995 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-96hvv" Nov 03 01:07:03.593297 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:03.591623 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-96hvv"] Nov 03 01:07:03.601695 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:03.599497 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f890f900-55cb-4e5f-b25d-8f5e899bc5c7-credential-socket\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-96hvv\" (UID: \"f890f900-55cb-4e5f-b25d-8f5e899bc5c7\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-96hvv" Nov 03 01:07:03.601695 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:03.599591 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdqz6\" (UniqueName: \"kubernetes.io/projected/f890f900-55cb-4e5f-b25d-8f5e899bc5c7-kube-api-access-pdqz6\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-96hvv\" (UID: \"f890f900-55cb-4e5f-b25d-8f5e899bc5c7\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-96hvv" Nov 03 01:07:03.601695 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:03.599662 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f890f900-55cb-4e5f-b25d-8f5e899bc5c7-istio-token\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-96hvv\" (UID: \"f890f900-55cb-4e5f-b25d-8f5e899bc5c7\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-96hvv" Nov 03 01:07:03.601695 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:03.599700 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f890f900-55cb-4e5f-b25d-8f5e899bc5c7-istio-podinfo\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-96hvv\" (UID: \"f890f900-55cb-4e5f-b25d-8f5e899bc5c7\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-96hvv" Nov 03 01:07:03.601695 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:03.599735 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f890f900-55cb-4e5f-b25d-8f5e899bc5c7-istio-envoy\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-96hvv\" (UID: \"f890f900-55cb-4e5f-b25d-8f5e899bc5c7\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-96hvv" Nov 03 01:07:03.601695 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:03.599765 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f890f900-55cb-4e5f-b25d-8f5e899bc5c7-istiod-ca-cert\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-96hvv\" (UID: \"f890f900-55cb-4e5f-b25d-8f5e899bc5c7\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-96hvv" Nov 03 01:07:03.601695 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:03.599903 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f890f900-55cb-4e5f-b25d-8f5e899bc5c7-workload-certs\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-96hvv\" (UID: \"f890f900-55cb-4e5f-b25d-8f5e899bc5c7\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-96hvv" Nov 03 01:07:03.601695 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:03.599984 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f890f900-55cb-4e5f-b25d-8f5e899bc5c7-istio-data\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-96hvv\" (UID: \"f890f900-55cb-4e5f-b25d-8f5e899bc5c7\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-96hvv" Nov 03 01:07:03.601695 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:03.600084 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f890f900-55cb-4e5f-b25d-8f5e899bc5c7-workload-socket\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-96hvv\" (UID: \"f890f900-55cb-4e5f-b25d-8f5e899bc5c7\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-96hvv" Nov 03 01:07:03.929410 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:03.886352 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f890f900-55cb-4e5f-b25d-8f5e899bc5c7-workload-socket\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-96hvv\" (UID: \"f890f900-55cb-4e5f-b25d-8f5e899bc5c7\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-96hvv" Nov 03 01:07:03.929410 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:03.886549 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f890f900-55cb-4e5f-b25d-8f5e899bc5c7-credential-socket\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-96hvv\" (UID: \"f890f900-55cb-4e5f-b25d-8f5e899bc5c7\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-96hvv" Nov 03 01:07:03.929410 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:03.886665 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdqz6\" (UniqueName: \"kubernetes.io/projected/f890f900-55cb-4e5f-b25d-8f5e899bc5c7-kube-api-access-pdqz6\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-96hvv\" (UID: \"f890f900-55cb-4e5f-b25d-8f5e899bc5c7\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-96hvv" Nov 03 01:07:03.929410 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:03.886760 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f890f900-55cb-4e5f-b25d-8f5e899bc5c7-istio-token\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-96hvv\" (UID: \"f890f900-55cb-4e5f-b25d-8f5e899bc5c7\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-96hvv" Nov 03 01:07:03.929410 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:03.886818 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f890f900-55cb-4e5f-b25d-8f5e899bc5c7-istio-podinfo\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-96hvv\" (UID: \"f890f900-55cb-4e5f-b25d-8f5e899bc5c7\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-96hvv" Nov 03 01:07:03.929410 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:03.886888 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f890f900-55cb-4e5f-b25d-8f5e899bc5c7-istio-envoy\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-96hvv\" (UID: \"f890f900-55cb-4e5f-b25d-8f5e899bc5c7\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-96hvv" Nov 03 01:07:03.929410 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:03.886929 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f890f900-55cb-4e5f-b25d-8f5e899bc5c7-istiod-ca-cert\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-96hvv\" (UID: \"f890f900-55cb-4e5f-b25d-8f5e899bc5c7\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-96hvv" Nov 03 01:07:03.929410 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:03.922057 4421 patch_prober.go:29] interesting pod/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-d55st container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" start-of-body= Nov 03 01:07:03.929410 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:03.922212 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-d55st" podUID="1548cadc-4d40-4310-89bd-0b2021af6faf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" Nov 03 01:07:03.956443 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:03.931033 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f890f900-55cb-4e5f-b25d-8f5e899bc5c7-workload-certs\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-96hvv\" (UID: \"f890f900-55cb-4e5f-b25d-8f5e899bc5c7\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-96hvv" Nov 03 01:07:03.956443 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:03.931776 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f890f900-55cb-4e5f-b25d-8f5e899bc5c7-workload-socket\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-96hvv\" (UID: \"f890f900-55cb-4e5f-b25d-8f5e899bc5c7\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-96hvv" Nov 03 01:07:03.956443 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:03.932121 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f890f900-55cb-4e5f-b25d-8f5e899bc5c7-credential-socket\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-96hvv\" (UID: \"f890f900-55cb-4e5f-b25d-8f5e899bc5c7\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-96hvv" Nov 03 01:07:03.956443 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:03.887157 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f890f900-55cb-4e5f-b25d-8f5e899bc5c7-workload-certs\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-96hvv\" (UID: \"f890f900-55cb-4e5f-b25d-8f5e899bc5c7\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-96hvv" Nov 03 01:07:03.956443 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:03.933696 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f890f900-55cb-4e5f-b25d-8f5e899bc5c7-istio-data\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-96hvv\" (UID: \"f890f900-55cb-4e5f-b25d-8f5e899bc5c7\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-96hvv" Nov 03 01:07:04.000836 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:03.999508 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f890f900-55cb-4e5f-b25d-8f5e899bc5c7-istio-data\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-96hvv\" (UID: \"f890f900-55cb-4e5f-b25d-8f5e899bc5c7\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-96hvv" Nov 03 01:07:04.023949 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:04.023598 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f890f900-55cb-4e5f-b25d-8f5e899bc5c7-istiod-ca-cert\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-96hvv\" (UID: \"f890f900-55cb-4e5f-b25d-8f5e899bc5c7\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-96hvv" Nov 03 01:07:04.094837 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:04.085196 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdqz6\" (UniqueName: \"kubernetes.io/projected/f890f900-55cb-4e5f-b25d-8f5e899bc5c7-kube-api-access-pdqz6\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-96hvv\" (UID: \"f890f900-55cb-4e5f-b25d-8f5e899bc5c7\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-96hvv" Nov 03 01:07:04.094837 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:04.086586 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f890f900-55cb-4e5f-b25d-8f5e899bc5c7-istio-token\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-96hvv\" (UID: \"f890f900-55cb-4e5f-b25d-8f5e899bc5c7\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-96hvv" Nov 03 01:07:04.101889 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:04.100878 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f890f900-55cb-4e5f-b25d-8f5e899bc5c7-istio-envoy\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-96hvv\" (UID: \"f890f900-55cb-4e5f-b25d-8f5e899bc5c7\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-96hvv" Nov 03 01:07:04.185978 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:04.185863 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f890f900-55cb-4e5f-b25d-8f5e899bc5c7-istio-podinfo\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-96hvv\" (UID: \"f890f900-55cb-4e5f-b25d-8f5e899bc5c7\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-96hvv" Nov 03 01:07:04.396529 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:04.396496 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-96hvv" Nov 03 01:07:04.460392 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:04.460268 4421 patch_prober.go:29] interesting pod/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-gh5gs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" start-of-body= Nov 03 01:07:04.460637 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:04.460597 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-gh5gs" podUID="5c6e134c-a6fe-4794-9c80-b52ebb11abb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" Nov 03 01:07:04.774390 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:04.744914 4421 patch_prober.go:29] interesting pod/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-d55st container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" start-of-body= Nov 03 01:07:04.774760 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:04.774648 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-d55st" podUID="1548cadc-4d40-4310-89bd-0b2021af6faf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" Nov 03 01:07:04.885342 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:04.885028 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:07:04.885342 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:04.885125 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:07:04.885342 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:04.885176 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:07:04.887010 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:04.886988 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-96hvv"] Nov 03 01:07:05.288773 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:05.288737 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-96hvv" event={"ID":"f890f900-55cb-4e5f-b25d-8f5e899bc5c7","Type":"ContainerStarted","Data":"abcc3adc4d6cdff2cfb43a436232425ed085e2f44b8634aac12f68463233d03c"} Nov 03 01:07:05.289070 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:05.288804 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-96hvv" event={"ID":"f890f900-55cb-4e5f-b25d-8f5e899bc5c7","Type":"ContainerStarted","Data":"d51c99bf88db8a11698844c5fe654e29dc7a88c3879c3d6e80eb0f81ce6184f9"} Nov 03 01:07:05.368301 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:05.367754 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-96hvv" podStartSLOduration=2.367733062 podStartE2EDuration="2.367733062s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:07:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:07:05.351474502 +0000 UTC m=+1567.522477615" watchObservedRunningTime="2026-11-03 01:07:05.367733062 +0000 UTC m=+1567.538736160" Nov 03 01:07:05.397559 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:05.397529 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-96hvv" Nov 03 01:07:05.454301 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:05.454021 4421 patch_prober.go:29] interesting pod/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-gh5gs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" start-of-body= Nov 03 01:07:05.454301 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:05.454087 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-gh5gs" podUID="5c6e134c-a6fe-4794-9c80-b52ebb11abb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" Nov 03 01:07:05.708351 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:05.704569 4421 patch_prober.go:29] interesting pod/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-d55st container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" start-of-body= Nov 03 01:07:05.708351 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:05.704634 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-d55st" podUID="1548cadc-4d40-4310-89bd-0b2021af6faf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" Nov 03 01:07:06.447696 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:06.418424 4421 patch_prober.go:29] interesting pod/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-96hvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.226:15021/healthz/ready\": dial tcp 10.134.0.226:15021: connect: connection refused" start-of-body= Nov 03 01:07:06.447696 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:06.418492 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-96hvv" podUID="f890f900-55cb-4e5f-b25d-8f5e899bc5c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.226:15021/healthz/ready\": dial tcp 10.134.0.226:15021: connect: connection refused" Nov 03 01:07:06.451744 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:06.451707 4421 patch_prober.go:29] interesting pod/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-gh5gs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" start-of-body= Nov 03 01:07:06.452035 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:06.452004 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-gh5gs" podUID="5c6e134c-a6fe-4794-9c80-b52ebb11abb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" Nov 03 01:07:06.736360 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:06.726812 4421 patch_prober.go:29] interesting pod/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-d55st container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" start-of-body= Nov 03 01:07:06.736360 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:06.726879 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-d55st" podUID="1548cadc-4d40-4310-89bd-0b2021af6faf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" Nov 03 01:07:06.910311 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:06.903350 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-4248ae-openshift-default-94bc8c67-b5pgk"] Nov 03 01:07:06.910311 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:06.903613 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-4248ae-openshift-default-94bc8c67-b5pgk" podUID="2afb12c9-7c7a-4527-95c4-0b2febd22e90" containerName="istio-proxy" containerID="cri-o://e80a07df9efede7a14c7c16e47016e346dcb62910ad6c53860f41ccecc0f403e" gracePeriod=30 Nov 03 01:07:07.397474 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:07.397435 4421 patch_prober.go:29] interesting pod/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-96hvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.226:15021/healthz/ready\": dial tcp 10.134.0.226:15021: connect: connection refused" start-of-body= Nov 03 01:07:07.397748 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:07.397716 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-96hvv" podUID="f890f900-55cb-4e5f-b25d-8f5e899bc5c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.226:15021/healthz/ready\": dial tcp 10.134.0.226:15021: connect: connection refused" Nov 03 01:07:07.461704 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:07.461663 4421 patch_prober.go:29] interesting pod/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-gh5gs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" start-of-body= Nov 03 01:07:07.491555 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:07.491514 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-gh5gs" podUID="5c6e134c-a6fe-4794-9c80-b52ebb11abb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" Nov 03 01:07:07.704808 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:07.704715 4421 patch_prober.go:29] interesting pod/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-d55st container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" start-of-body= Nov 03 01:07:07.705057 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:07.705028 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-d55st" podUID="1548cadc-4d40-4310-89bd-0b2021af6faf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" Nov 03 01:07:07.802651 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:07.802606 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-84e377-openshift-default-5554bf8669-lzzbj"] Nov 03 01:07:07.869714 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:07.869684 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-84e377-openshift-default-5554bf8669-lzzbj" Nov 03 01:07:07.900307 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:07.898052 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-84e377-openshift-default-5554bf8669-lzzbj"] Nov 03 01:07:08.041569 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:08.041532 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9815803-c1dc-4c21-a8f3-2a3c9c89f1b9-istio-token\") pod \"e2e-explicit-84e377-openshift-default-5554bf8669-lzzbj\" (UID: \"b9815803-c1dc-4c21-a8f3-2a3c9c89f1b9\") " pod="openshift-ingress/e2e-explicit-84e377-openshift-default-5554bf8669-lzzbj" Nov 03 01:07:08.041916 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:08.041889 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9815803-c1dc-4c21-a8f3-2a3c9c89f1b9-istio-podinfo\") pod \"e2e-explicit-84e377-openshift-default-5554bf8669-lzzbj\" (UID: \"b9815803-c1dc-4c21-a8f3-2a3c9c89f1b9\") " pod="openshift-ingress/e2e-explicit-84e377-openshift-default-5554bf8669-lzzbj" Nov 03 01:07:08.042152 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:08.042133 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9815803-c1dc-4c21-a8f3-2a3c9c89f1b9-credential-socket\") pod \"e2e-explicit-84e377-openshift-default-5554bf8669-lzzbj\" (UID: \"b9815803-c1dc-4c21-a8f3-2a3c9c89f1b9\") " pod="openshift-ingress/e2e-explicit-84e377-openshift-default-5554bf8669-lzzbj" Nov 03 01:07:08.042372 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:08.042354 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9815803-c1dc-4c21-a8f3-2a3c9c89f1b9-workload-certs\") pod \"e2e-explicit-84e377-openshift-default-5554bf8669-lzzbj\" (UID: \"b9815803-c1dc-4c21-a8f3-2a3c9c89f1b9\") " pod="openshift-ingress/e2e-explicit-84e377-openshift-default-5554bf8669-lzzbj" Nov 03 01:07:08.042627 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:08.042606 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9815803-c1dc-4c21-a8f3-2a3c9c89f1b9-workload-socket\") pod \"e2e-explicit-84e377-openshift-default-5554bf8669-lzzbj\" (UID: \"b9815803-c1dc-4c21-a8f3-2a3c9c89f1b9\") " pod="openshift-ingress/e2e-explicit-84e377-openshift-default-5554bf8669-lzzbj" Nov 03 01:07:08.042922 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:08.042886 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9815803-c1dc-4c21-a8f3-2a3c9c89f1b9-istiod-ca-cert\") pod \"e2e-explicit-84e377-openshift-default-5554bf8669-lzzbj\" (UID: \"b9815803-c1dc-4c21-a8f3-2a3c9c89f1b9\") " pod="openshift-ingress/e2e-explicit-84e377-openshift-default-5554bf8669-lzzbj" Nov 03 01:07:08.043132 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:08.043094 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9815803-c1dc-4c21-a8f3-2a3c9c89f1b9-istio-envoy\") pod \"e2e-explicit-84e377-openshift-default-5554bf8669-lzzbj\" (UID: \"b9815803-c1dc-4c21-a8f3-2a3c9c89f1b9\") " pod="openshift-ingress/e2e-explicit-84e377-openshift-default-5554bf8669-lzzbj" Nov 03 01:07:08.043566 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:08.043541 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9815803-c1dc-4c21-a8f3-2a3c9c89f1b9-istio-data\") pod \"e2e-explicit-84e377-openshift-default-5554bf8669-lzzbj\" (UID: \"b9815803-c1dc-4c21-a8f3-2a3c9c89f1b9\") " pod="openshift-ingress/e2e-explicit-84e377-openshift-default-5554bf8669-lzzbj" Nov 03 01:07:08.144313 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:08.118333 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-84e377-openshift-default-dockercfg-znvh2\"" Nov 03 01:07:08.284338 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:08.280116 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8b42x\" (UniqueName: \"kubernetes.io/projected/b9815803-c1dc-4c21-a8f3-2a3c9c89f1b9-kube-api-access-8b42x\") pod \"e2e-explicit-84e377-openshift-default-5554bf8669-lzzbj\" (UID: \"b9815803-c1dc-4c21-a8f3-2a3c9c89f1b9\") " pod="openshift-ingress/e2e-explicit-84e377-openshift-default-5554bf8669-lzzbj" Nov 03 01:07:08.562380 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:08.558378 4421 patch_prober.go:29] interesting pod/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-96hvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.226:15021/healthz/ready\": dial tcp 10.134.0.226:15021: connect: connection refused" start-of-body= Nov 03 01:07:08.562380 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:08.558438 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-96hvv" podUID="f890f900-55cb-4e5f-b25d-8f5e899bc5c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.226:15021/healthz/ready\": dial tcp 10.134.0.226:15021: connect: connection refused" Nov 03 01:07:08.562380 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:08.559999 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8b42x\" (UniqueName: \"kubernetes.io/projected/b9815803-c1dc-4c21-a8f3-2a3c9c89f1b9-kube-api-access-8b42x\") pod \"e2e-explicit-84e377-openshift-default-5554bf8669-lzzbj\" (UID: \"b9815803-c1dc-4c21-a8f3-2a3c9c89f1b9\") " pod="openshift-ingress/e2e-explicit-84e377-openshift-default-5554bf8669-lzzbj" Nov 03 01:07:08.562380 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:08.560194 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9815803-c1dc-4c21-a8f3-2a3c9c89f1b9-istio-token\") pod \"e2e-explicit-84e377-openshift-default-5554bf8669-lzzbj\" (UID: \"b9815803-c1dc-4c21-a8f3-2a3c9c89f1b9\") " pod="openshift-ingress/e2e-explicit-84e377-openshift-default-5554bf8669-lzzbj" Nov 03 01:07:08.562380 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:08.560258 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9815803-c1dc-4c21-a8f3-2a3c9c89f1b9-istio-podinfo\") pod \"e2e-explicit-84e377-openshift-default-5554bf8669-lzzbj\" (UID: \"b9815803-c1dc-4c21-a8f3-2a3c9c89f1b9\") " pod="openshift-ingress/e2e-explicit-84e377-openshift-default-5554bf8669-lzzbj" Nov 03 01:07:08.562380 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:08.560363 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9815803-c1dc-4c21-a8f3-2a3c9c89f1b9-credential-socket\") pod \"e2e-explicit-84e377-openshift-default-5554bf8669-lzzbj\" (UID: \"b9815803-c1dc-4c21-a8f3-2a3c9c89f1b9\") " pod="openshift-ingress/e2e-explicit-84e377-openshift-default-5554bf8669-lzzbj" Nov 03 01:07:08.562380 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:08.560460 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9815803-c1dc-4c21-a8f3-2a3c9c89f1b9-workload-certs\") pod \"e2e-explicit-84e377-openshift-default-5554bf8669-lzzbj\" (UID: \"b9815803-c1dc-4c21-a8f3-2a3c9c89f1b9\") " pod="openshift-ingress/e2e-explicit-84e377-openshift-default-5554bf8669-lzzbj" Nov 03 01:07:08.562380 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:08.560563 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9815803-c1dc-4c21-a8f3-2a3c9c89f1b9-workload-socket\") pod \"e2e-explicit-84e377-openshift-default-5554bf8669-lzzbj\" (UID: \"b9815803-c1dc-4c21-a8f3-2a3c9c89f1b9\") " pod="openshift-ingress/e2e-explicit-84e377-openshift-default-5554bf8669-lzzbj" Nov 03 01:07:08.562380 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:08.560663 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9815803-c1dc-4c21-a8f3-2a3c9c89f1b9-istiod-ca-cert\") pod \"e2e-explicit-84e377-openshift-default-5554bf8669-lzzbj\" (UID: \"b9815803-c1dc-4c21-a8f3-2a3c9c89f1b9\") " pod="openshift-ingress/e2e-explicit-84e377-openshift-default-5554bf8669-lzzbj" Nov 03 01:07:08.562380 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:08.560734 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9815803-c1dc-4c21-a8f3-2a3c9c89f1b9-istio-envoy\") pod \"e2e-explicit-84e377-openshift-default-5554bf8669-lzzbj\" (UID: \"b9815803-c1dc-4c21-a8f3-2a3c9c89f1b9\") " pod="openshift-ingress/e2e-explicit-84e377-openshift-default-5554bf8669-lzzbj" Nov 03 01:07:08.562380 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:08.560947 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9815803-c1dc-4c21-a8f3-2a3c9c89f1b9-istio-data\") pod \"e2e-explicit-84e377-openshift-default-5554bf8669-lzzbj\" (UID: \"b9815803-c1dc-4c21-a8f3-2a3c9c89f1b9\") " pod="openshift-ingress/e2e-explicit-84e377-openshift-default-5554bf8669-lzzbj" Nov 03 01:07:08.562380 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:08.561507 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9815803-c1dc-4c21-a8f3-2a3c9c89f1b9-istio-data\") pod \"e2e-explicit-84e377-openshift-default-5554bf8669-lzzbj\" (UID: \"b9815803-c1dc-4c21-a8f3-2a3c9c89f1b9\") " pod="openshift-ingress/e2e-explicit-84e377-openshift-default-5554bf8669-lzzbj" Nov 03 01:07:08.646478 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:08.595861 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9815803-c1dc-4c21-a8f3-2a3c9c89f1b9-workload-certs\") pod \"e2e-explicit-84e377-openshift-default-5554bf8669-lzzbj\" (UID: \"b9815803-c1dc-4c21-a8f3-2a3c9c89f1b9\") " pod="openshift-ingress/e2e-explicit-84e377-openshift-default-5554bf8669-lzzbj" Nov 03 01:07:08.646478 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:08.596141 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9815803-c1dc-4c21-a8f3-2a3c9c89f1b9-workload-socket\") pod \"e2e-explicit-84e377-openshift-default-5554bf8669-lzzbj\" (UID: \"b9815803-c1dc-4c21-a8f3-2a3c9c89f1b9\") " pod="openshift-ingress/e2e-explicit-84e377-openshift-default-5554bf8669-lzzbj" Nov 03 01:07:08.646478 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:08.596819 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9815803-c1dc-4c21-a8f3-2a3c9c89f1b9-istiod-ca-cert\") pod \"e2e-explicit-84e377-openshift-default-5554bf8669-lzzbj\" (UID: \"b9815803-c1dc-4c21-a8f3-2a3c9c89f1b9\") " pod="openshift-ingress/e2e-explicit-84e377-openshift-default-5554bf8669-lzzbj" Nov 03 01:07:08.646478 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:08.641799 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9815803-c1dc-4c21-a8f3-2a3c9c89f1b9-istio-envoy\") pod \"e2e-explicit-84e377-openshift-default-5554bf8669-lzzbj\" (UID: \"b9815803-c1dc-4c21-a8f3-2a3c9c89f1b9\") " pod="openshift-ingress/e2e-explicit-84e377-openshift-default-5554bf8669-lzzbj" Nov 03 01:07:08.688075 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:08.673909 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9815803-c1dc-4c21-a8f3-2a3c9c89f1b9-credential-socket\") pod \"e2e-explicit-84e377-openshift-default-5554bf8669-lzzbj\" (UID: \"b9815803-c1dc-4c21-a8f3-2a3c9c89f1b9\") " pod="openshift-ingress/e2e-explicit-84e377-openshift-default-5554bf8669-lzzbj" Nov 03 01:07:08.717611 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:08.693480 4421 patch_prober.go:29] interesting pod/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-gh5gs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" start-of-body= Nov 03 01:07:08.750947 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:08.739258 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9815803-c1dc-4c21-a8f3-2a3c9c89f1b9-istio-token\") pod \"e2e-explicit-84e377-openshift-default-5554bf8669-lzzbj\" (UID: \"b9815803-c1dc-4c21-a8f3-2a3c9c89f1b9\") " pod="openshift-ingress/e2e-explicit-84e377-openshift-default-5554bf8669-lzzbj" Nov 03 01:07:08.751190 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:08.745476 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9815803-c1dc-4c21-a8f3-2a3c9c89f1b9-istio-podinfo\") pod \"e2e-explicit-84e377-openshift-default-5554bf8669-lzzbj\" (UID: \"b9815803-c1dc-4c21-a8f3-2a3c9c89f1b9\") " pod="openshift-ingress/e2e-explicit-84e377-openshift-default-5554bf8669-lzzbj" Nov 03 01:07:08.751640 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:08.716604 4421 patch_prober.go:29] interesting pod/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-d55st container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" start-of-body= Nov 03 01:07:08.751810 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:08.751791 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-d55st" podUID="1548cadc-4d40-4310-89bd-0b2021af6faf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" Nov 03 01:07:08.752102 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:08.750655 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-gh5gs" podUID="5c6e134c-a6fe-4794-9c80-b52ebb11abb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" Nov 03 01:07:08.775599 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:08.775558 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8b42x\" (UniqueName: \"kubernetes.io/projected/b9815803-c1dc-4c21-a8f3-2a3c9c89f1b9-kube-api-access-8b42x\") pod \"e2e-explicit-84e377-openshift-default-5554bf8669-lzzbj\" (UID: \"b9815803-c1dc-4c21-a8f3-2a3c9c89f1b9\") " pod="openshift-ingress/e2e-explicit-84e377-openshift-default-5554bf8669-lzzbj" Nov 03 01:07:08.917707 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:08.917598 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-84e377-openshift-default-5554bf8669-lzzbj" Nov 03 01:07:09.377437 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:09.377397 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-84e377-openshift-default-5554bf8669-lzzbj"] Nov 03 01:07:09.381419 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:09.381379 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:07:09.381508 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:09.381467 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:07:09.381560 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:09.381508 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:07:09.397740 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:09.397681 4421 patch_prober.go:29] interesting pod/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-96hvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.226:15021/healthz/ready\": dial tcp 10.134.0.226:15021: connect: connection refused" start-of-body= Nov 03 01:07:09.397863 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:09.397774 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-96hvv" podUID="f890f900-55cb-4e5f-b25d-8f5e899bc5c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.226:15021/healthz/ready\": dial tcp 10.134.0.226:15021: connect: connection refused" Nov 03 01:07:09.475049 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:09.475014 4421 patch_prober.go:29] interesting pod/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-gh5gs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" start-of-body= Nov 03 01:07:09.475656 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:09.475246 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-gh5gs" podUID="5c6e134c-a6fe-4794-9c80-b52ebb11abb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" Nov 03 01:07:09.773308 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:09.772540 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-d55st" Nov 03 01:07:09.774850 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:09.774389 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-d55st" Nov 03 01:07:09.791540 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:09.791377 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-d55st" Nov 03 01:07:09.803058 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:09.803024 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-84e377-openshift-default-5554bf8669-lzzbj" event={"ID":"b9815803-c1dc-4c21-a8f3-2a3c9c89f1b9","Type":"ContainerStarted","Data":"8741545e1131e3da6e37b3db71aff7890695839c68673721250d33ee4b513d48"} Nov 03 01:07:09.803345 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:09.803322 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-84e377-openshift-default-5554bf8669-lzzbj" event={"ID":"b9815803-c1dc-4c21-a8f3-2a3c9c89f1b9","Type":"ContainerStarted","Data":"729da746a26192db0e74880b9d679cd1e70fb9242c4f9374cc7253aae5e3f9e8"} Nov 03 01:07:09.944300 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:09.923522 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-84e377-openshift-default-5554bf8669-lzzbj" Nov 03 01:07:10.026999 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:10.026857 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-84e377-openshift-default-5554bf8669-lzzbj" podStartSLOduration=3.026835873 podStartE2EDuration="3.026835873s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:07:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:07:10.016707196 +0000 UTC m=+1572.187710305" watchObservedRunningTime="2026-11-03 01:07:10.026835873 +0000 UTC m=+1572.197838970" Nov 03 01:07:10.396983 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:10.396857 4421 patch_prober.go:29] interesting pod/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-96hvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.226:15021/healthz/ready\": dial tcp 10.134.0.226:15021: connect: connection refused" start-of-body= Nov 03 01:07:10.396983 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:10.396922 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-96hvv" podUID="f890f900-55cb-4e5f-b25d-8f5e899bc5c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.226:15021/healthz/ready\": dial tcp 10.134.0.226:15021: connect: connection refused" Nov 03 01:07:10.449795 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:10.449761 4421 patch_prober.go:29] interesting pod/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-gh5gs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" start-of-body= Nov 03 01:07:10.449996 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:10.449822 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-gh5gs" podUID="5c6e134c-a6fe-4794-9c80-b52ebb11abb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" Nov 03 01:07:10.939133 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:10.939089 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-84e377-openshift-default-5554bf8669-lzzbj" Nov 03 01:07:11.086477 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:11.086439 4421 patch_prober.go:29] interesting pod/e2e-autores-4248ae-openshift-default-94bc8c67-b5pgk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 01:07:11.086661 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:11.086497 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-autores-4248ae-openshift-default-94bc8c67-b5pgk" podUID="2afb12c9-7c7a-4527-95c4-0b2febd22e90" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 01:07:11.397100 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:11.397064 4421 patch_prober.go:29] interesting pod/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-96hvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.226:15021/healthz/ready\": dial tcp 10.134.0.226:15021: connect: connection refused" start-of-body= Nov 03 01:07:11.397317 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:11.397128 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-96hvv" podUID="f890f900-55cb-4e5f-b25d-8f5e899bc5c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.226:15021/healthz/ready\": dial tcp 10.134.0.226:15021: connect: connection refused" Nov 03 01:07:11.455101 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:11.455070 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-gh5gs" Nov 03 01:07:11.455559 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:11.455539 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-gh5gs" Nov 03 01:07:11.456232 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:11.456213 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-gh5gs" Nov 03 01:07:11.838442 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:11.838409 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-84e377-openshift-default-5554bf8669-lzzbj" Nov 03 01:07:11.839229 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:11.839211 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-84e377-openshift-default-5554bf8669-lzzbj" Nov 03 01:07:12.397166 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:12.397125 4421 patch_prober.go:29] interesting pod/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-96hvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.226:15021/healthz/ready\": dial tcp 10.134.0.226:15021: connect: connection refused" start-of-body= Nov 03 01:07:12.397593 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:12.397207 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-96hvv" podUID="f890f900-55cb-4e5f-b25d-8f5e899bc5c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.226:15021/healthz/ready\": dial tcp 10.134.0.226:15021: connect: connection refused" Nov 03 01:07:12.763063 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:12.763032 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-4248ae-openshift-default-94bc8c67-b5pgk" Nov 03 01:07:12.855359 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:12.855320 4421 generic.go:356] "Generic (PLEG): container finished" podID="2afb12c9-7c7a-4527-95c4-0b2febd22e90" containerID="e80a07df9efede7a14c7c16e47016e346dcb62910ad6c53860f41ccecc0f403e" exitCode=0 Nov 03 01:07:12.856664 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:12.856637 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-4248ae-openshift-default-94bc8c67-b5pgk" Nov 03 01:07:12.857120 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:12.857095 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-4248ae-openshift-default-94bc8c67-b5pgk" event={"ID":"2afb12c9-7c7a-4527-95c4-0b2febd22e90","Type":"ContainerDied","Data":"e80a07df9efede7a14c7c16e47016e346dcb62910ad6c53860f41ccecc0f403e"} Nov 03 01:07:12.857207 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:12.857154 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-4248ae-openshift-default-94bc8c67-b5pgk" event={"ID":"2afb12c9-7c7a-4527-95c4-0b2febd22e90","Type":"ContainerDied","Data":"97f6cd7a1f4bb5a0fc1795e160b671084b1dc197eb9cb2a75b67b507d1751906"} Nov 03 01:07:12.857207 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:12.857186 4421 scope.go:122] "RemoveContainer" containerID="e80a07df9efede7a14c7c16e47016e346dcb62910ad6c53860f41ccecc0f403e" Nov 03 01:07:12.871412 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:12.871389 4421 scope.go:122] "RemoveContainer" containerID="20a772bbd0dc8daf02ef3acf699274dd1c38946168746fc7b03613467719f1b5" Nov 03 01:07:12.878493 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:12.878222 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2afb12c9-7c7a-4527-95c4-0b2febd22e90-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2afb12c9-7c7a-4527-95c4-0b2febd22e90-istio-podinfo\") pod \"2afb12c9-7c7a-4527-95c4-0b2febd22e90\" (UID: \"2afb12c9-7c7a-4527-95c4-0b2febd22e90\") " Nov 03 01:07:12.878493 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:12.878311 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2afb12c9-7c7a-4527-95c4-0b2febd22e90-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2afb12c9-7c7a-4527-95c4-0b2febd22e90-istio-envoy\") pod \"2afb12c9-7c7a-4527-95c4-0b2febd22e90\" (UID: \"2afb12c9-7c7a-4527-95c4-0b2febd22e90\") " Nov 03 01:07:12.878493 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:12.878336 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2afb12c9-7c7a-4527-95c4-0b2febd22e90-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2afb12c9-7c7a-4527-95c4-0b2febd22e90-credential-socket\") pod \"2afb12c9-7c7a-4527-95c4-0b2febd22e90\" (UID: \"2afb12c9-7c7a-4527-95c4-0b2febd22e90\") " Nov 03 01:07:12.878493 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:12.878390 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2afb12c9-7c7a-4527-95c4-0b2febd22e90-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2afb12c9-7c7a-4527-95c4-0b2febd22e90-workload-socket\") pod \"2afb12c9-7c7a-4527-95c4-0b2febd22e90\" (UID: \"2afb12c9-7c7a-4527-95c4-0b2febd22e90\") " Nov 03 01:07:12.884074 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:12.878671 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2afb12c9-7c7a-4527-95c4-0b2febd22e90-credential-socket" pod "2afb12c9-7c7a-4527-95c4-0b2febd22e90" (UID: "2afb12c9-7c7a-4527-95c4-0b2febd22e90"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:07:12.884074 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:12.878696 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2afb12c9-7c7a-4527-95c4-0b2febd22e90-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2afb12c9-7c7a-4527-95c4-0b2febd22e90-istio-data\") pod \"2afb12c9-7c7a-4527-95c4-0b2febd22e90\" (UID: \"2afb12c9-7c7a-4527-95c4-0b2febd22e90\") " Nov 03 01:07:12.884074 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:12.878751 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2afb12c9-7c7a-4527-95c4-0b2febd22e90-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2afb12c9-7c7a-4527-95c4-0b2febd22e90-istiod-ca-cert\") pod \"2afb12c9-7c7a-4527-95c4-0b2febd22e90\" (UID: \"2afb12c9-7c7a-4527-95c4-0b2febd22e90\") " Nov 03 01:07:12.884074 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:12.878853 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2afb12c9-7c7a-4527-95c4-0b2febd22e90-workload-socket" pod "2afb12c9-7c7a-4527-95c4-0b2febd22e90" (UID: "2afb12c9-7c7a-4527-95c4-0b2febd22e90"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:07:12.884074 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:12.878936 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2afb12c9-7c7a-4527-95c4-0b2febd22e90-istio-data" pod "2afb12c9-7c7a-4527-95c4-0b2febd22e90" (UID: "2afb12c9-7c7a-4527-95c4-0b2febd22e90"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:07:12.884074 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:12.878948 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2afb12c9-7c7a-4527-95c4-0b2febd22e90-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2afb12c9-7c7a-4527-95c4-0b2febd22e90-workload-certs\") pod \"2afb12c9-7c7a-4527-95c4-0b2febd22e90\" (UID: \"2afb12c9-7c7a-4527-95c4-0b2febd22e90\") " Nov 03 01:07:12.884074 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:12.879028 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2afb12c9-7c7a-4527-95c4-0b2febd22e90-kube-api-access-vwz2h\" (UniqueName: \"kubernetes.io/projected/2afb12c9-7c7a-4527-95c4-0b2febd22e90-kube-api-access-vwz2h\") pod \"2afb12c9-7c7a-4527-95c4-0b2febd22e90\" (UID: \"2afb12c9-7c7a-4527-95c4-0b2febd22e90\") " Nov 03 01:07:12.884074 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:12.879094 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2afb12c9-7c7a-4527-95c4-0b2febd22e90-istio-token\" (UniqueName: \"kubernetes.io/projected/2afb12c9-7c7a-4527-95c4-0b2febd22e90-istio-token\") pod \"2afb12c9-7c7a-4527-95c4-0b2febd22e90\" (UID: \"2afb12c9-7c7a-4527-95c4-0b2febd22e90\") " Nov 03 01:07:12.884074 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:12.879129 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2afb12c9-7c7a-4527-95c4-0b2febd22e90-workload-certs" pod "2afb12c9-7c7a-4527-95c4-0b2febd22e90" (UID: "2afb12c9-7c7a-4527-95c4-0b2febd22e90"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:07:12.884074 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:12.879202 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2afb12c9-7c7a-4527-95c4-0b2febd22e90-istiod-ca-cert" pod "2afb12c9-7c7a-4527-95c4-0b2febd22e90" (UID: "2afb12c9-7c7a-4527-95c4-0b2febd22e90"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:07:12.884074 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:12.879973 4421 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2afb12c9-7c7a-4527-95c4-0b2febd22e90-istio-data\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:07:12.884074 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:12.879996 4421 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2afb12c9-7c7a-4527-95c4-0b2febd22e90-istiod-ca-cert\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:07:12.884074 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:12.880012 4421 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2afb12c9-7c7a-4527-95c4-0b2febd22e90-workload-certs\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:07:12.884074 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:12.880028 4421 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2afb12c9-7c7a-4527-95c4-0b2febd22e90-credential-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:07:12.884074 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:12.880042 4421 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2afb12c9-7c7a-4527-95c4-0b2febd22e90-workload-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:07:12.884074 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:12.881471 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2afb12c9-7c7a-4527-95c4-0b2febd22e90-istio-podinfo" pod "2afb12c9-7c7a-4527-95c4-0b2febd22e90" (UID: "2afb12c9-7c7a-4527-95c4-0b2febd22e90"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:07:12.884074 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:12.881752 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2afb12c9-7c7a-4527-95c4-0b2febd22e90-kube-api-access-vwz2h" pod "2afb12c9-7c7a-4527-95c4-0b2febd22e90" (UID: "2afb12c9-7c7a-4527-95c4-0b2febd22e90"). InnerVolumeSpecName "kube-api-access-vwz2h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:07:12.884074 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:12.882075 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2afb12c9-7c7a-4527-95c4-0b2febd22e90-istio-envoy" pod "2afb12c9-7c7a-4527-95c4-0b2febd22e90" (UID: "2afb12c9-7c7a-4527-95c4-0b2febd22e90"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:07:12.884074 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:12.882095 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2afb12c9-7c7a-4527-95c4-0b2febd22e90-istio-token" pod "2afb12c9-7c7a-4527-95c4-0b2febd22e90" (UID: "2afb12c9-7c7a-4527-95c4-0b2febd22e90"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:07:12.886202 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:12.886182 4421 scope.go:122] "RemoveContainer" containerID="e80a07df9efede7a14c7c16e47016e346dcb62910ad6c53860f41ccecc0f403e" Nov 03 01:07:12.886610 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:07:12.886587 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e80a07df9efede7a14c7c16e47016e346dcb62910ad6c53860f41ccecc0f403e\": container with ID starting with e80a07df9efede7a14c7c16e47016e346dcb62910ad6c53860f41ccecc0f403e not found: ID does not exist" containerID="e80a07df9efede7a14c7c16e47016e346dcb62910ad6c53860f41ccecc0f403e" Nov 03 01:07:12.886698 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:12.886617 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e80a07df9efede7a14c7c16e47016e346dcb62910ad6c53860f41ccecc0f403e"} err="failed to get container status \"e80a07df9efede7a14c7c16e47016e346dcb62910ad6c53860f41ccecc0f403e\": rpc error: code = NotFound desc = could not find container \"e80a07df9efede7a14c7c16e47016e346dcb62910ad6c53860f41ccecc0f403e\": container with ID starting with e80a07df9efede7a14c7c16e47016e346dcb62910ad6c53860f41ccecc0f403e not found: ID does not exist" Nov 03 01:07:12.886698 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:12.886636 4421 scope.go:122] "RemoveContainer" containerID="20a772bbd0dc8daf02ef3acf699274dd1c38946168746fc7b03613467719f1b5" Nov 03 01:07:12.886945 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:07:12.886926 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20a772bbd0dc8daf02ef3acf699274dd1c38946168746fc7b03613467719f1b5\": container with ID starting with 20a772bbd0dc8daf02ef3acf699274dd1c38946168746fc7b03613467719f1b5 not found: ID does not exist" containerID="20a772bbd0dc8daf02ef3acf699274dd1c38946168746fc7b03613467719f1b5" Nov 03 01:07:12.887012 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:12.886958 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20a772bbd0dc8daf02ef3acf699274dd1c38946168746fc7b03613467719f1b5"} err="failed to get container status \"20a772bbd0dc8daf02ef3acf699274dd1c38946168746fc7b03613467719f1b5\": rpc error: code = NotFound desc = could not find container \"20a772bbd0dc8daf02ef3acf699274dd1c38946168746fc7b03613467719f1b5\": container with ID starting with 20a772bbd0dc8daf02ef3acf699274dd1c38946168746fc7b03613467719f1b5 not found: ID does not exist" Nov 03 01:07:12.982669 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:12.982499 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwz2h\" (UniqueName: \"kubernetes.io/projected/2afb12c9-7c7a-4527-95c4-0b2febd22e90-kube-api-access-vwz2h\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:07:12.982669 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:12.982534 4421 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2afb12c9-7c7a-4527-95c4-0b2febd22e90-istio-token\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:07:12.982669 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:12.982550 4421 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2afb12c9-7c7a-4527-95c4-0b2febd22e90-istio-podinfo\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:07:12.982669 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:12.982562 4421 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2afb12c9-7c7a-4527-95c4-0b2febd22e90-istio-envoy\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:07:13.191068 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:13.191012 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-4248ae-openshift-default-94bc8c67-b5pgk"] Nov 03 01:07:13.197743 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:13.197705 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-4248ae-openshift-default-94bc8c67-b5pgk"] Nov 03 01:07:13.407087 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:13.406995 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-96hvv" Nov 03 01:07:13.407866 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:13.407839 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-96hvv" Nov 03 01:07:13.408973 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:13.408951 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-96hvv" Nov 03 01:07:13.769017 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:13.768976 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-84e377-8468cbf5b9-wx4nd"] Nov 03 01:07:13.769976 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:13.769956 4421 cpu_manager.go:414] "containerMap: removing container" podUID="2afb12c9-7c7a-4527-95c4-0b2febd22e90" containerName="istio-proxy" Nov 03 01:07:13.770067 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:13.769978 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2afb12c9-7c7a-4527-95c4-0b2febd22e90" containerName="istio-proxy" Nov 03 01:07:13.770067 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:13.769997 4421 cpu_manager.go:414] "containerMap: removing container" podUID="2afb12c9-7c7a-4527-95c4-0b2febd22e90" containerName="istio-proxy" Nov 03 01:07:13.770067 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:13.770003 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2afb12c9-7c7a-4527-95c4-0b2febd22e90" containerName="istio-proxy" Nov 03 01:07:13.770178 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:13.770113 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="2afb12c9-7c7a-4527-95c4-0b2febd22e90" containerName="istio-proxy" Nov 03 01:07:13.775853 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:13.775824 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-84e377-8468cbf5b9-wx4nd" Nov 03 01:07:13.780190 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:13.780166 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-84e377-dockercfg-8rz6r\"" Nov 03 01:07:13.780627 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:13.780601 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-84e377\"" Nov 03 01:07:13.782456 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:13.782429 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-84e377-8468cbf5b9-wx4nd"] Nov 03 01:07:13.818407 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:13.818376 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-84e377-6bc89ccf99-pczgs"] Nov 03 01:07:13.819563 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:13.819531 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="2afb12c9-7c7a-4527-95c4-0b2febd22e90" containerName="istio-proxy" Nov 03 01:07:13.829123 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:13.827708 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-84e377-6bc89ccf99-pczgs" Nov 03 01:07:13.845615 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:13.838586 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-84e377-6bc89ccf99-pczgs"] Nov 03 01:07:14.032383 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:14.032215 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78d5c907-e2f2-4755-bb98-0b08e6e8dd47-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-84e377-6bc89ccf99-pczgs\" (UID: \"78d5c907-e2f2-4755-bb98-0b08e6e8dd47\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-84e377-6bc89ccf99-pczgs" Nov 03 01:07:14.032727 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:14.032691 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48055fe8-0181-47f8-bdc6-da665d9badac-plugins-config-volume\") pod \"payload-processing-e2e-explicit-84e377-8468cbf5b9-wx4nd\" (UID: \"48055fe8-0181-47f8-bdc6-da665d9badac\") " pod="openshift-ingress/payload-processing-e2e-explicit-84e377-8468cbf5b9-wx4nd" Nov 03 01:07:14.032986 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:14.032960 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nc7sn\" (UniqueName: \"kubernetes.io/projected/78d5c907-e2f2-4755-bb98-0b08e6e8dd47-kube-api-access-nc7sn\") pod \"payload-pre-processing-e2e-explicit-84e377-6bc89ccf99-pczgs\" (UID: \"78d5c907-e2f2-4755-bb98-0b08e6e8dd47\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-84e377-6bc89ccf99-pczgs" Nov 03 01:07:14.033232 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:14.033214 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kjf5\" (UniqueName: \"kubernetes.io/projected/48055fe8-0181-47f8-bdc6-da665d9badac-kube-api-access-5kjf5\") pod \"payload-processing-e2e-explicit-84e377-8468cbf5b9-wx4nd\" (UID: \"48055fe8-0181-47f8-bdc6-da665d9badac\") " pod="openshift-ingress/payload-processing-e2e-explicit-84e377-8468cbf5b9-wx4nd" Nov 03 01:07:14.218405 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:14.213379 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78d5c907-e2f2-4755-bb98-0b08e6e8dd47-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-84e377-6bc89ccf99-pczgs\" (UID: \"78d5c907-e2f2-4755-bb98-0b08e6e8dd47\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-84e377-6bc89ccf99-pczgs" Nov 03 01:07:14.218405 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:14.213528 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48055fe8-0181-47f8-bdc6-da665d9badac-plugins-config-volume\") pod \"payload-processing-e2e-explicit-84e377-8468cbf5b9-wx4nd\" (UID: \"48055fe8-0181-47f8-bdc6-da665d9badac\") " pod="openshift-ingress/payload-processing-e2e-explicit-84e377-8468cbf5b9-wx4nd" Nov 03 01:07:14.218405 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:14.213673 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nc7sn\" (UniqueName: \"kubernetes.io/projected/78d5c907-e2f2-4755-bb98-0b08e6e8dd47-kube-api-access-nc7sn\") pod \"payload-pre-processing-e2e-explicit-84e377-6bc89ccf99-pczgs\" (UID: \"78d5c907-e2f2-4755-bb98-0b08e6e8dd47\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-84e377-6bc89ccf99-pczgs" Nov 03 01:07:14.218405 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:14.215514 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kjf5\" (UniqueName: \"kubernetes.io/projected/48055fe8-0181-47f8-bdc6-da665d9badac-kube-api-access-5kjf5\") pod \"payload-processing-e2e-explicit-84e377-8468cbf5b9-wx4nd\" (UID: \"48055fe8-0181-47f8-bdc6-da665d9badac\") " pod="openshift-ingress/payload-processing-e2e-explicit-84e377-8468cbf5b9-wx4nd" Nov 03 01:07:14.218405 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:14.217986 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48055fe8-0181-47f8-bdc6-da665d9badac-plugins-config-volume\") pod \"payload-processing-e2e-explicit-84e377-8468cbf5b9-wx4nd\" (UID: \"48055fe8-0181-47f8-bdc6-da665d9badac\") " pod="openshift-ingress/payload-processing-e2e-explicit-84e377-8468cbf5b9-wx4nd" Nov 03 01:07:14.229341 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:14.229303 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78d5c907-e2f2-4755-bb98-0b08e6e8dd47-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-84e377-6bc89ccf99-pczgs\" (UID: \"78d5c907-e2f2-4755-bb98-0b08e6e8dd47\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-84e377-6bc89ccf99-pczgs" Nov 03 01:07:14.251156 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:14.251117 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nc7sn\" (UniqueName: \"kubernetes.io/projected/78d5c907-e2f2-4755-bb98-0b08e6e8dd47-kube-api-access-nc7sn\") pod \"payload-pre-processing-e2e-explicit-84e377-6bc89ccf99-pczgs\" (UID: \"78d5c907-e2f2-4755-bb98-0b08e6e8dd47\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-84e377-6bc89ccf99-pczgs" Nov 03 01:07:14.258334 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:14.252148 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kjf5\" (UniqueName: \"kubernetes.io/projected/48055fe8-0181-47f8-bdc6-da665d9badac-kube-api-access-5kjf5\") pod \"payload-processing-e2e-explicit-84e377-8468cbf5b9-wx4nd\" (UID: \"48055fe8-0181-47f8-bdc6-da665d9badac\") " pod="openshift-ingress/payload-processing-e2e-explicit-84e377-8468cbf5b9-wx4nd" Nov 03 01:07:14.265486 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:14.265447 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2afb12c9-7c7a-4527-95c4-0b2febd22e90" path="/var/lib/kubelet/pods/2afb12c9-7c7a-4527-95c4-0b2febd22e90/volumes" Nov 03 01:07:14.403141 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:14.403046 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-84e377-8468cbf5b9-wx4nd" Nov 03 01:07:14.537513 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:14.535401 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-84e377-6bc89ccf99-pczgs" Nov 03 01:07:14.689567 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:07:14.689485 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 01:07:14.710970 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:14.710940 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-84e377-8468cbf5b9-wx4nd"] Nov 03 01:07:14.849533 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:07:14.849505 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78d5c907_e2f2_4755_bb98_0b08e6e8dd47.slice/crio-b2d8ec56aa25b149e4d24d8705db9ab2940a7fac8943ff2c5269807d6218ede6 WatchSource:0}: Error finding container b2d8ec56aa25b149e4d24d8705db9ab2940a7fac8943ff2c5269807d6218ede6: Status 404 returned error can't find the container with id b2d8ec56aa25b149e4d24d8705db9ab2940a7fac8943ff2c5269807d6218ede6 Nov 03 01:07:14.852561 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:14.852536 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-84e377-6bc89ccf99-pczgs"] Nov 03 01:07:15.266152 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:15.266098 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-84e377-6bc89ccf99-pczgs" event={"ID":"78d5c907-e2f2-4755-bb98-0b08e6e8dd47","Type":"ContainerStarted","Data":"2536a5d67726786f8bbfae719a81e856cf14d957344569a005f1d7fb213587b9"} Nov 03 01:07:15.266362 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:15.266168 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-84e377-6bc89ccf99-pczgs" event={"ID":"78d5c907-e2f2-4755-bb98-0b08e6e8dd47","Type":"ContainerStarted","Data":"b2d8ec56aa25b149e4d24d8705db9ab2940a7fac8943ff2c5269807d6218ede6"} Nov 03 01:07:15.266362 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:15.266199 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-84e377-6bc89ccf99-pczgs" Nov 03 01:07:15.268054 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:15.268021 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-84e377-8468cbf5b9-wx4nd" event={"ID":"48055fe8-0181-47f8-bdc6-da665d9badac","Type":"ContainerStarted","Data":"86055a4842f72d08f0a2a93b1cfb937f92c7fc7179a83129f69eb5b3202c660f"} Nov 03 01:07:15.268198 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:15.268079 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-84e377-8468cbf5b9-wx4nd" event={"ID":"48055fe8-0181-47f8-bdc6-da665d9badac","Type":"ContainerStarted","Data":"3916d33ffbc540b110242ab2e55644094fbcd8e82120cde35f2c5447f1e84d8e"} Nov 03 01:07:15.268198 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:15.268148 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-84e377-8468cbf5b9-wx4nd" Nov 03 01:07:15.287752 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:15.287700 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-84e377-6bc89ccf99-pczgs" podStartSLOduration=2.287681632 podStartE2EDuration="2.287681632s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:07:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:07:15.282805215 +0000 UTC m=+1577.453808311" watchObservedRunningTime="2026-11-03 01:07:15.287681632 +0000 UTC m=+1577.458684729" Nov 03 01:07:15.304556 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:15.304499 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-84e377-8468cbf5b9-wx4nd" podStartSLOduration=2.304483961 podStartE2EDuration="2.304483961s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:07:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:07:15.298184394 +0000 UTC m=+1577.469187492" watchObservedRunningTime="2026-11-03 01:07:15.304483961 +0000 UTC m=+1577.475487056" Nov 03 01:07:18.591973 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:18.591931 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-78zwt"] Nov 03 01:07:18.598090 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:18.598061 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-78zwt" Nov 03 01:07:18.700052 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:18.698708 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f5666ca-5e70-47c9-84d1-20eb2475d196-istiod-ca-cert\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-78zwt\" (UID: \"8f5666ca-5e70-47c9-84d1-20eb2475d196\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-78zwt" Nov 03 01:07:18.700052 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:18.698794 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vr6z4\" (UniqueName: \"kubernetes.io/projected/8f5666ca-5e70-47c9-84d1-20eb2475d196-kube-api-access-vr6z4\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-78zwt\" (UID: \"8f5666ca-5e70-47c9-84d1-20eb2475d196\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-78zwt" Nov 03 01:07:18.700052 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:18.698842 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f5666ca-5e70-47c9-84d1-20eb2475d196-credential-socket\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-78zwt\" (UID: \"8f5666ca-5e70-47c9-84d1-20eb2475d196\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-78zwt" Nov 03 01:07:18.700052 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:18.698887 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f5666ca-5e70-47c9-84d1-20eb2475d196-istio-podinfo\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-78zwt\" (UID: \"8f5666ca-5e70-47c9-84d1-20eb2475d196\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-78zwt" Nov 03 01:07:18.700052 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:18.698925 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f5666ca-5e70-47c9-84d1-20eb2475d196-istio-data\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-78zwt\" (UID: \"8f5666ca-5e70-47c9-84d1-20eb2475d196\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-78zwt" Nov 03 01:07:18.700052 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:18.698959 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f5666ca-5e70-47c9-84d1-20eb2475d196-workload-certs\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-78zwt\" (UID: \"8f5666ca-5e70-47c9-84d1-20eb2475d196\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-78zwt" Nov 03 01:07:18.700052 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:18.699019 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f5666ca-5e70-47c9-84d1-20eb2475d196-workload-socket\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-78zwt\" (UID: \"8f5666ca-5e70-47c9-84d1-20eb2475d196\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-78zwt" Nov 03 01:07:18.700052 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:18.699184 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f5666ca-5e70-47c9-84d1-20eb2475d196-istio-token\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-78zwt\" (UID: \"8f5666ca-5e70-47c9-84d1-20eb2475d196\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-78zwt" Nov 03 01:07:18.700052 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:18.699293 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f5666ca-5e70-47c9-84d1-20eb2475d196-istio-envoy\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-78zwt\" (UID: \"8f5666ca-5e70-47c9-84d1-20eb2475d196\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-78zwt" Nov 03 01:07:18.718642 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:18.718557 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-78zwt"] Nov 03 01:07:18.800824 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:18.800787 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f5666ca-5e70-47c9-84d1-20eb2475d196-credential-socket\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-78zwt\" (UID: \"8f5666ca-5e70-47c9-84d1-20eb2475d196\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-78zwt" Nov 03 01:07:18.801032 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:18.800835 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f5666ca-5e70-47c9-84d1-20eb2475d196-istio-podinfo\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-78zwt\" (UID: \"8f5666ca-5e70-47c9-84d1-20eb2475d196\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-78zwt" Nov 03 01:07:18.801032 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:18.800871 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f5666ca-5e70-47c9-84d1-20eb2475d196-istio-data\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-78zwt\" (UID: \"8f5666ca-5e70-47c9-84d1-20eb2475d196\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-78zwt" Nov 03 01:07:18.801032 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:18.800893 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f5666ca-5e70-47c9-84d1-20eb2475d196-workload-certs\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-78zwt\" (UID: \"8f5666ca-5e70-47c9-84d1-20eb2475d196\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-78zwt" Nov 03 01:07:18.801032 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:18.800939 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f5666ca-5e70-47c9-84d1-20eb2475d196-workload-socket\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-78zwt\" (UID: \"8f5666ca-5e70-47c9-84d1-20eb2475d196\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-78zwt" Nov 03 01:07:18.801267 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:18.801106 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f5666ca-5e70-47c9-84d1-20eb2475d196-istio-token\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-78zwt\" (UID: \"8f5666ca-5e70-47c9-84d1-20eb2475d196\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-78zwt" Nov 03 01:07:18.801267 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:18.801213 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f5666ca-5e70-47c9-84d1-20eb2475d196-istio-envoy\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-78zwt\" (UID: \"8f5666ca-5e70-47c9-84d1-20eb2475d196\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-78zwt" Nov 03 01:07:18.801407 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:18.801270 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f5666ca-5e70-47c9-84d1-20eb2475d196-credential-socket\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-78zwt\" (UID: \"8f5666ca-5e70-47c9-84d1-20eb2475d196\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-78zwt" Nov 03 01:07:18.801407 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:18.801366 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f5666ca-5e70-47c9-84d1-20eb2475d196-istio-data\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-78zwt\" (UID: \"8f5666ca-5e70-47c9-84d1-20eb2475d196\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-78zwt" Nov 03 01:07:18.801569 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:18.801409 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f5666ca-5e70-47c9-84d1-20eb2475d196-istiod-ca-cert\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-78zwt\" (UID: \"8f5666ca-5e70-47c9-84d1-20eb2475d196\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-78zwt" Nov 03 01:07:18.801569 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:18.801442 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f5666ca-5e70-47c9-84d1-20eb2475d196-workload-certs\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-78zwt\" (UID: \"8f5666ca-5e70-47c9-84d1-20eb2475d196\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-78zwt" Nov 03 01:07:18.801569 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:18.801478 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f5666ca-5e70-47c9-84d1-20eb2475d196-workload-socket\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-78zwt\" (UID: \"8f5666ca-5e70-47c9-84d1-20eb2475d196\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-78zwt" Nov 03 01:07:18.801569 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:18.801508 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vr6z4\" (UniqueName: \"kubernetes.io/projected/8f5666ca-5e70-47c9-84d1-20eb2475d196-kube-api-access-vr6z4\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-78zwt\" (UID: \"8f5666ca-5e70-47c9-84d1-20eb2475d196\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-78zwt" Nov 03 01:07:18.802046 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:18.802023 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f5666ca-5e70-47c9-84d1-20eb2475d196-istiod-ca-cert\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-78zwt\" (UID: \"8f5666ca-5e70-47c9-84d1-20eb2475d196\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-78zwt" Nov 03 01:07:18.803445 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:18.803422 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f5666ca-5e70-47c9-84d1-20eb2475d196-istio-envoy\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-78zwt\" (UID: \"8f5666ca-5e70-47c9-84d1-20eb2475d196\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-78zwt" Nov 03 01:07:18.803540 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:18.803521 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f5666ca-5e70-47c9-84d1-20eb2475d196-istio-podinfo\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-78zwt\" (UID: \"8f5666ca-5e70-47c9-84d1-20eb2475d196\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-78zwt" Nov 03 01:07:18.809514 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:18.809487 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f5666ca-5e70-47c9-84d1-20eb2475d196-istio-token\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-78zwt\" (UID: \"8f5666ca-5e70-47c9-84d1-20eb2475d196\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-78zwt" Nov 03 01:07:18.809818 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:18.809799 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vr6z4\" (UniqueName: \"kubernetes.io/projected/8f5666ca-5e70-47c9-84d1-20eb2475d196-kube-api-access-vr6z4\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-78zwt\" (UID: \"8f5666ca-5e70-47c9-84d1-20eb2475d196\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-78zwt" Nov 03 01:07:19.023876 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:19.023842 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-78zwt" Nov 03 01:07:19.313669 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:07:19.313634 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f5666ca_5e70_47c9_84d1_20eb2475d196.slice/crio-0f872a01dfc55ba8345f52c52b0606554037e4fb771a4e18dd89bf6a4033e5bf WatchSource:0}: Error finding container 0f872a01dfc55ba8345f52c52b0606554037e4fb771a4e18dd89bf6a4033e5bf: Status 404 returned error can't find the container with id 0f872a01dfc55ba8345f52c52b0606554037e4fb771a4e18dd89bf6a4033e5bf Nov 03 01:07:19.315616 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:19.315469 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-78zwt"] Nov 03 01:07:19.316765 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:19.316725 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:07:19.316872 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:19.316799 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:07:19.316872 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:19.316829 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:07:20.340663 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:20.340622 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-78zwt" event={"ID":"8f5666ca-5e70-47c9-84d1-20eb2475d196","Type":"ContainerStarted","Data":"ceb17238f389cc4cbaf25cb81bb7734c5cd5e75cdc281ee51d8e6eced1a0af5e"} Nov 03 01:07:20.341093 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:20.340684 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-78zwt" event={"ID":"8f5666ca-5e70-47c9-84d1-20eb2475d196","Type":"ContainerStarted","Data":"0f872a01dfc55ba8345f52c52b0606554037e4fb771a4e18dd89bf6a4033e5bf"} Nov 03 01:07:20.366350 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:20.366287 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-78zwt" podStartSLOduration=2.366257391 podStartE2EDuration="2.366257391s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:07:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:07:20.36230653 +0000 UTC m=+1582.533309627" watchObservedRunningTime="2026-11-03 01:07:20.366257391 +0000 UTC m=+1582.537260486" Nov 03 01:07:21.030514 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:21.030473 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-78zwt" Nov 03 01:07:21.032466 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:21.032437 4421 patch_prober.go:29] interesting pod/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-78zwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.230:15021/healthz/ready\": dial tcp 10.134.0.230:15021: connect: connection refused" start-of-body= Nov 03 01:07:21.032587 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:21.032509 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-78zwt" podUID="8f5666ca-5e70-47c9-84d1-20eb2475d196" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.230:15021/healthz/ready\": dial tcp 10.134.0.230:15021: connect: connection refused" Nov 03 01:07:22.026266 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:22.026089 4421 patch_prober.go:29] interesting pod/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-78zwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.230:15021/healthz/ready\": dial tcp 10.134.0.230:15021: connect: connection refused" start-of-body= Nov 03 01:07:22.026266 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:22.026160 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-78zwt" podUID="8f5666ca-5e70-47c9-84d1-20eb2475d196" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.230:15021/healthz/ready\": dial tcp 10.134.0.230:15021: connect: connection refused" Nov 03 01:07:23.032294 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:23.031388 4421 patch_prober.go:29] interesting pod/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-78zwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.230:15021/healthz/ready\": dial tcp 10.134.0.230:15021: connect: connection refused" start-of-body= Nov 03 01:07:23.032294 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:23.031473 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-78zwt" podUID="8f5666ca-5e70-47c9-84d1-20eb2475d196" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.230:15021/healthz/ready\": dial tcp 10.134.0.230:15021: connect: connection refused" Nov 03 01:07:24.028299 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:24.027739 4421 patch_prober.go:29] interesting pod/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-78zwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.230:15021/healthz/ready\": dial tcp 10.134.0.230:15021: connect: connection refused" start-of-body= Nov 03 01:07:24.028299 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:24.027820 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-78zwt" podUID="8f5666ca-5e70-47c9-84d1-20eb2475d196" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.230:15021/healthz/ready\": dial tcp 10.134.0.230:15021: connect: connection refused" Nov 03 01:07:25.037301 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:25.027056 4421 patch_prober.go:29] interesting pod/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-78zwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.230:15021/healthz/ready\": dial tcp 10.134.0.230:15021: connect: connection refused" start-of-body= Nov 03 01:07:25.037301 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:25.027117 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-78zwt" podUID="8f5666ca-5e70-47c9-84d1-20eb2475d196" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.230:15021/healthz/ready\": dial tcp 10.134.0.230:15021: connect: connection refused" Nov 03 01:07:26.203455 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:26.203412 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-78zwt" Nov 03 01:07:26.205074 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:26.205038 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-78zwt" Nov 03 01:07:26.206221 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:26.206197 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-78zwt" Nov 03 01:07:26.282041 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:26.282011 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-84e377-8468cbf5b9-wx4nd" Nov 03 01:07:26.282361 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:26.282341 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-84e377-6bc89ccf99-pczgs" Nov 03 01:07:39.442997 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:39.442953 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-84e377/explicit-model-84e377-kserve-7458bff9f-7vtl7"] Nov 03 01:07:39.451676 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:39.451647 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-84e377/explicit-model-84e377-kserve-7458bff9f-7vtl7" Nov 03 01:07:39.459296 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:39.459247 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-84e377/explicit-model-84e377-kserve-7458bff9f-7vtl7"] Nov 03 01:07:39.461214 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:39.461188 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-84e377\"/\"kube-root-ca.crt\"" Nov 03 01:07:39.462104 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:39.462079 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-84e377\"/\"explicit-model-84e377-kserve-self-signed-certs\"" Nov 03 01:07:39.462221 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:39.462163 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-84e377\"/\"openshift-service-ca.crt\"" Nov 03 01:07:39.462325 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:39.462265 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-84e377\"/\"default-dockercfg-vvgc9\"" Nov 03 01:07:39.552264 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:39.552219 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbwb4\" (UniqueName: \"kubernetes.io/projected/96b6a080-1574-4a42-8197-dd1cb436757e-kube-api-access-cbwb4\") pod \"explicit-model-84e377-kserve-7458bff9f-7vtl7\" (UID: \"96b6a080-1574-4a42-8197-dd1cb436757e\") " pod="ai-tenant-e2e-explicit-84e377/explicit-model-84e377-kserve-7458bff9f-7vtl7" Nov 03 01:07:39.552498 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:39.552315 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96b6a080-1574-4a42-8197-dd1cb436757e-dshm\") pod \"explicit-model-84e377-kserve-7458bff9f-7vtl7\" (UID: \"96b6a080-1574-4a42-8197-dd1cb436757e\") " pod="ai-tenant-e2e-explicit-84e377/explicit-model-84e377-kserve-7458bff9f-7vtl7" Nov 03 01:07:39.552498 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:39.552426 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96b6a080-1574-4a42-8197-dd1cb436757e-model-cache\") pod \"explicit-model-84e377-kserve-7458bff9f-7vtl7\" (UID: \"96b6a080-1574-4a42-8197-dd1cb436757e\") " pod="ai-tenant-e2e-explicit-84e377/explicit-model-84e377-kserve-7458bff9f-7vtl7" Nov 03 01:07:39.552498 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:39.552480 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96b6a080-1574-4a42-8197-dd1cb436757e-tls-certs\") pod \"explicit-model-84e377-kserve-7458bff9f-7vtl7\" (UID: \"96b6a080-1574-4a42-8197-dd1cb436757e\") " pod="ai-tenant-e2e-explicit-84e377/explicit-model-84e377-kserve-7458bff9f-7vtl7" Nov 03 01:07:39.552695 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:39.552599 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96b6a080-1574-4a42-8197-dd1cb436757e-tmp-dir\") pod \"explicit-model-84e377-kserve-7458bff9f-7vtl7\" (UID: \"96b6a080-1574-4a42-8197-dd1cb436757e\") " pod="ai-tenant-e2e-explicit-84e377/explicit-model-84e377-kserve-7458bff9f-7vtl7" Nov 03 01:07:39.552695 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:39.552660 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96b6a080-1574-4a42-8197-dd1cb436757e-home\") pod \"explicit-model-84e377-kserve-7458bff9f-7vtl7\" (UID: \"96b6a080-1574-4a42-8197-dd1cb436757e\") " pod="ai-tenant-e2e-explicit-84e377/explicit-model-84e377-kserve-7458bff9f-7vtl7" Nov 03 01:07:39.655600 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:39.655554 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96b6a080-1574-4a42-8197-dd1cb436757e-dshm\") pod \"explicit-model-84e377-kserve-7458bff9f-7vtl7\" (UID: \"96b6a080-1574-4a42-8197-dd1cb436757e\") " pod="ai-tenant-e2e-explicit-84e377/explicit-model-84e377-kserve-7458bff9f-7vtl7" Nov 03 01:07:39.659299 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:39.655958 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96b6a080-1574-4a42-8197-dd1cb436757e-model-cache\") pod \"explicit-model-84e377-kserve-7458bff9f-7vtl7\" (UID: \"96b6a080-1574-4a42-8197-dd1cb436757e\") " pod="ai-tenant-e2e-explicit-84e377/explicit-model-84e377-kserve-7458bff9f-7vtl7" Nov 03 01:07:39.687541 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:39.687482 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96b6a080-1574-4a42-8197-dd1cb436757e-tls-certs\") pod \"explicit-model-84e377-kserve-7458bff9f-7vtl7\" (UID: \"96b6a080-1574-4a42-8197-dd1cb436757e\") " pod="ai-tenant-e2e-explicit-84e377/explicit-model-84e377-kserve-7458bff9f-7vtl7" Nov 03 01:07:39.687881 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:39.687859 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96b6a080-1574-4a42-8197-dd1cb436757e-tmp-dir\") pod \"explicit-model-84e377-kserve-7458bff9f-7vtl7\" (UID: \"96b6a080-1574-4a42-8197-dd1cb436757e\") " pod="ai-tenant-e2e-explicit-84e377/explicit-model-84e377-kserve-7458bff9f-7vtl7" Nov 03 01:07:39.688049 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:39.688032 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96b6a080-1574-4a42-8197-dd1cb436757e-home\") pod \"explicit-model-84e377-kserve-7458bff9f-7vtl7\" (UID: \"96b6a080-1574-4a42-8197-dd1cb436757e\") " pod="ai-tenant-e2e-explicit-84e377/explicit-model-84e377-kserve-7458bff9f-7vtl7" Nov 03 01:07:39.688465 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:39.688439 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbwb4\" (UniqueName: \"kubernetes.io/projected/96b6a080-1574-4a42-8197-dd1cb436757e-kube-api-access-cbwb4\") pod \"explicit-model-84e377-kserve-7458bff9f-7vtl7\" (UID: \"96b6a080-1574-4a42-8197-dd1cb436757e\") " pod="ai-tenant-e2e-explicit-84e377/explicit-model-84e377-kserve-7458bff9f-7vtl7" Nov 03 01:07:39.689683 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:39.662836 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96b6a080-1574-4a42-8197-dd1cb436757e-model-cache\") pod \"explicit-model-84e377-kserve-7458bff9f-7vtl7\" (UID: \"96b6a080-1574-4a42-8197-dd1cb436757e\") " pod="ai-tenant-e2e-explicit-84e377/explicit-model-84e377-kserve-7458bff9f-7vtl7" Nov 03 01:07:39.689683 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:39.668404 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96b6a080-1574-4a42-8197-dd1cb436757e-dshm\") pod \"explicit-model-84e377-kserve-7458bff9f-7vtl7\" (UID: \"96b6a080-1574-4a42-8197-dd1cb436757e\") " pod="ai-tenant-e2e-explicit-84e377/explicit-model-84e377-kserve-7458bff9f-7vtl7" Nov 03 01:07:39.689887 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:39.689817 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96b6a080-1574-4a42-8197-dd1cb436757e-tmp-dir\") pod \"explicit-model-84e377-kserve-7458bff9f-7vtl7\" (UID: \"96b6a080-1574-4a42-8197-dd1cb436757e\") " pod="ai-tenant-e2e-explicit-84e377/explicit-model-84e377-kserve-7458bff9f-7vtl7" Nov 03 01:07:39.690287 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:39.690246 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96b6a080-1574-4a42-8197-dd1cb436757e-home\") pod \"explicit-model-84e377-kserve-7458bff9f-7vtl7\" (UID: \"96b6a080-1574-4a42-8197-dd1cb436757e\") " pod="ai-tenant-e2e-explicit-84e377/explicit-model-84e377-kserve-7458bff9f-7vtl7" Nov 03 01:07:39.703598 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:39.703493 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96b6a080-1574-4a42-8197-dd1cb436757e-tls-certs\") pod \"explicit-model-84e377-kserve-7458bff9f-7vtl7\" (UID: \"96b6a080-1574-4a42-8197-dd1cb436757e\") " pod="ai-tenant-e2e-explicit-84e377/explicit-model-84e377-kserve-7458bff9f-7vtl7" Nov 03 01:07:39.749456 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:39.749411 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbwb4\" (UniqueName: \"kubernetes.io/projected/96b6a080-1574-4a42-8197-dd1cb436757e-kube-api-access-cbwb4\") pod \"explicit-model-84e377-kserve-7458bff9f-7vtl7\" (UID: \"96b6a080-1574-4a42-8197-dd1cb436757e\") " pod="ai-tenant-e2e-explicit-84e377/explicit-model-84e377-kserve-7458bff9f-7vtl7" Nov 03 01:07:39.779956 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:39.779627 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-84e377/explicit-model-84e377-kserve-7458bff9f-7vtl7" Nov 03 01:07:40.201960 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:07:40.201924 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod96b6a080_1574_4a42_8197_dd1cb436757e.slice/crio-68dc56786127825c2014483b81aefcd4ab87b8fb8b130d69721b76434d80aea6 WatchSource:0}: Error finding container 68dc56786127825c2014483b81aefcd4ab87b8fb8b130d69721b76434d80aea6: Status 404 returned error can't find the container with id 68dc56786127825c2014483b81aefcd4ab87b8fb8b130d69721b76434d80aea6 Nov 03 01:07:40.207611 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:40.207586 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-84e377/explicit-model-84e377-kserve-7458bff9f-7vtl7"] Nov 03 01:07:40.742214 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:40.742178 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-84e377/explicit-model-84e377-kserve-7458bff9f-7vtl7" event={"ID":"96b6a080-1574-4a42-8197-dd1cb436757e","Type":"ContainerStarted","Data":"e37ed6e38fc2d6173f17c75dc06b07baaeaf03510c6eaea79d11fa5ebef9d79f"} Nov 03 01:07:40.742699 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:40.742225 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-84e377/explicit-model-84e377-kserve-7458bff9f-7vtl7" event={"ID":"96b6a080-1574-4a42-8197-dd1cb436757e","Type":"ContainerStarted","Data":"68dc56786127825c2014483b81aefcd4ab87b8fb8b130d69721b76434d80aea6"} Nov 03 01:07:40.761250 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:40.761192 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-84e377/explicit-model-84e377-kserve-7458bff9f-7vtl7" podStartSLOduration=1.761177686 podStartE2EDuration="1.761177686s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:07:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:07:40.760029316 +0000 UTC m=+1602.931032413" watchObservedRunningTime="2026-11-03 01:07:40.761177686 +0000 UTC m=+1602.932180835" Nov 03 01:07:40.781149 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:40.781106 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-84e377/explicit-model-84e377-kserve-7458bff9f-7vtl7" Nov 03 01:07:41.028659 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:41.028619 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28b319b3-ch7k5"] Nov 03 01:07:41.034432 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:41.034400 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28b319b3-ch7k5"] Nov 03 01:07:41.984950 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:41.984902 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e13eb472-c996-46dd-b804-774416975943" path="/var/lib/kubelet/pods/e13eb472-c996-46dd-b804-774416975943/volumes" Nov 03 01:07:44.671516 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:07:44.671470 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 01:07:46.807495 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:46.807455 4421 generic.go:356] "Generic (PLEG): container finished" podID="894c4be7-9b51-47c4-87d7-05eee1e8bcbc" containerID="222e52cd77cb070a4e69d96d60d2a4c50bebd5e667232db677487b8cf0b86476" exitCode=0 Nov 03 01:07:46.807931 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:46.807530 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-x67zh" event={"ID":"894c4be7-9b51-47c4-87d7-05eee1e8bcbc","Type":"ContainerDied","Data":"222e52cd77cb070a4e69d96d60d2a4c50bebd5e667232db677487b8cf0b86476"} Nov 03 01:07:46.808044 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:46.808028 4421 scope.go:122] "RemoveContainer" containerID="222e52cd77cb070a4e69d96d60d2a4c50bebd5e667232db677487b8cf0b86476" Nov 03 01:07:46.809846 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:46.809815 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:07:46.809927 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:46.809895 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:07:46.809980 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:46.809927 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:07:47.893617 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:47.893580 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-x67zh" event={"ID":"894c4be7-9b51-47c4-87d7-05eee1e8bcbc","Type":"ContainerStarted","Data":"ff2d9570d49bca7bc41207d1b8e57265a35a82a4f07644979b5ccf8083a18db8"} Nov 03 01:07:48.825699 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:48.825650 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-x67zh" Nov 03 01:07:48.831043 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:48.831000 4421 patch_prober.go:29] interesting pod/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-x67zh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused" start-of-body= Nov 03 01:07:48.831265 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:48.831093 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-x67zh" podUID="894c4be7-9b51-47c4-87d7-05eee1e8bcbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused" Nov 03 01:07:49.781160 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:49.781116 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-84e377/explicit-model-84e377-kserve-7458bff9f-7vtl7" Nov 03 01:07:49.796253 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:49.796226 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-84e377/explicit-model-84e377-kserve-7458bff9f-7vtl7" Nov 03 01:07:49.838650 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:49.836443 4421 patch_prober.go:29] interesting pod/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-x67zh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused" start-of-body= Nov 03 01:07:49.838650 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:49.836518 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-x67zh" podUID="894c4be7-9b51-47c4-87d7-05eee1e8bcbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused" Nov 03 01:07:49.956068 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:49.956038 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-84e377/explicit-model-84e377-kserve-7458bff9f-7vtl7" Nov 03 01:07:50.056391 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:50.054966 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-84e377/explicit-model-84e377-kserve-7458bff9f-7vtl7"] Nov 03 01:07:50.079772 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:07:50.078998 4421 secret.go:189] Couldn't get secret ai-tenant-e2e-explicit-84e377/explicit-model-84e377-kserve-self-signed-certs: secret "explicit-model-84e377-kserve-self-signed-certs" not found Nov 03 01:07:50.079772 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:07:50.079077 4421 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96b6a080-1574-4a42-8197-dd1cb436757e-tls-certs podName:96b6a080-1574-4a42-8197-dd1cb436757e nodeName:}" failed. No retries permitted until 2026-11-03 01:07:50.579054693 +0000 UTC m=+1612.750057782 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/96b6a080-1574-4a42-8197-dd1cb436757e-tls-certs") pod "explicit-model-84e377-kserve-7458bff9f-7vtl7" (UID: "96b6a080-1574-4a42-8197-dd1cb436757e") : secret "explicit-model-84e377-kserve-self-signed-certs" not found Nov 03 01:07:50.716301 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:07:50.679678 4421 secret.go:189] Couldn't get secret ai-tenant-e2e-explicit-84e377/explicit-model-84e377-kserve-self-signed-certs: secret "explicit-model-84e377-kserve-self-signed-certs" not found Nov 03 01:07:50.716301 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:07:50.701389 4421 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96b6a080-1574-4a42-8197-dd1cb436757e-tls-certs podName:96b6a080-1574-4a42-8197-dd1cb436757e nodeName:}" failed. No retries permitted until 2026-11-03 01:07:51.701359017 +0000 UTC m=+1613.872362111 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/96b6a080-1574-4a42-8197-dd1cb436757e-tls-certs") pod "explicit-model-84e377-kserve-7458bff9f-7vtl7" (UID: "96b6a080-1574-4a42-8197-dd1cb436757e") : secret "explicit-model-84e377-kserve-self-signed-certs" not found Nov 03 01:07:50.854349 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:50.836292 4421 patch_prober.go:29] interesting pod/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-x67zh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused" start-of-body= Nov 03 01:07:50.976495 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:50.976391 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-x67zh" podUID="894c4be7-9b51-47c4-87d7-05eee1e8bcbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused" Nov 03 01:07:51.186454 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:51.186397 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/simulator-w2-f7680d-kserve-85bdc86cd6-8gxrg" podUID="2bdcf521-cebd-4d6b-8554-1b60db8b0f3c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:07:51.350431 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:51.350372 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-650be0/premium-w4-650be0-kserve-7bf8555dfb-z97n5" podUID="40601dc5-808f-456f-826f-6e0c35966ada" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:07:51.354433 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:51.354392 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/premium-w3-83755a-kserve-7c9d66988b-lt4xd" podUID="c0c6b02e-2ecb-428e-a13d-be6d7cd1ba39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:07:51.579300 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:51.575939 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/distinct-w3-83755a-kserve-86bdbbdd85-2cnnl" podUID="f4ae9624-d127-4a55-837d-7c9ff0267b37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:07:51.737680 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:07:51.737582 4421 secret.go:189] Couldn't get secret ai-tenant-e2e-explicit-84e377/explicit-model-84e377-kserve-self-signed-certs: secret "explicit-model-84e377-kserve-self-signed-certs" not found Nov 03 01:07:51.738096 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:07:51.738076 4421 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96b6a080-1574-4a42-8197-dd1cb436757e-tls-certs podName:96b6a080-1574-4a42-8197-dd1cb436757e nodeName:}" failed. No retries permitted until 2026-11-03 01:07:53.738051072 +0000 UTC m=+1615.909054164 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/96b6a080-1574-4a42-8197-dd1cb436757e-tls-certs") pod "explicit-model-84e377-kserve-7458bff9f-7vtl7" (UID: "96b6a080-1574-4a42-8197-dd1cb436757e") : secret "explicit-model-84e377-kserve-self-signed-certs" not found Nov 03 01:07:51.827236 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:51.826752 4421 patch_prober.go:29] interesting pod/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-x67zh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused" start-of-body= Nov 03 01:07:51.827236 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:51.826819 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-x67zh" podUID="894c4be7-9b51-47c4-87d7-05eee1e8bcbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused" Nov 03 01:07:51.957359 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:51.956167 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-explicit-84e377/explicit-model-84e377-kserve-7458bff9f-7vtl7" podUID="96b6a080-1574-4a42-8197-dd1cb436757e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.231:8000/ready\": context deadline exceeded" Nov 03 01:07:51.974513 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:51.968620 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-x67zh" Nov 03 01:07:52.830486 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:52.830447 4421 patch_prober.go:29] interesting pod/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-x67zh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused" start-of-body= Nov 03 01:07:52.830807 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:52.830774 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-x67zh" podUID="894c4be7-9b51-47c4-87d7-05eee1e8bcbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused" Nov 03 01:07:53.794812 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:07:53.794772 4421 secret.go:189] Couldn't get secret ai-tenant-e2e-explicit-84e377/explicit-model-84e377-kserve-self-signed-certs: secret "explicit-model-84e377-kserve-self-signed-certs" not found Nov 03 01:07:53.795242 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:07:53.794855 4421 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96b6a080-1574-4a42-8197-dd1cb436757e-tls-certs podName:96b6a080-1574-4a42-8197-dd1cb436757e nodeName:}" failed. No retries permitted until 2026-11-03 01:07:57.79484035 +0000 UTC m=+1619.965843425 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/96b6a080-1574-4a42-8197-dd1cb436757e-tls-certs") pod "explicit-model-84e377-kserve-7458bff9f-7vtl7" (UID: "96b6a080-1574-4a42-8197-dd1cb436757e") : secret "explicit-model-84e377-kserve-self-signed-certs" not found Nov 03 01:07:53.827762 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:53.827719 4421 patch_prober.go:29] interesting pod/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-x67zh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused" start-of-body= Nov 03 01:07:53.828057 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:53.828027 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-x67zh" podUID="894c4be7-9b51-47c4-87d7-05eee1e8bcbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused" Nov 03 01:07:54.986038 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:54.986007 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-x67zh" Nov 03 01:07:54.987600 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:07:54.987575 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-x67zh" Nov 03 01:07:57.860711 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:07:57.860611 4421 secret.go:189] Couldn't get secret ai-tenant-e2e-explicit-84e377/explicit-model-84e377-kserve-self-signed-certs: secret "explicit-model-84e377-kserve-self-signed-certs" not found Nov 03 01:07:57.860711 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:07:57.860697 4421 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96b6a080-1574-4a42-8197-dd1cb436757e-tls-certs podName:96b6a080-1574-4a42-8197-dd1cb436757e nodeName:}" failed. No retries permitted until 2026-11-03 01:08:05.860679378 +0000 UTC m=+1628.031682469 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/96b6a080-1574-4a42-8197-dd1cb436757e-tls-certs") pod "explicit-model-84e377-kserve-7458bff9f-7vtl7" (UID: "96b6a080-1574-4a42-8197-dd1cb436757e") : secret "explicit-model-84e377-kserve-self-signed-certs" not found Nov 03 01:08:05.957572 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:08:05.957535 4421 secret.go:189] Couldn't get secret ai-tenant-e2e-explicit-84e377/explicit-model-84e377-kserve-self-signed-certs: secret "explicit-model-84e377-kserve-self-signed-certs" not found Nov 03 01:08:05.958138 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:08:05.957638 4421 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96b6a080-1574-4a42-8197-dd1cb436757e-tls-certs podName:96b6a080-1574-4a42-8197-dd1cb436757e nodeName:}" failed. No retries permitted until 2026-11-03 01:08:21.957616917 +0000 UTC m=+1644.128619992 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/96b6a080-1574-4a42-8197-dd1cb436757e-tls-certs") pod "explicit-model-84e377-kserve-7458bff9f-7vtl7" (UID: "96b6a080-1574-4a42-8197-dd1cb436757e") : secret "explicit-model-84e377-kserve-self-signed-certs" not found Nov 03 01:08:06.394701 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:06.394666 4421 generic.go:356] "Generic (PLEG): container finished" podID="b9815803-c1dc-4c21-a8f3-2a3c9c89f1b9" containerID="8741545e1131e3da6e37b3db71aff7890695839c68673721250d33ee4b513d48" exitCode=0 Nov 03 01:08:06.394897 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:06.394737 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-84e377-openshift-default-5554bf8669-lzzbj" event={"ID":"b9815803-c1dc-4c21-a8f3-2a3c9c89f1b9","Type":"ContainerDied","Data":"8741545e1131e3da6e37b3db71aff7890695839c68673721250d33ee4b513d48"} Nov 03 01:08:06.395261 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:06.395240 4421 scope.go:122] "RemoveContainer" containerID="8741545e1131e3da6e37b3db71aff7890695839c68673721250d33ee4b513d48" Nov 03 01:08:06.397320 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:06.397251 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:08:06.397420 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:06.397351 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:08:06.397420 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:06.397390 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:08:07.364858 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:07.364807 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-84e377/explicit-model-84e377-kserve-7458bff9f-7vtl7" podUID="96b6a080-1574-4a42-8197-dd1cb436757e" containerName="main" containerID="cri-o://e37ed6e38fc2d6173f17c75dc06b07baaeaf03510c6eaea79d11fa5ebef9d79f" gracePeriod=45 Nov 03 01:08:07.432014 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:07.431974 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-84e377-openshift-default-5554bf8669-lzzbj" event={"ID":"b9815803-c1dc-4c21-a8f3-2a3c9c89f1b9","Type":"ContainerStarted","Data":"f68265b88dc94bbd4cf12137e75b6bb02ae75621d632862f27f97e292afeb39f"} Nov 03 01:08:07.927298 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:07.925049 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-84e377-openshift-default-5554bf8669-lzzbj" Nov 03 01:08:07.932484 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:07.932449 4421 patch_prober.go:29] interesting pod/e2e-explicit-84e377-openshift-default-5554bf8669-lzzbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" start-of-body= Nov 03 01:08:07.932668 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:07.932508 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-84e377-openshift-default-5554bf8669-lzzbj" podUID="b9815803-c1dc-4c21-a8f3-2a3c9c89f1b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" Nov 03 01:08:07.940330 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:07.940297 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-explicit-84e377/explicit-model-84e377-kserve-7458bff9f-7vtl7" podUID="96b6a080-1574-4a42-8197-dd1cb436757e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.231:8000/ready\": dial tcp 10.134.0.231:8000: connect: connection refused" Nov 03 01:08:08.180105 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:08.180041 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-84e377/explicit-model-84e377-kserve-7458bff9f-7vtl7" Nov 03 01:08:08.357301 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:08.356671 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96b6a080-1574-4a42-8197-dd1cb436757e-tmp-dir" pod "96b6a080-1574-4a42-8197-dd1cb436757e" (UID: "96b6a080-1574-4a42-8197-dd1cb436757e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:08:08.366342 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:08.358979 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96b6a080-1574-4a42-8197-dd1cb436757e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96b6a080-1574-4a42-8197-dd1cb436757e-tmp-dir\") pod \"96b6a080-1574-4a42-8197-dd1cb436757e\" (UID: \"96b6a080-1574-4a42-8197-dd1cb436757e\") " Nov 03 01:08:08.366342 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:08.359146 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96b6a080-1574-4a42-8197-dd1cb436757e-home\" (UniqueName: \"kubernetes.io/empty-dir/96b6a080-1574-4a42-8197-dd1cb436757e-home\") pod \"96b6a080-1574-4a42-8197-dd1cb436757e\" (UID: \"96b6a080-1574-4a42-8197-dd1cb436757e\") " Nov 03 01:08:08.366342 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:08.359365 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/96b6a080-1574-4a42-8197-dd1cb436757e-tls-certs\" (UniqueName: \"kubernetes.io/secret/96b6a080-1574-4a42-8197-dd1cb436757e-tls-certs\") pod \"96b6a080-1574-4a42-8197-dd1cb436757e\" (UID: \"96b6a080-1574-4a42-8197-dd1cb436757e\") " Nov 03 01:08:08.366342 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:08.359563 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96b6a080-1574-4a42-8197-dd1cb436757e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/96b6a080-1574-4a42-8197-dd1cb436757e-dshm\") pod \"96b6a080-1574-4a42-8197-dd1cb436757e\" (UID: \"96b6a080-1574-4a42-8197-dd1cb436757e\") " Nov 03 01:08:08.366342 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:08.359890 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96b6a080-1574-4a42-8197-dd1cb436757e-kube-api-access-cbwb4\" (UniqueName: \"kubernetes.io/projected/96b6a080-1574-4a42-8197-dd1cb436757e-kube-api-access-cbwb4\") pod \"96b6a080-1574-4a42-8197-dd1cb436757e\" (UID: \"96b6a080-1574-4a42-8197-dd1cb436757e\") " Nov 03 01:08:08.366342 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:08.359981 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96b6a080-1574-4a42-8197-dd1cb436757e-home" pod "96b6a080-1574-4a42-8197-dd1cb436757e" (UID: "96b6a080-1574-4a42-8197-dd1cb436757e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:08:08.366342 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:08.360217 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96b6a080-1574-4a42-8197-dd1cb436757e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96b6a080-1574-4a42-8197-dd1cb436757e-model-cache\") pod \"96b6a080-1574-4a42-8197-dd1cb436757e\" (UID: \"96b6a080-1574-4a42-8197-dd1cb436757e\") " Nov 03 01:08:08.366342 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:08.362246 4421 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96b6a080-1574-4a42-8197-dd1cb436757e-tmp-dir\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:08:08.366342 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:08.362288 4421 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96b6a080-1574-4a42-8197-dd1cb436757e-home\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:08:08.366342 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:08.362621 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96b6a080-1574-4a42-8197-dd1cb436757e-model-cache" pod "96b6a080-1574-4a42-8197-dd1cb436757e" (UID: "96b6a080-1574-4a42-8197-dd1cb436757e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:08:08.378973 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:08.378774 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96b6a080-1574-4a42-8197-dd1cb436757e-dshm" pod "96b6a080-1574-4a42-8197-dd1cb436757e" (UID: "96b6a080-1574-4a42-8197-dd1cb436757e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:08:08.378973 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:08.378882 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96b6a080-1574-4a42-8197-dd1cb436757e-kube-api-access-cbwb4" pod "96b6a080-1574-4a42-8197-dd1cb436757e" (UID: "96b6a080-1574-4a42-8197-dd1cb436757e"). InnerVolumeSpecName "kube-api-access-cbwb4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:08:08.379405 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:08.379382 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96b6a080-1574-4a42-8197-dd1cb436757e-tls-certs" pod "96b6a080-1574-4a42-8197-dd1cb436757e" (UID: "96b6a080-1574-4a42-8197-dd1cb436757e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:08:08.446339 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:08.446225 4421 generic.go:356] "Generic (PLEG): container finished" podID="96b6a080-1574-4a42-8197-dd1cb436757e" containerID="e37ed6e38fc2d6173f17c75dc06b07baaeaf03510c6eaea79d11fa5ebef9d79f" exitCode=0 Nov 03 01:08:08.447352 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:08.447336 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-84e377/explicit-model-84e377-kserve-7458bff9f-7vtl7" Nov 03 01:08:08.449426 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:08.449399 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-84e377/explicit-model-84e377-kserve-7458bff9f-7vtl7" event={"ID":"96b6a080-1574-4a42-8197-dd1cb436757e","Type":"ContainerDied","Data":"e37ed6e38fc2d6173f17c75dc06b07baaeaf03510c6eaea79d11fa5ebef9d79f"} Nov 03 01:08:08.449514 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:08.449443 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-84e377/explicit-model-84e377-kserve-7458bff9f-7vtl7" event={"ID":"96b6a080-1574-4a42-8197-dd1cb436757e","Type":"ContainerDied","Data":"68dc56786127825c2014483b81aefcd4ab87b8fb8b130d69721b76434d80aea6"} Nov 03 01:08:08.449514 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:08.449460 4421 scope.go:122] "RemoveContainer" containerID="e37ed6e38fc2d6173f17c75dc06b07baaeaf03510c6eaea79d11fa5ebef9d79f" Nov 03 01:08:08.462302 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:08.462263 4421 scope.go:122] "RemoveContainer" containerID="e37ed6e38fc2d6173f17c75dc06b07baaeaf03510c6eaea79d11fa5ebef9d79f" Nov 03 01:08:08.462645 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:08:08.462623 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e37ed6e38fc2d6173f17c75dc06b07baaeaf03510c6eaea79d11fa5ebef9d79f\": container with ID starting with e37ed6e38fc2d6173f17c75dc06b07baaeaf03510c6eaea79d11fa5ebef9d79f not found: ID does not exist" containerID="e37ed6e38fc2d6173f17c75dc06b07baaeaf03510c6eaea79d11fa5ebef9d79f" Nov 03 01:08:08.462712 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:08.462653 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e37ed6e38fc2d6173f17c75dc06b07baaeaf03510c6eaea79d11fa5ebef9d79f"} err="failed to get container status \"e37ed6e38fc2d6173f17c75dc06b07baaeaf03510c6eaea79d11fa5ebef9d79f\": rpc error: code = NotFound desc = could not find container \"e37ed6e38fc2d6173f17c75dc06b07baaeaf03510c6eaea79d11fa5ebef9d79f\": container with ID starting with e37ed6e38fc2d6173f17c75dc06b07baaeaf03510c6eaea79d11fa5ebef9d79f not found: ID does not exist" Nov 03 01:08:08.463640 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:08.463618 4421 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96b6a080-1574-4a42-8197-dd1cb436757e-model-cache\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:08:08.463727 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:08.463643 4421 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96b6a080-1574-4a42-8197-dd1cb436757e-tls-certs\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:08:08.463727 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:08.463653 4421 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96b6a080-1574-4a42-8197-dd1cb436757e-dshm\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:08:08.463727 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:08.463663 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbwb4\" (UniqueName: \"kubernetes.io/projected/96b6a080-1574-4a42-8197-dd1cb436757e-kube-api-access-cbwb4\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:08:08.474876 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:08.474836 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-84e377/explicit-model-84e377-kserve-7458bff9f-7vtl7"] Nov 03 01:08:08.482726 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:08.482694 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-84e377/explicit-model-84e377-kserve-7458bff9f-7vtl7"] Nov 03 01:08:08.925300 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:08.924767 4421 patch_prober.go:29] interesting pod/e2e-explicit-84e377-openshift-default-5554bf8669-lzzbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" start-of-body= Nov 03 01:08:08.925300 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:08.924842 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-84e377-openshift-default-5554bf8669-lzzbj" podUID="b9815803-c1dc-4c21-a8f3-2a3c9c89f1b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" Nov 03 01:08:09.930529 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:09.930412 4421 patch_prober.go:29] interesting pod/e2e-explicit-84e377-openshift-default-5554bf8669-lzzbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" start-of-body= Nov 03 01:08:09.930529 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:09.930497 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-84e377-openshift-default-5554bf8669-lzzbj" podUID="b9815803-c1dc-4c21-a8f3-2a3c9c89f1b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" Nov 03 01:08:09.982864 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:09.982826 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96b6a080-1574-4a42-8197-dd1cb436757e" path="/var/lib/kubelet/pods/96b6a080-1574-4a42-8197-dd1cb436757e/volumes" Nov 03 01:08:10.930850 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:10.930807 4421 patch_prober.go:29] interesting pod/e2e-explicit-84e377-openshift-default-5554bf8669-lzzbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" start-of-body= Nov 03 01:08:10.931465 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:10.930879 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-84e377-openshift-default-5554bf8669-lzzbj" podUID="b9815803-c1dc-4c21-a8f3-2a3c9c89f1b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" Nov 03 01:08:11.845701 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:11.845661 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-84e377-openshift-default-5554bf8669-lzzbj" Nov 03 01:08:11.923792 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:11.923753 4421 patch_prober.go:29] interesting pod/e2e-explicit-84e377-openshift-default-5554bf8669-lzzbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" start-of-body= Nov 03 01:08:11.924107 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:11.924069 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-84e377-openshift-default-5554bf8669-lzzbj" podUID="b9815803-c1dc-4c21-a8f3-2a3c9c89f1b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" Nov 03 01:08:12.927727 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:12.927697 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-84e377-openshift-default-5554bf8669-lzzbj" Nov 03 01:08:12.928807 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:12.928775 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-84e377-openshift-default-5554bf8669-lzzbj" Nov 03 01:08:13.024572 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:13.024528 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56b638a8-8pvz8"] Nov 03 01:08:13.029712 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:13.029682 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56b638a8-8pvz8"] Nov 03 01:08:14.012528 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:14.012486 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8fb996be-f95b-4a6f-9884-5e25ca6e19a3" path="/var/lib/kubelet/pods/8fb996be-f95b-4a6f-9884-5e25ca6e19a3/volumes" Nov 03 01:08:14.675068 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:08:14.675028 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 01:08:24.074301 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:24.061938 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77fa57c7-dsk6x"] Nov 03 01:08:24.122039 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:24.122005 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77fa57c7-dsk6x"] Nov 03 01:08:25.981450 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:25.981417 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="075b622b-9932-49cf-8c79-87ebcdc7e4bb" path="/var/lib/kubelet/pods/075b622b-9932-49cf-8c79-87ebcdc7e4bb/volumes" Nov 03 01:08:27.027051 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:27.027019 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a440037f-z754x"] Nov 03 01:08:27.032185 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:27.032160 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a440037f-z754x"] Nov 03 01:08:27.986692 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:27.986661 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="082dc2ba-188b-4d58-8d73-b53344fcd088" path="/var/lib/kubelet/pods/082dc2ba-188b-4d58-8d73-b53344fcd088/volumes" Nov 03 01:08:32.430110 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:32.430069 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-84e377-8468cbf5b9-wx4nd"] Nov 03 01:08:32.450676 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:32.450642 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-84e377-6bc89ccf99-pczgs"] Nov 03 01:08:36.024706 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:36.024667 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e6556d35-vr5qh"] Nov 03 01:08:36.029927 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:36.029894 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e6556d35-vr5qh"] Nov 03 01:08:37.452463 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:37.452411 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-84e377-6bc89ccf99-pczgs" podUID="78d5c907-e2f2-4755-bb98-0b08e6e8dd47" containerName="payload-pre-processing" containerID="cri-o://2536a5d67726786f8bbfae719a81e856cf14d957344569a005f1d7fb213587b9" gracePeriod=25 Nov 03 01:08:37.777356 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:37.777258 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-84e377-6bc89ccf99-pczgs" Nov 03 01:08:37.849676 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:37.849641 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78d5c907-e2f2-4755-bb98-0b08e6e8dd47-kube-api-access-nc7sn\" (UniqueName: \"kubernetes.io/projected/78d5c907-e2f2-4755-bb98-0b08e6e8dd47-kube-api-access-nc7sn\") pod \"78d5c907-e2f2-4755-bb98-0b08e6e8dd47\" (UID: \"78d5c907-e2f2-4755-bb98-0b08e6e8dd47\") " Nov 03 01:08:37.850082 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:37.850052 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78d5c907-e2f2-4755-bb98-0b08e6e8dd47-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78d5c907-e2f2-4755-bb98-0b08e6e8dd47-plugins-config-volume\") pod \"78d5c907-e2f2-4755-bb98-0b08e6e8dd47\" (UID: \"78d5c907-e2f2-4755-bb98-0b08e6e8dd47\") " Nov 03 01:08:37.851264 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:37.851237 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78d5c907-e2f2-4755-bb98-0b08e6e8dd47-plugins-config-volume" pod "78d5c907-e2f2-4755-bb98-0b08e6e8dd47" (UID: "78d5c907-e2f2-4755-bb98-0b08e6e8dd47"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:08:37.853380 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:37.853350 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78d5c907-e2f2-4755-bb98-0b08e6e8dd47-kube-api-access-nc7sn" pod "78d5c907-e2f2-4755-bb98-0b08e6e8dd47" (UID: "78d5c907-e2f2-4755-bb98-0b08e6e8dd47"). InnerVolumeSpecName "kube-api-access-nc7sn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:08:37.861421 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:37.861352 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-84e377-8468cbf5b9-wx4nd" podUID="48055fe8-0181-47f8-bdc6-da665d9badac" containerName="payload-processing" containerID="cri-o://86055a4842f72d08f0a2a93b1cfb937f92c7fc7179a83129f69eb5b3202c660f" gracePeriod=25 Nov 03 01:08:37.942756 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:37.942726 4421 generic.go:356] "Generic (PLEG): container finished" podID="78d5c907-e2f2-4755-bb98-0b08e6e8dd47" containerID="2536a5d67726786f8bbfae719a81e856cf14d957344569a005f1d7fb213587b9" exitCode=0 Nov 03 01:08:37.942953 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:37.942778 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-84e377-6bc89ccf99-pczgs" Nov 03 01:08:37.942953 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:37.942825 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-84e377-6bc89ccf99-pczgs" event={"ID":"78d5c907-e2f2-4755-bb98-0b08e6e8dd47","Type":"ContainerDied","Data":"2536a5d67726786f8bbfae719a81e856cf14d957344569a005f1d7fb213587b9"} Nov 03 01:08:37.942953 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:37.942888 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-84e377-6bc89ccf99-pczgs" event={"ID":"78d5c907-e2f2-4755-bb98-0b08e6e8dd47","Type":"ContainerDied","Data":"b2d8ec56aa25b149e4d24d8705db9ab2940a7fac8943ff2c5269807d6218ede6"} Nov 03 01:08:37.942953 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:37.942918 4421 scope.go:122] "RemoveContainer" containerID="2536a5d67726786f8bbfae719a81e856cf14d957344569a005f1d7fb213587b9" Nov 03 01:08:37.952656 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:37.952630 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nc7sn\" (UniqueName: \"kubernetes.io/projected/78d5c907-e2f2-4755-bb98-0b08e6e8dd47-kube-api-access-nc7sn\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:08:37.952771 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:37.952657 4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78d5c907-e2f2-4755-bb98-0b08e6e8dd47-plugins-config-volume\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:08:37.969421 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:37.969392 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-84e377-6bc89ccf99-pczgs"] Nov 03 01:08:37.979034 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:37.979017 4421 scope.go:122] "RemoveContainer" containerID="2536a5d67726786f8bbfae719a81e856cf14d957344569a005f1d7fb213587b9" Nov 03 01:08:37.979459 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:08:37.979435 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2536a5d67726786f8bbfae719a81e856cf14d957344569a005f1d7fb213587b9\": container with ID starting with 2536a5d67726786f8bbfae719a81e856cf14d957344569a005f1d7fb213587b9 not found: ID does not exist" containerID="2536a5d67726786f8bbfae719a81e856cf14d957344569a005f1d7fb213587b9" Nov 03 01:08:37.979591 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:37.979465 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2536a5d67726786f8bbfae719a81e856cf14d957344569a005f1d7fb213587b9"} err="failed to get container status \"2536a5d67726786f8bbfae719a81e856cf14d957344569a005f1d7fb213587b9\": rpc error: code = NotFound desc = could not find container \"2536a5d67726786f8bbfae719a81e856cf14d957344569a005f1d7fb213587b9\": container with ID starting with 2536a5d67726786f8bbfae719a81e856cf14d957344569a005f1d7fb213587b9 not found: ID does not exist" Nov 03 01:08:37.982605 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:37.982579 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7106c2fc-62bf-42cf-8faa-11f561707d0d" path="/var/lib/kubelet/pods/7106c2fc-62bf-42cf-8faa-11f561707d0d/volumes" Nov 03 01:08:37.983039 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:37.983017 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-84e377-6bc89ccf99-pczgs"] Nov 03 01:08:38.232127 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:38.232101 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-84e377-8468cbf5b9-wx4nd" Nov 03 01:08:38.304025 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:38.303932 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48055fe8-0181-47f8-bdc6-da665d9badac-kube-api-access-5kjf5\" (UniqueName: \"kubernetes.io/projected/48055fe8-0181-47f8-bdc6-da665d9badac-kube-api-access-5kjf5\") pod \"48055fe8-0181-47f8-bdc6-da665d9badac\" (UID: \"48055fe8-0181-47f8-bdc6-da665d9badac\") " Nov 03 01:08:38.304171 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:38.304154 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48055fe8-0181-47f8-bdc6-da665d9badac-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48055fe8-0181-47f8-bdc6-da665d9badac-plugins-config-volume\") pod \"48055fe8-0181-47f8-bdc6-da665d9badac\" (UID: \"48055fe8-0181-47f8-bdc6-da665d9badac\") " Nov 03 01:08:38.304572 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:38.304542 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48055fe8-0181-47f8-bdc6-da665d9badac-plugins-config-volume" pod "48055fe8-0181-47f8-bdc6-da665d9badac" (UID: "48055fe8-0181-47f8-bdc6-da665d9badac"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:08:38.305468 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:38.305441 4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48055fe8-0181-47f8-bdc6-da665d9badac-plugins-config-volume\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:08:38.306178 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:38.306145 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48055fe8-0181-47f8-bdc6-da665d9badac-kube-api-access-5kjf5" pod "48055fe8-0181-47f8-bdc6-da665d9badac" (UID: "48055fe8-0181-47f8-bdc6-da665d9badac"). InnerVolumeSpecName "kube-api-access-5kjf5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:08:38.407460 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:38.407425 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5kjf5\" (UniqueName: \"kubernetes.io/projected/48055fe8-0181-47f8-bdc6-da665d9badac-kube-api-access-5kjf5\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:08:38.956989 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:38.956948 4421 generic.go:356] "Generic (PLEG): container finished" podID="48055fe8-0181-47f8-bdc6-da665d9badac" containerID="86055a4842f72d08f0a2a93b1cfb937f92c7fc7179a83129f69eb5b3202c660f" exitCode=0 Nov 03 01:08:38.957475 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:38.957016 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-84e377-8468cbf5b9-wx4nd" Nov 03 01:08:38.957475 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:38.957033 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-84e377-8468cbf5b9-wx4nd" event={"ID":"48055fe8-0181-47f8-bdc6-da665d9badac","Type":"ContainerDied","Data":"86055a4842f72d08f0a2a93b1cfb937f92c7fc7179a83129f69eb5b3202c660f"} Nov 03 01:08:38.957475 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:38.957128 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-84e377-8468cbf5b9-wx4nd" event={"ID":"48055fe8-0181-47f8-bdc6-da665d9badac","Type":"ContainerDied","Data":"3916d33ffbc540b110242ab2e55644094fbcd8e82120cde35f2c5447f1e84d8e"} Nov 03 01:08:38.957475 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:38.957157 4421 scope.go:122] "RemoveContainer" containerID="86055a4842f72d08f0a2a93b1cfb937f92c7fc7179a83129f69eb5b3202c660f" Nov 03 01:08:38.969286 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:38.969251 4421 scope.go:122] "RemoveContainer" containerID="86055a4842f72d08f0a2a93b1cfb937f92c7fc7179a83129f69eb5b3202c660f" Nov 03 01:08:38.970374 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:08:38.970347 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86055a4842f72d08f0a2a93b1cfb937f92c7fc7179a83129f69eb5b3202c660f\": container with ID starting with 86055a4842f72d08f0a2a93b1cfb937f92c7fc7179a83129f69eb5b3202c660f not found: ID does not exist" containerID="86055a4842f72d08f0a2a93b1cfb937f92c7fc7179a83129f69eb5b3202c660f" Nov 03 01:08:38.970491 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:38.970382 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86055a4842f72d08f0a2a93b1cfb937f92c7fc7179a83129f69eb5b3202c660f"} err="failed to get container status \"86055a4842f72d08f0a2a93b1cfb937f92c7fc7179a83129f69eb5b3202c660f\": rpc error: code = NotFound desc = could not find container \"86055a4842f72d08f0a2a93b1cfb937f92c7fc7179a83129f69eb5b3202c660f\": container with ID starting with 86055a4842f72d08f0a2a93b1cfb937f92c7fc7179a83129f69eb5b3202c660f not found: ID does not exist" Nov 03 01:08:38.984096 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:38.984065 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-84e377-8468cbf5b9-wx4nd"] Nov 03 01:08:38.994187 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:38.994149 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-84e377-8468cbf5b9-wx4nd"] Nov 03 01:08:40.077305 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:40.073680 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48055fe8-0181-47f8-bdc6-da665d9badac" path="/var/lib/kubelet/pods/48055fe8-0181-47f8-bdc6-da665d9badac/volumes" Nov 03 01:08:40.077305 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:40.074440 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78d5c907-e2f2-4755-bb98-0b08e6e8dd47" path="/var/lib/kubelet/pods/78d5c907-e2f2-4755-bb98-0b08e6e8dd47/volumes" Nov 03 01:08:40.105844 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:40.105810 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf2d166c-q8psv"] Nov 03 01:08:40.187997 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:40.187738 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf2d166c-q8psv"] Nov 03 01:08:40.941301 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:40.940430 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8gm8n" podUID="2d367e12-e46f-4a68-9825-9b1c934d4d12" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 01:08:41.063302 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:41.037973 4421 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-83755a/premium-w3-83755a-kserve-7c9d66988b-lt4xd" podUID="c0c6b02e-2ecb-428e-a13d-be6d7cd1ba39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:08:41.192364 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:41.176323 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/simulator-w2-f7680d-kserve-85bdc86cd6-8gxrg" podUID="2bdcf521-cebd-4d6b-8554-1b60db8b0f3c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 03 01:08:41.355306 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:41.347394 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-650be0/premium-w4-650be0-kserve-7bf8555dfb-z97n5" podUID="40601dc5-808f-456f-826f-6e0c35966ada" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:08:41.373304 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:41.366125 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/premium-w3-83755a-kserve-7c9d66988b-lt4xd" podUID="c0c6b02e-2ecb-428e-a13d-be6d7cd1ba39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:08:41.628152 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:41.627400 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/distinct-w3-83755a-kserve-86bdbbdd85-2cnnl" podUID="f4ae9624-d127-4a55-837d-7c9ff0267b37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 01:08:41.638821 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:41.638370 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-650be0/simulator-w4-650be0-kserve-7bc588774f-gvz2k" podUID="a827fa88-f078-4312-8f12-2c799c11b461" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:08:41.986341 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:41.986226 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="965c6f6a-6363-4ee1-8c1c-e5252502790c" path="/var/lib/kubelet/pods/965c6f6a-6363-4ee1-8c1c-e5252502790c/volumes" Nov 03 01:08:43.491140 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:43.491102 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5151163-qxf58"] Nov 03 01:08:43.492977 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:43.492947 4421 cpu_manager.go:414] "containerMap: removing container" podUID="96b6a080-1574-4a42-8197-dd1cb436757e" containerName="main" Nov 03 01:08:43.492977 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:43.492976 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96b6a080-1574-4a42-8197-dd1cb436757e" containerName="main" Nov 03 01:08:43.493194 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:43.492995 4421 cpu_manager.go:414] "containerMap: removing container" podUID="48055fe8-0181-47f8-bdc6-da665d9badac" containerName="payload-processing" Nov 03 01:08:43.493194 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:43.493003 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48055fe8-0181-47f8-bdc6-da665d9badac" containerName="payload-processing" Nov 03 01:08:43.493194 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:43.493027 4421 cpu_manager.go:414] "containerMap: removing container" podUID="78d5c907-e2f2-4755-bb98-0b08e6e8dd47" containerName="payload-pre-processing" Nov 03 01:08:43.493194 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:43.493035 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78d5c907-e2f2-4755-bb98-0b08e6e8dd47" containerName="payload-pre-processing" Nov 03 01:08:43.493457 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:43.493199 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="96b6a080-1574-4a42-8197-dd1cb436757e" containerName="main" Nov 03 01:08:43.493457 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:43.493213 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="48055fe8-0181-47f8-bdc6-da665d9badac" containerName="payload-processing" Nov 03 01:08:43.493457 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:43.493227 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="78d5c907-e2f2-4755-bb98-0b08e6e8dd47" containerName="payload-pre-processing" Nov 03 01:08:43.510653 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:43.510620 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5151163-qxf58" Nov 03 01:08:43.512304 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:43.512256 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5151163-qxf58"] Nov 03 01:08:43.517932 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:43.517891 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 01:08:43.518121 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:43.518098 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6q4xp\"" Nov 03 01:08:43.518202 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:43.518102 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 01:08:43.518267 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:43.518259 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:08:43.628393 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:43.628353 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/527996cc-c052-44b9-a1e3-a0b7c3125cd9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a5151163-qxf58\" (UID: \"527996cc-c052-44b9-a1e3-a0b7c3125cd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5151163-qxf58" Nov 03 01:08:43.628643 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:43.628621 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/527996cc-c052-44b9-a1e3-a0b7c3125cd9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a5151163-qxf58\" (UID: \"527996cc-c052-44b9-a1e3-a0b7c3125cd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5151163-qxf58" Nov 03 01:08:43.628705 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:43.628651 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbdvf\" (UniqueName: \"kubernetes.io/projected/527996cc-c052-44b9-a1e3-a0b7c3125cd9-kube-api-access-fbdvf\") pod \"maas-api-revoke-sub-a5151163-qxf58\" (UID: \"527996cc-c052-44b9-a1e3-a0b7c3125cd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5151163-qxf58" Nov 03 01:08:43.730571 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:43.730521 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/527996cc-c052-44b9-a1e3-a0b7c3125cd9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a5151163-qxf58\" (UID: \"527996cc-c052-44b9-a1e3-a0b7c3125cd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5151163-qxf58" Nov 03 01:08:43.730571 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:43.730571 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbdvf\" (UniqueName: \"kubernetes.io/projected/527996cc-c052-44b9-a1e3-a0b7c3125cd9-kube-api-access-fbdvf\") pod \"maas-api-revoke-sub-a5151163-qxf58\" (UID: \"527996cc-c052-44b9-a1e3-a0b7c3125cd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5151163-qxf58" Nov 03 01:08:43.730852 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:43.730830 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/527996cc-c052-44b9-a1e3-a0b7c3125cd9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a5151163-qxf58\" (UID: \"527996cc-c052-44b9-a1e3-a0b7c3125cd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5151163-qxf58" Nov 03 01:08:43.731347 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:43.731324 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/527996cc-c052-44b9-a1e3-a0b7c3125cd9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a5151163-qxf58\" (UID: \"527996cc-c052-44b9-a1e3-a0b7c3125cd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5151163-qxf58" Nov 03 01:08:43.733837 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:43.733815 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/527996cc-c052-44b9-a1e3-a0b7c3125cd9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a5151163-qxf58\" (UID: \"527996cc-c052-44b9-a1e3-a0b7c3125cd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5151163-qxf58" Nov 03 01:08:43.739785 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:43.739760 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbdvf\" (UniqueName: \"kubernetes.io/projected/527996cc-c052-44b9-a1e3-a0b7c3125cd9-kube-api-access-fbdvf\") pod \"maas-api-revoke-sub-a5151163-qxf58\" (UID: \"527996cc-c052-44b9-a1e3-a0b7c3125cd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5151163-qxf58" Nov 03 01:08:43.845540 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:43.845502 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5151163-qxf58" Nov 03 01:08:44.134692 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:08:44.134650 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod527996cc_c052_44b9_a1e3_a0b7c3125cd9.slice/crio-625fda46414226b1296b824c181adfd308ace253527518e13eb610ab0707b3ee WatchSource:0}: Error finding container 625fda46414226b1296b824c181adfd308ace253527518e13eb610ab0707b3ee: Status 404 returned error can't find the container with id 625fda46414226b1296b824c181adfd308ace253527518e13eb610ab0707b3ee Nov 03 01:08:44.135071 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:44.135047 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5151163-qxf58"] Nov 03 01:08:44.300322 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:44.299206 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5151163-qxf58" event={"ID":"527996cc-c052-44b9-a1e3-a0b7c3125cd9","Type":"ContainerStarted","Data":"625fda46414226b1296b824c181adfd308ace253527518e13eb610ab0707b3ee"} Nov 03 01:08:44.755913 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:08:44.755874 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 01:08:45.308801 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:45.308767 4421 generic.go:356] "Generic (PLEG): container finished" podID="527996cc-c052-44b9-a1e3-a0b7c3125cd9" containerID="a8dac3bae2e9ab746c4fd02ee42a55b774238dfb69b22bca51f60ab1268d6be7" exitCode=0 Nov 03 01:08:45.308979 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:45.308851 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5151163-qxf58" event={"ID":"527996cc-c052-44b9-a1e3-a0b7c3125cd9","Type":"ContainerDied","Data":"a8dac3bae2e9ab746c4fd02ee42a55b774238dfb69b22bca51f60ab1268d6be7"} Nov 03 01:08:46.524028 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:46.523999 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5151163-qxf58" Nov 03 01:08:46.609076 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:46.608985 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/527996cc-c052-44b9-a1e3-a0b7c3125cd9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/527996cc-c052-44b9-a1e3-a0b7c3125cd9-maas-api-openshift-ca\") pod \"527996cc-c052-44b9-a1e3-a0b7c3125cd9\" (UID: \"527996cc-c052-44b9-a1e3-a0b7c3125cd9\") " Nov 03 01:08:46.609076 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:46.609061 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/527996cc-c052-44b9-a1e3-a0b7c3125cd9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/527996cc-c052-44b9-a1e3-a0b7c3125cd9-maas-api-xks-ca\") pod \"527996cc-c052-44b9-a1e3-a0b7c3125cd9\" (UID: \"527996cc-c052-44b9-a1e3-a0b7c3125cd9\") " Nov 03 01:08:46.609422 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:46.609256 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/527996cc-c052-44b9-a1e3-a0b7c3125cd9-kube-api-access-fbdvf\" (UniqueName: \"kubernetes.io/projected/527996cc-c052-44b9-a1e3-a0b7c3125cd9-kube-api-access-fbdvf\") pod \"527996cc-c052-44b9-a1e3-a0b7c3125cd9\" (UID: \"527996cc-c052-44b9-a1e3-a0b7c3125cd9\") " Nov 03 01:08:46.609546 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:46.609422 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/527996cc-c052-44b9-a1e3-a0b7c3125cd9-maas-api-openshift-ca" pod "527996cc-c052-44b9-a1e3-a0b7c3125cd9" (UID: "527996cc-c052-44b9-a1e3-a0b7c3125cd9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:08:46.610107 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:46.610080 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/527996cc-c052-44b9-a1e3-a0b7c3125cd9-maas-api-openshift-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:08:46.611505 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:46.611478 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/527996cc-c052-44b9-a1e3-a0b7c3125cd9-maas-api-xks-ca" pod "527996cc-c052-44b9-a1e3-a0b7c3125cd9" (UID: "527996cc-c052-44b9-a1e3-a0b7c3125cd9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:08:46.611781 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:46.611763 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/527996cc-c052-44b9-a1e3-a0b7c3125cd9-kube-api-access-fbdvf" pod "527996cc-c052-44b9-a1e3-a0b7c3125cd9" (UID: "527996cc-c052-44b9-a1e3-a0b7c3125cd9"). InnerVolumeSpecName "kube-api-access-fbdvf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:08:46.712401 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:46.712364 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbdvf\" (UniqueName: \"kubernetes.io/projected/527996cc-c052-44b9-a1e3-a0b7c3125cd9-kube-api-access-fbdvf\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:08:46.712704 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:46.712679 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/527996cc-c052-44b9-a1e3-a0b7c3125cd9-maas-api-xks-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:08:47.329255 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:47.329224 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5151163-qxf58" Nov 03 01:08:47.329255 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:47.329238 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5151163-qxf58" event={"ID":"527996cc-c052-44b9-a1e3-a0b7c3125cd9","Type":"ContainerDied","Data":"625fda46414226b1296b824c181adfd308ace253527518e13eb610ab0707b3ee"} Nov 03 01:08:47.329535 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:47.329296 4421 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="625fda46414226b1296b824c181adfd308ace253527518e13eb610ab0707b3ee" Nov 03 01:08:50.024312 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:50.024264 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-141f3283-5r4gf"] Nov 03 01:08:50.029957 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:50.029915 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-141f3283-5r4gf"] Nov 03 01:08:51.026058 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:51.026020 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35618553-t49t6"] Nov 03 01:08:51.033711 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:51.033669 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35618553-t49t6"] Nov 03 01:08:51.044483 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:51.044368 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24ca0955-cf9nt"] Nov 03 01:08:51.049607 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:51.049574 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24ca0955-cf9nt"] Nov 03 01:08:51.990946 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:51.990902 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e1f5804-5e9e-4cf3-b024-ec6d4cd9d739" path="/var/lib/kubelet/pods/8e1f5804-5e9e-4cf3-b024-ec6d4cd9d739/volumes" Nov 03 01:08:51.991577 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:51.991554 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0ae6246-255c-42fa-9210-adfaabe6f36f" path="/var/lib/kubelet/pods/d0ae6246-255c-42fa-9210-adfaabe6f36f/volumes" Nov 03 01:08:51.992104 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:51.992084 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1df22d7-a6c9-41d0-a379-ab5e852fbc0a" path="/var/lib/kubelet/pods/d1df22d7-a6c9-41d0-a379-ab5e852fbc0a/volumes" Nov 03 01:08:53.704462 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:53.704429 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09b5f664-9g7fw"] Nov 03 01:08:53.705198 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:53.705182 4421 cpu_manager.go:414] "containerMap: removing container" podUID="527996cc-c052-44b9-a1e3-a0b7c3125cd9" containerName="revoke-keys" Nov 03 01:08:53.705265 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:53.705199 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="527996cc-c052-44b9-a1e3-a0b7c3125cd9" containerName="revoke-keys" Nov 03 01:08:53.705384 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:53.705373 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="527996cc-c052-44b9-a1e3-a0b7c3125cd9" containerName="revoke-keys" Nov 03 01:08:53.710299 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:53.710262 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09b5f664-9g7fw" Nov 03 01:08:53.715409 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:53.715380 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6q4xp\"" Nov 03 01:08:53.715409 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:53.715393 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 01:08:53.715679 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:53.715494 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:08:53.715679 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:53.715573 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 01:08:53.717427 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:53.717402 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09b5f664-9g7fw"] Nov 03 01:08:53.815645 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:53.815606 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d81e03a-8d5d-435f-80dc-fd9e05310895-maas-api-xks-ca\") pod \"maas-api-revoke-sub-09b5f664-9g7fw\" (UID: \"0d81e03a-8d5d-435f-80dc-fd9e05310895\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09b5f664-9g7fw" Nov 03 01:08:53.815832 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:53.815656 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-284z8\" (UniqueName: \"kubernetes.io/projected/0d81e03a-8d5d-435f-80dc-fd9e05310895-kube-api-access-284z8\") pod \"maas-api-revoke-sub-09b5f664-9g7fw\" (UID: \"0d81e03a-8d5d-435f-80dc-fd9e05310895\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09b5f664-9g7fw" Nov 03 01:08:53.815832 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:53.815813 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d81e03a-8d5d-435f-80dc-fd9e05310895-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-09b5f664-9g7fw\" (UID: \"0d81e03a-8d5d-435f-80dc-fd9e05310895\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09b5f664-9g7fw" Nov 03 01:08:53.917679 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:53.917639 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d81e03a-8d5d-435f-80dc-fd9e05310895-maas-api-xks-ca\") pod \"maas-api-revoke-sub-09b5f664-9g7fw\" (UID: \"0d81e03a-8d5d-435f-80dc-fd9e05310895\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09b5f664-9g7fw" Nov 03 01:08:53.917859 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:53.917687 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-284z8\" (UniqueName: \"kubernetes.io/projected/0d81e03a-8d5d-435f-80dc-fd9e05310895-kube-api-access-284z8\") pod \"maas-api-revoke-sub-09b5f664-9g7fw\" (UID: \"0d81e03a-8d5d-435f-80dc-fd9e05310895\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09b5f664-9g7fw" Nov 03 01:08:53.917859 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:53.917761 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d81e03a-8d5d-435f-80dc-fd9e05310895-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-09b5f664-9g7fw\" (UID: \"0d81e03a-8d5d-435f-80dc-fd9e05310895\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09b5f664-9g7fw" Nov 03 01:08:53.918482 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:53.918461 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d81e03a-8d5d-435f-80dc-fd9e05310895-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-09b5f664-9g7fw\" (UID: \"0d81e03a-8d5d-435f-80dc-fd9e05310895\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09b5f664-9g7fw" Nov 03 01:08:53.920046 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:53.920014 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d81e03a-8d5d-435f-80dc-fd9e05310895-maas-api-xks-ca\") pod \"maas-api-revoke-sub-09b5f664-9g7fw\" (UID: \"0d81e03a-8d5d-435f-80dc-fd9e05310895\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09b5f664-9g7fw" Nov 03 01:08:53.926266 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:53.926242 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-284z8\" (UniqueName: \"kubernetes.io/projected/0d81e03a-8d5d-435f-80dc-fd9e05310895-kube-api-access-284z8\") pod \"maas-api-revoke-sub-09b5f664-9g7fw\" (UID: \"0d81e03a-8d5d-435f-80dc-fd9e05310895\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09b5f664-9g7fw" Nov 03 01:08:54.050552 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:54.046073 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09b5f664-9g7fw" Nov 03 01:08:54.359292 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:08:54.359230 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d81e03a_8d5d_435f_80dc_fd9e05310895.slice/crio-57e1acce69fab847fa2c81108a37da105505d0aa2c426381b78c1ab013509d5b WatchSource:0}: Error finding container 57e1acce69fab847fa2c81108a37da105505d0aa2c426381b78c1ab013509d5b: Status 404 returned error can't find the container with id 57e1acce69fab847fa2c81108a37da105505d0aa2c426381b78c1ab013509d5b Nov 03 01:08:54.360215 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:54.360185 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09b5f664-9g7fw"] Nov 03 01:08:54.439599 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:54.439561 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09b5f664-9g7fw" event={"ID":"0d81e03a-8d5d-435f-80dc-fd9e05310895","Type":"ContainerStarted","Data":"57e1acce69fab847fa2c81108a37da105505d0aa2c426381b78c1ab013509d5b"} Nov 03 01:08:54.499785 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:54.499746 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-84e377-openshift-default-5554bf8669-lzzbj"] Nov 03 01:08:54.500442 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:54.500150 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-84e377-openshift-default-5554bf8669-lzzbj" podUID="b9815803-c1dc-4c21-a8f3-2a3c9c89f1b9" containerName="istio-proxy" containerID="cri-o://f68265b88dc94bbd4cf12137e75b6bb02ae75621d632862f27f97e292afeb39f" gracePeriod=30 Nov 03 01:08:55.450264 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:55.450214 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09b5f664-9g7fw" event={"ID":"0d81e03a-8d5d-435f-80dc-fd9e05310895","Type":"ContainerStarted","Data":"78336998ab42c5b0599ae431c6b2a95628958b43bd6326f96952bd56cc3d7caf"} Nov 03 01:08:55.475900 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:55.475847 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09b5f664-9g7fw" podStartSLOduration=2.475830044 podStartE2EDuration="2.475830044s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:08:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:08:55.472407501 +0000 UTC m=+1677.643410600" watchObservedRunningTime="2026-11-03 01:08:55.475830044 +0000 UTC m=+1677.646833140" Nov 03 01:08:56.461137 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:56.461099 4421 generic.go:356] "Generic (PLEG): container finished" podID="0d81e03a-8d5d-435f-80dc-fd9e05310895" containerID="78336998ab42c5b0599ae431c6b2a95628958b43bd6326f96952bd56cc3d7caf" exitCode=0 Nov 03 01:08:56.461572 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:56.461172 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09b5f664-9g7fw" event={"ID":"0d81e03a-8d5d-435f-80dc-fd9e05310895","Type":"ContainerDied","Data":"78336998ab42c5b0599ae431c6b2a95628958b43bd6326f96952bd56cc3d7caf"} Nov 03 01:08:57.733177 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:57.733115 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09b5f664-9g7fw" Nov 03 01:08:57.779754 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:57.779714 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d81e03a-8d5d-435f-80dc-fd9e05310895-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d81e03a-8d5d-435f-80dc-fd9e05310895-maas-api-xks-ca\") pod \"0d81e03a-8d5d-435f-80dc-fd9e05310895\" (UID: \"0d81e03a-8d5d-435f-80dc-fd9e05310895\") " Nov 03 01:08:57.779951 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:57.779796 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d81e03a-8d5d-435f-80dc-fd9e05310895-kube-api-access-284z8\" (UniqueName: \"kubernetes.io/projected/0d81e03a-8d5d-435f-80dc-fd9e05310895-kube-api-access-284z8\") pod \"0d81e03a-8d5d-435f-80dc-fd9e05310895\" (UID: \"0d81e03a-8d5d-435f-80dc-fd9e05310895\") " Nov 03 01:08:57.779951 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:57.779858 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d81e03a-8d5d-435f-80dc-fd9e05310895-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d81e03a-8d5d-435f-80dc-fd9e05310895-maas-api-openshift-ca\") pod \"0d81e03a-8d5d-435f-80dc-fd9e05310895\" (UID: \"0d81e03a-8d5d-435f-80dc-fd9e05310895\") " Nov 03 01:08:57.780543 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:57.780511 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d81e03a-8d5d-435f-80dc-fd9e05310895-maas-api-openshift-ca" pod "0d81e03a-8d5d-435f-80dc-fd9e05310895" (UID: "0d81e03a-8d5d-435f-80dc-fd9e05310895"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:08:57.780744 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:57.780725 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d81e03a-8d5d-435f-80dc-fd9e05310895-maas-api-openshift-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:08:57.782181 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:57.782159 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d81e03a-8d5d-435f-80dc-fd9e05310895-maas-api-xks-ca" pod "0d81e03a-8d5d-435f-80dc-fd9e05310895" (UID: "0d81e03a-8d5d-435f-80dc-fd9e05310895"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:08:57.782591 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:57.782565 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d81e03a-8d5d-435f-80dc-fd9e05310895-kube-api-access-284z8" pod "0d81e03a-8d5d-435f-80dc-fd9e05310895" (UID: "0d81e03a-8d5d-435f-80dc-fd9e05310895"). InnerVolumeSpecName "kube-api-access-284z8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:08:57.882233 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:57.882201 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d81e03a-8d5d-435f-80dc-fd9e05310895-maas-api-xks-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:08:57.882233 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:57.882231 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-284z8\" (UniqueName: \"kubernetes.io/projected/0d81e03a-8d5d-435f-80dc-fd9e05310895-kube-api-access-284z8\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:08:57.931332 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:57.931264 4421 patch_prober.go:29] interesting pod/e2e-explicit-84e377-openshift-default-5554bf8669-lzzbj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused Nov 03 01:08:57.931964 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:57.931357 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-explicit-84e377-openshift-default-5554bf8669-lzzbj" podUID="b9815803-c1dc-4c21-a8f3-2a3c9c89f1b9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 01:08:58.480175 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:58.480135 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09b5f664-9g7fw" Nov 03 01:08:58.480440 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:58.480169 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09b5f664-9g7fw" event={"ID":"0d81e03a-8d5d-435f-80dc-fd9e05310895","Type":"ContainerDied","Data":"57e1acce69fab847fa2c81108a37da105505d0aa2c426381b78c1ab013509d5b"} Nov 03 01:08:58.480440 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:58.480261 4421 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="57e1acce69fab847fa2c81108a37da105505d0aa2c426381b78c1ab013509d5b" Nov 03 01:08:59.576873 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:59.576836 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-5ws4g"] Nov 03 01:08:59.577963 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:59.577946 4421 cpu_manager.go:414] "containerMap: removing container" podUID="0d81e03a-8d5d-435f-80dc-fd9e05310895" containerName="revoke-keys" Nov 03 01:08:59.578037 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:59.577966 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d81e03a-8d5d-435f-80dc-fd9e05310895" containerName="revoke-keys" Nov 03 01:08:59.578223 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:59.578211 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d81e03a-8d5d-435f-80dc-fd9e05310895" containerName="revoke-keys" Nov 03 01:08:59.582753 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:59.582712 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-5ws4g" Nov 03 01:08:59.589639 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:59.589597 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-531447-openshift-default-dockercfg-z7zxz\"" Nov 03 01:08:59.593170 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:59.593140 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-5ws4g"] Nov 03 01:08:59.612583 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:59.612548 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmlpq\" (UniqueName: \"kubernetes.io/projected/05c74cad-b107-403b-bffc-af74dea66076-kube-api-access-pmlpq\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-5ws4g\" (UID: \"05c74cad-b107-403b-bffc-af74dea66076\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-5ws4g" Nov 03 01:08:59.612762 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:59.612703 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05c74cad-b107-403b-bffc-af74dea66076-istio-envoy\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-5ws4g\" (UID: \"05c74cad-b107-403b-bffc-af74dea66076\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-5ws4g" Nov 03 01:08:59.612911 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:59.612887 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05c74cad-b107-403b-bffc-af74dea66076-istio-podinfo\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-5ws4g\" (UID: \"05c74cad-b107-403b-bffc-af74dea66076\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-5ws4g" Nov 03 01:08:59.613027 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:59.612959 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05c74cad-b107-403b-bffc-af74dea66076-istio-data\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-5ws4g\" (UID: \"05c74cad-b107-403b-bffc-af74dea66076\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-5ws4g" Nov 03 01:08:59.613093 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:59.613054 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05c74cad-b107-403b-bffc-af74dea66076-istio-token\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-5ws4g\" (UID: \"05c74cad-b107-403b-bffc-af74dea66076\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-5ws4g" Nov 03 01:08:59.613254 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:59.613217 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05c74cad-b107-403b-bffc-af74dea66076-credential-socket\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-5ws4g\" (UID: \"05c74cad-b107-403b-bffc-af74dea66076\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-5ws4g" Nov 03 01:08:59.613811 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:59.613775 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05c74cad-b107-403b-bffc-af74dea66076-workload-certs\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-5ws4g\" (UID: \"05c74cad-b107-403b-bffc-af74dea66076\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-5ws4g" Nov 03 01:08:59.613937 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:59.613825 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05c74cad-b107-403b-bffc-af74dea66076-workload-socket\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-5ws4g\" (UID: \"05c74cad-b107-403b-bffc-af74dea66076\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-5ws4g" Nov 03 01:08:59.613937 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:59.613906 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05c74cad-b107-403b-bffc-af74dea66076-istiod-ca-cert\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-5ws4g\" (UID: \"05c74cad-b107-403b-bffc-af74dea66076\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-5ws4g" Nov 03 01:08:59.720765 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:59.717234 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05c74cad-b107-403b-bffc-af74dea66076-istio-podinfo\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-5ws4g\" (UID: \"05c74cad-b107-403b-bffc-af74dea66076\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-5ws4g" Nov 03 01:08:59.720765 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:59.717309 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05c74cad-b107-403b-bffc-af74dea66076-istio-data\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-5ws4g\" (UID: \"05c74cad-b107-403b-bffc-af74dea66076\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-5ws4g" Nov 03 01:08:59.720765 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:59.717344 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05c74cad-b107-403b-bffc-af74dea66076-istio-token\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-5ws4g\" (UID: \"05c74cad-b107-403b-bffc-af74dea66076\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-5ws4g" Nov 03 01:08:59.720765 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:59.717403 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05c74cad-b107-403b-bffc-af74dea66076-credential-socket\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-5ws4g\" (UID: \"05c74cad-b107-403b-bffc-af74dea66076\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-5ws4g" Nov 03 01:08:59.720765 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:59.717465 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05c74cad-b107-403b-bffc-af74dea66076-workload-certs\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-5ws4g\" (UID: \"05c74cad-b107-403b-bffc-af74dea66076\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-5ws4g" Nov 03 01:08:59.720765 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:59.717491 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05c74cad-b107-403b-bffc-af74dea66076-workload-socket\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-5ws4g\" (UID: \"05c74cad-b107-403b-bffc-af74dea66076\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-5ws4g" Nov 03 01:08:59.720765 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:59.717561 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05c74cad-b107-403b-bffc-af74dea66076-istiod-ca-cert\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-5ws4g\" (UID: \"05c74cad-b107-403b-bffc-af74dea66076\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-5ws4g" Nov 03 01:08:59.720765 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:59.717656 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmlpq\" (UniqueName: \"kubernetes.io/projected/05c74cad-b107-403b-bffc-af74dea66076-kube-api-access-pmlpq\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-5ws4g\" (UID: \"05c74cad-b107-403b-bffc-af74dea66076\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-5ws4g" Nov 03 01:08:59.720765 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:59.717731 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05c74cad-b107-403b-bffc-af74dea66076-istio-envoy\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-5ws4g\" (UID: \"05c74cad-b107-403b-bffc-af74dea66076\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-5ws4g" Nov 03 01:08:59.720765 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:59.718665 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05c74cad-b107-403b-bffc-af74dea66076-credential-socket\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-5ws4g\" (UID: \"05c74cad-b107-403b-bffc-af74dea66076\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-5ws4g" Nov 03 01:08:59.762312 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:59.744193 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05c74cad-b107-403b-bffc-af74dea66076-workload-socket\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-5ws4g\" (UID: \"05c74cad-b107-403b-bffc-af74dea66076\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-5ws4g" Nov 03 01:08:59.762312 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:59.750033 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05c74cad-b107-403b-bffc-af74dea66076-istiod-ca-cert\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-5ws4g\" (UID: \"05c74cad-b107-403b-bffc-af74dea66076\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-5ws4g" Nov 03 01:08:59.762312 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:59.750419 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05c74cad-b107-403b-bffc-af74dea66076-workload-certs\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-5ws4g\" (UID: \"05c74cad-b107-403b-bffc-af74dea66076\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-5ws4g" Nov 03 01:08:59.762312 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:59.750727 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05c74cad-b107-403b-bffc-af74dea66076-istio-data\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-5ws4g\" (UID: \"05c74cad-b107-403b-bffc-af74dea66076\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-5ws4g" Nov 03 01:08:59.969349 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:08:59.966290 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05c74cad-b107-403b-bffc-af74dea66076-istio-podinfo\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-5ws4g\" (UID: \"05c74cad-b107-403b-bffc-af74dea66076\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-5ws4g" Nov 03 01:09:00.116066 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:00.107397 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05c74cad-b107-403b-bffc-af74dea66076-istio-token\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-5ws4g\" (UID: \"05c74cad-b107-403b-bffc-af74dea66076\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-5ws4g" Nov 03 01:09:00.137671 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:00.137629 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05c74cad-b107-403b-bffc-af74dea66076-istio-envoy\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-5ws4g\" (UID: \"05c74cad-b107-403b-bffc-af74dea66076\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-5ws4g" Nov 03 01:09:00.138322 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:00.138243 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmlpq\" (UniqueName: \"kubernetes.io/projected/05c74cad-b107-403b-bffc-af74dea66076-kube-api-access-pmlpq\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-5ws4g\" (UID: \"05c74cad-b107-403b-bffc-af74dea66076\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-5ws4g" Nov 03 01:09:00.182230 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:00.182201 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-84e377-openshift-default-5554bf8669-lzzbj" Nov 03 01:09:00.216203 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:00.216170 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-5ws4g" Nov 03 01:09:00.273307 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:00.262513 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9815803-c1dc-4c21-a8f3-2a3c9c89f1b9-kube-api-access-8b42x\" (UniqueName: \"kubernetes.io/projected/b9815803-c1dc-4c21-a8f3-2a3c9c89f1b9-kube-api-access-8b42x\") pod \"b9815803-c1dc-4c21-a8f3-2a3c9c89f1b9\" (UID: \"b9815803-c1dc-4c21-a8f3-2a3c9c89f1b9\") " Nov 03 01:09:00.273307 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:00.262738 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9815803-c1dc-4c21-a8f3-2a3c9c89f1b9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9815803-c1dc-4c21-a8f3-2a3c9c89f1b9-credential-socket\") pod \"b9815803-c1dc-4c21-a8f3-2a3c9c89f1b9\" (UID: \"b9815803-c1dc-4c21-a8f3-2a3c9c89f1b9\") " Nov 03 01:09:00.273307 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:00.262829 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b9815803-c1dc-4c21-a8f3-2a3c9c89f1b9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9815803-c1dc-4c21-a8f3-2a3c9c89f1b9-istio-podinfo\") pod \"b9815803-c1dc-4c21-a8f3-2a3c9c89f1b9\" (UID: \"b9815803-c1dc-4c21-a8f3-2a3c9c89f1b9\") " Nov 03 01:09:00.273307 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:00.262935 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9815803-c1dc-4c21-a8f3-2a3c9c89f1b9-istio-token\" (UniqueName: \"kubernetes.io/projected/b9815803-c1dc-4c21-a8f3-2a3c9c89f1b9-istio-token\") pod \"b9815803-c1dc-4c21-a8f3-2a3c9c89f1b9\" (UID: \"b9815803-c1dc-4c21-a8f3-2a3c9c89f1b9\") " Nov 03 01:09:00.273307 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:00.263006 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9815803-c1dc-4c21-a8f3-2a3c9c89f1b9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9815803-c1dc-4c21-a8f3-2a3c9c89f1b9-workload-socket\") pod \"b9815803-c1dc-4c21-a8f3-2a3c9c89f1b9\" (UID: \"b9815803-c1dc-4c21-a8f3-2a3c9c89f1b9\") " Nov 03 01:09:00.273307 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:00.263041 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9815803-c1dc-4c21-a8f3-2a3c9c89f1b9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9815803-c1dc-4c21-a8f3-2a3c9c89f1b9-istio-data\") pod \"b9815803-c1dc-4c21-a8f3-2a3c9c89f1b9\" (UID: \"b9815803-c1dc-4c21-a8f3-2a3c9c89f1b9\") " Nov 03 01:09:00.273307 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:00.263124 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9815803-c1dc-4c21-a8f3-2a3c9c89f1b9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9815803-c1dc-4c21-a8f3-2a3c9c89f1b9-istio-envoy\") pod \"b9815803-c1dc-4c21-a8f3-2a3c9c89f1b9\" (UID: \"b9815803-c1dc-4c21-a8f3-2a3c9c89f1b9\") " Nov 03 01:09:00.273307 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:00.263155 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9815803-c1dc-4c21-a8f3-2a3c9c89f1b9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9815803-c1dc-4c21-a8f3-2a3c9c89f1b9-workload-certs\") pod \"b9815803-c1dc-4c21-a8f3-2a3c9c89f1b9\" (UID: \"b9815803-c1dc-4c21-a8f3-2a3c9c89f1b9\") " Nov 03 01:09:00.273307 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:00.263226 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9815803-c1dc-4c21-a8f3-2a3c9c89f1b9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9815803-c1dc-4c21-a8f3-2a3c9c89f1b9-istiod-ca-cert\") pod \"b9815803-c1dc-4c21-a8f3-2a3c9c89f1b9\" (UID: \"b9815803-c1dc-4c21-a8f3-2a3c9c89f1b9\") " Nov 03 01:09:00.273307 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:00.264860 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9815803-c1dc-4c21-a8f3-2a3c9c89f1b9-istiod-ca-cert" pod "b9815803-c1dc-4c21-a8f3-2a3c9c89f1b9" (UID: "b9815803-c1dc-4c21-a8f3-2a3c9c89f1b9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:09:00.273307 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:00.267186 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9815803-c1dc-4c21-a8f3-2a3c9c89f1b9-workload-certs" pod "b9815803-c1dc-4c21-a8f3-2a3c9c89f1b9" (UID: "b9815803-c1dc-4c21-a8f3-2a3c9c89f1b9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:09:00.273307 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:00.267258 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9815803-c1dc-4c21-a8f3-2a3c9c89f1b9-credential-socket" pod "b9815803-c1dc-4c21-a8f3-2a3c9c89f1b9" (UID: "b9815803-c1dc-4c21-a8f3-2a3c9c89f1b9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:09:00.273307 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:00.267326 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9815803-c1dc-4c21-a8f3-2a3c9c89f1b9-istio-data" pod "b9815803-c1dc-4c21-a8f3-2a3c9c89f1b9" (UID: "b9815803-c1dc-4c21-a8f3-2a3c9c89f1b9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:09:00.273307 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:00.267634 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9815803-c1dc-4c21-a8f3-2a3c9c89f1b9-workload-socket" pod "b9815803-c1dc-4c21-a8f3-2a3c9c89f1b9" (UID: "b9815803-c1dc-4c21-a8f3-2a3c9c89f1b9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:09:00.279015 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:00.278950 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9815803-c1dc-4c21-a8f3-2a3c9c89f1b9-istio-token" pod "b9815803-c1dc-4c21-a8f3-2a3c9c89f1b9" (UID: "b9815803-c1dc-4c21-a8f3-2a3c9c89f1b9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:09:00.281583 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:00.281544 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9815803-c1dc-4c21-a8f3-2a3c9c89f1b9-istio-envoy" pod "b9815803-c1dc-4c21-a8f3-2a3c9c89f1b9" (UID: "b9815803-c1dc-4c21-a8f3-2a3c9c89f1b9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:09:00.282121 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:00.282061 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9815803-c1dc-4c21-a8f3-2a3c9c89f1b9-kube-api-access-8b42x" pod "b9815803-c1dc-4c21-a8f3-2a3c9c89f1b9" (UID: "b9815803-c1dc-4c21-a8f3-2a3c9c89f1b9"). InnerVolumeSpecName "kube-api-access-8b42x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:09:00.287116 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:00.287068 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b9815803-c1dc-4c21-a8f3-2a3c9c89f1b9-istio-podinfo" pod "b9815803-c1dc-4c21-a8f3-2a3c9c89f1b9" (UID: "b9815803-c1dc-4c21-a8f3-2a3c9c89f1b9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:09:00.366891 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:00.366698 4421 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9815803-c1dc-4c21-a8f3-2a3c9c89f1b9-credential-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:09:00.366891 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:00.366735 4421 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9815803-c1dc-4c21-a8f3-2a3c9c89f1b9-istio-podinfo\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:09:00.366891 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:00.366751 4421 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9815803-c1dc-4c21-a8f3-2a3c9c89f1b9-istio-token\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:09:00.366891 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:00.366768 4421 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9815803-c1dc-4c21-a8f3-2a3c9c89f1b9-workload-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:09:00.366891 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:00.366804 4421 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9815803-c1dc-4c21-a8f3-2a3c9c89f1b9-istio-data\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:09:00.366891 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:00.366818 4421 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9815803-c1dc-4c21-a8f3-2a3c9c89f1b9-istio-envoy\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:09:00.366891 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:00.366832 4421 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9815803-c1dc-4c21-a8f3-2a3c9c89f1b9-workload-certs\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:09:00.366891 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:00.366847 4421 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9815803-c1dc-4c21-a8f3-2a3c9c89f1b9-istiod-ca-cert\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:09:00.366891 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:00.366862 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8b42x\" (UniqueName: \"kubernetes.io/projected/b9815803-c1dc-4c21-a8f3-2a3c9c89f1b9-kube-api-access-8b42x\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:09:00.501561 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:00.501528 4421 generic.go:356] "Generic (PLEG): container finished" podID="b9815803-c1dc-4c21-a8f3-2a3c9c89f1b9" containerID="f68265b88dc94bbd4cf12137e75b6bb02ae75621d632862f27f97e292afeb39f" exitCode=0 Nov 03 01:09:00.501721 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:00.501652 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-84e377-openshift-default-5554bf8669-lzzbj" Nov 03 01:09:00.501790 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:00.501654 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-84e377-openshift-default-5554bf8669-lzzbj" event={"ID":"b9815803-c1dc-4c21-a8f3-2a3c9c89f1b9","Type":"ContainerDied","Data":"f68265b88dc94bbd4cf12137e75b6bb02ae75621d632862f27f97e292afeb39f"} Nov 03 01:09:00.501835 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:00.501782 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-84e377-openshift-default-5554bf8669-lzzbj" event={"ID":"b9815803-c1dc-4c21-a8f3-2a3c9c89f1b9","Type":"ContainerDied","Data":"729da746a26192db0e74880b9d679cd1e70fb9242c4f9374cc7253aae5e3f9e8"} Nov 03 01:09:00.501835 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:00.501815 4421 scope.go:122] "RemoveContainer" containerID="f68265b88dc94bbd4cf12137e75b6bb02ae75621d632862f27f97e292afeb39f" Nov 03 01:09:00.515333 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:00.515297 4421 scope.go:122] "RemoveContainer" containerID="8741545e1131e3da6e37b3db71aff7890695839c68673721250d33ee4b513d48" Nov 03 01:09:00.529102 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:00.529082 4421 scope.go:122] "RemoveContainer" containerID="f68265b88dc94bbd4cf12137e75b6bb02ae75621d632862f27f97e292afeb39f" Nov 03 01:09:00.529748 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:09:00.529644 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f68265b88dc94bbd4cf12137e75b6bb02ae75621d632862f27f97e292afeb39f\": container with ID starting with f68265b88dc94bbd4cf12137e75b6bb02ae75621d632862f27f97e292afeb39f not found: ID does not exist" containerID="f68265b88dc94bbd4cf12137e75b6bb02ae75621d632862f27f97e292afeb39f" Nov 03 01:09:00.529965 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:00.529932 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f68265b88dc94bbd4cf12137e75b6bb02ae75621d632862f27f97e292afeb39f"} err="failed to get container status \"f68265b88dc94bbd4cf12137e75b6bb02ae75621d632862f27f97e292afeb39f\": rpc error: code = NotFound desc = could not find container \"f68265b88dc94bbd4cf12137e75b6bb02ae75621d632862f27f97e292afeb39f\": container with ID starting with f68265b88dc94bbd4cf12137e75b6bb02ae75621d632862f27f97e292afeb39f not found: ID does not exist" Nov 03 01:09:00.530223 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:00.530201 4421 scope.go:122] "RemoveContainer" containerID="8741545e1131e3da6e37b3db71aff7890695839c68673721250d33ee4b513d48" Nov 03 01:09:00.530587 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:09:00.530562 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8741545e1131e3da6e37b3db71aff7890695839c68673721250d33ee4b513d48\": container with ID starting with 8741545e1131e3da6e37b3db71aff7890695839c68673721250d33ee4b513d48 not found: ID does not exist" containerID="8741545e1131e3da6e37b3db71aff7890695839c68673721250d33ee4b513d48" Nov 03 01:09:00.530673 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:00.530597 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8741545e1131e3da6e37b3db71aff7890695839c68673721250d33ee4b513d48"} err="failed to get container status \"8741545e1131e3da6e37b3db71aff7890695839c68673721250d33ee4b513d48\": rpc error: code = NotFound desc = could not find container \"8741545e1131e3da6e37b3db71aff7890695839c68673721250d33ee4b513d48\": container with ID starting with 8741545e1131e3da6e37b3db71aff7890695839c68673721250d33ee4b513d48 not found: ID does not exist" Nov 03 01:09:00.533679 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:00.533658 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-84e377-openshift-default-5554bf8669-lzzbj"] Nov 03 01:09:00.539001 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:00.538979 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-84e377-openshift-default-5554bf8669-lzzbj"] Nov 03 01:09:00.557093 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:00.557071 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-5ws4g"] Nov 03 01:09:00.557546 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:09:00.557525 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05c74cad_b107_403b_bffc_af74dea66076.slice/crio-2998d8d4c7d49fafdd2fa0383cd1a109d148edbd9edf8800baf6b2d775ea484d WatchSource:0}: Error finding container 2998d8d4c7d49fafdd2fa0383cd1a109d148edbd9edf8800baf6b2d775ea484d: Status 404 returned error can't find the container with id 2998d8d4c7d49fafdd2fa0383cd1a109d148edbd9edf8800baf6b2d775ea484d Nov 03 01:09:00.560536 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:00.560505 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:09:00.560624 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:00.560567 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:09:00.560624 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:00.560599 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:09:00.711427 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:00.711365 4421 scope.go:122] "RemoveContainer" containerID="c70e9a0de0311127a9a7c6f499cae8d29fba148e24b91ea9421b96d87359d62a" Nov 03 01:09:01.513129 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:01.513096 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-5ws4g" event={"ID":"05c74cad-b107-403b-bffc-af74dea66076","Type":"ContainerStarted","Data":"52f533bb4c8a15e2597427f81efc7c95ff1857307d03a59375f513687b8a55a0"} Nov 03 01:09:01.513338 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:01.513149 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-5ws4g" event={"ID":"05c74cad-b107-403b-bffc-af74dea66076","Type":"ContainerStarted","Data":"2998d8d4c7d49fafdd2fa0383cd1a109d148edbd9edf8800baf6b2d775ea484d"} Nov 03 01:09:01.540519 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:01.540466 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-5ws4g" podStartSLOduration=2.540451069 podStartE2EDuration="2.540451069s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:08:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:09:01.535176517 +0000 UTC m=+1683.706179610" watchObservedRunningTime="2026-11-03 01:09:01.540451069 +0000 UTC m=+1683.711454166" Nov 03 01:09:01.981193 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:01.981116 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9815803-c1dc-4c21-a8f3-2a3c9c89f1b9" path="/var/lib/kubelet/pods/b9815803-c1dc-4c21-a8f3-2a3c9c89f1b9/volumes" Nov 03 01:09:02.025723 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:02.025682 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39016e27-ldfgl"] Nov 03 01:09:02.030791 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:02.030756 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39016e27-ldfgl"] Nov 03 01:09:02.217408 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:02.217378 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-5ws4g" Nov 03 01:09:02.223646 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:02.223616 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-5ws4g" Nov 03 01:09:02.523965 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:02.523927 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-5ws4g" Nov 03 01:09:02.525010 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:02.524987 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-5ws4g" Nov 03 01:09:03.094007 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:03.089558 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a125e38-pmmf4"] Nov 03 01:09:03.099904 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:03.099869 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a125e38-pmmf4"] Nov 03 01:09:04.040301 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:04.040188 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="529f8d33-82a7-49d9-acdb-83e638d2e2a4" path="/var/lib/kubelet/pods/529f8d33-82a7-49d9-acdb-83e638d2e2a4/volumes" Nov 03 01:09:04.044303 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:04.041064 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="990696f8-a427-49f3-bc32-326d521e6fa8" path="/var/lib/kubelet/pods/990696f8-a427-49f3-bc32-326d521e6fa8/volumes" Nov 03 01:09:05.018125 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:05.018085 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-531447-7cf7bc4dcf-xt4bm"] Nov 03 01:09:05.019024 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:05.019005 4421 cpu_manager.go:414] "containerMap: removing container" podUID="b9815803-c1dc-4c21-a8f3-2a3c9c89f1b9" containerName="istio-proxy" Nov 03 01:09:05.019091 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:05.019024 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9815803-c1dc-4c21-a8f3-2a3c9c89f1b9" containerName="istio-proxy" Nov 03 01:09:05.019091 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:05.019044 4421 cpu_manager.go:414] "containerMap: removing container" podUID="b9815803-c1dc-4c21-a8f3-2a3c9c89f1b9" containerName="istio-proxy" Nov 03 01:09:05.019091 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:05.019049 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9815803-c1dc-4c21-a8f3-2a3c9c89f1b9" containerName="istio-proxy" Nov 03 01:09:05.019199 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:05.019184 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9815803-c1dc-4c21-a8f3-2a3c9c89f1b9" containerName="istio-proxy" Nov 03 01:09:05.019235 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:05.019203 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9815803-c1dc-4c21-a8f3-2a3c9c89f1b9" containerName="istio-proxy" Nov 03 01:09:05.025165 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:05.025141 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-531447-7cf7bc4dcf-xt4bm" Nov 03 01:09:05.031971 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:05.031939 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-531447\"" Nov 03 01:09:05.032711 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:05.032687 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-531447-dockercfg-8wpnx\"" Nov 03 01:09:05.035474 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:05.035435 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-531447-7cf7bc4dcf-xt4bm"] Nov 03 01:09:05.063768 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:05.063732 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-531447-667d774bfc-9shwq"] Nov 03 01:09:05.068707 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:05.068684 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-531447-667d774bfc-9shwq" Nov 03 01:09:05.081170 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:05.081136 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a3e520f-1e5b-4ae4-b263-c99521d89d9e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-531447-7cf7bc4dcf-xt4bm\" (UID: \"8a3e520f-1e5b-4ae4-b263-c99521d89d9e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-531447-7cf7bc4dcf-xt4bm" Nov 03 01:09:05.081403 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:05.081257 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-868qq\" (UniqueName: \"kubernetes.io/projected/8a3e520f-1e5b-4ae4-b263-c99521d89d9e-kube-api-access-868qq\") pod \"payload-pre-processing-e2e-shared-a-w2-531447-7cf7bc4dcf-xt4bm\" (UID: \"8a3e520f-1e5b-4ae4-b263-c99521d89d9e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-531447-7cf7bc4dcf-xt4bm" Nov 03 01:09:05.088592 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:05.088565 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-531447-667d774bfc-9shwq"] Nov 03 01:09:05.283349 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:05.259499 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a3e520f-1e5b-4ae4-b263-c99521d89d9e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-531447-7cf7bc4dcf-xt4bm\" (UID: \"8a3e520f-1e5b-4ae4-b263-c99521d89d9e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-531447-7cf7bc4dcf-xt4bm" Nov 03 01:09:05.297537 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:05.297482 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdq2n\" (UniqueName: \"kubernetes.io/projected/9f997097-9fbb-4cf2-b370-f48154901810-kube-api-access-bdq2n\") pod \"payload-processing-e2e-shared-a-w2-531447-667d774bfc-9shwq\" (UID: \"9f997097-9fbb-4cf2-b370-f48154901810\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-531447-667d774bfc-9shwq" Nov 03 01:09:05.313350 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:05.301366 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-868qq\" (UniqueName: \"kubernetes.io/projected/8a3e520f-1e5b-4ae4-b263-c99521d89d9e-kube-api-access-868qq\") pod \"payload-pre-processing-e2e-shared-a-w2-531447-7cf7bc4dcf-xt4bm\" (UID: \"8a3e520f-1e5b-4ae4-b263-c99521d89d9e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-531447-7cf7bc4dcf-xt4bm" Nov 03 01:09:05.313826 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:05.313796 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f997097-9fbb-4cf2-b370-f48154901810-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-531447-667d774bfc-9shwq\" (UID: \"9f997097-9fbb-4cf2-b370-f48154901810\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-531447-667d774bfc-9shwq" Nov 03 01:09:05.342303 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:05.265634 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a3e520f-1e5b-4ae4-b263-c99521d89d9e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-531447-7cf7bc4dcf-xt4bm\" (UID: \"8a3e520f-1e5b-4ae4-b263-c99521d89d9e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-531447-7cf7bc4dcf-xt4bm" Nov 03 01:09:05.361815 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:05.361594 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-868qq\" (UniqueName: \"kubernetes.io/projected/8a3e520f-1e5b-4ae4-b263-c99521d89d9e-kube-api-access-868qq\") pod \"payload-pre-processing-e2e-shared-a-w2-531447-7cf7bc4dcf-xt4bm\" (UID: \"8a3e520f-1e5b-4ae4-b263-c99521d89d9e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-531447-7cf7bc4dcf-xt4bm" Nov 03 01:09:05.423783 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:05.422105 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdq2n\" (UniqueName: \"kubernetes.io/projected/9f997097-9fbb-4cf2-b370-f48154901810-kube-api-access-bdq2n\") pod \"payload-processing-e2e-shared-a-w2-531447-667d774bfc-9shwq\" (UID: \"9f997097-9fbb-4cf2-b370-f48154901810\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-531447-667d774bfc-9shwq" Nov 03 01:09:05.423783 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:05.422308 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f997097-9fbb-4cf2-b370-f48154901810-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-531447-667d774bfc-9shwq\" (UID: \"9f997097-9fbb-4cf2-b370-f48154901810\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-531447-667d774bfc-9shwq" Nov 03 01:09:05.423783 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:05.423471 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f997097-9fbb-4cf2-b370-f48154901810-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-531447-667d774bfc-9shwq\" (UID: \"9f997097-9fbb-4cf2-b370-f48154901810\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-531447-667d774bfc-9shwq" Nov 03 01:09:05.450299 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:05.445161 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdq2n\" (UniqueName: \"kubernetes.io/projected/9f997097-9fbb-4cf2-b370-f48154901810-kube-api-access-bdq2n\") pod \"payload-processing-e2e-shared-a-w2-531447-667d774bfc-9shwq\" (UID: \"9f997097-9fbb-4cf2-b370-f48154901810\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-531447-667d774bfc-9shwq" Nov 03 01:09:05.645091 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:05.645003 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-531447-7cf7bc4dcf-xt4bm" Nov 03 01:09:05.692837 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:05.692798 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-531447-667d774bfc-9shwq" Nov 03 01:09:06.175347 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:06.162512 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-531447-7cf7bc4dcf-xt4bm"] Nov 03 01:09:06.256104 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:09:06.256077 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f997097_9fbb_4cf2_b370_f48154901810.slice/crio-8dd52f52e7e22dc9353573de40dae33ffaa21d2013eceeb850f7e2e1a62420d6 WatchSource:0}: Error finding container 8dd52f52e7e22dc9353573de40dae33ffaa21d2013eceeb850f7e2e1a62420d6: Status 404 returned error can't find the container with id 8dd52f52e7e22dc9353573de40dae33ffaa21d2013eceeb850f7e2e1a62420d6 Nov 03 01:09:06.258105 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:06.258077 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-531447-667d774bfc-9shwq"] Nov 03 01:09:06.566404 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:06.566367 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-531447-7cf7bc4dcf-xt4bm" event={"ID":"8a3e520f-1e5b-4ae4-b263-c99521d89d9e","Type":"ContainerStarted","Data":"b74c3fdc92d31740b8e8a002c49dba79a27ab542c30b0e2649687605e7824252"} Nov 03 01:09:06.566626 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:06.566428 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-531447-7cf7bc4dcf-xt4bm" event={"ID":"8a3e520f-1e5b-4ae4-b263-c99521d89d9e","Type":"ContainerStarted","Data":"785dddfd78f2f79b36b88062848563ed07db86e8dac2918b72dc906238242813"} Nov 03 01:09:06.566626 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:06.566487 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-531447-7cf7bc4dcf-xt4bm" Nov 03 01:09:06.575259 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:06.575223 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-531447-667d774bfc-9shwq" event={"ID":"9f997097-9fbb-4cf2-b370-f48154901810","Type":"ContainerStarted","Data":"338e3b01c784321a516c487e5b6c5637fb1b9a172718345ee311f93a221ef9ec"} Nov 03 01:09:06.575484 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:06.575335 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-531447-667d774bfc-9shwq" event={"ID":"9f997097-9fbb-4cf2-b370-f48154901810","Type":"ContainerStarted","Data":"8dd52f52e7e22dc9353573de40dae33ffaa21d2013eceeb850f7e2e1a62420d6"} Nov 03 01:09:06.576173 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:06.576153 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-531447-667d774bfc-9shwq" Nov 03 01:09:06.589178 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:06.589120 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-531447-7cf7bc4dcf-xt4bm" podStartSLOduration=2.589106068 podStartE2EDuration="2.589106068s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:09:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:09:06.586722822 +0000 UTC m=+1688.757725921" watchObservedRunningTime="2026-11-03 01:09:06.589106068 +0000 UTC m=+1688.760109210" Nov 03 01:09:06.605097 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:06.605041 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-531447-667d774bfc-9shwq" podStartSLOduration=1.6050221919999998 podStartE2EDuration="1.605022192s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:09:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:09:06.602722089 +0000 UTC m=+1688.773725182" watchObservedRunningTime="2026-11-03 01:09:06.605022192 +0000 UTC m=+1688.776025288" Nov 03 01:09:07.025049 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:07.025012 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15014ab9-hq8gm"] Nov 03 01:09:07.029964 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:07.029932 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15014ab9-hq8gm"] Nov 03 01:09:08.036306 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:08.036248 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40a29166-9958-4232-bdae-16802b39f316" path="/var/lib/kubelet/pods/40a29166-9958-4232-bdae-16802b39f316/volumes" Nov 03 01:09:11.026992 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:11.026942 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1e2ac75-c9cfl"] Nov 03 01:09:11.032096 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:11.032066 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1e2ac75-c9cfl"] Nov 03 01:09:11.988016 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:11.987981 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d053a07c-ed76-4510-98ea-1097312fcf38" path="/var/lib/kubelet/pods/d053a07c-ed76-4510-98ea-1097312fcf38/volumes" Nov 03 01:09:14.676538 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:09:14.676499 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 01:09:15.027575 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:15.027544 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff1448af-9knz8"] Nov 03 01:09:15.039924 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:15.039873 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff1448af-9knz8"] Nov 03 01:09:15.988977 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:15.988943 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4367a12b-c4d5-47ef-a064-8f2c1930d7c3" path="/var/lib/kubelet/pods/4367a12b-c4d5-47ef-a064-8f2c1930d7c3/volumes" Nov 03 01:09:16.150490 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:16.150416 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8308a673-mzz2s"] Nov 03 01:09:16.275993 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:16.275944 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8308a673-mzz2s"] Nov 03 01:09:17.602227 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:17.602193 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-531447-7cf7bc4dcf-xt4bm" Nov 03 01:09:17.982296 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:17.982199 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68f88e39-b2ce-4ba5-b235-caf9bdaff12f" path="/var/lib/kubelet/pods/68f88e39-b2ce-4ba5-b235-caf9bdaff12f/volumes" Nov 03 01:09:18.610861 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:18.610831 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-531447-667d774bfc-9shwq" Nov 03 01:09:21.955911 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:21.955869 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-821f5a-openshift-default-7bfc657d9-bv95q"] Nov 03 01:09:21.962231 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:21.962202 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-821f5a-openshift-default-7bfc657d9-bv95q" Nov 03 01:09:21.987423 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:21.986764 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff2ff304-350a-4e21-8415-1c28dcb90a7d-istiod-ca-cert\") pod \"e2e-extmod-a-821f5a-openshift-default-7bfc657d9-bv95q\" (UID: \"ff2ff304-350a-4e21-8415-1c28dcb90a7d\") " pod="openshift-ingress/e2e-extmod-a-821f5a-openshift-default-7bfc657d9-bv95q" Nov 03 01:09:21.987423 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:21.986902 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff2ff304-350a-4e21-8415-1c28dcb90a7d-istio-envoy\") pod \"e2e-extmod-a-821f5a-openshift-default-7bfc657d9-bv95q\" (UID: \"ff2ff304-350a-4e21-8415-1c28dcb90a7d\") " pod="openshift-ingress/e2e-extmod-a-821f5a-openshift-default-7bfc657d9-bv95q" Nov 03 01:09:21.987423 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:21.986953 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff2ff304-350a-4e21-8415-1c28dcb90a7d-istio-token\") pod \"e2e-extmod-a-821f5a-openshift-default-7bfc657d9-bv95q\" (UID: \"ff2ff304-350a-4e21-8415-1c28dcb90a7d\") " pod="openshift-ingress/e2e-extmod-a-821f5a-openshift-default-7bfc657d9-bv95q" Nov 03 01:09:21.987423 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:21.987004 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff2ff304-350a-4e21-8415-1c28dcb90a7d-istio-podinfo\") pod \"e2e-extmod-a-821f5a-openshift-default-7bfc657d9-bv95q\" (UID: \"ff2ff304-350a-4e21-8415-1c28dcb90a7d\") " pod="openshift-ingress/e2e-extmod-a-821f5a-openshift-default-7bfc657d9-bv95q" Nov 03 01:09:21.987423 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:21.987099 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff2ff304-350a-4e21-8415-1c28dcb90a7d-credential-socket\") pod \"e2e-extmod-a-821f5a-openshift-default-7bfc657d9-bv95q\" (UID: \"ff2ff304-350a-4e21-8415-1c28dcb90a7d\") " pod="openshift-ingress/e2e-extmod-a-821f5a-openshift-default-7bfc657d9-bv95q" Nov 03 01:09:21.987423 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:21.987149 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff2ff304-350a-4e21-8415-1c28dcb90a7d-istio-data\") pod \"e2e-extmod-a-821f5a-openshift-default-7bfc657d9-bv95q\" (UID: \"ff2ff304-350a-4e21-8415-1c28dcb90a7d\") " pod="openshift-ingress/e2e-extmod-a-821f5a-openshift-default-7bfc657d9-bv95q" Nov 03 01:09:21.987423 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:21.987220 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pj4gx\" (UniqueName: \"kubernetes.io/projected/ff2ff304-350a-4e21-8415-1c28dcb90a7d-kube-api-access-pj4gx\") pod \"e2e-extmod-a-821f5a-openshift-default-7bfc657d9-bv95q\" (UID: \"ff2ff304-350a-4e21-8415-1c28dcb90a7d\") " pod="openshift-ingress/e2e-extmod-a-821f5a-openshift-default-7bfc657d9-bv95q" Nov 03 01:09:21.987423 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:21.987312 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff2ff304-350a-4e21-8415-1c28dcb90a7d-workload-certs\") pod \"e2e-extmod-a-821f5a-openshift-default-7bfc657d9-bv95q\" (UID: \"ff2ff304-350a-4e21-8415-1c28dcb90a7d\") " pod="openshift-ingress/e2e-extmod-a-821f5a-openshift-default-7bfc657d9-bv95q" Nov 03 01:09:21.987423 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:21.987395 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff2ff304-350a-4e21-8415-1c28dcb90a7d-workload-socket\") pod \"e2e-extmod-a-821f5a-openshift-default-7bfc657d9-bv95q\" (UID: \"ff2ff304-350a-4e21-8415-1c28dcb90a7d\") " pod="openshift-ingress/e2e-extmod-a-821f5a-openshift-default-7bfc657d9-bv95q" Nov 03 01:09:21.989602 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:21.989571 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-821f5a-openshift-default-dockercfg-8ft6g\"" Nov 03 01:09:21.994866 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:21.994843 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-821f5a-openshift-default-7bfc657d9-bv95q"] Nov 03 01:09:22.150168 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:22.150122 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff2ff304-350a-4e21-8415-1c28dcb90a7d-istiod-ca-cert\") pod \"e2e-extmod-a-821f5a-openshift-default-7bfc657d9-bv95q\" (UID: \"ff2ff304-350a-4e21-8415-1c28dcb90a7d\") " pod="openshift-ingress/e2e-extmod-a-821f5a-openshift-default-7bfc657d9-bv95q" Nov 03 01:09:22.150650 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:22.150615 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff2ff304-350a-4e21-8415-1c28dcb90a7d-istio-envoy\") pod \"e2e-extmod-a-821f5a-openshift-default-7bfc657d9-bv95q\" (UID: \"ff2ff304-350a-4e21-8415-1c28dcb90a7d\") " pod="openshift-ingress/e2e-extmod-a-821f5a-openshift-default-7bfc657d9-bv95q" Nov 03 01:09:22.150866 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:22.150846 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff2ff304-350a-4e21-8415-1c28dcb90a7d-istio-token\") pod \"e2e-extmod-a-821f5a-openshift-default-7bfc657d9-bv95q\" (UID: \"ff2ff304-350a-4e21-8415-1c28dcb90a7d\") " pod="openshift-ingress/e2e-extmod-a-821f5a-openshift-default-7bfc657d9-bv95q" Nov 03 01:09:22.151067 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:22.151049 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff2ff304-350a-4e21-8415-1c28dcb90a7d-istio-podinfo\") pod \"e2e-extmod-a-821f5a-openshift-default-7bfc657d9-bv95q\" (UID: \"ff2ff304-350a-4e21-8415-1c28dcb90a7d\") " pod="openshift-ingress/e2e-extmod-a-821f5a-openshift-default-7bfc657d9-bv95q" Nov 03 01:09:22.151210 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:22.151180 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff2ff304-350a-4e21-8415-1c28dcb90a7d-istiod-ca-cert\") pod \"e2e-extmod-a-821f5a-openshift-default-7bfc657d9-bv95q\" (UID: \"ff2ff304-350a-4e21-8415-1c28dcb90a7d\") " pod="openshift-ingress/e2e-extmod-a-821f5a-openshift-default-7bfc657d9-bv95q" Nov 03 01:09:22.151380 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:22.151357 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff2ff304-350a-4e21-8415-1c28dcb90a7d-credential-socket\") pod \"e2e-extmod-a-821f5a-openshift-default-7bfc657d9-bv95q\" (UID: \"ff2ff304-350a-4e21-8415-1c28dcb90a7d\") " pod="openshift-ingress/e2e-extmod-a-821f5a-openshift-default-7bfc657d9-bv95q" Nov 03 01:09:22.157338 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:22.152630 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff2ff304-350a-4e21-8415-1c28dcb90a7d-credential-socket\") pod \"e2e-extmod-a-821f5a-openshift-default-7bfc657d9-bv95q\" (UID: \"ff2ff304-350a-4e21-8415-1c28dcb90a7d\") " pod="openshift-ingress/e2e-extmod-a-821f5a-openshift-default-7bfc657d9-bv95q" Nov 03 01:09:22.157338 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:22.154318 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff2ff304-350a-4e21-8415-1c28dcb90a7d-istio-data\") pod \"e2e-extmod-a-821f5a-openshift-default-7bfc657d9-bv95q\" (UID: \"ff2ff304-350a-4e21-8415-1c28dcb90a7d\") " pod="openshift-ingress/e2e-extmod-a-821f5a-openshift-default-7bfc657d9-bv95q" Nov 03 01:09:22.157338 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:22.154441 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pj4gx\" (UniqueName: \"kubernetes.io/projected/ff2ff304-350a-4e21-8415-1c28dcb90a7d-kube-api-access-pj4gx\") pod \"e2e-extmod-a-821f5a-openshift-default-7bfc657d9-bv95q\" (UID: \"ff2ff304-350a-4e21-8415-1c28dcb90a7d\") " pod="openshift-ingress/e2e-extmod-a-821f5a-openshift-default-7bfc657d9-bv95q" Nov 03 01:09:22.157338 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:22.154536 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff2ff304-350a-4e21-8415-1c28dcb90a7d-workload-certs\") pod \"e2e-extmod-a-821f5a-openshift-default-7bfc657d9-bv95q\" (UID: \"ff2ff304-350a-4e21-8415-1c28dcb90a7d\") " pod="openshift-ingress/e2e-extmod-a-821f5a-openshift-default-7bfc657d9-bv95q" Nov 03 01:09:22.157338 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:22.154634 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff2ff304-350a-4e21-8415-1c28dcb90a7d-workload-socket\") pod \"e2e-extmod-a-821f5a-openshift-default-7bfc657d9-bv95q\" (UID: \"ff2ff304-350a-4e21-8415-1c28dcb90a7d\") " pod="openshift-ingress/e2e-extmod-a-821f5a-openshift-default-7bfc657d9-bv95q" Nov 03 01:09:22.157338 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:22.155029 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff2ff304-350a-4e21-8415-1c28dcb90a7d-workload-socket\") pod \"e2e-extmod-a-821f5a-openshift-default-7bfc657d9-bv95q\" (UID: \"ff2ff304-350a-4e21-8415-1c28dcb90a7d\") " pod="openshift-ingress/e2e-extmod-a-821f5a-openshift-default-7bfc657d9-bv95q" Nov 03 01:09:22.157338 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:22.155245 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff2ff304-350a-4e21-8415-1c28dcb90a7d-istio-data\") pod \"e2e-extmod-a-821f5a-openshift-default-7bfc657d9-bv95q\" (UID: \"ff2ff304-350a-4e21-8415-1c28dcb90a7d\") " pod="openshift-ingress/e2e-extmod-a-821f5a-openshift-default-7bfc657d9-bv95q" Nov 03 01:09:22.201299 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:22.200920 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff2ff304-350a-4e21-8415-1c28dcb90a7d-workload-certs\") pod \"e2e-extmod-a-821f5a-openshift-default-7bfc657d9-bv95q\" (UID: \"ff2ff304-350a-4e21-8415-1c28dcb90a7d\") " pod="openshift-ingress/e2e-extmod-a-821f5a-openshift-default-7bfc657d9-bv95q" Nov 03 01:09:22.220165 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:22.220049 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff2ff304-350a-4e21-8415-1c28dcb90a7d-istio-token\") pod \"e2e-extmod-a-821f5a-openshift-default-7bfc657d9-bv95q\" (UID: \"ff2ff304-350a-4e21-8415-1c28dcb90a7d\") " pod="openshift-ingress/e2e-extmod-a-821f5a-openshift-default-7bfc657d9-bv95q" Nov 03 01:09:22.242341 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:22.222653 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pj4gx\" (UniqueName: \"kubernetes.io/projected/ff2ff304-350a-4e21-8415-1c28dcb90a7d-kube-api-access-pj4gx\") pod \"e2e-extmod-a-821f5a-openshift-default-7bfc657d9-bv95q\" (UID: \"ff2ff304-350a-4e21-8415-1c28dcb90a7d\") " pod="openshift-ingress/e2e-extmod-a-821f5a-openshift-default-7bfc657d9-bv95q" Nov 03 01:09:22.242341 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:22.225834 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff2ff304-350a-4e21-8415-1c28dcb90a7d-istio-podinfo\") pod \"e2e-extmod-a-821f5a-openshift-default-7bfc657d9-bv95q\" (UID: \"ff2ff304-350a-4e21-8415-1c28dcb90a7d\") " pod="openshift-ingress/e2e-extmod-a-821f5a-openshift-default-7bfc657d9-bv95q" Nov 03 01:09:22.243043 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:22.242972 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff2ff304-350a-4e21-8415-1c28dcb90a7d-istio-envoy\") pod \"e2e-extmod-a-821f5a-openshift-default-7bfc657d9-bv95q\" (UID: \"ff2ff304-350a-4e21-8415-1c28dcb90a7d\") " pod="openshift-ingress/e2e-extmod-a-821f5a-openshift-default-7bfc657d9-bv95q" Nov 03 01:09:22.315469 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:22.315425 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-821f5a-openshift-default-7bfc657d9-bv95q" Nov 03 01:09:22.853172 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:22.853114 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:09:22.853354 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:22.853267 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:09:22.853593 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:22.853566 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:09:22.863118 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:22.862818 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-821f5a-openshift-default-7bfc657d9-bv95q"] Nov 03 01:09:22.869899 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:22.869859 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-821f5a-openshift-default-7bfc657d9-bv95q" event={"ID":"ff2ff304-350a-4e21-8415-1c28dcb90a7d","Type":"ContainerStarted","Data":"c08ab3d5b43e3348801137be9f4f578011e53c020cbf1309aac421a75e352225"} Nov 03 01:09:23.880176 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:23.880136 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-821f5a-openshift-default-7bfc657d9-bv95q" event={"ID":"ff2ff304-350a-4e21-8415-1c28dcb90a7d","Type":"ContainerStarted","Data":"06d83afdaa519e9974b129def6ada0b1fa89a128b24d2597a074e887c7956f04"} Nov 03 01:09:23.903304 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:23.903220 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-821f5a-openshift-default-7bfc657d9-bv95q" podStartSLOduration=2.9032044519999998 podStartE2EDuration="2.903204452s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:09:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:09:23.899027196 +0000 UTC m=+1706.070030309" watchObservedRunningTime="2026-11-03 01:09:23.903204452 +0000 UTC m=+1706.074207547" Nov 03 01:09:24.316407 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:24.316364 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-821f5a-openshift-default-7bfc657d9-bv95q" Nov 03 01:09:24.322428 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:24.322403 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-821f5a-openshift-default-7bfc657d9-bv95q" Nov 03 01:09:24.924938 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:24.924561 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-821f5a-openshift-default-7bfc657d9-bv95q" Nov 03 01:09:24.933206 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:24.932456 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-821f5a-openshift-default-7bfc657d9-bv95q" Nov 03 01:09:27.336052 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:27.336011 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-821f5a-84858c594c-pszhq"] Nov 03 01:09:27.342083 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:27.342044 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-821f5a-84858c594c-pszhq" Nov 03 01:09:27.347510 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:27.347485 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-821f5a\"" Nov 03 01:09:27.352572 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:27.352540 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-821f5a-84858c594c-pszhq"] Nov 03 01:09:27.363171 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:27.363140 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-821f5a-7655cbb57c-wgbsw"] Nov 03 01:09:27.386737 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:27.386707 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-821f5a-7655cbb57c-wgbsw"] Nov 03 01:09:27.386962 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:27.386775 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-821f5a-7655cbb57c-wgbsw" Nov 03 01:09:27.391229 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:27.391203 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-821f5a-dockercfg-ckr25\"" Nov 03 01:09:27.502192 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:27.502157 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/542b6309-1f43-413e-a448-b604a68a5047-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-821f5a-84858c594c-pszhq\" (UID: \"542b6309-1f43-413e-a448-b604a68a5047\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-821f5a-84858c594c-pszhq" Nov 03 01:09:27.502397 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:27.502324 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcff7\" (UniqueName: \"kubernetes.io/projected/61568421-54f2-4ba7-973f-8efa5ab1e805-kube-api-access-tcff7\") pod \"payload-processing-e2e-extmod-a-821f5a-7655cbb57c-wgbsw\" (UID: \"61568421-54f2-4ba7-973f-8efa5ab1e805\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-821f5a-7655cbb57c-wgbsw" Nov 03 01:09:27.502555 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:27.502541 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61568421-54f2-4ba7-973f-8efa5ab1e805-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-821f5a-7655cbb57c-wgbsw\" (UID: \"61568421-54f2-4ba7-973f-8efa5ab1e805\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-821f5a-7655cbb57c-wgbsw" Nov 03 01:09:27.502622 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:27.502597 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkrjq\" (UniqueName: \"kubernetes.io/projected/542b6309-1f43-413e-a448-b604a68a5047-kube-api-access-bkrjq\") pod \"payload-pre-processing-e2e-extmod-a-821f5a-84858c594c-pszhq\" (UID: \"542b6309-1f43-413e-a448-b604a68a5047\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-821f5a-84858c594c-pszhq" Nov 03 01:09:27.614358 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:27.604134 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcff7\" (UniqueName: \"kubernetes.io/projected/61568421-54f2-4ba7-973f-8efa5ab1e805-kube-api-access-tcff7\") pod \"payload-processing-e2e-extmod-a-821f5a-7655cbb57c-wgbsw\" (UID: \"61568421-54f2-4ba7-973f-8efa5ab1e805\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-821f5a-7655cbb57c-wgbsw" Nov 03 01:09:27.614358 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:27.604487 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61568421-54f2-4ba7-973f-8efa5ab1e805-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-821f5a-7655cbb57c-wgbsw\" (UID: \"61568421-54f2-4ba7-973f-8efa5ab1e805\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-821f5a-7655cbb57c-wgbsw" Nov 03 01:09:27.614358 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:27.604583 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkrjq\" (UniqueName: \"kubernetes.io/projected/542b6309-1f43-413e-a448-b604a68a5047-kube-api-access-bkrjq\") pod \"payload-pre-processing-e2e-extmod-a-821f5a-84858c594c-pszhq\" (UID: \"542b6309-1f43-413e-a448-b604a68a5047\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-821f5a-84858c594c-pszhq" Nov 03 01:09:27.614358 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:27.604699 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/542b6309-1f43-413e-a448-b604a68a5047-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-821f5a-84858c594c-pszhq\" (UID: \"542b6309-1f43-413e-a448-b604a68a5047\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-821f5a-84858c594c-pszhq" Nov 03 01:09:27.628299 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:27.626959 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61568421-54f2-4ba7-973f-8efa5ab1e805-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-821f5a-7655cbb57c-wgbsw\" (UID: \"61568421-54f2-4ba7-973f-8efa5ab1e805\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-821f5a-7655cbb57c-wgbsw" Nov 03 01:09:27.628299 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:27.627568 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/542b6309-1f43-413e-a448-b604a68a5047-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-821f5a-84858c594c-pszhq\" (UID: \"542b6309-1f43-413e-a448-b604a68a5047\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-821f5a-84858c594c-pszhq" Nov 03 01:09:27.658302 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:27.654371 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcff7\" (UniqueName: \"kubernetes.io/projected/61568421-54f2-4ba7-973f-8efa5ab1e805-kube-api-access-tcff7\") pod \"payload-processing-e2e-extmod-a-821f5a-7655cbb57c-wgbsw\" (UID: \"61568421-54f2-4ba7-973f-8efa5ab1e805\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-821f5a-7655cbb57c-wgbsw" Nov 03 01:09:27.683356 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:27.674160 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkrjq\" (UniqueName: \"kubernetes.io/projected/542b6309-1f43-413e-a448-b604a68a5047-kube-api-access-bkrjq\") pod \"payload-pre-processing-e2e-extmod-a-821f5a-84858c594c-pszhq\" (UID: \"542b6309-1f43-413e-a448-b604a68a5047\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-821f5a-84858c594c-pszhq" Nov 03 01:09:27.692296 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:27.690186 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-821f5a-84858c594c-pszhq" Nov 03 01:09:27.817236 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:27.814645 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-821f5a-7655cbb57c-wgbsw" Nov 03 01:09:28.058200 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:28.058157 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52aa8264-p5dtw"] Nov 03 01:09:28.066929 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:28.066888 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17a596cf-nb48g"] Nov 03 01:09:28.073365 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:28.073327 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52aa8264-p5dtw"] Nov 03 01:09:28.079253 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:28.079221 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17a596cf-nb48g"] Nov 03 01:09:28.204407 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:09:28.204375 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod542b6309_1f43_413e_a448_b604a68a5047.slice/crio-f81ddb6f19713e21176140a10b46565bae8a88267aee6d8c928df1f525db761b WatchSource:0}: Error finding container f81ddb6f19713e21176140a10b46565bae8a88267aee6d8c928df1f525db761b: Status 404 returned error can't find the container with id f81ddb6f19713e21176140a10b46565bae8a88267aee6d8c928df1f525db761b Nov 03 01:09:28.204849 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:28.204826 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-821f5a-84858c594c-pszhq"] Nov 03 01:09:28.215445 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:09:28.215419 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61568421_54f2_4ba7_973f_8efa5ab1e805.slice/crio-fc1b5528c63670d6f2e2a33df763c6ba60686b0a6bc5a3604c8a34846018cfef WatchSource:0}: Error finding container fc1b5528c63670d6f2e2a33df763c6ba60686b0a6bc5a3604c8a34846018cfef: Status 404 returned error can't find the container with id fc1b5528c63670d6f2e2a33df763c6ba60686b0a6bc5a3604c8a34846018cfef Nov 03 01:09:28.215788 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:28.215766 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-821f5a-7655cbb57c-wgbsw"] Nov 03 01:09:28.961636 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:28.961593 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-821f5a-84858c594c-pszhq" event={"ID":"542b6309-1f43-413e-a448-b604a68a5047","Type":"ContainerStarted","Data":"4ed7957e2ccc19fddec9b2dc34613e524e0d7cb91be6a03bc5b59c895350bcf5"} Nov 03 01:09:28.962103 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:28.961656 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-821f5a-84858c594c-pszhq" event={"ID":"542b6309-1f43-413e-a448-b604a68a5047","Type":"ContainerStarted","Data":"f81ddb6f19713e21176140a10b46565bae8a88267aee6d8c928df1f525db761b"} Nov 03 01:09:28.962103 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:28.961698 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-821f5a-84858c594c-pszhq" Nov 03 01:09:28.963652 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:28.963624 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-821f5a-7655cbb57c-wgbsw" event={"ID":"61568421-54f2-4ba7-973f-8efa5ab1e805","Type":"ContainerStarted","Data":"97b45b47e745ec0e2a4a7e0edf92413d611bb2ab85559d64835d2cbe6e726fcd"} Nov 03 01:09:28.963806 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:28.963665 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-821f5a-7655cbb57c-wgbsw" event={"ID":"61568421-54f2-4ba7-973f-8efa5ab1e805","Type":"ContainerStarted","Data":"fc1b5528c63670d6f2e2a33df763c6ba60686b0a6bc5a3604c8a34846018cfef"} Nov 03 01:09:28.963806 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:28.963745 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-821f5a-7655cbb57c-wgbsw" Nov 03 01:09:28.979849 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:28.979802 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-821f5a-84858c594c-pszhq" podStartSLOduration=1.979789587 podStartE2EDuration="1.979789587s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:09:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:09:28.977438951 +0000 UTC m=+1711.148442047" watchObservedRunningTime="2026-11-03 01:09:28.979789587 +0000 UTC m=+1711.150792716" Nov 03 01:09:29.005837 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:29.005769 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-821f5a-7655cbb57c-wgbsw" podStartSLOduration=2.005756244 podStartE2EDuration="2.005756244s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:09:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:09:29.004402904 +0000 UTC m=+1711.175406000" watchObservedRunningTime="2026-11-03 01:09:29.005756244 +0000 UTC m=+1711.176759340" Nov 03 01:09:30.187806 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:30.187763 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50854d7f-ce7a-47eb-9eef-358b178c0dd9" path="/var/lib/kubelet/pods/50854d7f-ce7a-47eb-9eef-358b178c0dd9/volumes" Nov 03 01:09:30.189744 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:30.189713 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6586d098-8183-4430-99b7-1b5e0973b0ab" path="/var/lib/kubelet/pods/6586d098-8183-4430-99b7-1b5e0973b0ab/volumes" Nov 03 01:09:38.023763 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:38.023727 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a025f30-zpcwt"] Nov 03 01:09:38.028419 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:38.028396 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a025f30-zpcwt"] Nov 03 01:09:39.026550 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:39.026495 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a23337fa-mj5q4"] Nov 03 01:09:39.031472 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:39.031438 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a23337fa-mj5q4"] Nov 03 01:09:39.982823 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:39.982789 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="490ffe57-6ed5-434f-9253-3cbc4b6a1a18" path="/var/lib/kubelet/pods/490ffe57-6ed5-434f-9253-3cbc4b6a1a18/volumes" Nov 03 01:09:39.983464 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:39.983448 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6e8ed68-3a01-46cf-874e-61ec9ef563c1" path="/var/lib/kubelet/pods/c6e8ed68-3a01-46cf-874e-61ec9ef563c1/volumes" Nov 03 01:09:40.156111 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:40.156034 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-821f5a-7655cbb57c-wgbsw" Nov 03 01:09:40.156609 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:40.156148 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-821f5a-84858c594c-pszhq" Nov 03 01:09:41.180216 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:41.180173 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jvqst"] Nov 03 01:09:41.202088 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:41.202047 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jvqst" Nov 03 01:09:41.208285 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:41.203813 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jvqst"] Nov 03 01:09:41.314638 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:41.314592 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8669aad1-230a-4b5f-be97-2e7ecd2a9de3-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jvqst\" (UID: \"8669aad1-230a-4b5f-be97-2e7ecd2a9de3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jvqst" Nov 03 01:09:41.314830 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:41.314733 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/8669aad1-230a-4b5f-be97-2e7ecd2a9de3-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jvqst\" (UID: \"8669aad1-230a-4b5f-be97-2e7ecd2a9de3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jvqst" Nov 03 01:09:41.314830 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:41.314816 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/8669aad1-230a-4b5f-be97-2e7ecd2a9de3-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jvqst\" (UID: \"8669aad1-230a-4b5f-be97-2e7ecd2a9de3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jvqst" Nov 03 01:09:41.314924 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:41.314883 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8669aad1-230a-4b5f-be97-2e7ecd2a9de3-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jvqst\" (UID: \"8669aad1-230a-4b5f-be97-2e7ecd2a9de3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jvqst" Nov 03 01:09:41.314970 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:41.314950 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/8669aad1-230a-4b5f-be97-2e7ecd2a9de3-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jvqst\" (UID: \"8669aad1-230a-4b5f-be97-2e7ecd2a9de3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jvqst" Nov 03 01:09:41.315074 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:41.315060 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/8669aad1-230a-4b5f-be97-2e7ecd2a9de3-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jvqst\" (UID: \"8669aad1-230a-4b5f-be97-2e7ecd2a9de3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jvqst" Nov 03 01:09:41.315123 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:41.315111 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzcsz\" (UniqueName: \"kubernetes.io/projected/8669aad1-230a-4b5f-be97-2e7ecd2a9de3-kube-api-access-nzcsz\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jvqst\" (UID: \"8669aad1-230a-4b5f-be97-2e7ecd2a9de3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jvqst" Nov 03 01:09:41.416698 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:41.416656 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8669aad1-230a-4b5f-be97-2e7ecd2a9de3-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jvqst\" (UID: \"8669aad1-230a-4b5f-be97-2e7ecd2a9de3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jvqst" Nov 03 01:09:41.416878 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:41.416736 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/8669aad1-230a-4b5f-be97-2e7ecd2a9de3-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jvqst\" (UID: \"8669aad1-230a-4b5f-be97-2e7ecd2a9de3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jvqst" Nov 03 01:09:41.416878 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:41.416801 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/8669aad1-230a-4b5f-be97-2e7ecd2a9de3-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jvqst\" (UID: \"8669aad1-230a-4b5f-be97-2e7ecd2a9de3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jvqst" Nov 03 01:09:41.416878 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:41.416852 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8669aad1-230a-4b5f-be97-2e7ecd2a9de3-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jvqst\" (UID: \"8669aad1-230a-4b5f-be97-2e7ecd2a9de3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jvqst" Nov 03 01:09:41.417040 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:41.416881 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/8669aad1-230a-4b5f-be97-2e7ecd2a9de3-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jvqst\" (UID: \"8669aad1-230a-4b5f-be97-2e7ecd2a9de3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jvqst" Nov 03 01:09:41.417040 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:41.416998 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/8669aad1-230a-4b5f-be97-2e7ecd2a9de3-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jvqst\" (UID: \"8669aad1-230a-4b5f-be97-2e7ecd2a9de3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jvqst" Nov 03 01:09:41.417145 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:41.417071 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzcsz\" (UniqueName: \"kubernetes.io/projected/8669aad1-230a-4b5f-be97-2e7ecd2a9de3-kube-api-access-nzcsz\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jvqst\" (UID: \"8669aad1-230a-4b5f-be97-2e7ecd2a9de3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jvqst" Nov 03 01:09:41.417636 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:41.417610 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/8669aad1-230a-4b5f-be97-2e7ecd2a9de3-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jvqst\" (UID: \"8669aad1-230a-4b5f-be97-2e7ecd2a9de3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jvqst" Nov 03 01:09:41.419526 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:41.419495 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8669aad1-230a-4b5f-be97-2e7ecd2a9de3-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jvqst\" (UID: \"8669aad1-230a-4b5f-be97-2e7ecd2a9de3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jvqst" Nov 03 01:09:41.419913 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:41.419887 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/8669aad1-230a-4b5f-be97-2e7ecd2a9de3-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jvqst\" (UID: \"8669aad1-230a-4b5f-be97-2e7ecd2a9de3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jvqst" Nov 03 01:09:41.420013 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:41.419952 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/8669aad1-230a-4b5f-be97-2e7ecd2a9de3-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jvqst\" (UID: \"8669aad1-230a-4b5f-be97-2e7ecd2a9de3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jvqst" Nov 03 01:09:41.420356 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:41.420338 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/8669aad1-230a-4b5f-be97-2e7ecd2a9de3-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jvqst\" (UID: \"8669aad1-230a-4b5f-be97-2e7ecd2a9de3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jvqst" Nov 03 01:09:41.425750 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:41.425721 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzcsz\" (UniqueName: \"kubernetes.io/projected/8669aad1-230a-4b5f-be97-2e7ecd2a9de3-kube-api-access-nzcsz\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jvqst\" (UID: \"8669aad1-230a-4b5f-be97-2e7ecd2a9de3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jvqst" Nov 03 01:09:41.425832 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:41.425816 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8669aad1-230a-4b5f-be97-2e7ecd2a9de3-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jvqst\" (UID: \"8669aad1-230a-4b5f-be97-2e7ecd2a9de3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jvqst" Nov 03 01:09:41.534424 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:41.534386 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jvqst" Nov 03 01:09:42.230557 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:42.207057 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jvqst"] Nov 03 01:09:42.251581 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:42.244694 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:09:42.251581 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:42.244792 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:09:42.647300 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:42.642601 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jvqst" event={"ID":"8669aad1-230a-4b5f-be97-2e7ecd2a9de3","Type":"ContainerStarted","Data":"b49b100725f54df2c201f34a83803ee5f8796bcc1ecb6c9eed02761ba956f7eb"} Nov 03 01:09:42.861302 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:42.860293 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8gm8n" podUID="2d367e12-e46f-4a68-9825-9b1c934d4d12" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:09:43.116298 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:43.089530 4421 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-650be0/simulator-w4-650be0-kserve-7bc588774f-gvz2k" podUID="a827fa88-f078-4312-8f12-2c799c11b461" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": context deadline exceeded" Nov 03 01:09:43.116298 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:43.090574 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w0-50e623/auth-test-model-50e623-kserve-7db6664d89-zdq97" podUID="e387182e-8e68-421f-b3ec-143283631b5e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.223:8000/ready\": context deadline exceeded" Nov 03 01:09:43.131304 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:43.130330 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3b8268a-985qs"] Nov 03 01:09:43.271053 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:43.271009 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3b8268a-985qs"] Nov 03 01:09:43.662373 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:43.662264 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jvqst" event={"ID":"8669aad1-230a-4b5f-be97-2e7ecd2a9de3","Type":"ContainerStarted","Data":"933aca6d122054cbdea416716a95e170fef97edca0e9b76c7491b9be36487dc9"} Nov 03 01:09:43.662665 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:43.662638 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jvqst" Nov 03 01:09:43.666768 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:43.666743 4421 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-jvqst container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 01:09:43.666894 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:43.666786 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jvqst" podUID="8669aad1-230a-4b5f-be97-2e7ecd2a9de3" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 01:09:43.696131 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:43.696067 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jvqst" podStartSLOduration=2.696050578 podStartE2EDuration="2.696050578s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:09:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:09:43.688817011 +0000 UTC m=+1725.859820108" watchObservedRunningTime="2026-11-03 01:09:43.696050578 +0000 UTC m=+1725.867053677" Nov 03 01:09:43.984574 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:43.984492 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b6b9bed-b022-4614-923a-bcb39e9f5073" path="/var/lib/kubelet/pods/3b6b9bed-b022-4614-923a-bcb39e9f5073/volumes" Nov 03 01:09:44.672567 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:09:44.672530 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 01:09:44.673653 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:44.673631 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jvqst" Nov 03 01:09:45.596757 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:45.596722 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e51c8244-m4r72"] Nov 03 01:09:45.603757 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:45.603670 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e51c8244-m4r72" Nov 03 01:09:45.608145 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:45.608123 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e51c8244-m4r72"] Nov 03 01:09:45.608553 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:45.608527 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:09:45.608648 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:45.608537 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6q4xp\"" Nov 03 01:09:45.608648 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:45.608625 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 01:09:45.608648 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:45.608633 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 01:09:45.702821 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:45.702787 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f17732b-cea6-4c56-9f69-4796e6c86ed0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e51c8244-m4r72\" (UID: \"5f17732b-cea6-4c56-9f69-4796e6c86ed0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e51c8244-m4r72" Nov 03 01:09:45.703238 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:45.702943 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dkk5\" (UniqueName: \"kubernetes.io/projected/5f17732b-cea6-4c56-9f69-4796e6c86ed0-kube-api-access-5dkk5\") pod \"maas-api-revoke-sub-e51c8244-m4r72\" (UID: \"5f17732b-cea6-4c56-9f69-4796e6c86ed0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e51c8244-m4r72" Nov 03 01:09:45.703238 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:45.703114 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f17732b-cea6-4c56-9f69-4796e6c86ed0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e51c8244-m4r72\" (UID: \"5f17732b-cea6-4c56-9f69-4796e6c86ed0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e51c8244-m4r72" Nov 03 01:09:45.804795 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:45.804754 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f17732b-cea6-4c56-9f69-4796e6c86ed0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e51c8244-m4r72\" (UID: \"5f17732b-cea6-4c56-9f69-4796e6c86ed0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e51c8244-m4r72" Nov 03 01:09:45.805001 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:45.804972 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f17732b-cea6-4c56-9f69-4796e6c86ed0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e51c8244-m4r72\" (UID: \"5f17732b-cea6-4c56-9f69-4796e6c86ed0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e51c8244-m4r72" Nov 03 01:09:45.805215 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:45.805193 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dkk5\" (UniqueName: \"kubernetes.io/projected/5f17732b-cea6-4c56-9f69-4796e6c86ed0-kube-api-access-5dkk5\") pod \"maas-api-revoke-sub-e51c8244-m4r72\" (UID: \"5f17732b-cea6-4c56-9f69-4796e6c86ed0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e51c8244-m4r72" Nov 03 01:09:45.805775 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:45.805752 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f17732b-cea6-4c56-9f69-4796e6c86ed0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e51c8244-m4r72\" (UID: \"5f17732b-cea6-4c56-9f69-4796e6c86ed0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e51c8244-m4r72" Nov 03 01:09:45.807959 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:45.807937 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f17732b-cea6-4c56-9f69-4796e6c86ed0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e51c8244-m4r72\" (UID: \"5f17732b-cea6-4c56-9f69-4796e6c86ed0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e51c8244-m4r72" Nov 03 01:09:45.813421 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:45.813396 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dkk5\" (UniqueName: \"kubernetes.io/projected/5f17732b-cea6-4c56-9f69-4796e6c86ed0-kube-api-access-5dkk5\") pod \"maas-api-revoke-sub-e51c8244-m4r72\" (UID: \"5f17732b-cea6-4c56-9f69-4796e6c86ed0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e51c8244-m4r72" Nov 03 01:09:45.941502 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:45.935079 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e51c8244-m4r72" Nov 03 01:09:46.263610 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:09:46.263568 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f17732b_cea6_4c56_9f69_4796e6c86ed0.slice/crio-d284832d0d2bb1ad0160293ff7a89924093416d1a40dab5db14d35728a36d81a WatchSource:0}: Error finding container d284832d0d2bb1ad0160293ff7a89924093416d1a40dab5db14d35728a36d81a: Status 404 returned error can't find the container with id d284832d0d2bb1ad0160293ff7a89924093416d1a40dab5db14d35728a36d81a Nov 03 01:09:46.263742 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:46.263701 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e51c8244-m4r72"] Nov 03 01:09:46.694421 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:46.694386 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e51c8244-m4r72" event={"ID":"5f17732b-cea6-4c56-9f69-4796e6c86ed0","Type":"ContainerStarted","Data":"d284832d0d2bb1ad0160293ff7a89924093416d1a40dab5db14d35728a36d81a"} Nov 03 01:09:47.706293 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:47.706190 4421 generic.go:356] "Generic (PLEG): container finished" podID="5f17732b-cea6-4c56-9f69-4796e6c86ed0" containerID="9a4af3515a2e301e5de6d37314f1f35596dc6f1fd40ee6b065ed7877813b235b" exitCode=0 Nov 03 01:09:47.706714 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:47.706314 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e51c8244-m4r72" event={"ID":"5f17732b-cea6-4c56-9f69-4796e6c86ed0","Type":"ContainerDied","Data":"9a4af3515a2e301e5de6d37314f1f35596dc6f1fd40ee6b065ed7877813b235b"} Nov 03 01:09:48.905906 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:48.905877 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e51c8244-m4r72" Nov 03 01:09:49.006320 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:49.004706 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f17732b-cea6-4c56-9f69-4796e6c86ed0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f17732b-cea6-4c56-9f69-4796e6c86ed0-maas-api-openshift-ca\") pod \"5f17732b-cea6-4c56-9f69-4796e6c86ed0\" (UID: \"5f17732b-cea6-4c56-9f69-4796e6c86ed0\") " Nov 03 01:09:49.006320 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:49.004784 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f17732b-cea6-4c56-9f69-4796e6c86ed0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f17732b-cea6-4c56-9f69-4796e6c86ed0-maas-api-xks-ca\") pod \"5f17732b-cea6-4c56-9f69-4796e6c86ed0\" (UID: \"5f17732b-cea6-4c56-9f69-4796e6c86ed0\") " Nov 03 01:09:49.006320 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:49.005162 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f17732b-cea6-4c56-9f69-4796e6c86ed0-kube-api-access-5dkk5\" (UniqueName: \"kubernetes.io/projected/5f17732b-cea6-4c56-9f69-4796e6c86ed0-kube-api-access-5dkk5\") pod \"5f17732b-cea6-4c56-9f69-4796e6c86ed0\" (UID: \"5f17732b-cea6-4c56-9f69-4796e6c86ed0\") " Nov 03 01:09:49.007340 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:49.007244 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f17732b-cea6-4c56-9f69-4796e6c86ed0-maas-api-openshift-ca" pod "5f17732b-cea6-4c56-9f69-4796e6c86ed0" (UID: "5f17732b-cea6-4c56-9f69-4796e6c86ed0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:09:49.009583 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:49.009550 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f17732b-cea6-4c56-9f69-4796e6c86ed0-kube-api-access-5dkk5" pod "5f17732b-cea6-4c56-9f69-4796e6c86ed0" (UID: "5f17732b-cea6-4c56-9f69-4796e6c86ed0"). InnerVolumeSpecName "kube-api-access-5dkk5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:09:49.009793 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:49.009653 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f17732b-cea6-4c56-9f69-4796e6c86ed0-maas-api-xks-ca" pod "5f17732b-cea6-4c56-9f69-4796e6c86ed0" (UID: "5f17732b-cea6-4c56-9f69-4796e6c86ed0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:09:49.107692 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:49.107658 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5dkk5\" (UniqueName: \"kubernetes.io/projected/5f17732b-cea6-4c56-9f69-4796e6c86ed0-kube-api-access-5dkk5\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:09:49.107692 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:49.107689 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f17732b-cea6-4c56-9f69-4796e6c86ed0-maas-api-openshift-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:09:49.107692 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:49.107699 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f17732b-cea6-4c56-9f69-4796e6c86ed0-maas-api-xks-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:09:49.729602 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:49.729569 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e51c8244-m4r72" Nov 03 01:09:49.729795 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:49.729560 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e51c8244-m4r72" event={"ID":"5f17732b-cea6-4c56-9f69-4796e6c86ed0","Type":"ContainerDied","Data":"d284832d0d2bb1ad0160293ff7a89924093416d1a40dab5db14d35728a36d81a"} Nov 03 01:09:49.729795 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:49.729723 4421 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d284832d0d2bb1ad0160293ff7a89924093416d1a40dab5db14d35728a36d81a" Nov 03 01:09:52.027259 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:52.027224 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b050a313-fz7x5"] Nov 03 01:09:52.031954 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:52.031925 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b050a313-fz7x5"] Nov 03 01:09:53.988940 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:53.988903 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf46dbdd-d501-400c-b82a-aaf689363bab" path="/var/lib/kubelet/pods/bf46dbdd-d501-400c-b82a-aaf689363bab/volumes" Nov 03 01:09:54.032687 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:54.032642 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-97d101cc-vf5cc"] Nov 03 01:09:54.037156 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:54.037129 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-97d101cc-vf5cc"] Nov 03 01:09:55.833776 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:55.833742 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29819d6d-h2qkg"] Nov 03 01:09:55.834655 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:55.834635 4421 cpu_manager.go:414] "containerMap: removing container" podUID="5f17732b-cea6-4c56-9f69-4796e6c86ed0" containerName="revoke-keys" Nov 03 01:09:55.834655 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:55.834655 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f17732b-cea6-4c56-9f69-4796e6c86ed0" containerName="revoke-keys" Nov 03 01:09:55.834822 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:55.834806 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f17732b-cea6-4c56-9f69-4796e6c86ed0" containerName="revoke-keys" Nov 03 01:09:55.839197 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:55.839167 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29819d6d-h2qkg" Nov 03 01:09:55.842746 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:55.842722 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:09:55.843785 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:55.843761 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 01:09:55.843930 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:55.843783 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 01:09:55.844012 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:55.843996 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6q4xp\"" Nov 03 01:09:55.846428 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:55.846405 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29819d6d-h2qkg"] Nov 03 01:09:55.982766 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:55.982734 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32b8c1b0-8e81-4430-840b-ca5d03a7195d" path="/var/lib/kubelet/pods/32b8c1b0-8e81-4430-840b-ca5d03a7195d/volumes" Nov 03 01:09:55.994233 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:55.994205 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52975b66-9b47-417e-b48f-f6d460fbbcd0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-29819d6d-h2qkg\" (UID: \"52975b66-9b47-417e-b48f-f6d460fbbcd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29819d6d-h2qkg" Nov 03 01:09:55.994409 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:55.994391 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52975b66-9b47-417e-b48f-f6d460fbbcd0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-29819d6d-h2qkg\" (UID: \"52975b66-9b47-417e-b48f-f6d460fbbcd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29819d6d-h2qkg" Nov 03 01:09:55.994553 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:55.994537 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6qrd\" (UniqueName: \"kubernetes.io/projected/52975b66-9b47-417e-b48f-f6d460fbbcd0-kube-api-access-q6qrd\") pod \"maas-api-revoke-sub-29819d6d-h2qkg\" (UID: \"52975b66-9b47-417e-b48f-f6d460fbbcd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29819d6d-h2qkg" Nov 03 01:09:56.095842 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:56.095744 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52975b66-9b47-417e-b48f-f6d460fbbcd0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-29819d6d-h2qkg\" (UID: \"52975b66-9b47-417e-b48f-f6d460fbbcd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29819d6d-h2qkg" Nov 03 01:09:56.096016 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:56.095871 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6qrd\" (UniqueName: \"kubernetes.io/projected/52975b66-9b47-417e-b48f-f6d460fbbcd0-kube-api-access-q6qrd\") pod \"maas-api-revoke-sub-29819d6d-h2qkg\" (UID: \"52975b66-9b47-417e-b48f-f6d460fbbcd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29819d6d-h2qkg" Nov 03 01:09:56.096059 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:56.096033 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52975b66-9b47-417e-b48f-f6d460fbbcd0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-29819d6d-h2qkg\" (UID: \"52975b66-9b47-417e-b48f-f6d460fbbcd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29819d6d-h2qkg" Nov 03 01:09:56.096556 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:56.096526 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52975b66-9b47-417e-b48f-f6d460fbbcd0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-29819d6d-h2qkg\" (UID: \"52975b66-9b47-417e-b48f-f6d460fbbcd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29819d6d-h2qkg" Nov 03 01:09:56.098680 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:56.098655 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52975b66-9b47-417e-b48f-f6d460fbbcd0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-29819d6d-h2qkg\" (UID: \"52975b66-9b47-417e-b48f-f6d460fbbcd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29819d6d-h2qkg" Nov 03 01:09:56.104752 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:56.104728 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6qrd\" (UniqueName: \"kubernetes.io/projected/52975b66-9b47-417e-b48f-f6d460fbbcd0-kube-api-access-q6qrd\") pod \"maas-api-revoke-sub-29819d6d-h2qkg\" (UID: \"52975b66-9b47-417e-b48f-f6d460fbbcd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29819d6d-h2qkg" Nov 03 01:09:56.156596 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:56.156563 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29819d6d-h2qkg" Nov 03 01:09:56.449922 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:09:56.449878 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52975b66_9b47_417e_b48f_f6d460fbbcd0.slice/crio-1aa356e2c45320d947433bb4d9573c8228f7aad3005c7fc083cb7b425b4161c9 WatchSource:0}: Error finding container 1aa356e2c45320d947433bb4d9573c8228f7aad3005c7fc083cb7b425b4161c9: Status 404 returned error can't find the container with id 1aa356e2c45320d947433bb4d9573c8228f7aad3005c7fc083cb7b425b4161c9 Nov 03 01:09:56.450462 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:56.450434 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29819d6d-h2qkg"] Nov 03 01:09:56.814478 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:56.814439 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29819d6d-h2qkg" event={"ID":"52975b66-9b47-417e-b48f-f6d460fbbcd0","Type":"ContainerStarted","Data":"1aa356e2c45320d947433bb4d9573c8228f7aad3005c7fc083cb7b425b4161c9"} Nov 03 01:09:57.830386 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:57.830347 4421 generic.go:356] "Generic (PLEG): container finished" podID="52975b66-9b47-417e-b48f-f6d460fbbcd0" containerID="252e7fa5db41b396fb3f979d3fa010e54ab33429947cae9b2951732b0a36914a" exitCode=0 Nov 03 01:09:57.830808 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:57.830431 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29819d6d-h2qkg" event={"ID":"52975b66-9b47-417e-b48f-f6d460fbbcd0","Type":"ContainerDied","Data":"252e7fa5db41b396fb3f979d3fa010e54ab33429947cae9b2951732b0a36914a"} Nov 03 01:09:59.011466 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:59.011436 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29819d6d-h2qkg" Nov 03 01:09:59.031560 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:59.031518 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c50969f-s5cbm"] Nov 03 01:09:59.043033 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:59.042996 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c50969f-s5cbm"] Nov 03 01:09:59.157347 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:59.154991 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52975b66-9b47-417e-b48f-f6d460fbbcd0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52975b66-9b47-417e-b48f-f6d460fbbcd0-maas-api-openshift-ca\") pod \"52975b66-9b47-417e-b48f-f6d460fbbcd0\" (UID: \"52975b66-9b47-417e-b48f-f6d460fbbcd0\") " Nov 03 01:09:59.170298 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:59.163899 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52975b66-9b47-417e-b48f-f6d460fbbcd0-kube-api-access-q6qrd\" (UniqueName: \"kubernetes.io/projected/52975b66-9b47-417e-b48f-f6d460fbbcd0-kube-api-access-q6qrd\") pod \"52975b66-9b47-417e-b48f-f6d460fbbcd0\" (UID: \"52975b66-9b47-417e-b48f-f6d460fbbcd0\") " Nov 03 01:09:59.170298 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:59.164061 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/52975b66-9b47-417e-b48f-f6d460fbbcd0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52975b66-9b47-417e-b48f-f6d460fbbcd0-maas-api-xks-ca\") pod \"52975b66-9b47-417e-b48f-f6d460fbbcd0\" (UID: \"52975b66-9b47-417e-b48f-f6d460fbbcd0\") " Nov 03 01:09:59.179992 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:59.174440 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52975b66-9b47-417e-b48f-f6d460fbbcd0-maas-api-xks-ca" pod "52975b66-9b47-417e-b48f-f6d460fbbcd0" (UID: "52975b66-9b47-417e-b48f-f6d460fbbcd0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:09:59.183775 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:59.183716 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52975b66-9b47-417e-b48f-f6d460fbbcd0-kube-api-access-q6qrd" pod "52975b66-9b47-417e-b48f-f6d460fbbcd0" (UID: "52975b66-9b47-417e-b48f-f6d460fbbcd0"). InnerVolumeSpecName "kube-api-access-q6qrd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:09:59.192013 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:59.191941 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52975b66-9b47-417e-b48f-f6d460fbbcd0-maas-api-openshift-ca" pod "52975b66-9b47-417e-b48f-f6d460fbbcd0" (UID: "52975b66-9b47-417e-b48f-f6d460fbbcd0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:09:59.266887 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:59.266850 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52975b66-9b47-417e-b48f-f6d460fbbcd0-maas-api-xks-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:09:59.266887 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:59.266885 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52975b66-9b47-417e-b48f-f6d460fbbcd0-maas-api-openshift-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:09:59.266887 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:59.266896 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6qrd\" (UniqueName: \"kubernetes.io/projected/52975b66-9b47-417e-b48f-f6d460fbbcd0-kube-api-access-q6qrd\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:09:59.848469 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:59.848442 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29819d6d-h2qkg" Nov 03 01:09:59.848671 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:59.848473 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29819d6d-h2qkg" event={"ID":"52975b66-9b47-417e-b48f-f6d460fbbcd0","Type":"ContainerDied","Data":"1aa356e2c45320d947433bb4d9573c8228f7aad3005c7fc083cb7b425b4161c9"} Nov 03 01:09:59.848671 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:59.848527 4421 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1aa356e2c45320d947433bb4d9573c8228f7aad3005c7fc083cb7b425b4161c9" Nov 03 01:09:59.981858 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:09:59.981828 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8a50bdf-26dd-49d9-a899-a9dfec083da4" path="/var/lib/kubelet/pods/f8a50bdf-26dd-49d9-a899-a9dfec083da4/volumes" Nov 03 01:10:00.795919 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:00.795878 4421 scope.go:122] "RemoveContainer" containerID="c85acad2d042be5cbaeae365de5cf3f9d3418bc577a72985dd1e53dc7fd85259" Nov 03 01:10:00.808413 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:00.808387 4421 scope.go:122] "RemoveContainer" containerID="58e09291833fd3be000d16983f800524f4af36d51ae49c7b61c5389695631d79" Nov 03 01:10:00.857741 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:00.857712 4421 scope.go:122] "RemoveContainer" containerID="36e5aecd991fdc78ce856619b8d0b93118d230fad7ca6c5eaa63ed0bd80204e0" Nov 03 01:10:00.869315 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:00.869261 4421 scope.go:122] "RemoveContainer" containerID="685455d820d4891f21dc257ab41b7d22befe708357b99ac1d5f0536e34617f43" Nov 03 01:10:00.880242 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:00.880223 4421 scope.go:122] "RemoveContainer" containerID="acb81fddd8a1546c0524b7590ee8369a268d40b88a5c0fa8abc1a4f80a1d53e9" Nov 03 01:10:00.890526 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:00.890504 4421 scope.go:122] "RemoveContainer" containerID="40beb931d095cbd2da03c06c770f6ee0ecf0c04ed98c516087c46962b09fc06c" Nov 03 01:10:00.922929 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:00.922909 4421 scope.go:122] "RemoveContainer" containerID="6c9441f81a99800521b0bea31448f770fe1ef97866c8149dbaab8ff7b06e64f5" Nov 03 01:10:00.935130 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:00.935108 4421 scope.go:122] "RemoveContainer" containerID="4de6da265253ba73223a3bbf7ab5220d340592ac13ed66e28f4ff9e624e32bb1" Nov 03 01:10:00.946653 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:00.946633 4421 scope.go:122] "RemoveContainer" containerID="e396a3c0ced74d47e6d14ff45cc2d83b5863a36b517cd702248c2a11431c4e9b" Nov 03 01:10:00.957902 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:00.957883 4421 scope.go:122] "RemoveContainer" containerID="36b6d60e166b775d8583f375d29aac8b2f853d0f320d40e410178f0e047607fb" Nov 03 01:10:02.543215 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:02.543177 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-7hhxt"] Nov 03 01:10:02.544047 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:02.544030 4421 cpu_manager.go:414] "containerMap: removing container" podUID="52975b66-9b47-417e-b48f-f6d460fbbcd0" containerName="revoke-keys" Nov 03 01:10:02.544097 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:02.544048 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52975b66-9b47-417e-b48f-f6d460fbbcd0" containerName="revoke-keys" Nov 03 01:10:02.544221 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:02.544211 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="52975b66-9b47-417e-b48f-f6d460fbbcd0" containerName="revoke-keys" Nov 03 01:10:02.550171 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:02.550147 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-7hhxt" Nov 03 01:10:02.556394 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:02.556369 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-4a697f-openshift-default-dockercfg-wr6rf\"" Nov 03 01:10:02.560732 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:02.560698 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-7hhxt"] Nov 03 01:10:02.588944 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:02.588908 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2995b3fc-d4cd-49ba-a988-470f220f30f3-istio-token\") pod \"e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-7hhxt\" (UID: \"2995b3fc-d4cd-49ba-a988-470f220f30f3\") " pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-7hhxt" Nov 03 01:10:02.589138 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:02.588949 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2995b3fc-d4cd-49ba-a988-470f220f30f3-workload-certs\") pod \"e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-7hhxt\" (UID: \"2995b3fc-d4cd-49ba-a988-470f220f30f3\") " pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-7hhxt" Nov 03 01:10:02.589138 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:02.589120 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2995b3fc-d4cd-49ba-a988-470f220f30f3-istio-envoy\") pod \"e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-7hhxt\" (UID: \"2995b3fc-d4cd-49ba-a988-470f220f30f3\") " pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-7hhxt" Nov 03 01:10:02.589256 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:02.589182 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2995b3fc-d4cd-49ba-a988-470f220f30f3-istio-data\") pod \"e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-7hhxt\" (UID: \"2995b3fc-d4cd-49ba-a988-470f220f30f3\") " pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-7hhxt" Nov 03 01:10:02.589256 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:02.589244 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2995b3fc-d4cd-49ba-a988-470f220f30f3-istio-podinfo\") pod \"e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-7hhxt\" (UID: \"2995b3fc-d4cd-49ba-a988-470f220f30f3\") " pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-7hhxt" Nov 03 01:10:02.589951 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:02.589322 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2995b3fc-d4cd-49ba-a988-470f220f30f3-istiod-ca-cert\") pod \"e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-7hhxt\" (UID: \"2995b3fc-d4cd-49ba-a988-470f220f30f3\") " pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-7hhxt" Nov 03 01:10:02.589951 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:02.589373 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gq4hc\" (UniqueName: \"kubernetes.io/projected/2995b3fc-d4cd-49ba-a988-470f220f30f3-kube-api-access-gq4hc\") pod \"e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-7hhxt\" (UID: \"2995b3fc-d4cd-49ba-a988-470f220f30f3\") " pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-7hhxt" Nov 03 01:10:02.589951 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:02.589403 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2995b3fc-d4cd-49ba-a988-470f220f30f3-workload-socket\") pod \"e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-7hhxt\" (UID: \"2995b3fc-d4cd-49ba-a988-470f220f30f3\") " pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-7hhxt" Nov 03 01:10:02.589951 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:02.589490 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2995b3fc-d4cd-49ba-a988-470f220f30f3-credential-socket\") pod \"e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-7hhxt\" (UID: \"2995b3fc-d4cd-49ba-a988-470f220f30f3\") " pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-7hhxt" Nov 03 01:10:02.730538 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:02.730495 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2995b3fc-d4cd-49ba-a988-470f220f30f3-istio-podinfo\") pod \"e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-7hhxt\" (UID: \"2995b3fc-d4cd-49ba-a988-470f220f30f3\") " pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-7hhxt" Nov 03 01:10:02.740502 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:02.740448 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2995b3fc-d4cd-49ba-a988-470f220f30f3-istiod-ca-cert\") pod \"e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-7hhxt\" (UID: \"2995b3fc-d4cd-49ba-a988-470f220f30f3\") " pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-7hhxt" Nov 03 01:10:02.747356 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:02.742798 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gq4hc\" (UniqueName: \"kubernetes.io/projected/2995b3fc-d4cd-49ba-a988-470f220f30f3-kube-api-access-gq4hc\") pod \"e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-7hhxt\" (UID: \"2995b3fc-d4cd-49ba-a988-470f220f30f3\") " pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-7hhxt" Nov 03 01:10:02.747356 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:02.742848 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2995b3fc-d4cd-49ba-a988-470f220f30f3-workload-socket\") pod \"e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-7hhxt\" (UID: \"2995b3fc-d4cd-49ba-a988-470f220f30f3\") " pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-7hhxt" Nov 03 01:10:02.747356 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:02.742981 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2995b3fc-d4cd-49ba-a988-470f220f30f3-credential-socket\") pod \"e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-7hhxt\" (UID: \"2995b3fc-d4cd-49ba-a988-470f220f30f3\") " pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-7hhxt" Nov 03 01:10:02.747356 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:02.743190 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2995b3fc-d4cd-49ba-a988-470f220f30f3-istio-token\") pod \"e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-7hhxt\" (UID: \"2995b3fc-d4cd-49ba-a988-470f220f30f3\") " pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-7hhxt" Nov 03 01:10:02.747356 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:02.743230 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2995b3fc-d4cd-49ba-a988-470f220f30f3-workload-certs\") pod \"e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-7hhxt\" (UID: \"2995b3fc-d4cd-49ba-a988-470f220f30f3\") " pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-7hhxt" Nov 03 01:10:02.781295 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:02.757717 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2995b3fc-d4cd-49ba-a988-470f220f30f3-credential-socket\") pod \"e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-7hhxt\" (UID: \"2995b3fc-d4cd-49ba-a988-470f220f30f3\") " pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-7hhxt" Nov 03 01:10:02.781899 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:02.742678 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2995b3fc-d4cd-49ba-a988-470f220f30f3-istiod-ca-cert\") pod \"e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-7hhxt\" (UID: \"2995b3fc-d4cd-49ba-a988-470f220f30f3\") " pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-7hhxt" Nov 03 01:10:02.782794 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:02.782764 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2995b3fc-d4cd-49ba-a988-470f220f30f3-istio-envoy\") pod \"e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-7hhxt\" (UID: \"2995b3fc-d4cd-49ba-a988-470f220f30f3\") " pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-7hhxt" Nov 03 01:10:02.782964 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:02.782934 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2995b3fc-d4cd-49ba-a988-470f220f30f3-workload-socket\") pod \"e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-7hhxt\" (UID: \"2995b3fc-d4cd-49ba-a988-470f220f30f3\") " pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-7hhxt" Nov 03 01:10:02.818448 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:02.783025 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2995b3fc-d4cd-49ba-a988-470f220f30f3-istio-data\") pod \"e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-7hhxt\" (UID: \"2995b3fc-d4cd-49ba-a988-470f220f30f3\") " pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-7hhxt" Nov 03 01:10:02.818927 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:02.783329 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2995b3fc-d4cd-49ba-a988-470f220f30f3-istio-data\") pod \"e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-7hhxt\" (UID: \"2995b3fc-d4cd-49ba-a988-470f220f30f3\") " pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-7hhxt" Nov 03 01:10:02.831405 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:02.783533 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2995b3fc-d4cd-49ba-a988-470f220f30f3-workload-certs\") pod \"e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-7hhxt\" (UID: \"2995b3fc-d4cd-49ba-a988-470f220f30f3\") " pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-7hhxt" Nov 03 01:10:02.966562 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:02.949039 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gq4hc\" (UniqueName: \"kubernetes.io/projected/2995b3fc-d4cd-49ba-a988-470f220f30f3-kube-api-access-gq4hc\") pod \"e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-7hhxt\" (UID: \"2995b3fc-d4cd-49ba-a988-470f220f30f3\") " pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-7hhxt" Nov 03 01:10:02.966562 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:02.950559 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2995b3fc-d4cd-49ba-a988-470f220f30f3-istio-podinfo\") pod \"e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-7hhxt\" (UID: \"2995b3fc-d4cd-49ba-a988-470f220f30f3\") " pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-7hhxt" Nov 03 01:10:02.966562 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:02.957011 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2995b3fc-d4cd-49ba-a988-470f220f30f3-istio-token\") pod \"e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-7hhxt\" (UID: \"2995b3fc-d4cd-49ba-a988-470f220f30f3\") " pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-7hhxt" Nov 03 01:10:02.966562 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:02.957307 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2995b3fc-d4cd-49ba-a988-470f220f30f3-istio-envoy\") pod \"e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-7hhxt\" (UID: \"2995b3fc-d4cd-49ba-a988-470f220f30f3\") " pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-7hhxt" Nov 03 01:10:03.194356 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:03.179597 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-7hhxt" Nov 03 01:10:04.287678 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:04.287639 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-7hhxt"] Nov 03 01:10:04.288340 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:10:04.288254 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2995b3fc_d4cd_49ba_a988_470f220f30f3.slice/crio-cadd4bf7a0abdff58e4f19c35343660f4e10bbdc4acc4d28948379faf91bc44e WatchSource:0}: Error finding container cadd4bf7a0abdff58e4f19c35343660f4e10bbdc4acc4d28948379faf91bc44e: Status 404 returned error can't find the container with id cadd4bf7a0abdff58e4f19c35343660f4e10bbdc4acc4d28948379faf91bc44e Nov 03 01:10:04.291817 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:04.291778 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:10:04.292106 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:04.291869 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:10:04.292106 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:04.291926 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:10:05.098304 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:05.094801 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-7hhxt" event={"ID":"2995b3fc-d4cd-49ba-a988-470f220f30f3","Type":"ContainerStarted","Data":"8c4a39e5f8ae3453ec86398c3489eae2a8a04cf5e22eb2bcc3c2c3acfa27b030"} Nov 03 01:10:05.098304 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:05.094878 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-7hhxt" event={"ID":"2995b3fc-d4cd-49ba-a988-470f220f30f3","Type":"ContainerStarted","Data":"cadd4bf7a0abdff58e4f19c35343660f4e10bbdc4acc4d28948379faf91bc44e"} Nov 03 01:10:05.262298 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:05.256536 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-7hhxt" Nov 03 01:10:05.266610 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:05.265561 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-7hhxt" podStartSLOduration=3.265538474 podStartE2EDuration="3.265538474s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:10:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:10:05.254112734 +0000 UTC m=+1747.425115831" watchObservedRunningTime="2026-11-03 01:10:05.265538474 +0000 UTC m=+1747.436541571" Nov 03 01:10:05.430710 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:05.427462 4421 patch_prober.go:29] interesting pod/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-7hhxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused" start-of-body= Nov 03 01:10:05.430710 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:05.427603 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-7hhxt" podUID="2995b3fc-d4cd-49ba-a988-470f220f30f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused" Nov 03 01:10:06.180396 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:06.180360 4421 patch_prober.go:29] interesting pod/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-7hhxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused" start-of-body= Nov 03 01:10:06.180600 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:06.180418 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-7hhxt" podUID="2995b3fc-d4cd-49ba-a988-470f220f30f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused" Nov 03 01:10:07.180512 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:07.180474 4421 patch_prober.go:29] interesting pod/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-7hhxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused" start-of-body= Nov 03 01:10:07.180896 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:07.180528 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-7hhxt" podUID="2995b3fc-d4cd-49ba-a988-470f220f30f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused" Nov 03 01:10:07.835298 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:07.823369 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d75f4405-jw9b9"] Nov 03 01:10:07.839666 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:07.839634 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d75f4405-jw9b9" Nov 03 01:10:07.853306 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:07.846962 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6q4xp\"" Nov 03 01:10:07.853306 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:07.847004 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:10:07.853306 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:07.847160 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 01:10:07.858074 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:07.857670 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 01:10:07.888591 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:07.888553 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d75f4405-jw9b9"] Nov 03 01:10:07.968297 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:07.960441 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6861d029-4676-4e04-9dc1-3496949a7a7f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d75f4405-jw9b9\" (UID: \"6861d029-4676-4e04-9dc1-3496949a7a7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d75f4405-jw9b9" Nov 03 01:10:07.968297 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:07.960839 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6861d029-4676-4e04-9dc1-3496949a7a7f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d75f4405-jw9b9\" (UID: \"6861d029-4676-4e04-9dc1-3496949a7a7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d75f4405-jw9b9" Nov 03 01:10:07.968297 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:07.966583 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6f77\" (UniqueName: \"kubernetes.io/projected/6861d029-4676-4e04-9dc1-3496949a7a7f-kube-api-access-n6f77\") pod \"maas-api-revoke-sub-d75f4405-jw9b9\" (UID: \"6861d029-4676-4e04-9dc1-3496949a7a7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d75f4405-jw9b9" Nov 03 01:10:08.092588 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:08.092476 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6861d029-4676-4e04-9dc1-3496949a7a7f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d75f4405-jw9b9\" (UID: \"6861d029-4676-4e04-9dc1-3496949a7a7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d75f4405-jw9b9" Nov 03 01:10:08.092938 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:08.092909 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6861d029-4676-4e04-9dc1-3496949a7a7f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d75f4405-jw9b9\" (UID: \"6861d029-4676-4e04-9dc1-3496949a7a7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d75f4405-jw9b9" Nov 03 01:10:08.093179 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:08.093158 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6f77\" (UniqueName: \"kubernetes.io/projected/6861d029-4676-4e04-9dc1-3496949a7a7f-kube-api-access-n6f77\") pod \"maas-api-revoke-sub-d75f4405-jw9b9\" (UID: \"6861d029-4676-4e04-9dc1-3496949a7a7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d75f4405-jw9b9" Nov 03 01:10:08.095469 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:08.095443 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6861d029-4676-4e04-9dc1-3496949a7a7f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d75f4405-jw9b9\" (UID: \"6861d029-4676-4e04-9dc1-3496949a7a7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d75f4405-jw9b9" Nov 03 01:10:08.175294 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:08.149687 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6f77\" (UniqueName: \"kubernetes.io/projected/6861d029-4676-4e04-9dc1-3496949a7a7f-kube-api-access-n6f77\") pod \"maas-api-revoke-sub-d75f4405-jw9b9\" (UID: \"6861d029-4676-4e04-9dc1-3496949a7a7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d75f4405-jw9b9" Nov 03 01:10:08.196853 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:08.196813 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6861d029-4676-4e04-9dc1-3496949a7a7f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d75f4405-jw9b9\" (UID: \"6861d029-4676-4e04-9dc1-3496949a7a7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d75f4405-jw9b9" Nov 03 01:10:08.213046 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:08.213008 4421 patch_prober.go:29] interesting pod/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-7hhxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused" start-of-body= Nov 03 01:10:08.221905 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:08.221866 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-7hhxt" podUID="2995b3fc-d4cd-49ba-a988-470f220f30f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused" Nov 03 01:10:08.497682 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:08.497647 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d75f4405-jw9b9" Nov 03 01:10:08.836792 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:08.836755 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d75f4405-jw9b9"] Nov 03 01:10:08.838199 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:10:08.838160 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6861d029_4676_4e04_9dc1_3496949a7a7f.slice/crio-58b28137cb95cb993023c0645a26eccf69523278cb19d3eb638c501e71827f52 WatchSource:0}: Error finding container 58b28137cb95cb993023c0645a26eccf69523278cb19d3eb638c501e71827f52: Status 404 returned error can't find the container with id 58b28137cb95cb993023c0645a26eccf69523278cb19d3eb638c501e71827f52 Nov 03 01:10:09.180775 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:09.180692 4421 patch_prober.go:29] interesting pod/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-7hhxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused" start-of-body= Nov 03 01:10:09.180775 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:09.180754 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-7hhxt" podUID="2995b3fc-d4cd-49ba-a988-470f220f30f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused" Nov 03 01:10:09.268740 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:09.268701 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d75f4405-jw9b9" event={"ID":"6861d029-4676-4e04-9dc1-3496949a7a7f","Type":"ContainerStarted","Data":"58b28137cb95cb993023c0645a26eccf69523278cb19d3eb638c501e71827f52"} Nov 03 01:10:10.180410 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:10.180352 4421 patch_prober.go:29] interesting pod/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-7hhxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused" start-of-body= Nov 03 01:10:10.181069 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:10.180979 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-7hhxt" podUID="2995b3fc-d4cd-49ba-a988-470f220f30f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused" Nov 03 01:10:10.296526 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:10.296494 4421 generic.go:356] "Generic (PLEG): container finished" podID="6861d029-4676-4e04-9dc1-3496949a7a7f" containerID="e75355bc23ff18d818c4b0ec2e307288b309a3f58aa7f5c65c3cf77ae688c60e" exitCode=0 Nov 03 01:10:10.296960 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:10.296625 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d75f4405-jw9b9" event={"ID":"6861d029-4676-4e04-9dc1-3496949a7a7f","Type":"ContainerDied","Data":"e75355bc23ff18d818c4b0ec2e307288b309a3f58aa7f5c65c3cf77ae688c60e"} Nov 03 01:10:11.042384 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:11.041594 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-846d385a-rz8xm"] Nov 03 01:10:11.048587 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:11.048545 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-846d385a-rz8xm"] Nov 03 01:10:11.180388 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:11.180357 4421 patch_prober.go:29] interesting pod/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-7hhxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused" start-of-body= Nov 03 01:10:11.180588 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:11.180420 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-7hhxt" podUID="2995b3fc-d4cd-49ba-a988-470f220f30f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused" Nov 03 01:10:11.818556 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:11.818523 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d75f4405-jw9b9" Nov 03 01:10:11.984380 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:11.984342 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76467cc3-c3fc-4372-b6c7-af22e0af51e8" path="/var/lib/kubelet/pods/76467cc3-c3fc-4372-b6c7-af22e0af51e8/volumes" Nov 03 01:10:12.016145 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:12.016059 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6861d029-4676-4e04-9dc1-3496949a7a7f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6861d029-4676-4e04-9dc1-3496949a7a7f-maas-api-openshift-ca\") pod \"6861d029-4676-4e04-9dc1-3496949a7a7f\" (UID: \"6861d029-4676-4e04-9dc1-3496949a7a7f\") " Nov 03 01:10:12.016145 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:12.016133 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6861d029-4676-4e04-9dc1-3496949a7a7f-kube-api-access-n6f77\" (UniqueName: \"kubernetes.io/projected/6861d029-4676-4e04-9dc1-3496949a7a7f-kube-api-access-n6f77\") pod \"6861d029-4676-4e04-9dc1-3496949a7a7f\" (UID: \"6861d029-4676-4e04-9dc1-3496949a7a7f\") " Nov 03 01:10:12.016388 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:12.016319 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6861d029-4676-4e04-9dc1-3496949a7a7f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6861d029-4676-4e04-9dc1-3496949a7a7f-maas-api-xks-ca\") pod \"6861d029-4676-4e04-9dc1-3496949a7a7f\" (UID: \"6861d029-4676-4e04-9dc1-3496949a7a7f\") " Nov 03 01:10:12.016734 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:12.016594 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6861d029-4676-4e04-9dc1-3496949a7a7f-maas-api-openshift-ca" pod "6861d029-4676-4e04-9dc1-3496949a7a7f" (UID: "6861d029-4676-4e04-9dc1-3496949a7a7f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:10:12.017222 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:12.017194 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6861d029-4676-4e04-9dc1-3496949a7a7f-maas-api-openshift-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:10:12.019754 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:12.019727 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6861d029-4676-4e04-9dc1-3496949a7a7f-maas-api-xks-ca" pod "6861d029-4676-4e04-9dc1-3496949a7a7f" (UID: "6861d029-4676-4e04-9dc1-3496949a7a7f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:10:12.022560 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:12.022519 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6861d029-4676-4e04-9dc1-3496949a7a7f-kube-api-access-n6f77" pod "6861d029-4676-4e04-9dc1-3496949a7a7f" (UID: "6861d029-4676-4e04-9dc1-3496949a7a7f"). InnerVolumeSpecName "kube-api-access-n6f77". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:10:12.126438 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:12.126367 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n6f77\" (UniqueName: \"kubernetes.io/projected/6861d029-4676-4e04-9dc1-3496949a7a7f-kube-api-access-n6f77\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:10:12.126438 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:12.126404 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6861d029-4676-4e04-9dc1-3496949a7a7f-maas-api-xks-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:10:12.202023 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:12.201983 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-7hhxt" Nov 03 01:10:12.202996 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:12.202973 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-7hhxt" Nov 03 01:10:12.203928 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:12.203910 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-7hhxt" Nov 03 01:10:12.331736 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:12.331646 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d75f4405-jw9b9" Nov 03 01:10:12.331736 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:12.331646 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d75f4405-jw9b9" event={"ID":"6861d029-4676-4e04-9dc1-3496949a7a7f","Type":"ContainerDied","Data":"58b28137cb95cb993023c0645a26eccf69523278cb19d3eb638c501e71827f52"} Nov 03 01:10:12.331736 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:12.331718 4421 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="58b28137cb95cb993023c0645a26eccf69523278cb19d3eb638c501e71827f52" Nov 03 01:10:14.681999 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:10:14.681960 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 01:10:18.054783 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:18.054732 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16fcf300-d5mjj"] Nov 03 01:10:18.055945 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:18.055910 4421 cpu_manager.go:414] "containerMap: removing container" podUID="6861d029-4676-4e04-9dc1-3496949a7a7f" containerName="revoke-keys" Nov 03 01:10:18.055945 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:18.055936 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6861d029-4676-4e04-9dc1-3496949a7a7f" containerName="revoke-keys" Nov 03 01:10:18.056164 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:18.056137 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="6861d029-4676-4e04-9dc1-3496949a7a7f" containerName="revoke-keys" Nov 03 01:10:18.060936 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:18.060909 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16fcf300-d5mjj" Nov 03 01:10:18.066614 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:18.066521 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:10:18.066614 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:18.066560 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6q4xp\"" Nov 03 01:10:18.066614 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:18.066572 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 01:10:18.066858 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:18.066611 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 01:10:18.068098 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:18.068070 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16fcf300-d5mjj"] Nov 03 01:10:18.188149 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:18.188109 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1016d54-45b9-4ea7-b7b9-15aa57a18914-maas-api-xks-ca\") pod \"maas-api-revoke-sub-16fcf300-d5mjj\" (UID: \"e1016d54-45b9-4ea7-b7b9-15aa57a18914\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16fcf300-d5mjj" Nov 03 01:10:18.188453 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:18.188431 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxmb6\" (UniqueName: \"kubernetes.io/projected/e1016d54-45b9-4ea7-b7b9-15aa57a18914-kube-api-access-xxmb6\") pod \"maas-api-revoke-sub-16fcf300-d5mjj\" (UID: \"e1016d54-45b9-4ea7-b7b9-15aa57a18914\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16fcf300-d5mjj" Nov 03 01:10:18.188578 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:18.188500 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1016d54-45b9-4ea7-b7b9-15aa57a18914-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-16fcf300-d5mjj\" (UID: \"e1016d54-45b9-4ea7-b7b9-15aa57a18914\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16fcf300-d5mjj" Nov 03 01:10:18.290547 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:18.290509 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1016d54-45b9-4ea7-b7b9-15aa57a18914-maas-api-xks-ca\") pod \"maas-api-revoke-sub-16fcf300-d5mjj\" (UID: \"e1016d54-45b9-4ea7-b7b9-15aa57a18914\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16fcf300-d5mjj" Nov 03 01:10:18.290733 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:18.290690 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxmb6\" (UniqueName: \"kubernetes.io/projected/e1016d54-45b9-4ea7-b7b9-15aa57a18914-kube-api-access-xxmb6\") pod \"maas-api-revoke-sub-16fcf300-d5mjj\" (UID: \"e1016d54-45b9-4ea7-b7b9-15aa57a18914\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16fcf300-d5mjj" Nov 03 01:10:18.290778 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:18.290744 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1016d54-45b9-4ea7-b7b9-15aa57a18914-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-16fcf300-d5mjj\" (UID: \"e1016d54-45b9-4ea7-b7b9-15aa57a18914\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16fcf300-d5mjj" Nov 03 01:10:18.291457 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:18.291436 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1016d54-45b9-4ea7-b7b9-15aa57a18914-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-16fcf300-d5mjj\" (UID: \"e1016d54-45b9-4ea7-b7b9-15aa57a18914\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16fcf300-d5mjj" Nov 03 01:10:18.292951 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:18.292933 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1016d54-45b9-4ea7-b7b9-15aa57a18914-maas-api-xks-ca\") pod \"maas-api-revoke-sub-16fcf300-d5mjj\" (UID: \"e1016d54-45b9-4ea7-b7b9-15aa57a18914\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16fcf300-d5mjj" Nov 03 01:10:18.299594 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:18.299563 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxmb6\" (UniqueName: \"kubernetes.io/projected/e1016d54-45b9-4ea7-b7b9-15aa57a18914-kube-api-access-xxmb6\") pod \"maas-api-revoke-sub-16fcf300-d5mjj\" (UID: \"e1016d54-45b9-4ea7-b7b9-15aa57a18914\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16fcf300-d5mjj" Nov 03 01:10:18.389460 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:18.389368 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16fcf300-d5mjj" Nov 03 01:10:18.678517 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:10:18.678486 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1016d54_45b9_4ea7_b7b9_15aa57a18914.slice/crio-ee272562197c8b9f2b851b7dd6961339415e644204d806c6d2199ba122b5a16a WatchSource:0}: Error finding container ee272562197c8b9f2b851b7dd6961339415e644204d806c6d2199ba122b5a16a: Status 404 returned error can't find the container with id ee272562197c8b9f2b851b7dd6961339415e644204d806c6d2199ba122b5a16a Nov 03 01:10:18.679836 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:18.679810 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16fcf300-d5mjj"] Nov 03 01:10:19.482083 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:19.481963 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16fcf300-d5mjj" event={"ID":"e1016d54-45b9-4ea7-b7b9-15aa57a18914","Type":"ContainerStarted","Data":"122ef740c552b4ec3e5a77b97ab7baf1ae76f81d46f619e00261610ee9a3f342"} Nov 03 01:10:19.482083 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:19.482031 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16fcf300-d5mjj" event={"ID":"e1016d54-45b9-4ea7-b7b9-15aa57a18914","Type":"ContainerStarted","Data":"ee272562197c8b9f2b851b7dd6961339415e644204d806c6d2199ba122b5a16a"} Nov 03 01:10:19.505327 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:19.505250 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16fcf300-d5mjj" podStartSLOduration=1.5052364759999999 podStartE2EDuration="1.505236476s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:10:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:10:19.500295987 +0000 UTC m=+1761.671299084" watchObservedRunningTime="2026-11-03 01:10:19.505236476 +0000 UTC m=+1761.676239572" Nov 03 01:10:20.493895 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:20.493861 4421 generic.go:356] "Generic (PLEG): container finished" podID="e1016d54-45b9-4ea7-b7b9-15aa57a18914" containerID="122ef740c552b4ec3e5a77b97ab7baf1ae76f81d46f619e00261610ee9a3f342" exitCode=0 Nov 03 01:10:20.493895 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:20.493899 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16fcf300-d5mjj" event={"ID":"e1016d54-45b9-4ea7-b7b9-15aa57a18914","Type":"ContainerDied","Data":"122ef740c552b4ec3e5a77b97ab7baf1ae76f81d46f619e00261610ee9a3f342"} Nov 03 01:10:22.149234 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:22.149205 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16fcf300-d5mjj" Nov 03 01:10:22.248227 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:22.248189 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e1016d54-45b9-4ea7-b7b9-15aa57a18914-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1016d54-45b9-4ea7-b7b9-15aa57a18914-maas-api-xks-ca\") pod \"e1016d54-45b9-4ea7-b7b9-15aa57a18914\" (UID: \"e1016d54-45b9-4ea7-b7b9-15aa57a18914\") " Nov 03 01:10:22.248518 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:22.248502 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1016d54-45b9-4ea7-b7b9-15aa57a18914-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1016d54-45b9-4ea7-b7b9-15aa57a18914-maas-api-openshift-ca\") pod \"e1016d54-45b9-4ea7-b7b9-15aa57a18914\" (UID: \"e1016d54-45b9-4ea7-b7b9-15aa57a18914\") " Nov 03 01:10:22.248594 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:22.248565 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1016d54-45b9-4ea7-b7b9-15aa57a18914-kube-api-access-xxmb6\" (UniqueName: \"kubernetes.io/projected/e1016d54-45b9-4ea7-b7b9-15aa57a18914-kube-api-access-xxmb6\") pod \"e1016d54-45b9-4ea7-b7b9-15aa57a18914\" (UID: \"e1016d54-45b9-4ea7-b7b9-15aa57a18914\") " Nov 03 01:10:22.248921 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:22.248896 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1016d54-45b9-4ea7-b7b9-15aa57a18914-maas-api-openshift-ca" pod "e1016d54-45b9-4ea7-b7b9-15aa57a18914" (UID: "e1016d54-45b9-4ea7-b7b9-15aa57a18914"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:10:22.249517 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:22.249491 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1016d54-45b9-4ea7-b7b9-15aa57a18914-maas-api-openshift-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:10:22.250665 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:22.250629 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1016d54-45b9-4ea7-b7b9-15aa57a18914-kube-api-access-xxmb6" pod "e1016d54-45b9-4ea7-b7b9-15aa57a18914" (UID: "e1016d54-45b9-4ea7-b7b9-15aa57a18914"). InnerVolumeSpecName "kube-api-access-xxmb6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:10:22.250665 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:22.250644 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1016d54-45b9-4ea7-b7b9-15aa57a18914-maas-api-xks-ca" pod "e1016d54-45b9-4ea7-b7b9-15aa57a18914" (UID: "e1016d54-45b9-4ea7-b7b9-15aa57a18914"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:10:22.351482 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:22.351449 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1016d54-45b9-4ea7-b7b9-15aa57a18914-maas-api-xks-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:10:22.351482 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:22.351481 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxmb6\" (UniqueName: \"kubernetes.io/projected/e1016d54-45b9-4ea7-b7b9-15aa57a18914-kube-api-access-xxmb6\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:10:22.801195 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:22.801165 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16fcf300-d5mjj" Nov 03 01:10:22.801485 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:22.801205 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16fcf300-d5mjj" event={"ID":"e1016d54-45b9-4ea7-b7b9-15aa57a18914","Type":"ContainerDied","Data":"ee272562197c8b9f2b851b7dd6961339415e644204d806c6d2199ba122b5a16a"} Nov 03 01:10:22.801485 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:22.801269 4421 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ee272562197c8b9f2b851b7dd6961339415e644204d806c6d2199ba122b5a16a" Nov 03 01:10:25.188965 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:25.188922 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-g4cld"] Nov 03 01:10:25.189478 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:25.189143 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-g4cld" podUID="2cef07ce-2ada-4d6d-93ba-0241dd7e95a4" containerName="istio-proxy" containerID="cri-o://9741c9096d91d0e1cdaba0e6c7ce91be5e329a229144cda49463d44f4a65f862" gracePeriod=30 Nov 03 01:10:28.608190 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:28.607542 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9lp9t"] Nov 03 01:10:28.608190 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:28.607831 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9lp9t" podUID="d2a5f2d2-7356-496f-84ed-dfd3b2eea8e8" containerName="istio-proxy" containerID="cri-o://cc758e54f3e51c9d05b8ed7ec4dbb39fa8f4ee499be71add295e0cb4706cab9d" gracePeriod=30 Nov 03 01:10:28.617724 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:28.617693 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-r6t2p"] Nov 03 01:10:28.617958 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:28.617933 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-r6t2p" podUID="415fa10b-db0c-4ded-aedb-582612cbde2c" containerName="istio-proxy" containerID="cri-o://978bb67569194ec8835b6dbf6bd9aeb3af47eda4ceba67c6d19e125e1e57ff30" gracePeriod=30 Nov 03 01:10:29.895088 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:29.895056 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-50e623/auth-test-model-50e623-kserve-7db6664d89-zdq97"] Nov 03 01:10:30.636167 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:30.636133 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-g4cld" Nov 03 01:10:30.781023 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:30.780987 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cef07ce-2ada-4d6d-93ba-0241dd7e95a4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2cef07ce-2ada-4d6d-93ba-0241dd7e95a4-istio-envoy\") pod \"2cef07ce-2ada-4d6d-93ba-0241dd7e95a4\" (UID: \"2cef07ce-2ada-4d6d-93ba-0241dd7e95a4\") " Nov 03 01:10:30.784781 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:30.784744 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cef07ce-2ada-4d6d-93ba-0241dd7e95a4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2cef07ce-2ada-4d6d-93ba-0241dd7e95a4-workload-certs\") pod \"2cef07ce-2ada-4d6d-93ba-0241dd7e95a4\" (UID: \"2cef07ce-2ada-4d6d-93ba-0241dd7e95a4\") " Nov 03 01:10:30.785748 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:30.785719 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2cef07ce-2ada-4d6d-93ba-0241dd7e95a4-kube-api-access-np9gf\" (UniqueName: \"kubernetes.io/projected/2cef07ce-2ada-4d6d-93ba-0241dd7e95a4-kube-api-access-np9gf\") pod \"2cef07ce-2ada-4d6d-93ba-0241dd7e95a4\" (UID: \"2cef07ce-2ada-4d6d-93ba-0241dd7e95a4\") " Nov 03 01:10:30.785979 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:30.785949 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cef07ce-2ada-4d6d-93ba-0241dd7e95a4-workload-certs" pod "2cef07ce-2ada-4d6d-93ba-0241dd7e95a4" (UID: "2cef07ce-2ada-4d6d-93ba-0241dd7e95a4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:10:30.786747 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:30.786721 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2cef07ce-2ada-4d6d-93ba-0241dd7e95a4-istiod-ca-cert" pod "2cef07ce-2ada-4d6d-93ba-0241dd7e95a4" (UID: "2cef07ce-2ada-4d6d-93ba-0241dd7e95a4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:10:30.787400 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:30.787376 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2cef07ce-2ada-4d6d-93ba-0241dd7e95a4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2cef07ce-2ada-4d6d-93ba-0241dd7e95a4-istiod-ca-cert\") pod \"2cef07ce-2ada-4d6d-93ba-0241dd7e95a4\" (UID: \"2cef07ce-2ada-4d6d-93ba-0241dd7e95a4\") " Nov 03 01:10:30.787973 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:30.787945 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cef07ce-2ada-4d6d-93ba-0241dd7e95a4-credential-socket" pod "2cef07ce-2ada-4d6d-93ba-0241dd7e95a4" (UID: "2cef07ce-2ada-4d6d-93ba-0241dd7e95a4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:10:30.788100 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:30.788048 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cef07ce-2ada-4d6d-93ba-0241dd7e95a4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cef07ce-2ada-4d6d-93ba-0241dd7e95a4-credential-socket\") pod \"2cef07ce-2ada-4d6d-93ba-0241dd7e95a4\" (UID: \"2cef07ce-2ada-4d6d-93ba-0241dd7e95a4\") " Nov 03 01:10:30.788169 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:30.788109 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cef07ce-2ada-4d6d-93ba-0241dd7e95a4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2cef07ce-2ada-4d6d-93ba-0241dd7e95a4-istio-data\") pod \"2cef07ce-2ada-4d6d-93ba-0241dd7e95a4\" (UID: \"2cef07ce-2ada-4d6d-93ba-0241dd7e95a4\") " Nov 03 01:10:30.788253 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:30.788198 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2cef07ce-2ada-4d6d-93ba-0241dd7e95a4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2cef07ce-2ada-4d6d-93ba-0241dd7e95a4-istio-podinfo\") pod \"2cef07ce-2ada-4d6d-93ba-0241dd7e95a4\" (UID: \"2cef07ce-2ada-4d6d-93ba-0241dd7e95a4\") " Nov 03 01:10:30.788389 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:30.788291 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2cef07ce-2ada-4d6d-93ba-0241dd7e95a4-istio-token\" (UniqueName: \"kubernetes.io/projected/2cef07ce-2ada-4d6d-93ba-0241dd7e95a4-istio-token\") pod \"2cef07ce-2ada-4d6d-93ba-0241dd7e95a4\" (UID: \"2cef07ce-2ada-4d6d-93ba-0241dd7e95a4\") " Nov 03 01:10:30.788389 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:30.788380 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cef07ce-2ada-4d6d-93ba-0241dd7e95a4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cef07ce-2ada-4d6d-93ba-0241dd7e95a4-workload-socket\") pod \"2cef07ce-2ada-4d6d-93ba-0241dd7e95a4\" (UID: \"2cef07ce-2ada-4d6d-93ba-0241dd7e95a4\") " Nov 03 01:10:30.790245 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:30.789716 4421 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2cef07ce-2ada-4d6d-93ba-0241dd7e95a4-istiod-ca-cert\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:10:30.790245 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:30.789747 4421 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cef07ce-2ada-4d6d-93ba-0241dd7e95a4-credential-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:10:30.790613 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:30.789763 4421 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2cef07ce-2ada-4d6d-93ba-0241dd7e95a4-workload-certs\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:10:30.790732 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:30.790198 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cef07ce-2ada-4d6d-93ba-0241dd7e95a4-workload-socket" pod "2cef07ce-2ada-4d6d-93ba-0241dd7e95a4" (UID: "2cef07ce-2ada-4d6d-93ba-0241dd7e95a4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:10:30.790732 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:30.790491 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cef07ce-2ada-4d6d-93ba-0241dd7e95a4-istio-data" pod "2cef07ce-2ada-4d6d-93ba-0241dd7e95a4" (UID: "2cef07ce-2ada-4d6d-93ba-0241dd7e95a4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:10:30.791309 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:30.791246 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cef07ce-2ada-4d6d-93ba-0241dd7e95a4-istio-envoy" pod "2cef07ce-2ada-4d6d-93ba-0241dd7e95a4" (UID: "2cef07ce-2ada-4d6d-93ba-0241dd7e95a4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:10:30.791418 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:30.791395 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cef07ce-2ada-4d6d-93ba-0241dd7e95a4-kube-api-access-np9gf" pod "2cef07ce-2ada-4d6d-93ba-0241dd7e95a4" (UID: "2cef07ce-2ada-4d6d-93ba-0241dd7e95a4"). InnerVolumeSpecName "kube-api-access-np9gf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:10:30.792449 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:30.792427 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cef07ce-2ada-4d6d-93ba-0241dd7e95a4-istio-token" pod "2cef07ce-2ada-4d6d-93ba-0241dd7e95a4" (UID: "2cef07ce-2ada-4d6d-93ba-0241dd7e95a4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:10:30.793000 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:30.792978 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2cef07ce-2ada-4d6d-93ba-0241dd7e95a4-istio-podinfo" pod "2cef07ce-2ada-4d6d-93ba-0241dd7e95a4" (UID: "2cef07ce-2ada-4d6d-93ba-0241dd7e95a4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:10:30.892971 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:30.892924 4421 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cef07ce-2ada-4d6d-93ba-0241dd7e95a4-workload-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:10:30.892971 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:30.892973 4421 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2cef07ce-2ada-4d6d-93ba-0241dd7e95a4-istio-envoy\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:10:30.893206 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:30.892988 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-np9gf\" (UniqueName: \"kubernetes.io/projected/2cef07ce-2ada-4d6d-93ba-0241dd7e95a4-kube-api-access-np9gf\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:10:30.893206 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:30.893005 4421 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2cef07ce-2ada-4d6d-93ba-0241dd7e95a4-istio-data\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:10:30.893206 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:30.893047 4421 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2cef07ce-2ada-4d6d-93ba-0241dd7e95a4-istio-podinfo\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:10:30.893206 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:30.893061 4421 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2cef07ce-2ada-4d6d-93ba-0241dd7e95a4-istio-token\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:10:30.905806 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:30.905767 4421 generic.go:356] "Generic (PLEG): container finished" podID="2cef07ce-2ada-4d6d-93ba-0241dd7e95a4" containerID="9741c9096d91d0e1cdaba0e6c7ce91be5e329a229144cda49463d44f4a65f862" exitCode=0 Nov 03 01:10:30.906318 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:30.905822 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-g4cld" Nov 03 01:10:30.906318 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:30.905856 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-g4cld" event={"ID":"2cef07ce-2ada-4d6d-93ba-0241dd7e95a4","Type":"ContainerDied","Data":"9741c9096d91d0e1cdaba0e6c7ce91be5e329a229144cda49463d44f4a65f862"} Nov 03 01:10:30.906318 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:30.905957 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-g4cld" event={"ID":"2cef07ce-2ada-4d6d-93ba-0241dd7e95a4","Type":"ContainerDied","Data":"aa568a85dd45684564dd799cc85d05c975ea11110e85054785514682f3e7d619"} Nov 03 01:10:30.906318 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:30.905990 4421 scope.go:122] "RemoveContainer" containerID="9741c9096d91d0e1cdaba0e6c7ce91be5e329a229144cda49463d44f4a65f862" Nov 03 01:10:30.919317 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:30.919256 4421 scope.go:122] "RemoveContainer" containerID="9741c9096d91d0e1cdaba0e6c7ce91be5e329a229144cda49463d44f4a65f862" Nov 03 01:10:30.919597 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:10:30.919573 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9741c9096d91d0e1cdaba0e6c7ce91be5e329a229144cda49463d44f4a65f862\": container with ID starting with 9741c9096d91d0e1cdaba0e6c7ce91be5e329a229144cda49463d44f4a65f862 not found: ID does not exist" containerID="9741c9096d91d0e1cdaba0e6c7ce91be5e329a229144cda49463d44f4a65f862" Nov 03 01:10:30.919670 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:30.919610 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9741c9096d91d0e1cdaba0e6c7ce91be5e329a229144cda49463d44f4a65f862"} err="failed to get container status \"9741c9096d91d0e1cdaba0e6c7ce91be5e329a229144cda49463d44f4a65f862\": rpc error: code = NotFound desc = could not find container \"9741c9096d91d0e1cdaba0e6c7ce91be5e329a229144cda49463d44f4a65f862\": container with ID starting with 9741c9096d91d0e1cdaba0e6c7ce91be5e329a229144cda49463d44f4a65f862 not found: ID does not exist" Nov 03 01:10:30.973549 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:30.973258 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-g4cld"] Nov 03 01:10:30.978693 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:30.978659 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-g4cld"] Nov 03 01:10:31.982137 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:31.982102 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2cef07ce-2ada-4d6d-93ba-0241dd7e95a4" path="/var/lib/kubelet/pods/2cef07ce-2ada-4d6d-93ba-0241dd7e95a4/volumes" Nov 03 01:10:32.318462 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:32.318424 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-a23ed0-openshift-default-84685cd7b7-6vgvw"] Nov 03 01:10:32.319825 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:32.319796 4421 cpu_manager.go:414] "containerMap: removing container" podUID="e1016d54-45b9-4ea7-b7b9-15aa57a18914" containerName="revoke-keys" Nov 03 01:10:32.319825 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:32.319823 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1016d54-45b9-4ea7-b7b9-15aa57a18914" containerName="revoke-keys" Nov 03 01:10:32.320069 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:32.319913 4421 cpu_manager.go:414] "containerMap: removing container" podUID="2cef07ce-2ada-4d6d-93ba-0241dd7e95a4" containerName="istio-proxy" Nov 03 01:10:32.320069 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:32.319924 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2cef07ce-2ada-4d6d-93ba-0241dd7e95a4" containerName="istio-proxy" Nov 03 01:10:32.320394 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:32.320369 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="2cef07ce-2ada-4d6d-93ba-0241dd7e95a4" containerName="istio-proxy" Nov 03 01:10:32.320781 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:32.320427 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1016d54-45b9-4ea7-b7b9-15aa57a18914" containerName="revoke-keys" Nov 03 01:10:32.329910 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:32.329833 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-a23ed0-openshift-default-84685cd7b7-6vgvw" Nov 03 01:10:32.344236 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:32.344209 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-a23ed0-openshift-default-dockercfg-hblpl\"" Nov 03 01:10:32.350909 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:32.350884 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-a23ed0-openshift-default-84685cd7b7-6vgvw"] Nov 03 01:10:32.360128 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:32.360096 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/065c5377-f408-4f5c-be1c-19e4cea9e64e-istio-data\") pod \"e2e-noten-a23ed0-openshift-default-84685cd7b7-6vgvw\" (UID: \"065c5377-f408-4f5c-be1c-19e4cea9e64e\") " pod="openshift-ingress/e2e-noten-a23ed0-openshift-default-84685cd7b7-6vgvw" Nov 03 01:10:32.360303 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:32.360164 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/065c5377-f408-4f5c-be1c-19e4cea9e64e-credential-socket\") pod \"e2e-noten-a23ed0-openshift-default-84685cd7b7-6vgvw\" (UID: \"065c5377-f408-4f5c-be1c-19e4cea9e64e\") " pod="openshift-ingress/e2e-noten-a23ed0-openshift-default-84685cd7b7-6vgvw" Nov 03 01:10:32.360387 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:32.360302 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/065c5377-f408-4f5c-be1c-19e4cea9e64e-istio-envoy\") pod \"e2e-noten-a23ed0-openshift-default-84685cd7b7-6vgvw\" (UID: \"065c5377-f408-4f5c-be1c-19e4cea9e64e\") " pod="openshift-ingress/e2e-noten-a23ed0-openshift-default-84685cd7b7-6vgvw" Nov 03 01:10:32.360519 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:32.360485 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/065c5377-f408-4f5c-be1c-19e4cea9e64e-workload-certs\") pod \"e2e-noten-a23ed0-openshift-default-84685cd7b7-6vgvw\" (UID: \"065c5377-f408-4f5c-be1c-19e4cea9e64e\") " pod="openshift-ingress/e2e-noten-a23ed0-openshift-default-84685cd7b7-6vgvw" Nov 03 01:10:32.360572 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:32.360557 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/065c5377-f408-4f5c-be1c-19e4cea9e64e-istiod-ca-cert\") pod \"e2e-noten-a23ed0-openshift-default-84685cd7b7-6vgvw\" (UID: \"065c5377-f408-4f5c-be1c-19e4cea9e64e\") " pod="openshift-ingress/e2e-noten-a23ed0-openshift-default-84685cd7b7-6vgvw" Nov 03 01:10:32.360615 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:32.360601 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/065c5377-f408-4f5c-be1c-19e4cea9e64e-workload-socket\") pod \"e2e-noten-a23ed0-openshift-default-84685cd7b7-6vgvw\" (UID: \"065c5377-f408-4f5c-be1c-19e4cea9e64e\") " pod="openshift-ingress/e2e-noten-a23ed0-openshift-default-84685cd7b7-6vgvw" Nov 03 01:10:32.360698 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:32.360682 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/065c5377-f408-4f5c-be1c-19e4cea9e64e-istio-token\") pod \"e2e-noten-a23ed0-openshift-default-84685cd7b7-6vgvw\" (UID: \"065c5377-f408-4f5c-be1c-19e4cea9e64e\") " pod="openshift-ingress/e2e-noten-a23ed0-openshift-default-84685cd7b7-6vgvw" Nov 03 01:10:32.360864 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:32.360819 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/065c5377-f408-4f5c-be1c-19e4cea9e64e-istio-podinfo\") pod \"e2e-noten-a23ed0-openshift-default-84685cd7b7-6vgvw\" (UID: \"065c5377-f408-4f5c-be1c-19e4cea9e64e\") " pod="openshift-ingress/e2e-noten-a23ed0-openshift-default-84685cd7b7-6vgvw" Nov 03 01:10:32.360864 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:32.360852 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8qv9\" (UniqueName: \"kubernetes.io/projected/065c5377-f408-4f5c-be1c-19e4cea9e64e-kube-api-access-v8qv9\") pod \"e2e-noten-a23ed0-openshift-default-84685cd7b7-6vgvw\" (UID: \"065c5377-f408-4f5c-be1c-19e4cea9e64e\") " pod="openshift-ingress/e2e-noten-a23ed0-openshift-default-84685cd7b7-6vgvw" Nov 03 01:10:32.462158 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:32.462115 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/065c5377-f408-4f5c-be1c-19e4cea9e64e-workload-certs\") pod \"e2e-noten-a23ed0-openshift-default-84685cd7b7-6vgvw\" (UID: \"065c5377-f408-4f5c-be1c-19e4cea9e64e\") " pod="openshift-ingress/e2e-noten-a23ed0-openshift-default-84685cd7b7-6vgvw" Nov 03 01:10:32.462394 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:32.462193 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/065c5377-f408-4f5c-be1c-19e4cea9e64e-istiod-ca-cert\") pod \"e2e-noten-a23ed0-openshift-default-84685cd7b7-6vgvw\" (UID: \"065c5377-f408-4f5c-be1c-19e4cea9e64e\") " pod="openshift-ingress/e2e-noten-a23ed0-openshift-default-84685cd7b7-6vgvw" Nov 03 01:10:32.462394 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:32.462236 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/065c5377-f408-4f5c-be1c-19e4cea9e64e-workload-socket\") pod \"e2e-noten-a23ed0-openshift-default-84685cd7b7-6vgvw\" (UID: \"065c5377-f408-4f5c-be1c-19e4cea9e64e\") " pod="openshift-ingress/e2e-noten-a23ed0-openshift-default-84685cd7b7-6vgvw" Nov 03 01:10:32.462394 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:32.462335 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/065c5377-f408-4f5c-be1c-19e4cea9e64e-istio-token\") pod \"e2e-noten-a23ed0-openshift-default-84685cd7b7-6vgvw\" (UID: \"065c5377-f408-4f5c-be1c-19e4cea9e64e\") " pod="openshift-ingress/e2e-noten-a23ed0-openshift-default-84685cd7b7-6vgvw" Nov 03 01:10:32.462564 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:32.462468 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/065c5377-f408-4f5c-be1c-19e4cea9e64e-istio-podinfo\") pod \"e2e-noten-a23ed0-openshift-default-84685cd7b7-6vgvw\" (UID: \"065c5377-f408-4f5c-be1c-19e4cea9e64e\") " pod="openshift-ingress/e2e-noten-a23ed0-openshift-default-84685cd7b7-6vgvw" Nov 03 01:10:32.462659 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:32.462631 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/065c5377-f408-4f5c-be1c-19e4cea9e64e-workload-certs\") pod \"e2e-noten-a23ed0-openshift-default-84685cd7b7-6vgvw\" (UID: \"065c5377-f408-4f5c-be1c-19e4cea9e64e\") " pod="openshift-ingress/e2e-noten-a23ed0-openshift-default-84685cd7b7-6vgvw" Nov 03 01:10:32.462714 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:32.462640 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8qv9\" (UniqueName: \"kubernetes.io/projected/065c5377-f408-4f5c-be1c-19e4cea9e64e-kube-api-access-v8qv9\") pod \"e2e-noten-a23ed0-openshift-default-84685cd7b7-6vgvw\" (UID: \"065c5377-f408-4f5c-be1c-19e4cea9e64e\") " pod="openshift-ingress/e2e-noten-a23ed0-openshift-default-84685cd7b7-6vgvw" Nov 03 01:10:32.462714 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:32.462693 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/065c5377-f408-4f5c-be1c-19e4cea9e64e-workload-socket\") pod \"e2e-noten-a23ed0-openshift-default-84685cd7b7-6vgvw\" (UID: \"065c5377-f408-4f5c-be1c-19e4cea9e64e\") " pod="openshift-ingress/e2e-noten-a23ed0-openshift-default-84685cd7b7-6vgvw" Nov 03 01:10:32.462835 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:32.462787 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/065c5377-f408-4f5c-be1c-19e4cea9e64e-istio-data\") pod \"e2e-noten-a23ed0-openshift-default-84685cd7b7-6vgvw\" (UID: \"065c5377-f408-4f5c-be1c-19e4cea9e64e\") " pod="openshift-ingress/e2e-noten-a23ed0-openshift-default-84685cd7b7-6vgvw" Nov 03 01:10:32.462895 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:32.462847 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/065c5377-f408-4f5c-be1c-19e4cea9e64e-credential-socket\") pod \"e2e-noten-a23ed0-openshift-default-84685cd7b7-6vgvw\" (UID: \"065c5377-f408-4f5c-be1c-19e4cea9e64e\") " pod="openshift-ingress/e2e-noten-a23ed0-openshift-default-84685cd7b7-6vgvw" Nov 03 01:10:32.462998 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:32.462983 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/065c5377-f408-4f5c-be1c-19e4cea9e64e-istio-envoy\") pod \"e2e-noten-a23ed0-openshift-default-84685cd7b7-6vgvw\" (UID: \"065c5377-f408-4f5c-be1c-19e4cea9e64e\") " pod="openshift-ingress/e2e-noten-a23ed0-openshift-default-84685cd7b7-6vgvw" Nov 03 01:10:32.463062 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:32.463024 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/065c5377-f408-4f5c-be1c-19e4cea9e64e-istiod-ca-cert\") pod \"e2e-noten-a23ed0-openshift-default-84685cd7b7-6vgvw\" (UID: \"065c5377-f408-4f5c-be1c-19e4cea9e64e\") " pod="openshift-ingress/e2e-noten-a23ed0-openshift-default-84685cd7b7-6vgvw" Nov 03 01:10:32.463158 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:32.463136 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/065c5377-f408-4f5c-be1c-19e4cea9e64e-istio-data\") pod \"e2e-noten-a23ed0-openshift-default-84685cd7b7-6vgvw\" (UID: \"065c5377-f408-4f5c-be1c-19e4cea9e64e\") " pod="openshift-ingress/e2e-noten-a23ed0-openshift-default-84685cd7b7-6vgvw" Nov 03 01:10:32.463220 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:32.463198 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/065c5377-f408-4f5c-be1c-19e4cea9e64e-credential-socket\") pod \"e2e-noten-a23ed0-openshift-default-84685cd7b7-6vgvw\" (UID: \"065c5377-f408-4f5c-be1c-19e4cea9e64e\") " pod="openshift-ingress/e2e-noten-a23ed0-openshift-default-84685cd7b7-6vgvw" Nov 03 01:10:32.465054 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:32.465030 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/065c5377-f408-4f5c-be1c-19e4cea9e64e-istio-envoy\") pod \"e2e-noten-a23ed0-openshift-default-84685cd7b7-6vgvw\" (UID: \"065c5377-f408-4f5c-be1c-19e4cea9e64e\") " pod="openshift-ingress/e2e-noten-a23ed0-openshift-default-84685cd7b7-6vgvw" Nov 03 01:10:32.465181 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:32.465171 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/065c5377-f408-4f5c-be1c-19e4cea9e64e-istio-podinfo\") pod \"e2e-noten-a23ed0-openshift-default-84685cd7b7-6vgvw\" (UID: \"065c5377-f408-4f5c-be1c-19e4cea9e64e\") " pod="openshift-ingress/e2e-noten-a23ed0-openshift-default-84685cd7b7-6vgvw" Nov 03 01:10:32.470420 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:32.470397 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/065c5377-f408-4f5c-be1c-19e4cea9e64e-istio-token\") pod \"e2e-noten-a23ed0-openshift-default-84685cd7b7-6vgvw\" (UID: \"065c5377-f408-4f5c-be1c-19e4cea9e64e\") " pod="openshift-ingress/e2e-noten-a23ed0-openshift-default-84685cd7b7-6vgvw" Nov 03 01:10:32.470639 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:32.470620 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8qv9\" (UniqueName: \"kubernetes.io/projected/065c5377-f408-4f5c-be1c-19e4cea9e64e-kube-api-access-v8qv9\") pod \"e2e-noten-a23ed0-openshift-default-84685cd7b7-6vgvw\" (UID: \"065c5377-f408-4f5c-be1c-19e4cea9e64e\") " pod="openshift-ingress/e2e-noten-a23ed0-openshift-default-84685cd7b7-6vgvw" Nov 03 01:10:32.664355 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:32.663802 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-a23ed0-openshift-default-84685cd7b7-6vgvw" Nov 03 01:10:33.187898 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:10:33.187861 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod065c5377_f408_4f5c_be1c_19e4cea9e64e.slice/crio-9a2cd0efd5c522d44f9b416cd120a6c4b12eee74195e645d92b89cce4c39fb0f WatchSource:0}: Error finding container 9a2cd0efd5c522d44f9b416cd120a6c4b12eee74195e645d92b89cce4c39fb0f: Status 404 returned error can't find the container with id 9a2cd0efd5c522d44f9b416cd120a6c4b12eee74195e645d92b89cce4c39fb0f Nov 03 01:10:33.188258 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:33.188234 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-a23ed0-openshift-default-84685cd7b7-6vgvw"] Nov 03 01:10:33.190890 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:33.190852 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:10:33.190965 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:33.190928 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:10:33.190965 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:33.190960 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:10:33.946057 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:33.946027 4421 generic.go:356] "Generic (PLEG): container finished" podID="415fa10b-db0c-4ded-aedb-582612cbde2c" containerID="978bb67569194ec8835b6dbf6bd9aeb3af47eda4ceba67c6d19e125e1e57ff30" exitCode=0 Nov 03 01:10:33.946313 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:33.946100 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-r6t2p" event={"ID":"415fa10b-db0c-4ded-aedb-582612cbde2c","Type":"ContainerDied","Data":"978bb67569194ec8835b6dbf6bd9aeb3af47eda4ceba67c6d19e125e1e57ff30"} Nov 03 01:10:33.948581 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:33.948554 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-a23ed0-openshift-default-84685cd7b7-6vgvw" event={"ID":"065c5377-f408-4f5c-be1c-19e4cea9e64e","Type":"ContainerStarted","Data":"9202bd26541cdff38cf8d19d0e98ac483f3ae57b4238e4611a2d9af68350a4b9"} Nov 03 01:10:33.948726 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:33.948607 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-a23ed0-openshift-default-84685cd7b7-6vgvw" event={"ID":"065c5377-f408-4f5c-be1c-19e4cea9e64e","Type":"ContainerStarted","Data":"9a2cd0efd5c522d44f9b416cd120a6c4b12eee74195e645d92b89cce4c39fb0f"} Nov 03 01:10:33.950627 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:33.950598 4421 generic.go:356] "Generic (PLEG): container finished" podID="d2a5f2d2-7356-496f-84ed-dfd3b2eea8e8" containerID="cc758e54f3e51c9d05b8ed7ec4dbb39fa8f4ee499be71add295e0cb4706cab9d" exitCode=0 Nov 03 01:10:33.950767 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:33.950625 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9lp9t" event={"ID":"d2a5f2d2-7356-496f-84ed-dfd3b2eea8e8","Type":"ContainerDied","Data":"cc758e54f3e51c9d05b8ed7ec4dbb39fa8f4ee499be71add295e0cb4706cab9d"} Nov 03 01:10:33.950767 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:33.950684 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9lp9t" event={"ID":"d2a5f2d2-7356-496f-84ed-dfd3b2eea8e8","Type":"ContainerDied","Data":"bdf2f0c3cdfe0ccc797d7a8504b592b02a72f6ff572163d1d22d8b5d7c8cf5fc"} Nov 03 01:10:33.950767 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:33.950713 4421 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bdf2f0c3cdfe0ccc797d7a8504b592b02a72f6ff572163d1d22d8b5d7c8cf5fc" Nov 03 01:10:33.967105 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:33.967083 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9lp9t" Nov 03 01:10:33.971713 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:33.971254 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-r6t2p" Nov 03 01:10:33.972636 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:33.972589 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-a23ed0-openshift-default-84685cd7b7-6vgvw" podStartSLOduration=1.972577347 podStartE2EDuration="1.972577347s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:10:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:10:33.97086505 +0000 UTC m=+1776.141868151" watchObservedRunningTime="2026-11-03 01:10:33.972577347 +0000 UTC m=+1776.143580442" Nov 03 01:10:34.034008 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:34.033946 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d2a5f2d2-7356-496f-84ed-dfd3b2eea8e8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2a5f2d2-7356-496f-84ed-dfd3b2eea8e8-istio-podinfo\") pod \"d2a5f2d2-7356-496f-84ed-dfd3b2eea8e8\" (UID: \"d2a5f2d2-7356-496f-84ed-dfd3b2eea8e8\") " Nov 03 01:10:34.034220 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:34.034078 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2a5f2d2-7356-496f-84ed-dfd3b2eea8e8-kube-api-access-6vcq7\" (UniqueName: \"kubernetes.io/projected/d2a5f2d2-7356-496f-84ed-dfd3b2eea8e8-kube-api-access-6vcq7\") pod \"d2a5f2d2-7356-496f-84ed-dfd3b2eea8e8\" (UID: \"d2a5f2d2-7356-496f-84ed-dfd3b2eea8e8\") " Nov 03 01:10:34.034220 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:34.034116 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/415fa10b-db0c-4ded-aedb-582612cbde2c-kube-api-access-nzm6n\" (UniqueName: \"kubernetes.io/projected/415fa10b-db0c-4ded-aedb-582612cbde2c-kube-api-access-nzm6n\") pod \"415fa10b-db0c-4ded-aedb-582612cbde2c\" (UID: \"415fa10b-db0c-4ded-aedb-582612cbde2c\") " Nov 03 01:10:34.034220 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:34.034165 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/415fa10b-db0c-4ded-aedb-582612cbde2c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/415fa10b-db0c-4ded-aedb-582612cbde2c-workload-certs\") pod \"415fa10b-db0c-4ded-aedb-582612cbde2c\" (UID: \"415fa10b-db0c-4ded-aedb-582612cbde2c\") " Nov 03 01:10:34.034220 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:34.034215 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/415fa10b-db0c-4ded-aedb-582612cbde2c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/415fa10b-db0c-4ded-aedb-582612cbde2c-workload-socket\") pod \"415fa10b-db0c-4ded-aedb-582612cbde2c\" (UID: \"415fa10b-db0c-4ded-aedb-582612cbde2c\") " Nov 03 01:10:34.038016 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:34.037956 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/415fa10b-db0c-4ded-aedb-582612cbde2c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/415fa10b-db0c-4ded-aedb-582612cbde2c-istiod-ca-cert\") pod \"415fa10b-db0c-4ded-aedb-582612cbde2c\" (UID: \"415fa10b-db0c-4ded-aedb-582612cbde2c\") " Nov 03 01:10:34.038016 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:34.038011 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2a5f2d2-7356-496f-84ed-dfd3b2eea8e8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2a5f2d2-7356-496f-84ed-dfd3b2eea8e8-istiod-ca-cert\") pod \"d2a5f2d2-7356-496f-84ed-dfd3b2eea8e8\" (UID: \"d2a5f2d2-7356-496f-84ed-dfd3b2eea8e8\") " Nov 03 01:10:34.038216 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:34.038120 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/415fa10b-db0c-4ded-aedb-582612cbde2c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/415fa10b-db0c-4ded-aedb-582612cbde2c-credential-socket\") pod \"415fa10b-db0c-4ded-aedb-582612cbde2c\" (UID: \"415fa10b-db0c-4ded-aedb-582612cbde2c\") " Nov 03 01:10:34.038216 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:34.038161 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2a5f2d2-7356-496f-84ed-dfd3b2eea8e8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2a5f2d2-7356-496f-84ed-dfd3b2eea8e8-workload-socket\") pod \"d2a5f2d2-7356-496f-84ed-dfd3b2eea8e8\" (UID: \"d2a5f2d2-7356-496f-84ed-dfd3b2eea8e8\") " Nov 03 01:10:34.038359 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:34.038235 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2a5f2d2-7356-496f-84ed-dfd3b2eea8e8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2a5f2d2-7356-496f-84ed-dfd3b2eea8e8-credential-socket\") pod \"d2a5f2d2-7356-496f-84ed-dfd3b2eea8e8\" (UID: \"d2a5f2d2-7356-496f-84ed-dfd3b2eea8e8\") " Nov 03 01:10:34.038359 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:34.038330 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2a5f2d2-7356-496f-84ed-dfd3b2eea8e8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2a5f2d2-7356-496f-84ed-dfd3b2eea8e8-workload-certs\") pod \"d2a5f2d2-7356-496f-84ed-dfd3b2eea8e8\" (UID: \"d2a5f2d2-7356-496f-84ed-dfd3b2eea8e8\") " Nov 03 01:10:34.038483 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:34.038398 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/415fa10b-db0c-4ded-aedb-582612cbde2c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/415fa10b-db0c-4ded-aedb-582612cbde2c-istio-envoy\") pod \"415fa10b-db0c-4ded-aedb-582612cbde2c\" (UID: \"415fa10b-db0c-4ded-aedb-582612cbde2c\") " Nov 03 01:10:34.038483 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:34.038452 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/415fa10b-db0c-4ded-aedb-582612cbde2c-istio-token\" (UniqueName: \"kubernetes.io/projected/415fa10b-db0c-4ded-aedb-582612cbde2c-istio-token\") pod \"415fa10b-db0c-4ded-aedb-582612cbde2c\" (UID: \"415fa10b-db0c-4ded-aedb-582612cbde2c\") " Nov 03 01:10:34.038608 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:34.038502 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/415fa10b-db0c-4ded-aedb-582612cbde2c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/415fa10b-db0c-4ded-aedb-582612cbde2c-istio-podinfo\") pod \"415fa10b-db0c-4ded-aedb-582612cbde2c\" (UID: \"415fa10b-db0c-4ded-aedb-582612cbde2c\") " Nov 03 01:10:34.038669 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:34.038629 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2a5f2d2-7356-496f-84ed-dfd3b2eea8e8-istio-token\" (UniqueName: \"kubernetes.io/projected/d2a5f2d2-7356-496f-84ed-dfd3b2eea8e8-istio-token\") pod \"d2a5f2d2-7356-496f-84ed-dfd3b2eea8e8\" (UID: \"d2a5f2d2-7356-496f-84ed-dfd3b2eea8e8\") " Nov 03 01:10:34.038730 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:34.038672 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/415fa10b-db0c-4ded-aedb-582612cbde2c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/415fa10b-db0c-4ded-aedb-582612cbde2c-istio-data\") pod \"415fa10b-db0c-4ded-aedb-582612cbde2c\" (UID: \"415fa10b-db0c-4ded-aedb-582612cbde2c\") " Nov 03 01:10:34.038790 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:34.038736 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2a5f2d2-7356-496f-84ed-dfd3b2eea8e8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2a5f2d2-7356-496f-84ed-dfd3b2eea8e8-istio-envoy\") pod \"d2a5f2d2-7356-496f-84ed-dfd3b2eea8e8\" (UID: \"d2a5f2d2-7356-496f-84ed-dfd3b2eea8e8\") " Nov 03 01:10:34.038853 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:34.038815 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2a5f2d2-7356-496f-84ed-dfd3b2eea8e8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2a5f2d2-7356-496f-84ed-dfd3b2eea8e8-istio-data\") pod \"d2a5f2d2-7356-496f-84ed-dfd3b2eea8e8\" (UID: \"d2a5f2d2-7356-496f-84ed-dfd3b2eea8e8\") " Nov 03 01:10:34.038993 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:34.038966 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/415fa10b-db0c-4ded-aedb-582612cbde2c-workload-socket" pod "415fa10b-db0c-4ded-aedb-582612cbde2c" (UID: "415fa10b-db0c-4ded-aedb-582612cbde2c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:10:34.040462 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:34.040436 4421 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/415fa10b-db0c-4ded-aedb-582612cbde2c-workload-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:10:34.041128 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:34.041103 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/415fa10b-db0c-4ded-aedb-582612cbde2c-istiod-ca-cert" pod "415fa10b-db0c-4ded-aedb-582612cbde2c" (UID: "415fa10b-db0c-4ded-aedb-582612cbde2c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:10:34.041440 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:34.041422 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2a5f2d2-7356-496f-84ed-dfd3b2eea8e8-istiod-ca-cert" pod "d2a5f2d2-7356-496f-84ed-dfd3b2eea8e8" (UID: "d2a5f2d2-7356-496f-84ed-dfd3b2eea8e8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:10:34.041667 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:34.041646 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/415fa10b-db0c-4ded-aedb-582612cbde2c-credential-socket" pod "415fa10b-db0c-4ded-aedb-582612cbde2c" (UID: "415fa10b-db0c-4ded-aedb-582612cbde2c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:10:34.041967 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:34.041842 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2a5f2d2-7356-496f-84ed-dfd3b2eea8e8-workload-socket" pod "d2a5f2d2-7356-496f-84ed-dfd3b2eea8e8" (UID: "d2a5f2d2-7356-496f-84ed-dfd3b2eea8e8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:10:34.042063 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:34.042004 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2a5f2d2-7356-496f-84ed-dfd3b2eea8e8-credential-socket" pod "d2a5f2d2-7356-496f-84ed-dfd3b2eea8e8" (UID: "d2a5f2d2-7356-496f-84ed-dfd3b2eea8e8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:10:34.042202 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:34.042164 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2a5f2d2-7356-496f-84ed-dfd3b2eea8e8-workload-certs" pod "d2a5f2d2-7356-496f-84ed-dfd3b2eea8e8" (UID: "d2a5f2d2-7356-496f-84ed-dfd3b2eea8e8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:10:34.050093 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:34.049803 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2a5f2d2-7356-496f-84ed-dfd3b2eea8e8-istio-data" pod "d2a5f2d2-7356-496f-84ed-dfd3b2eea8e8" (UID: "d2a5f2d2-7356-496f-84ed-dfd3b2eea8e8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:10:34.050324 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:34.050261 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/415fa10b-db0c-4ded-aedb-582612cbde2c-istio-token" pod "415fa10b-db0c-4ded-aedb-582612cbde2c" (UID: "415fa10b-db0c-4ded-aedb-582612cbde2c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:10:34.050562 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:34.050529 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/415fa10b-db0c-4ded-aedb-582612cbde2c-workload-certs" pod "415fa10b-db0c-4ded-aedb-582612cbde2c" (UID: "415fa10b-db0c-4ded-aedb-582612cbde2c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:10:34.051215 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:34.051173 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/415fa10b-db0c-4ded-aedb-582612cbde2c-istio-data" pod "415fa10b-db0c-4ded-aedb-582612cbde2c" (UID: "415fa10b-db0c-4ded-aedb-582612cbde2c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:10:34.053720 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:34.053692 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/415fa10b-db0c-4ded-aedb-582612cbde2c-istio-podinfo" pod "415fa10b-db0c-4ded-aedb-582612cbde2c" (UID: "415fa10b-db0c-4ded-aedb-582612cbde2c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:10:34.054558 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:34.054520 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/415fa10b-db0c-4ded-aedb-582612cbde2c-kube-api-access-nzm6n" pod "415fa10b-db0c-4ded-aedb-582612cbde2c" (UID: "415fa10b-db0c-4ded-aedb-582612cbde2c"). InnerVolumeSpecName "kube-api-access-nzm6n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:10:34.054774 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:34.054744 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/415fa10b-db0c-4ded-aedb-582612cbde2c-istio-envoy" pod "415fa10b-db0c-4ded-aedb-582612cbde2c" (UID: "415fa10b-db0c-4ded-aedb-582612cbde2c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:10:34.054866 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:34.054805 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2a5f2d2-7356-496f-84ed-dfd3b2eea8e8-kube-api-access-6vcq7" pod "d2a5f2d2-7356-496f-84ed-dfd3b2eea8e8" (UID: "d2a5f2d2-7356-496f-84ed-dfd3b2eea8e8"). InnerVolumeSpecName "kube-api-access-6vcq7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:10:34.054866 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:34.054853 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d2a5f2d2-7356-496f-84ed-dfd3b2eea8e8-istio-podinfo" pod "d2a5f2d2-7356-496f-84ed-dfd3b2eea8e8" (UID: "d2a5f2d2-7356-496f-84ed-dfd3b2eea8e8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:10:34.056136 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:34.056109 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2a5f2d2-7356-496f-84ed-dfd3b2eea8e8-istio-envoy" pod "d2a5f2d2-7356-496f-84ed-dfd3b2eea8e8" (UID: "d2a5f2d2-7356-496f-84ed-dfd3b2eea8e8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:10:34.056473 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:34.056455 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2a5f2d2-7356-496f-84ed-dfd3b2eea8e8-istio-token" pod "d2a5f2d2-7356-496f-84ed-dfd3b2eea8e8" (UID: "d2a5f2d2-7356-496f-84ed-dfd3b2eea8e8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:10:34.100991 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:34.100954 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-020af9-openshift-default-d77df45d5-vjg9x"] Nov 03 01:10:34.101857 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:34.101838 4421 cpu_manager.go:414] "containerMap: removing container" podUID="415fa10b-db0c-4ded-aedb-582612cbde2c" containerName="istio-proxy" Nov 03 01:10:34.101917 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:34.101859 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="415fa10b-db0c-4ded-aedb-582612cbde2c" containerName="istio-proxy" Nov 03 01:10:34.101917 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:34.101910 4421 cpu_manager.go:414] "containerMap: removing container" podUID="d2a5f2d2-7356-496f-84ed-dfd3b2eea8e8" containerName="istio-proxy" Nov 03 01:10:34.101917 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:34.101916 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2a5f2d2-7356-496f-84ed-dfd3b2eea8e8" containerName="istio-proxy" Nov 03 01:10:34.102041 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:34.102032 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="415fa10b-db0c-4ded-aedb-582612cbde2c" containerName="istio-proxy" Nov 03 01:10:34.102078 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:34.102051 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2a5f2d2-7356-496f-84ed-dfd3b2eea8e8" containerName="istio-proxy" Nov 03 01:10:34.107406 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:34.107379 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-020af9-openshift-default-d77df45d5-vjg9x" Nov 03 01:10:34.117521 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:34.117472 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-020af9-openshift-default-d77df45d5-vjg9x"] Nov 03 01:10:34.118025 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:34.117995 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-020af9-openshift-default-dockercfg-6wsrt\"" Nov 03 01:10:34.179348 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:34.176310 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fb3426e-5493-4655-a9e9-e011e9073012-workload-certs\") pod \"e2e-shared-b-w0-020af9-openshift-default-d77df45d5-vjg9x\" (UID: \"8fb3426e-5493-4655-a9e9-e011e9073012\") " pod="openshift-ingress/e2e-shared-b-w0-020af9-openshift-default-d77df45d5-vjg9x" Nov 03 01:10:34.179348 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:34.176428 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fb3426e-5493-4655-a9e9-e011e9073012-istio-data\") pod \"e2e-shared-b-w0-020af9-openshift-default-d77df45d5-vjg9x\" (UID: \"8fb3426e-5493-4655-a9e9-e011e9073012\") " pod="openshift-ingress/e2e-shared-b-w0-020af9-openshift-default-d77df45d5-vjg9x" Nov 03 01:10:34.179348 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:34.176479 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fb3426e-5493-4655-a9e9-e011e9073012-istiod-ca-cert\") pod \"e2e-shared-b-w0-020af9-openshift-default-d77df45d5-vjg9x\" (UID: \"8fb3426e-5493-4655-a9e9-e011e9073012\") " pod="openshift-ingress/e2e-shared-b-w0-020af9-openshift-default-d77df45d5-vjg9x" Nov 03 01:10:34.179348 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:34.176532 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fb3426e-5493-4655-a9e9-e011e9073012-istio-token\") pod \"e2e-shared-b-w0-020af9-openshift-default-d77df45d5-vjg9x\" (UID: \"8fb3426e-5493-4655-a9e9-e011e9073012\") " pod="openshift-ingress/e2e-shared-b-w0-020af9-openshift-default-d77df45d5-vjg9x" Nov 03 01:10:34.179348 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:34.176755 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fb3426e-5493-4655-a9e9-e011e9073012-istio-podinfo\") pod \"e2e-shared-b-w0-020af9-openshift-default-d77df45d5-vjg9x\" (UID: \"8fb3426e-5493-4655-a9e9-e011e9073012\") " pod="openshift-ingress/e2e-shared-b-w0-020af9-openshift-default-d77df45d5-vjg9x" Nov 03 01:10:34.179348 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:34.176920 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpvxn\" (UniqueName: \"kubernetes.io/projected/8fb3426e-5493-4655-a9e9-e011e9073012-kube-api-access-jpvxn\") pod \"e2e-shared-b-w0-020af9-openshift-default-d77df45d5-vjg9x\" (UID: \"8fb3426e-5493-4655-a9e9-e011e9073012\") " pod="openshift-ingress/e2e-shared-b-w0-020af9-openshift-default-d77df45d5-vjg9x" Nov 03 01:10:34.179348 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:34.176960 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fb3426e-5493-4655-a9e9-e011e9073012-credential-socket\") pod \"e2e-shared-b-w0-020af9-openshift-default-d77df45d5-vjg9x\" (UID: \"8fb3426e-5493-4655-a9e9-e011e9073012\") " pod="openshift-ingress/e2e-shared-b-w0-020af9-openshift-default-d77df45d5-vjg9x" Nov 03 01:10:34.179348 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:34.177044 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fb3426e-5493-4655-a9e9-e011e9073012-workload-socket\") pod \"e2e-shared-b-w0-020af9-openshift-default-d77df45d5-vjg9x\" (UID: \"8fb3426e-5493-4655-a9e9-e011e9073012\") " pod="openshift-ingress/e2e-shared-b-w0-020af9-openshift-default-d77df45d5-vjg9x" Nov 03 01:10:34.179348 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:34.177068 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fb3426e-5493-4655-a9e9-e011e9073012-istio-envoy\") pod \"e2e-shared-b-w0-020af9-openshift-default-d77df45d5-vjg9x\" (UID: \"8fb3426e-5493-4655-a9e9-e011e9073012\") " pod="openshift-ingress/e2e-shared-b-w0-020af9-openshift-default-d77df45d5-vjg9x" Nov 03 01:10:34.179348 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:34.177205 4421 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2a5f2d2-7356-496f-84ed-dfd3b2eea8e8-credential-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:10:34.179348 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:34.177225 4421 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2a5f2d2-7356-496f-84ed-dfd3b2eea8e8-workload-certs\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:10:34.179348 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:34.177241 4421 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/415fa10b-db0c-4ded-aedb-582612cbde2c-istio-envoy\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:10:34.179348 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:34.177253 4421 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/415fa10b-db0c-4ded-aedb-582612cbde2c-istio-token\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:10:34.179348 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:34.177267 4421 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/415fa10b-db0c-4ded-aedb-582612cbde2c-istio-podinfo\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:10:34.179348 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:34.177298 4421 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2a5f2d2-7356-496f-84ed-dfd3b2eea8e8-istio-token\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:10:34.179348 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:34.177310 4421 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/415fa10b-db0c-4ded-aedb-582612cbde2c-istio-data\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:10:34.179348 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:34.177323 4421 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2a5f2d2-7356-496f-84ed-dfd3b2eea8e8-istio-envoy\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:10:34.179348 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:34.177338 4421 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2a5f2d2-7356-496f-84ed-dfd3b2eea8e8-istio-data\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:10:34.179348 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:34.177351 4421 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2a5f2d2-7356-496f-84ed-dfd3b2eea8e8-istio-podinfo\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:10:34.179348 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:34.177365 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vcq7\" (UniqueName: \"kubernetes.io/projected/d2a5f2d2-7356-496f-84ed-dfd3b2eea8e8-kube-api-access-6vcq7\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:10:34.179348 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:34.177380 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nzm6n\" (UniqueName: \"kubernetes.io/projected/415fa10b-db0c-4ded-aedb-582612cbde2c-kube-api-access-nzm6n\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:10:34.179348 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:34.177394 4421 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/415fa10b-db0c-4ded-aedb-582612cbde2c-workload-certs\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:10:34.179348 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:34.177408 4421 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/415fa10b-db0c-4ded-aedb-582612cbde2c-istiod-ca-cert\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:10:34.179348 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:34.177423 4421 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2a5f2d2-7356-496f-84ed-dfd3b2eea8e8-istiod-ca-cert\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:10:34.179348 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:34.177436 4421 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/415fa10b-db0c-4ded-aedb-582612cbde2c-credential-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:10:34.179348 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:34.177450 4421 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2a5f2d2-7356-496f-84ed-dfd3b2eea8e8-workload-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:10:34.279451 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:34.279407 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpvxn\" (UniqueName: \"kubernetes.io/projected/8fb3426e-5493-4655-a9e9-e011e9073012-kube-api-access-jpvxn\") pod \"e2e-shared-b-w0-020af9-openshift-default-d77df45d5-vjg9x\" (UID: \"8fb3426e-5493-4655-a9e9-e011e9073012\") " pod="openshift-ingress/e2e-shared-b-w0-020af9-openshift-default-d77df45d5-vjg9x" Nov 03 01:10:34.279638 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:34.279470 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fb3426e-5493-4655-a9e9-e011e9073012-credential-socket\") pod \"e2e-shared-b-w0-020af9-openshift-default-d77df45d5-vjg9x\" (UID: \"8fb3426e-5493-4655-a9e9-e011e9073012\") " pod="openshift-ingress/e2e-shared-b-w0-020af9-openshift-default-d77df45d5-vjg9x" Nov 03 01:10:34.279638 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:34.279546 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fb3426e-5493-4655-a9e9-e011e9073012-workload-socket\") pod \"e2e-shared-b-w0-020af9-openshift-default-d77df45d5-vjg9x\" (UID: \"8fb3426e-5493-4655-a9e9-e011e9073012\") " pod="openshift-ingress/e2e-shared-b-w0-020af9-openshift-default-d77df45d5-vjg9x" Nov 03 01:10:34.279638 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:34.279570 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fb3426e-5493-4655-a9e9-e011e9073012-istio-envoy\") pod \"e2e-shared-b-w0-020af9-openshift-default-d77df45d5-vjg9x\" (UID: \"8fb3426e-5493-4655-a9e9-e011e9073012\") " pod="openshift-ingress/e2e-shared-b-w0-020af9-openshift-default-d77df45d5-vjg9x" Nov 03 01:10:34.279638 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:34.279629 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fb3426e-5493-4655-a9e9-e011e9073012-workload-certs\") pod \"e2e-shared-b-w0-020af9-openshift-default-d77df45d5-vjg9x\" (UID: \"8fb3426e-5493-4655-a9e9-e011e9073012\") " pod="openshift-ingress/e2e-shared-b-w0-020af9-openshift-default-d77df45d5-vjg9x" Nov 03 01:10:34.279867 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:34.279704 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fb3426e-5493-4655-a9e9-e011e9073012-istio-data\") pod \"e2e-shared-b-w0-020af9-openshift-default-d77df45d5-vjg9x\" (UID: \"8fb3426e-5493-4655-a9e9-e011e9073012\") " pod="openshift-ingress/e2e-shared-b-w0-020af9-openshift-default-d77df45d5-vjg9x" Nov 03 01:10:34.279867 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:34.279749 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fb3426e-5493-4655-a9e9-e011e9073012-istiod-ca-cert\") pod \"e2e-shared-b-w0-020af9-openshift-default-d77df45d5-vjg9x\" (UID: \"8fb3426e-5493-4655-a9e9-e011e9073012\") " pod="openshift-ingress/e2e-shared-b-w0-020af9-openshift-default-d77df45d5-vjg9x" Nov 03 01:10:34.279867 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:34.279800 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fb3426e-5493-4655-a9e9-e011e9073012-istio-token\") pod \"e2e-shared-b-w0-020af9-openshift-default-d77df45d5-vjg9x\" (UID: \"8fb3426e-5493-4655-a9e9-e011e9073012\") " pod="openshift-ingress/e2e-shared-b-w0-020af9-openshift-default-d77df45d5-vjg9x" Nov 03 01:10:34.280045 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:34.280017 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fb3426e-5493-4655-a9e9-e011e9073012-istio-podinfo\") pod \"e2e-shared-b-w0-020af9-openshift-default-d77df45d5-vjg9x\" (UID: \"8fb3426e-5493-4655-a9e9-e011e9073012\") " pod="openshift-ingress/e2e-shared-b-w0-020af9-openshift-default-d77df45d5-vjg9x" Nov 03 01:10:34.280834 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:34.280793 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fb3426e-5493-4655-a9e9-e011e9073012-istio-data\") pod \"e2e-shared-b-w0-020af9-openshift-default-d77df45d5-vjg9x\" (UID: \"8fb3426e-5493-4655-a9e9-e011e9073012\") " pod="openshift-ingress/e2e-shared-b-w0-020af9-openshift-default-d77df45d5-vjg9x" Nov 03 01:10:34.281162 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:34.281060 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fb3426e-5493-4655-a9e9-e011e9073012-workload-certs\") pod \"e2e-shared-b-w0-020af9-openshift-default-d77df45d5-vjg9x\" (UID: \"8fb3426e-5493-4655-a9e9-e011e9073012\") " pod="openshift-ingress/e2e-shared-b-w0-020af9-openshift-default-d77df45d5-vjg9x" Nov 03 01:10:34.281796 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:34.281769 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fb3426e-5493-4655-a9e9-e011e9073012-istiod-ca-cert\") pod \"e2e-shared-b-w0-020af9-openshift-default-d77df45d5-vjg9x\" (UID: \"8fb3426e-5493-4655-a9e9-e011e9073012\") " pod="openshift-ingress/e2e-shared-b-w0-020af9-openshift-default-d77df45d5-vjg9x" Nov 03 01:10:34.281965 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:34.281824 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fb3426e-5493-4655-a9e9-e011e9073012-credential-socket\") pod \"e2e-shared-b-w0-020af9-openshift-default-d77df45d5-vjg9x\" (UID: \"8fb3426e-5493-4655-a9e9-e011e9073012\") " pod="openshift-ingress/e2e-shared-b-w0-020af9-openshift-default-d77df45d5-vjg9x" Nov 03 01:10:34.281965 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:34.281838 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fb3426e-5493-4655-a9e9-e011e9073012-workload-socket\") pod \"e2e-shared-b-w0-020af9-openshift-default-d77df45d5-vjg9x\" (UID: \"8fb3426e-5493-4655-a9e9-e011e9073012\") " pod="openshift-ingress/e2e-shared-b-w0-020af9-openshift-default-d77df45d5-vjg9x" Nov 03 01:10:34.288854 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:34.288823 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fb3426e-5493-4655-a9e9-e011e9073012-istio-envoy\") pod \"e2e-shared-b-w0-020af9-openshift-default-d77df45d5-vjg9x\" (UID: \"8fb3426e-5493-4655-a9e9-e011e9073012\") " pod="openshift-ingress/e2e-shared-b-w0-020af9-openshift-default-d77df45d5-vjg9x" Nov 03 01:10:34.290530 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:34.290502 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fb3426e-5493-4655-a9e9-e011e9073012-istio-podinfo\") pod \"e2e-shared-b-w0-020af9-openshift-default-d77df45d5-vjg9x\" (UID: \"8fb3426e-5493-4655-a9e9-e011e9073012\") " pod="openshift-ingress/e2e-shared-b-w0-020af9-openshift-default-d77df45d5-vjg9x" Nov 03 01:10:34.298858 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:34.298823 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fb3426e-5493-4655-a9e9-e011e9073012-istio-token\") pod \"e2e-shared-b-w0-020af9-openshift-default-d77df45d5-vjg9x\" (UID: \"8fb3426e-5493-4655-a9e9-e011e9073012\") " pod="openshift-ingress/e2e-shared-b-w0-020af9-openshift-default-d77df45d5-vjg9x" Nov 03 01:10:34.299009 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:34.298905 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpvxn\" (UniqueName: \"kubernetes.io/projected/8fb3426e-5493-4655-a9e9-e011e9073012-kube-api-access-jpvxn\") pod \"e2e-shared-b-w0-020af9-openshift-default-d77df45d5-vjg9x\" (UID: \"8fb3426e-5493-4655-a9e9-e011e9073012\") " pod="openshift-ingress/e2e-shared-b-w0-020af9-openshift-default-d77df45d5-vjg9x" Nov 03 01:10:34.538680 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:34.538641 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-020af9-openshift-default-d77df45d5-vjg9x" Nov 03 01:10:34.664627 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:34.664588 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-a23ed0-openshift-default-84685cd7b7-6vgvw" Nov 03 01:10:34.671180 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:34.671154 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-a23ed0-openshift-default-84685cd7b7-6vgvw" Nov 03 01:10:34.857445 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:10:34.857412 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fb3426e_5493_4655_a9e9_e011e9073012.slice/crio-288abde84800e5c9324a1a74d4d366ffde9a740cc3eee99e2466c7a129168281 WatchSource:0}: Error finding container 288abde84800e5c9324a1a74d4d366ffde9a740cc3eee99e2466c7a129168281: Status 404 returned error can't find the container with id 288abde84800e5c9324a1a74d4d366ffde9a740cc3eee99e2466c7a129168281 Nov 03 01:10:34.858995 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:34.858969 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-020af9-openshift-default-d77df45d5-vjg9x"] Nov 03 01:10:34.861076 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:34.861032 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:10:34.861193 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:34.861122 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:10:34.861193 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:34.861169 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:10:34.962897 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:34.962858 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-020af9-openshift-default-d77df45d5-vjg9x" event={"ID":"8fb3426e-5493-4655-a9e9-e011e9073012","Type":"ContainerStarted","Data":"288abde84800e5c9324a1a74d4d366ffde9a740cc3eee99e2466c7a129168281"} Nov 03 01:10:34.964890 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:34.964868 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-r6t2p" Nov 03 01:10:34.965019 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:34.964871 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-r6t2p" event={"ID":"415fa10b-db0c-4ded-aedb-582612cbde2c","Type":"ContainerDied","Data":"b44de0085e4e176ea05ecc2b81c25fb1de526b95bfe2105bb70a5306b23160d4"} Nov 03 01:10:34.965019 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:34.964992 4421 scope.go:122] "RemoveContainer" containerID="978bb67569194ec8835b6dbf6bd9aeb3af47eda4ceba67c6d19e125e1e57ff30" Nov 03 01:10:34.965128 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:34.965071 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9lp9t" Nov 03 01:10:34.967113 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:34.967085 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-a23ed0-openshift-default-84685cd7b7-6vgvw" Nov 03 01:10:34.975545 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:34.975518 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-a23ed0-openshift-default-84685cd7b7-6vgvw" Nov 03 01:10:35.096687 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:35.096650 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9lp9t"] Nov 03 01:10:35.110131 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:35.109334 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9lp9t"] Nov 03 01:10:35.117334 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:35.117247 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-r6t2p"] Nov 03 01:10:35.122711 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:35.122679 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-r6t2p"] Nov 03 01:10:36.104043 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:36.103989 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="415fa10b-db0c-4ded-aedb-582612cbde2c" path="/var/lib/kubelet/pods/415fa10b-db0c-4ded-aedb-582612cbde2c/volumes" Nov 03 01:10:36.111009 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:36.110970 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2a5f2d2-7356-496f-84ed-dfd3b2eea8e8" path="/var/lib/kubelet/pods/d2a5f2d2-7356-496f-84ed-dfd3b2eea8e8/volumes" Nov 03 01:10:36.142294 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:36.136147 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-020af9-openshift-default-d77df45d5-vjg9x" event={"ID":"8fb3426e-5493-4655-a9e9-e011e9073012","Type":"ContainerStarted","Data":"cb7345056853e5742c1811f4fc210b604d8848d627f1ea560ff1b0fc2d781373"} Nov 03 01:10:36.171500 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:36.171448 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-020af9-openshift-default-d77df45d5-vjg9x" podStartSLOduration=2.171434053 podStartE2EDuration="2.171434053s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:10:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:10:36.169030746 +0000 UTC m=+1778.340033878" watchObservedRunningTime="2026-11-03 01:10:36.171434053 +0000 UTC m=+1778.342437149" Nov 03 01:10:36.539514 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:36.539479 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-020af9-openshift-default-d77df45d5-vjg9x" Nov 03 01:10:36.540853 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:36.540824 4421 patch_prober.go:29] interesting pod/e2e-shared-b-w0-020af9-openshift-default-d77df45d5-vjg9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.247:15021/healthz/ready\": dial tcp 10.134.0.247:15021: connect: connection refused" start-of-body= Nov 03 01:10:36.540987 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:36.540884 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-020af9-openshift-default-d77df45d5-vjg9x" podUID="8fb3426e-5493-4655-a9e9-e011e9073012" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.247:15021/healthz/ready\": dial tcp 10.134.0.247:15021: connect: connection refused" Nov 03 01:10:37.539656 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:37.539615 4421 patch_prober.go:29] interesting pod/e2e-shared-b-w0-020af9-openshift-default-d77df45d5-vjg9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.247:15021/healthz/ready\": dial tcp 10.134.0.247:15021: connect: connection refused" start-of-body= Nov 03 01:10:37.540078 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:37.539673 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-020af9-openshift-default-d77df45d5-vjg9x" podUID="8fb3426e-5493-4655-a9e9-e011e9073012" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.247:15021/healthz/ready\": dial tcp 10.134.0.247:15021: connect: connection refused" Nov 03 01:10:38.547332 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:38.546261 4421 patch_prober.go:29] interesting pod/e2e-shared-b-w0-020af9-openshift-default-d77df45d5-vjg9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.247:15021/healthz/ready\": dial tcp 10.134.0.247:15021: connect: connection refused" start-of-body= Nov 03 01:10:38.547332 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:38.546354 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-020af9-openshift-default-d77df45d5-vjg9x" podUID="8fb3426e-5493-4655-a9e9-e011e9073012" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.247:15021/healthz/ready\": dial tcp 10.134.0.247:15021: connect: connection refused" Nov 03 01:10:39.554301 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:39.542826 4421 patch_prober.go:29] interesting pod/e2e-shared-b-w0-020af9-openshift-default-d77df45d5-vjg9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.247:15021/healthz/ready\": dial tcp 10.134.0.247:15021: connect: connection refused" start-of-body= Nov 03 01:10:39.554301 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:39.542925 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-020af9-openshift-default-d77df45d5-vjg9x" podUID="8fb3426e-5493-4655-a9e9-e011e9073012" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.247:15021/healthz/ready\": dial tcp 10.134.0.247:15021: connect: connection refused" Nov 03 01:10:39.599513 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:39.599435 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/distinct-w3-83755a-kserve-86bdbbdd85-2cnnl" podUID="f4ae9624-d127-4a55-837d-7c9ff0267b37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 01:10:39.643497 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:39.643436 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-650be0/simulator-w4-650be0-kserve-7bc588774f-gvz2k" podUID="a827fa88-f078-4312-8f12-2c799c11b461" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:10:40.095513 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:40.095466 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w0-50e623/auth-test-model-50e623-kserve-7db6664d89-zdq97" podUID="e387182e-8e68-421f-b3ec-143283631b5e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.223:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:10:40.339313 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:40.338236 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-650be0/premium-w4-650be0-kserve-7bf8555dfb-z97n5" podUID="40601dc5-808f-456f-826f-6e0c35966ada" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:10:40.505303 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:40.501399 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/simulator-w3-83755a-kserve-77d6664559-psqtg" podUID="b0a4c234-af58-4da0-a1a4-e1f167953f6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:10:40.540424 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:40.540388 4421 patch_prober.go:29] interesting pod/e2e-shared-b-w0-020af9-openshift-default-d77df45d5-vjg9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.247:15021/healthz/ready\": dial tcp 10.134.0.247:15021: connect: connection refused" start-of-body= Nov 03 01:10:40.540690 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:40.540665 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-020af9-openshift-default-d77df45d5-vjg9x" podUID="8fb3426e-5493-4655-a9e9-e011e9073012" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.247:15021/healthz/ready\": dial tcp 10.134.0.247:15021: connect: connection refused" Nov 03 01:10:40.629397 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:40.629335 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/distinct-w3-83755a-kserve-86bdbbdd85-2cnnl" podUID="f4ae9624-d127-4a55-837d-7c9ff0267b37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded" Nov 03 01:10:40.630713 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:40.630675 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-83755a/distinct-w3-83755a-kserve-86bdbbdd85-2cnnl" Nov 03 01:10:40.656090 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:40.655962 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-650be0/simulator-w4-650be0-kserve-7bc588774f-gvz2k" podUID="a827fa88-f078-4312-8f12-2c799c11b461" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:10:40.658019 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:40.657996 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-650be0/simulator-w4-650be0-kserve-7bc588774f-gvz2k" Nov 03 01:10:40.723637 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:40.723589 4421 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-f7680d/simulator-w2-f7680d-kserve-85bdc86cd6-8gxrg" podUID="2bdcf521-cebd-4d6b-8554-1b60db8b0f3c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:10:41.107519 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:41.107457 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w0-50e623/auth-test-model-50e623-kserve-7db6664d89-zdq97" podUID="e387182e-8e68-421f-b3ec-143283631b5e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.223:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:10:41.162300 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:41.132846 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w0-50e623/auth-test-model-50e623-kserve-7db6664d89-zdq97" Nov 03 01:10:41.293402 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:41.276233 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8gm8n" podUID="2d367e12-e46f-4a68-9825-9b1c934d4d12" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": context deadline exceeded" Nov 03 01:10:41.315204 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:41.315124 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-83755a/distinct-w3-83755a-kserve-86bdbbdd85-2cnnl" Nov 03 01:10:41.377451 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:41.377343 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/simulator-w2-f7680d-kserve-85bdc86cd6-8gxrg" podUID="2bdcf521-cebd-4d6b-8554-1b60db8b0f3c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 01:10:41.486436 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:41.486402 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-650be0/simulator-w4-650be0-kserve-7bc588774f-gvz2k" Nov 03 01:10:41.555970 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:41.555940 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w0-50e623/auth-test-model-50e623-kserve-7db6664d89-zdq97" Nov 03 01:10:41.557126 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:41.556902 4421 patch_prober.go:29] interesting pod/e2e-shared-b-w0-020af9-openshift-default-d77df45d5-vjg9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.247:15021/healthz/ready\": dial tcp 10.134.0.247:15021: connect: connection refused" start-of-body= Nov 03 01:10:41.557625 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:41.557569 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-020af9-openshift-default-d77df45d5-vjg9x" podUID="8fb3426e-5493-4655-a9e9-e011e9073012" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.247:15021/healthz/ready\": dial tcp 10.134.0.247:15021: connect: connection refused" Nov 03 01:10:42.539327 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:42.539265 4421 patch_prober.go:29] interesting pod/e2e-shared-b-w0-020af9-openshift-default-d77df45d5-vjg9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.247:15021/healthz/ready\": dial tcp 10.134.0.247:15021: connect: connection refused" start-of-body= Nov 03 01:10:42.539773 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:42.539366 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-020af9-openshift-default-d77df45d5-vjg9x" podUID="8fb3426e-5493-4655-a9e9-e011e9073012" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.247:15021/healthz/ready\": dial tcp 10.134.0.247:15021: connect: connection refused" Nov 03 01:10:43.539967 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:43.539923 4421 patch_prober.go:29] interesting pod/e2e-shared-b-w0-020af9-openshift-default-d77df45d5-vjg9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.247:15021/healthz/ready\": dial tcp 10.134.0.247:15021: connect: connection refused" start-of-body= Nov 03 01:10:43.540522 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:43.540016 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-020af9-openshift-default-d77df45d5-vjg9x" podUID="8fb3426e-5493-4655-a9e9-e011e9073012" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.247:15021/healthz/ready\": dial tcp 10.134.0.247:15021: connect: connection refused" Nov 03 01:10:44.545126 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:44.545094 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-020af9-openshift-default-d77df45d5-vjg9x" Nov 03 01:10:44.545697 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:44.545680 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-020af9-openshift-default-d77df45d5-vjg9x" Nov 03 01:10:44.546490 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:44.546466 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-020af9-openshift-default-d77df45d5-vjg9x" Nov 03 01:10:44.681894 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:10:44.681854 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 01:10:44.922657 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:44.922556 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w0-50e623/auth-test-model-50e623-kserve-7db6664d89-zdq97" podUID="e387182e-8e68-421f-b3ec-143283631b5e" containerName="main" containerID="cri-o://2663ce90c2fb9130b1280134a6f7b3f3b5f613972bf017716d1a7e7c48d6e23d" gracePeriod=45 Nov 03 01:10:45.070361 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:45.070296 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w0-50e623/auth-test-model-50e623-kserve-7db6664d89-zdq97" podUID="e387182e-8e68-421f-b3ec-143283631b5e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.223:8000/ready\": dial tcp 10.134.0.223:8000: connect: connection refused" Nov 03 01:10:45.257584 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:45.257552 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-50e623/auth-test-model-50e623-kserve-7db6664d89-zdq97" Nov 03 01:10:45.285804 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:45.285765 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e387182e-8e68-421f-b3ec-143283631b5e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e387182e-8e68-421f-b3ec-143283631b5e-dshm\") pod \"e387182e-8e68-421f-b3ec-143283631b5e\" (UID: \"e387182e-8e68-421f-b3ec-143283631b5e\") " Nov 03 01:10:45.286060 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:45.285868 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e387182e-8e68-421f-b3ec-143283631b5e-home\" (UniqueName: \"kubernetes.io/empty-dir/e387182e-8e68-421f-b3ec-143283631b5e-home\") pod \"e387182e-8e68-421f-b3ec-143283631b5e\" (UID: \"e387182e-8e68-421f-b3ec-143283631b5e\") " Nov 03 01:10:45.286060 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:45.285907 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e387182e-8e68-421f-b3ec-143283631b5e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e387182e-8e68-421f-b3ec-143283631b5e-tmp-dir\") pod \"e387182e-8e68-421f-b3ec-143283631b5e\" (UID: \"e387182e-8e68-421f-b3ec-143283631b5e\") " Nov 03 01:10:45.286060 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:45.285937 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e387182e-8e68-421f-b3ec-143283631b5e-tls-certs\" (UniqueName: \"kubernetes.io/secret/e387182e-8e68-421f-b3ec-143283631b5e-tls-certs\") pod \"e387182e-8e68-421f-b3ec-143283631b5e\" (UID: \"e387182e-8e68-421f-b3ec-143283631b5e\") " Nov 03 01:10:45.286060 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:45.286030 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e387182e-8e68-421f-b3ec-143283631b5e-kube-api-access-vn7kf\" (UniqueName: \"kubernetes.io/projected/e387182e-8e68-421f-b3ec-143283631b5e-kube-api-access-vn7kf\") pod \"e387182e-8e68-421f-b3ec-143283631b5e\" (UID: \"e387182e-8e68-421f-b3ec-143283631b5e\") " Nov 03 01:10:45.286401 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:45.286141 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e387182e-8e68-421f-b3ec-143283631b5e-home" pod "e387182e-8e68-421f-b3ec-143283631b5e" (UID: "e387182e-8e68-421f-b3ec-143283631b5e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:10:45.286401 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:45.286189 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e387182e-8e68-421f-b3ec-143283631b5e-tmp-dir" pod "e387182e-8e68-421f-b3ec-143283631b5e" (UID: "e387182e-8e68-421f-b3ec-143283631b5e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:10:45.286527 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:45.286458 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e387182e-8e68-421f-b3ec-143283631b5e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e387182e-8e68-421f-b3ec-143283631b5e-model-cache\") pod \"e387182e-8e68-421f-b3ec-143283631b5e\" (UID: \"e387182e-8e68-421f-b3ec-143283631b5e\") " Nov 03 01:10:45.286961 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:45.286925 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e387182e-8e68-421f-b3ec-143283631b5e-model-cache" pod "e387182e-8e68-421f-b3ec-143283631b5e" (UID: "e387182e-8e68-421f-b3ec-143283631b5e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:10:45.287434 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:45.287408 4421 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e387182e-8e68-421f-b3ec-143283631b5e-model-cache\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:10:45.287434 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:45.287436 4421 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e387182e-8e68-421f-b3ec-143283631b5e-home\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:10:45.287624 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:45.287450 4421 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e387182e-8e68-421f-b3ec-143283631b5e-tmp-dir\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:10:45.291956 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:45.291924 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e387182e-8e68-421f-b3ec-143283631b5e-dshm" pod "e387182e-8e68-421f-b3ec-143283631b5e" (UID: "e387182e-8e68-421f-b3ec-143283631b5e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:10:45.295726 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:45.295697 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e387182e-8e68-421f-b3ec-143283631b5e-tls-certs" pod "e387182e-8e68-421f-b3ec-143283631b5e" (UID: "e387182e-8e68-421f-b3ec-143283631b5e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:10:45.295840 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:45.295802 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e387182e-8e68-421f-b3ec-143283631b5e-kube-api-access-vn7kf" pod "e387182e-8e68-421f-b3ec-143283631b5e" (UID: "e387182e-8e68-421f-b3ec-143283631b5e"). InnerVolumeSpecName "kube-api-access-vn7kf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:10:45.399897 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:45.399865 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vn7kf\" (UniqueName: \"kubernetes.io/projected/e387182e-8e68-421f-b3ec-143283631b5e-kube-api-access-vn7kf\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:10:45.399897 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:45.399895 4421 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e387182e-8e68-421f-b3ec-143283631b5e-dshm\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:10:45.400146 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:45.399908 4421 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e387182e-8e68-421f-b3ec-143283631b5e-tls-certs\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:10:45.591232 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:45.591198 4421 generic.go:356] "Generic (PLEG): container finished" podID="e387182e-8e68-421f-b3ec-143283631b5e" containerID="2663ce90c2fb9130b1280134a6f7b3f3b5f613972bf017716d1a7e7c48d6e23d" exitCode=0 Nov 03 01:10:45.591689 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:45.591258 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-50e623/auth-test-model-50e623-kserve-7db6664d89-zdq97" Nov 03 01:10:45.591689 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:45.591313 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-50e623/auth-test-model-50e623-kserve-7db6664d89-zdq97" event={"ID":"e387182e-8e68-421f-b3ec-143283631b5e","Type":"ContainerDied","Data":"2663ce90c2fb9130b1280134a6f7b3f3b5f613972bf017716d1a7e7c48d6e23d"} Nov 03 01:10:45.591689 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:45.591356 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-50e623/auth-test-model-50e623-kserve-7db6664d89-zdq97" event={"ID":"e387182e-8e68-421f-b3ec-143283631b5e","Type":"ContainerDied","Data":"831cd8e349e5b7e201b1153d06936c2d3d8d98e91365dfd91d37005a319be9a8"} Nov 03 01:10:45.591689 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:45.591373 4421 scope.go:122] "RemoveContainer" containerID="2663ce90c2fb9130b1280134a6f7b3f3b5f613972bf017716d1a7e7c48d6e23d" Nov 03 01:10:45.604902 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:45.604882 4421 scope.go:122] "RemoveContainer" containerID="2663ce90c2fb9130b1280134a6f7b3f3b5f613972bf017716d1a7e7c48d6e23d" Nov 03 01:10:45.605189 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:10:45.605167 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2663ce90c2fb9130b1280134a6f7b3f3b5f613972bf017716d1a7e7c48d6e23d\": container with ID starting with 2663ce90c2fb9130b1280134a6f7b3f3b5f613972bf017716d1a7e7c48d6e23d not found: ID does not exist" containerID="2663ce90c2fb9130b1280134a6f7b3f3b5f613972bf017716d1a7e7c48d6e23d" Nov 03 01:10:45.605323 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:45.605194 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2663ce90c2fb9130b1280134a6f7b3f3b5f613972bf017716d1a7e7c48d6e23d"} err="failed to get container status \"2663ce90c2fb9130b1280134a6f7b3f3b5f613972bf017716d1a7e7c48d6e23d\": rpc error: code = NotFound desc = could not find container \"2663ce90c2fb9130b1280134a6f7b3f3b5f613972bf017716d1a7e7c48d6e23d\": container with ID starting with 2663ce90c2fb9130b1280134a6f7b3f3b5f613972bf017716d1a7e7c48d6e23d not found: ID does not exist" Nov 03 01:10:45.649891 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:45.649854 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-50e623/auth-test-model-50e623-kserve-7db6664d89-zdq97"] Nov 03 01:10:45.662790 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:45.662756 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w0-50e623/auth-test-model-50e623-kserve-7db6664d89-zdq97"] Nov 03 01:10:45.981597 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:45.981504 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e387182e-8e68-421f-b3ec-143283631b5e" path="/var/lib/kubelet/pods/e387182e-8e68-421f-b3ec-143283631b5e/volumes" Nov 03 01:10:46.015634 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:46.015600 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-t5bt4"] Nov 03 01:10:46.015847 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:46.015819 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-t5bt4" podUID="2b55c151-d235-4bcc-b602-eca0da4a7348" containerName="istio-proxy" containerID="cri-o://28c9237ffae3781146e130dd9c4cd0da86a188f0798d40ea2f29e94c02af4605" gracePeriod=30 Nov 03 01:10:46.230260 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:46.230229 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4263775b-xb4px"] Nov 03 01:10:46.231089 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:46.231073 4421 cpu_manager.go:414] "containerMap: removing container" podUID="e387182e-8e68-421f-b3ec-143283631b5e" containerName="main" Nov 03 01:10:46.231145 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:46.231091 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e387182e-8e68-421f-b3ec-143283631b5e" containerName="main" Nov 03 01:10:46.231218 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:46.231208 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="e387182e-8e68-421f-b3ec-143283631b5e" containerName="main" Nov 03 01:10:46.236684 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:46.236623 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4263775b-xb4px" Nov 03 01:10:46.240373 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:46.240353 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 01:10:46.240553 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:46.240411 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:10:46.240553 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:46.240496 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 01:10:46.240735 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:46.240635 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6q4xp\"" Nov 03 01:10:46.241907 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:46.241888 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4263775b-xb4px"] Nov 03 01:10:46.316172 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:46.316124 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99d7371e-75c2-4bbf-98ee-9c6bc419b779-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4263775b-xb4px\" (UID: \"99d7371e-75c2-4bbf-98ee-9c6bc419b779\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4263775b-xb4px" Nov 03 01:10:46.316382 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:46.316255 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rhwg\" (UniqueName: \"kubernetes.io/projected/99d7371e-75c2-4bbf-98ee-9c6bc419b779-kube-api-access-4rhwg\") pod \"maas-api-revoke-sub-4263775b-xb4px\" (UID: \"99d7371e-75c2-4bbf-98ee-9c6bc419b779\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4263775b-xb4px" Nov 03 01:10:46.316519 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:46.316501 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99d7371e-75c2-4bbf-98ee-9c6bc419b779-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4263775b-xb4px\" (UID: \"99d7371e-75c2-4bbf-98ee-9c6bc419b779\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4263775b-xb4px" Nov 03 01:10:46.418561 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:46.418505 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99d7371e-75c2-4bbf-98ee-9c6bc419b779-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4263775b-xb4px\" (UID: \"99d7371e-75c2-4bbf-98ee-9c6bc419b779\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4263775b-xb4px" Nov 03 01:10:46.418786 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:46.418628 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rhwg\" (UniqueName: \"kubernetes.io/projected/99d7371e-75c2-4bbf-98ee-9c6bc419b779-kube-api-access-4rhwg\") pod \"maas-api-revoke-sub-4263775b-xb4px\" (UID: \"99d7371e-75c2-4bbf-98ee-9c6bc419b779\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4263775b-xb4px" Nov 03 01:10:46.418937 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:46.418917 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99d7371e-75c2-4bbf-98ee-9c6bc419b779-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4263775b-xb4px\" (UID: \"99d7371e-75c2-4bbf-98ee-9c6bc419b779\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4263775b-xb4px" Nov 03 01:10:46.419264 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:46.419242 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99d7371e-75c2-4bbf-98ee-9c6bc419b779-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4263775b-xb4px\" (UID: \"99d7371e-75c2-4bbf-98ee-9c6bc419b779\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4263775b-xb4px" Nov 03 01:10:46.421583 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:46.421560 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99d7371e-75c2-4bbf-98ee-9c6bc419b779-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4263775b-xb4px\" (UID: \"99d7371e-75c2-4bbf-98ee-9c6bc419b779\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4263775b-xb4px" Nov 03 01:10:46.426805 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:46.426781 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rhwg\" (UniqueName: \"kubernetes.io/projected/99d7371e-75c2-4bbf-98ee-9c6bc419b779-kube-api-access-4rhwg\") pod \"maas-api-revoke-sub-4263775b-xb4px\" (UID: \"99d7371e-75c2-4bbf-98ee-9c6bc419b779\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4263775b-xb4px" Nov 03 01:10:46.556816 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:46.556777 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4263775b-xb4px" Nov 03 01:10:46.851491 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:46.851454 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4263775b-xb4px"] Nov 03 01:10:46.851996 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:10:46.851728 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99d7371e_75c2_4bbf_98ee_9c6bc419b779.slice/crio-ccef21b496b1c7a0333d019f47e5836d836799ec1d4d17fc0421b9cb1ebd59c2 WatchSource:0}: Error finding container ccef21b496b1c7a0333d019f47e5836d836799ec1d4d17fc0421b9cb1ebd59c2: Status 404 returned error can't find the container with id ccef21b496b1c7a0333d019f47e5836d836799ec1d4d17fc0421b9cb1ebd59c2 Nov 03 01:10:48.240301 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:48.239663 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4263775b-xb4px" event={"ID":"99d7371e-75c2-4bbf-98ee-9c6bc419b779","Type":"ContainerStarted","Data":"ee4dbcd3e39c067fbe13dfe1b63aedc2e2e7891bb3c5a330e26dab0d05f747b7"} Nov 03 01:10:48.240301 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:48.239732 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4263775b-xb4px" event={"ID":"99d7371e-75c2-4bbf-98ee-9c6bc419b779","Type":"ContainerStarted","Data":"ccef21b496b1c7a0333d019f47e5836d836799ec1d4d17fc0421b9cb1ebd59c2"} Nov 03 01:10:48.240301 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:48.239763 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-a23ed0-openshift-default-84685cd7b7-6vgvw"] Nov 03 01:10:48.240301 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:48.239975 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-a23ed0-openshift-default-84685cd7b7-6vgvw" podUID="065c5377-f408-4f5c-be1c-19e4cea9e64e" containerName="istio-proxy" containerID="cri-o://9202bd26541cdff38cf8d19d0e98ac483f3ae57b4238e4611a2d9af68350a4b9" gracePeriod=30 Nov 03 01:10:48.382652 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:48.382483 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4263775b-xb4px" podStartSLOduration=2.3824662549999998 podStartE2EDuration="2.382466255s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:10:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:10:48.343036703 +0000 UTC m=+1790.514039800" watchObservedRunningTime="2026-11-03 01:10:48.382466255 +0000 UTC m=+1790.553469350" Nov 03 01:10:49.218537 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:49.218502 4421 generic.go:356] "Generic (PLEG): container finished" podID="99d7371e-75c2-4bbf-98ee-9c6bc419b779" containerID="ee4dbcd3e39c067fbe13dfe1b63aedc2e2e7891bb3c5a330e26dab0d05f747b7" exitCode=0 Nov 03 01:10:49.218743 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:49.218663 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4263775b-xb4px" event={"ID":"99d7371e-75c2-4bbf-98ee-9c6bc419b779","Type":"ContainerDied","Data":"ee4dbcd3e39c067fbe13dfe1b63aedc2e2e7891bb3c5a330e26dab0d05f747b7"} Nov 03 01:10:49.983358 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:49.981451 4421 patch_prober.go:29] interesting pod/e2e-noten-a23ed0-openshift-default-84685cd7b7-6vgvw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 01:10:49.983358 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:49.981524 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-noten-a23ed0-openshift-default-84685cd7b7-6vgvw" podUID="065c5377-f408-4f5c-be1c-19e4cea9e64e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 01:10:50.413703 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:50.413675 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4263775b-xb4px" Nov 03 01:10:50.541823 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:50.541732 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99d7371e-75c2-4bbf-98ee-9c6bc419b779-kube-api-access-4rhwg\" (UniqueName: \"kubernetes.io/projected/99d7371e-75c2-4bbf-98ee-9c6bc419b779-kube-api-access-4rhwg\") pod \"99d7371e-75c2-4bbf-98ee-9c6bc419b779\" (UID: \"99d7371e-75c2-4bbf-98ee-9c6bc419b779\") " Nov 03 01:10:50.541998 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:50.541974 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99d7371e-75c2-4bbf-98ee-9c6bc419b779-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99d7371e-75c2-4bbf-98ee-9c6bc419b779-maas-api-openshift-ca\") pod \"99d7371e-75c2-4bbf-98ee-9c6bc419b779\" (UID: \"99d7371e-75c2-4bbf-98ee-9c6bc419b779\") " Nov 03 01:10:50.542140 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:50.542127 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/99d7371e-75c2-4bbf-98ee-9c6bc419b779-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99d7371e-75c2-4bbf-98ee-9c6bc419b779-maas-api-xks-ca\") pod \"99d7371e-75c2-4bbf-98ee-9c6bc419b779\" (UID: \"99d7371e-75c2-4bbf-98ee-9c6bc419b779\") " Nov 03 01:10:50.542329 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:50.542305 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99d7371e-75c2-4bbf-98ee-9c6bc419b779-maas-api-openshift-ca" pod "99d7371e-75c2-4bbf-98ee-9c6bc419b779" (UID: "99d7371e-75c2-4bbf-98ee-9c6bc419b779"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:10:50.542771 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:50.542748 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99d7371e-75c2-4bbf-98ee-9c6bc419b779-maas-api-openshift-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:10:50.544194 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:50.544165 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99d7371e-75c2-4bbf-98ee-9c6bc419b779-kube-api-access-4rhwg" pod "99d7371e-75c2-4bbf-98ee-9c6bc419b779" (UID: "99d7371e-75c2-4bbf-98ee-9c6bc419b779"). InnerVolumeSpecName "kube-api-access-4rhwg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:10:50.544371 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:50.544349 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99d7371e-75c2-4bbf-98ee-9c6bc419b779-maas-api-xks-ca" pod "99d7371e-75c2-4bbf-98ee-9c6bc419b779" (UID: "99d7371e-75c2-4bbf-98ee-9c6bc419b779"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:10:50.654326 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:50.652125 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99d7371e-75c2-4bbf-98ee-9c6bc419b779-maas-api-xks-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:10:50.654326 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:50.652167 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rhwg\" (UniqueName: \"kubernetes.io/projected/99d7371e-75c2-4bbf-98ee-9c6bc419b779-kube-api-access-4rhwg\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:10:51.243817 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:51.243775 4421 generic.go:356] "Generic (PLEG): container finished" podID="2b55c151-d235-4bcc-b602-eca0da4a7348" containerID="28c9237ffae3781146e130dd9c4cd0da86a188f0798d40ea2f29e94c02af4605" exitCode=0 Nov 03 01:10:51.244321 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:51.243848 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-t5bt4" event={"ID":"2b55c151-d235-4bcc-b602-eca0da4a7348","Type":"ContainerDied","Data":"28c9237ffae3781146e130dd9c4cd0da86a188f0798d40ea2f29e94c02af4605"} Nov 03 01:10:51.245701 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:51.245679 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4263775b-xb4px" event={"ID":"99d7371e-75c2-4bbf-98ee-9c6bc419b779","Type":"ContainerDied","Data":"ccef21b496b1c7a0333d019f47e5836d836799ec1d4d17fc0421b9cb1ebd59c2"} Nov 03 01:10:51.245796 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:51.245721 4421 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ccef21b496b1c7a0333d019f47e5836d836799ec1d4d17fc0421b9cb1ebd59c2" Nov 03 01:10:51.245796 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:51.245753 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4263775b-xb4px" Nov 03 01:10:51.291321 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:51.291261 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-t5bt4" Nov 03 01:10:51.342605 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:51.339772 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b55c151-d235-4bcc-b602-eca0da4a7348-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b55c151-d235-4bcc-b602-eca0da4a7348-credential-socket\") pod \"2b55c151-d235-4bcc-b602-eca0da4a7348\" (UID: \"2b55c151-d235-4bcc-b602-eca0da4a7348\") " Nov 03 01:10:51.342605 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:51.339833 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b55c151-d235-4bcc-b602-eca0da4a7348-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b55c151-d235-4bcc-b602-eca0da4a7348-istiod-ca-cert\") pod \"2b55c151-d235-4bcc-b602-eca0da4a7348\" (UID: \"2b55c151-d235-4bcc-b602-eca0da4a7348\") " Nov 03 01:10:51.342605 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:51.339866 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b55c151-d235-4bcc-b602-eca0da4a7348-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b55c151-d235-4bcc-b602-eca0da4a7348-istio-envoy\") pod \"2b55c151-d235-4bcc-b602-eca0da4a7348\" (UID: \"2b55c151-d235-4bcc-b602-eca0da4a7348\") " Nov 03 01:10:51.342605 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:51.339997 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b55c151-d235-4bcc-b602-eca0da4a7348-kube-api-access-bb9cl\" (UniqueName: \"kubernetes.io/projected/2b55c151-d235-4bcc-b602-eca0da4a7348-kube-api-access-bb9cl\") pod \"2b55c151-d235-4bcc-b602-eca0da4a7348\" (UID: \"2b55c151-d235-4bcc-b602-eca0da4a7348\") " Nov 03 01:10:51.342605 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:51.340052 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b55c151-d235-4bcc-b602-eca0da4a7348-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b55c151-d235-4bcc-b602-eca0da4a7348-workload-socket\") pod \"2b55c151-d235-4bcc-b602-eca0da4a7348\" (UID: \"2b55c151-d235-4bcc-b602-eca0da4a7348\") " Nov 03 01:10:51.342605 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:51.340235 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2b55c151-d235-4bcc-b602-eca0da4a7348-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b55c151-d235-4bcc-b602-eca0da4a7348-istio-podinfo\") pod \"2b55c151-d235-4bcc-b602-eca0da4a7348\" (UID: \"2b55c151-d235-4bcc-b602-eca0da4a7348\") " Nov 03 01:10:51.342605 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:51.340293 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b55c151-d235-4bcc-b602-eca0da4a7348-credential-socket" pod "2b55c151-d235-4bcc-b602-eca0da4a7348" (UID: "2b55c151-d235-4bcc-b602-eca0da4a7348"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:10:51.342605 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:51.340320 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b55c151-d235-4bcc-b602-eca0da4a7348-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b55c151-d235-4bcc-b602-eca0da4a7348-istio-data\") pod \"2b55c151-d235-4bcc-b602-eca0da4a7348\" (UID: \"2b55c151-d235-4bcc-b602-eca0da4a7348\") " Nov 03 01:10:51.342605 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:51.340356 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b55c151-d235-4bcc-b602-eca0da4a7348-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b55c151-d235-4bcc-b602-eca0da4a7348-workload-certs\") pod \"2b55c151-d235-4bcc-b602-eca0da4a7348\" (UID: \"2b55c151-d235-4bcc-b602-eca0da4a7348\") " Nov 03 01:10:51.342605 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:51.340389 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b55c151-d235-4bcc-b602-eca0da4a7348-istio-token\" (UniqueName: \"kubernetes.io/projected/2b55c151-d235-4bcc-b602-eca0da4a7348-istio-token\") pod \"2b55c151-d235-4bcc-b602-eca0da4a7348\" (UID: \"2b55c151-d235-4bcc-b602-eca0da4a7348\") " Nov 03 01:10:51.342605 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:51.341166 4421 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b55c151-d235-4bcc-b602-eca0da4a7348-credential-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:10:51.342605 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:51.341755 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b55c151-d235-4bcc-b602-eca0da4a7348-workload-socket" pod "2b55c151-d235-4bcc-b602-eca0da4a7348" (UID: "2b55c151-d235-4bcc-b602-eca0da4a7348"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:10:51.342605 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:51.342041 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b55c151-d235-4bcc-b602-eca0da4a7348-istiod-ca-cert" pod "2b55c151-d235-4bcc-b602-eca0da4a7348" (UID: "2b55c151-d235-4bcc-b602-eca0da4a7348"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:10:51.342605 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:51.342329 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b55c151-d235-4bcc-b602-eca0da4a7348-istio-data" pod "2b55c151-d235-4bcc-b602-eca0da4a7348" (UID: "2b55c151-d235-4bcc-b602-eca0da4a7348"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:10:51.342605 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:51.342512 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b55c151-d235-4bcc-b602-eca0da4a7348-workload-certs" pod "2b55c151-d235-4bcc-b602-eca0da4a7348" (UID: "2b55c151-d235-4bcc-b602-eca0da4a7348"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:10:51.343450 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:51.343114 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b55c151-d235-4bcc-b602-eca0da4a7348-kube-api-access-bb9cl" pod "2b55c151-d235-4bcc-b602-eca0da4a7348" (UID: "2b55c151-d235-4bcc-b602-eca0da4a7348"). InnerVolumeSpecName "kube-api-access-bb9cl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:10:51.346818 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:51.345339 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b55c151-d235-4bcc-b602-eca0da4a7348-istio-envoy" pod "2b55c151-d235-4bcc-b602-eca0da4a7348" (UID: "2b55c151-d235-4bcc-b602-eca0da4a7348"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:10:51.349905 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:51.347152 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b55c151-d235-4bcc-b602-eca0da4a7348-istio-token" pod "2b55c151-d235-4bcc-b602-eca0da4a7348" (UID: "2b55c151-d235-4bcc-b602-eca0da4a7348"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:10:51.351841 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:51.351802 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2b55c151-d235-4bcc-b602-eca0da4a7348-istio-podinfo" pod "2b55c151-d235-4bcc-b602-eca0da4a7348" (UID: "2b55c151-d235-4bcc-b602-eca0da4a7348"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:10:51.429130 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:51.429092 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-55d6nfg5v"] Nov 03 01:10:51.430360 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:51.430336 4421 cpu_manager.go:414] "containerMap: removing container" podUID="2b55c151-d235-4bcc-b602-eca0da4a7348" containerName="istio-proxy" Nov 03 01:10:51.430360 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:51.430358 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b55c151-d235-4bcc-b602-eca0da4a7348" containerName="istio-proxy" Nov 03 01:10:51.430509 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:51.430382 4421 cpu_manager.go:414] "containerMap: removing container" podUID="99d7371e-75c2-4bbf-98ee-9c6bc419b779" containerName="revoke-keys" Nov 03 01:10:51.430509 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:51.430388 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99d7371e-75c2-4bbf-98ee-9c6bc419b779" containerName="revoke-keys" Nov 03 01:10:51.430587 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:51.430526 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b55c151-d235-4bcc-b602-eca0da4a7348" containerName="istio-proxy" Nov 03 01:10:51.430587 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:51.430556 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="99d7371e-75c2-4bbf-98ee-9c6bc419b779" containerName="revoke-keys" Nov 03 01:10:51.435486 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:51.435466 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-55d6nfg5v" Nov 03 01:10:51.444828 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:51.444668 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-af8244\"/\"routing-overlay\"" Nov 03 01:10:51.444828 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:51.444686 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-af8244\"/\"kube-root-ca.crt\"" Nov 03 01:10:51.444828 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:51.444672 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-af8244\"/\"openshift-service-ca.crt\"" Nov 03 01:10:51.444828 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:51.444712 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-af8244\"/\"agc-cred-e8606c64533f\"" Nov 03 01:10:51.444828 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:51.444730 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-af8244\"/\"payload-processing-external-model-plugins-e2e-extmod-b-af8244\"" Nov 03 01:10:51.445191 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:51.444861 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-af8244\"/\"payload-processing-external-model-e2e--fa7ecc9b-dockercfg-95w8h\"" Nov 03 01:10:51.445241 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:51.445193 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-55d6nfg5v"] Nov 03 01:10:51.445546 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:51.445521 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsmnf\" (UniqueName: \"kubernetes.io/projected/31e81deb-0e7e-4c26-b43f-870959edffe8-kube-api-access-gsmnf\") pod \"payload-processing-external-model-e2e-extmod-b-af8244-55d6nfg5v\" (UID: \"31e81deb-0e7e-4c26-b43f-870959edffe8\") " pod="ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-55d6nfg5v" Nov 03 01:10:51.445755 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:51.445735 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/31e81deb-0e7e-4c26-b43f-870959edffe8-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-af8244-55d6nfg5v\" (UID: \"31e81deb-0e7e-4c26-b43f-870959edffe8\") " pod="ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-55d6nfg5v" Nov 03 01:10:51.445902 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:51.445887 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31e81deb-0e7e-4c26-b43f-870959edffe8-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-af8244-55d6nfg5v\" (UID: \"31e81deb-0e7e-4c26-b43f-870959edffe8\") " pod="ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-55d6nfg5v" Nov 03 01:10:51.446085 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:51.446063 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/31e81deb-0e7e-4c26-b43f-870959edffe8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-af8244-55d6nfg5v\" (UID: \"31e81deb-0e7e-4c26-b43f-870959edffe8\") " pod="ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-55d6nfg5v" Nov 03 01:10:51.446313 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:51.446264 4421 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b55c151-d235-4bcc-b602-eca0da4a7348-istio-podinfo\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:10:51.446313 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:51.446309 4421 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b55c151-d235-4bcc-b602-eca0da4a7348-istio-data\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:10:51.446422 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:51.446324 4421 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b55c151-d235-4bcc-b602-eca0da4a7348-workload-certs\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:10:51.446422 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:51.446333 4421 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b55c151-d235-4bcc-b602-eca0da4a7348-istio-token\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:10:51.446422 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:51.446342 4421 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b55c151-d235-4bcc-b602-eca0da4a7348-istiod-ca-cert\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:10:51.446422 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:51.446350 4421 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b55c151-d235-4bcc-b602-eca0da4a7348-istio-envoy\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:10:51.446422 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:51.446359 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bb9cl\" (UniqueName: \"kubernetes.io/projected/2b55c151-d235-4bcc-b602-eca0da4a7348-kube-api-access-bb9cl\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:10:51.446422 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:51.446369 4421 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b55c151-d235-4bcc-b602-eca0da4a7348-workload-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:10:51.558874 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:51.547648 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31e81deb-0e7e-4c26-b43f-870959edffe8-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-af8244-55d6nfg5v\" (UID: \"31e81deb-0e7e-4c26-b43f-870959edffe8\") " pod="ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-55d6nfg5v" Nov 03 01:10:51.558874 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:51.548373 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/31e81deb-0e7e-4c26-b43f-870959edffe8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-af8244-55d6nfg5v\" (UID: \"31e81deb-0e7e-4c26-b43f-870959edffe8\") " pod="ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-55d6nfg5v" Nov 03 01:10:51.558874 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:51.548500 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsmnf\" (UniqueName: \"kubernetes.io/projected/31e81deb-0e7e-4c26-b43f-870959edffe8-kube-api-access-gsmnf\") pod \"payload-processing-external-model-e2e-extmod-b-af8244-55d6nfg5v\" (UID: \"31e81deb-0e7e-4c26-b43f-870959edffe8\") " pod="ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-55d6nfg5v" Nov 03 01:10:51.558874 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:51.548642 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/31e81deb-0e7e-4c26-b43f-870959edffe8-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-af8244-55d6nfg5v\" (UID: \"31e81deb-0e7e-4c26-b43f-870959edffe8\") " pod="ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-55d6nfg5v" Nov 03 01:10:51.558874 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:51.548680 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31e81deb-0e7e-4c26-b43f-870959edffe8-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-af8244-55d6nfg5v\" (UID: \"31e81deb-0e7e-4c26-b43f-870959edffe8\") " pod="ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-55d6nfg5v" Nov 03 01:10:51.558874 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:51.549521 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/31e81deb-0e7e-4c26-b43f-870959edffe8-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-af8244-55d6nfg5v\" (UID: \"31e81deb-0e7e-4c26-b43f-870959edffe8\") " pod="ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-55d6nfg5v" Nov 03 01:10:51.558874 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:51.551370 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/31e81deb-0e7e-4c26-b43f-870959edffe8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-af8244-55d6nfg5v\" (UID: \"31e81deb-0e7e-4c26-b43f-870959edffe8\") " pod="ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-55d6nfg5v" Nov 03 01:10:51.609373 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:51.601754 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsmnf\" (UniqueName: \"kubernetes.io/projected/31e81deb-0e7e-4c26-b43f-870959edffe8-kube-api-access-gsmnf\") pod \"payload-processing-external-model-e2e-extmod-b-af8244-55d6nfg5v\" (UID: \"31e81deb-0e7e-4c26-b43f-870959edffe8\") " pod="ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-55d6nfg5v" Nov 03 01:10:51.762298 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:51.762047 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-55d6nfg5v" Nov 03 01:10:52.196057 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:10:52.196023 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31e81deb_0e7e_4c26_b43f_870959edffe8.slice/crio-c52238e5ac4a870c2f79cf0b2426bb80deb3c5c261cd44769cabd4b220caab76 WatchSource:0}: Error finding container c52238e5ac4a870c2f79cf0b2426bb80deb3c5c261cd44769cabd4b220caab76: Status 404 returned error can't find the container with id c52238e5ac4a870c2f79cf0b2426bb80deb3c5c261cd44769cabd4b220caab76 Nov 03 01:10:52.196352 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:52.196326 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-55d6nfg5v"] Nov 03 01:10:52.268803 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:52.268774 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-t5bt4" Nov 03 01:10:52.268803 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:52.268783 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-t5bt4" event={"ID":"2b55c151-d235-4bcc-b602-eca0da4a7348","Type":"ContainerDied","Data":"63b97dfc18d3a7aaa7edfda6c2ec84004adcafecad96f2c1069981bde452f7ce"} Nov 03 01:10:52.269371 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:52.268878 4421 scope.go:122] "RemoveContainer" containerID="28c9237ffae3781146e130dd9c4cd0da86a188f0798d40ea2f29e94c02af4605" Nov 03 01:10:52.271347 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:52.271203 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-55d6nfg5v" event={"ID":"31e81deb-0e7e-4c26-b43f-870959edffe8","Type":"ContainerStarted","Data":"c52238e5ac4a870c2f79cf0b2426bb80deb3c5c261cd44769cabd4b220caab76"} Nov 03 01:10:52.291764 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:52.291735 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-t5bt4"] Nov 03 01:10:52.297371 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:52.297345 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-t5bt4"] Nov 03 01:10:53.299902 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:53.299765 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-55d6nfg5v" event={"ID":"31e81deb-0e7e-4c26-b43f-870959edffe8","Type":"ContainerStarted","Data":"f9336e3115a14139ea618b94a58991993df29dd2e4466eef0d9432a13b292a20"} Nov 03 01:10:53.299902 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:53.299862 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-55d6nfg5v" Nov 03 01:10:53.326579 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:53.320379 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-55d6nfg5v" podStartSLOduration=2.320323828 podStartE2EDuration="2.320323828s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:10:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:10:53.31852724 +0000 UTC m=+1795.489530344" watchObservedRunningTime="2026-11-03 01:10:53.320323828 +0000 UTC m=+1795.491326924" Nov 03 01:10:53.665412 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:53.665384 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-a23ed0-openshift-default-84685cd7b7-6vgvw" Nov 03 01:10:53.760062 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:53.760018 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/065c5377-f408-4f5c-be1c-19e4cea9e64e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/065c5377-f408-4f5c-be1c-19e4cea9e64e-workload-socket\") pod \"065c5377-f408-4f5c-be1c-19e4cea9e64e\" (UID: \"065c5377-f408-4f5c-be1c-19e4cea9e64e\") " Nov 03 01:10:53.760251 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:53.760107 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/065c5377-f408-4f5c-be1c-19e4cea9e64e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/065c5377-f408-4f5c-be1c-19e4cea9e64e-istio-data\") pod \"065c5377-f408-4f5c-be1c-19e4cea9e64e\" (UID: \"065c5377-f408-4f5c-be1c-19e4cea9e64e\") " Nov 03 01:10:53.760251 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:53.760143 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/065c5377-f408-4f5c-be1c-19e4cea9e64e-istio-token\" (UniqueName: \"kubernetes.io/projected/065c5377-f408-4f5c-be1c-19e4cea9e64e-istio-token\") pod \"065c5377-f408-4f5c-be1c-19e4cea9e64e\" (UID: \"065c5377-f408-4f5c-be1c-19e4cea9e64e\") " Nov 03 01:10:53.760422 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:53.760262 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/065c5377-f408-4f5c-be1c-19e4cea9e64e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/065c5377-f408-4f5c-be1c-19e4cea9e64e-workload-certs\") pod \"065c5377-f408-4f5c-be1c-19e4cea9e64e\" (UID: \"065c5377-f408-4f5c-be1c-19e4cea9e64e\") " Nov 03 01:10:53.760422 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:53.760358 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/065c5377-f408-4f5c-be1c-19e4cea9e64e-workload-socket" pod "065c5377-f408-4f5c-be1c-19e4cea9e64e" (UID: "065c5377-f408-4f5c-be1c-19e4cea9e64e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:10:53.760422 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:53.760414 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/065c5377-f408-4f5c-be1c-19e4cea9e64e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/065c5377-f408-4f5c-be1c-19e4cea9e64e-istio-envoy\") pod \"065c5377-f408-4f5c-be1c-19e4cea9e64e\" (UID: \"065c5377-f408-4f5c-be1c-19e4cea9e64e\") " Nov 03 01:10:53.760600 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:53.760465 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/065c5377-f408-4f5c-be1c-19e4cea9e64e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/065c5377-f408-4f5c-be1c-19e4cea9e64e-istiod-ca-cert\") pod \"065c5377-f408-4f5c-be1c-19e4cea9e64e\" (UID: \"065c5377-f408-4f5c-be1c-19e4cea9e64e\") " Nov 03 01:10:53.760600 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:53.760526 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/065c5377-f408-4f5c-be1c-19e4cea9e64e-kube-api-access-v8qv9\" (UniqueName: \"kubernetes.io/projected/065c5377-f408-4f5c-be1c-19e4cea9e64e-kube-api-access-v8qv9\") pod \"065c5377-f408-4f5c-be1c-19e4cea9e64e\" (UID: \"065c5377-f408-4f5c-be1c-19e4cea9e64e\") " Nov 03 01:10:53.760600 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:53.760520 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/065c5377-f408-4f5c-be1c-19e4cea9e64e-istio-data" pod "065c5377-f408-4f5c-be1c-19e4cea9e64e" (UID: "065c5377-f408-4f5c-be1c-19e4cea9e64e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:10:53.760600 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:53.760569 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/065c5377-f408-4f5c-be1c-19e4cea9e64e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/065c5377-f408-4f5c-be1c-19e4cea9e64e-credential-socket\") pod \"065c5377-f408-4f5c-be1c-19e4cea9e64e\" (UID: \"065c5377-f408-4f5c-be1c-19e4cea9e64e\") " Nov 03 01:10:53.760966 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:53.760943 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/065c5377-f408-4f5c-be1c-19e4cea9e64e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/065c5377-f408-4f5c-be1c-19e4cea9e64e-istio-podinfo\") pod \"065c5377-f408-4f5c-be1c-19e4cea9e64e\" (UID: \"065c5377-f408-4f5c-be1c-19e4cea9e64e\") " Nov 03 01:10:53.761259 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:53.760962 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/065c5377-f408-4f5c-be1c-19e4cea9e64e-istiod-ca-cert" pod "065c5377-f408-4f5c-be1c-19e4cea9e64e" (UID: "065c5377-f408-4f5c-be1c-19e4cea9e64e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:10:53.761259 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:53.760967 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/065c5377-f408-4f5c-be1c-19e4cea9e64e-credential-socket" pod "065c5377-f408-4f5c-be1c-19e4cea9e64e" (UID: "065c5377-f408-4f5c-be1c-19e4cea9e64e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:10:53.761259 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:53.760665 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/065c5377-f408-4f5c-be1c-19e4cea9e64e-workload-certs" pod "065c5377-f408-4f5c-be1c-19e4cea9e64e" (UID: "065c5377-f408-4f5c-be1c-19e4cea9e64e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:10:53.762781 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:53.762758 4421 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/065c5377-f408-4f5c-be1c-19e4cea9e64e-credential-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:10:53.762947 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:53.762921 4421 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/065c5377-f408-4f5c-be1c-19e4cea9e64e-workload-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:10:53.762947 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:53.762948 4421 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/065c5377-f408-4f5c-be1c-19e4cea9e64e-istio-data\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:10:53.763122 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:53.762963 4421 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/065c5377-f408-4f5c-be1c-19e4cea9e64e-workload-certs\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:10:53.763122 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:53.762978 4421 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/065c5377-f408-4f5c-be1c-19e4cea9e64e-istiod-ca-cert\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:10:53.763478 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:53.763451 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/065c5377-f408-4f5c-be1c-19e4cea9e64e-istio-podinfo" pod "065c5377-f408-4f5c-be1c-19e4cea9e64e" (UID: "065c5377-f408-4f5c-be1c-19e4cea9e64e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:10:53.763478 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:53.763465 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/065c5377-f408-4f5c-be1c-19e4cea9e64e-istio-envoy" pod "065c5377-f408-4f5c-be1c-19e4cea9e64e" (UID: "065c5377-f408-4f5c-be1c-19e4cea9e64e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:10:53.763774 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:53.763747 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/065c5377-f408-4f5c-be1c-19e4cea9e64e-kube-api-access-v8qv9" pod "065c5377-f408-4f5c-be1c-19e4cea9e64e" (UID: "065c5377-f408-4f5c-be1c-19e4cea9e64e"). InnerVolumeSpecName "kube-api-access-v8qv9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:10:53.764142 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:53.764123 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/065c5377-f408-4f5c-be1c-19e4cea9e64e-istio-token" pod "065c5377-f408-4f5c-be1c-19e4cea9e64e" (UID: "065c5377-f408-4f5c-be1c-19e4cea9e64e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:10:53.864535 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:53.864502 4421 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/065c5377-f408-4f5c-be1c-19e4cea9e64e-istio-envoy\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:10:53.864535 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:53.864533 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8qv9\" (UniqueName: \"kubernetes.io/projected/065c5377-f408-4f5c-be1c-19e4cea9e64e-kube-api-access-v8qv9\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:10:53.864535 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:53.864544 4421 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/065c5377-f408-4f5c-be1c-19e4cea9e64e-istio-podinfo\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:10:53.864912 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:53.864555 4421 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/065c5377-f408-4f5c-be1c-19e4cea9e64e-istio-token\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:10:53.982557 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:53.982450 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b55c151-d235-4bcc-b602-eca0da4a7348" path="/var/lib/kubelet/pods/2b55c151-d235-4bcc-b602-eca0da4a7348/volumes" Nov 03 01:10:54.389966 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:54.389929 4421 generic.go:356] "Generic (PLEG): container finished" podID="065c5377-f408-4f5c-be1c-19e4cea9e64e" containerID="9202bd26541cdff38cf8d19d0e98ac483f3ae57b4238e4611a2d9af68350a4b9" exitCode=0 Nov 03 01:10:54.391733 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:54.391711 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-a23ed0-openshift-default-84685cd7b7-6vgvw" Nov 03 01:10:54.394911 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:54.394876 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-a23ed0-openshift-default-84685cd7b7-6vgvw" event={"ID":"065c5377-f408-4f5c-be1c-19e4cea9e64e","Type":"ContainerDied","Data":"9202bd26541cdff38cf8d19d0e98ac483f3ae57b4238e4611a2d9af68350a4b9"} Nov 03 01:10:54.395044 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:54.394945 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-a23ed0-openshift-default-84685cd7b7-6vgvw" event={"ID":"065c5377-f408-4f5c-be1c-19e4cea9e64e","Type":"ContainerDied","Data":"9a2cd0efd5c522d44f9b416cd120a6c4b12eee74195e645d92b89cce4c39fb0f"} Nov 03 01:10:54.395044 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:54.394970 4421 scope.go:122] "RemoveContainer" containerID="9202bd26541cdff38cf8d19d0e98ac483f3ae57b4238e4611a2d9af68350a4b9" Nov 03 01:10:54.416184 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:54.416150 4421 scope.go:122] "RemoveContainer" containerID="9202bd26541cdff38cf8d19d0e98ac483f3ae57b4238e4611a2d9af68350a4b9" Nov 03 01:10:54.416648 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:10:54.416615 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9202bd26541cdff38cf8d19d0e98ac483f3ae57b4238e4611a2d9af68350a4b9\": container with ID starting with 9202bd26541cdff38cf8d19d0e98ac483f3ae57b4238e4611a2d9af68350a4b9 not found: ID does not exist" containerID="9202bd26541cdff38cf8d19d0e98ac483f3ae57b4238e4611a2d9af68350a4b9" Nov 03 01:10:54.416783 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:54.416662 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9202bd26541cdff38cf8d19d0e98ac483f3ae57b4238e4611a2d9af68350a4b9"} err="failed to get container status \"9202bd26541cdff38cf8d19d0e98ac483f3ae57b4238e4611a2d9af68350a4b9\": rpc error: code = NotFound desc = could not find container \"9202bd26541cdff38cf8d19d0e98ac483f3ae57b4238e4611a2d9af68350a4b9\": container with ID starting with 9202bd26541cdff38cf8d19d0e98ac483f3ae57b4238e4611a2d9af68350a4b9 not found: ID does not exist" Nov 03 01:10:54.418942 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:54.418917 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-a23ed0-openshift-default-84685cd7b7-6vgvw"] Nov 03 01:10:54.424262 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:54.424234 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-a23ed0-openshift-default-84685cd7b7-6vgvw"] Nov 03 01:10:55.981907 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:55.981860 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="065c5377-f408-4f5c-be1c-19e4cea9e64e" path="/var/lib/kubelet/pods/065c5377-f408-4f5c-be1c-19e4cea9e64e/volumes" Nov 03 01:10:56.531554 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:56.531517 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9899cc30-t78rv"] Nov 03 01:10:56.532356 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:56.532339 4421 cpu_manager.go:414] "containerMap: removing container" podUID="065c5377-f408-4f5c-be1c-19e4cea9e64e" containerName="istio-proxy" Nov 03 01:10:56.532422 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:56.532357 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="065c5377-f408-4f5c-be1c-19e4cea9e64e" containerName="istio-proxy" Nov 03 01:10:56.532525 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:56.532516 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="065c5377-f408-4f5c-be1c-19e4cea9e64e" containerName="istio-proxy" Nov 03 01:10:56.537933 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:56.537916 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9899cc30-t78rv" Nov 03 01:10:56.541737 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:56.541701 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 01:10:56.542819 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:56.542794 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 01:10:56.542957 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:56.542846 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:10:56.542957 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:56.542928 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6q4xp\"" Nov 03 01:10:56.545077 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:56.545051 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9899cc30-t78rv"] Nov 03 01:10:56.623626 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:56.623590 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdwfh\" (UniqueName: \"kubernetes.io/projected/5fefa28b-08b0-4141-9f8a-8c411ac1990f-kube-api-access-qdwfh\") pod \"maas-api-revoke-sub-9899cc30-t78rv\" (UID: \"5fefa28b-08b0-4141-9f8a-8c411ac1990f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9899cc30-t78rv" Nov 03 01:10:56.623814 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:56.623721 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fefa28b-08b0-4141-9f8a-8c411ac1990f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9899cc30-t78rv\" (UID: \"5fefa28b-08b0-4141-9f8a-8c411ac1990f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9899cc30-t78rv" Nov 03 01:10:56.623907 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:56.623893 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fefa28b-08b0-4141-9f8a-8c411ac1990f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9899cc30-t78rv\" (UID: \"5fefa28b-08b0-4141-9f8a-8c411ac1990f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9899cc30-t78rv" Nov 03 01:10:56.725727 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:56.725691 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fefa28b-08b0-4141-9f8a-8c411ac1990f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9899cc30-t78rv\" (UID: \"5fefa28b-08b0-4141-9f8a-8c411ac1990f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9899cc30-t78rv" Nov 03 01:10:56.725943 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:56.725927 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdwfh\" (UniqueName: \"kubernetes.io/projected/5fefa28b-08b0-4141-9f8a-8c411ac1990f-kube-api-access-qdwfh\") pod \"maas-api-revoke-sub-9899cc30-t78rv\" (UID: \"5fefa28b-08b0-4141-9f8a-8c411ac1990f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9899cc30-t78rv" Nov 03 01:10:56.726020 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:56.725993 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fefa28b-08b0-4141-9f8a-8c411ac1990f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9899cc30-t78rv\" (UID: \"5fefa28b-08b0-4141-9f8a-8c411ac1990f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9899cc30-t78rv" Nov 03 01:10:56.726415 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:56.726393 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fefa28b-08b0-4141-9f8a-8c411ac1990f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9899cc30-t78rv\" (UID: \"5fefa28b-08b0-4141-9f8a-8c411ac1990f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9899cc30-t78rv" Nov 03 01:10:56.728501 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:56.728481 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fefa28b-08b0-4141-9f8a-8c411ac1990f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9899cc30-t78rv\" (UID: \"5fefa28b-08b0-4141-9f8a-8c411ac1990f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9899cc30-t78rv" Nov 03 01:10:56.734254 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:56.734231 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdwfh\" (UniqueName: \"kubernetes.io/projected/5fefa28b-08b0-4141-9f8a-8c411ac1990f-kube-api-access-qdwfh\") pod \"maas-api-revoke-sub-9899cc30-t78rv\" (UID: \"5fefa28b-08b0-4141-9f8a-8c411ac1990f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9899cc30-t78rv" Nov 03 01:10:56.864475 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:56.861603 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9899cc30-t78rv" Nov 03 01:10:57.199780 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:10:57.199748 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fefa28b_08b0_4141_9f8a_8c411ac1990f.slice/crio-440d7f555938758f6493670ba6ac9979b061d0430364f81674d75dceb0d37e81 WatchSource:0}: Error finding container 440d7f555938758f6493670ba6ac9979b061d0430364f81674d75dceb0d37e81: Status 404 returned error can't find the container with id 440d7f555938758f6493670ba6ac9979b061d0430364f81674d75dceb0d37e81 Nov 03 01:10:57.200117 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:57.200010 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9899cc30-t78rv"] Nov 03 01:10:57.456803 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:57.456710 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9899cc30-t78rv" event={"ID":"5fefa28b-08b0-4141-9f8a-8c411ac1990f","Type":"ContainerStarted","Data":"440d7f555938758f6493670ba6ac9979b061d0430364f81674d75dceb0d37e81"} Nov 03 01:10:58.468643 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:58.468604 4421 generic.go:356] "Generic (PLEG): container finished" podID="5fefa28b-08b0-4141-9f8a-8c411ac1990f" containerID="3c2f36f081b4a7265cfe781151e9c74decae9e970c3025789b5e68c257c7cb52" exitCode=0 Nov 03 01:10:58.469098 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:58.468661 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9899cc30-t78rv" event={"ID":"5fefa28b-08b0-4141-9f8a-8c411ac1990f","Type":"ContainerDied","Data":"3c2f36f081b4a7265cfe781151e9c74decae9e970c3025789b5e68c257c7cb52"} Nov 03 01:10:59.873174 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:59.873145 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9899cc30-t78rv" Nov 03 01:10:59.965406 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:59.965371 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fefa28b-08b0-4141-9f8a-8c411ac1990f-kube-api-access-qdwfh\" (UniqueName: \"kubernetes.io/projected/5fefa28b-08b0-4141-9f8a-8c411ac1990f-kube-api-access-qdwfh\") pod \"5fefa28b-08b0-4141-9f8a-8c411ac1990f\" (UID: \"5fefa28b-08b0-4141-9f8a-8c411ac1990f\") " Nov 03 01:10:59.965630 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:59.965537 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5fefa28b-08b0-4141-9f8a-8c411ac1990f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fefa28b-08b0-4141-9f8a-8c411ac1990f-maas-api-openshift-ca\") pod \"5fefa28b-08b0-4141-9f8a-8c411ac1990f\" (UID: \"5fefa28b-08b0-4141-9f8a-8c411ac1990f\") " Nov 03 01:10:59.965720 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:59.965672 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5fefa28b-08b0-4141-9f8a-8c411ac1990f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fefa28b-08b0-4141-9f8a-8c411ac1990f-maas-api-xks-ca\") pod \"5fefa28b-08b0-4141-9f8a-8c411ac1990f\" (UID: \"5fefa28b-08b0-4141-9f8a-8c411ac1990f\") " Nov 03 01:10:59.966745 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:59.966713 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fefa28b-08b0-4141-9f8a-8c411ac1990f-maas-api-openshift-ca" pod "5fefa28b-08b0-4141-9f8a-8c411ac1990f" (UID: "5fefa28b-08b0-4141-9f8a-8c411ac1990f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:10:59.968737 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:59.968074 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fefa28b-08b0-4141-9f8a-8c411ac1990f-kube-api-access-qdwfh" pod "5fefa28b-08b0-4141-9f8a-8c411ac1990f" (UID: "5fefa28b-08b0-4141-9f8a-8c411ac1990f"). InnerVolumeSpecName "kube-api-access-qdwfh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:10:59.988386 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:10:59.988354 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5fefa28b-08b0-4141-9f8a-8c411ac1990f-maas-api-xks-ca" pod "5fefa28b-08b0-4141-9f8a-8c411ac1990f" (UID: "5fefa28b-08b0-4141-9f8a-8c411ac1990f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:11:00.068669 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:11:00.068637 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fefa28b-08b0-4141-9f8a-8c411ac1990f-maas-api-xks-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:11:00.068669 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:11:00.068666 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdwfh\" (UniqueName: \"kubernetes.io/projected/5fefa28b-08b0-4141-9f8a-8c411ac1990f-kube-api-access-qdwfh\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:11:00.068669 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:11:00.068677 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fefa28b-08b0-4141-9f8a-8c411ac1990f-maas-api-openshift-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:11:00.578751 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:11:00.578711 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9899cc30-t78rv" event={"ID":"5fefa28b-08b0-4141-9f8a-8c411ac1990f","Type":"ContainerDied","Data":"440d7f555938758f6493670ba6ac9979b061d0430364f81674d75dceb0d37e81"} Nov 03 01:11:00.578938 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:11:00.578777 4421 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="440d7f555938758f6493670ba6ac9979b061d0430364f81674d75dceb0d37e81" Nov 03 01:11:00.578938 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:11:00.578846 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9899cc30-t78rv" Nov 03 01:11:01.081564 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:11:01.081533 4421 scope.go:122] "RemoveContainer" containerID="1f80ee5dda79c13cca4b34aef98dfd231fa34358bd15e8034dfd87f915d152ca" Nov 03 01:11:01.093826 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:11:01.093806 4421 scope.go:122] "RemoveContainer" containerID="6ed0341f3cd8b54982948ca9648afdb1c68aab15a02b6fbbe2a35cfa0a0a23b5" Nov 03 01:11:01.105817 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:11:01.105792 4421 scope.go:122] "RemoveContainer" containerID="5a900ba3e20c0dd7ad4283e577eac7f3a750d775012fb27cb02f11d746bf83d0" Nov 03 01:11:01.117727 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:11:01.117698 4421 scope.go:122] "RemoveContainer" containerID="3c9a423bdb5d8887e7ad9e46d5be762a8a11de2cebd9620a1f4e5b3c7076291d" Nov 03 01:11:01.130423 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:11:01.130405 4421 scope.go:122] "RemoveContainer" containerID="d776777727ce1f660b9f6772b72313f206ac51e3d494fa3234e07f02318a1463" Nov 03 01:11:01.141425 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:11:01.141397 4421 scope.go:122] "RemoveContainer" containerID="d44aabc392f84beba7ee975c0d56a4b0321d698f695880cdeb9d9d01eb133d67" Nov 03 01:11:01.151249 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:11:01.151233 4421 scope.go:122] "RemoveContainer" containerID="18068f51d0b3f50cab57a90bf91c5aac8331e19017464b56be3536ddcfec62cf" Nov 03 01:11:01.161719 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:11:01.161703 4421 scope.go:122] "RemoveContainer" containerID="cc758e54f3e51c9d05b8ed7ec4dbb39fa8f4ee499be71add295e0cb4706cab9d" Nov 03 01:11:01.171978 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:11:01.171958 4421 scope.go:122] "RemoveContainer" containerID="82e2680ea46199a1dc1751a9f8eb3f983b8297632bf5a133923e9474e90f101f" Nov 03 01:11:01.182318 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:11:01.182260 4421 scope.go:122] "RemoveContainer" containerID="2876ca3c7ee93ffe453542b4021df38c53516ad611837f6d53c3b99e90a2eb6b" Nov 03 01:11:01.192794 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:11:01.192773 4421 scope.go:122] "RemoveContainer" containerID="b86453dd86bfec9ca29cf35a4487ea0cbc414cca36e81aedc56a0bc1a27bf079" Nov 03 01:11:01.203337 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:11:01.203316 4421 scope.go:122] "RemoveContainer" containerID="08328ecc38835b3ce804edab79c0bc73f422eab44865c2b2408bf4112800c917" Nov 03 01:11:01.216222 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:11:01.216201 4421 scope.go:122] "RemoveContainer" containerID="0a1a9c233f7ed8f9c8448eb3d0fedbb4aea5b3c9891f201dc86e35fc7fd50be5" Nov 03 01:11:04.401722 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:11:04.401684 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-55d6nfg5v" Nov 03 01:11:14.707812 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:11:14.707762 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 01:11:20.943389 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:11:20.943349 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-d25759-857574b47-bpmv4"] Nov 03 01:11:20.944779 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:11:20.944756 4421 cpu_manager.go:414] "containerMap: removing container" podUID="5fefa28b-08b0-4141-9f8a-8c411ac1990f" containerName="revoke-keys" Nov 03 01:11:20.944856 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:11:20.944785 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fefa28b-08b0-4141-9f8a-8c411ac1990f" containerName="revoke-keys" Nov 03 01:11:20.944985 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:11:20.944971 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fefa28b-08b0-4141-9f8a-8c411ac1990f" containerName="revoke-keys" Nov 03 01:11:20.949678 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:11:20.949656 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d25759-857574b47-bpmv4" Nov 03 01:11:20.957690 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:11:20.957660 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-d25759-dockercfg-s57cs\"" Nov 03 01:11:20.967790 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:11:20.967763 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-d25759-857574b47-bpmv4"] Nov 03 01:11:21.009160 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:11:21.009116 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6g8z9\" (UniqueName: \"kubernetes.io/projected/d5576371-83ce-4e3e-b021-f0de09a49d6e-kube-api-access-6g8z9\") pod \"payload-pre-processing-e2e-extmod-a-d25759-857574b47-bpmv4\" (UID: \"d5576371-83ce-4e3e-b021-f0de09a49d6e\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d25759-857574b47-bpmv4" Nov 03 01:11:21.009599 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:11:21.009572 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5576371-83ce-4e3e-b021-f0de09a49d6e-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-d25759-857574b47-bpmv4\" (UID: \"d5576371-83ce-4e3e-b021-f0de09a49d6e\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d25759-857574b47-bpmv4" Nov 03 01:11:21.053299 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:11:21.051299 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-d25759\"" Nov 03 01:11:21.162302 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:11:21.135582 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5576371-83ce-4e3e-b021-f0de09a49d6e-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-d25759-857574b47-bpmv4\" (UID: \"d5576371-83ce-4e3e-b021-f0de09a49d6e\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d25759-857574b47-bpmv4" Nov 03 01:11:21.162302 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:11:21.135698 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6g8z9\" (UniqueName: \"kubernetes.io/projected/d5576371-83ce-4e3e-b021-f0de09a49d6e-kube-api-access-6g8z9\") pod \"payload-pre-processing-e2e-extmod-a-d25759-857574b47-bpmv4\" (UID: \"d5576371-83ce-4e3e-b021-f0de09a49d6e\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d25759-857574b47-bpmv4" Nov 03 01:11:21.162302 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:11:21.147644 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5576371-83ce-4e3e-b021-f0de09a49d6e-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-d25759-857574b47-bpmv4\" (UID: \"d5576371-83ce-4e3e-b021-f0de09a49d6e\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d25759-857574b47-bpmv4" Nov 03 01:11:21.194350 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:11:21.179068 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6g8z9\" (UniqueName: \"kubernetes.io/projected/d5576371-83ce-4e3e-b021-f0de09a49d6e-kube-api-access-6g8z9\") pod \"payload-pre-processing-e2e-extmod-a-d25759-857574b47-bpmv4\" (UID: \"d5576371-83ce-4e3e-b021-f0de09a49d6e\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d25759-857574b47-bpmv4" Nov 03 01:11:21.287299 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:11:21.281048 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d25759-857574b47-bpmv4" Nov 03 01:11:21.918499 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:11:21.918466 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5576371_83ce_4e3e_b021_f0de09a49d6e.slice/crio-7be4139f8a63280d178cd0ade1485046ab552e0c907f32cf192f65243b723054 WatchSource:0}: Error finding container 7be4139f8a63280d178cd0ade1485046ab552e0c907f32cf192f65243b723054: Status 404 returned error can't find the container with id 7be4139f8a63280d178cd0ade1485046ab552e0c907f32cf192f65243b723054 Nov 03 01:11:21.919226 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:11:21.919199 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-d25759-857574b47-bpmv4"] Nov 03 01:11:22.180477 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:11:22.180368 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d25759-857574b47-bpmv4" event={"ID":"d5576371-83ce-4e3e-b021-f0de09a49d6e","Type":"ContainerStarted","Data":"28f81a1d05bfcf330a3f478d3b32e252f1a8ecc18e619bef8e09c3c8c3c7e3dd"} Nov 03 01:11:22.180477 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:11:22.180434 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d25759-857574b47-bpmv4" event={"ID":"d5576371-83ce-4e3e-b021-f0de09a49d6e","Type":"ContainerStarted","Data":"7be4139f8a63280d178cd0ade1485046ab552e0c907f32cf192f65243b723054"} Nov 03 01:11:22.180477 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:11:22.180463 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d25759-857574b47-bpmv4" Nov 03 01:11:22.199563 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:11:22.199515 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d25759-857574b47-bpmv4" podStartSLOduration=2.199500815 podStartE2EDuration="2.199500815s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:11:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:11:22.195436938 +0000 UTC m=+1824.366440035" watchObservedRunningTime="2026-11-03 01:11:22.199500815 +0000 UTC m=+1824.370504002" Nov 03 01:11:30.620296 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:11:30.620196 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-2dshc"] Nov 03 01:11:30.627941 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:11:30.627812 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-2dshc" Nov 03 01:11:30.641352 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:11:30.638359 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-2dshc"] Nov 03 01:11:30.706607 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:11:30.706562 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eeca2c25-c047-4693-8c4d-def02cfabfa8-workload-certs\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-2dshc\" (UID: \"eeca2c25-c047-4693-8c4d-def02cfabfa8\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-2dshc" Nov 03 01:11:30.706804 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:11:30.706622 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eeca2c25-c047-4693-8c4d-def02cfabfa8-istio-podinfo\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-2dshc\" (UID: \"eeca2c25-c047-4693-8c4d-def02cfabfa8\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-2dshc" Nov 03 01:11:30.706876 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:11:30.706803 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eeca2c25-c047-4693-8c4d-def02cfabfa8-workload-socket\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-2dshc\" (UID: \"eeca2c25-c047-4693-8c4d-def02cfabfa8\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-2dshc" Nov 03 01:11:30.706941 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:11:30.706887 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eeca2c25-c047-4693-8c4d-def02cfabfa8-istio-data\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-2dshc\" (UID: \"eeca2c25-c047-4693-8c4d-def02cfabfa8\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-2dshc" Nov 03 01:11:30.707041 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:11:30.707025 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eeca2c25-c047-4693-8c4d-def02cfabfa8-istio-envoy\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-2dshc\" (UID: \"eeca2c25-c047-4693-8c4d-def02cfabfa8\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-2dshc" Nov 03 01:11:30.707176 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:11:30.707150 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rzn5\" (UniqueName: \"kubernetes.io/projected/eeca2c25-c047-4693-8c4d-def02cfabfa8-kube-api-access-4rzn5\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-2dshc\" (UID: \"eeca2c25-c047-4693-8c4d-def02cfabfa8\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-2dshc" Nov 03 01:11:30.707343 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:11:30.707320 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eeca2c25-c047-4693-8c4d-def02cfabfa8-istio-token\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-2dshc\" (UID: \"eeca2c25-c047-4693-8c4d-def02cfabfa8\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-2dshc" Nov 03 01:11:30.707407 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:11:30.707366 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eeca2c25-c047-4693-8c4d-def02cfabfa8-istiod-ca-cert\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-2dshc\" (UID: \"eeca2c25-c047-4693-8c4d-def02cfabfa8\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-2dshc" Nov 03 01:11:30.707500 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:11:30.707484 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eeca2c25-c047-4693-8c4d-def02cfabfa8-credential-socket\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-2dshc\" (UID: \"eeca2c25-c047-4693-8c4d-def02cfabfa8\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-2dshc" Nov 03 01:11:30.808618 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:11:30.808583 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eeca2c25-c047-4693-8c4d-def02cfabfa8-workload-socket\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-2dshc\" (UID: \"eeca2c25-c047-4693-8c4d-def02cfabfa8\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-2dshc" Nov 03 01:11:30.808800 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:11:30.808643 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eeca2c25-c047-4693-8c4d-def02cfabfa8-istio-data\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-2dshc\" (UID: \"eeca2c25-c047-4693-8c4d-def02cfabfa8\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-2dshc" Nov 03 01:11:30.808800 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:11:30.808741 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eeca2c25-c047-4693-8c4d-def02cfabfa8-istio-envoy\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-2dshc\" (UID: \"eeca2c25-c047-4693-8c4d-def02cfabfa8\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-2dshc" Nov 03 01:11:30.808900 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:11:30.808826 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rzn5\" (UniqueName: \"kubernetes.io/projected/eeca2c25-c047-4693-8c4d-def02cfabfa8-kube-api-access-4rzn5\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-2dshc\" (UID: \"eeca2c25-c047-4693-8c4d-def02cfabfa8\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-2dshc" Nov 03 01:11:30.808945 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:11:30.808897 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eeca2c25-c047-4693-8c4d-def02cfabfa8-istio-token\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-2dshc\" (UID: \"eeca2c25-c047-4693-8c4d-def02cfabfa8\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-2dshc" Nov 03 01:11:30.808945 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:11:30.808927 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eeca2c25-c047-4693-8c4d-def02cfabfa8-istiod-ca-cert\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-2dshc\" (UID: \"eeca2c25-c047-4693-8c4d-def02cfabfa8\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-2dshc" Nov 03 01:11:30.809030 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:11:30.809010 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eeca2c25-c047-4693-8c4d-def02cfabfa8-credential-socket\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-2dshc\" (UID: \"eeca2c25-c047-4693-8c4d-def02cfabfa8\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-2dshc" Nov 03 01:11:30.809074 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:11:30.809050 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eeca2c25-c047-4693-8c4d-def02cfabfa8-workload-socket\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-2dshc\" (UID: \"eeca2c25-c047-4693-8c4d-def02cfabfa8\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-2dshc" Nov 03 01:11:30.809129 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:11:30.809094 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eeca2c25-c047-4693-8c4d-def02cfabfa8-istio-data\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-2dshc\" (UID: \"eeca2c25-c047-4693-8c4d-def02cfabfa8\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-2dshc" Nov 03 01:11:30.809332 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:11:30.809294 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eeca2c25-c047-4693-8c4d-def02cfabfa8-workload-certs\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-2dshc\" (UID: \"eeca2c25-c047-4693-8c4d-def02cfabfa8\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-2dshc" Nov 03 01:11:30.809492 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:11:30.809339 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eeca2c25-c047-4693-8c4d-def02cfabfa8-istio-podinfo\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-2dshc\" (UID: \"eeca2c25-c047-4693-8c4d-def02cfabfa8\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-2dshc" Nov 03 01:11:30.809492 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:11:30.809340 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eeca2c25-c047-4693-8c4d-def02cfabfa8-credential-socket\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-2dshc\" (UID: \"eeca2c25-c047-4693-8c4d-def02cfabfa8\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-2dshc" Nov 03 01:11:30.809633 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:11:30.809599 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eeca2c25-c047-4693-8c4d-def02cfabfa8-workload-certs\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-2dshc\" (UID: \"eeca2c25-c047-4693-8c4d-def02cfabfa8\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-2dshc" Nov 03 01:11:30.809932 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:11:30.809910 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eeca2c25-c047-4693-8c4d-def02cfabfa8-istiod-ca-cert\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-2dshc\" (UID: \"eeca2c25-c047-4693-8c4d-def02cfabfa8\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-2dshc" Nov 03 01:11:30.811254 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:11:30.811229 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eeca2c25-c047-4693-8c4d-def02cfabfa8-istio-envoy\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-2dshc\" (UID: \"eeca2c25-c047-4693-8c4d-def02cfabfa8\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-2dshc" Nov 03 01:11:30.811711 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:11:30.811692 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eeca2c25-c047-4693-8c4d-def02cfabfa8-istio-podinfo\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-2dshc\" (UID: \"eeca2c25-c047-4693-8c4d-def02cfabfa8\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-2dshc" Nov 03 01:11:30.820853 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:11:30.820829 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eeca2c25-c047-4693-8c4d-def02cfabfa8-istio-token\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-2dshc\" (UID: \"eeca2c25-c047-4693-8c4d-def02cfabfa8\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-2dshc" Nov 03 01:11:30.821980 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:11:30.821954 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rzn5\" (UniqueName: \"kubernetes.io/projected/eeca2c25-c047-4693-8c4d-def02cfabfa8-kube-api-access-4rzn5\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-2dshc\" (UID: \"eeca2c25-c047-4693-8c4d-def02cfabfa8\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-2dshc" Nov 03 01:11:30.975064 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:11:30.974986 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-2dshc" Nov 03 01:11:31.462662 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:11:31.448569 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-2dshc"] Nov 03 01:11:31.479298 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:11:31.472366 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:11:31.479298 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:11:31.472474 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:11:31.479298 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:11:31.472520 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:11:31.659298 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:11:31.658342 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-2dshc" event={"ID":"eeca2c25-c047-4693-8c4d-def02cfabfa8","Type":"ContainerStarted","Data":"3782690ef640fd4a6e96c80f18381371464d1deea166af781f233a6f8476a646"} Nov 03 01:11:32.667907 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:11:32.667869 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-2dshc" event={"ID":"eeca2c25-c047-4693-8c4d-def02cfabfa8","Type":"ContainerStarted","Data":"8f392df94677232b2a092579d3a645a15c11416e9ffd51cb37fdaa431a76347e"} Nov 03 01:11:32.688244 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:11:32.688163 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-2dshc" podStartSLOduration=2.688142504 podStartE2EDuration="2.688142504s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:11:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:11:32.687016234 +0000 UTC m=+1834.858019370" watchObservedRunningTime="2026-11-03 01:11:32.688142504 +0000 UTC m=+1834.859145603" Nov 03 01:11:33.002441 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:11:32.986677 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-2dshc" Nov 03 01:11:33.002441 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:11:32.991398 4421 patch_prober.go:29] interesting pod/e2e-shared-a-w3-717217-openshift-default-578c69554c-2dshc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.252:15021/healthz/ready\": dial tcp 10.134.0.252:15021: connect: connection refused" start-of-body= Nov 03 01:11:33.002441 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:11:32.991477 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-2dshc" podUID="eeca2c25-c047-4693-8c4d-def02cfabfa8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.252:15021/healthz/ready\": dial tcp 10.134.0.252:15021: connect: connection refused" Nov 03 01:11:33.191809 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:11:33.191775 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d25759-857574b47-bpmv4" Nov 03 01:11:33.979317 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:11:33.979235 4421 patch_prober.go:29] interesting pod/e2e-shared-a-w3-717217-openshift-default-578c69554c-2dshc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.252:15021/healthz/ready\": dial tcp 10.134.0.252:15021: connect: connection refused" start-of-body= Nov 03 01:11:33.979815 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:11:33.979351 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-2dshc" podUID="eeca2c25-c047-4693-8c4d-def02cfabfa8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.252:15021/healthz/ready\": dial tcp 10.134.0.252:15021: connect: connection refused" Nov 03 01:11:34.975430 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:11:34.975392 4421 patch_prober.go:29] interesting pod/e2e-shared-a-w3-717217-openshift-default-578c69554c-2dshc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.252:15021/healthz/ready\": dial tcp 10.134.0.252:15021: connect: connection refused" start-of-body= Nov 03 01:11:34.975614 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:11:34.975475 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-2dshc" podUID="eeca2c25-c047-4693-8c4d-def02cfabfa8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.252:15021/healthz/ready\": dial tcp 10.134.0.252:15021: connect: connection refused" Nov 03 01:11:35.976024 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:11:35.975991 4421 patch_prober.go:29] interesting pod/e2e-shared-a-w3-717217-openshift-default-578c69554c-2dshc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.252:15021/healthz/ready\": dial tcp 10.134.0.252:15021: connect: connection refused" start-of-body= Nov 03 01:11:35.976631 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:11:35.976038 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-2dshc" podUID="eeca2c25-c047-4693-8c4d-def02cfabfa8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.252:15021/healthz/ready\": dial tcp 10.134.0.252:15021: connect: connection refused" Nov 03 01:11:36.985307 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:11:36.985251 4421 patch_prober.go:29] interesting pod/e2e-shared-a-w3-717217-openshift-default-578c69554c-2dshc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.252:15021/healthz/ready\": dial tcp 10.134.0.252:15021: connect: connection refused" start-of-body= Nov 03 01:11:36.985818 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:11:36.985327 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-2dshc" podUID="eeca2c25-c047-4693-8c4d-def02cfabfa8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.252:15021/healthz/ready\": dial tcp 10.134.0.252:15021: connect: connection refused" Nov 03 01:11:37.991473 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:11:37.991434 4421 patch_prober.go:29] interesting pod/e2e-shared-a-w3-717217-openshift-default-578c69554c-2dshc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.252:15021/healthz/ready\": dial tcp 10.134.0.252:15021: connect: connection refused" start-of-body= Nov 03 01:11:38.000418 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:11:38.000371 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-2dshc" podUID="eeca2c25-c047-4693-8c4d-def02cfabfa8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.252:15021/healthz/ready\": dial tcp 10.134.0.252:15021: connect: connection refused" Nov 03 01:11:38.980917 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:11:38.980888 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-2dshc" Nov 03 01:11:38.981194 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:11:38.981170 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-2dshc" Nov 03 01:11:38.982313 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:11:38.982258 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-2dshc" Nov 03 01:11:41.439164 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:11:41.439086 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-d25759-5c4fc584b4-xldtp"] Nov 03 01:11:41.445739 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:11:41.445718 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d25759-5c4fc584b4-xldtp" Nov 03 01:11:41.457647 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:11:41.457618 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-d25759-serving-cert\"" Nov 03 01:11:41.457914 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:11:41.457704 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-d25759-5c4fc584b4-xldtp"] Nov 03 01:11:41.458024 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:11:41.457799 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 03 01:11:41.459178 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:11:41.459155 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 01:11:41.459344 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:11:41.459209 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-nfcxl\"" Nov 03 01:11:41.459344 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:11:41.459245 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 01:11:41.501973 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:11:41.501935 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f9b330f-a209-4b16-a14a-6441781d4889-metrics-certs\") pod \"maas-api-e2e-extmod-a-d25759-5c4fc584b4-xldtp\" (UID: \"4f9b330f-a209-4b16-a14a-6441781d4889\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d25759-5c4fc584b4-xldtp" Nov 03 01:11:41.502217 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:11:41.502199 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f9b330f-a209-4b16-a14a-6441781d4889-maas-api-tls\") pod \"maas-api-e2e-extmod-a-d25759-5c4fc584b4-xldtp\" (UID: \"4f9b330f-a209-4b16-a14a-6441781d4889\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d25759-5c4fc584b4-xldtp" Nov 03 01:11:41.502355 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:11:41.502240 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fvpx\" (UniqueName: \"kubernetes.io/projected/4f9b330f-a209-4b16-a14a-6441781d4889-kube-api-access-5fvpx\") pod \"maas-api-e2e-extmod-a-d25759-5c4fc584b4-xldtp\" (UID: \"4f9b330f-a209-4b16-a14a-6441781d4889\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d25759-5c4fc584b4-xldtp" Nov 03 01:11:41.502442 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:11:41.502427 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f9b330f-a209-4b16-a14a-6441781d4889-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-d25759-5c4fc584b4-xldtp\" (UID: \"4f9b330f-a209-4b16-a14a-6441781d4889\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d25759-5c4fc584b4-xldtp" Nov 03 01:11:41.607525 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:11:41.606833 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f9b330f-a209-4b16-a14a-6441781d4889-maas-api-tls\") pod \"maas-api-e2e-extmod-a-d25759-5c4fc584b4-xldtp\" (UID: \"4f9b330f-a209-4b16-a14a-6441781d4889\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d25759-5c4fc584b4-xldtp" Nov 03 01:11:41.607525 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:11:41.606966 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fvpx\" (UniqueName: \"kubernetes.io/projected/4f9b330f-a209-4b16-a14a-6441781d4889-kube-api-access-5fvpx\") pod \"maas-api-e2e-extmod-a-d25759-5c4fc584b4-xldtp\" (UID: \"4f9b330f-a209-4b16-a14a-6441781d4889\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d25759-5c4fc584b4-xldtp" Nov 03 01:11:41.612229 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:11:41.608592 4421 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-d25759-serving-cert: secret "maas-api-e2e-extmod-a-d25759-serving-cert" not found Nov 03 01:11:41.612229 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:11:41.608676 4421 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f9b330f-a209-4b16-a14a-6441781d4889-maas-api-tls podName:4f9b330f-a209-4b16-a14a-6441781d4889 nodeName:}" failed. No retries permitted until 2026-11-03 01:11:42.108653615 +0000 UTC m=+1844.279656708 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4f9b330f-a209-4b16-a14a-6441781d4889-maas-api-tls") pod "maas-api-e2e-extmod-a-d25759-5c4fc584b4-xldtp" (UID: "4f9b330f-a209-4b16-a14a-6441781d4889") : secret "maas-api-e2e-extmod-a-d25759-serving-cert" not found Nov 03 01:11:41.623682 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:11:41.618480 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f9b330f-a209-4b16-a14a-6441781d4889-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-d25759-5c4fc584b4-xldtp\" (UID: \"4f9b330f-a209-4b16-a14a-6441781d4889\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d25759-5c4fc584b4-xldtp" Nov 03 01:11:41.623682 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:11:41.618843 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f9b330f-a209-4b16-a14a-6441781d4889-metrics-certs\") pod \"maas-api-e2e-extmod-a-d25759-5c4fc584b4-xldtp\" (UID: \"4f9b330f-a209-4b16-a14a-6441781d4889\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d25759-5c4fc584b4-xldtp" Nov 03 01:11:41.623682 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:11:41.622538 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f9b330f-a209-4b16-a14a-6441781d4889-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-d25759-5c4fc584b4-xldtp\" (UID: \"4f9b330f-a209-4b16-a14a-6441781d4889\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d25759-5c4fc584b4-xldtp" Nov 03 01:11:41.644214 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:11:41.641535 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fvpx\" (UniqueName: \"kubernetes.io/projected/4f9b330f-a209-4b16-a14a-6441781d4889-kube-api-access-5fvpx\") pod \"maas-api-e2e-extmod-a-d25759-5c4fc584b4-xldtp\" (UID: \"4f9b330f-a209-4b16-a14a-6441781d4889\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d25759-5c4fc584b4-xldtp" Nov 03 01:11:41.671353 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:11:41.671316 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f9b330f-a209-4b16-a14a-6441781d4889-metrics-certs\") pod \"maas-api-e2e-extmod-a-d25759-5c4fc584b4-xldtp\" (UID: \"4f9b330f-a209-4b16-a14a-6441781d4889\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d25759-5c4fc584b4-xldtp" Nov 03 01:11:42.130401 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:11:42.130364 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f9b330f-a209-4b16-a14a-6441781d4889-maas-api-tls\") pod \"maas-api-e2e-extmod-a-d25759-5c4fc584b4-xldtp\" (UID: \"4f9b330f-a209-4b16-a14a-6441781d4889\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d25759-5c4fc584b4-xldtp" Nov 03 01:11:42.133000 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:11:42.132976 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f9b330f-a209-4b16-a14a-6441781d4889-maas-api-tls\") pod \"maas-api-e2e-extmod-a-d25759-5c4fc584b4-xldtp\" (UID: \"4f9b330f-a209-4b16-a14a-6441781d4889\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d25759-5c4fc584b4-xldtp" Nov 03 01:11:42.381349 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:11:42.378382 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d25759-5c4fc584b4-xldtp" Nov 03 01:11:42.942635 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:11:42.942605 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f9b330f_a209_4b16_a14a_6441781d4889.slice/crio-5853776eb5fcd897175b7810ed2d1a5b67ff81055cbb66399a510750a3260141 WatchSource:0}: Error finding container 5853776eb5fcd897175b7810ed2d1a5b67ff81055cbb66399a510750a3260141: Status 404 returned error can't find the container with id 5853776eb5fcd897175b7810ed2d1a5b67ff81055cbb66399a510750a3260141 Nov 03 01:11:42.943086 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:11:42.943063 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-d25759-5c4fc584b4-xldtp"] Nov 03 01:11:42.944677 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:11:42.944659 4421 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 01:11:43.863473 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:11:43.863438 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d25759-5c4fc584b4-xldtp" event={"ID":"4f9b330f-a209-4b16-a14a-6441781d4889","Type":"ContainerStarted","Data":"5853776eb5fcd897175b7810ed2d1a5b67ff81055cbb66399a510750a3260141"} Nov 03 01:11:44.648685 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:11:44.648647 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-ddd653-5958b7b98-78g9l"] Nov 03 01:11:44.661476 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:11:44.661436 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ddd653-5958b7b98-78g9l" Nov 03 01:11:44.670007 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:11:44.669976 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-ddd653-5958b7b98-78g9l"] Nov 03 01:11:44.671746 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:11:44.671718 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-ddd653-serving-cert\"" Nov 03 01:11:44.700891 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:11:44.700801 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 01:11:44.705353 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:11:44.705215 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/816d576b-324a-492c-b81f-cf3d4a6dfe92-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-ddd653-5958b7b98-78g9l\" (UID: \"816d576b-324a-492c-b81f-cf3d4a6dfe92\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ddd653-5958b7b98-78g9l" Nov 03 01:11:44.706116 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:11:44.705753 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/816d576b-324a-492c-b81f-cf3d4a6dfe92-maas-api-tls\") pod \"maas-api-e2e-extmod-a-ddd653-5958b7b98-78g9l\" (UID: \"816d576b-324a-492c-b81f-cf3d4a6dfe92\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ddd653-5958b7b98-78g9l" Nov 03 01:11:44.706116 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:11:44.705861 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/816d576b-324a-492c-b81f-cf3d4a6dfe92-metrics-certs\") pod \"maas-api-e2e-extmod-a-ddd653-5958b7b98-78g9l\" (UID: \"816d576b-324a-492c-b81f-cf3d4a6dfe92\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ddd653-5958b7b98-78g9l" Nov 03 01:11:44.706116 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:11:44.705901 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vw8kk\" (UniqueName: \"kubernetes.io/projected/816d576b-324a-492c-b81f-cf3d4a6dfe92-kube-api-access-vw8kk\") pod \"maas-api-e2e-extmod-a-ddd653-5958b7b98-78g9l\" (UID: \"816d576b-324a-492c-b81f-cf3d4a6dfe92\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ddd653-5958b7b98-78g9l" Nov 03 01:11:44.854293 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:11:44.833863 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/816d576b-324a-492c-b81f-cf3d4a6dfe92-maas-api-tls\") pod \"maas-api-e2e-extmod-a-ddd653-5958b7b98-78g9l\" (UID: \"816d576b-324a-492c-b81f-cf3d4a6dfe92\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ddd653-5958b7b98-78g9l" Nov 03 01:11:44.854293 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:11:44.833989 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/816d576b-324a-492c-b81f-cf3d4a6dfe92-metrics-certs\") pod \"maas-api-e2e-extmod-a-ddd653-5958b7b98-78g9l\" (UID: \"816d576b-324a-492c-b81f-cf3d4a6dfe92\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ddd653-5958b7b98-78g9l" Nov 03 01:11:44.854293 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:11:44.834032 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vw8kk\" (UniqueName: \"kubernetes.io/projected/816d576b-324a-492c-b81f-cf3d4a6dfe92-kube-api-access-vw8kk\") pod \"maas-api-e2e-extmod-a-ddd653-5958b7b98-78g9l\" (UID: \"816d576b-324a-492c-b81f-cf3d4a6dfe92\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ddd653-5958b7b98-78g9l" Nov 03 01:11:44.854293 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:11:44.834185 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/816d576b-324a-492c-b81f-cf3d4a6dfe92-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-ddd653-5958b7b98-78g9l\" (UID: \"816d576b-324a-492c-b81f-cf3d4a6dfe92\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ddd653-5958b7b98-78g9l" Nov 03 01:11:44.882986 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:11:44.864296 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/816d576b-324a-492c-b81f-cf3d4a6dfe92-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-ddd653-5958b7b98-78g9l\" (UID: \"816d576b-324a-492c-b81f-cf3d4a6dfe92\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ddd653-5958b7b98-78g9l" Nov 03 01:11:44.914999 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:11:44.914892 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/816d576b-324a-492c-b81f-cf3d4a6dfe92-metrics-certs\") pod \"maas-api-e2e-extmod-a-ddd653-5958b7b98-78g9l\" (UID: \"816d576b-324a-492c-b81f-cf3d4a6dfe92\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ddd653-5958b7b98-78g9l" Nov 03 01:11:44.917854 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:11:44.917537 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vw8kk\" (UniqueName: \"kubernetes.io/projected/816d576b-324a-492c-b81f-cf3d4a6dfe92-kube-api-access-vw8kk\") pod \"maas-api-e2e-extmod-a-ddd653-5958b7b98-78g9l\" (UID: \"816d576b-324a-492c-b81f-cf3d4a6dfe92\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ddd653-5958b7b98-78g9l" Nov 03 01:11:44.923587 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:11:44.923550 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/816d576b-324a-492c-b81f-cf3d4a6dfe92-maas-api-tls\") pod \"maas-api-e2e-extmod-a-ddd653-5958b7b98-78g9l\" (UID: \"816d576b-324a-492c-b81f-cf3d4a6dfe92\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ddd653-5958b7b98-78g9l" Nov 03 01:11:45.007966 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:11:45.007933 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ddd653-5958b7b98-78g9l" Nov 03 01:11:45.492298 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:11:45.431488 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-ddd653-5958b7b98-78g9l"] Nov 03 01:11:46.009898 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:11:46.009841 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ddd653-5958b7b98-78g9l" event={"ID":"816d576b-324a-492c-b81f-cf3d4a6dfe92","Type":"ContainerStarted","Data":"4f57e64b1e44ecc8c7ae00b90e7670c8a2bd49fdaa7477b2563db7ee5e63ea11"} Nov 03 01:11:47.016921 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:11:47.016883 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d25759-5c4fc584b4-xldtp" event={"ID":"4f9b330f-a209-4b16-a14a-6441781d4889","Type":"ContainerStarted","Data":"9ba2c56d8a8c68719f768359b827e5b00b3c4c86b500534989047631adf023b6"} Nov 03 01:11:47.017502 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:11:47.016995 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d25759-5c4fc584b4-xldtp" Nov 03 01:11:47.018888 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:11:47.018832 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ddd653-5958b7b98-78g9l" event={"ID":"816d576b-324a-492c-b81f-cf3d4a6dfe92","Type":"ContainerStarted","Data":"3b2d5350ed7342b7dab9a48c5cbd8e79cae127963aae8a75e2d2008ea195e362"} Nov 03 01:11:47.018888 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:11:47.018879 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ddd653-5958b7b98-78g9l" Nov 03 01:11:47.038059 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:11:47.037952 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d25759-5c4fc584b4-xldtp" podStartSLOduration=2.5985084240000003 podStartE2EDuration="6.037937598s" totalImagesPullingTime="3.439429174s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:11:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:11:47.036487832 +0000 UTC m=+1849.207490928" watchObservedRunningTime="2026-11-03 01:11:47.037937598 +0000 UTC m=+1849.208940693" Nov 03 01:11:47.056919 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:11:47.056869 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ddd653-5958b7b98-78g9l" podStartSLOduration=1.841174953 podStartE2EDuration="3.056855038s" totalImagesPullingTime="1.215680085s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:11:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:11:47.05418523 +0000 UTC m=+1849.225188324" watchObservedRunningTime="2026-11-03 01:11:47.056855038 +0000 UTC m=+1849.227858134" Nov 03 01:11:53.033148 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:11:53.033112 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d25759-5c4fc584b4-xldtp" Nov 03 01:11:53.033617 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:11:53.033564 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ddd653-5958b7b98-78g9l" Nov 03 01:12:01.666082 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:12:01.666043 4421 scope.go:122] "RemoveContainer" containerID="9387a663d5d0703d7e6862c17a2abc112b8d3451fdd0dccf563a954e5e955bb4" Nov 03 01:12:02.250351 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:12:02.202994 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/simulator-w2-f7680d-kserve-85bdc86cd6-8gxrg" podUID="2bdcf521-cebd-4d6b-8554-1b60db8b0f3c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:12:02.325888 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:12:02.325837 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/distinct-w3-83755a-kserve-86bdbbdd85-2cnnl" podUID="f4ae9624-d127-4a55-837d-7c9ff0267b37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:12:02.343296 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:12:02.328301 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-650be0/simulator-w4-650be0-kserve-7bc588774f-gvz2k" podUID="a827fa88-f078-4312-8f12-2c799c11b461" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Nov 03 01:12:02.416474 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:12:02.416404 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/premium-w3-83755a-kserve-7c9d66988b-lt4xd" podUID="c0c6b02e-2ecb-428e-a13d-be6d7cd1ba39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 01:12:02.489440 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:12:02.489371 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/simulator-w3-83755a-kserve-77d6664559-psqtg" podUID="b0a4c234-af58-4da0-a1a4-e1f167953f6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:12:03.229487 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:12:03.229444 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-6d9bxhzmz"] Nov 03 01:12:03.236946 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:12:03.236916 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-6d9bxhzmz" Nov 03 01:12:03.244049 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:12:03.243197 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-57079b\"/\"agc-cred-06b235044840\"" Nov 03 01:12:03.244049 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:12:03.243594 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-57079b\"/\"payload-processing-external-model-e2e--4a3ed774-dockercfg-nwdwk\"" Nov 03 01:12:03.244380 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:12:03.244211 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-57079b\"/\"routing-overlay\"" Nov 03 01:12:03.244380 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:12:03.244294 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-57079b\"/\"kube-root-ca.crt\"" Nov 03 01:12:03.244380 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:12:03.244331 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-57079b\"/\"payload-processing-external-model-plugins-e2e-extmod-b-57079b\"" Nov 03 01:12:03.244572 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:12:03.244384 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-57079b\"/\"openshift-service-ca.crt\"" Nov 03 01:12:03.245881 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:12:03.245852 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-6d9bxhzmz"] Nov 03 01:12:03.441316 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:12:03.421300 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8e9e1695-dbf3-4d60-8121-2321028e461b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-57079b-6d9bxhzmz\" (UID: \"8e9e1695-dbf3-4d60-8121-2321028e461b\") " pod="ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-6d9bxhzmz" Nov 03 01:12:03.442263 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:12:03.442225 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5tln\" (UniqueName: \"kubernetes.io/projected/8e9e1695-dbf3-4d60-8121-2321028e461b-kube-api-access-z5tln\") pod \"payload-processing-external-model-e2e-extmod-b-57079b-6d9bxhzmz\" (UID: \"8e9e1695-dbf3-4d60-8121-2321028e461b\") " pod="ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-6d9bxhzmz" Nov 03 01:12:03.442559 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:12:03.442515 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8e9e1695-dbf3-4d60-8121-2321028e461b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-57079b-6d9bxhzmz\" (UID: \"8e9e1695-dbf3-4d60-8121-2321028e461b\") " pod="ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-6d9bxhzmz" Nov 03 01:12:03.442733 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:12:03.442715 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e9e1695-dbf3-4d60-8121-2321028e461b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-57079b-6d9bxhzmz\" (UID: \"8e9e1695-dbf3-4d60-8121-2321028e461b\") " pod="ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-6d9bxhzmz" Nov 03 01:12:03.553296 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:12:03.547605 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8e9e1695-dbf3-4d60-8121-2321028e461b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-57079b-6d9bxhzmz\" (UID: \"8e9e1695-dbf3-4d60-8121-2321028e461b\") " pod="ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-6d9bxhzmz" Nov 03 01:12:03.553296 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:12:03.547663 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5tln\" (UniqueName: \"kubernetes.io/projected/8e9e1695-dbf3-4d60-8121-2321028e461b-kube-api-access-z5tln\") pod \"payload-processing-external-model-e2e-extmod-b-57079b-6d9bxhzmz\" (UID: \"8e9e1695-dbf3-4d60-8121-2321028e461b\") " pod="ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-6d9bxhzmz" Nov 03 01:12:03.553296 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:12:03.547733 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8e9e1695-dbf3-4d60-8121-2321028e461b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-57079b-6d9bxhzmz\" (UID: \"8e9e1695-dbf3-4d60-8121-2321028e461b\") " pod="ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-6d9bxhzmz" Nov 03 01:12:03.553296 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:12:03.547772 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e9e1695-dbf3-4d60-8121-2321028e461b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-57079b-6d9bxhzmz\" (UID: \"8e9e1695-dbf3-4d60-8121-2321028e461b\") " pod="ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-6d9bxhzmz" Nov 03 01:12:03.553296 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:12:03.549139 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e9e1695-dbf3-4d60-8121-2321028e461b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-57079b-6d9bxhzmz\" (UID: \"8e9e1695-dbf3-4d60-8121-2321028e461b\") " pod="ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-6d9bxhzmz" Nov 03 01:12:03.553296 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:12:03.549770 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8e9e1695-dbf3-4d60-8121-2321028e461b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-57079b-6d9bxhzmz\" (UID: \"8e9e1695-dbf3-4d60-8121-2321028e461b\") " pod="ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-6d9bxhzmz" Nov 03 01:12:03.614366 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:12:03.609104 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8e9e1695-dbf3-4d60-8121-2321028e461b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-57079b-6d9bxhzmz\" (UID: \"8e9e1695-dbf3-4d60-8121-2321028e461b\") " pod="ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-6d9bxhzmz" Nov 03 01:12:03.614366 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:12:03.610778 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5tln\" (UniqueName: \"kubernetes.io/projected/8e9e1695-dbf3-4d60-8121-2321028e461b-kube-api-access-z5tln\") pod \"payload-processing-external-model-e2e-extmod-b-57079b-6d9bxhzmz\" (UID: \"8e9e1695-dbf3-4d60-8121-2321028e461b\") " pod="ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-6d9bxhzmz" Nov 03 01:12:03.861073 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:12:03.860979 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-6d9bxhzmz" Nov 03 01:12:04.174559 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:12:04.174519 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e9e1695_dbf3_4d60_8121_2321028e461b.slice/crio-71fc91298d5d7497a5e22ad0b92695e8e23b72c2db3041f497be528bf84f3312 WatchSource:0}: Error finding container 71fc91298d5d7497a5e22ad0b92695e8e23b72c2db3041f497be528bf84f3312: Status 404 returned error can't find the container with id 71fc91298d5d7497a5e22ad0b92695e8e23b72c2db3041f497be528bf84f3312 Nov 03 01:12:04.178127 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:12:04.177993 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-6d9bxhzmz"] Nov 03 01:12:04.576241 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:12:04.576201 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-6d9bxhzmz" event={"ID":"8e9e1695-dbf3-4d60-8121-2321028e461b","Type":"ContainerStarted","Data":"c7aa980a92da77cd0f941bb008ae5f28e5423e7c557e441620a6d2cf02a91089"} Nov 03 01:12:04.576696 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:12:04.576266 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-6d9bxhzmz" event={"ID":"8e9e1695-dbf3-4d60-8121-2321028e461b","Type":"ContainerStarted","Data":"71fc91298d5d7497a5e22ad0b92695e8e23b72c2db3041f497be528bf84f3312"} Nov 03 01:12:04.576696 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:12:04.576342 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-6d9bxhzmz" Nov 03 01:12:04.600320 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:12:04.600233 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-6d9bxhzmz" podStartSLOduration=1.600217016 podStartE2EDuration="1.600217016s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:12:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:12:04.595064323 +0000 UTC m=+1866.766067420" watchObservedRunningTime="2026-11-03 01:12:04.600217016 +0000 UTC m=+1866.771220146" Nov 03 01:12:14.675325 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:12:14.675250 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 01:12:15.588768 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:12:15.588732 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-6d9bxhzmz" Nov 03 01:12:44.700783 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:12:44.700732 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 01:13:14.672781 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:13:14.672692 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 01:13:44.672911 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:13:44.672871 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 01:13:47.026178 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:13:47.026125 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5151163-qxf58"] Nov 03 01:13:47.031233 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:13:47.031198 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5151163-qxf58"] Nov 03 01:13:48.033710 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:13:48.033677 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="527996cc-c052-44b9-a1e3-a0b7c3125cd9" path="/var/lib/kubelet/pods/527996cc-c052-44b9-a1e3-a0b7c3125cd9/volumes" Nov 03 01:13:58.041963 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:13:58.041925 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09b5f664-9g7fw"] Nov 03 01:13:58.043883 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:13:58.043845 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09b5f664-9g7fw"] Nov 03 01:13:59.983519 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:13:59.983482 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d81e03a-8d5d-435f-80dc-fd9e05310895" path="/var/lib/kubelet/pods/0d81e03a-8d5d-435f-80dc-fd9e05310895/volumes" Nov 03 01:14:00.951040 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:00.951008 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-pvzvs"] Nov 03 01:14:00.951258 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:00.951233 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-pvzvs" podUID="ae4f838b-7b65-477f-8e03-56a00c7635b1" containerName="istio-proxy" containerID="cri-o://e9f4aa83b36003440bc9b7c255040bae3b1312e0b5c93dee8db857bd60d70fc6" gracePeriod=30 Nov 03 01:14:00.956916 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:00.956888 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-d55st"] Nov 03 01:14:00.957098 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:00.957078 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-d55st" podUID="1548cadc-4d40-4310-89bd-0b2021af6faf" containerName="istio-proxy" containerID="cri-o://3d718dedeb1f8cbb8158b0c7caec274cceb7566dc5f44a6623c5e5e618eaa192" gracePeriod=30 Nov 03 01:14:06.245810 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:06.245364 4421 generic.go:356] "Generic (PLEG): container finished" podID="1548cadc-4d40-4310-89bd-0b2021af6faf" containerID="3d718dedeb1f8cbb8158b0c7caec274cceb7566dc5f44a6623c5e5e618eaa192" exitCode=0 Nov 03 01:14:06.245810 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:06.245659 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-d55st" event={"ID":"1548cadc-4d40-4310-89bd-0b2021af6faf","Type":"ContainerDied","Data":"3d718dedeb1f8cbb8158b0c7caec274cceb7566dc5f44a6623c5e5e618eaa192"} Nov 03 01:14:06.249046 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:06.249010 4421 generic.go:356] "Generic (PLEG): container finished" podID="ae4f838b-7b65-477f-8e03-56a00c7635b1" containerID="e9f4aa83b36003440bc9b7c255040bae3b1312e0b5c93dee8db857bd60d70fc6" exitCode=0 Nov 03 01:14:06.249189 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:06.249138 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-pvzvs" event={"ID":"ae4f838b-7b65-477f-8e03-56a00c7635b1","Type":"ContainerDied","Data":"e9f4aa83b36003440bc9b7c255040bae3b1312e0b5c93dee8db857bd60d70fc6"} Nov 03 01:14:06.351539 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:06.351508 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-pvzvs" Nov 03 01:14:06.392925 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:06.392824 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-56df2l9sl"] Nov 03 01:14:06.393917 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:06.393896 4421 cpu_manager.go:414] "containerMap: removing container" podUID="ae4f838b-7b65-477f-8e03-56a00c7635b1" containerName="istio-proxy" Nov 03 01:14:06.394017 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:06.393918 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae4f838b-7b65-477f-8e03-56a00c7635b1" containerName="istio-proxy" Nov 03 01:14:06.394079 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:06.394064 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae4f838b-7b65-477f-8e03-56a00c7635b1" containerName="istio-proxy" Nov 03 01:14:06.405064 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:06.405029 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-5779tv6bk"] Nov 03 01:14:06.405240 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:06.405166 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-56df2l9sl" Nov 03 01:14:06.409395 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:06.409371 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-af8244\"/\"agc-cred-3b03a14485a2\"" Nov 03 01:14:06.412548 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:06.412513 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-5779tv6bk"] Nov 03 01:14:06.412664 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:06.412623 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-5779tv6bk" Nov 03 01:14:06.412802 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:06.412773 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-56df2l9sl"] Nov 03 01:14:06.429540 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:06.429510 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-d55st" Nov 03 01:14:06.470836 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:06.470799 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae4f838b-7b65-477f-8e03-56a00c7635b1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae4f838b-7b65-477f-8e03-56a00c7635b1-workload-socket\") pod \"ae4f838b-7b65-477f-8e03-56a00c7635b1\" (UID: \"ae4f838b-7b65-477f-8e03-56a00c7635b1\") " Nov 03 01:14:06.471082 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:06.470844 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ae4f838b-7b65-477f-8e03-56a00c7635b1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae4f838b-7b65-477f-8e03-56a00c7635b1-istio-podinfo\") pod \"ae4f838b-7b65-477f-8e03-56a00c7635b1\" (UID: \"ae4f838b-7b65-477f-8e03-56a00c7635b1\") " Nov 03 01:14:06.471082 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:06.470904 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae4f838b-7b65-477f-8e03-56a00c7635b1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae4f838b-7b65-477f-8e03-56a00c7635b1-istiod-ca-cert\") pod \"ae4f838b-7b65-477f-8e03-56a00c7635b1\" (UID: \"ae4f838b-7b65-477f-8e03-56a00c7635b1\") " Nov 03 01:14:06.471082 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:06.470965 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae4f838b-7b65-477f-8e03-56a00c7635b1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae4f838b-7b65-477f-8e03-56a00c7635b1-istio-envoy\") pod \"ae4f838b-7b65-477f-8e03-56a00c7635b1\" (UID: \"ae4f838b-7b65-477f-8e03-56a00c7635b1\") " Nov 03 01:14:06.471082 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:06.470992 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1548cadc-4d40-4310-89bd-0b2021af6faf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1548cadc-4d40-4310-89bd-0b2021af6faf-credential-socket\") pod \"1548cadc-4d40-4310-89bd-0b2021af6faf\" (UID: \"1548cadc-4d40-4310-89bd-0b2021af6faf\") " Nov 03 01:14:06.471082 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:06.471081 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1548cadc-4d40-4310-89bd-0b2021af6faf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1548cadc-4d40-4310-89bd-0b2021af6faf-istiod-ca-cert\") pod \"1548cadc-4d40-4310-89bd-0b2021af6faf\" (UID: \"1548cadc-4d40-4310-89bd-0b2021af6faf\") " Nov 03 01:14:06.471514 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:06.471120 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1548cadc-4d40-4310-89bd-0b2021af6faf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1548cadc-4d40-4310-89bd-0b2021af6faf-istio-podinfo\") pod \"1548cadc-4d40-4310-89bd-0b2021af6faf\" (UID: \"1548cadc-4d40-4310-89bd-0b2021af6faf\") " Nov 03 01:14:06.471514 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:06.471209 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1548cadc-4d40-4310-89bd-0b2021af6faf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1548cadc-4d40-4310-89bd-0b2021af6faf-workload-certs\") pod \"1548cadc-4d40-4310-89bd-0b2021af6faf\" (UID: \"1548cadc-4d40-4310-89bd-0b2021af6faf\") " Nov 03 01:14:06.471514 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:06.471250 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae4f838b-7b65-477f-8e03-56a00c7635b1-kube-api-access-fc6h7\" (UniqueName: \"kubernetes.io/projected/ae4f838b-7b65-477f-8e03-56a00c7635b1-kube-api-access-fc6h7\") pod \"ae4f838b-7b65-477f-8e03-56a00c7635b1\" (UID: \"ae4f838b-7b65-477f-8e03-56a00c7635b1\") " Nov 03 01:14:06.471514 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:06.471350 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae4f838b-7b65-477f-8e03-56a00c7635b1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae4f838b-7b65-477f-8e03-56a00c7635b1-credential-socket\") pod \"ae4f838b-7b65-477f-8e03-56a00c7635b1\" (UID: \"ae4f838b-7b65-477f-8e03-56a00c7635b1\") " Nov 03 01:14:06.471514 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:06.471382 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1548cadc-4d40-4310-89bd-0b2021af6faf-istio-token\" (UniqueName: \"kubernetes.io/projected/1548cadc-4d40-4310-89bd-0b2021af6faf-istio-token\") pod \"1548cadc-4d40-4310-89bd-0b2021af6faf\" (UID: \"1548cadc-4d40-4310-89bd-0b2021af6faf\") " Nov 03 01:14:06.471514 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:06.471484 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae4f838b-7b65-477f-8e03-56a00c7635b1-istio-token\" (UniqueName: \"kubernetes.io/projected/ae4f838b-7b65-477f-8e03-56a00c7635b1-istio-token\") pod \"ae4f838b-7b65-477f-8e03-56a00c7635b1\" (UID: \"ae4f838b-7b65-477f-8e03-56a00c7635b1\") " Nov 03 01:14:06.471844 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:06.471567 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae4f838b-7b65-477f-8e03-56a00c7635b1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae4f838b-7b65-477f-8e03-56a00c7635b1-istio-data\") pod \"ae4f838b-7b65-477f-8e03-56a00c7635b1\" (UID: \"ae4f838b-7b65-477f-8e03-56a00c7635b1\") " Nov 03 01:14:06.471844 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:06.471643 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1548cadc-4d40-4310-89bd-0b2021af6faf-kube-api-access-zbswp\" (UniqueName: \"kubernetes.io/projected/1548cadc-4d40-4310-89bd-0b2021af6faf-kube-api-access-zbswp\") pod \"1548cadc-4d40-4310-89bd-0b2021af6faf\" (UID: \"1548cadc-4d40-4310-89bd-0b2021af6faf\") " Nov 03 01:14:06.477454 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:06.476928 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae4f838b-7b65-477f-8e03-56a00c7635b1-credential-socket" pod "ae4f838b-7b65-477f-8e03-56a00c7635b1" (UID: "ae4f838b-7b65-477f-8e03-56a00c7635b1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:14:06.477454 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:06.477053 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1548cadc-4d40-4310-89bd-0b2021af6faf-istiod-ca-cert" pod "1548cadc-4d40-4310-89bd-0b2021af6faf" (UID: "1548cadc-4d40-4310-89bd-0b2021af6faf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:14:06.477454 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:06.477171 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1548cadc-4d40-4310-89bd-0b2021af6faf-workload-certs" pod "1548cadc-4d40-4310-89bd-0b2021af6faf" (UID: "1548cadc-4d40-4310-89bd-0b2021af6faf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:14:06.477454 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:06.477337 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae4f838b-7b65-477f-8e03-56a00c7635b1-workload-socket" pod "ae4f838b-7b65-477f-8e03-56a00c7635b1" (UID: "ae4f838b-7b65-477f-8e03-56a00c7635b1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:14:06.478248 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:06.477993 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1548cadc-4d40-4310-89bd-0b2021af6faf-credential-socket" pod "1548cadc-4d40-4310-89bd-0b2021af6faf" (UID: "1548cadc-4d40-4310-89bd-0b2021af6faf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:14:06.478427 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:06.478392 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-dg98g"] Nov 03 01:14:06.478631 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:06.478611 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae4f838b-7b65-477f-8e03-56a00c7635b1-istiod-ca-cert" pod "ae4f838b-7b65-477f-8e03-56a00c7635b1" (UID: "ae4f838b-7b65-477f-8e03-56a00c7635b1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:14:06.479050 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:06.479031 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae4f838b-7b65-477f-8e03-56a00c7635b1-istio-data" pod "ae4f838b-7b65-477f-8e03-56a00c7635b1" (UID: "ae4f838b-7b65-477f-8e03-56a00c7635b1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:14:06.479424 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:06.479391 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae4f838b-7b65-477f-8e03-56a00c7635b1-istio-token" pod "ae4f838b-7b65-477f-8e03-56a00c7635b1" (UID: "ae4f838b-7b65-477f-8e03-56a00c7635b1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:14:06.479616 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:06.479598 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1548cadc-4d40-4310-89bd-0b2021af6faf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1548cadc-4d40-4310-89bd-0b2021af6faf-istio-data\") pod \"1548cadc-4d40-4310-89bd-0b2021af6faf\" (UID: \"1548cadc-4d40-4310-89bd-0b2021af6faf\") " Nov 03 01:14:06.479780 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:06.479766 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1548cadc-4d40-4310-89bd-0b2021af6faf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1548cadc-4d40-4310-89bd-0b2021af6faf-istio-envoy\") pod \"1548cadc-4d40-4310-89bd-0b2021af6faf\" (UID: \"1548cadc-4d40-4310-89bd-0b2021af6faf\") " Nov 03 01:14:06.479906 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:06.479893 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1548cadc-4d40-4310-89bd-0b2021af6faf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1548cadc-4d40-4310-89bd-0b2021af6faf-workload-socket\") pod \"1548cadc-4d40-4310-89bd-0b2021af6faf\" (UID: \"1548cadc-4d40-4310-89bd-0b2021af6faf\") " Nov 03 01:14:06.480178 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:06.480163 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae4f838b-7b65-477f-8e03-56a00c7635b1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae4f838b-7b65-477f-8e03-56a00c7635b1-workload-certs\") pod \"ae4f838b-7b65-477f-8e03-56a00c7635b1\" (UID: \"ae4f838b-7b65-477f-8e03-56a00c7635b1\") " Nov 03 01:14:06.480672 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:06.480641 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kl9bw\" (UniqueName: \"kubernetes.io/projected/e6622a05-c65e-4d3e-bd0f-286f852413bd-kube-api-access-kl9bw\") pod \"payload-processing-external-model-e2e-extmod-b-af8244-56df2l9sl\" (UID: \"e6622a05-c65e-4d3e-bd0f-286f852413bd\") " pod="ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-56df2l9sl" Nov 03 01:14:06.480779 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:06.480683 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/666777c0-fa6d-4f7b-b32c-5799477477af-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-57079b-5779tv6bk\" (UID: \"666777c0-fa6d-4f7b-b32c-5799477477af\") " pod="ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-5779tv6bk" Nov 03 01:14:06.480779 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:06.480722 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgfwd\" (UniqueName: \"kubernetes.io/projected/666777c0-fa6d-4f7b-b32c-5799477477af-kube-api-access-cgfwd\") pod \"payload-processing-external-model-e2e-extmod-b-57079b-5779tv6bk\" (UID: \"666777c0-fa6d-4f7b-b32c-5799477477af\") " pod="ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-5779tv6bk" Nov 03 01:14:06.480897 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:06.480777 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e6622a05-c65e-4d3e-bd0f-286f852413bd-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-af8244-56df2l9sl\" (UID: \"e6622a05-c65e-4d3e-bd0f-286f852413bd\") " pod="ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-56df2l9sl" Nov 03 01:14:06.480897 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:06.480856 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/666777c0-fa6d-4f7b-b32c-5799477477af-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-57079b-5779tv6bk\" (UID: \"666777c0-fa6d-4f7b-b32c-5799477477af\") " pod="ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-5779tv6bk" Nov 03 01:14:06.481208 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:06.479810 4421 cpu_manager.go:414] "containerMap: removing container" podUID="1548cadc-4d40-4310-89bd-0b2021af6faf" containerName="istio-proxy" Nov 03 01:14:06.481343 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:06.481212 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1548cadc-4d40-4310-89bd-0b2021af6faf" containerName="istio-proxy" Nov 03 01:14:06.481343 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:06.481219 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/666777c0-fa6d-4f7b-b32c-5799477477af-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-57079b-5779tv6bk\" (UID: \"666777c0-fa6d-4f7b-b32c-5799477477af\") " pod="ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-5779tv6bk" Nov 03 01:14:06.481463 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:06.481371 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6622a05-c65e-4d3e-bd0f-286f852413bd-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-af8244-56df2l9sl\" (UID: \"e6622a05-c65e-4d3e-bd0f-286f852413bd\") " pod="ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-56df2l9sl" Nov 03 01:14:06.481463 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:06.481399 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e6622a05-c65e-4d3e-bd0f-286f852413bd-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-af8244-56df2l9sl\" (UID: \"e6622a05-c65e-4d3e-bd0f-286f852413bd\") " pod="ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-56df2l9sl" Nov 03 01:14:06.481631 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:06.481596 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="1548cadc-4d40-4310-89bd-0b2021af6faf" containerName="istio-proxy" Nov 03 01:14:06.481793 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:06.481734 4421 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae4f838b-7b65-477f-8e03-56a00c7635b1-istio-data\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:14:06.481793 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:06.481752 4421 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae4f838b-7b65-477f-8e03-56a00c7635b1-workload-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:14:06.481793 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:06.481767 4421 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae4f838b-7b65-477f-8e03-56a00c7635b1-istiod-ca-cert\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:14:06.481793 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:06.481780 4421 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1548cadc-4d40-4310-89bd-0b2021af6faf-credential-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:14:06.482029 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:06.481794 4421 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1548cadc-4d40-4310-89bd-0b2021af6faf-istiod-ca-cert\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:14:06.482029 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:06.481807 4421 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1548cadc-4d40-4310-89bd-0b2021af6faf-workload-certs\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:14:06.482029 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:06.481820 4421 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae4f838b-7b65-477f-8e03-56a00c7635b1-credential-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:14:06.482029 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:06.481833 4421 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae4f838b-7b65-477f-8e03-56a00c7635b1-istio-token\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:14:06.482242 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:06.480042 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1548cadc-4d40-4310-89bd-0b2021af6faf-istio-data" pod "1548cadc-4d40-4310-89bd-0b2021af6faf" (UID: "1548cadc-4d40-4310-89bd-0b2021af6faf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:14:06.482242 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:06.480975 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ae4f838b-7b65-477f-8e03-56a00c7635b1-istio-podinfo" pod "ae4f838b-7b65-477f-8e03-56a00c7635b1" (UID: "ae4f838b-7b65-477f-8e03-56a00c7635b1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:14:06.482242 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:06.481183 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae4f838b-7b65-477f-8e03-56a00c7635b1-workload-certs" pod "ae4f838b-7b65-477f-8e03-56a00c7635b1" (UID: "ae4f838b-7b65-477f-8e03-56a00c7635b1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:14:06.482242 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:06.482146 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1548cadc-4d40-4310-89bd-0b2021af6faf-workload-socket" pod "1548cadc-4d40-4310-89bd-0b2021af6faf" (UID: "1548cadc-4d40-4310-89bd-0b2021af6faf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:14:06.488739 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:06.488711 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1548cadc-4d40-4310-89bd-0b2021af6faf-istio-podinfo" pod "1548cadc-4d40-4310-89bd-0b2021af6faf" (UID: "1548cadc-4d40-4310-89bd-0b2021af6faf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:14:06.489028 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:06.489004 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1548cadc-4d40-4310-89bd-0b2021af6faf-kube-api-access-zbswp" pod "1548cadc-4d40-4310-89bd-0b2021af6faf" (UID: "1548cadc-4d40-4310-89bd-0b2021af6faf"). InnerVolumeSpecName "kube-api-access-zbswp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:14:06.489231 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:06.489209 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1548cadc-4d40-4310-89bd-0b2021af6faf-istio-token" pod "1548cadc-4d40-4310-89bd-0b2021af6faf" (UID: "1548cadc-4d40-4310-89bd-0b2021af6faf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:14:06.490520 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:06.490457 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae4f838b-7b65-477f-8e03-56a00c7635b1-kube-api-access-fc6h7" pod "ae4f838b-7b65-477f-8e03-56a00c7635b1" (UID: "ae4f838b-7b65-477f-8e03-56a00c7635b1"). InnerVolumeSpecName "kube-api-access-fc6h7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:14:06.491556 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:06.491527 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1548cadc-4d40-4310-89bd-0b2021af6faf-istio-envoy" pod "1548cadc-4d40-4310-89bd-0b2021af6faf" (UID: "1548cadc-4d40-4310-89bd-0b2021af6faf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:14:06.492974 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:06.492951 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-dg98g" Nov 03 01:14:06.498045 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:06.496297 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-dg98g"] Nov 03 01:14:06.507149 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:06.507120 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae4f838b-7b65-477f-8e03-56a00c7635b1-istio-envoy" pod "ae4f838b-7b65-477f-8e03-56a00c7635b1" (UID: "ae4f838b-7b65-477f-8e03-56a00c7635b1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:14:06.509101 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:06.509076 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-57079b\"/\"agc-cred-83631bdec3fc\"" Nov 03 01:14:06.509214 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:06.509193 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-e1f08d-openshift-default-dockercfg-kd24b\"" Nov 03 01:14:06.591263 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:06.588373 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7edd679-63c6-4949-b3e5-e4e09274d522-workload-socket\") pod \"e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-dg98g\" (UID: \"d7edd679-63c6-4949-b3e5-e4e09274d522\") " pod="openshift-ingress/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-dg98g" Nov 03 01:14:06.591263 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:06.588580 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7edd679-63c6-4949-b3e5-e4e09274d522-istio-envoy\") pod \"e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-dg98g\" (UID: \"d7edd679-63c6-4949-b3e5-e4e09274d522\") " pod="openshift-ingress/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-dg98g" Nov 03 01:14:06.591263 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:06.588646 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7edd679-63c6-4949-b3e5-e4e09274d522-istio-token\") pod \"e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-dg98g\" (UID: \"d7edd679-63c6-4949-b3e5-e4e09274d522\") " pod="openshift-ingress/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-dg98g" Nov 03 01:14:06.591263 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:06.588679 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7edd679-63c6-4949-b3e5-e4e09274d522-istio-podinfo\") pod \"e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-dg98g\" (UID: \"d7edd679-63c6-4949-b3e5-e4e09274d522\") " pod="openshift-ingress/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-dg98g" Nov 03 01:14:06.591263 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:06.588738 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kl9bw\" (UniqueName: \"kubernetes.io/projected/e6622a05-c65e-4d3e-bd0f-286f852413bd-kube-api-access-kl9bw\") pod \"payload-processing-external-model-e2e-extmod-b-af8244-56df2l9sl\" (UID: \"e6622a05-c65e-4d3e-bd0f-286f852413bd\") " pod="ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-56df2l9sl" Nov 03 01:14:06.591263 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:06.588764 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/666777c0-fa6d-4f7b-b32c-5799477477af-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-57079b-5779tv6bk\" (UID: \"666777c0-fa6d-4f7b-b32c-5799477477af\") " pod="ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-5779tv6bk" Nov 03 01:14:06.591263 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:06.588790 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7edd679-63c6-4949-b3e5-e4e09274d522-workload-certs\") pod \"e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-dg98g\" (UID: \"d7edd679-63c6-4949-b3e5-e4e09274d522\") " pod="openshift-ingress/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-dg98g" Nov 03 01:14:06.591263 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:06.588826 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgfwd\" (UniqueName: \"kubernetes.io/projected/666777c0-fa6d-4f7b-b32c-5799477477af-kube-api-access-cgfwd\") pod \"payload-processing-external-model-e2e-extmod-b-57079b-5779tv6bk\" (UID: \"666777c0-fa6d-4f7b-b32c-5799477477af\") " pod="ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-5779tv6bk" Nov 03 01:14:06.591263 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:06.588874 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e6622a05-c65e-4d3e-bd0f-286f852413bd-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-af8244-56df2l9sl\" (UID: \"e6622a05-c65e-4d3e-bd0f-286f852413bd\") " pod="ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-56df2l9sl" Nov 03 01:14:06.591263 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:06.588965 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/666777c0-fa6d-4f7b-b32c-5799477477af-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-57079b-5779tv6bk\" (UID: \"666777c0-fa6d-4f7b-b32c-5799477477af\") " pod="ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-5779tv6bk" Nov 03 01:14:06.591263 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:06.589005 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwkh7\" (UniqueName: \"kubernetes.io/projected/d7edd679-63c6-4949-b3e5-e4e09274d522-kube-api-access-jwkh7\") pod \"e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-dg98g\" (UID: \"d7edd679-63c6-4949-b3e5-e4e09274d522\") " pod="openshift-ingress/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-dg98g" Nov 03 01:14:06.591263 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:06.589184 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7edd679-63c6-4949-b3e5-e4e09274d522-istio-data\") pod \"e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-dg98g\" (UID: \"d7edd679-63c6-4949-b3e5-e4e09274d522\") " pod="openshift-ingress/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-dg98g" Nov 03 01:14:06.591263 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:06.589360 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7edd679-63c6-4949-b3e5-e4e09274d522-istiod-ca-cert\") pod \"e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-dg98g\" (UID: \"d7edd679-63c6-4949-b3e5-e4e09274d522\") " pod="openshift-ingress/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-dg98g" Nov 03 01:14:06.592327 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:06.592243 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e6622a05-c65e-4d3e-bd0f-286f852413bd-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-af8244-56df2l9sl\" (UID: \"e6622a05-c65e-4d3e-bd0f-286f852413bd\") " pod="ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-56df2l9sl" Nov 03 01:14:06.592777 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:06.592749 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/666777c0-fa6d-4f7b-b32c-5799477477af-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-57079b-5779tv6bk\" (UID: \"666777c0-fa6d-4f7b-b32c-5799477477af\") " pod="ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-5779tv6bk" Nov 03 01:14:06.592922 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:06.592893 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7edd679-63c6-4949-b3e5-e4e09274d522-credential-socket\") pod \"e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-dg98g\" (UID: \"d7edd679-63c6-4949-b3e5-e4e09274d522\") " pod="openshift-ingress/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-dg98g" Nov 03 01:14:06.592966 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:06.592946 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6622a05-c65e-4d3e-bd0f-286f852413bd-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-af8244-56df2l9sl\" (UID: \"e6622a05-c65e-4d3e-bd0f-286f852413bd\") " pod="ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-56df2l9sl" Nov 03 01:14:06.593010 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:06.592987 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e6622a05-c65e-4d3e-bd0f-286f852413bd-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-af8244-56df2l9sl\" (UID: \"e6622a05-c65e-4d3e-bd0f-286f852413bd\") " pod="ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-56df2l9sl" Nov 03 01:14:06.593412 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:06.593206 4421 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1548cadc-4d40-4310-89bd-0b2021af6faf-istio-podinfo\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:14:06.593412 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:06.593227 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fc6h7\" (UniqueName: \"kubernetes.io/projected/ae4f838b-7b65-477f-8e03-56a00c7635b1-kube-api-access-fc6h7\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:14:06.593412 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:06.593244 4421 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1548cadc-4d40-4310-89bd-0b2021af6faf-istio-token\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:14:06.593412 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:06.593260 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbswp\" (UniqueName: \"kubernetes.io/projected/1548cadc-4d40-4310-89bd-0b2021af6faf-kube-api-access-zbswp\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:14:06.593412 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:06.593285 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/666777c0-fa6d-4f7b-b32c-5799477477af-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-57079b-5779tv6bk\" (UID: \"666777c0-fa6d-4f7b-b32c-5799477477af\") " pod="ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-5779tv6bk" Nov 03 01:14:06.593412 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:06.593312 4421 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1548cadc-4d40-4310-89bd-0b2021af6faf-istio-data\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:14:06.593412 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:06.593327 4421 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1548cadc-4d40-4310-89bd-0b2021af6faf-istio-envoy\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:14:06.593412 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:06.593368 4421 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1548cadc-4d40-4310-89bd-0b2021af6faf-workload-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:14:06.593412 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:06.593387 4421 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae4f838b-7b65-477f-8e03-56a00c7635b1-workload-certs\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:14:06.593412 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:06.593400 4421 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae4f838b-7b65-477f-8e03-56a00c7635b1-istio-podinfo\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:14:06.593412 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:06.593415 4421 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae4f838b-7b65-477f-8e03-56a00c7635b1-istio-envoy\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:14:06.593959 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:06.593695 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/666777c0-fa6d-4f7b-b32c-5799477477af-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-57079b-5779tv6bk\" (UID: \"666777c0-fa6d-4f7b-b32c-5799477477af\") " pod="ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-5779tv6bk" Nov 03 01:14:06.594223 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:06.594197 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6622a05-c65e-4d3e-bd0f-286f852413bd-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-af8244-56df2l9sl\" (UID: \"e6622a05-c65e-4d3e-bd0f-286f852413bd\") " pod="ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-56df2l9sl" Nov 03 01:14:06.595928 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:06.595901 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/666777c0-fa6d-4f7b-b32c-5799477477af-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-57079b-5779tv6bk\" (UID: \"666777c0-fa6d-4f7b-b32c-5799477477af\") " pod="ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-5779tv6bk" Nov 03 01:14:06.596015 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:06.595950 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e6622a05-c65e-4d3e-bd0f-286f852413bd-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-af8244-56df2l9sl\" (UID: \"e6622a05-c65e-4d3e-bd0f-286f852413bd\") " pod="ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-56df2l9sl" Nov 03 01:14:06.608034 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:06.608003 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kl9bw\" (UniqueName: \"kubernetes.io/projected/e6622a05-c65e-4d3e-bd0f-286f852413bd-kube-api-access-kl9bw\") pod \"payload-processing-external-model-e2e-extmod-b-af8244-56df2l9sl\" (UID: \"e6622a05-c65e-4d3e-bd0f-286f852413bd\") " pod="ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-56df2l9sl" Nov 03 01:14:06.615959 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:06.615932 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgfwd\" (UniqueName: \"kubernetes.io/projected/666777c0-fa6d-4f7b-b32c-5799477477af-kube-api-access-cgfwd\") pod \"payload-processing-external-model-e2e-extmod-b-57079b-5779tv6bk\" (UID: \"666777c0-fa6d-4f7b-b32c-5799477477af\") " pod="ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-5779tv6bk" Nov 03 01:14:06.695316 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:06.695241 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7edd679-63c6-4949-b3e5-e4e09274d522-workload-certs\") pod \"e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-dg98g\" (UID: \"d7edd679-63c6-4949-b3e5-e4e09274d522\") " pod="openshift-ingress/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-dg98g" Nov 03 01:14:06.695542 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:06.695419 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwkh7\" (UniqueName: \"kubernetes.io/projected/d7edd679-63c6-4949-b3e5-e4e09274d522-kube-api-access-jwkh7\") pod \"e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-dg98g\" (UID: \"d7edd679-63c6-4949-b3e5-e4e09274d522\") " pod="openshift-ingress/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-dg98g" Nov 03 01:14:06.695542 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:06.695513 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7edd679-63c6-4949-b3e5-e4e09274d522-istio-data\") pod \"e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-dg98g\" (UID: \"d7edd679-63c6-4949-b3e5-e4e09274d522\") " pod="openshift-ingress/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-dg98g" Nov 03 01:14:06.695750 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:06.695619 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7edd679-63c6-4949-b3e5-e4e09274d522-istiod-ca-cert\") pod \"e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-dg98g\" (UID: \"d7edd679-63c6-4949-b3e5-e4e09274d522\") " pod="openshift-ingress/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-dg98g" Nov 03 01:14:06.695750 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:06.695699 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7edd679-63c6-4949-b3e5-e4e09274d522-credential-socket\") pod \"e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-dg98g\" (UID: \"d7edd679-63c6-4949-b3e5-e4e09274d522\") " pod="openshift-ingress/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-dg98g" Nov 03 01:14:06.695857 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:06.695762 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7edd679-63c6-4949-b3e5-e4e09274d522-workload-certs\") pod \"e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-dg98g\" (UID: \"d7edd679-63c6-4949-b3e5-e4e09274d522\") " pod="openshift-ingress/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-dg98g" Nov 03 01:14:06.695857 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:06.695807 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7edd679-63c6-4949-b3e5-e4e09274d522-workload-socket\") pod \"e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-dg98g\" (UID: \"d7edd679-63c6-4949-b3e5-e4e09274d522\") " pod="openshift-ingress/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-dg98g" Nov 03 01:14:06.695857 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:06.695835 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7edd679-63c6-4949-b3e5-e4e09274d522-istio-envoy\") pod \"e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-dg98g\" (UID: \"d7edd679-63c6-4949-b3e5-e4e09274d522\") " pod="openshift-ingress/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-dg98g" Nov 03 01:14:06.696029 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:06.695877 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7edd679-63c6-4949-b3e5-e4e09274d522-istio-data\") pod \"e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-dg98g\" (UID: \"d7edd679-63c6-4949-b3e5-e4e09274d522\") " pod="openshift-ingress/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-dg98g" Nov 03 01:14:06.696029 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:06.695890 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7edd679-63c6-4949-b3e5-e4e09274d522-istio-token\") pod \"e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-dg98g\" (UID: \"d7edd679-63c6-4949-b3e5-e4e09274d522\") " pod="openshift-ingress/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-dg98g" Nov 03 01:14:06.696029 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:06.695932 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7edd679-63c6-4949-b3e5-e4e09274d522-istio-podinfo\") pod \"e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-dg98g\" (UID: \"d7edd679-63c6-4949-b3e5-e4e09274d522\") " pod="openshift-ingress/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-dg98g" Nov 03 01:14:06.696188 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:06.696041 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7edd679-63c6-4949-b3e5-e4e09274d522-credential-socket\") pod \"e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-dg98g\" (UID: \"d7edd679-63c6-4949-b3e5-e4e09274d522\") " pod="openshift-ingress/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-dg98g" Nov 03 01:14:06.696188 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:06.696123 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7edd679-63c6-4949-b3e5-e4e09274d522-workload-socket\") pod \"e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-dg98g\" (UID: \"d7edd679-63c6-4949-b3e5-e4e09274d522\") " pod="openshift-ingress/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-dg98g" Nov 03 01:14:06.696460 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:06.696428 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7edd679-63c6-4949-b3e5-e4e09274d522-istiod-ca-cert\") pod \"e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-dg98g\" (UID: \"d7edd679-63c6-4949-b3e5-e4e09274d522\") " pod="openshift-ingress/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-dg98g" Nov 03 01:14:06.698181 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:06.698151 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7edd679-63c6-4949-b3e5-e4e09274d522-istio-envoy\") pod \"e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-dg98g\" (UID: \"d7edd679-63c6-4949-b3e5-e4e09274d522\") " pod="openshift-ingress/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-dg98g" Nov 03 01:14:06.698443 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:06.698422 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7edd679-63c6-4949-b3e5-e4e09274d522-istio-podinfo\") pod \"e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-dg98g\" (UID: \"d7edd679-63c6-4949-b3e5-e4e09274d522\") " pod="openshift-ingress/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-dg98g" Nov 03 01:14:06.707163 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:06.707136 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7edd679-63c6-4949-b3e5-e4e09274d522-istio-token\") pod \"e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-dg98g\" (UID: \"d7edd679-63c6-4949-b3e5-e4e09274d522\") " pod="openshift-ingress/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-dg98g" Nov 03 01:14:06.707329 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:06.707267 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwkh7\" (UniqueName: \"kubernetes.io/projected/d7edd679-63c6-4949-b3e5-e4e09274d522-kube-api-access-jwkh7\") pod \"e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-dg98g\" (UID: \"d7edd679-63c6-4949-b3e5-e4e09274d522\") " pod="openshift-ingress/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-dg98g" Nov 03 01:14:06.742382 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:06.742347 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-56df2l9sl" Nov 03 01:14:06.750254 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:06.750176 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-5779tv6bk" Nov 03 01:14:06.833176 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:06.828572 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-dg98g" Nov 03 01:14:07.232828 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:14:07.232789 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7edd679_63c6_4949_b3e5_e4e09274d522.slice/crio-892cd0cb29ffba4246bd60daf0780c582832cbc32d34f4d7c49297772f53f984 WatchSource:0}: Error finding container 892cd0cb29ffba4246bd60daf0780c582832cbc32d34f4d7c49297772f53f984: Status 404 returned error can't find the container with id 892cd0cb29ffba4246bd60daf0780c582832cbc32d34f4d7c49297772f53f984 Nov 03 01:14:07.233496 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:07.233471 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-dg98g"] Nov 03 01:14:07.236826 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:07.236746 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:14:07.236948 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:07.236841 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:14:07.236948 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:07.236904 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:14:07.261658 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:07.261256 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-d55st" Nov 03 01:14:07.261658 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:07.261259 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-d55st" event={"ID":"1548cadc-4d40-4310-89bd-0b2021af6faf","Type":"ContainerDied","Data":"1875d298785835728da841941970bdded2454e057d20c3f9562acbe2706029bb"} Nov 03 01:14:07.261658 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:07.261461 4421 scope.go:122] "RemoveContainer" containerID="3d718dedeb1f8cbb8158b0c7caec274cceb7566dc5f44a6623c5e5e618eaa192" Nov 03 01:14:07.263934 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:07.263862 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-dg98g" event={"ID":"d7edd679-63c6-4949-b3e5-e4e09274d522","Type":"ContainerStarted","Data":"892cd0cb29ffba4246bd60daf0780c582832cbc32d34f4d7c49297772f53f984"} Nov 03 01:14:07.266044 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:07.266016 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-pvzvs" event={"ID":"ae4f838b-7b65-477f-8e03-56a00c7635b1","Type":"ContainerDied","Data":"fdc471991de0be67dc118d68cd9ba04b57a365a332a9a47750f8f86827bf765b"} Nov 03 01:14:07.266155 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:07.266127 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-pvzvs" Nov 03 01:14:07.268738 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:07.268711 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-56df2l9sl"] Nov 03 01:14:07.299941 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:07.299916 4421 scope.go:122] "RemoveContainer" containerID="e9f4aa83b36003440bc9b7c255040bae3b1312e0b5c93dee8db857bd60d70fc6" Nov 03 01:14:07.346462 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:07.344583 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-5779tv6bk"] Nov 03 01:14:07.354034 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:07.354008 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-pvzvs"] Nov 03 01:14:07.362643 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:07.362614 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-pvzvs"] Nov 03 01:14:07.373012 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:07.372908 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-d55st"] Nov 03 01:14:07.378639 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:07.378613 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-d55st"] Nov 03 01:14:07.994063 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:07.994029 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1548cadc-4d40-4310-89bd-0b2021af6faf" path="/var/lib/kubelet/pods/1548cadc-4d40-4310-89bd-0b2021af6faf/volumes" Nov 03 01:14:07.994894 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:07.994871 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae4f838b-7b65-477f-8e03-56a00c7635b1" path="/var/lib/kubelet/pods/ae4f838b-7b65-477f-8e03-56a00c7635b1/volumes" Nov 03 01:14:08.278824 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:08.278783 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-5779tv6bk" event={"ID":"666777c0-fa6d-4f7b-b32c-5799477477af","Type":"ContainerStarted","Data":"efc5aaf72beeb404204da947498b1297a0429f4c389cbc0b1fd27147a23acc8c"} Nov 03 01:14:08.279299 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:08.278852 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-5779tv6bk" event={"ID":"666777c0-fa6d-4f7b-b32c-5799477477af","Type":"ContainerStarted","Data":"379444f5f40bfdc983501d82084876e63df73ab72342fd6055d603070cab1388"} Nov 03 01:14:08.279299 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:08.278884 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-5779tv6bk" Nov 03 01:14:08.280722 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:08.280693 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-dg98g" event={"ID":"d7edd679-63c6-4949-b3e5-e4e09274d522","Type":"ContainerStarted","Data":"3e48bdcc23749d41fff58606c10e327e7b4b81376eb8d44e28bb4d33eee1db46"} Nov 03 01:14:08.282673 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:08.282646 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-56df2l9sl" event={"ID":"e6622a05-c65e-4d3e-bd0f-286f852413bd","Type":"ContainerStarted","Data":"183ab6f6737edb534ee318667050da7b3d80a2221b7748b4a38ccaa3eab053d0"} Nov 03 01:14:08.282789 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:08.282692 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-56df2l9sl" event={"ID":"e6622a05-c65e-4d3e-bd0f-286f852413bd","Type":"ContainerStarted","Data":"dded4c7372167fd7993f553704d6a456f3b73d9581fd3745df3caff410786049"} Nov 03 01:14:08.282789 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:08.282745 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-56df2l9sl" Nov 03 01:14:08.299995 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:08.299934 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-5779tv6bk" podStartSLOduration=37.29991602 podStartE2EDuration="37.29991602s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:13:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:14:08.296756661 +0000 UTC m=+1990.467759757" watchObservedRunningTime="2026-11-03 01:14:08.29991602 +0000 UTC m=+1990.470919117" Nov 03 01:14:08.319301 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:08.319225 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-56df2l9sl" podStartSLOduration=120.319206349 podStartE2EDuration="2m0.319206349s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:12:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:14:08.315601401 +0000 UTC m=+1990.486604500" watchObservedRunningTime="2026-11-03 01:14:08.319206349 +0000 UTC m=+1990.490209446" Nov 03 01:14:08.346629 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:08.346567 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-dg98g" podStartSLOduration=117.346546073 podStartE2EDuration="1m57.346546073s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:12:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:14:08.34121371 +0000 UTC m=+1990.512216817" watchObservedRunningTime="2026-11-03 01:14:08.346546073 +0000 UTC m=+1990.517549169" Nov 03 01:14:08.835630 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:08.835596 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-dg98g" Nov 03 01:14:08.842529 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:08.842499 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-dg98g" Nov 03 01:14:09.293217 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:09.293173 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-dg98g" Nov 03 01:14:09.294245 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:09.294220 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-dg98g" Nov 03 01:14:12.085896 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:12.085863 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-e1f08d-597cb67847-6rjh6"] Nov 03 01:14:12.092974 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:12.092941 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e1f08d-597cb67847-6rjh6" Nov 03 01:14:12.096260 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:12.096225 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-e1f08d-597cb67847-6rjh6"] Nov 03 01:14:12.096899 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:12.096876 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-e1f08d\"" Nov 03 01:14:12.098167 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:12.098143 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-e1f08d-dockercfg-bct88\"" Nov 03 01:14:12.301684 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:12.246619 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4e62211-7bb6-42d2-8e73-c95d43c2737c-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-e1f08d-597cb67847-6rjh6\" (UID: \"f4e62211-7bb6-42d2-8e73-c95d43c2737c\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e1f08d-597cb67847-6rjh6" Nov 03 01:14:12.301684 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:12.246758 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbqcd\" (UniqueName: \"kubernetes.io/projected/f4e62211-7bb6-42d2-8e73-c95d43c2737c-kube-api-access-rbqcd\") pod \"payload-pre-processing-e2e-extmod-b-e1f08d-597cb67847-6rjh6\" (UID: \"f4e62211-7bb6-42d2-8e73-c95d43c2737c\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e1f08d-597cb67847-6rjh6" Nov 03 01:14:12.390345 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:12.380521 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbqcd\" (UniqueName: \"kubernetes.io/projected/f4e62211-7bb6-42d2-8e73-c95d43c2737c-kube-api-access-rbqcd\") pod \"payload-pre-processing-e2e-extmod-b-e1f08d-597cb67847-6rjh6\" (UID: \"f4e62211-7bb6-42d2-8e73-c95d43c2737c\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e1f08d-597cb67847-6rjh6" Nov 03 01:14:12.390345 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:12.380936 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4e62211-7bb6-42d2-8e73-c95d43c2737c-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-e1f08d-597cb67847-6rjh6\" (UID: \"f4e62211-7bb6-42d2-8e73-c95d43c2737c\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e1f08d-597cb67847-6rjh6" Nov 03 01:14:12.390345 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:12.381827 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4e62211-7bb6-42d2-8e73-c95d43c2737c-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-e1f08d-597cb67847-6rjh6\" (UID: \"f4e62211-7bb6-42d2-8e73-c95d43c2737c\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e1f08d-597cb67847-6rjh6" Nov 03 01:14:12.455054 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:12.455018 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbqcd\" (UniqueName: \"kubernetes.io/projected/f4e62211-7bb6-42d2-8e73-c95d43c2737c-kube-api-access-rbqcd\") pod \"payload-pre-processing-e2e-extmod-b-e1f08d-597cb67847-6rjh6\" (UID: \"f4e62211-7bb6-42d2-8e73-c95d43c2737c\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e1f08d-597cb67847-6rjh6" Nov 03 01:14:12.715910 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:12.715808 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e1f08d-597cb67847-6rjh6" Nov 03 01:14:13.034753 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:14:13.032729 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4e62211_7bb6_42d2_8e73_c95d43c2737c.slice/crio-d82e62ce7f215da719e8e43425737a12336fab57ad8975f51d150ec74f76d586 WatchSource:0}: Error finding container d82e62ce7f215da719e8e43425737a12336fab57ad8975f51d150ec74f76d586: Status 404 returned error can't find the container with id d82e62ce7f215da719e8e43425737a12336fab57ad8975f51d150ec74f76d586 Nov 03 01:14:13.053173 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:13.053144 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-e1f08d-597cb67847-6rjh6"] Nov 03 01:14:13.455090 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:13.454994 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e1f08d-597cb67847-6rjh6" event={"ID":"f4e62211-7bb6-42d2-8e73-c95d43c2737c","Type":"ContainerStarted","Data":"478ff80e0d5bd352e46475f00afacd38b6bd4378d7eb0aff9a2223fb40a44aa2"} Nov 03 01:14:13.455090 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:13.455058 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e1f08d-597cb67847-6rjh6" event={"ID":"f4e62211-7bb6-42d2-8e73-c95d43c2737c","Type":"ContainerStarted","Data":"d82e62ce7f215da719e8e43425737a12336fab57ad8975f51d150ec74f76d586"} Nov 03 01:14:13.455090 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:13.455080 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e1f08d-597cb67847-6rjh6" Nov 03 01:14:13.471053 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:13.471003 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e1f08d-597cb67847-6rjh6" podStartSLOduration=1.470988995 podStartE2EDuration="1.470988995s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:14:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:14:13.469625422 +0000 UTC m=+1995.640628583" watchObservedRunningTime="2026-11-03 01:14:13.470988995 +0000 UTC m=+1995.641992093" Nov 03 01:14:14.893562 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:14:14.893515 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 01:14:18.981400 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:18.981365 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-5779tv6bk"] Nov 03 01:14:18.984806 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:18.984781 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-5779tv6bk" Nov 03 01:14:19.296047 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:19.296014 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-56df2l9sl" Nov 03 01:14:19.355298 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:19.355249 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-55d6nfg5v"] Nov 03 01:14:23.982627 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:23.982586 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-5779tv6bk" podUID="666777c0-fa6d-4f7b-b32c-5799477477af" containerName="payload-processing" containerID="cri-o://efc5aaf72beeb404204da947498b1297a0429f4c389cbc0b1fd27147a23acc8c" gracePeriod=25 Nov 03 01:14:24.359943 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:24.359911 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-5779tv6bk" Nov 03 01:14:24.396438 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:24.396404 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-e1f08d-f5fb4c64f-mmvp6"] Nov 03 01:14:24.397807 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:24.397784 4421 cpu_manager.go:414] "containerMap: removing container" podUID="666777c0-fa6d-4f7b-b32c-5799477477af" containerName="payload-processing" Nov 03 01:14:24.397882 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:24.397810 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="666777c0-fa6d-4f7b-b32c-5799477477af" containerName="payload-processing" Nov 03 01:14:24.398086 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:24.398070 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="666777c0-fa6d-4f7b-b32c-5799477477af" containerName="payload-processing" Nov 03 01:14:24.411529 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:24.410732 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/666777c0-fa6d-4f7b-b32c-5799477477af-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/666777c0-fa6d-4f7b-b32c-5799477477af-plugins-config-volume\") pod \"666777c0-fa6d-4f7b-b32c-5799477477af\" (UID: \"666777c0-fa6d-4f7b-b32c-5799477477af\") " Nov 03 01:14:24.411529 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:24.410808 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/666777c0-fa6d-4f7b-b32c-5799477477af-kube-api-access-cgfwd\" (UniqueName: \"kubernetes.io/projected/666777c0-fa6d-4f7b-b32c-5799477477af-kube-api-access-cgfwd\") pod \"666777c0-fa6d-4f7b-b32c-5799477477af\" (UID: \"666777c0-fa6d-4f7b-b32c-5799477477af\") " Nov 03 01:14:24.411529 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:24.410840 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/666777c0-fa6d-4f7b-b32c-5799477477af-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/666777c0-fa6d-4f7b-b32c-5799477477af-routing-overlay-volume\") pod \"666777c0-fa6d-4f7b-b32c-5799477477af\" (UID: \"666777c0-fa6d-4f7b-b32c-5799477477af\") " Nov 03 01:14:24.411529 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:24.410869 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-e1f08d-f5fb4c64f-mmvp6"] Nov 03 01:14:24.411529 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:24.410932 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/666777c0-fa6d-4f7b-b32c-5799477477af-provider-credentials\" (UniqueName: \"kubernetes.io/projected/666777c0-fa6d-4f7b-b32c-5799477477af-provider-credentials\") pod \"666777c0-fa6d-4f7b-b32c-5799477477af\" (UID: \"666777c0-fa6d-4f7b-b32c-5799477477af\") " Nov 03 01:14:24.411529 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:24.410986 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-e1f08d-f5fb4c64f-mmvp6" Nov 03 01:14:24.413838 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:24.413045 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/666777c0-fa6d-4f7b-b32c-5799477477af-plugins-config-volume" pod "666777c0-fa6d-4f7b-b32c-5799477477af" (UID: "666777c0-fa6d-4f7b-b32c-5799477477af"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:14:24.413838 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:24.413202 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/666777c0-fa6d-4f7b-b32c-5799477477af-routing-overlay-volume" pod "666777c0-fa6d-4f7b-b32c-5799477477af" (UID: "666777c0-fa6d-4f7b-b32c-5799477477af"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:14:24.416375 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:24.415289 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/666777c0-fa6d-4f7b-b32c-5799477477af-provider-credentials" pod "666777c0-fa6d-4f7b-b32c-5799477477af" (UID: "666777c0-fa6d-4f7b-b32c-5799477477af"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:14:24.416787 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:24.416762 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/666777c0-fa6d-4f7b-b32c-5799477477af-kube-api-access-cgfwd" pod "666777c0-fa6d-4f7b-b32c-5799477477af" (UID: "666777c0-fa6d-4f7b-b32c-5799477477af"). InnerVolumeSpecName "kube-api-access-cgfwd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:14:24.507452 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:24.507419 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e1f08d-597cb67847-6rjh6" Nov 03 01:14:24.513680 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:24.513653 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e60b0573-2ad3-40d9-898e-9c644f41dbc9-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-e1f08d-f5fb4c64f-mmvp6\" (UID: \"e60b0573-2ad3-40d9-898e-9c644f41dbc9\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-e1f08d-f5fb4c64f-mmvp6" Nov 03 01:14:24.513920 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:24.513896 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57js7\" (UniqueName: \"kubernetes.io/projected/e60b0573-2ad3-40d9-898e-9c644f41dbc9-kube-api-access-57js7\") pod \"payload-processing-e2e-extmod-b-e1f08d-f5fb4c64f-mmvp6\" (UID: \"e60b0573-2ad3-40d9-898e-9c644f41dbc9\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-e1f08d-f5fb4c64f-mmvp6" Nov 03 01:14:24.514197 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:24.514177 4421 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/666777c0-fa6d-4f7b-b32c-5799477477af-provider-credentials\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:14:24.514310 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:24.514203 4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/666777c0-fa6d-4f7b-b32c-5799477477af-plugins-config-volume\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:14:24.514310 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:24.514221 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cgfwd\" (UniqueName: \"kubernetes.io/projected/666777c0-fa6d-4f7b-b32c-5799477477af-kube-api-access-cgfwd\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:14:24.514310 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:24.514235 4421 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/666777c0-fa6d-4f7b-b32c-5799477477af-routing-overlay-volume\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:14:24.593957 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:24.593836 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-55d6nfg5v" podUID="31e81deb-0e7e-4c26-b43f-870959edffe8" containerName="payload-processing" containerID="cri-o://f9336e3115a14139ea618b94a58991993df29dd2e4466eef0d9432a13b292a20" gracePeriod=25 Nov 03 01:14:24.615866 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:24.615823 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e60b0573-2ad3-40d9-898e-9c644f41dbc9-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-e1f08d-f5fb4c64f-mmvp6\" (UID: \"e60b0573-2ad3-40d9-898e-9c644f41dbc9\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-e1f08d-f5fb4c64f-mmvp6" Nov 03 01:14:24.616158 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:24.616142 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57js7\" (UniqueName: \"kubernetes.io/projected/e60b0573-2ad3-40d9-898e-9c644f41dbc9-kube-api-access-57js7\") pod \"payload-processing-e2e-extmod-b-e1f08d-f5fb4c64f-mmvp6\" (UID: \"e60b0573-2ad3-40d9-898e-9c644f41dbc9\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-e1f08d-f5fb4c64f-mmvp6" Nov 03 01:14:24.616715 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:24.616691 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e60b0573-2ad3-40d9-898e-9c644f41dbc9-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-e1f08d-f5fb4c64f-mmvp6\" (UID: \"e60b0573-2ad3-40d9-898e-9c644f41dbc9\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-e1f08d-f5fb4c64f-mmvp6" Nov 03 01:14:24.627524 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:24.627500 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57js7\" (UniqueName: \"kubernetes.io/projected/e60b0573-2ad3-40d9-898e-9c644f41dbc9-kube-api-access-57js7\") pod \"payload-processing-e2e-extmod-b-e1f08d-f5fb4c64f-mmvp6\" (UID: \"e60b0573-2ad3-40d9-898e-9c644f41dbc9\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-e1f08d-f5fb4c64f-mmvp6" Nov 03 01:14:24.668173 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:24.668134 4421 generic.go:356] "Generic (PLEG): container finished" podID="666777c0-fa6d-4f7b-b32c-5799477477af" containerID="efc5aaf72beeb404204da947498b1297a0429f4c389cbc0b1fd27147a23acc8c" exitCode=0 Nov 03 01:14:24.668428 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:24.668235 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-5779tv6bk" Nov 03 01:14:24.668428 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:24.668318 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-5779tv6bk" event={"ID":"666777c0-fa6d-4f7b-b32c-5799477477af","Type":"ContainerDied","Data":"efc5aaf72beeb404204da947498b1297a0429f4c389cbc0b1fd27147a23acc8c"} Nov 03 01:14:24.668428 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:24.668378 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-5779tv6bk" event={"ID":"666777c0-fa6d-4f7b-b32c-5799477477af","Type":"ContainerDied","Data":"379444f5f40bfdc983501d82084876e63df73ab72342fd6055d603070cab1388"} Nov 03 01:14:24.668428 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:24.668397 4421 scope.go:122] "RemoveContainer" containerID="efc5aaf72beeb404204da947498b1297a0429f4c389cbc0b1fd27147a23acc8c" Nov 03 01:14:24.701864 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:24.701827 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-5779tv6bk"] Nov 03 01:14:24.707829 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:24.707798 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-5779tv6bk"] Nov 03 01:14:24.710710 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:24.710605 4421 scope.go:122] "RemoveContainer" containerID="efc5aaf72beeb404204da947498b1297a0429f4c389cbc0b1fd27147a23acc8c" Nov 03 01:14:24.711234 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:14:24.711159 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efc5aaf72beeb404204da947498b1297a0429f4c389cbc0b1fd27147a23acc8c\": container with ID starting with efc5aaf72beeb404204da947498b1297a0429f4c389cbc0b1fd27147a23acc8c not found: ID does not exist" containerID="efc5aaf72beeb404204da947498b1297a0429f4c389cbc0b1fd27147a23acc8c" Nov 03 01:14:24.711234 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:24.711194 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efc5aaf72beeb404204da947498b1297a0429f4c389cbc0b1fd27147a23acc8c"} err="failed to get container status \"efc5aaf72beeb404204da947498b1297a0429f4c389cbc0b1fd27147a23acc8c\": rpc error: code = NotFound desc = could not find container \"efc5aaf72beeb404204da947498b1297a0429f4c389cbc0b1fd27147a23acc8c\": container with ID starting with efc5aaf72beeb404204da947498b1297a0429f4c389cbc0b1fd27147a23acc8c not found: ID does not exist" Nov 03 01:14:24.749760 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:24.749327 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-e1f08d-f5fb4c64f-mmvp6" Nov 03 01:14:25.367709 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:25.367676 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-55d6nfg5v" Nov 03 01:14:25.418008 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:25.409229 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-78c872f4s"] Nov 03 01:14:25.419327 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:25.419249 4421 cpu_manager.go:414] "containerMap: removing container" podUID="31e81deb-0e7e-4c26-b43f-870959edffe8" containerName="payload-processing" Nov 03 01:14:25.419327 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:25.419295 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31e81deb-0e7e-4c26-b43f-870959edffe8" containerName="payload-processing" Nov 03 01:14:25.419573 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:25.419522 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="31e81deb-0e7e-4c26-b43f-870959edffe8" containerName="payload-processing" Nov 03 01:14:25.424490 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:25.424459 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-78c872f4s"] Nov 03 01:14:25.424632 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:25.424588 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-78c872f4s" Nov 03 01:14:25.429053 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:25.429024 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-3c5983\"/\"agc-cred-803162a56fb2\"" Nov 03 01:14:25.429053 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:25.429041 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-3c5983\"/\"agc-cred-371bb2bc1d0f\"" Nov 03 01:14:25.429053 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:25.429035 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-3c5983\"/\"agc-cred-7fc5c1cab2ff\"" Nov 03 01:14:25.429449 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:25.429094 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-3c5983\"/\"payload-processing-external-model-e2e--ba2bb6eb-dockercfg-zkrj5\"" Nov 03 01:14:25.443433 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:25.443405 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-3c5983\"/\"payload-processing-external-model-plugins-e2e-extmod-b-3c5983\"" Nov 03 01:14:25.443568 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:25.443467 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-3c5983\"/\"openshift-service-ca.crt\"" Nov 03 01:14:25.443637 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:25.443565 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-3c5983\"/\"routing-overlay\"" Nov 03 01:14:25.443637 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:25.443582 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-3c5983\"/\"kube-root-ca.crt\"" Nov 03 01:14:25.456502 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:25.456480 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31e81deb-0e7e-4c26-b43f-870959edffe8-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/31e81deb-0e7e-4c26-b43f-870959edffe8-routing-overlay-volume\") pod \"31e81deb-0e7e-4c26-b43f-870959edffe8\" (UID: \"31e81deb-0e7e-4c26-b43f-870959edffe8\") " Nov 03 01:14:25.456627 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:25.456575 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31e81deb-0e7e-4c26-b43f-870959edffe8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31e81deb-0e7e-4c26-b43f-870959edffe8-plugins-config-volume\") pod \"31e81deb-0e7e-4c26-b43f-870959edffe8\" (UID: \"31e81deb-0e7e-4c26-b43f-870959edffe8\") " Nov 03 01:14:25.456753 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:25.456733 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31e81deb-0e7e-4c26-b43f-870959edffe8-provider-credentials\" (UniqueName: \"kubernetes.io/projected/31e81deb-0e7e-4c26-b43f-870959edffe8-provider-credentials\") pod \"31e81deb-0e7e-4c26-b43f-870959edffe8\" (UID: \"31e81deb-0e7e-4c26-b43f-870959edffe8\") " Nov 03 01:14:25.456903 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:25.456884 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31e81deb-0e7e-4c26-b43f-870959edffe8-routing-overlay-volume" pod "31e81deb-0e7e-4c26-b43f-870959edffe8" (UID: "31e81deb-0e7e-4c26-b43f-870959edffe8"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:14:25.456974 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:25.456955 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31e81deb-0e7e-4c26-b43f-870959edffe8-kube-api-access-gsmnf\" (UniqueName: \"kubernetes.io/projected/31e81deb-0e7e-4c26-b43f-870959edffe8-kube-api-access-gsmnf\") pod \"31e81deb-0e7e-4c26-b43f-870959edffe8\" (UID: \"31e81deb-0e7e-4c26-b43f-870959edffe8\") " Nov 03 01:14:25.457031 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:25.457011 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31e81deb-0e7e-4c26-b43f-870959edffe8-plugins-config-volume" pod "31e81deb-0e7e-4c26-b43f-870959edffe8" (UID: "31e81deb-0e7e-4c26-b43f-870959edffe8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:14:25.457834 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:25.457805 4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31e81deb-0e7e-4c26-b43f-870959edffe8-plugins-config-volume\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:14:25.457984 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:25.457839 4421 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/31e81deb-0e7e-4c26-b43f-870959edffe8-routing-overlay-volume\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:14:25.459396 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:25.459371 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31e81deb-0e7e-4c26-b43f-870959edffe8-provider-credentials" pod "31e81deb-0e7e-4c26-b43f-870959edffe8" (UID: "31e81deb-0e7e-4c26-b43f-870959edffe8"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:14:25.459493 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:25.459410 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31e81deb-0e7e-4c26-b43f-870959edffe8-kube-api-access-gsmnf" pod "31e81deb-0e7e-4c26-b43f-870959edffe8" (UID: "31e81deb-0e7e-4c26-b43f-870959edffe8"). InnerVolumeSpecName "kube-api-access-gsmnf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:14:25.502908 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:25.502879 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-e1f08d-f5fb4c64f-mmvp6"] Nov 03 01:14:25.586973 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:25.564732 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c78x9\" (UniqueName: \"kubernetes.io/projected/6386694a-6d4c-4a69-8388-52f34a3484d2-kube-api-access-c78x9\") pod \"payload-processing-external-model-e2e-extmod-b-3c5983-78c872f4s\" (UID: \"6386694a-6d4c-4a69-8388-52f34a3484d2\") " pod="ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-78c872f4s" Nov 03 01:14:25.586973 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:25.564864 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6386694a-6d4c-4a69-8388-52f34a3484d2-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3c5983-78c872f4s\" (UID: \"6386694a-6d4c-4a69-8388-52f34a3484d2\") " pod="ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-78c872f4s" Nov 03 01:14:25.586973 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:25.564912 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6386694a-6d4c-4a69-8388-52f34a3484d2-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-3c5983-78c872f4s\" (UID: \"6386694a-6d4c-4a69-8388-52f34a3484d2\") " pod="ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-78c872f4s" Nov 03 01:14:25.586973 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:25.565223 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6386694a-6d4c-4a69-8388-52f34a3484d2-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3c5983-78c872f4s\" (UID: \"6386694a-6d4c-4a69-8388-52f34a3484d2\") " pod="ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-78c872f4s" Nov 03 01:14:25.591724 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:25.587694 4421 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/31e81deb-0e7e-4c26-b43f-870959edffe8-provider-credentials\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:14:25.595319 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:25.587730 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gsmnf\" (UniqueName: \"kubernetes.io/projected/31e81deb-0e7e-4c26-b43f-870959edffe8-kube-api-access-gsmnf\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:14:25.681438 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:25.681404 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-e1f08d-f5fb4c64f-mmvp6" event={"ID":"e60b0573-2ad3-40d9-898e-9c644f41dbc9","Type":"ContainerStarted","Data":"10fdc578bbe9186f314c3b276a87a7bc0f70a128b08a8b765a3dd3f0816e7642"} Nov 03 01:14:25.682858 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:25.682832 4421 generic.go:356] "Generic (PLEG): container finished" podID="31e81deb-0e7e-4c26-b43f-870959edffe8" containerID="f9336e3115a14139ea618b94a58991993df29dd2e4466eef0d9432a13b292a20" exitCode=0 Nov 03 01:14:25.682977 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:25.682886 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-55d6nfg5v" Nov 03 01:14:25.682977 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:25.682915 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-55d6nfg5v" event={"ID":"31e81deb-0e7e-4c26-b43f-870959edffe8","Type":"ContainerDied","Data":"f9336e3115a14139ea618b94a58991993df29dd2e4466eef0d9432a13b292a20"} Nov 03 01:14:25.682977 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:25.682974 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-55d6nfg5v" event={"ID":"31e81deb-0e7e-4c26-b43f-870959edffe8","Type":"ContainerDied","Data":"c52238e5ac4a870c2f79cf0b2426bb80deb3c5c261cd44769cabd4b220caab76"} Nov 03 01:14:25.683114 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:25.682998 4421 scope.go:122] "RemoveContainer" containerID="f9336e3115a14139ea618b94a58991993df29dd2e4466eef0d9432a13b292a20" Nov 03 01:14:25.694580 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:25.694557 4421 scope.go:122] "RemoveContainer" containerID="f9336e3115a14139ea618b94a58991993df29dd2e4466eef0d9432a13b292a20" Nov 03 01:14:25.694848 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:14:25.694826 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9336e3115a14139ea618b94a58991993df29dd2e4466eef0d9432a13b292a20\": container with ID starting with f9336e3115a14139ea618b94a58991993df29dd2e4466eef0d9432a13b292a20 not found: ID does not exist" containerID="f9336e3115a14139ea618b94a58991993df29dd2e4466eef0d9432a13b292a20" Nov 03 01:14:25.694952 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:25.694855 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9336e3115a14139ea618b94a58991993df29dd2e4466eef0d9432a13b292a20"} err="failed to get container status \"f9336e3115a14139ea618b94a58991993df29dd2e4466eef0d9432a13b292a20\": rpc error: code = NotFound desc = could not find container \"f9336e3115a14139ea618b94a58991993df29dd2e4466eef0d9432a13b292a20\": container with ID starting with f9336e3115a14139ea618b94a58991993df29dd2e4466eef0d9432a13b292a20 not found: ID does not exist" Nov 03 01:14:25.696411 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:25.696387 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6386694a-6d4c-4a69-8388-52f34a3484d2-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-3c5983-78c872f4s\" (UID: \"6386694a-6d4c-4a69-8388-52f34a3484d2\") " pod="ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-78c872f4s" Nov 03 01:14:25.696622 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:25.696606 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6386694a-6d4c-4a69-8388-52f34a3484d2-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3c5983-78c872f4s\" (UID: \"6386694a-6d4c-4a69-8388-52f34a3484d2\") " pod="ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-78c872f4s" Nov 03 01:14:25.696706 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:25.696690 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c78x9\" (UniqueName: \"kubernetes.io/projected/6386694a-6d4c-4a69-8388-52f34a3484d2-kube-api-access-c78x9\") pod \"payload-processing-external-model-e2e-extmod-b-3c5983-78c872f4s\" (UID: \"6386694a-6d4c-4a69-8388-52f34a3484d2\") " pod="ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-78c872f4s" Nov 03 01:14:25.696774 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:25.696760 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6386694a-6d4c-4a69-8388-52f34a3484d2-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3c5983-78c872f4s\" (UID: \"6386694a-6d4c-4a69-8388-52f34a3484d2\") " pod="ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-78c872f4s" Nov 03 01:14:25.697500 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:25.697477 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6386694a-6d4c-4a69-8388-52f34a3484d2-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3c5983-78c872f4s\" (UID: \"6386694a-6d4c-4a69-8388-52f34a3484d2\") " pod="ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-78c872f4s" Nov 03 01:14:25.697580 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:25.697525 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6386694a-6d4c-4a69-8388-52f34a3484d2-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3c5983-78c872f4s\" (UID: \"6386694a-6d4c-4a69-8388-52f34a3484d2\") " pod="ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-78c872f4s" Nov 03 01:14:25.699257 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:25.699232 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6386694a-6d4c-4a69-8388-52f34a3484d2-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-3c5983-78c872f4s\" (UID: \"6386694a-6d4c-4a69-8388-52f34a3484d2\") " pod="ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-78c872f4s" Nov 03 01:14:25.707787 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:25.707756 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c78x9\" (UniqueName: \"kubernetes.io/projected/6386694a-6d4c-4a69-8388-52f34a3484d2-kube-api-access-c78x9\") pod \"payload-processing-external-model-e2e-extmod-b-3c5983-78c872f4s\" (UID: \"6386694a-6d4c-4a69-8388-52f34a3484d2\") " pod="ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-78c872f4s" Nov 03 01:14:25.710351 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:25.710327 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-55d6nfg5v"] Nov 03 01:14:25.715207 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:25.715183 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-55d6nfg5v"] Nov 03 01:14:25.740960 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:25.740932 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-78c872f4s" Nov 03 01:14:25.983053 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:25.982976 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31e81deb-0e7e-4c26-b43f-870959edffe8" path="/var/lib/kubelet/pods/31e81deb-0e7e-4c26-b43f-870959edffe8/volumes" Nov 03 01:14:25.983527 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:25.983511 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="666777c0-fa6d-4f7b-b32c-5799477477af" path="/var/lib/kubelet/pods/666777c0-fa6d-4f7b-b32c-5799477477af/volumes" Nov 03 01:14:26.029317 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:14:26.029263 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6386694a_6d4c_4a69_8388_52f34a3484d2.slice/crio-cd33dc082959a7b63d830f265e870c53035554a14b5a464c521655419b5f28c1 WatchSource:0}: Error finding container cd33dc082959a7b63d830f265e870c53035554a14b5a464c521655419b5f28c1: Status 404 returned error can't find the container with id cd33dc082959a7b63d830f265e870c53035554a14b5a464c521655419b5f28c1 Nov 03 01:14:26.031073 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:26.031042 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-78c872f4s"] Nov 03 01:14:26.692703 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:26.692659 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-78c872f4s" event={"ID":"6386694a-6d4c-4a69-8388-52f34a3484d2","Type":"ContainerStarted","Data":"4eb085a159c0f13628c7c239b1185a0a65c96f79a44a1a91fe0f133a69e5b85f"} Nov 03 01:14:26.693232 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:26.692725 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-78c872f4s" event={"ID":"6386694a-6d4c-4a69-8388-52f34a3484d2","Type":"ContainerStarted","Data":"cd33dc082959a7b63d830f265e870c53035554a14b5a464c521655419b5f28c1"} Nov 03 01:14:26.693232 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:26.692754 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-78c872f4s" Nov 03 01:14:26.694607 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:26.694578 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-e1f08d-f5fb4c64f-mmvp6" event={"ID":"e60b0573-2ad3-40d9-898e-9c644f41dbc9","Type":"ContainerStarted","Data":"4f2d454b26668b3c077e1a50e676853a63d03300af6ff64682ea5dd916a39fae"} Nov 03 01:14:26.694732 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:26.694649 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-e1f08d-f5fb4c64f-mmvp6" Nov 03 01:14:26.713202 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:26.713143 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-78c872f4s" podStartSLOduration=9.713129047 podStartE2EDuration="9.713129047s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:14:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:14:26.708936246 +0000 UTC m=+2008.879939354" watchObservedRunningTime="2026-11-03 01:14:26.713129047 +0000 UTC m=+2008.884132142" Nov 03 01:14:26.728677 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:26.728625 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-e1f08d-f5fb4c64f-mmvp6" podStartSLOduration=14.728611761 podStartE2EDuration="14.728611761s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:14:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:14:26.725404352 +0000 UTC m=+2008.896407449" watchObservedRunningTime="2026-11-03 01:14:26.728611761 +0000 UTC m=+2008.899614856" Nov 03 01:14:30.849854 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:30.849818 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dqlwj"] Nov 03 01:14:30.850462 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:30.850060 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dqlwj" podUID="52eafa3f-6965-4775-9394-013e1bf9c157" containerName="istio-proxy" containerID="cri-o://a971182eccbea82a0f2ffb674b6846ca3497101e4b4aeee3015b0c0d36ff706b" gracePeriod=30 Nov 03 01:14:36.169510 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:36.169478 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dqlwj" Nov 03 01:14:36.216905 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:36.216832 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-e1f08d-67989d5bdc-8k4qm"] Nov 03 01:14:36.220699 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:36.220598 4421 cpu_manager.go:414] "containerMap: removing container" podUID="52eafa3f-6965-4775-9394-013e1bf9c157" containerName="istio-proxy" Nov 03 01:14:36.220699 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:36.220627 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52eafa3f-6965-4775-9394-013e1bf9c157" containerName="istio-proxy" Nov 03 01:14:36.221788 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:36.221161 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="52eafa3f-6965-4775-9394-013e1bf9c157" containerName="istio-proxy" Nov 03 01:14:36.224068 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:36.223858 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52eafa3f-6965-4775-9394-013e1bf9c157-kube-api-access-qlxqf\" (UniqueName: \"kubernetes.io/projected/52eafa3f-6965-4775-9394-013e1bf9c157-kube-api-access-qlxqf\") pod \"52eafa3f-6965-4775-9394-013e1bf9c157\" (UID: \"52eafa3f-6965-4775-9394-013e1bf9c157\") " Nov 03 01:14:36.225257 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:36.224947 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52eafa3f-6965-4775-9394-013e1bf9c157-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52eafa3f-6965-4775-9394-013e1bf9c157-istiod-ca-cert\") pod \"52eafa3f-6965-4775-9394-013e1bf9c157\" (UID: \"52eafa3f-6965-4775-9394-013e1bf9c157\") " Nov 03 01:14:36.225257 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:36.225038 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52eafa3f-6965-4775-9394-013e1bf9c157-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52eafa3f-6965-4775-9394-013e1bf9c157-workload-socket\") pod \"52eafa3f-6965-4775-9394-013e1bf9c157\" (UID: \"52eafa3f-6965-4775-9394-013e1bf9c157\") " Nov 03 01:14:36.225257 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:36.225161 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52eafa3f-6965-4775-9394-013e1bf9c157-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52eafa3f-6965-4775-9394-013e1bf9c157-credential-socket\") pod \"52eafa3f-6965-4775-9394-013e1bf9c157\" (UID: \"52eafa3f-6965-4775-9394-013e1bf9c157\") " Nov 03 01:14:36.225257 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:36.225201 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/52eafa3f-6965-4775-9394-013e1bf9c157-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52eafa3f-6965-4775-9394-013e1bf9c157-istio-podinfo\") pod \"52eafa3f-6965-4775-9394-013e1bf9c157\" (UID: \"52eafa3f-6965-4775-9394-013e1bf9c157\") " Nov 03 01:14:36.226971 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:36.225551 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52eafa3f-6965-4775-9394-013e1bf9c157-istiod-ca-cert" pod "52eafa3f-6965-4775-9394-013e1bf9c157" (UID: "52eafa3f-6965-4775-9394-013e1bf9c157"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:14:36.226971 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:36.225602 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52eafa3f-6965-4775-9394-013e1bf9c157-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52eafa3f-6965-4775-9394-013e1bf9c157-istio-envoy\") pod \"52eafa3f-6965-4775-9394-013e1bf9c157\" (UID: \"52eafa3f-6965-4775-9394-013e1bf9c157\") " Nov 03 01:14:36.226971 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:36.225737 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52eafa3f-6965-4775-9394-013e1bf9c157-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52eafa3f-6965-4775-9394-013e1bf9c157-istio-data\") pod \"52eafa3f-6965-4775-9394-013e1bf9c157\" (UID: \"52eafa3f-6965-4775-9394-013e1bf9c157\") " Nov 03 01:14:36.226971 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:36.225770 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52eafa3f-6965-4775-9394-013e1bf9c157-istio-token\" (UniqueName: \"kubernetes.io/projected/52eafa3f-6965-4775-9394-013e1bf9c157-istio-token\") pod \"52eafa3f-6965-4775-9394-013e1bf9c157\" (UID: \"52eafa3f-6965-4775-9394-013e1bf9c157\") " Nov 03 01:14:36.226971 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:36.225838 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52eafa3f-6965-4775-9394-013e1bf9c157-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52eafa3f-6965-4775-9394-013e1bf9c157-workload-certs\") pod \"52eafa3f-6965-4775-9394-013e1bf9c157\" (UID: \"52eafa3f-6965-4775-9394-013e1bf9c157\") " Nov 03 01:14:36.226971 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:36.225895 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52eafa3f-6965-4775-9394-013e1bf9c157-workload-socket" pod "52eafa3f-6965-4775-9394-013e1bf9c157" (UID: "52eafa3f-6965-4775-9394-013e1bf9c157"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:14:36.226971 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:36.226753 4421 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52eafa3f-6965-4775-9394-013e1bf9c157-istiod-ca-cert\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:14:36.226971 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:36.226776 4421 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52eafa3f-6965-4775-9394-013e1bf9c157-workload-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:14:36.227552 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:36.227477 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52eafa3f-6965-4775-9394-013e1bf9c157-istio-data" pod "52eafa3f-6965-4775-9394-013e1bf9c157" (UID: "52eafa3f-6965-4775-9394-013e1bf9c157"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:14:36.227687 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:36.227660 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52eafa3f-6965-4775-9394-013e1bf9c157-credential-socket" pod "52eafa3f-6965-4775-9394-013e1bf9c157" (UID: "52eafa3f-6965-4775-9394-013e1bf9c157"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:14:36.227823 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:36.227800 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52eafa3f-6965-4775-9394-013e1bf9c157-workload-certs" pod "52eafa3f-6965-4775-9394-013e1bf9c157" (UID: "52eafa3f-6965-4775-9394-013e1bf9c157"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:14:36.229943 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:36.229041 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-e1f08d-67989d5bdc-8k4qm"] Nov 03 01:14:36.229943 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:36.229149 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e1f08d-67989d5bdc-8k4qm" Nov 03 01:14:36.229943 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:36.229833 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52eafa3f-6965-4775-9394-013e1bf9c157-kube-api-access-qlxqf" pod "52eafa3f-6965-4775-9394-013e1bf9c157" (UID: "52eafa3f-6965-4775-9394-013e1bf9c157"). InnerVolumeSpecName "kube-api-access-qlxqf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:14:36.233424 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:36.233406 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-e1f08d-serving-cert\"" Nov 03 01:14:36.236684 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:36.236662 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52eafa3f-6965-4775-9394-013e1bf9c157-istio-token" pod "52eafa3f-6965-4775-9394-013e1bf9c157" (UID: "52eafa3f-6965-4775-9394-013e1bf9c157"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:14:36.236798 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:36.236661 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52eafa3f-6965-4775-9394-013e1bf9c157-istio-envoy" pod "52eafa3f-6965-4775-9394-013e1bf9c157" (UID: "52eafa3f-6965-4775-9394-013e1bf9c157"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:14:36.238045 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:36.238020 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/52eafa3f-6965-4775-9394-013e1bf9c157-istio-podinfo" pod "52eafa3f-6965-4775-9394-013e1bf9c157" (UID: "52eafa3f-6965-4775-9394-013e1bf9c157"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:14:36.328677 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:36.328637 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxq8q\" (UniqueName: \"kubernetes.io/projected/d748c793-a88a-43c7-ab53-1ff9242d53d3-kube-api-access-xxq8q\") pod \"maas-api-e2e-extmod-b-e1f08d-67989d5bdc-8k4qm\" (UID: \"d748c793-a88a-43c7-ab53-1ff9242d53d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e1f08d-67989d5bdc-8k4qm" Nov 03 01:14:36.328863 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:36.328761 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d748c793-a88a-43c7-ab53-1ff9242d53d3-metrics-certs\") pod \"maas-api-e2e-extmod-b-e1f08d-67989d5bdc-8k4qm\" (UID: \"d748c793-a88a-43c7-ab53-1ff9242d53d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e1f08d-67989d5bdc-8k4qm" Nov 03 01:14:36.328917 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:36.328898 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d748c793-a88a-43c7-ab53-1ff9242d53d3-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-e1f08d-67989d5bdc-8k4qm\" (UID: \"d748c793-a88a-43c7-ab53-1ff9242d53d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e1f08d-67989d5bdc-8k4qm" Nov 03 01:14:36.328977 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:36.328935 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d748c793-a88a-43c7-ab53-1ff9242d53d3-maas-api-tls\") pod \"maas-api-e2e-extmod-b-e1f08d-67989d5bdc-8k4qm\" (UID: \"d748c793-a88a-43c7-ab53-1ff9242d53d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e1f08d-67989d5bdc-8k4qm" Nov 03 01:14:36.329213 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:36.329193 4421 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52eafa3f-6965-4775-9394-013e1bf9c157-credential-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:14:36.329337 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:36.329218 4421 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52eafa3f-6965-4775-9394-013e1bf9c157-istio-podinfo\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:14:36.329337 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:36.329233 4421 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52eafa3f-6965-4775-9394-013e1bf9c157-istio-envoy\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:14:36.329337 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:36.329245 4421 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52eafa3f-6965-4775-9394-013e1bf9c157-istio-data\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:14:36.329337 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:36.329258 4421 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52eafa3f-6965-4775-9394-013e1bf9c157-istio-token\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:14:36.329337 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:36.329287 4421 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52eafa3f-6965-4775-9394-013e1bf9c157-workload-certs\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:14:36.329337 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:36.329301 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qlxqf\" (UniqueName: \"kubernetes.io/projected/52eafa3f-6965-4775-9394-013e1bf9c157-kube-api-access-qlxqf\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:14:36.431317 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:36.431223 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxq8q\" (UniqueName: \"kubernetes.io/projected/d748c793-a88a-43c7-ab53-1ff9242d53d3-kube-api-access-xxq8q\") pod \"maas-api-e2e-extmod-b-e1f08d-67989d5bdc-8k4qm\" (UID: \"d748c793-a88a-43c7-ab53-1ff9242d53d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e1f08d-67989d5bdc-8k4qm" Nov 03 01:14:36.431317 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:36.431299 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d748c793-a88a-43c7-ab53-1ff9242d53d3-metrics-certs\") pod \"maas-api-e2e-extmod-b-e1f08d-67989d5bdc-8k4qm\" (UID: \"d748c793-a88a-43c7-ab53-1ff9242d53d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e1f08d-67989d5bdc-8k4qm" Nov 03 01:14:36.431546 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:36.431407 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d748c793-a88a-43c7-ab53-1ff9242d53d3-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-e1f08d-67989d5bdc-8k4qm\" (UID: \"d748c793-a88a-43c7-ab53-1ff9242d53d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e1f08d-67989d5bdc-8k4qm" Nov 03 01:14:36.431546 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:36.431449 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d748c793-a88a-43c7-ab53-1ff9242d53d3-maas-api-tls\") pod \"maas-api-e2e-extmod-b-e1f08d-67989d5bdc-8k4qm\" (UID: \"d748c793-a88a-43c7-ab53-1ff9242d53d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e1f08d-67989d5bdc-8k4qm" Nov 03 01:14:36.432125 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:36.432101 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d748c793-a88a-43c7-ab53-1ff9242d53d3-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-e1f08d-67989d5bdc-8k4qm\" (UID: \"d748c793-a88a-43c7-ab53-1ff9242d53d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e1f08d-67989d5bdc-8k4qm" Nov 03 01:14:36.433852 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:36.433829 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d748c793-a88a-43c7-ab53-1ff9242d53d3-maas-api-tls\") pod \"maas-api-e2e-extmod-b-e1f08d-67989d5bdc-8k4qm\" (UID: \"d748c793-a88a-43c7-ab53-1ff9242d53d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e1f08d-67989d5bdc-8k4qm" Nov 03 01:14:36.433951 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:36.433894 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d748c793-a88a-43c7-ab53-1ff9242d53d3-metrics-certs\") pod \"maas-api-e2e-extmod-b-e1f08d-67989d5bdc-8k4qm\" (UID: \"d748c793-a88a-43c7-ab53-1ff9242d53d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e1f08d-67989d5bdc-8k4qm" Nov 03 01:14:36.438689 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:36.438665 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxq8q\" (UniqueName: \"kubernetes.io/projected/d748c793-a88a-43c7-ab53-1ff9242d53d3-kube-api-access-xxq8q\") pod \"maas-api-e2e-extmod-b-e1f08d-67989d5bdc-8k4qm\" (UID: \"d748c793-a88a-43c7-ab53-1ff9242d53d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e1f08d-67989d5bdc-8k4qm" Nov 03 01:14:36.606234 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:36.606198 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e1f08d-67989d5bdc-8k4qm" Nov 03 01:14:36.854192 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:36.854155 4421 generic.go:356] "Generic (PLEG): container finished" podID="52eafa3f-6965-4775-9394-013e1bf9c157" containerID="a971182eccbea82a0f2ffb674b6846ca3497101e4b4aeee3015b0c0d36ff706b" exitCode=0 Nov 03 01:14:36.854441 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:36.854245 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dqlwj" Nov 03 01:14:36.854441 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:36.854248 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dqlwj" event={"ID":"52eafa3f-6965-4775-9394-013e1bf9c157","Type":"ContainerDied","Data":"a971182eccbea82a0f2ffb674b6846ca3497101e4b4aeee3015b0c0d36ff706b"} Nov 03 01:14:36.854441 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:36.854438 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dqlwj" event={"ID":"52eafa3f-6965-4775-9394-013e1bf9c157","Type":"ContainerDied","Data":"7245bcd7ef71be82b5a5561cd7004ed9ff60b70a5289a66effe689a870883a82"} Nov 03 01:14:36.854629 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:36.854466 4421 scope.go:122] "RemoveContainer" containerID="a971182eccbea82a0f2ffb674b6846ca3497101e4b4aeee3015b0c0d36ff706b" Nov 03 01:14:36.867241 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:36.867218 4421 scope.go:122] "RemoveContainer" containerID="a971182eccbea82a0f2ffb674b6846ca3497101e4b4aeee3015b0c0d36ff706b" Nov 03 01:14:36.867608 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:14:36.867586 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a971182eccbea82a0f2ffb674b6846ca3497101e4b4aeee3015b0c0d36ff706b\": container with ID starting with a971182eccbea82a0f2ffb674b6846ca3497101e4b4aeee3015b0c0d36ff706b not found: ID does not exist" containerID="a971182eccbea82a0f2ffb674b6846ca3497101e4b4aeee3015b0c0d36ff706b" Nov 03 01:14:36.867690 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:36.867620 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a971182eccbea82a0f2ffb674b6846ca3497101e4b4aeee3015b0c0d36ff706b"} err="failed to get container status \"a971182eccbea82a0f2ffb674b6846ca3497101e4b4aeee3015b0c0d36ff706b\": rpc error: code = NotFound desc = could not find container \"a971182eccbea82a0f2ffb674b6846ca3497101e4b4aeee3015b0c0d36ff706b\": container with ID starting with a971182eccbea82a0f2ffb674b6846ca3497101e4b4aeee3015b0c0d36ff706b not found: ID does not exist" Nov 03 01:14:36.882740 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:36.882709 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dqlwj"] Nov 03 01:14:36.888065 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:36.888043 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dqlwj"] Nov 03 01:14:36.902811 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:14:36.902786 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd748c793_a88a_43c7_ab53_1ff9242d53d3.slice/crio-89addcab389e6a6abb0b5ed4b46dc53f21313014ec573933307d96d729e3d600 WatchSource:0}: Error finding container 89addcab389e6a6abb0b5ed4b46dc53f21313014ec573933307d96d729e3d600: Status 404 returned error can't find the container with id 89addcab389e6a6abb0b5ed4b46dc53f21313014ec573933307d96d729e3d600 Nov 03 01:14:36.903353 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:36.903293 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-e1f08d-67989d5bdc-8k4qm"] Nov 03 01:14:37.744471 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:37.744389 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-e1f08d-f5fb4c64f-mmvp6" Nov 03 01:14:37.744869 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:37.744818 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-78c872f4s" Nov 03 01:14:37.867159 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:37.867128 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e1f08d-67989d5bdc-8k4qm" event={"ID":"d748c793-a88a-43c7-ab53-1ff9242d53d3","Type":"ContainerStarted","Data":"fd49fb070443f3e2208b05f6999c758dde0ff220265133572835ec4b2e935692"} Nov 03 01:14:37.867412 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:37.867185 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e1f08d-67989d5bdc-8k4qm" event={"ID":"d748c793-a88a-43c7-ab53-1ff9242d53d3","Type":"ContainerStarted","Data":"89addcab389e6a6abb0b5ed4b46dc53f21313014ec573933307d96d729e3d600"} Nov 03 01:14:37.867412 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:37.867225 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e1f08d-67989d5bdc-8k4qm" Nov 03 01:14:37.890705 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:37.890640 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e1f08d-67989d5bdc-8k4qm" podStartSLOduration=6.404787996 podStartE2EDuration="6.890626342s" totalImagesPullingTime="485.838346ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:14:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:14:37.888017906 +0000 UTC m=+2020.059021018" watchObservedRunningTime="2026-11-03 01:14:37.890626342 +0000 UTC m=+2020.061629437" Nov 03 01:14:37.982256 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:37.982226 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52eafa3f-6965-4775-9394-013e1bf9c157" path="/var/lib/kubelet/pods/52eafa3f-6965-4775-9394-013e1bf9c157/volumes" Nov 03 01:14:43.885097 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:43.885015 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e1f08d-67989d5bdc-8k4qm" Nov 03 01:14:44.674599 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:14:44.674548 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 01:14:49.003379 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:49.003347 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-56df2l9sl"] Nov 03 01:14:49.021486 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:49.021452 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-6d9bxhzmz"] Nov 03 01:14:49.049627 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:49.049578 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e51c8244-m4r72"] Nov 03 01:14:49.060346 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:49.060307 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e51c8244-m4r72"] Nov 03 01:14:49.983534 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:49.983498 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f17732b-cea6-4c56-9f69-4796e6c86ed0" path="/var/lib/kubelet/pods/5f17732b-cea6-4c56-9f69-4796e6c86ed0/volumes" Nov 03 01:14:54.004037 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:54.003992 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-56df2l9sl" podUID="e6622a05-c65e-4d3e-bd0f-286f852413bd" containerName="payload-processing" containerID="cri-o://183ab6f6737edb534ee318667050da7b3d80a2221b7748b4a38ccaa3eab053d0" gracePeriod=25 Nov 03 01:14:54.022600 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:54.022528 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-6d9bxhzmz" podUID="8e9e1695-dbf3-4d60-8121-2321028e461b" containerName="payload-processing" containerID="cri-o://c7aa980a92da77cd0f941bb008ae5f28e5423e7c557e441620a6d2cf02a91089" gracePeriod=25 Nov 03 01:14:54.429748 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:54.429715 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-56df2l9sl" Nov 03 01:14:54.447135 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:54.447108 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-6d9bxhzmz" Nov 03 01:14:54.486248 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:54.486217 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-mxvfl"] Nov 03 01:14:54.487441 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:54.487410 4421 cpu_manager.go:414] "containerMap: removing container" podUID="e6622a05-c65e-4d3e-bd0f-286f852413bd" containerName="payload-processing" Nov 03 01:14:54.487441 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:54.487438 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6622a05-c65e-4d3e-bd0f-286f852413bd" containerName="payload-processing" Nov 03 01:14:54.487703 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:54.487463 4421 cpu_manager.go:414] "containerMap: removing container" podUID="8e9e1695-dbf3-4d60-8121-2321028e461b" containerName="payload-processing" Nov 03 01:14:54.487703 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:54.487471 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e9e1695-dbf3-4d60-8121-2321028e461b" containerName="payload-processing" Nov 03 01:14:54.487703 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:54.487694 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6622a05-c65e-4d3e-bd0f-286f852413bd" containerName="payload-processing" Nov 03 01:14:54.487905 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:54.487710 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e9e1695-dbf3-4d60-8121-2321028e461b" containerName="payload-processing" Nov 03 01:14:54.492260 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:54.492234 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-mxvfl" Nov 03 01:14:54.499611 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:54.499583 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-d25759-openshift-default-dockercfg-8fmhw\"" Nov 03 01:14:54.503803 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:54.503769 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6622a05-c65e-4d3e-bd0f-286f852413bd-provider-credentials\" (UniqueName: \"kubernetes.io/projected/e6622a05-c65e-4d3e-bd0f-286f852413bd-provider-credentials\") pod \"e6622a05-c65e-4d3e-bd0f-286f852413bd\" (UID: \"e6622a05-c65e-4d3e-bd0f-286f852413bd\") " Nov 03 01:14:54.504061 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:54.504038 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6622a05-c65e-4d3e-bd0f-286f852413bd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6622a05-c65e-4d3e-bd0f-286f852413bd-plugins-config-volume\") pod \"e6622a05-c65e-4d3e-bd0f-286f852413bd\" (UID: \"e6622a05-c65e-4d3e-bd0f-286f852413bd\") " Nov 03 01:14:54.504171 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:54.504137 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6622a05-c65e-4d3e-bd0f-286f852413bd-kube-api-access-kl9bw\" (UniqueName: \"kubernetes.io/projected/e6622a05-c65e-4d3e-bd0f-286f852413bd-kube-api-access-kl9bw\") pod \"e6622a05-c65e-4d3e-bd0f-286f852413bd\" (UID: \"e6622a05-c65e-4d3e-bd0f-286f852413bd\") " Nov 03 01:14:54.504244 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:54.504181 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e9e1695-dbf3-4d60-8121-2321028e461b-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8e9e1695-dbf3-4d60-8121-2321028e461b-routing-overlay-volume\") pod \"8e9e1695-dbf3-4d60-8121-2321028e461b\" (UID: \"8e9e1695-dbf3-4d60-8121-2321028e461b\") " Nov 03 01:14:54.504244 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:54.504225 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e9e1695-dbf3-4d60-8121-2321028e461b-kube-api-access-z5tln\" (UniqueName: \"kubernetes.io/projected/8e9e1695-dbf3-4d60-8121-2321028e461b-kube-api-access-z5tln\") pod \"8e9e1695-dbf3-4d60-8121-2321028e461b\" (UID: \"8e9e1695-dbf3-4d60-8121-2321028e461b\") " Nov 03 01:14:54.504444 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:54.504287 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e9e1695-dbf3-4d60-8121-2321028e461b-provider-credentials\" (UniqueName: \"kubernetes.io/projected/8e9e1695-dbf3-4d60-8121-2321028e461b-provider-credentials\") pod \"8e9e1695-dbf3-4d60-8121-2321028e461b\" (UID: \"8e9e1695-dbf3-4d60-8121-2321028e461b\") " Nov 03 01:14:54.504444 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:54.504387 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6622a05-c65e-4d3e-bd0f-286f852413bd-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e6622a05-c65e-4d3e-bd0f-286f852413bd-routing-overlay-volume\") pod \"e6622a05-c65e-4d3e-bd0f-286f852413bd\" (UID: \"e6622a05-c65e-4d3e-bd0f-286f852413bd\") " Nov 03 01:14:54.504444 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:54.504429 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e9e1695-dbf3-4d60-8121-2321028e461b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e9e1695-dbf3-4d60-8121-2321028e461b-plugins-config-volume\") pod \"8e9e1695-dbf3-4d60-8121-2321028e461b\" (UID: \"8e9e1695-dbf3-4d60-8121-2321028e461b\") " Nov 03 01:14:54.505056 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:54.505032 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6622a05-c65e-4d3e-bd0f-286f852413bd-plugins-config-volume" pod "e6622a05-c65e-4d3e-bd0f-286f852413bd" (UID: "e6622a05-c65e-4d3e-bd0f-286f852413bd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:14:54.505531 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:54.505506 4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6622a05-c65e-4d3e-bd0f-286f852413bd-plugins-config-volume\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:14:54.505629 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:54.505531 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e9e1695-dbf3-4d60-8121-2321028e461b-plugins-config-volume" pod "8e9e1695-dbf3-4d60-8121-2321028e461b" (UID: "8e9e1695-dbf3-4d60-8121-2321028e461b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:14:54.507787 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:54.507507 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6622a05-c65e-4d3e-bd0f-286f852413bd-routing-overlay-volume" pod "e6622a05-c65e-4d3e-bd0f-286f852413bd" (UID: "e6622a05-c65e-4d3e-bd0f-286f852413bd"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:14:54.508661 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:54.508629 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e9e1695-dbf3-4d60-8121-2321028e461b-routing-overlay-volume" pod "8e9e1695-dbf3-4d60-8121-2321028e461b" (UID: "8e9e1695-dbf3-4d60-8121-2321028e461b"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:14:54.508880 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:54.508849 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e9e1695-dbf3-4d60-8121-2321028e461b-provider-credentials" pod "8e9e1695-dbf3-4d60-8121-2321028e461b" (UID: "8e9e1695-dbf3-4d60-8121-2321028e461b"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:14:54.510366 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:54.510336 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6622a05-c65e-4d3e-bd0f-286f852413bd-provider-credentials" pod "e6622a05-c65e-4d3e-bd0f-286f852413bd" (UID: "e6622a05-c65e-4d3e-bd0f-286f852413bd"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:14:54.530807 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:54.530719 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-mxvfl"] Nov 03 01:14:54.536030 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:54.535997 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e9e1695-dbf3-4d60-8121-2321028e461b-kube-api-access-z5tln" pod "8e9e1695-dbf3-4d60-8121-2321028e461b" (UID: "8e9e1695-dbf3-4d60-8121-2321028e461b"). InnerVolumeSpecName "kube-api-access-z5tln". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:14:54.538089 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:54.538063 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6622a05-c65e-4d3e-bd0f-286f852413bd-kube-api-access-kl9bw" pod "e6622a05-c65e-4d3e-bd0f-286f852413bd" (UID: "e6622a05-c65e-4d3e-bd0f-286f852413bd"). InnerVolumeSpecName "kube-api-access-kl9bw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:14:54.607646 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:54.607601 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a1e781a-b4bc-4e84-85ef-18b48bf77f99-istio-envoy\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-mxvfl\" (UID: \"5a1e781a-b4bc-4e84-85ef-18b48bf77f99\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-mxvfl" Nov 03 01:14:54.607879 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:54.607773 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a1e781a-b4bc-4e84-85ef-18b48bf77f99-istio-token\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-mxvfl\" (UID: \"5a1e781a-b4bc-4e84-85ef-18b48bf77f99\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-mxvfl" Nov 03 01:14:54.607879 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:54.607815 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a1e781a-b4bc-4e84-85ef-18b48bf77f99-istio-podinfo\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-mxvfl\" (UID: \"5a1e781a-b4bc-4e84-85ef-18b48bf77f99\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-mxvfl" Nov 03 01:14:54.608027 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:54.607908 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a1e781a-b4bc-4e84-85ef-18b48bf77f99-credential-socket\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-mxvfl\" (UID: \"5a1e781a-b4bc-4e84-85ef-18b48bf77f99\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-mxvfl" Nov 03 01:14:54.608027 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:54.607975 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a1e781a-b4bc-4e84-85ef-18b48bf77f99-workload-certs\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-mxvfl\" (UID: \"5a1e781a-b4bc-4e84-85ef-18b48bf77f99\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-mxvfl" Nov 03 01:14:54.608151 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:54.608084 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcwwl\" (UniqueName: \"kubernetes.io/projected/5a1e781a-b4bc-4e84-85ef-18b48bf77f99-kube-api-access-kcwwl\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-mxvfl\" (UID: \"5a1e781a-b4bc-4e84-85ef-18b48bf77f99\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-mxvfl" Nov 03 01:14:54.608225 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:54.608186 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a1e781a-b4bc-4e84-85ef-18b48bf77f99-workload-socket\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-mxvfl\" (UID: \"5a1e781a-b4bc-4e84-85ef-18b48bf77f99\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-mxvfl" Nov 03 01:14:54.608225 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:54.608204 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a1e781a-b4bc-4e84-85ef-18b48bf77f99-istio-data\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-mxvfl\" (UID: \"5a1e781a-b4bc-4e84-85ef-18b48bf77f99\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-mxvfl" Nov 03 01:14:54.608368 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:54.608316 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a1e781a-b4bc-4e84-85ef-18b48bf77f99-istiod-ca-cert\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-mxvfl\" (UID: \"5a1e781a-b4bc-4e84-85ef-18b48bf77f99\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-mxvfl" Nov 03 01:14:54.608448 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:54.608433 4421 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e6622a05-c65e-4d3e-bd0f-286f852413bd-routing-overlay-volume\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:14:54.608517 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:54.608450 4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e9e1695-dbf3-4d60-8121-2321028e461b-plugins-config-volume\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:14:54.608517 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:54.608459 4421 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e6622a05-c65e-4d3e-bd0f-286f852413bd-provider-credentials\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:14:54.608517 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:54.608470 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kl9bw\" (UniqueName: \"kubernetes.io/projected/e6622a05-c65e-4d3e-bd0f-286f852413bd-kube-api-access-kl9bw\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:14:54.608517 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:54.608484 4421 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8e9e1695-dbf3-4d60-8121-2321028e461b-routing-overlay-volume\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:14:54.608517 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:54.608493 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5tln\" (UniqueName: \"kubernetes.io/projected/8e9e1695-dbf3-4d60-8121-2321028e461b-kube-api-access-z5tln\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:14:54.608517 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:54.608503 4421 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8e9e1695-dbf3-4d60-8121-2321028e461b-provider-credentials\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:14:54.710038 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:54.709989 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a1e781a-b4bc-4e84-85ef-18b48bf77f99-istio-token\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-mxvfl\" (UID: \"5a1e781a-b4bc-4e84-85ef-18b48bf77f99\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-mxvfl" Nov 03 01:14:54.710295 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:54.710047 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a1e781a-b4bc-4e84-85ef-18b48bf77f99-istio-podinfo\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-mxvfl\" (UID: \"5a1e781a-b4bc-4e84-85ef-18b48bf77f99\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-mxvfl" Nov 03 01:14:54.710295 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:54.710101 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a1e781a-b4bc-4e84-85ef-18b48bf77f99-credential-socket\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-mxvfl\" (UID: \"5a1e781a-b4bc-4e84-85ef-18b48bf77f99\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-mxvfl" Nov 03 01:14:54.710295 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:54.710202 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a1e781a-b4bc-4e84-85ef-18b48bf77f99-workload-certs\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-mxvfl\" (UID: \"5a1e781a-b4bc-4e84-85ef-18b48bf77f99\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-mxvfl" Nov 03 01:14:54.710493 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:54.710294 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcwwl\" (UniqueName: \"kubernetes.io/projected/5a1e781a-b4bc-4e84-85ef-18b48bf77f99-kube-api-access-kcwwl\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-mxvfl\" (UID: \"5a1e781a-b4bc-4e84-85ef-18b48bf77f99\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-mxvfl" Nov 03 01:14:54.710493 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:54.710393 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a1e781a-b4bc-4e84-85ef-18b48bf77f99-workload-socket\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-mxvfl\" (UID: \"5a1e781a-b4bc-4e84-85ef-18b48bf77f99\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-mxvfl" Nov 03 01:14:54.710493 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:54.710425 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a1e781a-b4bc-4e84-85ef-18b48bf77f99-istio-data\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-mxvfl\" (UID: \"5a1e781a-b4bc-4e84-85ef-18b48bf77f99\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-mxvfl" Nov 03 01:14:54.710663 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:54.710524 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a1e781a-b4bc-4e84-85ef-18b48bf77f99-istiod-ca-cert\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-mxvfl\" (UID: \"5a1e781a-b4bc-4e84-85ef-18b48bf77f99\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-mxvfl" Nov 03 01:14:54.710726 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:54.710663 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a1e781a-b4bc-4e84-85ef-18b48bf77f99-istio-envoy\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-mxvfl\" (UID: \"5a1e781a-b4bc-4e84-85ef-18b48bf77f99\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-mxvfl" Nov 03 01:14:54.710726 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:54.710669 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a1e781a-b4bc-4e84-85ef-18b48bf77f99-credential-socket\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-mxvfl\" (UID: \"5a1e781a-b4bc-4e84-85ef-18b48bf77f99\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-mxvfl" Nov 03 01:14:54.710726 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:54.710708 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a1e781a-b4bc-4e84-85ef-18b48bf77f99-workload-certs\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-mxvfl\" (UID: \"5a1e781a-b4bc-4e84-85ef-18b48bf77f99\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-mxvfl" Nov 03 01:14:54.710894 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:54.710746 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a1e781a-b4bc-4e84-85ef-18b48bf77f99-workload-socket\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-mxvfl\" (UID: \"5a1e781a-b4bc-4e84-85ef-18b48bf77f99\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-mxvfl" Nov 03 01:14:54.710961 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:54.710890 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a1e781a-b4bc-4e84-85ef-18b48bf77f99-istio-data\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-mxvfl\" (UID: \"5a1e781a-b4bc-4e84-85ef-18b48bf77f99\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-mxvfl" Nov 03 01:14:54.711287 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:54.711252 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a1e781a-b4bc-4e84-85ef-18b48bf77f99-istiod-ca-cert\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-mxvfl\" (UID: \"5a1e781a-b4bc-4e84-85ef-18b48bf77f99\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-mxvfl" Nov 03 01:14:54.712988 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:54.712952 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a1e781a-b4bc-4e84-85ef-18b48bf77f99-istio-envoy\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-mxvfl\" (UID: \"5a1e781a-b4bc-4e84-85ef-18b48bf77f99\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-mxvfl" Nov 03 01:14:54.713233 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:54.713216 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a1e781a-b4bc-4e84-85ef-18b48bf77f99-istio-podinfo\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-mxvfl\" (UID: \"5a1e781a-b4bc-4e84-85ef-18b48bf77f99\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-mxvfl" Nov 03 01:14:54.718627 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:54.718599 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a1e781a-b4bc-4e84-85ef-18b48bf77f99-istio-token\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-mxvfl\" (UID: \"5a1e781a-b4bc-4e84-85ef-18b48bf77f99\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-mxvfl" Nov 03 01:14:54.718728 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:54.718713 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcwwl\" (UniqueName: \"kubernetes.io/projected/5a1e781a-b4bc-4e84-85ef-18b48bf77f99-kube-api-access-kcwwl\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-mxvfl\" (UID: \"5a1e781a-b4bc-4e84-85ef-18b48bf77f99\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-mxvfl" Nov 03 01:14:54.822143 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:54.821784 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-mxvfl" Nov 03 01:14:55.096717 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:55.096625 4421 generic.go:356] "Generic (PLEG): container finished" podID="8e9e1695-dbf3-4d60-8121-2321028e461b" containerID="c7aa980a92da77cd0f941bb008ae5f28e5423e7c557e441620a6d2cf02a91089" exitCode=0 Nov 03 01:14:55.096717 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:55.096692 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-6d9bxhzmz" Nov 03 01:14:55.097327 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:55.096712 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-6d9bxhzmz" event={"ID":"8e9e1695-dbf3-4d60-8121-2321028e461b","Type":"ContainerDied","Data":"c7aa980a92da77cd0f941bb008ae5f28e5423e7c557e441620a6d2cf02a91089"} Nov 03 01:14:55.097327 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:55.096778 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-6d9bxhzmz" event={"ID":"8e9e1695-dbf3-4d60-8121-2321028e461b","Type":"ContainerDied","Data":"71fc91298d5d7497a5e22ad0b92695e8e23b72c2db3041f497be528bf84f3312"} Nov 03 01:14:55.097327 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:55.096801 4421 scope.go:122] "RemoveContainer" containerID="c7aa980a92da77cd0f941bb008ae5f28e5423e7c557e441620a6d2cf02a91089" Nov 03 01:14:55.098707 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:55.098487 4421 generic.go:356] "Generic (PLEG): container finished" podID="e6622a05-c65e-4d3e-bd0f-286f852413bd" containerID="183ab6f6737edb534ee318667050da7b3d80a2221b7748b4a38ccaa3eab053d0" exitCode=0 Nov 03 01:14:55.098707 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:55.098543 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-56df2l9sl" Nov 03 01:14:55.098707 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:55.098564 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-56df2l9sl" event={"ID":"e6622a05-c65e-4d3e-bd0f-286f852413bd","Type":"ContainerDied","Data":"183ab6f6737edb534ee318667050da7b3d80a2221b7748b4a38ccaa3eab053d0"} Nov 03 01:14:55.098707 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:55.098632 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-56df2l9sl" event={"ID":"e6622a05-c65e-4d3e-bd0f-286f852413bd","Type":"ContainerDied","Data":"dded4c7372167fd7993f553704d6a456f3b73d9581fd3745df3caff410786049"} Nov 03 01:14:55.111943 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:55.111916 4421 scope.go:122] "RemoveContainer" containerID="c7aa980a92da77cd0f941bb008ae5f28e5423e7c557e441620a6d2cf02a91089" Nov 03 01:14:55.112406 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:14:55.112379 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7aa980a92da77cd0f941bb008ae5f28e5423e7c557e441620a6d2cf02a91089\": container with ID starting with c7aa980a92da77cd0f941bb008ae5f28e5423e7c557e441620a6d2cf02a91089 not found: ID does not exist" containerID="c7aa980a92da77cd0f941bb008ae5f28e5423e7c557e441620a6d2cf02a91089" Nov 03 01:14:55.112565 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:55.112420 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7aa980a92da77cd0f941bb008ae5f28e5423e7c557e441620a6d2cf02a91089"} err="failed to get container status \"c7aa980a92da77cd0f941bb008ae5f28e5423e7c557e441620a6d2cf02a91089\": rpc error: code = NotFound desc = could not find container \"c7aa980a92da77cd0f941bb008ae5f28e5423e7c557e441620a6d2cf02a91089\": container with ID starting with c7aa980a92da77cd0f941bb008ae5f28e5423e7c557e441620a6d2cf02a91089 not found: ID does not exist" Nov 03 01:14:55.112565 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:55.112444 4421 scope.go:122] "RemoveContainer" containerID="183ab6f6737edb534ee318667050da7b3d80a2221b7748b4a38ccaa3eab053d0" Nov 03 01:14:55.125419 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:55.125396 4421 scope.go:122] "RemoveContainer" containerID="183ab6f6737edb534ee318667050da7b3d80a2221b7748b4a38ccaa3eab053d0" Nov 03 01:14:55.125745 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:14:55.125724 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"183ab6f6737edb534ee318667050da7b3d80a2221b7748b4a38ccaa3eab053d0\": container with ID starting with 183ab6f6737edb534ee318667050da7b3d80a2221b7748b4a38ccaa3eab053d0 not found: ID does not exist" containerID="183ab6f6737edb534ee318667050da7b3d80a2221b7748b4a38ccaa3eab053d0" Nov 03 01:14:55.125812 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:55.125758 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"183ab6f6737edb534ee318667050da7b3d80a2221b7748b4a38ccaa3eab053d0"} err="failed to get container status \"183ab6f6737edb534ee318667050da7b3d80a2221b7748b4a38ccaa3eab053d0\": rpc error: code = NotFound desc = could not find container \"183ab6f6737edb534ee318667050da7b3d80a2221b7748b4a38ccaa3eab053d0\": container with ID starting with 183ab6f6737edb534ee318667050da7b3d80a2221b7748b4a38ccaa3eab053d0 not found: ID does not exist" Nov 03 01:14:55.134557 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:55.134526 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-56df2l9sl"] Nov 03 01:14:55.139473 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:55.139450 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-56df2l9sl"] Nov 03 01:14:55.150085 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:55.150056 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-6d9bxhzmz"] Nov 03 01:14:55.154799 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:55.154762 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-6d9bxhzmz"] Nov 03 01:14:55.170654 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:14:55.170626 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a1e781a_b4bc_4e84_85ef_18b48bf77f99.slice/crio-93248c0a06bc1359bef6ef5cac03f2d17106b351d924265d56b6919069ca06cf WatchSource:0}: Error finding container 93248c0a06bc1359bef6ef5cac03f2d17106b351d924265d56b6919069ca06cf: Status 404 returned error can't find the container with id 93248c0a06bc1359bef6ef5cac03f2d17106b351d924265d56b6919069ca06cf Nov 03 01:14:55.171835 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:55.171816 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-mxvfl"] Nov 03 01:14:55.174111 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:55.174077 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:14:55.174211 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:55.174142 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:14:55.174211 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:55.174172 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:14:55.981765 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:55.981732 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e9e1695-dbf3-4d60-8121-2321028e461b" path="/var/lib/kubelet/pods/8e9e1695-dbf3-4d60-8121-2321028e461b/volumes" Nov 03 01:14:55.982325 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:55.982310 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6622a05-c65e-4d3e-bd0f-286f852413bd" path="/var/lib/kubelet/pods/e6622a05-c65e-4d3e-bd0f-286f852413bd/volumes" Nov 03 01:14:56.109490 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:56.109451 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-mxvfl" event={"ID":"5a1e781a-b4bc-4e84-85ef-18b48bf77f99","Type":"ContainerStarted","Data":"0f6d27237cee407e3c0c756d4a58ba7518d85cbc28093f1f3b69d9ebf144cf95"} Nov 03 01:14:56.109936 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:56.109522 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-mxvfl" event={"ID":"5a1e781a-b4bc-4e84-85ef-18b48bf77f99","Type":"ContainerStarted","Data":"93248c0a06bc1359bef6ef5cac03f2d17106b351d924265d56b6919069ca06cf"} Nov 03 01:14:56.143639 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:56.143580 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-mxvfl" podStartSLOduration=164.143559564 podStartE2EDuration="2m44.143559564s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:12:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:14:56.13019519 +0000 UTC m=+2038.301198300" watchObservedRunningTime="2026-11-03 01:14:56.143559564 +0000 UTC m=+2038.314562662" Nov 03 01:14:56.830227 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:56.829315 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-mxvfl" Nov 03 01:14:56.836308 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:56.834402 4421 patch_prober.go:29] interesting pod/e2e-extmod-a-d25759-openshift-default-d47fd9785-mxvfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.7:15021/healthz/ready\": dial tcp 10.134.1.7:15021: connect: connection refused" start-of-body= Nov 03 01:14:56.836308 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:56.834463 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-mxvfl" podUID="5a1e781a-b4bc-4e84-85ef-18b48bf77f99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.7:15021/healthz/ready\": dial tcp 10.134.1.7:15021: connect: connection refused" Nov 03 01:14:57.825302 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:57.823381 4421 patch_prober.go:29] interesting pod/e2e-extmod-a-d25759-openshift-default-d47fd9785-mxvfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.7:15021/healthz/ready\": dial tcp 10.134.1.7:15021: connect: connection refused" start-of-body= Nov 03 01:14:57.825302 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:57.823445 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-mxvfl" podUID="5a1e781a-b4bc-4e84-85ef-18b48bf77f99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.7:15021/healthz/ready\": dial tcp 10.134.1.7:15021: connect: connection refused" Nov 03 01:14:58.830297 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:58.827801 4421 patch_prober.go:29] interesting pod/e2e-extmod-a-d25759-openshift-default-d47fd9785-mxvfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.7:15021/healthz/ready\": dial tcp 10.134.1.7:15021: connect: connection refused" start-of-body= Nov 03 01:14:58.830297 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:58.827871 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-mxvfl" podUID="5a1e781a-b4bc-4e84-85ef-18b48bf77f99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.7:15021/healthz/ready\": dial tcp 10.134.1.7:15021: connect: connection refused" Nov 03 01:14:59.827300 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:59.826673 4421 patch_prober.go:29] interesting pod/e2e-extmod-a-d25759-openshift-default-d47fd9785-mxvfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.7:15021/healthz/ready\": dial tcp 10.134.1.7:15021: connect: connection refused" start-of-body= Nov 03 01:14:59.827300 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:14:59.826733 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-mxvfl" podUID="5a1e781a-b4bc-4e84-85ef-18b48bf77f99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.7:15021/healthz/ready\": dial tcp 10.134.1.7:15021: connect: connection refused" Nov 03 01:15:00.037139 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:00.037091 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29819d6d-h2qkg"] Nov 03 01:15:00.042008 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:00.041977 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29819d6d-h2qkg"] Nov 03 01:15:00.148970 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:00.148889 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-531447-29894475-zr4nt"] Nov 03 01:15:00.155396 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:00.155372 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-531447-29894475-zr4nt" Nov 03 01:15:00.170610 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:00.169592 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6q4xp\"" Nov 03 01:15:00.176703 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:00.176662 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-531447-29894475-zr4nt"] Nov 03 01:15:00.304014 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:00.303464 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd14e9bc-4127-471b-8502-dc36b4244615-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-531447-29894475-zr4nt\" (UID: \"dd14e9bc-4127-471b-8502-dc36b4244615\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-531447-29894475-zr4nt" Nov 03 01:15:00.304014 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:00.303539 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfb29\" (UniqueName: \"kubernetes.io/projected/dd14e9bc-4127-471b-8502-dc36b4244615-kube-api-access-zfb29\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-531447-29894475-zr4nt\" (UID: \"dd14e9bc-4127-471b-8502-dc36b4244615\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-531447-29894475-zr4nt" Nov 03 01:15:00.405198 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:00.405109 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd14e9bc-4127-471b-8502-dc36b4244615-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-531447-29894475-zr4nt\" (UID: \"dd14e9bc-4127-471b-8502-dc36b4244615\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-531447-29894475-zr4nt" Nov 03 01:15:00.405198 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:00.405163 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfb29\" (UniqueName: \"kubernetes.io/projected/dd14e9bc-4127-471b-8502-dc36b4244615-kube-api-access-zfb29\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-531447-29894475-zr4nt\" (UID: \"dd14e9bc-4127-471b-8502-dc36b4244615\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-531447-29894475-zr4nt" Nov 03 01:15:00.429309 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:00.429000 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd14e9bc-4127-471b-8502-dc36b4244615-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-531447-29894475-zr4nt\" (UID: \"dd14e9bc-4127-471b-8502-dc36b4244615\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-531447-29894475-zr4nt" Nov 03 01:15:00.433427 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:00.433373 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfb29\" (UniqueName: \"kubernetes.io/projected/dd14e9bc-4127-471b-8502-dc36b4244615-kube-api-access-zfb29\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-531447-29894475-zr4nt\" (UID: \"dd14e9bc-4127-471b-8502-dc36b4244615\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-531447-29894475-zr4nt" Nov 03 01:15:00.481302 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:00.480579 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-531447-29894475-zr4nt" Nov 03 01:15:00.834142 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:00.832843 4421 patch_prober.go:29] interesting pod/e2e-extmod-a-d25759-openshift-default-d47fd9785-mxvfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.7:15021/healthz/ready\": dial tcp 10.134.1.7:15021: connect: connection refused" start-of-body= Nov 03 01:15:00.834142 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:00.832910 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-mxvfl" podUID="5a1e781a-b4bc-4e84-85ef-18b48bf77f99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.7:15021/healthz/ready\": dial tcp 10.134.1.7:15021: connect: connection refused" Nov 03 01:15:00.886965 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:15:00.886928 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd14e9bc_4127_471b_8502_dc36b4244615.slice/crio-42ffcee8e556f7803b36364d605621a67311b5456c9dd814a2017fb1453c9498 WatchSource:0}: Error finding container 42ffcee8e556f7803b36364d605621a67311b5456c9dd814a2017fb1453c9498: Status 404 returned error can't find the container with id 42ffcee8e556f7803b36364d605621a67311b5456c9dd814a2017fb1453c9498 Nov 03 01:15:00.889315 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:00.889288 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-531447-29894475-zr4nt"] Nov 03 01:15:01.190954 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:01.190867 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-531447-29894475-zr4nt" event={"ID":"dd14e9bc-4127-471b-8502-dc36b4244615","Type":"ContainerStarted","Data":"42ffcee8e556f7803b36364d605621a67311b5456c9dd814a2017fb1453c9498"} Nov 03 01:15:01.830494 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:01.827431 4421 patch_prober.go:29] interesting pod/e2e-extmod-a-d25759-openshift-default-d47fd9785-mxvfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.7:15021/healthz/ready\": dial tcp 10.134.1.7:15021: connect: connection refused" start-of-body= Nov 03 01:15:01.830494 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:01.827494 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-mxvfl" podUID="5a1e781a-b4bc-4e84-85ef-18b48bf77f99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.7:15021/healthz/ready\": dial tcp 10.134.1.7:15021: connect: connection refused" Nov 03 01:15:02.052827 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:02.052797 4421 scope.go:122] "RemoveContainer" containerID="78336998ab42c5b0599ae431c6b2a95628958b43bd6326f96952bd56cc3d7caf" Nov 03 01:15:02.053529 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:02.053501 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52975b66-9b47-417e-b48f-f6d460fbbcd0" path="/var/lib/kubelet/pods/52975b66-9b47-417e-b48f-f6d460fbbcd0/volumes" Nov 03 01:15:02.064310 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:02.064250 4421 scope.go:122] "RemoveContainer" containerID="a8dac3bae2e9ab746c4fd02ee42a55b774238dfb69b22bca51f60ab1268d6be7" Nov 03 01:15:02.200554 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:02.200519 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-531447-29894475-zr4nt" event={"ID":"dd14e9bc-4127-471b-8502-dc36b4244615","Type":"ContainerStarted","Data":"88d98162d8db82eaf95a9da9f9d84996413ee518691f7fad2be3d1774e32ba96"} Nov 03 01:15:02.829647 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:02.829613 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-mxvfl" Nov 03 01:15:02.830123 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:02.830100 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-mxvfl" Nov 03 01:15:02.831003 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:02.830985 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-mxvfl" Nov 03 01:15:02.857147 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:02.857095 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-531447-29894475-zr4nt" podStartSLOduration=2.857081689 podStartE2EDuration="2.857081689s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:15:02.232298362 +0000 UTC m=+2044.403301459" watchObservedRunningTime="2026-11-03 01:15:02.857081689 +0000 UTC m=+2045.028084784" Nov 03 01:15:03.210380 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:03.210263 4421 generic.go:356] "Generic (PLEG): container finished" podID="dd14e9bc-4127-471b-8502-dc36b4244615" containerID="88d98162d8db82eaf95a9da9f9d84996413ee518691f7fad2be3d1774e32ba96" exitCode=60 Nov 03 01:15:03.210380 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:03.210349 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-531447-29894475-zr4nt" event={"ID":"dd14e9bc-4127-471b-8502-dc36b4244615","Type":"ContainerDied","Data":"88d98162d8db82eaf95a9da9f9d84996413ee518691f7fad2be3d1774e32ba96"} Nov 03 01:15:03.210912 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:03.210688 4421 scope.go:122] "RemoveContainer" containerID="88d98162d8db82eaf95a9da9f9d84996413ee518691f7fad2be3d1774e32ba96" Nov 03 01:15:04.226498 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:04.226458 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-531447-29894475-zr4nt" event={"ID":"dd14e9bc-4127-471b-8502-dc36b4244615","Type":"ContainerStarted","Data":"d96ee0d4adde67651141bd0a6d079f65bb179e0083f13a66a09e7e65851d734c"} Nov 03 01:15:05.510302 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:05.479217 4421 generic.go:356] "Generic (PLEG): container finished" podID="dd14e9bc-4127-471b-8502-dc36b4244615" containerID="d96ee0d4adde67651141bd0a6d079f65bb179e0083f13a66a09e7e65851d734c" exitCode=60 Nov 03 01:15:05.674231 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:05.510904 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-531447-29894475-zr4nt" event={"ID":"dd14e9bc-4127-471b-8502-dc36b4244615","Type":"ContainerDied","Data":"d96ee0d4adde67651141bd0a6d079f65bb179e0083f13a66a09e7e65851d734c"} Nov 03 01:15:05.675362 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:05.511643 4421 scope.go:122] "RemoveContainer" containerID="d96ee0d4adde67651141bd0a6d079f65bb179e0083f13a66a09e7e65851d734c" Nov 03 01:15:05.679805 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:15:05.679693 4421 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-531447-29894475-zr4nt_odh-ai-gateway-infra(dd14e9bc-4127-471b-8502-dc36b4244615)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-531447-29894475-zr4nt" podUID="dd14e9bc-4127-471b-8502-dc36b4244615" Nov 03 01:15:05.681868 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:05.681346 4421 scope.go:122] "RemoveContainer" containerID="88d98162d8db82eaf95a9da9f9d84996413ee518691f7fad2be3d1774e32ba96" Nov 03 01:15:06.322125 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:06.320710 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/distinct-w3-83755a-kserve-86bdbbdd85-2cnnl" podUID="f4ae9624-d127-4a55-837d-7c9ff0267b37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:15:06.415297 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:06.400824 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-650be0/premium-w4-650be0-kserve-7bf8555dfb-z97n5" podUID="40601dc5-808f-456f-826f-6e0c35966ada" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 01:15:06.446298 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:06.445415 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/premium-w3-83755a-kserve-7c9d66988b-lt4xd" podUID="c0c6b02e-2ecb-428e-a13d-be6d7cd1ba39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:15:06.684473 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:06.684384 4421 scope.go:122] "RemoveContainer" containerID="d96ee0d4adde67651141bd0a6d079f65bb179e0083f13a66a09e7e65851d734c" Nov 03 01:15:06.684857 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:15:06.684821 4421 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-531447-29894475-zr4nt_odh-ai-gateway-infra(dd14e9bc-4127-471b-8502-dc36b4244615)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-531447-29894475-zr4nt" podUID="dd14e9bc-4127-471b-8502-dc36b4244615" Nov 03 01:15:12.108351 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:12.108130 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d75f4405-jw9b9"] Nov 03 01:15:12.160298 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:12.158178 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d75f4405-jw9b9"] Nov 03 01:15:13.989697 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:13.989645 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6861d029-4676-4e04-9dc1-3496949a7a7f" path="/var/lib/kubelet/pods/6861d029-4676-4e04-9dc1-3496949a7a7f/volumes" Nov 03 01:15:15.058299 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:15:15.056808 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 01:15:16.972077 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:16.972045 4421 scope.go:122] "RemoveContainer" containerID="d96ee0d4adde67651141bd0a6d079f65bb179e0083f13a66a09e7e65851d734c" Nov 03 01:15:18.199213 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:18.199171 4421 generic.go:356] "Generic (PLEG): container finished" podID="dd14e9bc-4127-471b-8502-dc36b4244615" containerID="d79ad8b736d9e0bf60f394ee099218fea9050d87431ebc8f376986812aa45efc" exitCode=60 Nov 03 01:15:18.199745 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:18.199240 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-531447-29894475-zr4nt" event={"ID":"dd14e9bc-4127-471b-8502-dc36b4244615","Type":"ContainerDied","Data":"d79ad8b736d9e0bf60f394ee099218fea9050d87431ebc8f376986812aa45efc"} Nov 03 01:15:18.199745 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:18.199351 4421 scope.go:122] "RemoveContainer" containerID="d96ee0d4adde67651141bd0a6d079f65bb179e0083f13a66a09e7e65851d734c" Nov 03 01:15:18.199854 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:18.199843 4421 scope.go:122] "RemoveContainer" containerID="d79ad8b736d9e0bf60f394ee099218fea9050d87431ebc8f376986812aa45efc" Nov 03 01:15:18.200397 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:15:18.200368 4421 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-531447-29894475-zr4nt_odh-ai-gateway-infra(dd14e9bc-4127-471b-8502-dc36b4244615)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-531447-29894475-zr4nt" podUID="dd14e9bc-4127-471b-8502-dc36b4244615" Nov 03 01:15:19.214244 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:19.214206 4421 generic.go:356] "Generic (PLEG): container finished" podID="d7edd679-63c6-4949-b3e5-e4e09274d522" containerID="3e48bdcc23749d41fff58606c10e327e7b4b81376eb8d44e28bb4d33eee1db46" exitCode=0 Nov 03 01:15:19.214860 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:19.214292 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-dg98g" event={"ID":"d7edd679-63c6-4949-b3e5-e4e09274d522","Type":"ContainerDied","Data":"3e48bdcc23749d41fff58606c10e327e7b4b81376eb8d44e28bb4d33eee1db46"} Nov 03 01:15:19.214949 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:19.214868 4421 scope.go:122] "RemoveContainer" containerID="3e48bdcc23749d41fff58606c10e327e7b4b81376eb8d44e28bb4d33eee1db46" Nov 03 01:15:19.218042 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:19.218006 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:15:19.218211 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:19.218088 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:15:19.218211 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:19.218121 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:15:19.238075 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:19.238042 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-531447-29894475-zr4nt"] Nov 03 01:15:19.536413 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:19.536377 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-531447-29894475-zr4nt" Nov 03 01:15:19.574553 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:19.574518 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-650be0-29894475-9gm47"] Nov 03 01:15:19.575457 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:19.575441 4421 cpu_manager.go:414] "containerMap: removing container" podUID="dd14e9bc-4127-471b-8502-dc36b4244615" containerName="cleanup" Nov 03 01:15:19.575536 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:19.575459 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd14e9bc-4127-471b-8502-dc36b4244615" containerName="cleanup" Nov 03 01:15:19.575536 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:19.575468 4421 cpu_manager.go:414] "containerMap: removing container" podUID="dd14e9bc-4127-471b-8502-dc36b4244615" containerName="cleanup" Nov 03 01:15:19.575536 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:19.575473 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd14e9bc-4127-471b-8502-dc36b4244615" containerName="cleanup" Nov 03 01:15:19.575664 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:19.575654 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd14e9bc-4127-471b-8502-dc36b4244615" containerName="cleanup" Nov 03 01:15:19.575705 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:19.575664 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd14e9bc-4127-471b-8502-dc36b4244615" containerName="cleanup" Nov 03 01:15:19.575705 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:19.575673 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd14e9bc-4127-471b-8502-dc36b4244615" containerName="cleanup" Nov 03 01:15:19.581650 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:19.581625 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-650be0-29894475-9gm47" Nov 03 01:15:19.620391 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:19.620348 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-650be0-29894475-9gm47"] Nov 03 01:15:19.625088 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:19.625055 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd14e9bc-4127-471b-8502-dc36b4244615-kube-api-access-zfb29\" (UniqueName: \"kubernetes.io/projected/dd14e9bc-4127-471b-8502-dc36b4244615-kube-api-access-zfb29\") pod \"dd14e9bc-4127-471b-8502-dc36b4244615\" (UID: \"dd14e9bc-4127-471b-8502-dc36b4244615\") " Nov 03 01:15:19.625374 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:19.625352 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd14e9bc-4127-471b-8502-dc36b4244615-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd14e9bc-4127-471b-8502-dc36b4244615-maas-api-service-ca\") pod \"dd14e9bc-4127-471b-8502-dc36b4244615\" (UID: \"dd14e9bc-4127-471b-8502-dc36b4244615\") " Nov 03 01:15:19.625760 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:19.625730 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd14e9bc-4127-471b-8502-dc36b4244615-maas-api-service-ca" pod "dd14e9bc-4127-471b-8502-dc36b4244615" (UID: "dd14e9bc-4127-471b-8502-dc36b4244615"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:15:19.626180 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:19.626150 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjntx\" (UniqueName: \"kubernetes.io/projected/528ec640-b0fa-40e0-a2ef-03c7d4479ae5-kube-api-access-wjntx\") pod \"maas-api-key-cleanup-e2e-worker-w4-650be0-29894475-9gm47\" (UID: \"528ec640-b0fa-40e0-a2ef-03c7d4479ae5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-650be0-29894475-9gm47" Nov 03 01:15:19.626397 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:19.626376 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/528ec640-b0fa-40e0-a2ef-03c7d4479ae5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-650be0-29894475-9gm47\" (UID: \"528ec640-b0fa-40e0-a2ef-03c7d4479ae5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-650be0-29894475-9gm47" Nov 03 01:15:19.626613 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:19.626594 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd14e9bc-4127-471b-8502-dc36b4244615-maas-api-service-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:15:19.627807 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:19.627779 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd14e9bc-4127-471b-8502-dc36b4244615-kube-api-access-zfb29" pod "dd14e9bc-4127-471b-8502-dc36b4244615" (UID: "dd14e9bc-4127-471b-8502-dc36b4244615"). InnerVolumeSpecName "kube-api-access-zfb29". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:15:19.757356 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:19.757211 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjntx\" (UniqueName: \"kubernetes.io/projected/528ec640-b0fa-40e0-a2ef-03c7d4479ae5-kube-api-access-wjntx\") pod \"maas-api-key-cleanup-e2e-worker-w4-650be0-29894475-9gm47\" (UID: \"528ec640-b0fa-40e0-a2ef-03c7d4479ae5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-650be0-29894475-9gm47" Nov 03 01:15:19.757557 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:19.757387 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/528ec640-b0fa-40e0-a2ef-03c7d4479ae5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-650be0-29894475-9gm47\" (UID: \"528ec640-b0fa-40e0-a2ef-03c7d4479ae5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-650be0-29894475-9gm47" Nov 03 01:15:19.763300 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:19.757749 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zfb29\" (UniqueName: \"kubernetes.io/projected/dd14e9bc-4127-471b-8502-dc36b4244615-kube-api-access-zfb29\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:15:19.763300 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:19.758635 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/528ec640-b0fa-40e0-a2ef-03c7d4479ae5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-650be0-29894475-9gm47\" (UID: \"528ec640-b0fa-40e0-a2ef-03c7d4479ae5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-650be0-29894475-9gm47" Nov 03 01:15:19.774300 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:19.771358 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjntx\" (UniqueName: \"kubernetes.io/projected/528ec640-b0fa-40e0-a2ef-03c7d4479ae5-kube-api-access-wjntx\") pod \"maas-api-key-cleanup-e2e-worker-w4-650be0-29894475-9gm47\" (UID: \"528ec640-b0fa-40e0-a2ef-03c7d4479ae5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-650be0-29894475-9gm47" Nov 03 01:15:19.923460 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:19.923425 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-650be0-29894475-9gm47" Nov 03 01:15:20.219800 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:20.219749 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-650be0-29894475-9gm47"] Nov 03 01:15:20.249498 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:20.249452 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-dg98g" event={"ID":"d7edd679-63c6-4949-b3e5-e4e09274d522","Type":"ContainerStarted","Data":"dc2c5fd1d391f1fc4ff87efd006b69a9cf60f5724b64cbd4ae1c64e60006617d"} Nov 03 01:15:20.252836 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:20.252802 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-650be0-29894475-9gm47" event={"ID":"528ec640-b0fa-40e0-a2ef-03c7d4479ae5","Type":"ContainerStarted","Data":"34466cdbd6f989f55b5e4b785512b0526709311d8fe8aa15bae4b1129250ab90"} Nov 03 01:15:20.254488 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:20.254443 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-531447-29894475-zr4nt" event={"ID":"dd14e9bc-4127-471b-8502-dc36b4244615","Type":"ContainerDied","Data":"42ffcee8e556f7803b36364d605621a67311b5456c9dd814a2017fb1453c9498"} Nov 03 01:15:20.254597 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:20.254507 4421 scope.go:122] "RemoveContainer" containerID="d79ad8b736d9e0bf60f394ee099218fea9050d87431ebc8f376986812aa45efc" Nov 03 01:15:20.254597 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:20.254528 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-531447-29894475-zr4nt" Nov 03 01:15:20.345339 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:20.338440 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-531447-29894475-zr4nt"] Nov 03 01:15:20.346283 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:20.346223 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-531447-29894475-zr4nt"] Nov 03 01:15:20.836190 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:20.835624 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-dg98g" Nov 03 01:15:20.837743 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:20.837708 4421 patch_prober.go:29] interesting pod/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-dg98g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.2:15021/healthz/ready\": dial tcp 10.134.1.2:15021: connect: connection refused" start-of-body= Nov 03 01:15:20.837882 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:20.837769 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-dg98g" podUID="d7edd679-63c6-4949-b3e5-e4e09274d522" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.2:15021/healthz/ready\": dial tcp 10.134.1.2:15021: connect: connection refused" Nov 03 01:15:21.268464 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:21.268418 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-650be0-29894475-9gm47" event={"ID":"528ec640-b0fa-40e0-a2ef-03c7d4479ae5","Type":"ContainerStarted","Data":"0ec412cf7e90349aa9221c325395b88006db9f6d7ee475def940abe64ff60660"} Nov 03 01:15:21.299500 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:21.299445 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-650be0-29894475-9gm47" podStartSLOduration=21.299428163 podStartE2EDuration="21.299428163s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:15:21.286941529 +0000 UTC m=+2063.457944628" watchObservedRunningTime="2026-11-03 01:15:21.299428163 +0000 UTC m=+2063.470431259" Nov 03 01:15:21.844362 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:21.840728 4421 patch_prober.go:29] interesting pod/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-dg98g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.2:15021/healthz/ready\": dial tcp 10.134.1.2:15021: connect: connection refused" start-of-body= Nov 03 01:15:21.844362 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:21.840788 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-dg98g" podUID="d7edd679-63c6-4949-b3e5-e4e09274d522" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.2:15021/healthz/ready\": dial tcp 10.134.1.2:15021: connect: connection refused" Nov 03 01:15:22.152341 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:22.151601 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd14e9bc-4127-471b-8502-dc36b4244615" path="/var/lib/kubelet/pods/dd14e9bc-4127-471b-8502-dc36b4244615/volumes" Nov 03 01:15:22.152341 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:22.152173 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16fcf300-d5mjj"] Nov 03 01:15:22.152341 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:22.152216 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16fcf300-d5mjj"] Nov 03 01:15:22.290041 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:22.289990 4421 generic.go:356] "Generic (PLEG): container finished" podID="528ec640-b0fa-40e0-a2ef-03c7d4479ae5" containerID="0ec412cf7e90349aa9221c325395b88006db9f6d7ee475def940abe64ff60660" exitCode=60 Nov 03 01:15:22.290551 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:22.290098 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-650be0-29894475-9gm47" event={"ID":"528ec640-b0fa-40e0-a2ef-03c7d4479ae5","Type":"ContainerDied","Data":"0ec412cf7e90349aa9221c325395b88006db9f6d7ee475def940abe64ff60660"} Nov 03 01:15:22.290668 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:22.290652 4421 scope.go:122] "RemoveContainer" containerID="0ec412cf7e90349aa9221c325395b88006db9f6d7ee475def940abe64ff60660" Nov 03 01:15:22.843792 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:22.843688 4421 patch_prober.go:29] interesting pod/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-dg98g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.2:15021/healthz/ready\": dial tcp 10.134.1.2:15021: connect: connection refused" start-of-body= Nov 03 01:15:22.843792 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:22.843737 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-dg98g" podUID="d7edd679-63c6-4949-b3e5-e4e09274d522" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.2:15021/healthz/ready\": dial tcp 10.134.1.2:15021: connect: connection refused" Nov 03 01:15:23.378401 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:23.376251 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-650be0-29894475-9gm47" event={"ID":"528ec640-b0fa-40e0-a2ef-03c7d4479ae5","Type":"ContainerStarted","Data":"33277f72fe5a34ae9e377d4b434f9514986510c12161cfaeb15d7a3aeb5dcd61"} Nov 03 01:15:23.836045 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:23.836002 4421 patch_prober.go:29] interesting pod/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-dg98g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.2:15021/healthz/ready\": dial tcp 10.134.1.2:15021: connect: connection refused" start-of-body= Nov 03 01:15:23.836389 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:23.836347 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-dg98g" podUID="d7edd679-63c6-4949-b3e5-e4e09274d522" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.2:15021/healthz/ready\": dial tcp 10.134.1.2:15021: connect: connection refused" Nov 03 01:15:23.982628 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:23.982593 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1016d54-45b9-4ea7-b7b9-15aa57a18914" path="/var/lib/kubelet/pods/e1016d54-45b9-4ea7-b7b9-15aa57a18914/volumes" Nov 03 01:15:24.296026 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:24.294763 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-dg98g" Nov 03 01:15:24.391922 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:24.391895 4421 generic.go:356] "Generic (PLEG): container finished" podID="528ec640-b0fa-40e0-a2ef-03c7d4479ae5" containerID="33277f72fe5a34ae9e377d4b434f9514986510c12161cfaeb15d7a3aeb5dcd61" exitCode=60 Nov 03 01:15:24.392424 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:24.392400 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-650be0-29894475-9gm47" event={"ID":"528ec640-b0fa-40e0-a2ef-03c7d4479ae5","Type":"ContainerDied","Data":"33277f72fe5a34ae9e377d4b434f9514986510c12161cfaeb15d7a3aeb5dcd61"} Nov 03 01:15:24.392629 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:24.392613 4421 scope.go:122] "RemoveContainer" containerID="0ec412cf7e90349aa9221c325395b88006db9f6d7ee475def940abe64ff60660" Nov 03 01:15:24.392867 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:24.392850 4421 scope.go:122] "RemoveContainer" containerID="33277f72fe5a34ae9e377d4b434f9514986510c12161cfaeb15d7a3aeb5dcd61" Nov 03 01:15:24.393418 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:15:24.393385 4421 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-650be0-29894475-9gm47_odh-ai-gateway-infra(528ec640-b0fa-40e0-a2ef-03c7d4479ae5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-650be0-29894475-9gm47" podUID="528ec640-b0fa-40e0-a2ef-03c7d4479ae5" Nov 03 01:15:24.839299 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:24.838841 4421 patch_prober.go:29] interesting pod/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-dg98g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.2:15021/healthz/ready\": dial tcp 10.134.1.2:15021: connect: connection refused" start-of-body= Nov 03 01:15:24.839299 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:24.838910 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-dg98g" podUID="d7edd679-63c6-4949-b3e5-e4e09274d522" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.2:15021/healthz/ready\": dial tcp 10.134.1.2:15021: connect: connection refused" Nov 03 01:15:25.403323 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:25.403290 4421 scope.go:122] "RemoveContainer" containerID="33277f72fe5a34ae9e377d4b434f9514986510c12161cfaeb15d7a3aeb5dcd61" Nov 03 01:15:25.403758 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:15:25.403734 4421 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-650be0-29894475-9gm47_odh-ai-gateway-infra(528ec640-b0fa-40e0-a2ef-03c7d4479ae5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-650be0-29894475-9gm47" podUID="528ec640-b0fa-40e0-a2ef-03c7d4479ae5" Nov 03 01:15:25.835444 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:25.835407 4421 patch_prober.go:29] interesting pod/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-dg98g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.2:15021/healthz/ready\": dial tcp 10.134.1.2:15021: connect: connection refused" start-of-body= Nov 03 01:15:25.835614 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:25.835466 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-dg98g" podUID="d7edd679-63c6-4949-b3e5-e4e09274d522" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.2:15021/healthz/ready\": dial tcp 10.134.1.2:15021: connect: connection refused" Nov 03 01:15:26.954591 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:26.954529 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-dg98g" Nov 03 01:15:26.956398 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:26.956374 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-dg98g" Nov 03 01:15:33.994600 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:33.994561 4421 scope.go:122] "RemoveContainer" containerID="33277f72fe5a34ae9e377d4b434f9514986510c12161cfaeb15d7a3aeb5dcd61" Nov 03 01:15:34.602143 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:34.602093 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-650be0-29894475-9gm47" event={"ID":"528ec640-b0fa-40e0-a2ef-03c7d4479ae5","Type":"ContainerStarted","Data":"8a9e9c3db624d00a671b1b76f66777d2ea0627f95a84d3f5aa57b3f8c8188a1b"} Nov 03 01:15:34.764526 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:34.764471 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xj4dr"] Nov 03 01:15:34.765131 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:34.765093 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xj4dr" podUID="88bdc98e-5c06-4a49-9c50-013ca50e932d" containerName="istio-proxy" containerID="cri-o://c081f5813e845276d7dd8cc9dbebadb59bd176953e9d5ba8a95457e539da8d93" gracePeriod=30 Nov 03 01:15:35.612265 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:35.612230 4421 generic.go:356] "Generic (PLEG): container finished" podID="528ec640-b0fa-40e0-a2ef-03c7d4479ae5" containerID="8a9e9c3db624d00a671b1b76f66777d2ea0627f95a84d3f5aa57b3f8c8188a1b" exitCode=60 Nov 03 01:15:35.612729 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:35.612306 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-650be0-29894475-9gm47" event={"ID":"528ec640-b0fa-40e0-a2ef-03c7d4479ae5","Type":"ContainerDied","Data":"8a9e9c3db624d00a671b1b76f66777d2ea0627f95a84d3f5aa57b3f8c8188a1b"} Nov 03 01:15:35.612729 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:35.612365 4421 scope.go:122] "RemoveContainer" containerID="33277f72fe5a34ae9e377d4b434f9514986510c12161cfaeb15d7a3aeb5dcd61" Nov 03 01:15:35.612894 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:35.612881 4421 scope.go:122] "RemoveContainer" containerID="8a9e9c3db624d00a671b1b76f66777d2ea0627f95a84d3f5aa57b3f8c8188a1b" Nov 03 01:15:35.613367 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:15:35.613331 4421 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-650be0-29894475-9gm47_odh-ai-gateway-infra(528ec640-b0fa-40e0-a2ef-03c7d4479ae5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-650be0-29894475-9gm47" podUID="528ec640-b0fa-40e0-a2ef-03c7d4479ae5" Nov 03 01:15:35.632842 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:35.632813 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-650be0-29894475-9gm47"] Nov 03 01:15:36.823985 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:36.823955 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-650be0-29894475-9gm47" Nov 03 01:15:36.856065 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:36.856024 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e85f3c00-prm5z"] Nov 03 01:15:36.857082 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:36.857057 4421 cpu_manager.go:414] "containerMap: removing container" podUID="528ec640-b0fa-40e0-a2ef-03c7d4479ae5" containerName="cleanup" Nov 03 01:15:36.857082 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:36.857081 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="528ec640-b0fa-40e0-a2ef-03c7d4479ae5" containerName="cleanup" Nov 03 01:15:36.857255 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:36.857102 4421 cpu_manager.go:414] "containerMap: removing container" podUID="528ec640-b0fa-40e0-a2ef-03c7d4479ae5" containerName="cleanup" Nov 03 01:15:36.857255 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:36.857111 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="528ec640-b0fa-40e0-a2ef-03c7d4479ae5" containerName="cleanup" Nov 03 01:15:36.857255 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:36.857153 4421 cpu_manager.go:414] "containerMap: removing container" podUID="dd14e9bc-4127-471b-8502-dc36b4244615" containerName="cleanup" Nov 03 01:15:36.857255 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:36.857160 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd14e9bc-4127-471b-8502-dc36b4244615" containerName="cleanup" Nov 03 01:15:36.857255 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:36.857174 4421 cpu_manager.go:414] "containerMap: removing container" podUID="528ec640-b0fa-40e0-a2ef-03c7d4479ae5" containerName="cleanup" Nov 03 01:15:36.857255 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:36.857181 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="528ec640-b0fa-40e0-a2ef-03c7d4479ae5" containerName="cleanup" Nov 03 01:15:36.857491 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:36.857442 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="528ec640-b0fa-40e0-a2ef-03c7d4479ae5" containerName="cleanup" Nov 03 01:15:36.857491 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:36.857458 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="528ec640-b0fa-40e0-a2ef-03c7d4479ae5" containerName="cleanup" Nov 03 01:15:36.862105 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:36.862082 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e85f3c00-prm5z" Nov 03 01:15:36.867013 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:36.866905 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:15:36.872181 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:36.872159 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e85f3c00-prm5z"] Nov 03 01:15:36.902519 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:36.902480 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/528ec640-b0fa-40e0-a2ef-03c7d4479ae5-kube-api-access-wjntx\" (UniqueName: \"kubernetes.io/projected/528ec640-b0fa-40e0-a2ef-03c7d4479ae5-kube-api-access-wjntx\") pod \"528ec640-b0fa-40e0-a2ef-03c7d4479ae5\" (UID: \"528ec640-b0fa-40e0-a2ef-03c7d4479ae5\") " Nov 03 01:15:36.902704 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:36.902634 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/528ec640-b0fa-40e0-a2ef-03c7d4479ae5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/528ec640-b0fa-40e0-a2ef-03c7d4479ae5-maas-api-service-ca\") pod \"528ec640-b0fa-40e0-a2ef-03c7d4479ae5\" (UID: \"528ec640-b0fa-40e0-a2ef-03c7d4479ae5\") " Nov 03 01:15:36.903106 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:36.903077 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/528ec640-b0fa-40e0-a2ef-03c7d4479ae5-maas-api-service-ca" pod "528ec640-b0fa-40e0-a2ef-03c7d4479ae5" (UID: "528ec640-b0fa-40e0-a2ef-03c7d4479ae5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:15:36.903348 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:36.903320 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/613397a3-4bde-4b90-ad05-e73fff87a5fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e85f3c00-prm5z\" (UID: \"613397a3-4bde-4b90-ad05-e73fff87a5fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e85f3c00-prm5z" Nov 03 01:15:36.903521 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:36.903500 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/613397a3-4bde-4b90-ad05-e73fff87a5fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e85f3c00-prm5z\" (UID: \"613397a3-4bde-4b90-ad05-e73fff87a5fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e85f3c00-prm5z" Nov 03 01:15:36.903604 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:36.903549 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjfbp\" (UniqueName: \"kubernetes.io/projected/613397a3-4bde-4b90-ad05-e73fff87a5fa-kube-api-access-zjfbp\") pod \"maas-api-revoke-sub-e85f3c00-prm5z\" (UID: \"613397a3-4bde-4b90-ad05-e73fff87a5fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e85f3c00-prm5z" Nov 03 01:15:36.903865 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:36.903844 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/528ec640-b0fa-40e0-a2ef-03c7d4479ae5-maas-api-service-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:15:36.904949 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:36.904917 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/528ec640-b0fa-40e0-a2ef-03c7d4479ae5-kube-api-access-wjntx" pod "528ec640-b0fa-40e0-a2ef-03c7d4479ae5" (UID: "528ec640-b0fa-40e0-a2ef-03c7d4479ae5"). InnerVolumeSpecName "kube-api-access-wjntx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:15:37.005027 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:37.004976 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/613397a3-4bde-4b90-ad05-e73fff87a5fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e85f3c00-prm5z\" (UID: \"613397a3-4bde-4b90-ad05-e73fff87a5fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e85f3c00-prm5z" Nov 03 01:15:37.005231 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:37.005072 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/613397a3-4bde-4b90-ad05-e73fff87a5fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e85f3c00-prm5z\" (UID: \"613397a3-4bde-4b90-ad05-e73fff87a5fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e85f3c00-prm5z" Nov 03 01:15:37.005231 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:37.005095 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjfbp\" (UniqueName: \"kubernetes.io/projected/613397a3-4bde-4b90-ad05-e73fff87a5fa-kube-api-access-zjfbp\") pod \"maas-api-revoke-sub-e85f3c00-prm5z\" (UID: \"613397a3-4bde-4b90-ad05-e73fff87a5fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e85f3c00-prm5z" Nov 03 01:15:37.005231 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:37.005170 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjntx\" (UniqueName: \"kubernetes.io/projected/528ec640-b0fa-40e0-a2ef-03c7d4479ae5-kube-api-access-wjntx\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:15:37.005944 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:37.005904 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/613397a3-4bde-4b90-ad05-e73fff87a5fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e85f3c00-prm5z\" (UID: \"613397a3-4bde-4b90-ad05-e73fff87a5fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e85f3c00-prm5z" Nov 03 01:15:37.007618 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:37.007591 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/613397a3-4bde-4b90-ad05-e73fff87a5fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e85f3c00-prm5z\" (UID: \"613397a3-4bde-4b90-ad05-e73fff87a5fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e85f3c00-prm5z" Nov 03 01:15:37.037315 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:37.030378 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjfbp\" (UniqueName: \"kubernetes.io/projected/613397a3-4bde-4b90-ad05-e73fff87a5fa-kube-api-access-zjfbp\") pod \"maas-api-revoke-sub-e85f3c00-prm5z\" (UID: \"613397a3-4bde-4b90-ad05-e73fff87a5fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e85f3c00-prm5z" Nov 03 01:15:37.187112 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:37.187029 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e85f3c00-prm5z" Nov 03 01:15:37.471735 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:15:37.471702 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod613397a3_4bde_4b90_ad05_e73fff87a5fa.slice/crio-e1b3ed559715a1eea494a89aa29a11a260d06a83b93d2c08944b00e4abfdcb59 WatchSource:0}: Error finding container e1b3ed559715a1eea494a89aa29a11a260d06a83b93d2c08944b00e4abfdcb59: Status 404 returned error can't find the container with id e1b3ed559715a1eea494a89aa29a11a260d06a83b93d2c08944b00e4abfdcb59 Nov 03 01:15:37.471892 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:37.471865 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e85f3c00-prm5z"] Nov 03 01:15:37.637346 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:37.637310 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-650be0-29894475-9gm47" event={"ID":"528ec640-b0fa-40e0-a2ef-03c7d4479ae5","Type":"ContainerDied","Data":"34466cdbd6f989f55b5e4b785512b0526709311d8fe8aa15bae4b1129250ab90"} Nov 03 01:15:37.637508 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:37.637317 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-650be0-29894475-9gm47" Nov 03 01:15:37.637508 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:37.637385 4421 scope.go:122] "RemoveContainer" containerID="8a9e9c3db624d00a671b1b76f66777d2ea0627f95a84d3f5aa57b3f8c8188a1b" Nov 03 01:15:37.638873 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:37.638845 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e85f3c00-prm5z" event={"ID":"613397a3-4bde-4b90-ad05-e73fff87a5fa","Type":"ContainerStarted","Data":"e1b3ed559715a1eea494a89aa29a11a260d06a83b93d2c08944b00e4abfdcb59"} Nov 03 01:15:37.669477 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:37.669449 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-650be0-29894475-9gm47"] Nov 03 01:15:37.682390 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:37.682354 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-650be0-29894475-9gm47"] Nov 03 01:15:37.981976 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:37.981904 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="528ec640-b0fa-40e0-a2ef-03c7d4479ae5" path="/var/lib/kubelet/pods/528ec640-b0fa-40e0-a2ef-03c7d4479ae5/volumes" Nov 03 01:15:38.665012 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:38.664973 4421 generic.go:356] "Generic (PLEG): container finished" podID="613397a3-4bde-4b90-ad05-e73fff87a5fa" containerID="487c72a1af88c682515c46c7a05334f46a99bbe8e738ce4ff6f60c427a364ad5" exitCode=0 Nov 03 01:15:38.665215 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:38.665056 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e85f3c00-prm5z" event={"ID":"613397a3-4bde-4b90-ad05-e73fff87a5fa","Type":"ContainerDied","Data":"487c72a1af88c682515c46c7a05334f46a99bbe8e738ce4ff6f60c427a364ad5"} Nov 03 01:15:39.909860 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:39.909826 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e85f3c00-prm5z" Nov 03 01:15:39.945337 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:39.945205 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f933cbe-wz7pk"] Nov 03 01:15:39.946583 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:39.946548 4421 cpu_manager.go:414] "containerMap: removing container" podUID="613397a3-4bde-4b90-ad05-e73fff87a5fa" containerName="revoke-keys" Nov 03 01:15:39.946583 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:39.946580 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="613397a3-4bde-4b90-ad05-e73fff87a5fa" containerName="revoke-keys" Nov 03 01:15:39.946847 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:39.946832 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="528ec640-b0fa-40e0-a2ef-03c7d4479ae5" containerName="cleanup" Nov 03 01:15:39.946918 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:39.946864 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="613397a3-4bde-4b90-ad05-e73fff87a5fa" containerName="revoke-keys" Nov 03 01:15:39.952299 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:39.951876 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f933cbe-wz7pk" Nov 03 01:15:39.964092 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:39.964063 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f933cbe-wz7pk"] Nov 03 01:15:40.012295 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:40.001212 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/613397a3-4bde-4b90-ad05-e73fff87a5fa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/613397a3-4bde-4b90-ad05-e73fff87a5fa-maas-api-xks-ca\") pod \"613397a3-4bde-4b90-ad05-e73fff87a5fa\" (UID: \"613397a3-4bde-4b90-ad05-e73fff87a5fa\") " Nov 03 01:15:40.012295 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:40.001312 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/613397a3-4bde-4b90-ad05-e73fff87a5fa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/613397a3-4bde-4b90-ad05-e73fff87a5fa-maas-api-openshift-ca\") pod \"613397a3-4bde-4b90-ad05-e73fff87a5fa\" (UID: \"613397a3-4bde-4b90-ad05-e73fff87a5fa\") " Nov 03 01:15:40.012295 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:40.001389 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/613397a3-4bde-4b90-ad05-e73fff87a5fa-kube-api-access-zjfbp\" (UniqueName: \"kubernetes.io/projected/613397a3-4bde-4b90-ad05-e73fff87a5fa-kube-api-access-zjfbp\") pod \"613397a3-4bde-4b90-ad05-e73fff87a5fa\" (UID: \"613397a3-4bde-4b90-ad05-e73fff87a5fa\") " Nov 03 01:15:40.088399 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:40.050616 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/613397a3-4bde-4b90-ad05-e73fff87a5fa-maas-api-openshift-ca" pod "613397a3-4bde-4b90-ad05-e73fff87a5fa" (UID: "613397a3-4bde-4b90-ad05-e73fff87a5fa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:15:40.114812 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:40.113210 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b673d7f4-7e8e-4573-8208-c9ca7fe1b3f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6f933cbe-wz7pk\" (UID: \"b673d7f4-7e8e-4573-8208-c9ca7fe1b3f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f933cbe-wz7pk" Nov 03 01:15:40.116269 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:40.113264 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kls6k\" (UniqueName: \"kubernetes.io/projected/b673d7f4-7e8e-4573-8208-c9ca7fe1b3f6-kube-api-access-kls6k\") pod \"maas-api-revoke-sub-6f933cbe-wz7pk\" (UID: \"b673d7f4-7e8e-4573-8208-c9ca7fe1b3f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f933cbe-wz7pk" Nov 03 01:15:40.116269 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:40.115261 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b673d7f4-7e8e-4573-8208-c9ca7fe1b3f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6f933cbe-wz7pk\" (UID: \"b673d7f4-7e8e-4573-8208-c9ca7fe1b3f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f933cbe-wz7pk" Nov 03 01:15:40.116269 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:40.115595 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/613397a3-4bde-4b90-ad05-e73fff87a5fa-maas-api-openshift-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:15:40.220996 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:40.220896 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b673d7f4-7e8e-4573-8208-c9ca7fe1b3f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6f933cbe-wz7pk\" (UID: \"b673d7f4-7e8e-4573-8208-c9ca7fe1b3f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f933cbe-wz7pk" Nov 03 01:15:40.221664 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:40.221629 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kls6k\" (UniqueName: \"kubernetes.io/projected/b673d7f4-7e8e-4573-8208-c9ca7fe1b3f6-kube-api-access-kls6k\") pod \"maas-api-revoke-sub-6f933cbe-wz7pk\" (UID: \"b673d7f4-7e8e-4573-8208-c9ca7fe1b3f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f933cbe-wz7pk" Nov 03 01:15:40.222114 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:40.222085 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b673d7f4-7e8e-4573-8208-c9ca7fe1b3f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6f933cbe-wz7pk\" (UID: \"b673d7f4-7e8e-4573-8208-c9ca7fe1b3f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f933cbe-wz7pk" Nov 03 01:15:40.223786 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:40.223744 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b673d7f4-7e8e-4573-8208-c9ca7fe1b3f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6f933cbe-wz7pk\" (UID: \"b673d7f4-7e8e-4573-8208-c9ca7fe1b3f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f933cbe-wz7pk" Nov 03 01:15:40.230091 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:40.224224 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b673d7f4-7e8e-4573-8208-c9ca7fe1b3f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6f933cbe-wz7pk\" (UID: \"b673d7f4-7e8e-4573-8208-c9ca7fe1b3f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f933cbe-wz7pk" Nov 03 01:15:40.233682 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:40.233618 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/613397a3-4bde-4b90-ad05-e73fff87a5fa-kube-api-access-zjfbp" pod "613397a3-4bde-4b90-ad05-e73fff87a5fa" (UID: "613397a3-4bde-4b90-ad05-e73fff87a5fa"). InnerVolumeSpecName "kube-api-access-zjfbp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:15:40.236368 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:40.236298 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kls6k\" (UniqueName: \"kubernetes.io/projected/b673d7f4-7e8e-4573-8208-c9ca7fe1b3f6-kube-api-access-kls6k\") pod \"maas-api-revoke-sub-6f933cbe-wz7pk\" (UID: \"b673d7f4-7e8e-4573-8208-c9ca7fe1b3f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f933cbe-wz7pk" Nov 03 01:15:40.236699 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:40.236674 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/613397a3-4bde-4b90-ad05-e73fff87a5fa-maas-api-xks-ca" pod "613397a3-4bde-4b90-ad05-e73fff87a5fa" (UID: "613397a3-4bde-4b90-ad05-e73fff87a5fa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:15:40.324361 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:40.324330 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/613397a3-4bde-4b90-ad05-e73fff87a5fa-maas-api-xks-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:15:40.324557 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:40.324369 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjfbp\" (UniqueName: \"kubernetes.io/projected/613397a3-4bde-4b90-ad05-e73fff87a5fa-kube-api-access-zjfbp\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:15:40.347348 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:40.347313 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xj4dr" Nov 03 01:15:40.385590 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:40.385553 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-5779vg5bt"] Nov 03 01:15:40.386796 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:40.386766 4421 cpu_manager.go:414] "containerMap: removing container" podUID="88bdc98e-5c06-4a49-9c50-013ca50e932d" containerName="istio-proxy" Nov 03 01:15:40.386796 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:40.386796 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88bdc98e-5c06-4a49-9c50-013ca50e932d" containerName="istio-proxy" Nov 03 01:15:40.387058 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:40.387036 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="88bdc98e-5c06-4a49-9c50-013ca50e932d" containerName="istio-proxy" Nov 03 01:15:40.392015 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:40.391987 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-7697grqf4"] Nov 03 01:15:40.392297 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:40.392130 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-5779vg5bt" Nov 03 01:15:40.394506 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:40.394298 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f933cbe-wz7pk" Nov 03 01:15:40.407321 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:40.399240 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-57079b\"/\"agc-cred-83631bdec3fc\"" Nov 03 01:15:40.407321 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:40.399320 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-57079b\"/\"agc-cred-06b235044840\"" Nov 03 01:15:40.407321 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:40.399365 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-57079b\"/\"openshift-service-ca.crt\"" Nov 03 01:15:40.407321 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:40.399391 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-57079b\"/\"payload-processing-external-model-plugins-e2e-extmod-b-57079b\"" Nov 03 01:15:40.407321 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:40.399917 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-57079b\"/\"routing-overlay\"" Nov 03 01:15:40.407321 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:40.399858 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-57079b\"/\"kube-root-ca.crt\"" Nov 03 01:15:40.407321 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:40.402939 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-57079b\"/\"payload-processing-external-model-e2e--4a3ed774-dockercfg-nwdwk\"" Nov 03 01:15:40.407321 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:40.404599 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-7697grqf4" Nov 03 01:15:40.408352 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:40.408325 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-5779vg5bt"] Nov 03 01:15:40.414867 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:40.414834 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-7697grqf4"] Nov 03 01:15:40.425389 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:40.425363 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88bdc98e-5c06-4a49-9c50-013ca50e932d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88bdc98e-5c06-4a49-9c50-013ca50e932d-istio-envoy\") pod \"88bdc98e-5c06-4a49-9c50-013ca50e932d\" (UID: \"88bdc98e-5c06-4a49-9c50-013ca50e932d\") " Nov 03 01:15:40.425512 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:40.425411 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88bdc98e-5c06-4a49-9c50-013ca50e932d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88bdc98e-5c06-4a49-9c50-013ca50e932d-istiod-ca-cert\") pod \"88bdc98e-5c06-4a49-9c50-013ca50e932d\" (UID: \"88bdc98e-5c06-4a49-9c50-013ca50e932d\") " Nov 03 01:15:40.425512 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:40.425467 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88bdc98e-5c06-4a49-9c50-013ca50e932d-kube-api-access-k5td7\" (UniqueName: \"kubernetes.io/projected/88bdc98e-5c06-4a49-9c50-013ca50e932d-kube-api-access-k5td7\") pod \"88bdc98e-5c06-4a49-9c50-013ca50e932d\" (UID: \"88bdc98e-5c06-4a49-9c50-013ca50e932d\") " Nov 03 01:15:40.425651 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:40.425561 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88bdc98e-5c06-4a49-9c50-013ca50e932d-istio-token\" (UniqueName: \"kubernetes.io/projected/88bdc98e-5c06-4a49-9c50-013ca50e932d-istio-token\") pod \"88bdc98e-5c06-4a49-9c50-013ca50e932d\" (UID: \"88bdc98e-5c06-4a49-9c50-013ca50e932d\") " Nov 03 01:15:40.425651 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:40.425596 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88bdc98e-5c06-4a49-9c50-013ca50e932d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88bdc98e-5c06-4a49-9c50-013ca50e932d-workload-certs\") pod \"88bdc98e-5c06-4a49-9c50-013ca50e932d\" (UID: \"88bdc98e-5c06-4a49-9c50-013ca50e932d\") " Nov 03 01:15:40.425766 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:40.425655 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88bdc98e-5c06-4a49-9c50-013ca50e932d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88bdc98e-5c06-4a49-9c50-013ca50e932d-workload-socket\") pod \"88bdc98e-5c06-4a49-9c50-013ca50e932d\" (UID: \"88bdc98e-5c06-4a49-9c50-013ca50e932d\") " Nov 03 01:15:40.425766 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:40.425685 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/88bdc98e-5c06-4a49-9c50-013ca50e932d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88bdc98e-5c06-4a49-9c50-013ca50e932d-istio-podinfo\") pod \"88bdc98e-5c06-4a49-9c50-013ca50e932d\" (UID: \"88bdc98e-5c06-4a49-9c50-013ca50e932d\") " Nov 03 01:15:40.425864 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:40.425766 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88bdc98e-5c06-4a49-9c50-013ca50e932d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88bdc98e-5c06-4a49-9c50-013ca50e932d-istio-data\") pod \"88bdc98e-5c06-4a49-9c50-013ca50e932d\" (UID: \"88bdc98e-5c06-4a49-9c50-013ca50e932d\") " Nov 03 01:15:40.425941 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:40.425920 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88bdc98e-5c06-4a49-9c50-013ca50e932d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88bdc98e-5c06-4a49-9c50-013ca50e932d-credential-socket\") pod \"88bdc98e-5c06-4a49-9c50-013ca50e932d\" (UID: \"88bdc98e-5c06-4a49-9c50-013ca50e932d\") " Nov 03 01:15:40.426172 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:40.426150 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88bdc98e-5c06-4a49-9c50-013ca50e932d-workload-certs" pod "88bdc98e-5c06-4a49-9c50-013ca50e932d" (UID: "88bdc98e-5c06-4a49-9c50-013ca50e932d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:15:40.426439 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:40.426415 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88bdc98e-5c06-4a49-9c50-013ca50e932d-istiod-ca-cert" pod "88bdc98e-5c06-4a49-9c50-013ca50e932d" (UID: "88bdc98e-5c06-4a49-9c50-013ca50e932d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:15:40.426556 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:40.426534 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbl66\" (UniqueName: \"kubernetes.io/projected/c9a4975c-6ec7-43c0-aaea-32c45c400d42-kube-api-access-gbl66\") pod \"payload-processing-external-model-e2e-extmod-b-57079b-5779vg5bt\" (UID: \"c9a4975c-6ec7-43c0-aaea-32c45c400d42\") " pod="ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-5779vg5bt" Nov 03 01:15:40.426716 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:40.426695 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9a4975c-6ec7-43c0-aaea-32c45c400d42-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-57079b-5779vg5bt\" (UID: \"c9a4975c-6ec7-43c0-aaea-32c45c400d42\") " pod="ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-5779vg5bt" Nov 03 01:15:40.426774 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:40.426751 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c9a4975c-6ec7-43c0-aaea-32c45c400d42-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-57079b-5779vg5bt\" (UID: \"c9a4975c-6ec7-43c0-aaea-32c45c400d42\") " pod="ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-5779vg5bt" Nov 03 01:15:40.426839 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:40.426826 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c9a4975c-6ec7-43c0-aaea-32c45c400d42-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-57079b-5779vg5bt\" (UID: \"c9a4975c-6ec7-43c0-aaea-32c45c400d42\") " pod="ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-5779vg5bt" Nov 03 01:15:40.426969 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:40.426627 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88bdc98e-5c06-4a49-9c50-013ca50e932d-credential-socket" pod "88bdc98e-5c06-4a49-9c50-013ca50e932d" (UID: "88bdc98e-5c06-4a49-9c50-013ca50e932d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:15:40.427044 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:40.426790 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88bdc98e-5c06-4a49-9c50-013ca50e932d-istio-data" pod "88bdc98e-5c06-4a49-9c50-013ca50e932d" (UID: "88bdc98e-5c06-4a49-9c50-013ca50e932d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:15:40.427044 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:40.426942 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88bdc98e-5c06-4a49-9c50-013ca50e932d-workload-socket" pod "88bdc98e-5c06-4a49-9c50-013ca50e932d" (UID: "88bdc98e-5c06-4a49-9c50-013ca50e932d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:15:40.427376 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:40.427350 4421 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88bdc98e-5c06-4a49-9c50-013ca50e932d-credential-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:15:40.427489 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:40.427378 4421 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88bdc98e-5c06-4a49-9c50-013ca50e932d-istiod-ca-cert\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:15:40.427489 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:40.427395 4421 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88bdc98e-5c06-4a49-9c50-013ca50e932d-workload-certs\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:15:40.427489 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:40.427410 4421 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88bdc98e-5c06-4a49-9c50-013ca50e932d-workload-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:15:40.427489 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:40.427425 4421 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88bdc98e-5c06-4a49-9c50-013ca50e932d-istio-data\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:15:40.428252 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:40.428232 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88bdc98e-5c06-4a49-9c50-013ca50e932d-istio-envoy" pod "88bdc98e-5c06-4a49-9c50-013ca50e932d" (UID: "88bdc98e-5c06-4a49-9c50-013ca50e932d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:15:40.431593 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:40.431570 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/88bdc98e-5c06-4a49-9c50-013ca50e932d-istio-podinfo" pod "88bdc98e-5c06-4a49-9c50-013ca50e932d" (UID: "88bdc98e-5c06-4a49-9c50-013ca50e932d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:15:40.431737 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:40.431619 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88bdc98e-5c06-4a49-9c50-013ca50e932d-kube-api-access-k5td7" pod "88bdc98e-5c06-4a49-9c50-013ca50e932d" (UID: "88bdc98e-5c06-4a49-9c50-013ca50e932d"). InnerVolumeSpecName "kube-api-access-k5td7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:15:40.432851 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:40.432825 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88bdc98e-5c06-4a49-9c50-013ca50e932d-istio-token" pod "88bdc98e-5c06-4a49-9c50-013ca50e932d" (UID: "88bdc98e-5c06-4a49-9c50-013ca50e932d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:15:40.773522 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:40.773472 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbl66\" (UniqueName: \"kubernetes.io/projected/c9a4975c-6ec7-43c0-aaea-32c45c400d42-kube-api-access-gbl66\") pod \"payload-processing-external-model-e2e-extmod-b-57079b-5779vg5bt\" (UID: \"c9a4975c-6ec7-43c0-aaea-32c45c400d42\") " pod="ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-5779vg5bt" Nov 03 01:15:40.782307 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:40.782231 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9a4975c-6ec7-43c0-aaea-32c45c400d42-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-57079b-5779vg5bt\" (UID: \"c9a4975c-6ec7-43c0-aaea-32c45c400d42\") " pod="ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-5779vg5bt" Nov 03 01:15:40.782621 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:40.782581 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c9a4975c-6ec7-43c0-aaea-32c45c400d42-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-57079b-5779vg5bt\" (UID: \"c9a4975c-6ec7-43c0-aaea-32c45c400d42\") " pod="ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-5779vg5bt" Nov 03 01:15:40.782849 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:40.782830 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c46f150a-c198-412d-b0ab-b35753457acb-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-3c5983-7697grqf4\" (UID: \"c46f150a-c198-412d-b0ab-b35753457acb\") " pod="ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-7697grqf4" Nov 03 01:15:40.783017 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:40.783002 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c9a4975c-6ec7-43c0-aaea-32c45c400d42-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-57079b-5779vg5bt\" (UID: \"c9a4975c-6ec7-43c0-aaea-32c45c400d42\") " pod="ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-5779vg5bt" Nov 03 01:15:40.783173 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:40.783158 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c46f150a-c198-412d-b0ab-b35753457acb-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3c5983-7697grqf4\" (UID: \"c46f150a-c198-412d-b0ab-b35753457acb\") " pod="ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-7697grqf4" Nov 03 01:15:40.783329 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:40.783309 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptzfv\" (UniqueName: \"kubernetes.io/projected/c46f150a-c198-412d-b0ab-b35753457acb-kube-api-access-ptzfv\") pod \"payload-processing-external-model-e2e-extmod-b-3c5983-7697grqf4\" (UID: \"c46f150a-c198-412d-b0ab-b35753457acb\") " pod="ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-7697grqf4" Nov 03 01:15:40.783864 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:40.783836 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c46f150a-c198-412d-b0ab-b35753457acb-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3c5983-7697grqf4\" (UID: \"c46f150a-c198-412d-b0ab-b35753457acb\") " pod="ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-7697grqf4" Nov 03 01:15:40.784229 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:40.784209 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5td7\" (UniqueName: \"kubernetes.io/projected/88bdc98e-5c06-4a49-9c50-013ca50e932d-kube-api-access-k5td7\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:15:40.784379 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:40.784365 4421 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88bdc98e-5c06-4a49-9c50-013ca50e932d-istio-token\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:15:40.784473 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:40.784462 4421 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88bdc98e-5c06-4a49-9c50-013ca50e932d-istio-podinfo\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:15:40.784551 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:40.784542 4421 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88bdc98e-5c06-4a49-9c50-013ca50e932d-istio-envoy\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:15:40.785568 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:40.785545 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9a4975c-6ec7-43c0-aaea-32c45c400d42-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-57079b-5779vg5bt\" (UID: \"c9a4975c-6ec7-43c0-aaea-32c45c400d42\") " pod="ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-5779vg5bt" Nov 03 01:15:40.786384 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:40.786363 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c9a4975c-6ec7-43c0-aaea-32c45c400d42-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-57079b-5779vg5bt\" (UID: \"c9a4975c-6ec7-43c0-aaea-32c45c400d42\") " pod="ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-5779vg5bt" Nov 03 01:15:40.828606 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:40.791245 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c9a4975c-6ec7-43c0-aaea-32c45c400d42-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-57079b-5779vg5bt\" (UID: \"c9a4975c-6ec7-43c0-aaea-32c45c400d42\") " pod="ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-5779vg5bt" Nov 03 01:15:40.875299 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:40.871795 4421 generic.go:356] "Generic (PLEG): container finished" podID="88bdc98e-5c06-4a49-9c50-013ca50e932d" containerID="c081f5813e845276d7dd8cc9dbebadb59bd176953e9d5ba8a95457e539da8d93" exitCode=0 Nov 03 01:15:40.875299 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:40.871897 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xj4dr" event={"ID":"88bdc98e-5c06-4a49-9c50-013ca50e932d","Type":"ContainerDied","Data":"c081f5813e845276d7dd8cc9dbebadb59bd176953e9d5ba8a95457e539da8d93"} Nov 03 01:15:40.875299 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:40.871947 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xj4dr" event={"ID":"88bdc98e-5c06-4a49-9c50-013ca50e932d","Type":"ContainerDied","Data":"32674b1c68d04f489cd6d37f3a84c9b9bd9294957c2f54c16e58f1cd1a2f9ec1"} Nov 03 01:15:40.875299 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:40.871994 4421 scope.go:122] "RemoveContainer" containerID="c081f5813e845276d7dd8cc9dbebadb59bd176953e9d5ba8a95457e539da8d93" Nov 03 01:15:40.875299 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:40.872130 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xj4dr" Nov 03 01:15:40.905765 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:40.896962 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c46f150a-c198-412d-b0ab-b35753457acb-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3c5983-7697grqf4\" (UID: \"c46f150a-c198-412d-b0ab-b35753457acb\") " pod="ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-7697grqf4" Nov 03 01:15:40.905765 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:40.897213 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c46f150a-c198-412d-b0ab-b35753457acb-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-3c5983-7697grqf4\" (UID: \"c46f150a-c198-412d-b0ab-b35753457acb\") " pod="ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-7697grqf4" Nov 03 01:15:40.905765 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:40.897324 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c46f150a-c198-412d-b0ab-b35753457acb-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3c5983-7697grqf4\" (UID: \"c46f150a-c198-412d-b0ab-b35753457acb\") " pod="ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-7697grqf4" Nov 03 01:15:40.905765 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:40.897375 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptzfv\" (UniqueName: \"kubernetes.io/projected/c46f150a-c198-412d-b0ab-b35753457acb-kube-api-access-ptzfv\") pod \"payload-processing-external-model-e2e-extmod-b-3c5983-7697grqf4\" (UID: \"c46f150a-c198-412d-b0ab-b35753457acb\") " pod="ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-7697grqf4" Nov 03 01:15:40.905765 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:40.899170 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c46f150a-c198-412d-b0ab-b35753457acb-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3c5983-7697grqf4\" (UID: \"c46f150a-c198-412d-b0ab-b35753457acb\") " pod="ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-7697grqf4" Nov 03 01:15:40.905765 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:40.900099 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c46f150a-c198-412d-b0ab-b35753457acb-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3c5983-7697grqf4\" (UID: \"c46f150a-c198-412d-b0ab-b35753457acb\") " pod="ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-7697grqf4" Nov 03 01:15:40.908491 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:40.908430 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c46f150a-c198-412d-b0ab-b35753457acb-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-3c5983-7697grqf4\" (UID: \"c46f150a-c198-412d-b0ab-b35753457acb\") " pod="ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-7697grqf4" Nov 03 01:15:40.922240 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:40.922203 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbl66\" (UniqueName: \"kubernetes.io/projected/c9a4975c-6ec7-43c0-aaea-32c45c400d42-kube-api-access-gbl66\") pod \"payload-processing-external-model-e2e-extmod-b-57079b-5779vg5bt\" (UID: \"c9a4975c-6ec7-43c0-aaea-32c45c400d42\") " pod="ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-5779vg5bt" Nov 03 01:15:40.922739 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:40.922525 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e85f3c00-prm5z" event={"ID":"613397a3-4bde-4b90-ad05-e73fff87a5fa","Type":"ContainerDied","Data":"e1b3ed559715a1eea494a89aa29a11a260d06a83b93d2c08944b00e4abfdcb59"} Nov 03 01:15:40.922739 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:40.922609 4421 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e1b3ed559715a1eea494a89aa29a11a260d06a83b93d2c08944b00e4abfdcb59" Nov 03 01:15:40.922739 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:40.922686 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e85f3c00-prm5z" Nov 03 01:15:40.924040 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:40.923951 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptzfv\" (UniqueName: \"kubernetes.io/projected/c46f150a-c198-412d-b0ab-b35753457acb-kube-api-access-ptzfv\") pod \"payload-processing-external-model-e2e-extmod-b-3c5983-7697grqf4\" (UID: \"c46f150a-c198-412d-b0ab-b35753457acb\") " pod="ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-7697grqf4" Nov 03 01:15:40.966397 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:40.966372 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f933cbe-wz7pk"] Nov 03 01:15:40.974018 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:40.973989 4421 scope.go:122] "RemoveContainer" containerID="c081f5813e845276d7dd8cc9dbebadb59bd176953e9d5ba8a95457e539da8d93" Nov 03 01:15:40.974323 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:15:40.974257 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c081f5813e845276d7dd8cc9dbebadb59bd176953e9d5ba8a95457e539da8d93\": container with ID starting with c081f5813e845276d7dd8cc9dbebadb59bd176953e9d5ba8a95457e539da8d93 not found: ID does not exist" containerID="c081f5813e845276d7dd8cc9dbebadb59bd176953e9d5ba8a95457e539da8d93" Nov 03 01:15:40.974446 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:40.974332 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c081f5813e845276d7dd8cc9dbebadb59bd176953e9d5ba8a95457e539da8d93"} err="failed to get container status \"c081f5813e845276d7dd8cc9dbebadb59bd176953e9d5ba8a95457e539da8d93\": rpc error: code = NotFound desc = could not find container \"c081f5813e845276d7dd8cc9dbebadb59bd176953e9d5ba8a95457e539da8d93\": container with ID starting with c081f5813e845276d7dd8cc9dbebadb59bd176953e9d5ba8a95457e539da8d93 not found: ID does not exist" Nov 03 01:15:41.005382 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:41.005340 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xj4dr"] Nov 03 01:15:41.010062 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:41.010033 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xj4dr"] Nov 03 01:15:41.026446 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:41.026376 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-5779vg5bt" Nov 03 01:15:41.034069 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:41.034044 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-7697grqf4" Nov 03 01:15:41.342240 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:41.342202 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-5779vg5bt"] Nov 03 01:15:41.366496 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:41.365331 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-7697grqf4"] Nov 03 01:15:41.932568 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:41.932466 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f933cbe-wz7pk" event={"ID":"b673d7f4-7e8e-4573-8208-c9ca7fe1b3f6","Type":"ContainerStarted","Data":"72f41cea4e90ff8b1b7a4d6d0af68b4b02b38c4baaf881c8069941248efa7a4f"} Nov 03 01:15:41.932568 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:41.932521 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f933cbe-wz7pk" event={"ID":"b673d7f4-7e8e-4573-8208-c9ca7fe1b3f6","Type":"ContainerStarted","Data":"3a8d985a55caee7d025ffc076b19ee4610e1f20f7aa7d2c039ed4a8de7c2815f"} Nov 03 01:15:41.934334 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:41.934309 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-5779vg5bt" event={"ID":"c9a4975c-6ec7-43c0-aaea-32c45c400d42","Type":"ContainerStarted","Data":"01e63850bbc26fefd2e501e72191bad3541984d2eb209565405433d8fecaf94a"} Nov 03 01:15:41.934486 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:41.934353 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-5779vg5bt" event={"ID":"c9a4975c-6ec7-43c0-aaea-32c45c400d42","Type":"ContainerStarted","Data":"3562f44655e92707752ac3b89423cdcbd8cafc4ee045670272455f440b7dc8cb"} Nov 03 01:15:41.934486 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:41.934433 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-5779vg5bt" Nov 03 01:15:41.937044 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:41.937023 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-7697grqf4" event={"ID":"c46f150a-c198-412d-b0ab-b35753457acb","Type":"ContainerStarted","Data":"f057b165e4c7a935b23b7e664abd0f5ced22f4170fd5c68510b356c944960288"} Nov 03 01:15:41.937140 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:41.937057 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-7697grqf4" event={"ID":"c46f150a-c198-412d-b0ab-b35753457acb","Type":"ContainerStarted","Data":"d98785d9f464b5888085a7c55451c72ed23435ab71b13674cd9c58ad7f03e38d"} Nov 03 01:15:41.937140 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:41.937107 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-7697grqf4" Nov 03 01:15:41.950998 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:41.950953 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f933cbe-wz7pk" podStartSLOduration=32.950939415 podStartE2EDuration="32.950939415s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:15:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:15:41.949840724 +0000 UTC m=+2084.120843823" watchObservedRunningTime="2026-11-03 01:15:41.950939415 +0000 UTC m=+2084.121942510" Nov 03 01:15:41.971358 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:41.971306 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-7697grqf4" podStartSLOduration=6.971292314 podStartE2EDuration="6.971292314s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:15:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:15:41.966835128 +0000 UTC m=+2084.137838254" watchObservedRunningTime="2026-11-03 01:15:41.971292314 +0000 UTC m=+2084.142295407" Nov 03 01:15:41.982071 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:41.982038 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88bdc98e-5c06-4a49-9c50-013ca50e932d" path="/var/lib/kubelet/pods/88bdc98e-5c06-4a49-9c50-013ca50e932d/volumes" Nov 03 01:15:41.989371 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:41.989327 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-5779vg5bt" podStartSLOduration=1.989313762 podStartE2EDuration="1.989313762s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:15:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:15:41.985644951 +0000 UTC m=+2084.156648048" watchObservedRunningTime="2026-11-03 01:15:41.989313762 +0000 UTC m=+2084.160316858" Nov 03 01:15:42.946786 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:42.946748 4421 generic.go:356] "Generic (PLEG): container finished" podID="b673d7f4-7e8e-4573-8208-c9ca7fe1b3f6" containerID="72f41cea4e90ff8b1b7a4d6d0af68b4b02b38c4baaf881c8069941248efa7a4f" exitCode=0 Nov 03 01:15:42.947193 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:42.946825 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f933cbe-wz7pk" event={"ID":"b673d7f4-7e8e-4573-8208-c9ca7fe1b3f6","Type":"ContainerDied","Data":"72f41cea4e90ff8b1b7a4d6d0af68b4b02b38c4baaf881c8069941248efa7a4f"} Nov 03 01:15:44.173983 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:44.173961 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f933cbe-wz7pk" Nov 03 01:15:44.211704 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:44.211614 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b673d7f4-7e8e-4573-8208-c9ca7fe1b3f6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b673d7f4-7e8e-4573-8208-c9ca7fe1b3f6-maas-api-xks-ca\") pod \"b673d7f4-7e8e-4573-8208-c9ca7fe1b3f6\" (UID: \"b673d7f4-7e8e-4573-8208-c9ca7fe1b3f6\") " Nov 03 01:15:44.211845 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:44.211818 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b673d7f4-7e8e-4573-8208-c9ca7fe1b3f6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b673d7f4-7e8e-4573-8208-c9ca7fe1b3f6-maas-api-openshift-ca\") pod \"b673d7f4-7e8e-4573-8208-c9ca7fe1b3f6\" (UID: \"b673d7f4-7e8e-4573-8208-c9ca7fe1b3f6\") " Nov 03 01:15:44.211917 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:44.211858 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b673d7f4-7e8e-4573-8208-c9ca7fe1b3f6-kube-api-access-kls6k\" (UniqueName: \"kubernetes.io/projected/b673d7f4-7e8e-4573-8208-c9ca7fe1b3f6-kube-api-access-kls6k\") pod \"b673d7f4-7e8e-4573-8208-c9ca7fe1b3f6\" (UID: \"b673d7f4-7e8e-4573-8208-c9ca7fe1b3f6\") " Nov 03 01:15:44.212222 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:44.212187 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b673d7f4-7e8e-4573-8208-c9ca7fe1b3f6-maas-api-openshift-ca" pod "b673d7f4-7e8e-4573-8208-c9ca7fe1b3f6" (UID: "b673d7f4-7e8e-4573-8208-c9ca7fe1b3f6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:15:44.212740 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:44.212719 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b673d7f4-7e8e-4573-8208-c9ca7fe1b3f6-maas-api-openshift-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:15:44.214701 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:44.214670 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b673d7f4-7e8e-4573-8208-c9ca7fe1b3f6-maas-api-xks-ca" pod "b673d7f4-7e8e-4573-8208-c9ca7fe1b3f6" (UID: "b673d7f4-7e8e-4573-8208-c9ca7fe1b3f6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:15:44.214797 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:44.214777 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b673d7f4-7e8e-4573-8208-c9ca7fe1b3f6-kube-api-access-kls6k" pod "b673d7f4-7e8e-4573-8208-c9ca7fe1b3f6" (UID: "b673d7f4-7e8e-4573-8208-c9ca7fe1b3f6"). InnerVolumeSpecName "kube-api-access-kls6k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:15:44.315098 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:44.314781 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kls6k\" (UniqueName: \"kubernetes.io/projected/b673d7f4-7e8e-4573-8208-c9ca7fe1b3f6-kube-api-access-kls6k\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:15:44.315098 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:44.314820 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b673d7f4-7e8e-4573-8208-c9ca7fe1b3f6-maas-api-xks-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:15:44.734517 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:15:44.734476 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 01:15:44.967354 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:44.967305 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f933cbe-wz7pk" event={"ID":"b673d7f4-7e8e-4573-8208-c9ca7fe1b3f6","Type":"ContainerDied","Data":"3a8d985a55caee7d025ffc076b19ee4610e1f20f7aa7d2c039ed4a8de7c2815f"} Nov 03 01:15:44.967354 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:44.967340 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f933cbe-wz7pk" Nov 03 01:15:44.967354 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:44.967362 4421 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3a8d985a55caee7d025ffc076b19ee4610e1f20f7aa7d2c039ed4a8de7c2815f" Nov 03 01:15:45.200410 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:45.200377 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-7697grqf4"] Nov 03 01:15:47.001360 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:47.001325 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf8dfc0e-z8qwx"] Nov 03 01:15:47.002240 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:47.002220 4421 cpu_manager.go:414] "containerMap: removing container" podUID="b673d7f4-7e8e-4573-8208-c9ca7fe1b3f6" containerName="revoke-keys" Nov 03 01:15:47.002240 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:47.002238 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b673d7f4-7e8e-4573-8208-c9ca7fe1b3f6" containerName="revoke-keys" Nov 03 01:15:47.002490 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:47.002397 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="b673d7f4-7e8e-4573-8208-c9ca7fe1b3f6" containerName="revoke-keys" Nov 03 01:15:47.006369 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:47.006351 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf8dfc0e-z8qwx" Nov 03 01:15:47.012348 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:47.012189 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:15:47.012348 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:47.012302 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6q4xp\"" Nov 03 01:15:47.016938 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:47.016915 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf8dfc0e-z8qwx"] Nov 03 01:15:47.152766 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:47.152213 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/adda682e-735e-42a1-ac06-80e472ae394a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cf8dfc0e-z8qwx\" (UID: \"adda682e-735e-42a1-ac06-80e472ae394a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf8dfc0e-z8qwx" Nov 03 01:15:47.161673 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:47.161625 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/adda682e-735e-42a1-ac06-80e472ae394a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cf8dfc0e-z8qwx\" (UID: \"adda682e-735e-42a1-ac06-80e472ae394a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf8dfc0e-z8qwx" Nov 03 01:15:47.161888 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:47.161847 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8b88\" (UniqueName: \"kubernetes.io/projected/adda682e-735e-42a1-ac06-80e472ae394a-kube-api-access-m8b88\") pod \"maas-api-revoke-sub-cf8dfc0e-z8qwx\" (UID: \"adda682e-735e-42a1-ac06-80e472ae394a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf8dfc0e-z8qwx" Nov 03 01:15:47.263926 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:47.263821 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/adda682e-735e-42a1-ac06-80e472ae394a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cf8dfc0e-z8qwx\" (UID: \"adda682e-735e-42a1-ac06-80e472ae394a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf8dfc0e-z8qwx" Nov 03 01:15:47.263926 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:47.263898 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8b88\" (UniqueName: \"kubernetes.io/projected/adda682e-735e-42a1-ac06-80e472ae394a-kube-api-access-m8b88\") pod \"maas-api-revoke-sub-cf8dfc0e-z8qwx\" (UID: \"adda682e-735e-42a1-ac06-80e472ae394a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf8dfc0e-z8qwx" Nov 03 01:15:47.264152 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:47.263965 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/adda682e-735e-42a1-ac06-80e472ae394a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cf8dfc0e-z8qwx\" (UID: \"adda682e-735e-42a1-ac06-80e472ae394a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf8dfc0e-z8qwx" Nov 03 01:15:47.264722 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:47.264693 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/adda682e-735e-42a1-ac06-80e472ae394a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cf8dfc0e-z8qwx\" (UID: \"adda682e-735e-42a1-ac06-80e472ae394a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf8dfc0e-z8qwx" Nov 03 01:15:47.266509 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:47.266482 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/adda682e-735e-42a1-ac06-80e472ae394a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cf8dfc0e-z8qwx\" (UID: \"adda682e-735e-42a1-ac06-80e472ae394a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf8dfc0e-z8qwx" Nov 03 01:15:47.273546 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:47.273520 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8b88\" (UniqueName: \"kubernetes.io/projected/adda682e-735e-42a1-ac06-80e472ae394a-kube-api-access-m8b88\") pod \"maas-api-revoke-sub-cf8dfc0e-z8qwx\" (UID: \"adda682e-735e-42a1-ac06-80e472ae394a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf8dfc0e-z8qwx" Nov 03 01:15:47.326588 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:47.326547 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf8dfc0e-z8qwx" Nov 03 01:15:47.620521 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:15:47.620487 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadda682e_735e_42a1_ac06_80e472ae394a.slice/crio-2ab9f75a311d847b3f3e9ada9a25a5a314822091bd094ed5e0a1d135f06df907 WatchSource:0}: Error finding container 2ab9f75a311d847b3f3e9ada9a25a5a314822091bd094ed5e0a1d135f06df907: Status 404 returned error can't find the container with id 2ab9f75a311d847b3f3e9ada9a25a5a314822091bd094ed5e0a1d135f06df907 Nov 03 01:15:47.621190 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:47.621166 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf8dfc0e-z8qwx"] Nov 03 01:15:48.009823 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:48.009785 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf8dfc0e-z8qwx" event={"ID":"adda682e-735e-42a1-ac06-80e472ae394a","Type":"ContainerStarted","Data":"2ab9f75a311d847b3f3e9ada9a25a5a314822091bd094ed5e0a1d135f06df907"} Nov 03 01:15:49.020853 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:49.020819 4421 generic.go:356] "Generic (PLEG): container finished" podID="adda682e-735e-42a1-ac06-80e472ae394a" containerID="d370c6f01369231bb6918b93173892863d5f868e89f92258affd730f451b8f45" exitCode=0 Nov 03 01:15:49.021504 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:49.020904 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf8dfc0e-z8qwx" event={"ID":"adda682e-735e-42a1-ac06-80e472ae394a","Type":"ContainerDied","Data":"d370c6f01369231bb6918b93173892863d5f868e89f92258affd730f451b8f45"} Nov 03 01:15:50.204159 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:50.201246 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-7697grqf4" podUID="c46f150a-c198-412d-b0ab-b35753457acb" containerName="payload-processing" containerID="cri-o://f057b165e4c7a935b23b7e664abd0f5ced22f4170fd5c68510b356c944960288" gracePeriod=25 Nov 03 01:15:50.373229 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:50.373199 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf8dfc0e-z8qwx" Nov 03 01:15:50.454213 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:50.454178 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/adda682e-735e-42a1-ac06-80e472ae394a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/adda682e-735e-42a1-ac06-80e472ae394a-maas-api-xks-ca\") pod \"adda682e-735e-42a1-ac06-80e472ae394a\" (UID: \"adda682e-735e-42a1-ac06-80e472ae394a\") " Nov 03 01:15:50.454471 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:50.454315 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/adda682e-735e-42a1-ac06-80e472ae394a-kube-api-access-m8b88\" (UniqueName: \"kubernetes.io/projected/adda682e-735e-42a1-ac06-80e472ae394a-kube-api-access-m8b88\") pod \"adda682e-735e-42a1-ac06-80e472ae394a\" (UID: \"adda682e-735e-42a1-ac06-80e472ae394a\") " Nov 03 01:15:50.454624 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:50.454602 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/adda682e-735e-42a1-ac06-80e472ae394a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/adda682e-735e-42a1-ac06-80e472ae394a-maas-api-openshift-ca\") pod \"adda682e-735e-42a1-ac06-80e472ae394a\" (UID: \"adda682e-735e-42a1-ac06-80e472ae394a\") " Nov 03 01:15:50.455935 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:50.455898 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/adda682e-735e-42a1-ac06-80e472ae394a-maas-api-openshift-ca" pod "adda682e-735e-42a1-ac06-80e472ae394a" (UID: "adda682e-735e-42a1-ac06-80e472ae394a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:15:50.457548 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:50.457517 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adda682e-735e-42a1-ac06-80e472ae394a-kube-api-access-m8b88" pod "adda682e-735e-42a1-ac06-80e472ae394a" (UID: "adda682e-735e-42a1-ac06-80e472ae394a"). InnerVolumeSpecName "kube-api-access-m8b88". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:15:50.457783 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:50.457749 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/adda682e-735e-42a1-ac06-80e472ae394a-maas-api-xks-ca" pod "adda682e-735e-42a1-ac06-80e472ae394a" (UID: "adda682e-735e-42a1-ac06-80e472ae394a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:15:50.516551 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:50.516517 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-7697grqf4" Nov 03 01:15:50.550135 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:50.550047 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-679574xvq"] Nov 03 01:15:50.551462 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:50.551412 4421 cpu_manager.go:414] "containerMap: removing container" podUID="c46f150a-c198-412d-b0ab-b35753457acb" containerName="payload-processing" Nov 03 01:15:50.551462 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:50.551462 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c46f150a-c198-412d-b0ab-b35753457acb" containerName="payload-processing" Nov 03 01:15:50.551674 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:50.551482 4421 cpu_manager.go:414] "containerMap: removing container" podUID="adda682e-735e-42a1-ac06-80e472ae394a" containerName="revoke-keys" Nov 03 01:15:50.551674 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:50.551491 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="adda682e-735e-42a1-ac06-80e472ae394a" containerName="revoke-keys" Nov 03 01:15:50.551984 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:50.551966 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="c46f150a-c198-412d-b0ab-b35753457acb" containerName="payload-processing" Nov 03 01:15:50.552071 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:50.551989 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="adda682e-735e-42a1-ac06-80e472ae394a" containerName="revoke-keys" Nov 03 01:15:50.556384 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:50.556354 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-679574xvq" Nov 03 01:15:50.556899 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:50.556864 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c46f150a-c198-412d-b0ab-b35753457acb-provider-credentials\" (UniqueName: \"kubernetes.io/projected/c46f150a-c198-412d-b0ab-b35753457acb-provider-credentials\") pod \"c46f150a-c198-412d-b0ab-b35753457acb\" (UID: \"c46f150a-c198-412d-b0ab-b35753457acb\") " Nov 03 01:15:50.557045 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:50.557024 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c46f150a-c198-412d-b0ab-b35753457acb-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c46f150a-c198-412d-b0ab-b35753457acb-routing-overlay-volume\") pod \"c46f150a-c198-412d-b0ab-b35753457acb\" (UID: \"c46f150a-c198-412d-b0ab-b35753457acb\") " Nov 03 01:15:50.557125 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:50.557065 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c46f150a-c198-412d-b0ab-b35753457acb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c46f150a-c198-412d-b0ab-b35753457acb-plugins-config-volume\") pod \"c46f150a-c198-412d-b0ab-b35753457acb\" (UID: \"c46f150a-c198-412d-b0ab-b35753457acb\") " Nov 03 01:15:50.557233 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:50.557216 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c46f150a-c198-412d-b0ab-b35753457acb-kube-api-access-ptzfv\" (UniqueName: \"kubernetes.io/projected/c46f150a-c198-412d-b0ab-b35753457acb-kube-api-access-ptzfv\") pod \"c46f150a-c198-412d-b0ab-b35753457acb\" (UID: \"c46f150a-c198-412d-b0ab-b35753457acb\") " Nov 03 01:15:50.559151 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:50.557533 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c46f150a-c198-412d-b0ab-b35753457acb-routing-overlay-volume" pod "c46f150a-c198-412d-b0ab-b35753457acb" (UID: "c46f150a-c198-412d-b0ab-b35753457acb"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:15:50.559151 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:50.558185 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c46f150a-c198-412d-b0ab-b35753457acb-plugins-config-volume" pod "c46f150a-c198-412d-b0ab-b35753457acb" (UID: "c46f150a-c198-412d-b0ab-b35753457acb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:15:50.559151 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:50.558391 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/adda682e-735e-42a1-ac06-80e472ae394a-maas-api-xks-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:15:50.559151 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:50.558414 4421 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c46f150a-c198-412d-b0ab-b35753457acb-routing-overlay-volume\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:15:50.559151 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:50.558430 4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c46f150a-c198-412d-b0ab-b35753457acb-plugins-config-volume\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:15:50.559151 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:50.558444 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8b88\" (UniqueName: \"kubernetes.io/projected/adda682e-735e-42a1-ac06-80e472ae394a-kube-api-access-m8b88\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:15:50.559151 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:50.558458 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/adda682e-735e-42a1-ac06-80e472ae394a-maas-api-openshift-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:15:50.562354 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:50.562315 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c46f150a-c198-412d-b0ab-b35753457acb-kube-api-access-ptzfv" pod "c46f150a-c198-412d-b0ab-b35753457acb" (UID: "c46f150a-c198-412d-b0ab-b35753457acb"). InnerVolumeSpecName "kube-api-access-ptzfv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:15:50.568649 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:50.568464 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-3c5983\"/\"agc-cred-07505ed8483e\"" Nov 03 01:15:50.568649 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:50.568609 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c46f150a-c198-412d-b0ab-b35753457acb-provider-credentials" pod "c46f150a-c198-412d-b0ab-b35753457acb" (UID: "c46f150a-c198-412d-b0ab-b35753457acb"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:15:50.571193 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:50.571170 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-679574xvq"] Nov 03 01:15:50.668297 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:50.664741 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a8667bca-ac8c-4f83-8d87-71c67bb3bf66-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-3c5983-679574xvq\" (UID: \"a8667bca-ac8c-4f83-8d87-71c67bb3bf66\") " pod="ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-679574xvq" Nov 03 01:15:50.668297 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:50.664857 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52h8n\" (UniqueName: \"kubernetes.io/projected/a8667bca-ac8c-4f83-8d87-71c67bb3bf66-kube-api-access-52h8n\") pod \"payload-processing-external-model-e2e-extmod-b-3c5983-679574xvq\" (UID: \"a8667bca-ac8c-4f83-8d87-71c67bb3bf66\") " pod="ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-679574xvq" Nov 03 01:15:50.668297 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:50.665091 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8667bca-ac8c-4f83-8d87-71c67bb3bf66-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3c5983-679574xvq\" (UID: \"a8667bca-ac8c-4f83-8d87-71c67bb3bf66\") " pod="ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-679574xvq" Nov 03 01:15:50.668297 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:50.665139 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a8667bca-ac8c-4f83-8d87-71c67bb3bf66-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3c5983-679574xvq\" (UID: \"a8667bca-ac8c-4f83-8d87-71c67bb3bf66\") " pod="ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-679574xvq" Nov 03 01:15:50.668297 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:50.665290 4421 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c46f150a-c198-412d-b0ab-b35753457acb-provider-credentials\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:15:50.668297 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:50.665308 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptzfv\" (UniqueName: \"kubernetes.io/projected/c46f150a-c198-412d-b0ab-b35753457acb-kube-api-access-ptzfv\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:15:50.767026 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:50.766989 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a8667bca-ac8c-4f83-8d87-71c67bb3bf66-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-3c5983-679574xvq\" (UID: \"a8667bca-ac8c-4f83-8d87-71c67bb3bf66\") " pod="ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-679574xvq" Nov 03 01:15:50.767234 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:50.767104 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52h8n\" (UniqueName: \"kubernetes.io/projected/a8667bca-ac8c-4f83-8d87-71c67bb3bf66-kube-api-access-52h8n\") pod \"payload-processing-external-model-e2e-extmod-b-3c5983-679574xvq\" (UID: \"a8667bca-ac8c-4f83-8d87-71c67bb3bf66\") " pod="ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-679574xvq" Nov 03 01:15:50.767396 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:50.767376 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8667bca-ac8c-4f83-8d87-71c67bb3bf66-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3c5983-679574xvq\" (UID: \"a8667bca-ac8c-4f83-8d87-71c67bb3bf66\") " pod="ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-679574xvq" Nov 03 01:15:50.767478 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:50.767432 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a8667bca-ac8c-4f83-8d87-71c67bb3bf66-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3c5983-679574xvq\" (UID: \"a8667bca-ac8c-4f83-8d87-71c67bb3bf66\") " pod="ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-679574xvq" Nov 03 01:15:50.768232 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:50.768199 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8667bca-ac8c-4f83-8d87-71c67bb3bf66-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3c5983-679574xvq\" (UID: \"a8667bca-ac8c-4f83-8d87-71c67bb3bf66\") " pod="ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-679574xvq" Nov 03 01:15:50.768393 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:50.768239 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a8667bca-ac8c-4f83-8d87-71c67bb3bf66-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3c5983-679574xvq\" (UID: \"a8667bca-ac8c-4f83-8d87-71c67bb3bf66\") " pod="ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-679574xvq" Nov 03 01:15:50.769766 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:50.769747 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a8667bca-ac8c-4f83-8d87-71c67bb3bf66-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-3c5983-679574xvq\" (UID: \"a8667bca-ac8c-4f83-8d87-71c67bb3bf66\") " pod="ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-679574xvq" Nov 03 01:15:50.775143 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:50.775121 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-52h8n\" (UniqueName: \"kubernetes.io/projected/a8667bca-ac8c-4f83-8d87-71c67bb3bf66-kube-api-access-52h8n\") pod \"payload-processing-external-model-e2e-extmod-b-3c5983-679574xvq\" (UID: \"a8667bca-ac8c-4f83-8d87-71c67bb3bf66\") " pod="ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-679574xvq" Nov 03 01:15:50.886539 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:50.886449 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-679574xvq" Nov 03 01:15:51.094994 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:51.087563 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4263775b-xb4px"] Nov 03 01:15:51.094994 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:51.090070 4421 generic.go:356] "Generic (PLEG): container finished" podID="c46f150a-c198-412d-b0ab-b35753457acb" containerID="f057b165e4c7a935b23b7e664abd0f5ced22f4170fd5c68510b356c944960288" exitCode=0 Nov 03 01:15:51.094994 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:51.090250 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-7697grqf4" event={"ID":"c46f150a-c198-412d-b0ab-b35753457acb","Type":"ContainerDied","Data":"f057b165e4c7a935b23b7e664abd0f5ced22f4170fd5c68510b356c944960288"} Nov 03 01:15:51.094994 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:51.090323 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-7697grqf4" event={"ID":"c46f150a-c198-412d-b0ab-b35753457acb","Type":"ContainerDied","Data":"d98785d9f464b5888085a7c55451c72ed23435ab71b13674cd9c58ad7f03e38d"} Nov 03 01:15:51.094994 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:51.090357 4421 scope.go:122] "RemoveContainer" containerID="f057b165e4c7a935b23b7e664abd0f5ced22f4170fd5c68510b356c944960288" Nov 03 01:15:51.094994 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:51.090530 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-7697grqf4" Nov 03 01:15:51.098935 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:51.098481 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf8dfc0e-z8qwx" event={"ID":"adda682e-735e-42a1-ac06-80e472ae394a","Type":"ContainerDied","Data":"2ab9f75a311d847b3f3e9ada9a25a5a314822091bd094ed5e0a1d135f06df907"} Nov 03 01:15:51.098935 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:51.098534 4421 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2ab9f75a311d847b3f3e9ada9a25a5a314822091bd094ed5e0a1d135f06df907" Nov 03 01:15:51.098935 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:51.098606 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf8dfc0e-z8qwx" Nov 03 01:15:51.100372 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:51.100345 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4263775b-xb4px"] Nov 03 01:15:51.134054 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:51.134026 4421 scope.go:122] "RemoveContainer" containerID="f057b165e4c7a935b23b7e664abd0f5ced22f4170fd5c68510b356c944960288" Nov 03 01:15:51.134819 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:15:51.134779 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f057b165e4c7a935b23b7e664abd0f5ced22f4170fd5c68510b356c944960288\": container with ID starting with f057b165e4c7a935b23b7e664abd0f5ced22f4170fd5c68510b356c944960288 not found: ID does not exist" containerID="f057b165e4c7a935b23b7e664abd0f5ced22f4170fd5c68510b356c944960288" Nov 03 01:15:51.134940 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:51.134830 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f057b165e4c7a935b23b7e664abd0f5ced22f4170fd5c68510b356c944960288"} err="failed to get container status \"f057b165e4c7a935b23b7e664abd0f5ced22f4170fd5c68510b356c944960288\": rpc error: code = NotFound desc = could not find container \"f057b165e4c7a935b23b7e664abd0f5ced22f4170fd5c68510b356c944960288\": container with ID starting with f057b165e4c7a935b23b7e664abd0f5ced22f4170fd5c68510b356c944960288 not found: ID does not exist" Nov 03 01:15:51.137406 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:51.137356 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-7697grqf4"] Nov 03 01:15:51.141649 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:51.141628 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-7697grqf4"] Nov 03 01:15:51.184107 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:15:51.184074 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8667bca_ac8c_4f83_8d87_71c67bb3bf66.slice/crio-6368b296693656fb7e13cd582de0e1cb7afdee58da216d9a8b9c1c387d269842 WatchSource:0}: Error finding container 6368b296693656fb7e13cd582de0e1cb7afdee58da216d9a8b9c1c387d269842: Status 404 returned error can't find the container with id 6368b296693656fb7e13cd582de0e1cb7afdee58da216d9a8b9c1c387d269842 Nov 03 01:15:51.186200 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:51.186173 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-679574xvq"] Nov 03 01:15:51.612321 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:51.605689 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-5779vg5bt"] Nov 03 01:15:51.670144 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:51.669803 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-5779vg5bt" Nov 03 01:15:52.073196 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:52.073155 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99d7371e-75c2-4bbf-98ee-9c6bc419b779" path="/var/lib/kubelet/pods/99d7371e-75c2-4bbf-98ee-9c6bc419b779/volumes" Nov 03 01:15:52.074266 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:52.074240 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c46f150a-c198-412d-b0ab-b35753457acb" path="/var/lib/kubelet/pods/c46f150a-c198-412d-b0ab-b35753457acb/volumes" Nov 03 01:15:52.243298 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:52.237318 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a395bce9-flxhm"] Nov 03 01:15:52.301296 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:52.287406 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a395bce9-flxhm" Nov 03 01:15:52.338332 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:52.334457 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:15:52.339942 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:52.339914 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a395bce9-flxhm"] Nov 03 01:15:52.341123 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:52.341100 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6q4xp\"" Nov 03 01:15:52.354341 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:52.350964 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-679574xvq" event={"ID":"a8667bca-ac8c-4f83-8d87-71c67bb3bf66","Type":"ContainerStarted","Data":"573418613b9ccea8128a348f70efcca45594781ece2a4438bb9dd192a23531a2"} Nov 03 01:15:52.354341 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:52.351034 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-679574xvq" event={"ID":"a8667bca-ac8c-4f83-8d87-71c67bb3bf66","Type":"ContainerStarted","Data":"6368b296693656fb7e13cd582de0e1cb7afdee58da216d9a8b9c1c387d269842"} Nov 03 01:15:52.354341 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:52.352231 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-679574xvq" Nov 03 01:15:52.382057 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:52.381999 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-679574xvq" podStartSLOduration=7.3819856 podStartE2EDuration="7.3819856s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:15:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:15:52.381690718 +0000 UTC m=+2094.552693814" watchObservedRunningTime="2026-11-03 01:15:52.3819856 +0000 UTC m=+2094.552988696" Nov 03 01:15:52.406129 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:52.406092 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0d13eee-83f0-4cd5-80d0-0a7e6d60db18-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a395bce9-flxhm\" (UID: \"e0d13eee-83f0-4cd5-80d0-0a7e6d60db18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a395bce9-flxhm" Nov 03 01:15:52.406326 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:52.406176 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxfck\" (UniqueName: \"kubernetes.io/projected/e0d13eee-83f0-4cd5-80d0-0a7e6d60db18-kube-api-access-sxfck\") pod \"maas-api-revoke-sub-a395bce9-flxhm\" (UID: \"e0d13eee-83f0-4cd5-80d0-0a7e6d60db18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a395bce9-flxhm" Nov 03 01:15:52.406384 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:52.406349 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0d13eee-83f0-4cd5-80d0-0a7e6d60db18-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a395bce9-flxhm\" (UID: \"e0d13eee-83f0-4cd5-80d0-0a7e6d60db18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a395bce9-flxhm" Nov 03 01:15:52.508865 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:52.508822 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0d13eee-83f0-4cd5-80d0-0a7e6d60db18-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a395bce9-flxhm\" (UID: \"e0d13eee-83f0-4cd5-80d0-0a7e6d60db18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a395bce9-flxhm" Nov 03 01:15:52.510673 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:52.510644 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxfck\" (UniqueName: \"kubernetes.io/projected/e0d13eee-83f0-4cd5-80d0-0a7e6d60db18-kube-api-access-sxfck\") pod \"maas-api-revoke-sub-a395bce9-flxhm\" (UID: \"e0d13eee-83f0-4cd5-80d0-0a7e6d60db18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a395bce9-flxhm" Nov 03 01:15:52.510825 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:52.510736 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0d13eee-83f0-4cd5-80d0-0a7e6d60db18-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a395bce9-flxhm\" (UID: \"e0d13eee-83f0-4cd5-80d0-0a7e6d60db18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a395bce9-flxhm" Nov 03 01:15:52.511466 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:52.510294 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0d13eee-83f0-4cd5-80d0-0a7e6d60db18-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a395bce9-flxhm\" (UID: \"e0d13eee-83f0-4cd5-80d0-0a7e6d60db18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a395bce9-flxhm" Nov 03 01:15:52.523090 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:52.523064 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0d13eee-83f0-4cd5-80d0-0a7e6d60db18-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a395bce9-flxhm\" (UID: \"e0d13eee-83f0-4cd5-80d0-0a7e6d60db18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a395bce9-flxhm" Nov 03 01:15:52.537395 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:52.537362 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxfck\" (UniqueName: \"kubernetes.io/projected/e0d13eee-83f0-4cd5-80d0-0a7e6d60db18-kube-api-access-sxfck\") pod \"maas-api-revoke-sub-a395bce9-flxhm\" (UID: \"e0d13eee-83f0-4cd5-80d0-0a7e6d60db18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a395bce9-flxhm" Nov 03 01:15:52.663428 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:52.663341 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a395bce9-flxhm" Nov 03 01:15:52.947786 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:15:52.947740 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0d13eee_83f0_4cd5_80d0_0a7e6d60db18.slice/crio-855b957bcac6c7a86e01195bddb68d5cfd0282cf0862c8b605931ffdc45bfb6b WatchSource:0}: Error finding container 855b957bcac6c7a86e01195bddb68d5cfd0282cf0862c8b605931ffdc45bfb6b: Status 404 returned error can't find the container with id 855b957bcac6c7a86e01195bddb68d5cfd0282cf0862c8b605931ffdc45bfb6b Nov 03 01:15:52.948948 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:52.948922 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a395bce9-flxhm"] Nov 03 01:15:53.406521 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:53.406481 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a395bce9-flxhm" event={"ID":"e0d13eee-83f0-4cd5-80d0-0a7e6d60db18","Type":"ContainerStarted","Data":"855b957bcac6c7a86e01195bddb68d5cfd0282cf0862c8b605931ffdc45bfb6b"} Nov 03 01:15:54.416733 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:54.416703 4421 generic.go:356] "Generic (PLEG): container finished" podID="e0d13eee-83f0-4cd5-80d0-0a7e6d60db18" containerID="16c5a0a4f80f28c7b6d31df3a91689ef3b72a917d930e31fa2739d3b51a4fb36" exitCode=0 Nov 03 01:15:54.417192 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:54.416799 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a395bce9-flxhm" event={"ID":"e0d13eee-83f0-4cd5-80d0-0a7e6d60db18","Type":"ContainerDied","Data":"16c5a0a4f80f28c7b6d31df3a91689ef3b72a917d930e31fa2739d3b51a4fb36"} Nov 03 01:15:55.063732 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:15:55.063687 4421 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-3c5983/agc-cred-371bb2bc1d0f: secret "agc-cred-371bb2bc1d0f" not found Nov 03 01:15:55.063732 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:15:55.063723 4421 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-3c5983/agc-cred-7fc5c1cab2ff: secret "agc-cred-7fc5c1cab2ff" not found Nov 03 01:15:55.066559 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:15:55.065084 4421 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-78c872f4s: [secret "agc-cred-371bb2bc1d0f" not found, secret "agc-cred-7fc5c1cab2ff" not found] Nov 03 01:15:55.066757 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:15:55.066648 4421 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6386694a-6d4c-4a69-8388-52f34a3484d2-provider-credentials podName:6386694a-6d4c-4a69-8388-52f34a3484d2 nodeName:}" failed. No retries permitted until 2026-11-03 01:15:55.566619315 +0000 UTC m=+2097.737622412 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/6386694a-6d4c-4a69-8388-52f34a3484d2-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-3c5983-78c872f4s" (UID: "6386694a-6d4c-4a69-8388-52f34a3484d2") : [secret "agc-cred-371bb2bc1d0f" not found, secret "agc-cred-7fc5c1cab2ff" not found] Nov 03 01:15:55.574507 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:15:55.574468 4421 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-3c5983/agc-cred-371bb2bc1d0f: secret "agc-cred-371bb2bc1d0f" not found Nov 03 01:15:55.574507 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:15:55.574504 4421 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-3c5983/agc-cred-7fc5c1cab2ff: secret "agc-cred-7fc5c1cab2ff" not found Nov 03 01:15:55.575071 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:15:55.574520 4421 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-78c872f4s: [secret "agc-cred-371bb2bc1d0f" not found, secret "agc-cred-7fc5c1cab2ff" not found] Nov 03 01:15:55.575071 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:15:55.574582 4421 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6386694a-6d4c-4a69-8388-52f34a3484d2-provider-credentials podName:6386694a-6d4c-4a69-8388-52f34a3484d2 nodeName:}" failed. No retries permitted until 2026-11-03 01:15:56.574559264 +0000 UTC m=+2098.745562354 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/6386694a-6d4c-4a69-8388-52f34a3484d2-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-3c5983-78c872f4s" (UID: "6386694a-6d4c-4a69-8388-52f34a3484d2") : [secret "agc-cred-371bb2bc1d0f" not found, secret "agc-cred-7fc5c1cab2ff" not found] Nov 03 01:15:55.680349 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:55.680318 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a395bce9-flxhm" Nov 03 01:15:55.777630 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:55.777580 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0d13eee-83f0-4cd5-80d0-0a7e6d60db18-kube-api-access-sxfck\" (UniqueName: \"kubernetes.io/projected/e0d13eee-83f0-4cd5-80d0-0a7e6d60db18-kube-api-access-sxfck\") pod \"e0d13eee-83f0-4cd5-80d0-0a7e6d60db18\" (UID: \"e0d13eee-83f0-4cd5-80d0-0a7e6d60db18\") " Nov 03 01:15:55.777816 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:55.777712 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e0d13eee-83f0-4cd5-80d0-0a7e6d60db18-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0d13eee-83f0-4cd5-80d0-0a7e6d60db18-maas-api-xks-ca\") pod \"e0d13eee-83f0-4cd5-80d0-0a7e6d60db18\" (UID: \"e0d13eee-83f0-4cd5-80d0-0a7e6d60db18\") " Nov 03 01:15:55.777816 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:55.777755 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0d13eee-83f0-4cd5-80d0-0a7e6d60db18-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0d13eee-83f0-4cd5-80d0-0a7e6d60db18-maas-api-openshift-ca\") pod \"e0d13eee-83f0-4cd5-80d0-0a7e6d60db18\" (UID: \"e0d13eee-83f0-4cd5-80d0-0a7e6d60db18\") " Nov 03 01:15:55.778162 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:55.778129 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0d13eee-83f0-4cd5-80d0-0a7e6d60db18-maas-api-openshift-ca" pod "e0d13eee-83f0-4cd5-80d0-0a7e6d60db18" (UID: "e0d13eee-83f0-4cd5-80d0-0a7e6d60db18"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:15:55.779053 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:55.779024 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0d13eee-83f0-4cd5-80d0-0a7e6d60db18-maas-api-openshift-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:15:55.779970 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:55.779945 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0d13eee-83f0-4cd5-80d0-0a7e6d60db18-kube-api-access-sxfck" pod "e0d13eee-83f0-4cd5-80d0-0a7e6d60db18" (UID: "e0d13eee-83f0-4cd5-80d0-0a7e6d60db18"). InnerVolumeSpecName "kube-api-access-sxfck". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:15:55.780081 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:55.779988 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0d13eee-83f0-4cd5-80d0-0a7e6d60db18-maas-api-xks-ca" pod "e0d13eee-83f0-4cd5-80d0-0a7e6d60db18" (UID: "e0d13eee-83f0-4cd5-80d0-0a7e6d60db18"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:15:55.880461 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:55.880428 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxfck\" (UniqueName: \"kubernetes.io/projected/e0d13eee-83f0-4cd5-80d0-0a7e6d60db18-kube-api-access-sxfck\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:15:55.880461 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:55.880457 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0d13eee-83f0-4cd5-80d0-0a7e6d60db18-maas-api-xks-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:15:56.442621 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:56.442590 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a395bce9-flxhm" Nov 03 01:15:56.442811 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:56.442626 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a395bce9-flxhm" event={"ID":"e0d13eee-83f0-4cd5-80d0-0a7e6d60db18","Type":"ContainerDied","Data":"855b957bcac6c7a86e01195bddb68d5cfd0282cf0862c8b605931ffdc45bfb6b"} Nov 03 01:15:56.442811 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:56.442677 4421 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="855b957bcac6c7a86e01195bddb68d5cfd0282cf0862c8b605931ffdc45bfb6b" Nov 03 01:15:56.598064 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:15:56.598016 4421 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-3c5983/agc-cred-371bb2bc1d0f: secret "agc-cred-371bb2bc1d0f" not found Nov 03 01:15:56.598064 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:15:56.598062 4421 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-3c5983/agc-cred-7fc5c1cab2ff: secret "agc-cred-7fc5c1cab2ff" not found Nov 03 01:15:56.598650 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:15:56.598082 4421 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-78c872f4s: [secret "agc-cred-371bb2bc1d0f" not found, secret "agc-cred-7fc5c1cab2ff" not found] Nov 03 01:15:56.598650 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:15:56.598169 4421 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6386694a-6d4c-4a69-8388-52f34a3484d2-provider-credentials podName:6386694a-6d4c-4a69-8388-52f34a3484d2 nodeName:}" failed. No retries permitted until 2026-11-03 01:15:58.598147741 +0000 UTC m=+2100.769150830 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/6386694a-6d4c-4a69-8388-52f34a3484d2-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-3c5983-78c872f4s" (UID: "6386694a-6d4c-4a69-8388-52f34a3484d2") : [secret "agc-cred-371bb2bc1d0f" not found, secret "agc-cred-7fc5c1cab2ff" not found] Nov 03 01:15:56.606886 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:56.606826 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-5779vg5bt" podUID="c9a4975c-6ec7-43c0-aaea-32c45c400d42" containerName="payload-processing" containerID="cri-o://01e63850bbc26fefd2e501e72191bad3541984d2eb209565405433d8fecaf94a" gracePeriod=25 Nov 03 01:15:56.961872 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:56.961844 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-5779vg5bt" Nov 03 01:15:56.993986 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:56.993912 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-6d9btqzxk"] Nov 03 01:15:56.995340 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:56.995311 4421 cpu_manager.go:414] "containerMap: removing container" podUID="c9a4975c-6ec7-43c0-aaea-32c45c400d42" containerName="payload-processing" Nov 03 01:15:56.995340 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:56.995338 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9a4975c-6ec7-43c0-aaea-32c45c400d42" containerName="payload-processing" Nov 03 01:15:56.995503 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:56.995385 4421 cpu_manager.go:414] "containerMap: removing container" podUID="e0d13eee-83f0-4cd5-80d0-0a7e6d60db18" containerName="revoke-keys" Nov 03 01:15:56.995503 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:56.995394 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0d13eee-83f0-4cd5-80d0-0a7e6d60db18" containerName="revoke-keys" Nov 03 01:15:56.995618 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:56.995603 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9a4975c-6ec7-43c0-aaea-32c45c400d42" containerName="payload-processing" Nov 03 01:15:56.995658 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:56.995627 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0d13eee-83f0-4cd5-80d0-0a7e6d60db18" containerName="revoke-keys" Nov 03 01:15:57.000309 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:57.000267 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-6d9btqzxk" Nov 03 01:15:57.011962 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:57.011931 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-6d9btqzxk"] Nov 03 01:15:57.111602 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:57.111567 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9a4975c-6ec7-43c0-aaea-32c45c400d42-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9a4975c-6ec7-43c0-aaea-32c45c400d42-plugins-config-volume\") pod \"c9a4975c-6ec7-43c0-aaea-32c45c400d42\" (UID: \"c9a4975c-6ec7-43c0-aaea-32c45c400d42\") " Nov 03 01:15:57.111835 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:57.111823 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9a4975c-6ec7-43c0-aaea-32c45c400d42-provider-credentials\" (UniqueName: \"kubernetes.io/projected/c9a4975c-6ec7-43c0-aaea-32c45c400d42-provider-credentials\") pod \"c9a4975c-6ec7-43c0-aaea-32c45c400d42\" (UID: \"c9a4975c-6ec7-43c0-aaea-32c45c400d42\") " Nov 03 01:15:57.111923 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:57.111909 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9a4975c-6ec7-43c0-aaea-32c45c400d42-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c9a4975c-6ec7-43c0-aaea-32c45c400d42-routing-overlay-volume\") pod \"c9a4975c-6ec7-43c0-aaea-32c45c400d42\" (UID: \"c9a4975c-6ec7-43c0-aaea-32c45c400d42\") " Nov 03 01:15:57.111975 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:57.111959 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9a4975c-6ec7-43c0-aaea-32c45c400d42-kube-api-access-gbl66\" (UniqueName: \"kubernetes.io/projected/c9a4975c-6ec7-43c0-aaea-32c45c400d42-kube-api-access-gbl66\") pod \"c9a4975c-6ec7-43c0-aaea-32c45c400d42\" (UID: \"c9a4975c-6ec7-43c0-aaea-32c45c400d42\") " Nov 03 01:15:57.112080 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:57.112043 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9a4975c-6ec7-43c0-aaea-32c45c400d42-plugins-config-volume" pod "c9a4975c-6ec7-43c0-aaea-32c45c400d42" (UID: "c9a4975c-6ec7-43c0-aaea-32c45c400d42"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:15:57.112385 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:57.112360 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9a4975c-6ec7-43c0-aaea-32c45c400d42-routing-overlay-volume" pod "c9a4975c-6ec7-43c0-aaea-32c45c400d42" (UID: "c9a4975c-6ec7-43c0-aaea-32c45c400d42"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:15:57.112560 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:57.112539 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/396f111c-07d7-440c-853a-a9c48739892f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-57079b-6d9btqzxk\" (UID: \"396f111c-07d7-440c-853a-a9c48739892f\") " pod="ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-6d9btqzxk" Nov 03 01:15:57.112819 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:57.112799 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/396f111c-07d7-440c-853a-a9c48739892f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-57079b-6d9btqzxk\" (UID: \"396f111c-07d7-440c-853a-a9c48739892f\") " pod="ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-6d9btqzxk" Nov 03 01:15:57.113221 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:57.113197 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/396f111c-07d7-440c-853a-a9c48739892f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-57079b-6d9btqzxk\" (UID: \"396f111c-07d7-440c-853a-a9c48739892f\") " pod="ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-6d9btqzxk" Nov 03 01:15:57.113516 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:57.113490 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgsrh\" (UniqueName: \"kubernetes.io/projected/396f111c-07d7-440c-853a-a9c48739892f-kube-api-access-mgsrh\") pod \"payload-processing-external-model-e2e-extmod-b-57079b-6d9btqzxk\" (UID: \"396f111c-07d7-440c-853a-a9c48739892f\") " pod="ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-6d9btqzxk" Nov 03 01:15:57.113843 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:57.113822 4421 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c9a4975c-6ec7-43c0-aaea-32c45c400d42-routing-overlay-volume\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:15:57.113918 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:57.113851 4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9a4975c-6ec7-43c0-aaea-32c45c400d42-plugins-config-volume\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:15:57.114424 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:57.114399 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9a4975c-6ec7-43c0-aaea-32c45c400d42-provider-credentials" pod "c9a4975c-6ec7-43c0-aaea-32c45c400d42" (UID: "c9a4975c-6ec7-43c0-aaea-32c45c400d42"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:15:57.114725 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:57.114688 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9a4975c-6ec7-43c0-aaea-32c45c400d42-kube-api-access-gbl66" pod "c9a4975c-6ec7-43c0-aaea-32c45c400d42" (UID: "c9a4975c-6ec7-43c0-aaea-32c45c400d42"). InnerVolumeSpecName "kube-api-access-gbl66". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:15:57.215354 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:57.215260 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/396f111c-07d7-440c-853a-a9c48739892f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-57079b-6d9btqzxk\" (UID: \"396f111c-07d7-440c-853a-a9c48739892f\") " pod="ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-6d9btqzxk" Nov 03 01:15:57.215577 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:57.215489 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/396f111c-07d7-440c-853a-a9c48739892f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-57079b-6d9btqzxk\" (UID: \"396f111c-07d7-440c-853a-a9c48739892f\") " pod="ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-6d9btqzxk" Nov 03 01:15:57.215650 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:57.215599 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/396f111c-07d7-440c-853a-a9c48739892f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-57079b-6d9btqzxk\" (UID: \"396f111c-07d7-440c-853a-a9c48739892f\") " pod="ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-6d9btqzxk" Nov 03 01:15:57.215650 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:57.215642 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgsrh\" (UniqueName: \"kubernetes.io/projected/396f111c-07d7-440c-853a-a9c48739892f-kube-api-access-mgsrh\") pod \"payload-processing-external-model-e2e-extmod-b-57079b-6d9btqzxk\" (UID: \"396f111c-07d7-440c-853a-a9c48739892f\") " pod="ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-6d9btqzxk" Nov 03 01:15:57.215793 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:57.215776 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbl66\" (UniqueName: \"kubernetes.io/projected/c9a4975c-6ec7-43c0-aaea-32c45c400d42-kube-api-access-gbl66\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:15:57.215847 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:57.215801 4421 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c9a4975c-6ec7-43c0-aaea-32c45c400d42-provider-credentials\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:15:57.216212 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:57.216181 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/396f111c-07d7-440c-853a-a9c48739892f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-57079b-6d9btqzxk\" (UID: \"396f111c-07d7-440c-853a-a9c48739892f\") " pod="ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-6d9btqzxk" Nov 03 01:15:57.216396 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:57.216267 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/396f111c-07d7-440c-853a-a9c48739892f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-57079b-6d9btqzxk\" (UID: \"396f111c-07d7-440c-853a-a9c48739892f\") " pod="ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-6d9btqzxk" Nov 03 01:15:57.217904 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:57.217885 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/396f111c-07d7-440c-853a-a9c48739892f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-57079b-6d9btqzxk\" (UID: \"396f111c-07d7-440c-853a-a9c48739892f\") " pod="ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-6d9btqzxk" Nov 03 01:15:57.223628 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:57.223605 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgsrh\" (UniqueName: \"kubernetes.io/projected/396f111c-07d7-440c-853a-a9c48739892f-kube-api-access-mgsrh\") pod \"payload-processing-external-model-e2e-extmod-b-57079b-6d9btqzxk\" (UID: \"396f111c-07d7-440c-853a-a9c48739892f\") " pod="ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-6d9btqzxk" Nov 03 01:15:57.326371 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:57.326331 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-6d9btqzxk" Nov 03 01:15:57.459497 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:57.459451 4421 generic.go:356] "Generic (PLEG): container finished" podID="c9a4975c-6ec7-43c0-aaea-32c45c400d42" containerID="01e63850bbc26fefd2e501e72191bad3541984d2eb209565405433d8fecaf94a" exitCode=0 Nov 03 01:15:57.459758 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:57.459731 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-5779vg5bt" event={"ID":"c9a4975c-6ec7-43c0-aaea-32c45c400d42","Type":"ContainerDied","Data":"01e63850bbc26fefd2e501e72191bad3541984d2eb209565405433d8fecaf94a"} Nov 03 01:15:57.459851 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:57.459800 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-5779vg5bt" event={"ID":"c9a4975c-6ec7-43c0-aaea-32c45c400d42","Type":"ContainerDied","Data":"3562f44655e92707752ac3b89423cdcbd8cafc4ee045670272455f440b7dc8cb"} Nov 03 01:15:57.459851 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:57.459836 4421 scope.go:122] "RemoveContainer" containerID="01e63850bbc26fefd2e501e72191bad3541984d2eb209565405433d8fecaf94a" Nov 03 01:15:57.460025 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:57.460008 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-5779vg5bt" Nov 03 01:15:57.477989 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:57.477967 4421 scope.go:122] "RemoveContainer" containerID="01e63850bbc26fefd2e501e72191bad3541984d2eb209565405433d8fecaf94a" Nov 03 01:15:57.478532 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:15:57.478506 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01e63850bbc26fefd2e501e72191bad3541984d2eb209565405433d8fecaf94a\": container with ID starting with 01e63850bbc26fefd2e501e72191bad3541984d2eb209565405433d8fecaf94a not found: ID does not exist" containerID="01e63850bbc26fefd2e501e72191bad3541984d2eb209565405433d8fecaf94a" Nov 03 01:15:57.478650 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:57.478543 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01e63850bbc26fefd2e501e72191bad3541984d2eb209565405433d8fecaf94a"} err="failed to get container status \"01e63850bbc26fefd2e501e72191bad3541984d2eb209565405433d8fecaf94a\": rpc error: code = NotFound desc = could not find container \"01e63850bbc26fefd2e501e72191bad3541984d2eb209565405433d8fecaf94a\": container with ID starting with 01e63850bbc26fefd2e501e72191bad3541984d2eb209565405433d8fecaf94a not found: ID does not exist" Nov 03 01:15:57.491425 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:57.491389 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-5779vg5bt"] Nov 03 01:15:57.496492 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:57.496464 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-5779vg5bt"] Nov 03 01:15:57.626974 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:15:57.626939 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod396f111c_07d7_440c_853a_a9c48739892f.slice/crio-7b410e97b9ab6be01b8af7774bfc768e1a90ef1a465532fd6598ceab0466a7ac WatchSource:0}: Error finding container 7b410e97b9ab6be01b8af7774bfc768e1a90ef1a465532fd6598ceab0466a7ac: Status 404 returned error can't find the container with id 7b410e97b9ab6be01b8af7774bfc768e1a90ef1a465532fd6598ceab0466a7ac Nov 03 01:15:57.627959 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:57.627929 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-6d9btqzxk"] Nov 03 01:15:57.982889 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:57.982813 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9a4975c-6ec7-43c0-aaea-32c45c400d42" path="/var/lib/kubelet/pods/c9a4975c-6ec7-43c0-aaea-32c45c400d42/volumes" Nov 03 01:15:58.469835 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:58.469792 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-6d9btqzxk" event={"ID":"396f111c-07d7-440c-853a-a9c48739892f","Type":"ContainerStarted","Data":"0fe0c20f7f170e58a8e1d6d722358d643d6e2cd129942a6ba643f5d7ce0ca1e5"} Nov 03 01:15:58.470047 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:58.469864 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-6d9btqzxk" Nov 03 01:15:58.470047 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:58.469892 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-6d9btqzxk" event={"ID":"396f111c-07d7-440c-853a-a9c48739892f","Type":"ContainerStarted","Data":"7b410e97b9ab6be01b8af7774bfc768e1a90ef1a465532fd6598ceab0466a7ac"} Nov 03 01:15:58.493338 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:58.493251 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-6d9btqzxk" podStartSLOduration=7.493232775 podStartE2EDuration="7.493232775s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:15:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:15:58.487475018 +0000 UTC m=+2100.658478117" watchObservedRunningTime="2026-11-03 01:15:58.493232775 +0000 UTC m=+2100.664235872" Nov 03 01:15:58.689208 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:15:58.689178 4421 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-3c5983/agc-cred-371bb2bc1d0f: secret "agc-cred-371bb2bc1d0f" not found Nov 03 01:15:58.689208 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:15:58.689211 4421 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-3c5983/agc-cred-7fc5c1cab2ff: secret "agc-cred-7fc5c1cab2ff" not found Nov 03 01:15:58.689620 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:15:58.689227 4421 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-78c872f4s: [secret "agc-cred-371bb2bc1d0f" not found, secret "agc-cred-7fc5c1cab2ff" not found] Nov 03 01:15:58.689620 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:15:58.689303 4421 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6386694a-6d4c-4a69-8388-52f34a3484d2-provider-credentials podName:6386694a-6d4c-4a69-8388-52f34a3484d2 nodeName:}" failed. No retries permitted until 2026-11-03 01:16:02.689268741 +0000 UTC m=+2104.860271829 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/6386694a-6d4c-4a69-8388-52f34a3484d2-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-3c5983-78c872f4s" (UID: "6386694a-6d4c-4a69-8388-52f34a3484d2") : [secret "agc-cred-371bb2bc1d0f" not found, secret "agc-cred-7fc5c1cab2ff" not found] Nov 03 01:15:59.236212 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:59.236170 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-66662d95-t4k4k"] Nov 03 01:15:59.241056 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:59.241034 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66662d95-t4k4k" Nov 03 01:15:59.247760 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:59.247688 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6q4xp\"" Nov 03 01:15:59.248509 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:59.248481 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:15:59.256517 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:59.256484 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-66662d95-t4k4k"] Nov 03 01:15:59.300829 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:59.300790 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a36f95f-aa93-4151-893b-fcc702a5fb9a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-66662d95-t4k4k\" (UID: \"3a36f95f-aa93-4151-893b-fcc702a5fb9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66662d95-t4k4k" Nov 03 01:15:59.301028 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:59.301005 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a36f95f-aa93-4151-893b-fcc702a5fb9a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-66662d95-t4k4k\" (UID: \"3a36f95f-aa93-4151-893b-fcc702a5fb9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66662d95-t4k4k" Nov 03 01:15:59.301221 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:59.301205 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2429\" (UniqueName: \"kubernetes.io/projected/3a36f95f-aa93-4151-893b-fcc702a5fb9a-kube-api-access-d2429\") pod \"maas-api-revoke-sub-66662d95-t4k4k\" (UID: \"3a36f95f-aa93-4151-893b-fcc702a5fb9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66662d95-t4k4k" Nov 03 01:15:59.403504 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:59.403470 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2429\" (UniqueName: \"kubernetes.io/projected/3a36f95f-aa93-4151-893b-fcc702a5fb9a-kube-api-access-d2429\") pod \"maas-api-revoke-sub-66662d95-t4k4k\" (UID: \"3a36f95f-aa93-4151-893b-fcc702a5fb9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66662d95-t4k4k" Nov 03 01:15:59.403692 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:59.403642 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a36f95f-aa93-4151-893b-fcc702a5fb9a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-66662d95-t4k4k\" (UID: \"3a36f95f-aa93-4151-893b-fcc702a5fb9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66662d95-t4k4k" Nov 03 01:15:59.403857 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:59.403840 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a36f95f-aa93-4151-893b-fcc702a5fb9a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-66662d95-t4k4k\" (UID: \"3a36f95f-aa93-4151-893b-fcc702a5fb9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66662d95-t4k4k" Nov 03 01:15:59.404288 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:59.404257 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a36f95f-aa93-4151-893b-fcc702a5fb9a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-66662d95-t4k4k\" (UID: \"3a36f95f-aa93-4151-893b-fcc702a5fb9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66662d95-t4k4k" Nov 03 01:15:59.406225 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:59.406201 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a36f95f-aa93-4151-893b-fcc702a5fb9a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-66662d95-t4k4k\" (UID: \"3a36f95f-aa93-4151-893b-fcc702a5fb9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66662d95-t4k4k" Nov 03 01:15:59.412786 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:59.412766 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2429\" (UniqueName: \"kubernetes.io/projected/3a36f95f-aa93-4151-893b-fcc702a5fb9a-kube-api-access-d2429\") pod \"maas-api-revoke-sub-66662d95-t4k4k\" (UID: \"3a36f95f-aa93-4151-893b-fcc702a5fb9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66662d95-t4k4k" Nov 03 01:15:59.585327 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:59.585231 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66662d95-t4k4k" Nov 03 01:15:59.875822 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:15:59.875788 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a36f95f_aa93_4151_893b_fcc702a5fb9a.slice/crio-fca128260d8cb25e67687cd81e0e78656cd6ed8fa18a47bf0fe6b4a6a7155ff8 WatchSource:0}: Error finding container fca128260d8cb25e67687cd81e0e78656cd6ed8fa18a47bf0fe6b4a6a7155ff8: Status 404 returned error can't find the container with id fca128260d8cb25e67687cd81e0e78656cd6ed8fa18a47bf0fe6b4a6a7155ff8 Nov 03 01:15:59.876895 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:15:59.876811 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-66662d95-t4k4k"] Nov 03 01:16:00.023627 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:00.023595 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9899cc30-t78rv"] Nov 03 01:16:00.029500 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:00.029465 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9899cc30-t78rv"] Nov 03 01:16:00.490843 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:00.490752 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66662d95-t4k4k" event={"ID":"3a36f95f-aa93-4151-893b-fcc702a5fb9a","Type":"ContainerStarted","Data":"fca128260d8cb25e67687cd81e0e78656cd6ed8fa18a47bf0fe6b4a6a7155ff8"} Nov 03 01:16:01.502232 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:01.502186 4421 generic.go:356] "Generic (PLEG): container finished" podID="3a36f95f-aa93-4151-893b-fcc702a5fb9a" containerID="b7edec4b4715df42d415edaa293dffb409339ef52b7aedee62d053817135a056" exitCode=0 Nov 03 01:16:01.502748 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:01.502309 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66662d95-t4k4k" event={"ID":"3a36f95f-aa93-4151-893b-fcc702a5fb9a","Type":"ContainerDied","Data":"b7edec4b4715df42d415edaa293dffb409339ef52b7aedee62d053817135a056"} Nov 03 01:16:02.056624 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:02.056400 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5fefa28b-08b0-4141-9f8a-8c411ac1990f" path="/var/lib/kubelet/pods/5fefa28b-08b0-4141-9f8a-8c411ac1990f/volumes" Nov 03 01:16:02.174794 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:02.174760 4421 scope.go:122] "RemoveContainer" containerID="252e7fa5db41b396fb3f979d3fa010e54ab33429947cae9b2951732b0a36914a" Nov 03 01:16:02.194885 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:02.194247 4421 scope.go:122] "RemoveContainer" containerID="9a4af3515a2e301e5de6d37314f1f35596dc6f1fd40ee6b065ed7877813b235b" Nov 03 01:16:02.566071 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:02.566050 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-tt9jg"] Nov 03 01:16:02.566800 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:02.566305 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-tt9jg" podUID="67dd8eb6-fd6b-4a86-a8e4-b393e62044bf" containerName="istio-proxy" containerID="cri-o://91e7a11e0d31b71fc299a4faa5c90530dbcef8e9a2a767fd2524b061064134b6" gracePeriod=30 Nov 03 01:16:02.732089 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:16:02.732047 4421 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-3c5983/agc-cred-371bb2bc1d0f: secret "agc-cred-371bb2bc1d0f" not found Nov 03 01:16:02.732267 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:16:02.732120 4421 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-3c5983/agc-cred-7fc5c1cab2ff: secret "agc-cred-7fc5c1cab2ff" not found Nov 03 01:16:02.732267 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:16:02.732140 4421 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-78c872f4s: [secret "agc-cred-371bb2bc1d0f" not found, secret "agc-cred-7fc5c1cab2ff" not found] Nov 03 01:16:02.732267 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:16:02.732199 4421 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6386694a-6d4c-4a69-8388-52f34a3484d2-provider-credentials podName:6386694a-6d4c-4a69-8388-52f34a3484d2 nodeName:}" failed. No retries permitted until 2026-11-03 01:16:10.732183136 +0000 UTC m=+2112.903186210 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/6386694a-6d4c-4a69-8388-52f34a3484d2-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-3c5983-78c872f4s" (UID: "6386694a-6d4c-4a69-8388-52f34a3484d2") : [secret "agc-cred-371bb2bc1d0f" not found, secret "agc-cred-7fc5c1cab2ff" not found] Nov 03 01:16:02.788983 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:02.788951 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-h2p2p"] Nov 03 01:16:02.789199 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:02.789174 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-h2p2p" podUID="5bd6d60e-043b-47ba-beb1-08b1eaace690" containerName="istio-proxy" containerID="cri-o://bd1518b1ffc13c09f9f41b5e8c7aebdfa23b77711a7dd2f01a1878c5a1f832e5" gracePeriod=30 Nov 03 01:16:02.833066 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:02.833028 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66662d95-t4k4k" Nov 03 01:16:02.934246 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:02.934215 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a36f95f-aa93-4151-893b-fcc702a5fb9a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a36f95f-aa93-4151-893b-fcc702a5fb9a-maas-api-openshift-ca\") pod \"3a36f95f-aa93-4151-893b-fcc702a5fb9a\" (UID: \"3a36f95f-aa93-4151-893b-fcc702a5fb9a\") " Nov 03 01:16:02.934488 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:02.934392 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a36f95f-aa93-4151-893b-fcc702a5fb9a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a36f95f-aa93-4151-893b-fcc702a5fb9a-maas-api-xks-ca\") pod \"3a36f95f-aa93-4151-893b-fcc702a5fb9a\" (UID: \"3a36f95f-aa93-4151-893b-fcc702a5fb9a\") " Nov 03 01:16:02.934642 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:02.934616 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a36f95f-aa93-4151-893b-fcc702a5fb9a-maas-api-openshift-ca" pod "3a36f95f-aa93-4151-893b-fcc702a5fb9a" (UID: "3a36f95f-aa93-4151-893b-fcc702a5fb9a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:16:02.934740 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:02.934659 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a36f95f-aa93-4151-893b-fcc702a5fb9a-kube-api-access-d2429\" (UniqueName: \"kubernetes.io/projected/3a36f95f-aa93-4151-893b-fcc702a5fb9a-kube-api-access-d2429\") pod \"3a36f95f-aa93-4151-893b-fcc702a5fb9a\" (UID: \"3a36f95f-aa93-4151-893b-fcc702a5fb9a\") " Nov 03 01:16:02.935717 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:02.935688 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a36f95f-aa93-4151-893b-fcc702a5fb9a-maas-api-openshift-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:16:02.936959 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:02.936925 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a36f95f-aa93-4151-893b-fcc702a5fb9a-kube-api-access-d2429" pod "3a36f95f-aa93-4151-893b-fcc702a5fb9a" (UID: "3a36f95f-aa93-4151-893b-fcc702a5fb9a"). InnerVolumeSpecName "kube-api-access-d2429". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:16:02.938475 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:02.938450 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a36f95f-aa93-4151-893b-fcc702a5fb9a-maas-api-xks-ca" pod "3a36f95f-aa93-4151-893b-fcc702a5fb9a" (UID: "3a36f95f-aa93-4151-893b-fcc702a5fb9a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:16:03.038210 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:03.038179 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a36f95f-aa93-4151-893b-fcc702a5fb9a-maas-api-xks-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:16:03.038210 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:03.038207 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2429\" (UniqueName: \"kubernetes.io/projected/3a36f95f-aa93-4151-893b-fcc702a5fb9a-kube-api-access-d2429\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:16:03.552621 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:03.552585 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66662d95-t4k4k" Nov 03 01:16:03.552621 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:03.552615 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66662d95-t4k4k" event={"ID":"3a36f95f-aa93-4151-893b-fcc702a5fb9a","Type":"ContainerDied","Data":"fca128260d8cb25e67687cd81e0e78656cd6ed8fa18a47bf0fe6b4a6a7155ff8"} Nov 03 01:16:03.552908 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:03.552662 4421 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fca128260d8cb25e67687cd81e0e78656cd6ed8fa18a47bf0fe6b4a6a7155ff8" Nov 03 01:16:03.988833 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:03.988734 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-6d9btqzxk"] Nov 03 01:16:03.993571 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:03.993543 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-6d9btqzxk" Nov 03 01:16:04.481301 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:04.468609 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-679574xvq" Nov 03 01:16:04.640299 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:04.635752 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-78c872f4s"] Nov 03 01:16:07.881232 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:07.881192 4421 patch_prober.go:29] interesting pod/e2e-worker-w3-83755a-openshift-default-c86f9565c-h2p2p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Nov 03 01:16:07.881671 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:07.881254 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-h2p2p" podUID="5bd6d60e-043b-47ba-beb1-08b1eaace690" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Nov 03 01:16:07.932586 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:07.932557 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-tt9jg" Nov 03 01:16:07.966604 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:07.966562 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-b894qjmvh"] Nov 03 01:16:07.967536 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:07.967512 4421 cpu_manager.go:414] "containerMap: removing container" podUID="67dd8eb6-fd6b-4a86-a8e4-b393e62044bf" containerName="istio-proxy" Nov 03 01:16:07.967536 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:07.967532 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67dd8eb6-fd6b-4a86-a8e4-b393e62044bf" containerName="istio-proxy" Nov 03 01:16:07.967763 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:07.967558 4421 cpu_manager.go:414] "containerMap: removing container" podUID="3a36f95f-aa93-4151-893b-fcc702a5fb9a" containerName="revoke-keys" Nov 03 01:16:07.967763 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:07.967566 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a36f95f-aa93-4151-893b-fcc702a5fb9a" containerName="revoke-keys" Nov 03 01:16:07.967763 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:07.967705 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="67dd8eb6-fd6b-4a86-a8e4-b393e62044bf" containerName="istio-proxy" Nov 03 01:16:07.967763 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:07.967717 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a36f95f-aa93-4151-893b-fcc702a5fb9a" containerName="revoke-keys" Nov 03 01:16:07.982138 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:07.981565 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-b894qjmvh" Nov 03 01:16:07.985531 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:07.985497 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-d25759\"/\"openshift-service-ca.crt\"" Nov 03 01:16:07.985700 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:07.985583 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-d25759\"/\"e2e-cfl-e257d5-provider-cred\"" Nov 03 01:16:07.985700 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:07.985651 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-d25759\"/\"payload-processing-external-model-e2e--4d491765-dockercfg-btrfs\"" Nov 03 01:16:07.985700 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:07.985694 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-d25759\"/\"kube-root-ca.crt\"" Nov 03 01:16:07.986048 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:07.985937 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-d25759\"/\"payload-processing-external-model-plugins-e2e-extmod-a-d25759\"" Nov 03 01:16:07.986149 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:07.986089 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-d25759\"/\"routing-overlay\"" Nov 03 01:16:08.006078 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:08.006055 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-b894qjmvh"] Nov 03 01:16:08.015547 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:08.015523 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67dd8eb6-fd6b-4a86-a8e4-b393e62044bf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67dd8eb6-fd6b-4a86-a8e4-b393e62044bf-workload-socket\") pod \"67dd8eb6-fd6b-4a86-a8e4-b393e62044bf\" (UID: \"67dd8eb6-fd6b-4a86-a8e4-b393e62044bf\") " Nov 03 01:16:08.015691 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:08.015557 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67dd8eb6-fd6b-4a86-a8e4-b393e62044bf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67dd8eb6-fd6b-4a86-a8e4-b393e62044bf-istio-data\") pod \"67dd8eb6-fd6b-4a86-a8e4-b393e62044bf\" (UID: \"67dd8eb6-fd6b-4a86-a8e4-b393e62044bf\") " Nov 03 01:16:08.015691 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:08.015623 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67dd8eb6-fd6b-4a86-a8e4-b393e62044bf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67dd8eb6-fd6b-4a86-a8e4-b393e62044bf-workload-certs\") pod \"67dd8eb6-fd6b-4a86-a8e4-b393e62044bf\" (UID: \"67dd8eb6-fd6b-4a86-a8e4-b393e62044bf\") " Nov 03 01:16:08.015810 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:08.015713 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/67dd8eb6-fd6b-4a86-a8e4-b393e62044bf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67dd8eb6-fd6b-4a86-a8e4-b393e62044bf-istio-podinfo\") pod \"67dd8eb6-fd6b-4a86-a8e4-b393e62044bf\" (UID: \"67dd8eb6-fd6b-4a86-a8e4-b393e62044bf\") " Nov 03 01:16:08.015810 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:08.015792 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67dd8eb6-fd6b-4a86-a8e4-b393e62044bf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67dd8eb6-fd6b-4a86-a8e4-b393e62044bf-istio-envoy\") pod \"67dd8eb6-fd6b-4a86-a8e4-b393e62044bf\" (UID: \"67dd8eb6-fd6b-4a86-a8e4-b393e62044bf\") " Nov 03 01:16:08.015932 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:08.015897 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67dd8eb6-fd6b-4a86-a8e4-b393e62044bf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67dd8eb6-fd6b-4a86-a8e4-b393e62044bf-istiod-ca-cert\") pod \"67dd8eb6-fd6b-4a86-a8e4-b393e62044bf\" (UID: \"67dd8eb6-fd6b-4a86-a8e4-b393e62044bf\") " Nov 03 01:16:08.015932 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:08.015916 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67dd8eb6-fd6b-4a86-a8e4-b393e62044bf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67dd8eb6-fd6b-4a86-a8e4-b393e62044bf-credential-socket\") pod \"67dd8eb6-fd6b-4a86-a8e4-b393e62044bf\" (UID: \"67dd8eb6-fd6b-4a86-a8e4-b393e62044bf\") " Nov 03 01:16:08.016035 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:08.015966 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67dd8eb6-fd6b-4a86-a8e4-b393e62044bf-istio-token\" (UniqueName: \"kubernetes.io/projected/67dd8eb6-fd6b-4a86-a8e4-b393e62044bf-istio-token\") pod \"67dd8eb6-fd6b-4a86-a8e4-b393e62044bf\" (UID: \"67dd8eb6-fd6b-4a86-a8e4-b393e62044bf\") " Nov 03 01:16:08.016089 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:08.016055 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67dd8eb6-fd6b-4a86-a8e4-b393e62044bf-kube-api-access-jt5xh\" (UniqueName: \"kubernetes.io/projected/67dd8eb6-fd6b-4a86-a8e4-b393e62044bf-kube-api-access-jt5xh\") pod \"67dd8eb6-fd6b-4a86-a8e4-b393e62044bf\" (UID: \"67dd8eb6-fd6b-4a86-a8e4-b393e62044bf\") " Nov 03 01:16:08.016655 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:08.016630 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cdc5358a-3f4b-43e8-b889-38fa85711fbd-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d25759-b894qjmvh\" (UID: \"cdc5358a-3f4b-43e8-b889-38fa85711fbd\") " pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-b894qjmvh" Nov 03 01:16:08.016761 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:08.016660 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67dd8eb6-fd6b-4a86-a8e4-b393e62044bf-istiod-ca-cert" pod "67dd8eb6-fd6b-4a86-a8e4-b393e62044bf" (UID: "67dd8eb6-fd6b-4a86-a8e4-b393e62044bf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:16:08.016846 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:08.016830 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67dd8eb6-fd6b-4a86-a8e4-b393e62044bf-credential-socket" pod "67dd8eb6-fd6b-4a86-a8e4-b393e62044bf" (UID: "67dd8eb6-fd6b-4a86-a8e4-b393e62044bf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:16:08.016971 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:08.016948 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cdc5358a-3f4b-43e8-b889-38fa85711fbd-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-d25759-b894qjmvh\" (UID: \"cdc5358a-3f4b-43e8-b889-38fa85711fbd\") " pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-b894qjmvh" Nov 03 01:16:08.017185 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:08.017139 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8vp6\" (UniqueName: \"kubernetes.io/projected/cdc5358a-3f4b-43e8-b889-38fa85711fbd-kube-api-access-x8vp6\") pod \"payload-processing-external-model-e2e-extmod-a-d25759-b894qjmvh\" (UID: \"cdc5358a-3f4b-43e8-b889-38fa85711fbd\") " pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-b894qjmvh" Nov 03 01:16:08.017256 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:08.017218 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdc5358a-3f4b-43e8-b889-38fa85711fbd-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d25759-b894qjmvh\" (UID: \"cdc5358a-3f4b-43e8-b889-38fa85711fbd\") " pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-b894qjmvh" Nov 03 01:16:08.017724 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:08.017702 4421 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67dd8eb6-fd6b-4a86-a8e4-b393e62044bf-istiod-ca-cert\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:16:08.017818 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:08.017730 4421 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67dd8eb6-fd6b-4a86-a8e4-b393e62044bf-credential-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:16:08.017880 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:08.017839 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67dd8eb6-fd6b-4a86-a8e4-b393e62044bf-workload-socket" pod "67dd8eb6-fd6b-4a86-a8e4-b393e62044bf" (UID: "67dd8eb6-fd6b-4a86-a8e4-b393e62044bf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:16:08.018066 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:08.018044 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67dd8eb6-fd6b-4a86-a8e4-b393e62044bf-istio-data" pod "67dd8eb6-fd6b-4a86-a8e4-b393e62044bf" (UID: "67dd8eb6-fd6b-4a86-a8e4-b393e62044bf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:16:08.018188 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:08.018168 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67dd8eb6-fd6b-4a86-a8e4-b393e62044bf-workload-certs" pod "67dd8eb6-fd6b-4a86-a8e4-b393e62044bf" (UID: "67dd8eb6-fd6b-4a86-a8e4-b393e62044bf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:16:08.019730 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:08.019624 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/67dd8eb6-fd6b-4a86-a8e4-b393e62044bf-istio-podinfo" pod "67dd8eb6-fd6b-4a86-a8e4-b393e62044bf" (UID: "67dd8eb6-fd6b-4a86-a8e4-b393e62044bf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:16:08.020166 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:08.020142 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67dd8eb6-fd6b-4a86-a8e4-b393e62044bf-istio-token" pod "67dd8eb6-fd6b-4a86-a8e4-b393e62044bf" (UID: "67dd8eb6-fd6b-4a86-a8e4-b393e62044bf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:16:08.020472 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:08.020437 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67dd8eb6-fd6b-4a86-a8e4-b393e62044bf-kube-api-access-jt5xh" pod "67dd8eb6-fd6b-4a86-a8e4-b393e62044bf" (UID: "67dd8eb6-fd6b-4a86-a8e4-b393e62044bf"). InnerVolumeSpecName "kube-api-access-jt5xh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:16:08.020692 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:08.020657 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67dd8eb6-fd6b-4a86-a8e4-b393e62044bf-istio-envoy" pod "67dd8eb6-fd6b-4a86-a8e4-b393e62044bf" (UID: "67dd8eb6-fd6b-4a86-a8e4-b393e62044bf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:16:08.200347 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:08.198969 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cdc5358a-3f4b-43e8-b889-38fa85711fbd-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d25759-b894qjmvh\" (UID: \"cdc5358a-3f4b-43e8-b889-38fa85711fbd\") " pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-b894qjmvh" Nov 03 01:16:08.200347 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:08.199090 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cdc5358a-3f4b-43e8-b889-38fa85711fbd-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-d25759-b894qjmvh\" (UID: \"cdc5358a-3f4b-43e8-b889-38fa85711fbd\") " pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-b894qjmvh" Nov 03 01:16:08.200347 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:08.199166 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8vp6\" (UniqueName: \"kubernetes.io/projected/cdc5358a-3f4b-43e8-b889-38fa85711fbd-kube-api-access-x8vp6\") pod \"payload-processing-external-model-e2e-extmod-a-d25759-b894qjmvh\" (UID: \"cdc5358a-3f4b-43e8-b889-38fa85711fbd\") " pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-b894qjmvh" Nov 03 01:16:08.200347 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:08.199193 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdc5358a-3f4b-43e8-b889-38fa85711fbd-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d25759-b894qjmvh\" (UID: \"cdc5358a-3f4b-43e8-b889-38fa85711fbd\") " pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-b894qjmvh" Nov 03 01:16:08.200347 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:08.199297 4421 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67dd8eb6-fd6b-4a86-a8e4-b393e62044bf-workload-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:16:08.200347 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:08.199309 4421 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67dd8eb6-fd6b-4a86-a8e4-b393e62044bf-istio-data\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:16:08.200347 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:08.199318 4421 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67dd8eb6-fd6b-4a86-a8e4-b393e62044bf-workload-certs\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:16:08.200347 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:08.199327 4421 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67dd8eb6-fd6b-4a86-a8e4-b393e62044bf-istio-podinfo\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:16:08.200347 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:08.199335 4421 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67dd8eb6-fd6b-4a86-a8e4-b393e62044bf-istio-envoy\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:16:08.200347 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:08.199345 4421 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67dd8eb6-fd6b-4a86-a8e4-b393e62044bf-istio-token\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:16:08.200347 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:08.199356 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jt5xh\" (UniqueName: \"kubernetes.io/projected/67dd8eb6-fd6b-4a86-a8e4-b393e62044bf-kube-api-access-jt5xh\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:16:08.201189 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:08.200913 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdc5358a-3f4b-43e8-b889-38fa85711fbd-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d25759-b894qjmvh\" (UID: \"cdc5358a-3f4b-43e8-b889-38fa85711fbd\") " pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-b894qjmvh" Nov 03 01:16:08.201189 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:08.200913 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cdc5358a-3f4b-43e8-b889-38fa85711fbd-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d25759-b894qjmvh\" (UID: \"cdc5358a-3f4b-43e8-b889-38fa85711fbd\") " pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-b894qjmvh" Nov 03 01:16:08.205227 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:08.205198 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cdc5358a-3f4b-43e8-b889-38fa85711fbd-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-d25759-b894qjmvh\" (UID: \"cdc5358a-3f4b-43e8-b889-38fa85711fbd\") " pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-b894qjmvh" Nov 03 01:16:08.208628 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:08.208580 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8vp6\" (UniqueName: \"kubernetes.io/projected/cdc5358a-3f4b-43e8-b889-38fa85711fbd-kube-api-access-x8vp6\") pod \"payload-processing-external-model-e2e-extmod-a-d25759-b894qjmvh\" (UID: \"cdc5358a-3f4b-43e8-b889-38fa85711fbd\") " pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-b894qjmvh" Nov 03 01:16:08.215226 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:08.215204 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-h2p2p" Nov 03 01:16:08.261053 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:08.261012 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-7nssr"] Nov 03 01:16:08.262517 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:08.262489 4421 cpu_manager.go:414] "containerMap: removing container" podUID="5bd6d60e-043b-47ba-beb1-08b1eaace690" containerName="istio-proxy" Nov 03 01:16:08.262517 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:08.262515 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5bd6d60e-043b-47ba-beb1-08b1eaace690" containerName="istio-proxy" Nov 03 01:16:08.262840 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:08.262795 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="5bd6d60e-043b-47ba-beb1-08b1eaace690" containerName="istio-proxy" Nov 03 01:16:08.267581 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:08.267556 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-7nssr" Nov 03 01:16:08.283560 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:08.283448 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-7nssr"] Nov 03 01:16:08.300069 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:08.300036 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5bd6d60e-043b-47ba-beb1-08b1eaace690-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bd6d60e-043b-47ba-beb1-08b1eaace690-istiod-ca-cert\") pod \"5bd6d60e-043b-47ba-beb1-08b1eaace690\" (UID: \"5bd6d60e-043b-47ba-beb1-08b1eaace690\") " Nov 03 01:16:08.300211 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:08.300142 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5bd6d60e-043b-47ba-beb1-08b1eaace690-istio-token\" (UniqueName: \"kubernetes.io/projected/5bd6d60e-043b-47ba-beb1-08b1eaace690-istio-token\") pod \"5bd6d60e-043b-47ba-beb1-08b1eaace690\" (UID: \"5bd6d60e-043b-47ba-beb1-08b1eaace690\") " Nov 03 01:16:08.300211 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:08.300173 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bd6d60e-043b-47ba-beb1-08b1eaace690-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bd6d60e-043b-47ba-beb1-08b1eaace690-credential-socket\") pod \"5bd6d60e-043b-47ba-beb1-08b1eaace690\" (UID: \"5bd6d60e-043b-47ba-beb1-08b1eaace690\") " Nov 03 01:16:08.300211 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:08.300204 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5bd6d60e-043b-47ba-beb1-08b1eaace690-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bd6d60e-043b-47ba-beb1-08b1eaace690-istio-podinfo\") pod \"5bd6d60e-043b-47ba-beb1-08b1eaace690\" (UID: \"5bd6d60e-043b-47ba-beb1-08b1eaace690\") " Nov 03 01:16:08.300457 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:08.300260 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bd6d60e-043b-47ba-beb1-08b1eaace690-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bd6d60e-043b-47ba-beb1-08b1eaace690-workload-socket\") pod \"5bd6d60e-043b-47ba-beb1-08b1eaace690\" (UID: \"5bd6d60e-043b-47ba-beb1-08b1eaace690\") " Nov 03 01:16:08.300457 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:08.300323 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bd6d60e-043b-47ba-beb1-08b1eaace690-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bd6d60e-043b-47ba-beb1-08b1eaace690-istio-data\") pod \"5bd6d60e-043b-47ba-beb1-08b1eaace690\" (UID: \"5bd6d60e-043b-47ba-beb1-08b1eaace690\") " Nov 03 01:16:08.300457 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:08.300373 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5bd6d60e-043b-47ba-beb1-08b1eaace690-kube-api-access-tnlld\" (UniqueName: \"kubernetes.io/projected/5bd6d60e-043b-47ba-beb1-08b1eaace690-kube-api-access-tnlld\") pod \"5bd6d60e-043b-47ba-beb1-08b1eaace690\" (UID: \"5bd6d60e-043b-47ba-beb1-08b1eaace690\") " Nov 03 01:16:08.300457 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:08.300409 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bd6d60e-043b-47ba-beb1-08b1eaace690-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bd6d60e-043b-47ba-beb1-08b1eaace690-workload-certs\") pod \"5bd6d60e-043b-47ba-beb1-08b1eaace690\" (UID: \"5bd6d60e-043b-47ba-beb1-08b1eaace690\") " Nov 03 01:16:08.300678 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:08.300529 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bd6d60e-043b-47ba-beb1-08b1eaace690-istiod-ca-cert" pod "5bd6d60e-043b-47ba-beb1-08b1eaace690" (UID: "5bd6d60e-043b-47ba-beb1-08b1eaace690"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:16:08.300782 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:08.300761 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bd6d60e-043b-47ba-beb1-08b1eaace690-istio-data" pod "5bd6d60e-043b-47ba-beb1-08b1eaace690" (UID: "5bd6d60e-043b-47ba-beb1-08b1eaace690"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:16:08.300782 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:08.300774 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bd6d60e-043b-47ba-beb1-08b1eaace690-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bd6d60e-043b-47ba-beb1-08b1eaace690-istio-envoy\") pod \"5bd6d60e-043b-47ba-beb1-08b1eaace690\" (UID: \"5bd6d60e-043b-47ba-beb1-08b1eaace690\") " Nov 03 01:16:08.301108 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:08.301077 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bd6d60e-043b-47ba-beb1-08b1eaace690-workload-certs" pod "5bd6d60e-043b-47ba-beb1-08b1eaace690" (UID: "5bd6d60e-043b-47ba-beb1-08b1eaace690"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:16:08.301261 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:08.301243 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnpt2\" (UniqueName: \"kubernetes.io/projected/4a72680d-8a9d-4aed-89c9-50206468ea74-kube-api-access-lnpt2\") pod \"e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-7nssr\" (UID: \"4a72680d-8a9d-4aed-89c9-50206468ea74\") " pod="openshift-ingress/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-7nssr" Nov 03 01:16:08.301387 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:08.301353 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a72680d-8a9d-4aed-89c9-50206468ea74-istio-data\") pod \"e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-7nssr\" (UID: \"4a72680d-8a9d-4aed-89c9-50206468ea74\") " pod="openshift-ingress/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-7nssr" Nov 03 01:16:08.301458 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:08.301417 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a72680d-8a9d-4aed-89c9-50206468ea74-istiod-ca-cert\") pod \"e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-7nssr\" (UID: \"4a72680d-8a9d-4aed-89c9-50206468ea74\") " pod="openshift-ingress/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-7nssr" Nov 03 01:16:08.301458 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:08.301453 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a72680d-8a9d-4aed-89c9-50206468ea74-istio-envoy\") pod \"e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-7nssr\" (UID: \"4a72680d-8a9d-4aed-89c9-50206468ea74\") " pod="openshift-ingress/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-7nssr" Nov 03 01:16:08.301579 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:08.301543 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a72680d-8a9d-4aed-89c9-50206468ea74-istio-token\") pod \"e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-7nssr\" (UID: \"4a72680d-8a9d-4aed-89c9-50206468ea74\") " pod="openshift-ingress/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-7nssr" Nov 03 01:16:08.301642 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:08.301605 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a72680d-8a9d-4aed-89c9-50206468ea74-workload-certs\") pod \"e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-7nssr\" (UID: \"4a72680d-8a9d-4aed-89c9-50206468ea74\") " pod="openshift-ingress/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-7nssr" Nov 03 01:16:08.301712 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:08.301659 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a72680d-8a9d-4aed-89c9-50206468ea74-workload-socket\") pod \"e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-7nssr\" (UID: \"4a72680d-8a9d-4aed-89c9-50206468ea74\") " pod="openshift-ingress/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-7nssr" Nov 03 01:16:08.301712 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:08.301681 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a72680d-8a9d-4aed-89c9-50206468ea74-credential-socket\") pod \"e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-7nssr\" (UID: \"4a72680d-8a9d-4aed-89c9-50206468ea74\") " pod="openshift-ingress/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-7nssr" Nov 03 01:16:08.301826 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:08.301720 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a72680d-8a9d-4aed-89c9-50206468ea74-istio-podinfo\") pod \"e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-7nssr\" (UID: \"4a72680d-8a9d-4aed-89c9-50206468ea74\") " pod="openshift-ingress/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-7nssr" Nov 03 01:16:08.301826 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:08.301808 4421 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bd6d60e-043b-47ba-beb1-08b1eaace690-istiod-ca-cert\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:16:08.301826 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:08.301819 4421 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bd6d60e-043b-47ba-beb1-08b1eaace690-istio-data\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:16:08.302001 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:08.301827 4421 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bd6d60e-043b-47ba-beb1-08b1eaace690-workload-certs\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:16:08.302001 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:08.301962 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bd6d60e-043b-47ba-beb1-08b1eaace690-workload-socket" pod "5bd6d60e-043b-47ba-beb1-08b1eaace690" (UID: "5bd6d60e-043b-47ba-beb1-08b1eaace690"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:16:08.302113 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:08.302009 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bd6d60e-043b-47ba-beb1-08b1eaace690-credential-socket" pod "5bd6d60e-043b-47ba-beb1-08b1eaace690" (UID: "5bd6d60e-043b-47ba-beb1-08b1eaace690"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:16:08.303101 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:08.303073 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bd6d60e-043b-47ba-beb1-08b1eaace690-istio-envoy" pod "5bd6d60e-043b-47ba-beb1-08b1eaace690" (UID: "5bd6d60e-043b-47ba-beb1-08b1eaace690"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:16:08.303252 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:08.303235 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5bd6d60e-043b-47ba-beb1-08b1eaace690-istio-podinfo" pod "5bd6d60e-043b-47ba-beb1-08b1eaace690" (UID: "5bd6d60e-043b-47ba-beb1-08b1eaace690"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:16:08.303892 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:08.303865 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bd6d60e-043b-47ba-beb1-08b1eaace690-kube-api-access-tnlld" pod "5bd6d60e-043b-47ba-beb1-08b1eaace690" (UID: "5bd6d60e-043b-47ba-beb1-08b1eaace690"). InnerVolumeSpecName "kube-api-access-tnlld". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:16:08.304880 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:08.304855 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bd6d60e-043b-47ba-beb1-08b1eaace690-istio-token" pod "5bd6d60e-043b-47ba-beb1-08b1eaace690" (UID: "5bd6d60e-043b-47ba-beb1-08b1eaace690"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:16:08.309047 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:08.309025 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-b894qjmvh" Nov 03 01:16:08.405693 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:08.405653 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnpt2\" (UniqueName: \"kubernetes.io/projected/4a72680d-8a9d-4aed-89c9-50206468ea74-kube-api-access-lnpt2\") pod \"e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-7nssr\" (UID: \"4a72680d-8a9d-4aed-89c9-50206468ea74\") " pod="openshift-ingress/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-7nssr" Nov 03 01:16:08.405827 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:08.405778 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a72680d-8a9d-4aed-89c9-50206468ea74-istio-data\") pod \"e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-7nssr\" (UID: \"4a72680d-8a9d-4aed-89c9-50206468ea74\") " pod="openshift-ingress/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-7nssr" Nov 03 01:16:08.405867 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:08.405854 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a72680d-8a9d-4aed-89c9-50206468ea74-istiod-ca-cert\") pod \"e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-7nssr\" (UID: \"4a72680d-8a9d-4aed-89c9-50206468ea74\") " pod="openshift-ingress/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-7nssr" Nov 03 01:16:08.405922 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:08.405907 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a72680d-8a9d-4aed-89c9-50206468ea74-istio-envoy\") pod \"e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-7nssr\" (UID: \"4a72680d-8a9d-4aed-89c9-50206468ea74\") " pod="openshift-ingress/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-7nssr" Nov 03 01:16:08.406032 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:08.406018 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a72680d-8a9d-4aed-89c9-50206468ea74-istio-token\") pod \"e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-7nssr\" (UID: \"4a72680d-8a9d-4aed-89c9-50206468ea74\") " pod="openshift-ingress/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-7nssr" Nov 03 01:16:08.406121 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:08.406106 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a72680d-8a9d-4aed-89c9-50206468ea74-workload-certs\") pod \"e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-7nssr\" (UID: \"4a72680d-8a9d-4aed-89c9-50206468ea74\") " pod="openshift-ingress/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-7nssr" Nov 03 01:16:08.406190 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:08.406174 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a72680d-8a9d-4aed-89c9-50206468ea74-workload-socket\") pod \"e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-7nssr\" (UID: \"4a72680d-8a9d-4aed-89c9-50206468ea74\") " pod="openshift-ingress/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-7nssr" Nov 03 01:16:08.406228 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:08.406212 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a72680d-8a9d-4aed-89c9-50206468ea74-credential-socket\") pod \"e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-7nssr\" (UID: \"4a72680d-8a9d-4aed-89c9-50206468ea74\") " pod="openshift-ingress/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-7nssr" Nov 03 01:16:08.406362 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:08.406268 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a72680d-8a9d-4aed-89c9-50206468ea74-istio-podinfo\") pod \"e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-7nssr\" (UID: \"4a72680d-8a9d-4aed-89c9-50206468ea74\") " pod="openshift-ingress/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-7nssr" Nov 03 01:16:08.406427 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:08.406411 4421 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bd6d60e-043b-47ba-beb1-08b1eaace690-istio-envoy\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:16:08.406461 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:08.406431 4421 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bd6d60e-043b-47ba-beb1-08b1eaace690-istio-token\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:16:08.406461 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:08.406446 4421 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bd6d60e-043b-47ba-beb1-08b1eaace690-credential-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:16:08.406525 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:08.406461 4421 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bd6d60e-043b-47ba-beb1-08b1eaace690-istio-podinfo\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:16:08.406525 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:08.406475 4421 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bd6d60e-043b-47ba-beb1-08b1eaace690-workload-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:16:08.406525 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:08.406488 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tnlld\" (UniqueName: \"kubernetes.io/projected/5bd6d60e-043b-47ba-beb1-08b1eaace690-kube-api-access-tnlld\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:16:08.406937 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:08.406903 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a72680d-8a9d-4aed-89c9-50206468ea74-istio-data\") pod \"e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-7nssr\" (UID: \"4a72680d-8a9d-4aed-89c9-50206468ea74\") " pod="openshift-ingress/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-7nssr" Nov 03 01:16:08.407232 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:08.407205 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a72680d-8a9d-4aed-89c9-50206468ea74-workload-certs\") pod \"e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-7nssr\" (UID: \"4a72680d-8a9d-4aed-89c9-50206468ea74\") " pod="openshift-ingress/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-7nssr" Nov 03 01:16:08.407344 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:08.407242 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a72680d-8a9d-4aed-89c9-50206468ea74-istiod-ca-cert\") pod \"e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-7nssr\" (UID: \"4a72680d-8a9d-4aed-89c9-50206468ea74\") " pod="openshift-ingress/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-7nssr" Nov 03 01:16:08.407408 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:08.407340 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a72680d-8a9d-4aed-89c9-50206468ea74-workload-socket\") pod \"e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-7nssr\" (UID: \"4a72680d-8a9d-4aed-89c9-50206468ea74\") " pod="openshift-ingress/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-7nssr" Nov 03 01:16:08.407619 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:08.407596 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a72680d-8a9d-4aed-89c9-50206468ea74-credential-socket\") pod \"e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-7nssr\" (UID: \"4a72680d-8a9d-4aed-89c9-50206468ea74\") " pod="openshift-ingress/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-7nssr" Nov 03 01:16:08.409463 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:08.409446 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a72680d-8a9d-4aed-89c9-50206468ea74-istio-podinfo\") pod \"e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-7nssr\" (UID: \"4a72680d-8a9d-4aed-89c9-50206468ea74\") " pod="openshift-ingress/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-7nssr" Nov 03 01:16:08.410020 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:08.409994 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a72680d-8a9d-4aed-89c9-50206468ea74-istio-envoy\") pod \"e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-7nssr\" (UID: \"4a72680d-8a9d-4aed-89c9-50206468ea74\") " pod="openshift-ingress/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-7nssr" Nov 03 01:16:08.415191 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:08.415157 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnpt2\" (UniqueName: \"kubernetes.io/projected/4a72680d-8a9d-4aed-89c9-50206468ea74-kube-api-access-lnpt2\") pod \"e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-7nssr\" (UID: \"4a72680d-8a9d-4aed-89c9-50206468ea74\") " pod="openshift-ingress/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-7nssr" Nov 03 01:16:08.416875 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:08.416846 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a72680d-8a9d-4aed-89c9-50206468ea74-istio-token\") pod \"e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-7nssr\" (UID: \"4a72680d-8a9d-4aed-89c9-50206468ea74\") " pod="openshift-ingress/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-7nssr" Nov 03 01:16:08.591145 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:08.591108 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-7nssr" Nov 03 01:16:08.613310 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:16:08.613265 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdc5358a_3f4b_43e8_b889_38fa85711fbd.slice/crio-bd593175aa1002d05064fc4e0e48fac329e182c7beb5c0f41328add6cd243dfe WatchSource:0}: Error finding container bd593175aa1002d05064fc4e0e48fac329e182c7beb5c0f41328add6cd243dfe: Status 404 returned error can't find the container with id bd593175aa1002d05064fc4e0e48fac329e182c7beb5c0f41328add6cd243dfe Nov 03 01:16:08.614317 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:08.614288 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-b894qjmvh"] Nov 03 01:16:08.693325 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:08.690465 4421 generic.go:356] "Generic (PLEG): container finished" podID="5bd6d60e-043b-47ba-beb1-08b1eaace690" containerID="bd1518b1ffc13c09f9f41b5e8c7aebdfa23b77711a7dd2f01a1878c5a1f832e5" exitCode=0 Nov 03 01:16:08.693325 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:08.690698 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-h2p2p" Nov 03 01:16:08.693325 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:08.691125 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-h2p2p" event={"ID":"5bd6d60e-043b-47ba-beb1-08b1eaace690","Type":"ContainerDied","Data":"bd1518b1ffc13c09f9f41b5e8c7aebdfa23b77711a7dd2f01a1878c5a1f832e5"} Nov 03 01:16:08.693325 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:08.691186 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-h2p2p" event={"ID":"5bd6d60e-043b-47ba-beb1-08b1eaace690","Type":"ContainerDied","Data":"0757afa1f0249ed383eb1f991017a880274a253cf6fd37a8049a09bc9a40f6d5"} Nov 03 01:16:08.693325 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:08.691219 4421 scope.go:122] "RemoveContainer" containerID="bd1518b1ffc13c09f9f41b5e8c7aebdfa23b77711a7dd2f01a1878c5a1f832e5" Nov 03 01:16:08.698394 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:08.698365 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-b894qjmvh" event={"ID":"cdc5358a-3f4b-43e8-b889-38fa85711fbd","Type":"ContainerStarted","Data":"bd593175aa1002d05064fc4e0e48fac329e182c7beb5c0f41328add6cd243dfe"} Nov 03 01:16:08.700151 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:08.700126 4421 generic.go:356] "Generic (PLEG): container finished" podID="67dd8eb6-fd6b-4a86-a8e4-b393e62044bf" containerID="91e7a11e0d31b71fc299a4faa5c90530dbcef8e9a2a767fd2524b061064134b6" exitCode=0 Nov 03 01:16:08.700315 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:08.700185 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-tt9jg" Nov 03 01:16:08.700389 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:08.700330 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-tt9jg" event={"ID":"67dd8eb6-fd6b-4a86-a8e4-b393e62044bf","Type":"ContainerDied","Data":"91e7a11e0d31b71fc299a4faa5c90530dbcef8e9a2a767fd2524b061064134b6"} Nov 03 01:16:08.700389 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:08.700362 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-tt9jg" event={"ID":"67dd8eb6-fd6b-4a86-a8e4-b393e62044bf","Type":"ContainerDied","Data":"b37b51d30e6995b3e8a27acef67d3d5f3f973637afe83d9e4b7b600d1f185423"} Nov 03 01:16:08.715673 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:08.715563 4421 scope.go:122] "RemoveContainer" containerID="bd1518b1ffc13c09f9f41b5e8c7aebdfa23b77711a7dd2f01a1878c5a1f832e5" Nov 03 01:16:08.728838 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:16:08.727066 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd1518b1ffc13c09f9f41b5e8c7aebdfa23b77711a7dd2f01a1878c5a1f832e5\": container with ID starting with bd1518b1ffc13c09f9f41b5e8c7aebdfa23b77711a7dd2f01a1878c5a1f832e5 not found: ID does not exist" containerID="bd1518b1ffc13c09f9f41b5e8c7aebdfa23b77711a7dd2f01a1878c5a1f832e5" Nov 03 01:16:08.728838 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:08.727119 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd1518b1ffc13c09f9f41b5e8c7aebdfa23b77711a7dd2f01a1878c5a1f832e5"} err="failed to get container status \"bd1518b1ffc13c09f9f41b5e8c7aebdfa23b77711a7dd2f01a1878c5a1f832e5\": rpc error: code = NotFound desc = could not find container \"bd1518b1ffc13c09f9f41b5e8c7aebdfa23b77711a7dd2f01a1878c5a1f832e5\": container with ID starting with bd1518b1ffc13c09f9f41b5e8c7aebdfa23b77711a7dd2f01a1878c5a1f832e5 not found: ID does not exist" Nov 03 01:16:08.728838 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:08.727146 4421 scope.go:122] "RemoveContainer" containerID="91e7a11e0d31b71fc299a4faa5c90530dbcef8e9a2a767fd2524b061064134b6" Nov 03 01:16:08.740308 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:08.740263 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-h2p2p"] Nov 03 01:16:08.748343 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:08.748297 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-h2p2p"] Nov 03 01:16:08.760772 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:08.760749 4421 scope.go:122] "RemoveContainer" containerID="91e7a11e0d31b71fc299a4faa5c90530dbcef8e9a2a767fd2524b061064134b6" Nov 03 01:16:08.765512 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:16:08.765465 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91e7a11e0d31b71fc299a4faa5c90530dbcef8e9a2a767fd2524b061064134b6\": container with ID starting with 91e7a11e0d31b71fc299a4faa5c90530dbcef8e9a2a767fd2524b061064134b6 not found: ID does not exist" containerID="91e7a11e0d31b71fc299a4faa5c90530dbcef8e9a2a767fd2524b061064134b6" Nov 03 01:16:08.765657 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:08.765527 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91e7a11e0d31b71fc299a4faa5c90530dbcef8e9a2a767fd2524b061064134b6"} err="failed to get container status \"91e7a11e0d31b71fc299a4faa5c90530dbcef8e9a2a767fd2524b061064134b6\": rpc error: code = NotFound desc = could not find container \"91e7a11e0d31b71fc299a4faa5c90530dbcef8e9a2a767fd2524b061064134b6\": container with ID starting with 91e7a11e0d31b71fc299a4faa5c90530dbcef8e9a2a767fd2524b061064134b6 not found: ID does not exist" Nov 03 01:16:08.781512 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:08.776181 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-tt9jg"] Nov 03 01:16:08.800300 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:08.793577 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-tt9jg"] Nov 03 01:16:08.893540 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:08.893509 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-7nssr"] Nov 03 01:16:08.894193 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:16:08.894169 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a72680d_8a9d_4aed_89c9_50206468ea74.slice/crio-cc44f065d5a69a61292301609539afaddff33c6402f64107ef35008d0c4983c4 WatchSource:0}: Error finding container cc44f065d5a69a61292301609539afaddff33c6402f64107ef35008d0c4983c4: Status 404 returned error can't find the container with id cc44f065d5a69a61292301609539afaddff33c6402f64107ef35008d0c4983c4 Nov 03 01:16:08.897397 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:08.897362 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:16:08.897476 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:08.897431 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:16:08.897476 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:08.897460 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:16:08.989809 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:08.989768 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-6d9btqzxk" podUID="396f111c-07d7-440c-853a-a9c48739892f" containerName="payload-processing" containerID="cri-o://0fe0c20f7f170e58a8e1d6d722358d643d6e2cd129942a6ba643f5d7ce0ca1e5" gracePeriod=25 Nov 03 01:16:09.288958 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:09.288928 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-6d9btqzxk" Nov 03 01:16:09.323508 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:09.323466 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-hbjwh"] Nov 03 01:16:09.324869 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:09.324842 4421 cpu_manager.go:414] "containerMap: removing container" podUID="396f111c-07d7-440c-853a-a9c48739892f" containerName="payload-processing" Nov 03 01:16:09.324984 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:09.324872 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="396f111c-07d7-440c-853a-a9c48739892f" containerName="payload-processing" Nov 03 01:16:09.325438 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:09.325167 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="396f111c-07d7-440c-853a-a9c48739892f" containerName="payload-processing" Nov 03 01:16:09.348346 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:09.343486 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-hbjwh" Nov 03 01:16:09.348346 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:09.344117 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-hbjwh"] Nov 03 01:16:09.357880 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:09.357842 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/396f111c-07d7-440c-853a-a9c48739892f-provider-credentials\" (UniqueName: \"kubernetes.io/projected/396f111c-07d7-440c-853a-a9c48739892f-provider-credentials\") pod \"396f111c-07d7-440c-853a-a9c48739892f\" (UID: \"396f111c-07d7-440c-853a-a9c48739892f\") " Nov 03 01:16:09.358045 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:09.357990 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/396f111c-07d7-440c-853a-a9c48739892f-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/396f111c-07d7-440c-853a-a9c48739892f-routing-overlay-volume\") pod \"396f111c-07d7-440c-853a-a9c48739892f\" (UID: \"396f111c-07d7-440c-853a-a9c48739892f\") " Nov 03 01:16:09.358104 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:09.358084 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/396f111c-07d7-440c-853a-a9c48739892f-kube-api-access-mgsrh\" (UniqueName: \"kubernetes.io/projected/396f111c-07d7-440c-853a-a9c48739892f-kube-api-access-mgsrh\") pod \"396f111c-07d7-440c-853a-a9c48739892f\" (UID: \"396f111c-07d7-440c-853a-a9c48739892f\") " Nov 03 01:16:09.358235 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:09.358218 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/396f111c-07d7-440c-853a-a9c48739892f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/396f111c-07d7-440c-853a-a9c48739892f-plugins-config-volume\") pod \"396f111c-07d7-440c-853a-a9c48739892f\" (UID: \"396f111c-07d7-440c-853a-a9c48739892f\") " Nov 03 01:16:09.358966 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:09.358749 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/396f111c-07d7-440c-853a-a9c48739892f-routing-overlay-volume" pod "396f111c-07d7-440c-853a-a9c48739892f" (UID: "396f111c-07d7-440c-853a-a9c48739892f"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:16:09.359190 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:09.359167 4421 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/396f111c-07d7-440c-853a-a9c48739892f-routing-overlay-volume\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:16:09.359580 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:09.359556 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/396f111c-07d7-440c-853a-a9c48739892f-plugins-config-volume" pod "396f111c-07d7-440c-853a-a9c48739892f" (UID: "396f111c-07d7-440c-853a-a9c48739892f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:16:09.360708 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:09.360680 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/396f111c-07d7-440c-853a-a9c48739892f-provider-credentials" pod "396f111c-07d7-440c-853a-a9c48739892f" (UID: "396f111c-07d7-440c-853a-a9c48739892f"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:16:09.361417 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:09.361378 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/396f111c-07d7-440c-853a-a9c48739892f-kube-api-access-mgsrh" pod "396f111c-07d7-440c-853a-a9c48739892f" (UID: "396f111c-07d7-440c-853a-a9c48739892f"). InnerVolumeSpecName "kube-api-access-mgsrh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:16:09.460837 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:09.460735 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f17e4c5-9bbd-4c90-bc08-c20d373da97d-workload-certs\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-hbjwh\" (UID: \"2f17e4c5-9bbd-4c90-bc08-c20d373da97d\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-hbjwh" Nov 03 01:16:09.460837 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:09.460785 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f17e4c5-9bbd-4c90-bc08-c20d373da97d-workload-socket\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-hbjwh\" (UID: \"2f17e4c5-9bbd-4c90-bc08-c20d373da97d\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-hbjwh" Nov 03 01:16:09.461042 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:09.460993 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f17e4c5-9bbd-4c90-bc08-c20d373da97d-istiod-ca-cert\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-hbjwh\" (UID: \"2f17e4c5-9bbd-4c90-bc08-c20d373da97d\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-hbjwh" Nov 03 01:16:09.461091 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:09.461076 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f17e4c5-9bbd-4c90-bc08-c20d373da97d-istio-podinfo\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-hbjwh\" (UID: \"2f17e4c5-9bbd-4c90-bc08-c20d373da97d\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-hbjwh" Nov 03 01:16:09.461176 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:09.461159 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f17e4c5-9bbd-4c90-bc08-c20d373da97d-istio-data\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-hbjwh\" (UID: \"2f17e4c5-9bbd-4c90-bc08-c20d373da97d\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-hbjwh" Nov 03 01:16:09.461246 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:09.461229 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f17e4c5-9bbd-4c90-bc08-c20d373da97d-istio-envoy\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-hbjwh\" (UID: \"2f17e4c5-9bbd-4c90-bc08-c20d373da97d\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-hbjwh" Nov 03 01:16:09.461360 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:09.461333 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f17e4c5-9bbd-4c90-bc08-c20d373da97d-credential-socket\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-hbjwh\" (UID: \"2f17e4c5-9bbd-4c90-bc08-c20d373da97d\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-hbjwh" Nov 03 01:16:09.461577 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:09.461559 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ns47l\" (UniqueName: \"kubernetes.io/projected/2f17e4c5-9bbd-4c90-bc08-c20d373da97d-kube-api-access-ns47l\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-hbjwh\" (UID: \"2f17e4c5-9bbd-4c90-bc08-c20d373da97d\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-hbjwh" Nov 03 01:16:09.461620 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:09.461595 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f17e4c5-9bbd-4c90-bc08-c20d373da97d-istio-token\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-hbjwh\" (UID: \"2f17e4c5-9bbd-4c90-bc08-c20d373da97d\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-hbjwh" Nov 03 01:16:09.461827 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:09.461811 4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/396f111c-07d7-440c-853a-a9c48739892f-plugins-config-volume\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:16:09.461862 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:09.461835 4421 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/396f111c-07d7-440c-853a-a9c48739892f-provider-credentials\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:16:09.461862 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:09.461852 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mgsrh\" (UniqueName: \"kubernetes.io/projected/396f111c-07d7-440c-853a-a9c48739892f-kube-api-access-mgsrh\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:16:09.563551 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:09.563516 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f17e4c5-9bbd-4c90-bc08-c20d373da97d-istiod-ca-cert\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-hbjwh\" (UID: \"2f17e4c5-9bbd-4c90-bc08-c20d373da97d\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-hbjwh" Nov 03 01:16:09.563740 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:09.563572 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f17e4c5-9bbd-4c90-bc08-c20d373da97d-istio-podinfo\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-hbjwh\" (UID: \"2f17e4c5-9bbd-4c90-bc08-c20d373da97d\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-hbjwh" Nov 03 01:16:09.563740 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:09.563617 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f17e4c5-9bbd-4c90-bc08-c20d373da97d-istio-data\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-hbjwh\" (UID: \"2f17e4c5-9bbd-4c90-bc08-c20d373da97d\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-hbjwh" Nov 03 01:16:09.563740 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:09.563667 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f17e4c5-9bbd-4c90-bc08-c20d373da97d-istio-envoy\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-hbjwh\" (UID: \"2f17e4c5-9bbd-4c90-bc08-c20d373da97d\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-hbjwh" Nov 03 01:16:09.563740 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:09.563726 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f17e4c5-9bbd-4c90-bc08-c20d373da97d-credential-socket\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-hbjwh\" (UID: \"2f17e4c5-9bbd-4c90-bc08-c20d373da97d\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-hbjwh" Nov 03 01:16:09.564007 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:09.563899 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ns47l\" (UniqueName: \"kubernetes.io/projected/2f17e4c5-9bbd-4c90-bc08-c20d373da97d-kube-api-access-ns47l\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-hbjwh\" (UID: \"2f17e4c5-9bbd-4c90-bc08-c20d373da97d\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-hbjwh" Nov 03 01:16:09.564007 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:09.563929 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f17e4c5-9bbd-4c90-bc08-c20d373da97d-istio-token\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-hbjwh\" (UID: \"2f17e4c5-9bbd-4c90-bc08-c20d373da97d\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-hbjwh" Nov 03 01:16:09.564174 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:09.564129 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f17e4c5-9bbd-4c90-bc08-c20d373da97d-istio-data\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-hbjwh\" (UID: \"2f17e4c5-9bbd-4c90-bc08-c20d373da97d\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-hbjwh" Nov 03 01:16:09.564254 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:09.564175 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f17e4c5-9bbd-4c90-bc08-c20d373da97d-credential-socket\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-hbjwh\" (UID: \"2f17e4c5-9bbd-4c90-bc08-c20d373da97d\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-hbjwh" Nov 03 01:16:09.564363 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:09.564336 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f17e4c5-9bbd-4c90-bc08-c20d373da97d-workload-certs\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-hbjwh\" (UID: \"2f17e4c5-9bbd-4c90-bc08-c20d373da97d\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-hbjwh" Nov 03 01:16:09.564428 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:09.564379 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f17e4c5-9bbd-4c90-bc08-c20d373da97d-workload-socket\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-hbjwh\" (UID: \"2f17e4c5-9bbd-4c90-bc08-c20d373da97d\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-hbjwh" Nov 03 01:16:09.564491 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:09.564447 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f17e4c5-9bbd-4c90-bc08-c20d373da97d-istiod-ca-cert\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-hbjwh\" (UID: \"2f17e4c5-9bbd-4c90-bc08-c20d373da97d\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-hbjwh" Nov 03 01:16:09.564675 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:09.564650 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f17e4c5-9bbd-4c90-bc08-c20d373da97d-workload-certs\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-hbjwh\" (UID: \"2f17e4c5-9bbd-4c90-bc08-c20d373da97d\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-hbjwh" Nov 03 01:16:09.564779 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:09.564669 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f17e4c5-9bbd-4c90-bc08-c20d373da97d-workload-socket\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-hbjwh\" (UID: \"2f17e4c5-9bbd-4c90-bc08-c20d373da97d\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-hbjwh" Nov 03 01:16:09.566537 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:09.566515 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f17e4c5-9bbd-4c90-bc08-c20d373da97d-istio-envoy\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-hbjwh\" (UID: \"2f17e4c5-9bbd-4c90-bc08-c20d373da97d\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-hbjwh" Nov 03 01:16:09.566637 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:09.566546 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f17e4c5-9bbd-4c90-bc08-c20d373da97d-istio-podinfo\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-hbjwh\" (UID: \"2f17e4c5-9bbd-4c90-bc08-c20d373da97d\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-hbjwh" Nov 03 01:16:09.580557 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:09.580525 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f17e4c5-9bbd-4c90-bc08-c20d373da97d-istio-token\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-hbjwh\" (UID: \"2f17e4c5-9bbd-4c90-bc08-c20d373da97d\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-hbjwh" Nov 03 01:16:09.580766 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:09.580743 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ns47l\" (UniqueName: \"kubernetes.io/projected/2f17e4c5-9bbd-4c90-bc08-c20d373da97d-kube-api-access-ns47l\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-hbjwh\" (UID: \"2f17e4c5-9bbd-4c90-bc08-c20d373da97d\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-hbjwh" Nov 03 01:16:09.640848 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:09.638480 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-78c872f4s" podUID="6386694a-6d4c-4a69-8388-52f34a3484d2" containerName="payload-processing" containerID="cri-o://4eb085a159c0f13628c7c239b1185a0a65c96f79a44a1a91fe0f133a69e5b85f" gracePeriod=25 Nov 03 01:16:09.678516 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:09.678479 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-hbjwh" Nov 03 01:16:09.750890 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:09.750545 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-b894qjmvh" event={"ID":"cdc5358a-3f4b-43e8-b889-38fa85711fbd","Type":"ContainerStarted","Data":"92f5951089164ddf89736b0835e7932b8d849a9d88c9ba381365e1e11c6fc873"} Nov 03 01:16:09.750890 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:09.750631 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-b894qjmvh" Nov 03 01:16:09.753148 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:09.753121 4421 generic.go:356] "Generic (PLEG): container finished" podID="396f111c-07d7-440c-853a-a9c48739892f" containerID="0fe0c20f7f170e58a8e1d6d722358d643d6e2cd129942a6ba643f5d7ce0ca1e5" exitCode=0 Nov 03 01:16:09.753263 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:09.753159 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-6d9btqzxk" Nov 03 01:16:09.753263 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:09.753197 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-6d9btqzxk" event={"ID":"396f111c-07d7-440c-853a-a9c48739892f","Type":"ContainerDied","Data":"0fe0c20f7f170e58a8e1d6d722358d643d6e2cd129942a6ba643f5d7ce0ca1e5"} Nov 03 01:16:09.753263 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:09.753246 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-6d9btqzxk" event={"ID":"396f111c-07d7-440c-853a-a9c48739892f","Type":"ContainerDied","Data":"7b410e97b9ab6be01b8af7774bfc768e1a90ef1a465532fd6598ceab0466a7ac"} Nov 03 01:16:09.753483 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:09.753287 4421 scope.go:122] "RemoveContainer" containerID="0fe0c20f7f170e58a8e1d6d722358d643d6e2cd129942a6ba643f5d7ce0ca1e5" Nov 03 01:16:09.755882 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:09.755859 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-7nssr" event={"ID":"4a72680d-8a9d-4aed-89c9-50206468ea74","Type":"ContainerStarted","Data":"313995a8e8d94d5c2a3e4375572a13e07e40738c5db13afca7b4c3b3dca9312f"} Nov 03 01:16:09.756001 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:09.755902 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-7nssr" event={"ID":"4a72680d-8a9d-4aed-89c9-50206468ea74","Type":"ContainerStarted","Data":"cc44f065d5a69a61292301609539afaddff33c6402f64107ef35008d0c4983c4"} Nov 03 01:16:09.774026 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:09.773972 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-b894qjmvh" podStartSLOduration=44.77395728 podStartE2EDuration="44.77395728s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:15:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:16:09.769644152 +0000 UTC m=+2111.940647247" watchObservedRunningTime="2026-11-03 01:16:09.77395728 +0000 UTC m=+2111.944960376" Nov 03 01:16:09.797711 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:09.797208 4421 scope.go:122] "RemoveContainer" containerID="0fe0c20f7f170e58a8e1d6d722358d643d6e2cd129942a6ba643f5d7ce0ca1e5" Nov 03 01:16:09.797711 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:16:09.797605 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0fe0c20f7f170e58a8e1d6d722358d643d6e2cd129942a6ba643f5d7ce0ca1e5\": container with ID starting with 0fe0c20f7f170e58a8e1d6d722358d643d6e2cd129942a6ba643f5d7ce0ca1e5 not found: ID does not exist" containerID="0fe0c20f7f170e58a8e1d6d722358d643d6e2cd129942a6ba643f5d7ce0ca1e5" Nov 03 01:16:09.797711 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:09.797645 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fe0c20f7f170e58a8e1d6d722358d643d6e2cd129942a6ba643f5d7ce0ca1e5"} err="failed to get container status \"0fe0c20f7f170e58a8e1d6d722358d643d6e2cd129942a6ba643f5d7ce0ca1e5\": rpc error: code = NotFound desc = could not find container \"0fe0c20f7f170e58a8e1d6d722358d643d6e2cd129942a6ba643f5d7ce0ca1e5\": container with ID starting with 0fe0c20f7f170e58a8e1d6d722358d643d6e2cd129942a6ba643f5d7ce0ca1e5 not found: ID does not exist" Nov 03 01:16:09.801183 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:09.801136 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-7nssr" podStartSLOduration=58.801124902 podStartE2EDuration="58.801124902s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:15:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:16:09.794741856 +0000 UTC m=+2111.965744951" watchObservedRunningTime="2026-11-03 01:16:09.801124902 +0000 UTC m=+2111.972127998" Nov 03 01:16:09.854632 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:09.851782 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-6d9btqzxk"] Nov 03 01:16:09.862074 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:09.860584 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-6d9btqzxk"] Nov 03 01:16:10.049377 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:10.049121 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="396f111c-07d7-440c-853a-a9c48739892f" path="/var/lib/kubelet/pods/396f111c-07d7-440c-853a-a9c48739892f/volumes" Nov 03 01:16:10.050358 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:10.050012 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5bd6d60e-043b-47ba-beb1-08b1eaace690" path="/var/lib/kubelet/pods/5bd6d60e-043b-47ba-beb1-08b1eaace690/volumes" Nov 03 01:16:10.050766 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:10.050744 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67dd8eb6-fd6b-4a86-a8e4-b393e62044bf" path="/var/lib/kubelet/pods/67dd8eb6-fd6b-4a86-a8e4-b393e62044bf/volumes" Nov 03 01:16:10.269576 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:16:10.269544 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f17e4c5_9bbd_4c90_bc08_c20d373da97d.slice/crio-ca3ea94c6b990eaf04ab7004baf2f8c2a883bc63de501168e6f641d9b55ebb12 WatchSource:0}: Error finding container ca3ea94c6b990eaf04ab7004baf2f8c2a883bc63de501168e6f641d9b55ebb12: Status 404 returned error can't find the container with id ca3ea94c6b990eaf04ab7004baf2f8c2a883bc63de501168e6f641d9b55ebb12 Nov 03 01:16:10.272348 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:10.272253 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-hbjwh"] Nov 03 01:16:10.274156 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:10.274109 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:16:10.274261 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:10.274209 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:16:10.274327 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:10.274257 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:16:10.301250 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:10.301226 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-78c872f4s" Nov 03 01:16:10.358405 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:10.355812 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6386694a-6d4c-4a69-8388-52f34a3484d2-provider-credentials\" (UniqueName: \"kubernetes.io/projected/6386694a-6d4c-4a69-8388-52f34a3484d2-provider-credentials\") pod \"6386694a-6d4c-4a69-8388-52f34a3484d2\" (UID: \"6386694a-6d4c-4a69-8388-52f34a3484d2\") " Nov 03 01:16:10.358405 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:10.355983 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6386694a-6d4c-4a69-8388-52f34a3484d2-kube-api-access-c78x9\" (UniqueName: \"kubernetes.io/projected/6386694a-6d4c-4a69-8388-52f34a3484d2-kube-api-access-c78x9\") pod \"6386694a-6d4c-4a69-8388-52f34a3484d2\" (UID: \"6386694a-6d4c-4a69-8388-52f34a3484d2\") " Nov 03 01:16:10.358405 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:10.356137 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6386694a-6d4c-4a69-8388-52f34a3484d2-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6386694a-6d4c-4a69-8388-52f34a3484d2-routing-overlay-volume\") pod \"6386694a-6d4c-4a69-8388-52f34a3484d2\" (UID: \"6386694a-6d4c-4a69-8388-52f34a3484d2\") " Nov 03 01:16:10.358405 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:10.356210 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6386694a-6d4c-4a69-8388-52f34a3484d2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6386694a-6d4c-4a69-8388-52f34a3484d2-plugins-config-volume\") pod \"6386694a-6d4c-4a69-8388-52f34a3484d2\" (UID: \"6386694a-6d4c-4a69-8388-52f34a3484d2\") " Nov 03 01:16:10.358405 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:10.356662 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6386694a-6d4c-4a69-8388-52f34a3484d2-routing-overlay-volume" pod "6386694a-6d4c-4a69-8388-52f34a3484d2" (UID: "6386694a-6d4c-4a69-8388-52f34a3484d2"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:16:10.358405 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:10.357250 4421 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6386694a-6d4c-4a69-8388-52f34a3484d2-routing-overlay-volume\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:16:10.359170 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:10.359128 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6386694a-6d4c-4a69-8388-52f34a3484d2-plugins-config-volume" pod "6386694a-6d4c-4a69-8388-52f34a3484d2" (UID: "6386694a-6d4c-4a69-8388-52f34a3484d2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:16:10.370854 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:10.370608 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6386694a-6d4c-4a69-8388-52f34a3484d2-provider-credentials" pod "6386694a-6d4c-4a69-8388-52f34a3484d2" (UID: "6386694a-6d4c-4a69-8388-52f34a3484d2"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:16:10.370854 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:10.370665 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6386694a-6d4c-4a69-8388-52f34a3484d2-kube-api-access-c78x9" pod "6386694a-6d4c-4a69-8388-52f34a3484d2" (UID: "6386694a-6d4c-4a69-8388-52f34a3484d2"). InnerVolumeSpecName "kube-api-access-c78x9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:16:10.482295 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:10.478042 4421 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6386694a-6d4c-4a69-8388-52f34a3484d2-provider-credentials\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:16:10.482295 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:10.478089 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c78x9\" (UniqueName: \"kubernetes.io/projected/6386694a-6d4c-4a69-8388-52f34a3484d2-kube-api-access-c78x9\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:16:10.482295 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:10.478106 4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6386694a-6d4c-4a69-8388-52f34a3484d2-plugins-config-volume\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:16:10.608349 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:10.601040 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-7nssr" Nov 03 01:16:10.608883 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:10.608855 4421 patch_prober.go:29] interesting pod/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-7nssr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.20:15021/healthz/ready\": dial tcp 10.134.1.20:15021: connect: connection refused" start-of-body= Nov 03 01:16:10.609084 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:10.609057 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-7nssr" podUID="4a72680d-8a9d-4aed-89c9-50206468ea74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.20:15021/healthz/ready\": dial tcp 10.134.1.20:15021: connect: connection refused" Nov 03 01:16:10.837949 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:10.837895 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-hbjwh" event={"ID":"2f17e4c5-9bbd-4c90-bc08-c20d373da97d","Type":"ContainerStarted","Data":"c737a942d8498e9c0f5f94824e2a69553bb3e6ad5d8cc508dd534eea3b00525f"} Nov 03 01:16:10.838216 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:10.838195 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-hbjwh" event={"ID":"2f17e4c5-9bbd-4c90-bc08-c20d373da97d","Type":"ContainerStarted","Data":"ca3ea94c6b990eaf04ab7004baf2f8c2a883bc63de501168e6f641d9b55ebb12"} Nov 03 01:16:10.962537 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:10.962092 4421 generic.go:356] "Generic (PLEG): container finished" podID="6386694a-6d4c-4a69-8388-52f34a3484d2" containerID="4eb085a159c0f13628c7c239b1185a0a65c96f79a44a1a91fe0f133a69e5b85f" exitCode=0 Nov 03 01:16:10.964679 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:10.964423 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-78c872f4s" Nov 03 01:16:10.966207 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:10.964446 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-78c872f4s" event={"ID":"6386694a-6d4c-4a69-8388-52f34a3484d2","Type":"ContainerDied","Data":"4eb085a159c0f13628c7c239b1185a0a65c96f79a44a1a91fe0f133a69e5b85f"} Nov 03 01:16:10.966445 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:10.966421 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-78c872f4s" event={"ID":"6386694a-6d4c-4a69-8388-52f34a3484d2","Type":"ContainerDied","Data":"cd33dc082959a7b63d830f265e870c53035554a14b5a464c521655419b5f28c1"} Nov 03 01:16:10.966587 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:10.966565 4421 scope.go:122] "RemoveContainer" containerID="4eb085a159c0f13628c7c239b1185a0a65c96f79a44a1a91fe0f133a69e5b85f" Nov 03 01:16:11.068742 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:11.068150 4421 scope.go:122] "RemoveContainer" containerID="4eb085a159c0f13628c7c239b1185a0a65c96f79a44a1a91fe0f133a69e5b85f" Nov 03 01:16:11.068742 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:16:11.068591 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4eb085a159c0f13628c7c239b1185a0a65c96f79a44a1a91fe0f133a69e5b85f\": container with ID starting with 4eb085a159c0f13628c7c239b1185a0a65c96f79a44a1a91fe0f133a69e5b85f not found: ID does not exist" containerID="4eb085a159c0f13628c7c239b1185a0a65c96f79a44a1a91fe0f133a69e5b85f" Nov 03 01:16:11.068742 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:11.068624 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4eb085a159c0f13628c7c239b1185a0a65c96f79a44a1a91fe0f133a69e5b85f"} err="failed to get container status \"4eb085a159c0f13628c7c239b1185a0a65c96f79a44a1a91fe0f133a69e5b85f\": rpc error: code = NotFound desc = could not find container \"4eb085a159c0f13628c7c239b1185a0a65c96f79a44a1a91fe0f133a69e5b85f\": container with ID starting with 4eb085a159c0f13628c7c239b1185a0a65c96f79a44a1a91fe0f133a69e5b85f not found: ID does not exist" Nov 03 01:16:11.085896 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:11.085834 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-hbjwh" podStartSLOduration=239.085813166 podStartE2EDuration="3m59.085813166s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:12:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:16:10.911681305 +0000 UTC m=+2113.082684401" watchObservedRunningTime="2026-11-03 01:16:11.085813166 +0000 UTC m=+2113.256816263" Nov 03 01:16:11.086636 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:11.086609 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-78c872f4s"] Nov 03 01:16:11.153441 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:11.139296 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-78c872f4s"] Nov 03 01:16:11.592342 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:11.592199 4421 patch_prober.go:29] interesting pod/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-7nssr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.20:15021/healthz/ready\": dial tcp 10.134.1.20:15021: connect: connection refused" start-of-body= Nov 03 01:16:11.592342 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:11.592265 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-7nssr" podUID="4a72680d-8a9d-4aed-89c9-50206468ea74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.20:15021/healthz/ready\": dial tcp 10.134.1.20:15021: connect: connection refused" Nov 03 01:16:11.679852 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:11.679813 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-hbjwh" Nov 03 01:16:11.681783 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:11.681754 4421 patch_prober.go:29] interesting pod/e2e-extmod-a-d25759-openshift-default-d47fd9785-hbjwh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.21:15021/healthz/ready\": dial tcp 10.134.1.21:15021: connect: connection refused" start-of-body= Nov 03 01:16:11.682035 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:11.681996 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-hbjwh" podUID="2f17e4c5-9bbd-4c90-bc08-c20d373da97d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.21:15021/healthz/ready\": dial tcp 10.134.1.21:15021: connect: connection refused" Nov 03 01:16:11.982465 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:11.982384 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6386694a-6d4c-4a69-8388-52f34a3484d2" path="/var/lib/kubelet/pods/6386694a-6d4c-4a69-8388-52f34a3484d2/volumes" Nov 03 01:16:12.592194 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:12.592152 4421 patch_prober.go:29] interesting pod/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-7nssr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.20:15021/healthz/ready\": dial tcp 10.134.1.20:15021: connect: connection refused" start-of-body= Nov 03 01:16:12.592650 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:12.592217 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-7nssr" podUID="4a72680d-8a9d-4aed-89c9-50206468ea74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.20:15021/healthz/ready\": dial tcp 10.134.1.20:15021: connect: connection refused" Nov 03 01:16:12.680484 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:12.680440 4421 patch_prober.go:29] interesting pod/e2e-extmod-a-d25759-openshift-default-d47fd9785-hbjwh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.21:15021/healthz/ready\": dial tcp 10.134.1.21:15021: connect: connection refused" start-of-body= Nov 03 01:16:12.680664 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:12.680508 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-hbjwh" podUID="2f17e4c5-9bbd-4c90-bc08-c20d373da97d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.21:15021/healthz/ready\": dial tcp 10.134.1.21:15021: connect: connection refused" Nov 03 01:16:13.592094 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:13.592058 4421 patch_prober.go:29] interesting pod/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-7nssr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.20:15021/healthz/ready\": dial tcp 10.134.1.20:15021: connect: connection refused" start-of-body= Nov 03 01:16:13.592259 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:13.592110 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-7nssr" podUID="4a72680d-8a9d-4aed-89c9-50206468ea74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.20:15021/healthz/ready\": dial tcp 10.134.1.20:15021: connect: connection refused" Nov 03 01:16:13.682302 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:13.681879 4421 patch_prober.go:29] interesting pod/e2e-extmod-a-d25759-openshift-default-d47fd9785-hbjwh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.21:15021/healthz/ready\": dial tcp 10.134.1.21:15021: connect: connection refused" start-of-body= Nov 03 01:16:13.682524 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:13.681980 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-hbjwh" podUID="2f17e4c5-9bbd-4c90-bc08-c20d373da97d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.21:15021/healthz/ready\": dial tcp 10.134.1.21:15021: connect: connection refused" Nov 03 01:16:14.591635 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:14.591600 4421 patch_prober.go:29] interesting pod/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-7nssr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.20:15021/healthz/ready\": dial tcp 10.134.1.20:15021: connect: connection refused" start-of-body= Nov 03 01:16:14.591889 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:14.591860 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-7nssr" podUID="4a72680d-8a9d-4aed-89c9-50206468ea74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.20:15021/healthz/ready\": dial tcp 10.134.1.20:15021: connect: connection refused" Nov 03 01:16:14.680517 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:14.680160 4421 patch_prober.go:29] interesting pod/e2e-extmod-a-d25759-openshift-default-d47fd9785-hbjwh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.21:15021/healthz/ready\": dial tcp 10.134.1.21:15021: connect: connection refused" start-of-body= Nov 03 01:16:14.680517 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:14.680230 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-hbjwh" podUID="2f17e4c5-9bbd-4c90-bc08-c20d373da97d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.21:15021/healthz/ready\": dial tcp 10.134.1.21:15021: connect: connection refused" Nov 03 01:16:14.683376 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:16:14.683345 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 01:16:15.593294 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:15.592533 4421 patch_prober.go:29] interesting pod/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-7nssr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.20:15021/healthz/ready\": dial tcp 10.134.1.20:15021: connect: connection refused" start-of-body= Nov 03 01:16:15.593294 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:15.592596 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-7nssr" podUID="4a72680d-8a9d-4aed-89c9-50206468ea74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.20:15021/healthz/ready\": dial tcp 10.134.1.20:15021: connect: connection refused" Nov 03 01:16:15.681685 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:15.681641 4421 patch_prober.go:29] interesting pod/e2e-extmod-a-d25759-openshift-default-d47fd9785-hbjwh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.21:15021/healthz/ready\": dial tcp 10.134.1.21:15021: connect: connection refused" start-of-body= Nov 03 01:16:15.682170 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:15.681709 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-hbjwh" podUID="2f17e4c5-9bbd-4c90-bc08-c20d373da97d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.21:15021/healthz/ready\": dial tcp 10.134.1.21:15021: connect: connection refused" Nov 03 01:16:16.597132 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:16.597095 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-7nssr" Nov 03 01:16:16.597663 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:16.597639 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-7nssr" Nov 03 01:16:16.598392 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:16.598374 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-7nssr" Nov 03 01:16:16.686427 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:16.686389 4421 patch_prober.go:29] interesting pod/e2e-extmod-a-d25759-openshift-default-d47fd9785-hbjwh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.21:15021/healthz/ready\": dial tcp 10.134.1.21:15021: connect: connection refused" start-of-body= Nov 03 01:16:16.687032 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:16.686992 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-hbjwh" podUID="2f17e4c5-9bbd-4c90-bc08-c20d373da97d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.21:15021/healthz/ready\": dial tcp 10.134.1.21:15021: connect: connection refused" Nov 03 01:16:17.694972 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:17.694933 4421 patch_prober.go:29] interesting pod/e2e-extmod-a-d25759-openshift-default-d47fd9785-hbjwh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.21:15021/healthz/ready\": dial tcp 10.134.1.21:15021: connect: connection refused" start-of-body= Nov 03 01:16:17.695647 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:17.695612 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-hbjwh" podUID="2f17e4c5-9bbd-4c90-bc08-c20d373da97d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.21:15021/healthz/ready\": dial tcp 10.134.1.21:15021: connect: connection refused" Nov 03 01:16:18.807149 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:18.803051 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-hbjwh" Nov 03 01:16:18.807149 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:18.805034 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-hbjwh" Nov 03 01:16:18.813289 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:18.812319 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-hbjwh" Nov 03 01:16:19.639399 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:19.639364 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-5d6bztwql"] Nov 03 01:16:19.640206 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:19.640190 4421 cpu_manager.go:414] "containerMap: removing container" podUID="6386694a-6d4c-4a69-8388-52f34a3484d2" containerName="payload-processing" Nov 03 01:16:19.640255 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:19.640208 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6386694a-6d4c-4a69-8388-52f34a3484d2" containerName="payload-processing" Nov 03 01:16:19.640395 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:19.640384 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="6386694a-6d4c-4a69-8388-52f34a3484d2" containerName="payload-processing" Nov 03 01:16:19.645749 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:19.645731 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-5d6bztwql" Nov 03 01:16:19.656656 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:19.656625 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-5d6bztwql"] Nov 03 01:16:19.667717 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:19.667683 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-7c759a\"/\"payload-processing-external-model-e2e--d95606a1-dockercfg-8xzgj\"" Nov 03 01:16:19.667901 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:19.667775 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-7c759a\"/\"kube-root-ca.crt\"" Nov 03 01:16:19.667901 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:19.667841 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-7c759a\"/\"payload-processing-external-model-plugins-e2e-extmod-a-7c759a\"" Nov 03 01:16:19.667901 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:19.667893 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-7c759a\"/\"openshift-service-ca.crt\"" Nov 03 01:16:19.668095 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:19.667993 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-7c759a\"/\"routing-overlay\"" Nov 03 01:16:19.668095 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:19.668068 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-7c759a\"/\"e2e-cmp-legacy-2e4c55-provider-cred\"" Nov 03 01:16:19.792301 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:19.786913 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/477f510e-003c-4491-bc5d-a3a313f6af13-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-7c759a-5d6bztwql\" (UID: \"477f510e-003c-4491-bc5d-a3a313f6af13\") " pod="ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-5d6bztwql" Nov 03 01:16:19.833513 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:19.833469 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/477f510e-003c-4491-bc5d-a3a313f6af13-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-7c759a-5d6bztwql\" (UID: \"477f510e-003c-4491-bc5d-a3a313f6af13\") " pod="ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-5d6bztwql" Nov 03 01:16:19.834157 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:19.834127 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vcb5\" (UniqueName: \"kubernetes.io/projected/477f510e-003c-4491-bc5d-a3a313f6af13-kube-api-access-7vcb5\") pod \"payload-processing-external-model-e2e-extmod-a-7c759a-5d6bztwql\" (UID: \"477f510e-003c-4491-bc5d-a3a313f6af13\") " pod="ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-5d6bztwql" Nov 03 01:16:19.834385 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:19.834293 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/477f510e-003c-4491-bc5d-a3a313f6af13-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-7c759a-5d6bztwql\" (UID: \"477f510e-003c-4491-bc5d-a3a313f6af13\") " pod="ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-5d6bztwql" Nov 03 01:16:19.974103 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:19.962131 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/477f510e-003c-4491-bc5d-a3a313f6af13-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-7c759a-5d6bztwql\" (UID: \"477f510e-003c-4491-bc5d-a3a313f6af13\") " pod="ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-5d6bztwql" Nov 03 01:16:19.974103 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:19.962230 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vcb5\" (UniqueName: \"kubernetes.io/projected/477f510e-003c-4491-bc5d-a3a313f6af13-kube-api-access-7vcb5\") pod \"payload-processing-external-model-e2e-extmod-a-7c759a-5d6bztwql\" (UID: \"477f510e-003c-4491-bc5d-a3a313f6af13\") " pod="ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-5d6bztwql" Nov 03 01:16:19.974103 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:19.962261 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/477f510e-003c-4491-bc5d-a3a313f6af13-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-7c759a-5d6bztwql\" (UID: \"477f510e-003c-4491-bc5d-a3a313f6af13\") " pod="ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-5d6bztwql" Nov 03 01:16:19.985828 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:19.974743 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/477f510e-003c-4491-bc5d-a3a313f6af13-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-7c759a-5d6bztwql\" (UID: \"477f510e-003c-4491-bc5d-a3a313f6af13\") " pod="ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-5d6bztwql" Nov 03 01:16:19.996815 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:19.996778 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/477f510e-003c-4491-bc5d-a3a313f6af13-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-7c759a-5d6bztwql\" (UID: \"477f510e-003c-4491-bc5d-a3a313f6af13\") " pod="ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-5d6bztwql" Nov 03 01:16:20.009803 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:19.997911 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/477f510e-003c-4491-bc5d-a3a313f6af13-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-7c759a-5d6bztwql\" (UID: \"477f510e-003c-4491-bc5d-a3a313f6af13\") " pod="ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-5d6bztwql" Nov 03 01:16:20.040344 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:20.040288 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vcb5\" (UniqueName: \"kubernetes.io/projected/477f510e-003c-4491-bc5d-a3a313f6af13-kube-api-access-7vcb5\") pod \"payload-processing-external-model-e2e-extmod-a-7c759a-5d6bztwql\" (UID: \"477f510e-003c-4491-bc5d-a3a313f6af13\") " pod="ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-5d6bztwql" Nov 03 01:16:20.050304 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:20.050246 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/477f510e-003c-4491-bc5d-a3a313f6af13-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-7c759a-5d6bztwql\" (UID: \"477f510e-003c-4491-bc5d-a3a313f6af13\") " pod="ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-5d6bztwql" Nov 03 01:16:20.275669 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:20.275641 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-5d6bztwql" Nov 03 01:16:20.566156 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:16:20.566127 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod477f510e_003c_4491_bc5d_a3a313f6af13.slice/crio-15715cdd65e896f3af9bcbcdaa9e6249f523e864501d6740544ba72e8d56b2bd WatchSource:0}: Error finding container 15715cdd65e896f3af9bcbcdaa9e6249f523e864501d6740544ba72e8d56b2bd: Status 404 returned error can't find the container with id 15715cdd65e896f3af9bcbcdaa9e6249f523e864501d6740544ba72e8d56b2bd Nov 03 01:16:20.566512 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:20.566488 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-5d6bztwql"] Nov 03 01:16:20.969089 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:20.968968 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-b894qjmvh" Nov 03 01:16:21.401294 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:21.400982 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-5d6bztwql" event={"ID":"477f510e-003c-4491-bc5d-a3a313f6af13","Type":"ContainerStarted","Data":"bed238877949d81d49d67a64b2b9529010a7418da8863010860f97645a6e3b70"} Nov 03 01:16:21.401294 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:21.401049 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-5d6bztwql" event={"ID":"477f510e-003c-4491-bc5d-a3a313f6af13","Type":"ContainerStarted","Data":"15715cdd65e896f3af9bcbcdaa9e6249f523e864501d6740544ba72e8d56b2bd"} Nov 03 01:16:21.401294 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:21.401094 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-5d6bztwql" Nov 03 01:16:21.484830 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:21.446792 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-5d6bztwql" podStartSLOduration=2.446767355 podStartE2EDuration="2.446767355s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:16:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:16:21.420688807 +0000 UTC m=+2123.591691905" watchObservedRunningTime="2026-11-03 01:16:21.446767355 +0000 UTC m=+2123.617770452" Nov 03 01:16:33.422700 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:33.422667 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-5d6bztwql" Nov 03 01:16:34.373606 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:34.373568 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-v8jpj"] Nov 03 01:16:34.373869 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:34.373819 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-v8jpj" podUID="9ca37a4e-e37b-453c-a937-a35fedc2af8c" containerName="istio-proxy" containerID="cri-o://be6f5ed2a61a6b545363fe16cb2e24b3dc7a069500d992a17cfcddcd50d0da54" gracePeriod=30 Nov 03 01:16:34.383543 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:34.379459 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-78zwt"] Nov 03 01:16:34.383543 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:34.379696 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-78zwt" podUID="8f5666ca-5e70-47c9-84d1-20eb2475d196" containerName="istio-proxy" containerID="cri-o://ceb17238f389cc4cbaf25cb81bb7734c5cd5e75cdc281ee51d8e6eced1a0af5e" gracePeriod=30 Nov 03 01:16:34.385578 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:34.385549 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-c5x7c"] Nov 03 01:16:34.385837 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:34.385788 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-c5x7c" podUID="652b215e-11c7-4343-86ec-bf27918c1c35" containerName="istio-proxy" containerID="cri-o://6a3743d0f74bb4c4289533a869fa36f0b7a71f52e3823905476b32e05870bf03" gracePeriod=30 Nov 03 01:16:35.015062 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:35.013666 4421 patch_prober.go:29] interesting pod/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-v8jpj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 01:16:35.015062 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:35.013743 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-v8jpj" podUID="9ca37a4e-e37b-453c-a937-a35fedc2af8c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 01:16:36.731942 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:36.731850 4421 generic.go:356] "Generic (PLEG): container finished" podID="ff2ff304-350a-4e21-8415-1c28dcb90a7d" containerID="06d83afdaa519e9974b129def6ada0b1fa89a128b24d2597a074e887c7956f04" exitCode=0 Nov 03 01:16:36.731942 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:36.731923 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-821f5a-openshift-default-7bfc657d9-bv95q" event={"ID":"ff2ff304-350a-4e21-8415-1c28dcb90a7d","Type":"ContainerDied","Data":"06d83afdaa519e9974b129def6ada0b1fa89a128b24d2597a074e887c7956f04"} Nov 03 01:16:36.732508 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:36.732492 4421 scope.go:122] "RemoveContainer" containerID="06d83afdaa519e9974b129def6ada0b1fa89a128b24d2597a074e887c7956f04" Nov 03 01:16:36.734723 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:36.734690 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:16:36.734809 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:36.734762 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:16:36.734809 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:36.734797 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:16:37.760519 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:37.760481 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-821f5a-openshift-default-7bfc657d9-bv95q" event={"ID":"ff2ff304-350a-4e21-8415-1c28dcb90a7d","Type":"ContainerStarted","Data":"9d6092a60568bf598c4fbde5716d6486743852949a4adfa6b39d6adaafcdcb95"} Nov 03 01:16:38.318013 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:38.317966 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-821f5a-openshift-default-7bfc657d9-bv95q" Nov 03 01:16:38.319413 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:38.319387 4421 patch_prober.go:29] interesting pod/e2e-extmod-a-821f5a-openshift-default-7bfc657d9-bv95q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" start-of-body= Nov 03 01:16:38.320196 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:38.319637 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-821f5a-openshift-default-7bfc657d9-bv95q" podUID="ff2ff304-350a-4e21-8415-1c28dcb90a7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" Nov 03 01:16:39.331038 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:39.330965 4421 patch_prober.go:29] interesting pod/e2e-extmod-a-821f5a-openshift-default-7bfc657d9-bv95q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" start-of-body= Nov 03 01:16:39.331632 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:39.331047 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-821f5a-openshift-default-7bfc657d9-bv95q" podUID="ff2ff304-350a-4e21-8415-1c28dcb90a7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" Nov 03 01:16:39.844893 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:39.844842 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-v8jpj" Nov 03 01:16:39.852326 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:39.852250 4421 generic.go:356] "Generic (PLEG): container finished" podID="8f5666ca-5e70-47c9-84d1-20eb2475d196" containerID="ceb17238f389cc4cbaf25cb81bb7734c5cd5e75cdc281ee51d8e6eced1a0af5e" exitCode=0 Nov 03 01:16:39.852504 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:39.852440 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-78zwt" event={"ID":"8f5666ca-5e70-47c9-84d1-20eb2475d196","Type":"ContainerDied","Data":"ceb17238f389cc4cbaf25cb81bb7734c5cd5e75cdc281ee51d8e6eced1a0af5e"} Nov 03 01:16:39.855478 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:39.855449 4421 generic.go:356] "Generic (PLEG): container finished" podID="652b215e-11c7-4343-86ec-bf27918c1c35" containerID="6a3743d0f74bb4c4289533a869fa36f0b7a71f52e3823905476b32e05870bf03" exitCode=0 Nov 03 01:16:39.855603 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:39.855536 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-c5x7c" event={"ID":"652b215e-11c7-4343-86ec-bf27918c1c35","Type":"ContainerDied","Data":"6a3743d0f74bb4c4289533a869fa36f0b7a71f52e3823905476b32e05870bf03"} Nov 03 01:16:39.858067 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:39.858031 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ca37a4e-e37b-453c-a937-a35fedc2af8c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ca37a4e-e37b-453c-a937-a35fedc2af8c-credential-socket\") pod \"9ca37a4e-e37b-453c-a937-a35fedc2af8c\" (UID: \"9ca37a4e-e37b-453c-a937-a35fedc2af8c\") " Nov 03 01:16:39.858211 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:39.858102 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ca37a4e-e37b-453c-a937-a35fedc2af8c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ca37a4e-e37b-453c-a937-a35fedc2af8c-workload-certs\") pod \"9ca37a4e-e37b-453c-a937-a35fedc2af8c\" (UID: \"9ca37a4e-e37b-453c-a937-a35fedc2af8c\") " Nov 03 01:16:39.858211 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:39.858199 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ca37a4e-e37b-453c-a937-a35fedc2af8c-kube-api-access-4x8n4\" (UniqueName: \"kubernetes.io/projected/9ca37a4e-e37b-453c-a937-a35fedc2af8c-kube-api-access-4x8n4\") pod \"9ca37a4e-e37b-453c-a937-a35fedc2af8c\" (UID: \"9ca37a4e-e37b-453c-a937-a35fedc2af8c\") " Nov 03 01:16:39.858352 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:39.858216 4421 generic.go:356] "Generic (PLEG): container finished" podID="9ca37a4e-e37b-453c-a937-a35fedc2af8c" containerID="be6f5ed2a61a6b545363fe16cb2e24b3dc7a069500d992a17cfcddcd50d0da54" exitCode=0 Nov 03 01:16:39.858352 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:39.858246 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ca37a4e-e37b-453c-a937-a35fedc2af8c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ca37a4e-e37b-453c-a937-a35fedc2af8c-istiod-ca-cert\") pod \"9ca37a4e-e37b-453c-a937-a35fedc2af8c\" (UID: \"9ca37a4e-e37b-453c-a937-a35fedc2af8c\") " Nov 03 01:16:39.858352 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:39.858319 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-v8jpj" event={"ID":"9ca37a4e-e37b-453c-a937-a35fedc2af8c","Type":"ContainerDied","Data":"be6f5ed2a61a6b545363fe16cb2e24b3dc7a069500d992a17cfcddcd50d0da54"} Nov 03 01:16:39.858516 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:39.858361 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-v8jpj" event={"ID":"9ca37a4e-e37b-453c-a937-a35fedc2af8c","Type":"ContainerDied","Data":"99f44fa0a7b48f58111b3c30a7b722fba3d410269bb9249b7a5ee9c1f76f5c1a"} Nov 03 01:16:39.858516 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:39.858360 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ca37a4e-e37b-453c-a937-a35fedc2af8c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ca37a4e-e37b-453c-a937-a35fedc2af8c-workload-socket\") pod \"9ca37a4e-e37b-453c-a937-a35fedc2af8c\" (UID: \"9ca37a4e-e37b-453c-a937-a35fedc2af8c\") " Nov 03 01:16:39.858516 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:39.858508 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ca37a4e-e37b-453c-a937-a35fedc2af8c-istio-token\" (UniqueName: \"kubernetes.io/projected/9ca37a4e-e37b-453c-a937-a35fedc2af8c-istio-token\") pod \"9ca37a4e-e37b-453c-a937-a35fedc2af8c\" (UID: \"9ca37a4e-e37b-453c-a937-a35fedc2af8c\") " Nov 03 01:16:39.858684 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:39.858588 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ca37a4e-e37b-453c-a937-a35fedc2af8c-workload-socket" pod "9ca37a4e-e37b-453c-a937-a35fedc2af8c" (UID: "9ca37a4e-e37b-453c-a937-a35fedc2af8c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:16:39.858684 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:39.858676 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ca37a4e-e37b-453c-a937-a35fedc2af8c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ca37a4e-e37b-453c-a937-a35fedc2af8c-istio-data\") pod \"9ca37a4e-e37b-453c-a937-a35fedc2af8c\" (UID: \"9ca37a4e-e37b-453c-a937-a35fedc2af8c\") " Nov 03 01:16:39.858795 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:39.858705 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9ca37a4e-e37b-453c-a937-a35fedc2af8c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ca37a4e-e37b-453c-a937-a35fedc2af8c-istio-podinfo\") pod \"9ca37a4e-e37b-453c-a937-a35fedc2af8c\" (UID: \"9ca37a4e-e37b-453c-a937-a35fedc2af8c\") " Nov 03 01:16:39.858795 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:39.858740 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ca37a4e-e37b-453c-a937-a35fedc2af8c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ca37a4e-e37b-453c-a937-a35fedc2af8c-istio-envoy\") pod \"9ca37a4e-e37b-453c-a937-a35fedc2af8c\" (UID: \"9ca37a4e-e37b-453c-a937-a35fedc2af8c\") " Nov 03 01:16:39.858901 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:39.858801 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ca37a4e-e37b-453c-a937-a35fedc2af8c-credential-socket" pod "9ca37a4e-e37b-453c-a937-a35fedc2af8c" (UID: "9ca37a4e-e37b-453c-a937-a35fedc2af8c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:16:39.859046 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:39.858980 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ca37a4e-e37b-453c-a937-a35fedc2af8c-workload-certs" pod "9ca37a4e-e37b-453c-a937-a35fedc2af8c" (UID: "9ca37a4e-e37b-453c-a937-a35fedc2af8c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:16:39.859317 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:39.859242 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ca37a4e-e37b-453c-a937-a35fedc2af8c-istiod-ca-cert" pod "9ca37a4e-e37b-453c-a937-a35fedc2af8c" (UID: "9ca37a4e-e37b-453c-a937-a35fedc2af8c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:16:39.859427 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:39.859403 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-v8jpj" Nov 03 01:16:39.859638 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:39.859605 4421 scope.go:122] "RemoveContainer" containerID="be6f5ed2a61a6b545363fe16cb2e24b3dc7a069500d992a17cfcddcd50d0da54" Nov 03 01:16:39.860061 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:39.860034 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ca37a4e-e37b-453c-a937-a35fedc2af8c-istio-data" pod "9ca37a4e-e37b-453c-a937-a35fedc2af8c" (UID: "9ca37a4e-e37b-453c-a937-a35fedc2af8c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:16:39.861790 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:39.861768 4421 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ca37a4e-e37b-453c-a937-a35fedc2af8c-istio-data\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:16:39.861894 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:39.861794 4421 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ca37a4e-e37b-453c-a937-a35fedc2af8c-credential-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:16:39.861894 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:39.861803 4421 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ca37a4e-e37b-453c-a937-a35fedc2af8c-workload-certs\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:16:39.861894 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:39.861811 4421 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ca37a4e-e37b-453c-a937-a35fedc2af8c-istiod-ca-cert\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:16:39.861894 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:39.861820 4421 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ca37a4e-e37b-453c-a937-a35fedc2af8c-workload-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:16:39.862174 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:39.862145 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9ca37a4e-e37b-453c-a937-a35fedc2af8c-istio-podinfo" pod "9ca37a4e-e37b-453c-a937-a35fedc2af8c" (UID: "9ca37a4e-e37b-453c-a937-a35fedc2af8c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:16:39.862962 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:39.862934 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ca37a4e-e37b-453c-a937-a35fedc2af8c-istio-token" pod "9ca37a4e-e37b-453c-a937-a35fedc2af8c" (UID: "9ca37a4e-e37b-453c-a937-a35fedc2af8c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:16:39.863189 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:39.863126 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ca37a4e-e37b-453c-a937-a35fedc2af8c-istio-envoy" pod "9ca37a4e-e37b-453c-a937-a35fedc2af8c" (UID: "9ca37a4e-e37b-453c-a937-a35fedc2af8c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:16:39.864154 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:39.864128 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ca37a4e-e37b-453c-a937-a35fedc2af8c-kube-api-access-4x8n4" pod "9ca37a4e-e37b-453c-a937-a35fedc2af8c" (UID: "9ca37a4e-e37b-453c-a937-a35fedc2af8c"). InnerVolumeSpecName "kube-api-access-4x8n4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:16:39.897017 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:39.892396 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-821f5a/payload-processing-external-model-e2e-extmod-a-821f5a-6f64zw8pg"] Nov 03 01:16:39.897017 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:39.893970 4421 cpu_manager.go:414] "containerMap: removing container" podUID="9ca37a4e-e37b-453c-a937-a35fedc2af8c" containerName="istio-proxy" Nov 03 01:16:39.897017 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:39.893995 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ca37a4e-e37b-453c-a937-a35fedc2af8c" containerName="istio-proxy" Nov 03 01:16:39.897017 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:39.894018 4421 cpu_manager.go:414] "containerMap: removing container" podUID="9ca37a4e-e37b-453c-a937-a35fedc2af8c" containerName="istio-proxy" Nov 03 01:16:39.897017 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:39.894027 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ca37a4e-e37b-453c-a937-a35fedc2af8c" containerName="istio-proxy" Nov 03 01:16:39.897017 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:39.894333 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ca37a4e-e37b-453c-a937-a35fedc2af8c" containerName="istio-proxy" Nov 03 01:16:39.903392 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:39.901591 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-821f5a/payload-processing-external-model-e2e-extmod-a-821f5a-6f64zw8pg" Nov 03 01:16:39.937441 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:39.905081 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-821f5a/payload-processing-external-model-e2e-extmod-a-821f5a-6f64zw8pg"] Nov 03 01:16:39.937441 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:39.935419 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-821f5a-openshift-default-7bfc657d9-bv95q" Nov 03 01:16:39.937441 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:39.935564 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-821f5a\"/\"e2e-cmp-legacy-a89f97-provider-cred\"" Nov 03 01:16:39.937441 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:39.935709 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-821f5a\"/\"routing-overlay\"" Nov 03 01:16:39.937441 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:39.935789 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-821f5a\"/\"payload-processing-external-model-e2e--1074f2f6-dockercfg-xg5md\"" Nov 03 01:16:39.937441 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:39.935880 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-821f5a\"/\"kube-root-ca.crt\"" Nov 03 01:16:39.937441 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:39.935934 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-821f5a\"/\"openshift-service-ca.crt\"" Nov 03 01:16:39.937441 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:39.936001 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-821f5a\"/\"payload-processing-external-model-plugins-e2e-extmod-a-821f5a\"" Nov 03 01:16:39.944928 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:39.944899 4421 scope.go:122] "RemoveContainer" containerID="2f7c8781a132b4ccff1df6db612942c2b474bda21f3232903c72e69ab3c02219" Nov 03 01:16:39.945241 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:39.945223 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-78zwt" Nov 03 01:16:39.946592 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:39.946573 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-c5x7c" Nov 03 01:16:39.965922 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:39.963935 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/652b215e-11c7-4343-86ec-bf27918c1c35-istiod-ca-cert" pod "652b215e-11c7-4343-86ec-bf27918c1c35" (UID: "652b215e-11c7-4343-86ec-bf27918c1c35"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:16:39.965922 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:39.964009 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/652b215e-11c7-4343-86ec-bf27918c1c35-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/652b215e-11c7-4343-86ec-bf27918c1c35-istiod-ca-cert\") pod \"652b215e-11c7-4343-86ec-bf27918c1c35\" (UID: \"652b215e-11c7-4343-86ec-bf27918c1c35\") " Nov 03 01:16:39.965922 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:39.964088 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f5666ca-5e70-47c9-84d1-20eb2475d196-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f5666ca-5e70-47c9-84d1-20eb2475d196-istio-data\") pod \"8f5666ca-5e70-47c9-84d1-20eb2475d196\" (UID: \"8f5666ca-5e70-47c9-84d1-20eb2475d196\") " Nov 03 01:16:39.965922 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:39.964159 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/652b215e-11c7-4343-86ec-bf27918c1c35-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/652b215e-11c7-4343-86ec-bf27918c1c35-workload-socket\") pod \"652b215e-11c7-4343-86ec-bf27918c1c35\" (UID: \"652b215e-11c7-4343-86ec-bf27918c1c35\") " Nov 03 01:16:39.965922 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:39.964200 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/652b215e-11c7-4343-86ec-bf27918c1c35-istio-token\" (UniqueName: \"kubernetes.io/projected/652b215e-11c7-4343-86ec-bf27918c1c35-istio-token\") pod \"652b215e-11c7-4343-86ec-bf27918c1c35\" (UID: \"652b215e-11c7-4343-86ec-bf27918c1c35\") " Nov 03 01:16:39.965922 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:39.964304 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/652b215e-11c7-4343-86ec-bf27918c1c35-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/652b215e-11c7-4343-86ec-bf27918c1c35-istio-data\") pod \"652b215e-11c7-4343-86ec-bf27918c1c35\" (UID: \"652b215e-11c7-4343-86ec-bf27918c1c35\") " Nov 03 01:16:39.965922 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:39.964416 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f5666ca-5e70-47c9-84d1-20eb2475d196-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f5666ca-5e70-47c9-84d1-20eb2475d196-workload-socket\") pod \"8f5666ca-5e70-47c9-84d1-20eb2475d196\" (UID: \"8f5666ca-5e70-47c9-84d1-20eb2475d196\") " Nov 03 01:16:39.965922 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:39.964477 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f5666ca-5e70-47c9-84d1-20eb2475d196-istio-token\" (UniqueName: \"kubernetes.io/projected/8f5666ca-5e70-47c9-84d1-20eb2475d196-istio-token\") pod \"8f5666ca-5e70-47c9-84d1-20eb2475d196\" (UID: \"8f5666ca-5e70-47c9-84d1-20eb2475d196\") " Nov 03 01:16:39.965922 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:39.964652 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f5666ca-5e70-47c9-84d1-20eb2475d196-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f5666ca-5e70-47c9-84d1-20eb2475d196-workload-certs\") pod \"8f5666ca-5e70-47c9-84d1-20eb2475d196\" (UID: \"8f5666ca-5e70-47c9-84d1-20eb2475d196\") " Nov 03 01:16:39.965922 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:39.964738 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/652b215e-11c7-4343-86ec-bf27918c1c35-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/652b215e-11c7-4343-86ec-bf27918c1c35-workload-certs\") pod \"652b215e-11c7-4343-86ec-bf27918c1c35\" (UID: \"652b215e-11c7-4343-86ec-bf27918c1c35\") " Nov 03 01:16:39.965922 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:39.964785 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f5666ca-5e70-47c9-84d1-20eb2475d196-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f5666ca-5e70-47c9-84d1-20eb2475d196-credential-socket\") pod \"8f5666ca-5e70-47c9-84d1-20eb2475d196\" (UID: \"8f5666ca-5e70-47c9-84d1-20eb2475d196\") " Nov 03 01:16:39.965922 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:39.964886 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8f5666ca-5e70-47c9-84d1-20eb2475d196-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f5666ca-5e70-47c9-84d1-20eb2475d196-istio-podinfo\") pod \"8f5666ca-5e70-47c9-84d1-20eb2475d196\" (UID: \"8f5666ca-5e70-47c9-84d1-20eb2475d196\") " Nov 03 01:16:39.965922 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:39.964949 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/652b215e-11c7-4343-86ec-bf27918c1c35-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/652b215e-11c7-4343-86ec-bf27918c1c35-istio-envoy\") pod \"652b215e-11c7-4343-86ec-bf27918c1c35\" (UID: \"652b215e-11c7-4343-86ec-bf27918c1c35\") " Nov 03 01:16:39.965922 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:39.965023 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f5666ca-5e70-47c9-84d1-20eb2475d196-kube-api-access-vr6z4\" (UniqueName: \"kubernetes.io/projected/8f5666ca-5e70-47c9-84d1-20eb2475d196-kube-api-access-vr6z4\") pod \"8f5666ca-5e70-47c9-84d1-20eb2475d196\" (UID: \"8f5666ca-5e70-47c9-84d1-20eb2475d196\") " Nov 03 01:16:39.965922 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:39.965095 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f5666ca-5e70-47c9-84d1-20eb2475d196-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f5666ca-5e70-47c9-84d1-20eb2475d196-istiod-ca-cert\") pod \"8f5666ca-5e70-47c9-84d1-20eb2475d196\" (UID: \"8f5666ca-5e70-47c9-84d1-20eb2475d196\") " Nov 03 01:16:39.965922 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:39.965223 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/652b215e-11c7-4343-86ec-bf27918c1c35-kube-api-access-qmwf5\" (UniqueName: \"kubernetes.io/projected/652b215e-11c7-4343-86ec-bf27918c1c35-kube-api-access-qmwf5\") pod \"652b215e-11c7-4343-86ec-bf27918c1c35\" (UID: \"652b215e-11c7-4343-86ec-bf27918c1c35\") " Nov 03 01:16:39.965922 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:39.965348 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/652b215e-11c7-4343-86ec-bf27918c1c35-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/652b215e-11c7-4343-86ec-bf27918c1c35-credential-socket\") pod \"652b215e-11c7-4343-86ec-bf27918c1c35\" (UID: \"652b215e-11c7-4343-86ec-bf27918c1c35\") " Nov 03 01:16:39.965922 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:39.965412 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/652b215e-11c7-4343-86ec-bf27918c1c35-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/652b215e-11c7-4343-86ec-bf27918c1c35-istio-podinfo\") pod \"652b215e-11c7-4343-86ec-bf27918c1c35\" (UID: \"652b215e-11c7-4343-86ec-bf27918c1c35\") " Nov 03 01:16:39.965922 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:39.965480 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f5666ca-5e70-47c9-84d1-20eb2475d196-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f5666ca-5e70-47c9-84d1-20eb2475d196-istio-envoy\") pod \"8f5666ca-5e70-47c9-84d1-20eb2475d196\" (UID: \"8f5666ca-5e70-47c9-84d1-20eb2475d196\") " Nov 03 01:16:39.970688 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:39.966013 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msh9d\" (UniqueName: \"kubernetes.io/projected/ad8efe73-1c3e-4086-b626-212210ac2d0d-kube-api-access-msh9d\") pod \"payload-processing-external-model-e2e-extmod-a-821f5a-6f64zw8pg\" (UID: \"ad8efe73-1c3e-4086-b626-212210ac2d0d\") " pod="ai-tenant-e2e-extmod-a-821f5a/payload-processing-external-model-e2e-extmod-a-821f5a-6f64zw8pg" Nov 03 01:16:39.970688 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:39.966187 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad8efe73-1c3e-4086-b626-212210ac2d0d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-821f5a-6f64zw8pg\" (UID: \"ad8efe73-1c3e-4086-b626-212210ac2d0d\") " pod="ai-tenant-e2e-extmod-a-821f5a/payload-processing-external-model-e2e-extmod-a-821f5a-6f64zw8pg" Nov 03 01:16:39.970688 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:39.966268 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ad8efe73-1c3e-4086-b626-212210ac2d0d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-821f5a-6f64zw8pg\" (UID: \"ad8efe73-1c3e-4086-b626-212210ac2d0d\") " pod="ai-tenant-e2e-extmod-a-821f5a/payload-processing-external-model-e2e-extmod-a-821f5a-6f64zw8pg" Nov 03 01:16:39.970688 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:39.966486 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ad8efe73-1c3e-4086-b626-212210ac2d0d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-821f5a-6f64zw8pg\" (UID: \"ad8efe73-1c3e-4086-b626-212210ac2d0d\") " pod="ai-tenant-e2e-extmod-a-821f5a/payload-processing-external-model-e2e-extmod-a-821f5a-6f64zw8pg" Nov 03 01:16:39.970688 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:39.967039 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4x8n4\" (UniqueName: \"kubernetes.io/projected/9ca37a4e-e37b-453c-a937-a35fedc2af8c-kube-api-access-4x8n4\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:16:39.970688 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:39.967081 4421 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/652b215e-11c7-4343-86ec-bf27918c1c35-istiod-ca-cert\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:16:39.970688 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:39.967098 4421 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ca37a4e-e37b-453c-a937-a35fedc2af8c-istio-token\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:16:39.970688 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:39.967111 4421 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ca37a4e-e37b-453c-a937-a35fedc2af8c-istio-podinfo\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:16:39.970688 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:39.967136 4421 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ca37a4e-e37b-453c-a937-a35fedc2af8c-istio-envoy\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:16:39.970688 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:39.967720 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f5666ca-5e70-47c9-84d1-20eb2475d196-istio-data" pod "8f5666ca-5e70-47c9-84d1-20eb2475d196" (UID: "8f5666ca-5e70-47c9-84d1-20eb2475d196"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:16:39.970688 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:39.969435 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f5666ca-5e70-47c9-84d1-20eb2475d196-workload-certs" pod "8f5666ca-5e70-47c9-84d1-20eb2475d196" (UID: "8f5666ca-5e70-47c9-84d1-20eb2475d196"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:16:39.970688 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:39.969598 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/652b215e-11c7-4343-86ec-bf27918c1c35-istio-data" pod "652b215e-11c7-4343-86ec-bf27918c1c35" (UID: "652b215e-11c7-4343-86ec-bf27918c1c35"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:16:39.970688 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:39.969696 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f5666ca-5e70-47c9-84d1-20eb2475d196-workload-socket" pod "8f5666ca-5e70-47c9-84d1-20eb2475d196" (UID: "8f5666ca-5e70-47c9-84d1-20eb2475d196"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:16:39.970688 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:39.970295 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f5666ca-5e70-47c9-84d1-20eb2475d196-istiod-ca-cert" pod "8f5666ca-5e70-47c9-84d1-20eb2475d196" (UID: "8f5666ca-5e70-47c9-84d1-20eb2475d196"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:16:39.972419 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:39.971773 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/652b215e-11c7-4343-86ec-bf27918c1c35-workload-socket" pod "652b215e-11c7-4343-86ec-bf27918c1c35" (UID: "652b215e-11c7-4343-86ec-bf27918c1c35"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:16:39.972419 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:39.972301 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/652b215e-11c7-4343-86ec-bf27918c1c35-credential-socket" pod "652b215e-11c7-4343-86ec-bf27918c1c35" (UID: "652b215e-11c7-4343-86ec-bf27918c1c35"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:16:39.972554 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:39.972499 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/652b215e-11c7-4343-86ec-bf27918c1c35-workload-certs" pod "652b215e-11c7-4343-86ec-bf27918c1c35" (UID: "652b215e-11c7-4343-86ec-bf27918c1c35"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:16:39.972787 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:39.972761 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f5666ca-5e70-47c9-84d1-20eb2475d196-credential-socket" pod "8f5666ca-5e70-47c9-84d1-20eb2475d196" (UID: "8f5666ca-5e70-47c9-84d1-20eb2475d196"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:16:39.977444 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:39.976450 4421 scope.go:122] "RemoveContainer" containerID="be6f5ed2a61a6b545363fe16cb2e24b3dc7a069500d992a17cfcddcd50d0da54" Nov 03 01:16:39.977444 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:39.977415 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f5666ca-5e70-47c9-84d1-20eb2475d196-kube-api-access-vr6z4" pod "8f5666ca-5e70-47c9-84d1-20eb2475d196" (UID: "8f5666ca-5e70-47c9-84d1-20eb2475d196"). InnerVolumeSpecName "kube-api-access-vr6z4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:16:39.978390 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:16:39.978301 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be6f5ed2a61a6b545363fe16cb2e24b3dc7a069500d992a17cfcddcd50d0da54\": container with ID starting with be6f5ed2a61a6b545363fe16cb2e24b3dc7a069500d992a17cfcddcd50d0da54 not found: ID does not exist" containerID="be6f5ed2a61a6b545363fe16cb2e24b3dc7a069500d992a17cfcddcd50d0da54" Nov 03 01:16:39.978390 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:39.978342 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be6f5ed2a61a6b545363fe16cb2e24b3dc7a069500d992a17cfcddcd50d0da54"} err="failed to get container status \"be6f5ed2a61a6b545363fe16cb2e24b3dc7a069500d992a17cfcddcd50d0da54\": rpc error: code = NotFound desc = could not find container \"be6f5ed2a61a6b545363fe16cb2e24b3dc7a069500d992a17cfcddcd50d0da54\": container with ID starting with be6f5ed2a61a6b545363fe16cb2e24b3dc7a069500d992a17cfcddcd50d0da54 not found: ID does not exist" Nov 03 01:16:39.978390 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:39.978369 4421 scope.go:122] "RemoveContainer" containerID="2f7c8781a132b4ccff1df6db612942c2b474bda21f3232903c72e69ab3c02219" Nov 03 01:16:39.978980 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:16:39.978904 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f7c8781a132b4ccff1df6db612942c2b474bda21f3232903c72e69ab3c02219\": container with ID starting with 2f7c8781a132b4ccff1df6db612942c2b474bda21f3232903c72e69ab3c02219 not found: ID does not exist" containerID="2f7c8781a132b4ccff1df6db612942c2b474bda21f3232903c72e69ab3c02219" Nov 03 01:16:39.978980 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:39.978940 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f7c8781a132b4ccff1df6db612942c2b474bda21f3232903c72e69ab3c02219"} err="failed to get container status \"2f7c8781a132b4ccff1df6db612942c2b474bda21f3232903c72e69ab3c02219\": rpc error: code = NotFound desc = could not find container \"2f7c8781a132b4ccff1df6db612942c2b474bda21f3232903c72e69ab3c02219\": container with ID starting with 2f7c8781a132b4ccff1df6db612942c2b474bda21f3232903c72e69ab3c02219 not found: ID does not exist" Nov 03 01:16:39.984404 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:39.984368 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/652b215e-11c7-4343-86ec-bf27918c1c35-istio-envoy" pod "652b215e-11c7-4343-86ec-bf27918c1c35" (UID: "652b215e-11c7-4343-86ec-bf27918c1c35"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:16:39.985586 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:39.984739 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/652b215e-11c7-4343-86ec-bf27918c1c35-istio-token" pod "652b215e-11c7-4343-86ec-bf27918c1c35" (UID: "652b215e-11c7-4343-86ec-bf27918c1c35"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:16:39.985926 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:39.985885 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/652b215e-11c7-4343-86ec-bf27918c1c35-istio-podinfo" pod "652b215e-11c7-4343-86ec-bf27918c1c35" (UID: "652b215e-11c7-4343-86ec-bf27918c1c35"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:16:39.985926 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:39.985899 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f5666ca-5e70-47c9-84d1-20eb2475d196-istio-envoy" pod "8f5666ca-5e70-47c9-84d1-20eb2475d196" (UID: "8f5666ca-5e70-47c9-84d1-20eb2475d196"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:16:39.985926 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:39.985916 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8f5666ca-5e70-47c9-84d1-20eb2475d196-istio-podinfo" pod "8f5666ca-5e70-47c9-84d1-20eb2475d196" (UID: "8f5666ca-5e70-47c9-84d1-20eb2475d196"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:16:39.988710 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:39.986329 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/652b215e-11c7-4343-86ec-bf27918c1c35-kube-api-access-qmwf5" pod "652b215e-11c7-4343-86ec-bf27918c1c35" (UID: "652b215e-11c7-4343-86ec-bf27918c1c35"). InnerVolumeSpecName "kube-api-access-qmwf5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:16:40.051105 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:40.051074 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-wxr6x"] Nov 03 01:16:40.052696 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:40.052666 4421 cpu_manager.go:414] "containerMap: removing container" podUID="652b215e-11c7-4343-86ec-bf27918c1c35" containerName="istio-proxy" Nov 03 01:16:40.052909 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:40.052893 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="652b215e-11c7-4343-86ec-bf27918c1c35" containerName="istio-proxy" Nov 03 01:16:40.053097 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:40.053084 4421 cpu_manager.go:414] "containerMap: removing container" podUID="8f5666ca-5e70-47c9-84d1-20eb2475d196" containerName="istio-proxy" Nov 03 01:16:40.053200 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:40.053188 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f5666ca-5e70-47c9-84d1-20eb2475d196" containerName="istio-proxy" Nov 03 01:16:40.053584 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:40.053567 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f5666ca-5e70-47c9-84d1-20eb2475d196" containerName="istio-proxy" Nov 03 01:16:40.053707 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:40.053694 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ca37a4e-e37b-453c-a937-a35fedc2af8c" containerName="istio-proxy" Nov 03 01:16:40.053818 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:40.053806 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="652b215e-11c7-4343-86ec-bf27918c1c35" containerName="istio-proxy" Nov 03 01:16:40.062534 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:40.062493 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f5666ca-5e70-47c9-84d1-20eb2475d196-istio-token" pod "8f5666ca-5e70-47c9-84d1-20eb2475d196" (UID: "8f5666ca-5e70-47c9-84d1-20eb2475d196"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:16:40.069295 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:40.066430 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-wxr6x" Nov 03 01:16:40.069295 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:40.068794 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msh9d\" (UniqueName: \"kubernetes.io/projected/ad8efe73-1c3e-4086-b626-212210ac2d0d-kube-api-access-msh9d\") pod \"payload-processing-external-model-e2e-extmod-a-821f5a-6f64zw8pg\" (UID: \"ad8efe73-1c3e-4086-b626-212210ac2d0d\") " pod="ai-tenant-e2e-extmod-a-821f5a/payload-processing-external-model-e2e-extmod-a-821f5a-6f64zw8pg" Nov 03 01:16:40.069295 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:40.068977 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad8efe73-1c3e-4086-b626-212210ac2d0d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-821f5a-6f64zw8pg\" (UID: \"ad8efe73-1c3e-4086-b626-212210ac2d0d\") " pod="ai-tenant-e2e-extmod-a-821f5a/payload-processing-external-model-e2e-extmod-a-821f5a-6f64zw8pg" Nov 03 01:16:40.069295 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:40.069070 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ad8efe73-1c3e-4086-b626-212210ac2d0d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-821f5a-6f64zw8pg\" (UID: \"ad8efe73-1c3e-4086-b626-212210ac2d0d\") " pod="ai-tenant-e2e-extmod-a-821f5a/payload-processing-external-model-e2e-extmod-a-821f5a-6f64zw8pg" Nov 03 01:16:40.069295 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:40.069198 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ad8efe73-1c3e-4086-b626-212210ac2d0d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-821f5a-6f64zw8pg\" (UID: \"ad8efe73-1c3e-4086-b626-212210ac2d0d\") " pod="ai-tenant-e2e-extmod-a-821f5a/payload-processing-external-model-e2e-extmod-a-821f5a-6f64zw8pg" Nov 03 01:16:40.070458 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:40.070007 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad8efe73-1c3e-4086-b626-212210ac2d0d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-821f5a-6f64zw8pg\" (UID: \"ad8efe73-1c3e-4086-b626-212210ac2d0d\") " pod="ai-tenant-e2e-extmod-a-821f5a/payload-processing-external-model-e2e-extmod-a-821f5a-6f64zw8pg" Nov 03 01:16:40.070458 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:40.070160 4421 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f5666ca-5e70-47c9-84d1-20eb2475d196-workload-certs\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:16:40.070458 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:40.070172 4421 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/652b215e-11c7-4343-86ec-bf27918c1c35-workload-certs\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:16:40.070458 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:40.070184 4421 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f5666ca-5e70-47c9-84d1-20eb2475d196-credential-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:16:40.070458 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:40.070194 4421 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f5666ca-5e70-47c9-84d1-20eb2475d196-istio-podinfo\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:16:40.070458 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:40.070203 4421 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/652b215e-11c7-4343-86ec-bf27918c1c35-istio-envoy\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:16:40.070458 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:40.070212 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vr6z4\" (UniqueName: \"kubernetes.io/projected/8f5666ca-5e70-47c9-84d1-20eb2475d196-kube-api-access-vr6z4\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:16:40.070458 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:40.070221 4421 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f5666ca-5e70-47c9-84d1-20eb2475d196-istiod-ca-cert\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:16:40.070458 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:40.070230 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmwf5\" (UniqueName: \"kubernetes.io/projected/652b215e-11c7-4343-86ec-bf27918c1c35-kube-api-access-qmwf5\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:16:40.070458 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:40.070239 4421 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/652b215e-11c7-4343-86ec-bf27918c1c35-credential-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:16:40.070458 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:40.070249 4421 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/652b215e-11c7-4343-86ec-bf27918c1c35-istio-podinfo\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:16:40.070458 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:40.070259 4421 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f5666ca-5e70-47c9-84d1-20eb2475d196-istio-envoy\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:16:40.070458 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:40.070267 4421 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f5666ca-5e70-47c9-84d1-20eb2475d196-istio-data\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:16:40.070458 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:40.070312 4421 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/652b215e-11c7-4343-86ec-bf27918c1c35-workload-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:16:40.070458 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:40.070321 4421 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/652b215e-11c7-4343-86ec-bf27918c1c35-istio-token\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:16:40.070458 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:40.070329 4421 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/652b215e-11c7-4343-86ec-bf27918c1c35-istio-data\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:16:40.070458 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:40.070338 4421 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f5666ca-5e70-47c9-84d1-20eb2475d196-workload-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:16:40.070458 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:40.070346 4421 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f5666ca-5e70-47c9-84d1-20eb2475d196-istio-token\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:16:40.073052 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:40.071480 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ad8efe73-1c3e-4086-b626-212210ac2d0d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-821f5a-6f64zw8pg\" (UID: \"ad8efe73-1c3e-4086-b626-212210ac2d0d\") " pod="ai-tenant-e2e-extmod-a-821f5a/payload-processing-external-model-e2e-extmod-a-821f5a-6f64zw8pg" Nov 03 01:16:40.073173 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:40.066332 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-wxr6x"] Nov 03 01:16:40.073617 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:40.073598 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-zxs8d"] Nov 03 01:16:40.078655 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:40.078537 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-zxs8d" Nov 03 01:16:40.082252 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:40.082222 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-zxs8d"] Nov 03 01:16:40.083218 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:40.083192 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ad8efe73-1c3e-4086-b626-212210ac2d0d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-821f5a-6f64zw8pg\" (UID: \"ad8efe73-1c3e-4086-b626-212210ac2d0d\") " pod="ai-tenant-e2e-extmod-a-821f5a/payload-processing-external-model-e2e-extmod-a-821f5a-6f64zw8pg" Nov 03 01:16:40.090190 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:40.090163 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-msh9d\" (UniqueName: \"kubernetes.io/projected/ad8efe73-1c3e-4086-b626-212210ac2d0d-kube-api-access-msh9d\") pod \"payload-processing-external-model-e2e-extmod-a-821f5a-6f64zw8pg\" (UID: \"ad8efe73-1c3e-4086-b626-212210ac2d0d\") " pod="ai-tenant-e2e-extmod-a-821f5a/payload-processing-external-model-e2e-extmod-a-821f5a-6f64zw8pg" Nov 03 01:16:40.187998 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:40.187970 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d56b90c8-05c3-4c59-a204-79eb70d4a172-istio-envoy\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-wxr6x\" (UID: \"d56b90c8-05c3-4c59-a204-79eb70d4a172\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-wxr6x" Nov 03 01:16:40.188262 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:40.188241 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d56b90c8-05c3-4c59-a204-79eb70d4a172-workload-socket\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-wxr6x\" (UID: \"d56b90c8-05c3-4c59-a204-79eb70d4a172\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-wxr6x" Nov 03 01:16:40.190194 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:40.188472 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e5cb5e3-4c37-4ae7-9c6b-ef145cb87eb3-credential-socket\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-zxs8d\" (UID: \"6e5cb5e3-4c37-4ae7-9c6b-ef145cb87eb3\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-zxs8d" Nov 03 01:16:40.190362 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:40.190248 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d56b90c8-05c3-4c59-a204-79eb70d4a172-credential-socket\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-wxr6x\" (UID: \"d56b90c8-05c3-4c59-a204-79eb70d4a172\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-wxr6x" Nov 03 01:16:40.190453 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:40.190437 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e5cb5e3-4c37-4ae7-9c6b-ef145cb87eb3-istio-podinfo\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-zxs8d\" (UID: \"6e5cb5e3-4c37-4ae7-9c6b-ef145cb87eb3\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-zxs8d" Nov 03 01:16:40.190553 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:40.190477 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e5cb5e3-4c37-4ae7-9c6b-ef145cb87eb3-istio-token\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-zxs8d\" (UID: \"6e5cb5e3-4c37-4ae7-9c6b-ef145cb87eb3\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-zxs8d" Nov 03 01:16:40.190620 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:40.190594 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d56b90c8-05c3-4c59-a204-79eb70d4a172-workload-certs\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-wxr6x\" (UID: \"d56b90c8-05c3-4c59-a204-79eb70d4a172\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-wxr6x" Nov 03 01:16:40.190684 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:40.190661 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e5cb5e3-4c37-4ae7-9c6b-ef145cb87eb3-workload-socket\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-zxs8d\" (UID: \"6e5cb5e3-4c37-4ae7-9c6b-ef145cb87eb3\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-zxs8d" Nov 03 01:16:40.190926 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:40.190887 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e5cb5e3-4c37-4ae7-9c6b-ef145cb87eb3-istio-envoy\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-zxs8d\" (UID: \"6e5cb5e3-4c37-4ae7-9c6b-ef145cb87eb3\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-zxs8d" Nov 03 01:16:40.192136 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:40.192103 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e5cb5e3-4c37-4ae7-9c6b-ef145cb87eb3-istiod-ca-cert\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-zxs8d\" (UID: \"6e5cb5e3-4c37-4ae7-9c6b-ef145cb87eb3\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-zxs8d" Nov 03 01:16:40.192255 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:40.192238 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d56b90c8-05c3-4c59-a204-79eb70d4a172-istio-podinfo\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-wxr6x\" (UID: \"d56b90c8-05c3-4c59-a204-79eb70d4a172\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-wxr6x" Nov 03 01:16:40.192342 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:40.192264 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xd59p\" (UniqueName: \"kubernetes.io/projected/6e5cb5e3-4c37-4ae7-9c6b-ef145cb87eb3-kube-api-access-xd59p\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-zxs8d\" (UID: \"6e5cb5e3-4c37-4ae7-9c6b-ef145cb87eb3\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-zxs8d" Nov 03 01:16:40.192410 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:40.192351 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d56b90c8-05c3-4c59-a204-79eb70d4a172-istio-token\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-wxr6x\" (UID: \"d56b90c8-05c3-4c59-a204-79eb70d4a172\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-wxr6x" Nov 03 01:16:40.192410 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:40.192385 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjnw8\" (UniqueName: \"kubernetes.io/projected/d56b90c8-05c3-4c59-a204-79eb70d4a172-kube-api-access-tjnw8\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-wxr6x\" (UID: \"d56b90c8-05c3-4c59-a204-79eb70d4a172\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-wxr6x" Nov 03 01:16:40.192535 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:40.192414 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d56b90c8-05c3-4c59-a204-79eb70d4a172-istio-data\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-wxr6x\" (UID: \"d56b90c8-05c3-4c59-a204-79eb70d4a172\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-wxr6x" Nov 03 01:16:40.192535 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:40.192437 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e5cb5e3-4c37-4ae7-9c6b-ef145cb87eb3-workload-certs\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-zxs8d\" (UID: \"6e5cb5e3-4c37-4ae7-9c6b-ef145cb87eb3\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-zxs8d" Nov 03 01:16:40.192535 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:40.192464 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e5cb5e3-4c37-4ae7-9c6b-ef145cb87eb3-istio-data\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-zxs8d\" (UID: \"6e5cb5e3-4c37-4ae7-9c6b-ef145cb87eb3\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-zxs8d" Nov 03 01:16:40.192535 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:40.192494 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d56b90c8-05c3-4c59-a204-79eb70d4a172-istiod-ca-cert\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-wxr6x\" (UID: \"d56b90c8-05c3-4c59-a204-79eb70d4a172\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-wxr6x" Nov 03 01:16:40.244958 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:40.244922 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-v8jpj"] Nov 03 01:16:40.250653 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:40.250622 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-v8jpj"] Nov 03 01:16:40.262931 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:40.262901 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-821f5a/payload-processing-external-model-e2e-extmod-a-821f5a-6f64zw8pg" Nov 03 01:16:40.295309 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:40.294376 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e5cb5e3-4c37-4ae7-9c6b-ef145cb87eb3-istio-envoy\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-zxs8d\" (UID: \"6e5cb5e3-4c37-4ae7-9c6b-ef145cb87eb3\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-zxs8d" Nov 03 01:16:40.295309 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:40.294423 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e5cb5e3-4c37-4ae7-9c6b-ef145cb87eb3-istiod-ca-cert\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-zxs8d\" (UID: \"6e5cb5e3-4c37-4ae7-9c6b-ef145cb87eb3\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-zxs8d" Nov 03 01:16:40.295309 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:40.294473 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d56b90c8-05c3-4c59-a204-79eb70d4a172-istio-podinfo\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-wxr6x\" (UID: \"d56b90c8-05c3-4c59-a204-79eb70d4a172\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-wxr6x" Nov 03 01:16:40.295309 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:40.294491 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xd59p\" (UniqueName: \"kubernetes.io/projected/6e5cb5e3-4c37-4ae7-9c6b-ef145cb87eb3-kube-api-access-xd59p\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-zxs8d\" (UID: \"6e5cb5e3-4c37-4ae7-9c6b-ef145cb87eb3\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-zxs8d" Nov 03 01:16:40.295309 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:40.294510 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d56b90c8-05c3-4c59-a204-79eb70d4a172-istio-token\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-wxr6x\" (UID: \"d56b90c8-05c3-4c59-a204-79eb70d4a172\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-wxr6x" Nov 03 01:16:40.295309 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:40.294533 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjnw8\" (UniqueName: \"kubernetes.io/projected/d56b90c8-05c3-4c59-a204-79eb70d4a172-kube-api-access-tjnw8\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-wxr6x\" (UID: \"d56b90c8-05c3-4c59-a204-79eb70d4a172\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-wxr6x" Nov 03 01:16:40.295309 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:40.294567 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d56b90c8-05c3-4c59-a204-79eb70d4a172-istio-data\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-wxr6x\" (UID: \"d56b90c8-05c3-4c59-a204-79eb70d4a172\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-wxr6x" Nov 03 01:16:40.295309 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:40.294584 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e5cb5e3-4c37-4ae7-9c6b-ef145cb87eb3-workload-certs\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-zxs8d\" (UID: \"6e5cb5e3-4c37-4ae7-9c6b-ef145cb87eb3\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-zxs8d" Nov 03 01:16:40.295309 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:40.294603 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e5cb5e3-4c37-4ae7-9c6b-ef145cb87eb3-istio-data\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-zxs8d\" (UID: \"6e5cb5e3-4c37-4ae7-9c6b-ef145cb87eb3\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-zxs8d" Nov 03 01:16:40.295309 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:40.294626 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d56b90c8-05c3-4c59-a204-79eb70d4a172-istiod-ca-cert\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-wxr6x\" (UID: \"d56b90c8-05c3-4c59-a204-79eb70d4a172\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-wxr6x" Nov 03 01:16:40.295309 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:40.294697 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d56b90c8-05c3-4c59-a204-79eb70d4a172-istio-envoy\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-wxr6x\" (UID: \"d56b90c8-05c3-4c59-a204-79eb70d4a172\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-wxr6x" Nov 03 01:16:40.295309 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:40.294745 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d56b90c8-05c3-4c59-a204-79eb70d4a172-workload-socket\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-wxr6x\" (UID: \"d56b90c8-05c3-4c59-a204-79eb70d4a172\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-wxr6x" Nov 03 01:16:40.295309 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:40.294809 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e5cb5e3-4c37-4ae7-9c6b-ef145cb87eb3-credential-socket\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-zxs8d\" (UID: \"6e5cb5e3-4c37-4ae7-9c6b-ef145cb87eb3\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-zxs8d" Nov 03 01:16:40.295309 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:40.294870 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d56b90c8-05c3-4c59-a204-79eb70d4a172-credential-socket\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-wxr6x\" (UID: \"d56b90c8-05c3-4c59-a204-79eb70d4a172\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-wxr6x" Nov 03 01:16:40.295309 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:40.294950 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e5cb5e3-4c37-4ae7-9c6b-ef145cb87eb3-istio-podinfo\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-zxs8d\" (UID: \"6e5cb5e3-4c37-4ae7-9c6b-ef145cb87eb3\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-zxs8d" Nov 03 01:16:40.295309 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:40.294984 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e5cb5e3-4c37-4ae7-9c6b-ef145cb87eb3-istio-token\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-zxs8d\" (UID: \"6e5cb5e3-4c37-4ae7-9c6b-ef145cb87eb3\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-zxs8d" Nov 03 01:16:40.295309 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:40.295118 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d56b90c8-05c3-4c59-a204-79eb70d4a172-workload-certs\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-wxr6x\" (UID: \"d56b90c8-05c3-4c59-a204-79eb70d4a172\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-wxr6x" Nov 03 01:16:40.295309 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:40.295132 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d56b90c8-05c3-4c59-a204-79eb70d4a172-istio-data\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-wxr6x\" (UID: \"d56b90c8-05c3-4c59-a204-79eb70d4a172\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-wxr6x" Nov 03 01:16:40.295309 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:40.295141 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e5cb5e3-4c37-4ae7-9c6b-ef145cb87eb3-istiod-ca-cert\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-zxs8d\" (UID: \"6e5cb5e3-4c37-4ae7-9c6b-ef145cb87eb3\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-zxs8d" Nov 03 01:16:40.295309 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:40.295257 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e5cb5e3-4c37-4ae7-9c6b-ef145cb87eb3-workload-socket\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-zxs8d\" (UID: \"6e5cb5e3-4c37-4ae7-9c6b-ef145cb87eb3\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-zxs8d" Nov 03 01:16:40.296643 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:40.295401 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d56b90c8-05c3-4c59-a204-79eb70d4a172-workload-certs\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-wxr6x\" (UID: \"d56b90c8-05c3-4c59-a204-79eb70d4a172\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-wxr6x" Nov 03 01:16:40.296643 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:40.295516 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e5cb5e3-4c37-4ae7-9c6b-ef145cb87eb3-istio-data\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-zxs8d\" (UID: \"6e5cb5e3-4c37-4ae7-9c6b-ef145cb87eb3\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-zxs8d" Nov 03 01:16:40.296643 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:40.296101 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d56b90c8-05c3-4c59-a204-79eb70d4a172-workload-socket\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-wxr6x\" (UID: \"d56b90c8-05c3-4c59-a204-79eb70d4a172\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-wxr6x" Nov 03 01:16:40.296643 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:40.296117 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d56b90c8-05c3-4c59-a204-79eb70d4a172-istiod-ca-cert\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-wxr6x\" (UID: \"d56b90c8-05c3-4c59-a204-79eb70d4a172\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-wxr6x" Nov 03 01:16:40.296643 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:40.296319 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d56b90c8-05c3-4c59-a204-79eb70d4a172-credential-socket\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-wxr6x\" (UID: \"d56b90c8-05c3-4c59-a204-79eb70d4a172\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-wxr6x" Nov 03 01:16:40.296925 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:40.296778 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e5cb5e3-4c37-4ae7-9c6b-ef145cb87eb3-credential-socket\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-zxs8d\" (UID: \"6e5cb5e3-4c37-4ae7-9c6b-ef145cb87eb3\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-zxs8d" Nov 03 01:16:40.296925 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:40.296787 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e5cb5e3-4c37-4ae7-9c6b-ef145cb87eb3-workload-certs\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-zxs8d\" (UID: \"6e5cb5e3-4c37-4ae7-9c6b-ef145cb87eb3\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-zxs8d" Nov 03 01:16:40.297130 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:40.297110 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e5cb5e3-4c37-4ae7-9c6b-ef145cb87eb3-istio-envoy\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-zxs8d\" (UID: \"6e5cb5e3-4c37-4ae7-9c6b-ef145cb87eb3\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-zxs8d" Nov 03 01:16:40.297497 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:40.297481 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e5cb5e3-4c37-4ae7-9c6b-ef145cb87eb3-workload-socket\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-zxs8d\" (UID: \"6e5cb5e3-4c37-4ae7-9c6b-ef145cb87eb3\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-zxs8d" Nov 03 01:16:40.298057 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:40.298036 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d56b90c8-05c3-4c59-a204-79eb70d4a172-istio-envoy\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-wxr6x\" (UID: \"d56b90c8-05c3-4c59-a204-79eb70d4a172\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-wxr6x" Nov 03 01:16:40.299048 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:40.299026 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e5cb5e3-4c37-4ae7-9c6b-ef145cb87eb3-istio-podinfo\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-zxs8d\" (UID: \"6e5cb5e3-4c37-4ae7-9c6b-ef145cb87eb3\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-zxs8d" Nov 03 01:16:40.299483 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:40.299460 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d56b90c8-05c3-4c59-a204-79eb70d4a172-istio-podinfo\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-wxr6x\" (UID: \"d56b90c8-05c3-4c59-a204-79eb70d4a172\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-wxr6x" Nov 03 01:16:40.304262 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:40.304232 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xd59p\" (UniqueName: \"kubernetes.io/projected/6e5cb5e3-4c37-4ae7-9c6b-ef145cb87eb3-kube-api-access-xd59p\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-zxs8d\" (UID: \"6e5cb5e3-4c37-4ae7-9c6b-ef145cb87eb3\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-zxs8d" Nov 03 01:16:40.304735 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:40.304710 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjnw8\" (UniqueName: \"kubernetes.io/projected/d56b90c8-05c3-4c59-a204-79eb70d4a172-kube-api-access-tjnw8\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-wxr6x\" (UID: \"d56b90c8-05c3-4c59-a204-79eb70d4a172\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-wxr6x" Nov 03 01:16:40.304839 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:40.304710 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d56b90c8-05c3-4c59-a204-79eb70d4a172-istio-token\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-wxr6x\" (UID: \"d56b90c8-05c3-4c59-a204-79eb70d4a172\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-wxr6x" Nov 03 01:16:40.304917 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:40.304860 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e5cb5e3-4c37-4ae7-9c6b-ef145cb87eb3-istio-token\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-zxs8d\" (UID: \"6e5cb5e3-4c37-4ae7-9c6b-ef145cb87eb3\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-zxs8d" Nov 03 01:16:40.334628 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:40.332975 4421 patch_prober.go:29] interesting pod/e2e-extmod-a-821f5a-openshift-default-7bfc657d9-bv95q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" start-of-body= Nov 03 01:16:40.334628 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:40.333042 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-821f5a-openshift-default-7bfc657d9-bv95q" podUID="ff2ff304-350a-4e21-8415-1c28dcb90a7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" Nov 03 01:16:40.481675 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:40.480599 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-zxs8d" Nov 03 01:16:40.503125 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:40.495065 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-wxr6x" Nov 03 01:16:40.915372 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:40.915225 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-821f5a/payload-processing-external-model-e2e-extmod-a-821f5a-6f64zw8pg"] Nov 03 01:16:41.423299 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:41.417543 4421 patch_prober.go:29] interesting pod/e2e-extmod-a-821f5a-openshift-default-7bfc657d9-bv95q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" start-of-body= Nov 03 01:16:41.423299 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:41.417601 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-821f5a-openshift-default-7bfc657d9-bv95q" podUID="ff2ff304-350a-4e21-8415-1c28dcb90a7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" Nov 03 01:16:41.470094 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:41.470061 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-zxs8d"] Nov 03 01:16:41.474812 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:41.474779 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:16:41.474993 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:41.474976 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:16:41.475113 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:41.475100 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:16:41.486250 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:41.485770 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-c5x7c" event={"ID":"652b215e-11c7-4343-86ec-bf27918c1c35","Type":"ContainerDied","Data":"e7a1205c7936651f3f6a8ebf6a2717e903be8147833542e7c096d4e91d86e3ba"} Nov 03 01:16:41.486250 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:41.485837 4421 scope.go:122] "RemoveContainer" containerID="6a3743d0f74bb4c4289533a869fa36f0b7a71f52e3823905476b32e05870bf03" Nov 03 01:16:41.486250 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:41.486094 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-c5x7c" Nov 03 01:16:41.499768 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:41.495998 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-821f5a/payload-processing-external-model-e2e-extmod-a-821f5a-6f64zw8pg" event={"ID":"ad8efe73-1c3e-4086-b626-212210ac2d0d","Type":"ContainerStarted","Data":"6ec60dfb51ef1e1336b94ac806f7999cf9c090e9f45c5170d568c39737c482c7"} Nov 03 01:16:41.499768 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:41.498487 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-78zwt" event={"ID":"8f5666ca-5e70-47c9-84d1-20eb2475d196","Type":"ContainerDied","Data":"0f872a01dfc55ba8345f52c52b0606554037e4fb771a4e18dd89bf6a4033e5bf"} Nov 03 01:16:41.499768 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:41.498686 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-78zwt" Nov 03 01:16:41.567979 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:41.567897 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-wxr6x"] Nov 03 01:16:41.571784 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:41.571626 4421 scope.go:122] "RemoveContainer" containerID="ceb17238f389cc4cbaf25cb81bb7734c5cd5e75cdc281ee51d8e6eced1a0af5e" Nov 03 01:16:41.575972 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:16:41.575941 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd56b90c8_05c3_4c59_a204_79eb70d4a172.slice/crio-6747158065bb2d75cfea8f16ff5d4a2a88df4c6717714b41e088255be4d98f2e WatchSource:0}: Error finding container 6747158065bb2d75cfea8f16ff5d4a2a88df4c6717714b41e088255be4d98f2e: Status 404 returned error can't find the container with id 6747158065bb2d75cfea8f16ff5d4a2a88df4c6717714b41e088255be4d98f2e Nov 03 01:16:41.579366 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:41.579232 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:16:41.579366 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:41.579329 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:16:41.579547 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:41.579405 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:16:41.580961 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:41.579958 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-c5x7c"] Nov 03 01:16:41.589140 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:41.589098 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-c5x7c"] Nov 03 01:16:41.597608 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:41.597544 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-78zwt"] Nov 03 01:16:41.603968 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:41.603940 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-78zwt"] Nov 03 01:16:41.987426 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:41.987331 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="652b215e-11c7-4343-86ec-bf27918c1c35" path="/var/lib/kubelet/pods/652b215e-11c7-4343-86ec-bf27918c1c35/volumes" Nov 03 01:16:41.988117 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:41.988099 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f5666ca-5e70-47c9-84d1-20eb2475d196" path="/var/lib/kubelet/pods/8f5666ca-5e70-47c9-84d1-20eb2475d196/volumes" Nov 03 01:16:41.988778 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:41.988762 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ca37a4e-e37b-453c-a937-a35fedc2af8c" path="/var/lib/kubelet/pods/9ca37a4e-e37b-453c-a937-a35fedc2af8c/volumes" Nov 03 01:16:42.337995 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:42.337129 4421 patch_prober.go:29] interesting pod/e2e-extmod-a-821f5a-openshift-default-7bfc657d9-bv95q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" start-of-body= Nov 03 01:16:42.337995 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:42.337188 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-821f5a-openshift-default-7bfc657d9-bv95q" podUID="ff2ff304-350a-4e21-8415-1c28dcb90a7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" Nov 03 01:16:42.585998 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:42.585958 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-wxr6x" event={"ID":"d56b90c8-05c3-4c59-a204-79eb70d4a172","Type":"ContainerStarted","Data":"456097de640039b54e16601e495e6a3f66020f72fa020b0ddbba108503f0990f"} Nov 03 01:16:42.586695 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:42.586650 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-wxr6x" event={"ID":"d56b90c8-05c3-4c59-a204-79eb70d4a172","Type":"ContainerStarted","Data":"6747158065bb2d75cfea8f16ff5d4a2a88df4c6717714b41e088255be4d98f2e"} Nov 03 01:16:42.589866 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:42.589793 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-821f5a/payload-processing-external-model-e2e-extmod-a-821f5a-6f64zw8pg" event={"ID":"ad8efe73-1c3e-4086-b626-212210ac2d0d","Type":"ContainerStarted","Data":"095437fd34efd6562aabc9a4c25a05dca49ea18ece22e6fef4669d419ddec627"} Nov 03 01:16:42.590332 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:42.590304 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-821f5a/payload-processing-external-model-e2e-extmod-a-821f5a-6f64zw8pg" Nov 03 01:16:42.592562 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:42.592536 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-zxs8d" event={"ID":"6e5cb5e3-4c37-4ae7-9c6b-ef145cb87eb3","Type":"ContainerStarted","Data":"00666c657a79b88b3f1f97e09cdcc20c75e999c4caa98f003675fa116d6d4a56"} Nov 03 01:16:42.592651 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:42.592580 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-zxs8d" event={"ID":"6e5cb5e3-4c37-4ae7-9c6b-ef145cb87eb3","Type":"ContainerStarted","Data":"a5d285de4bbcecc1e22a9e8d80eb805514374c126a9e96896163e4ea55db8f3a"} Nov 03 01:16:42.661321 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:42.660373 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-wxr6x" podStartSLOduration=225.660350243 podStartE2EDuration="3m45.660350243s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:12:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:16:42.621807637 +0000 UTC m=+2144.792810735" watchObservedRunningTime="2026-11-03 01:16:42.660350243 +0000 UTC m=+2144.831353340" Nov 03 01:16:42.744365 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:42.741878 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-821f5a/payload-processing-external-model-e2e-extmod-a-821f5a-6f64zw8pg" podStartSLOduration=17.741856675 podStartE2EDuration="17.741856675s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:16:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:16:42.652762898 +0000 UTC m=+2144.823766010" watchObservedRunningTime="2026-11-03 01:16:42.741856675 +0000 UTC m=+2144.912859772" Nov 03 01:16:42.804838 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:42.804770 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-zxs8d" podStartSLOduration=225.804746497 podStartE2EDuration="3m45.804746497s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:12:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:16:42.691785781 +0000 UTC m=+2144.862788878" watchObservedRunningTime="2026-11-03 01:16:42.804746497 +0000 UTC m=+2144.975749594" Nov 03 01:16:43.322299 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:43.318973 4421 patch_prober.go:29] interesting pod/e2e-extmod-a-821f5a-openshift-default-7bfc657d9-bv95q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" start-of-body= Nov 03 01:16:43.322299 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:43.319044 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-821f5a-openshift-default-7bfc657d9-bv95q" podUID="ff2ff304-350a-4e21-8415-1c28dcb90a7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" Nov 03 01:16:43.501130 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:43.501092 4421 patch_prober.go:29] interesting pod/e2e-extmod-a-d25759-openshift-default-d47fd9785-wxr6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.24:15021/healthz/ready\": dial tcp 10.134.1.24:15021: connect: connection refused" start-of-body= Nov 03 01:16:43.523134 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:43.523098 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-wxr6x" podUID="d56b90c8-05c3-4c59-a204-79eb70d4a172" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.24:15021/healthz/ready\": dial tcp 10.134.1.24:15021: connect: connection refused" Nov 03 01:16:43.523677 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:43.507329 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-wxr6x" Nov 03 01:16:43.523908 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:43.523894 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-zxs8d" Nov 03 01:16:43.524038 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:43.523012 4421 patch_prober.go:29] interesting pod/e2e-extmod-a-d25759-openshift-default-d47fd9785-zxs8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.25:15021/healthz/ready\": dial tcp 10.134.1.25:15021: connect: connection refused" start-of-body= Nov 03 01:16:43.524173 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:43.524152 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-zxs8d" podUID="6e5cb5e3-4c37-4ae7-9c6b-ef145cb87eb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.25:15021/healthz/ready\": dial tcp 10.134.1.25:15021: connect: connection refused" Nov 03 01:16:44.370295 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:44.369643 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-821f5a-openshift-default-7bfc657d9-bv95q" Nov 03 01:16:44.383652 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:44.383618 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-821f5a-openshift-default-7bfc657d9-bv95q" Nov 03 01:16:44.496684 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:44.496138 4421 patch_prober.go:29] interesting pod/e2e-extmod-a-d25759-openshift-default-d47fd9785-zxs8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.25:15021/healthz/ready\": dial tcp 10.134.1.25:15021: connect: connection refused" start-of-body= Nov 03 01:16:44.496684 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:44.496197 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-zxs8d" podUID="6e5cb5e3-4c37-4ae7-9c6b-ef145cb87eb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.25:15021/healthz/ready\": dial tcp 10.134.1.25:15021: connect: connection refused" Nov 03 01:16:44.496684 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:44.496604 4421 patch_prober.go:29] interesting pod/e2e-extmod-a-d25759-openshift-default-d47fd9785-wxr6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.24:15021/healthz/ready\": dial tcp 10.134.1.24:15021: connect: connection refused" start-of-body= Nov 03 01:16:44.496684 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:44.496634 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-wxr6x" podUID="d56b90c8-05c3-4c59-a204-79eb70d4a172" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.24:15021/healthz/ready\": dial tcp 10.134.1.24:15021: connect: connection refused" Nov 03 01:16:44.691545 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:16:44.691458 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 01:16:45.494996 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:45.494950 4421 patch_prober.go:29] interesting pod/e2e-extmod-a-d25759-openshift-default-d47fd9785-zxs8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.25:15021/healthz/ready\": dial tcp 10.134.1.25:15021: connect: connection refused" start-of-body= Nov 03 01:16:45.495435 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:45.495006 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-zxs8d" podUID="6e5cb5e3-4c37-4ae7-9c6b-ef145cb87eb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.25:15021/healthz/ready\": dial tcp 10.134.1.25:15021: connect: connection refused" Nov 03 01:16:45.496449 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:45.496417 4421 patch_prober.go:29] interesting pod/e2e-extmod-a-d25759-openshift-default-d47fd9785-wxr6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.24:15021/healthz/ready\": dial tcp 10.134.1.24:15021: connect: connection refused" start-of-body= Nov 03 01:16:45.496521 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:45.496463 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-wxr6x" podUID="d56b90c8-05c3-4c59-a204-79eb70d4a172" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.24:15021/healthz/ready\": dial tcp 10.134.1.24:15021: connect: connection refused" Nov 03 01:16:46.073323 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:46.073289 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-2dshc"] Nov 03 01:16:46.073527 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:46.073498 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-2dshc" podUID="eeca2c25-c047-4693-8c4d-def02cfabfa8" containerName="istio-proxy" containerID="cri-o://8f392df94677232b2a092579d3a645a15c11416e9ffd51cb37fdaa431a76347e" gracePeriod=30 Nov 03 01:16:46.494927 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:46.494823 4421 patch_prober.go:29] interesting pod/e2e-extmod-a-d25759-openshift-default-d47fd9785-zxs8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.25:15021/healthz/ready\": dial tcp 10.134.1.25:15021: connect: connection refused" start-of-body= Nov 03 01:16:46.494927 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:46.494888 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-zxs8d" podUID="6e5cb5e3-4c37-4ae7-9c6b-ef145cb87eb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.25:15021/healthz/ready\": dial tcp 10.134.1.25:15021: connect: connection refused" Nov 03 01:16:46.495494 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:46.495395 4421 patch_prober.go:29] interesting pod/e2e-extmod-a-d25759-openshift-default-d47fd9785-wxr6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.24:15021/healthz/ready\": dial tcp 10.134.1.24:15021: connect: connection refused" start-of-body= Nov 03 01:16:46.495494 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:46.495434 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-wxr6x" podUID="d56b90c8-05c3-4c59-a204-79eb70d4a172" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.24:15021/healthz/ready\": dial tcp 10.134.1.24:15021: connect: connection refused" Nov 03 01:16:47.503315 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:47.500452 4421 patch_prober.go:29] interesting pod/e2e-extmod-a-d25759-openshift-default-d47fd9785-zxs8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.25:15021/healthz/ready\": dial tcp 10.134.1.25:15021: connect: connection refused" start-of-body= Nov 03 01:16:47.503315 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:47.500518 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-zxs8d" podUID="6e5cb5e3-4c37-4ae7-9c6b-ef145cb87eb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.25:15021/healthz/ready\": dial tcp 10.134.1.25:15021: connect: connection refused" Nov 03 01:16:47.503315 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:47.500920 4421 patch_prober.go:29] interesting pod/e2e-extmod-a-d25759-openshift-default-d47fd9785-wxr6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.24:15021/healthz/ready\": dial tcp 10.134.1.24:15021: connect: connection refused" start-of-body= Nov 03 01:16:47.503315 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:47.500955 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-wxr6x" podUID="d56b90c8-05c3-4c59-a204-79eb70d4a172" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.24:15021/healthz/ready\": dial tcp 10.134.1.24:15021: connect: connection refused" Nov 03 01:16:48.495864 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:48.495821 4421 patch_prober.go:29] interesting pod/e2e-extmod-a-d25759-openshift-default-d47fd9785-wxr6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.24:15021/healthz/ready\": dial tcp 10.134.1.24:15021: connect: connection refused" start-of-body= Nov 03 01:16:48.496160 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:48.496132 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-wxr6x" podUID="d56b90c8-05c3-4c59-a204-79eb70d4a172" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.24:15021/healthz/ready\": dial tcp 10.134.1.24:15021: connect: connection refused" Nov 03 01:16:48.496320 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:48.495956 4421 patch_prober.go:29] interesting pod/e2e-extmod-a-d25759-openshift-default-d47fd9785-zxs8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.25:15021/healthz/ready\": dial tcp 10.134.1.25:15021: connect: connection refused" start-of-body= Nov 03 01:16:48.496446 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:48.496428 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-zxs8d" podUID="6e5cb5e3-4c37-4ae7-9c6b-ef145cb87eb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.25:15021/healthz/ready\": dial tcp 10.134.1.25:15021: connect: connection refused" Nov 03 01:16:49.501137 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:49.501104 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-zxs8d" Nov 03 01:16:49.501933 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:49.501913 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-zxs8d" Nov 03 01:16:49.502118 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:49.502103 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-wxr6x" Nov 03 01:16:49.502502 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:49.502474 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-wxr6x" Nov 03 01:16:49.502713 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:49.502695 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-zxs8d" Nov 03 01:16:49.503181 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:49.503164 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-wxr6x" Nov 03 01:16:51.390621 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:51.390587 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-2dshc" Nov 03 01:16:51.479999 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:51.479955 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-5lkrv"] Nov 03 01:16:51.482912 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:51.481314 4421 cpu_manager.go:414] "containerMap: removing container" podUID="eeca2c25-c047-4693-8c4d-def02cfabfa8" containerName="istio-proxy" Nov 03 01:16:51.482912 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:51.481340 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eeca2c25-c047-4693-8c4d-def02cfabfa8" containerName="istio-proxy" Nov 03 01:16:51.482912 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:51.481591 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="eeca2c25-c047-4693-8c4d-def02cfabfa8" containerName="istio-proxy" Nov 03 01:16:51.504084 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:51.504048 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-5lkrv" Nov 03 01:16:51.510996 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:51.510864 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-5lkrv"] Nov 03 01:16:51.542158 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:51.536786 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eeca2c25-c047-4693-8c4d-def02cfabfa8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eeca2c25-c047-4693-8c4d-def02cfabfa8-credential-socket\") pod \"eeca2c25-c047-4693-8c4d-def02cfabfa8\" (UID: \"eeca2c25-c047-4693-8c4d-def02cfabfa8\") " Nov 03 01:16:51.542158 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:51.536872 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eeca2c25-c047-4693-8c4d-def02cfabfa8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eeca2c25-c047-4693-8c4d-def02cfabfa8-istio-data\") pod \"eeca2c25-c047-4693-8c4d-def02cfabfa8\" (UID: \"eeca2c25-c047-4693-8c4d-def02cfabfa8\") " Nov 03 01:16:51.542158 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:51.536925 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eeca2c25-c047-4693-8c4d-def02cfabfa8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eeca2c25-c047-4693-8c4d-def02cfabfa8-istiod-ca-cert\") pod \"eeca2c25-c047-4693-8c4d-def02cfabfa8\" (UID: \"eeca2c25-c047-4693-8c4d-def02cfabfa8\") " Nov 03 01:16:51.542158 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:51.536958 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eeca2c25-c047-4693-8c4d-def02cfabfa8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eeca2c25-c047-4693-8c4d-def02cfabfa8-workload-certs\") pod \"eeca2c25-c047-4693-8c4d-def02cfabfa8\" (UID: \"eeca2c25-c047-4693-8c4d-def02cfabfa8\") " Nov 03 01:16:51.542158 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:51.537021 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eeca2c25-c047-4693-8c4d-def02cfabfa8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eeca2c25-c047-4693-8c4d-def02cfabfa8-istio-envoy\") pod \"eeca2c25-c047-4693-8c4d-def02cfabfa8\" (UID: \"eeca2c25-c047-4693-8c4d-def02cfabfa8\") " Nov 03 01:16:51.542158 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:51.537347 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eeca2c25-c047-4693-8c4d-def02cfabfa8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eeca2c25-c047-4693-8c4d-def02cfabfa8-workload-socket\") pod \"eeca2c25-c047-4693-8c4d-def02cfabfa8\" (UID: \"eeca2c25-c047-4693-8c4d-def02cfabfa8\") " Nov 03 01:16:51.542158 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:51.537438 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/eeca2c25-c047-4693-8c4d-def02cfabfa8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eeca2c25-c047-4693-8c4d-def02cfabfa8-istio-podinfo\") pod \"eeca2c25-c047-4693-8c4d-def02cfabfa8\" (UID: \"eeca2c25-c047-4693-8c4d-def02cfabfa8\") " Nov 03 01:16:51.542158 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:51.537490 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eeca2c25-c047-4693-8c4d-def02cfabfa8-kube-api-access-4rzn5\" (UniqueName: \"kubernetes.io/projected/eeca2c25-c047-4693-8c4d-def02cfabfa8-kube-api-access-4rzn5\") pod \"eeca2c25-c047-4693-8c4d-def02cfabfa8\" (UID: \"eeca2c25-c047-4693-8c4d-def02cfabfa8\") " Nov 03 01:16:51.542158 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:51.537618 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eeca2c25-c047-4693-8c4d-def02cfabfa8-istio-token\" (UniqueName: \"kubernetes.io/projected/eeca2c25-c047-4693-8c4d-def02cfabfa8-istio-token\") pod \"eeca2c25-c047-4693-8c4d-def02cfabfa8\" (UID: \"eeca2c25-c047-4693-8c4d-def02cfabfa8\") " Nov 03 01:16:51.544363 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:51.543935 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eeca2c25-c047-4693-8c4d-def02cfabfa8-istio-data" pod "eeca2c25-c047-4693-8c4d-def02cfabfa8" (UID: "eeca2c25-c047-4693-8c4d-def02cfabfa8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:16:51.544896 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:51.544872 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eeca2c25-c047-4693-8c4d-def02cfabfa8-workload-socket" pod "eeca2c25-c047-4693-8c4d-def02cfabfa8" (UID: "eeca2c25-c047-4693-8c4d-def02cfabfa8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:16:51.547257 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:51.546973 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eeca2c25-c047-4693-8c4d-def02cfabfa8-istiod-ca-cert" pod "eeca2c25-c047-4693-8c4d-def02cfabfa8" (UID: "eeca2c25-c047-4693-8c4d-def02cfabfa8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:16:51.551100 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:51.549313 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eeca2c25-c047-4693-8c4d-def02cfabfa8-credential-socket" pod "eeca2c25-c047-4693-8c4d-def02cfabfa8" (UID: "eeca2c25-c047-4693-8c4d-def02cfabfa8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:16:51.551100 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:51.549376 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eeca2c25-c047-4693-8c4d-def02cfabfa8-workload-certs" pod "eeca2c25-c047-4693-8c4d-def02cfabfa8" (UID: "eeca2c25-c047-4693-8c4d-def02cfabfa8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:16:51.551100 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:51.549739 4421 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eeca2c25-c047-4693-8c4d-def02cfabfa8-credential-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:16:51.551100 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:51.549765 4421 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eeca2c25-c047-4693-8c4d-def02cfabfa8-istio-data\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:16:51.551100 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:51.549782 4421 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eeca2c25-c047-4693-8c4d-def02cfabfa8-istiod-ca-cert\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:16:51.551100 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:51.549797 4421 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eeca2c25-c047-4693-8c4d-def02cfabfa8-workload-certs\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:16:51.551100 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:51.549814 4421 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eeca2c25-c047-4693-8c4d-def02cfabfa8-workload-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:16:51.560619 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:51.556315 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eeca2c25-c047-4693-8c4d-def02cfabfa8-kube-api-access-4rzn5" pod "eeca2c25-c047-4693-8c4d-def02cfabfa8" (UID: "eeca2c25-c047-4693-8c4d-def02cfabfa8"). InnerVolumeSpecName "kube-api-access-4rzn5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:16:51.560619 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:51.556507 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eeca2c25-c047-4693-8c4d-def02cfabfa8-istio-envoy" pod "eeca2c25-c047-4693-8c4d-def02cfabfa8" (UID: "eeca2c25-c047-4693-8c4d-def02cfabfa8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:16:51.570316 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:51.570235 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eeca2c25-c047-4693-8c4d-def02cfabfa8-istio-podinfo" pod "eeca2c25-c047-4693-8c4d-def02cfabfa8" (UID: "eeca2c25-c047-4693-8c4d-def02cfabfa8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:16:51.591386 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:51.587019 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eeca2c25-c047-4693-8c4d-def02cfabfa8-istio-token" pod "eeca2c25-c047-4693-8c4d-def02cfabfa8" (UID: "eeca2c25-c047-4693-8c4d-def02cfabfa8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:16:51.651232 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:51.651189 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09633e29-ea24-4847-b1c5-81d5843899fb-workload-socket\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-5lkrv\" (UID: \"09633e29-ea24-4847-b1c5-81d5843899fb\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-5lkrv" Nov 03 01:16:51.651232 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:51.651233 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09633e29-ea24-4847-b1c5-81d5843899fb-istio-token\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-5lkrv\" (UID: \"09633e29-ea24-4847-b1c5-81d5843899fb\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-5lkrv" Nov 03 01:16:51.651541 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:51.651399 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09633e29-ea24-4847-b1c5-81d5843899fb-istio-data\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-5lkrv\" (UID: \"09633e29-ea24-4847-b1c5-81d5843899fb\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-5lkrv" Nov 03 01:16:51.651541 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:51.651493 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mc9x2\" (UniqueName: \"kubernetes.io/projected/09633e29-ea24-4847-b1c5-81d5843899fb-kube-api-access-mc9x2\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-5lkrv\" (UID: \"09633e29-ea24-4847-b1c5-81d5843899fb\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-5lkrv" Nov 03 01:16:51.651764 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:51.651746 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09633e29-ea24-4847-b1c5-81d5843899fb-credential-socket\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-5lkrv\" (UID: \"09633e29-ea24-4847-b1c5-81d5843899fb\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-5lkrv" Nov 03 01:16:51.651893 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:51.651875 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09633e29-ea24-4847-b1c5-81d5843899fb-istiod-ca-cert\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-5lkrv\" (UID: \"09633e29-ea24-4847-b1c5-81d5843899fb\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-5lkrv" Nov 03 01:16:51.651987 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:51.651968 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09633e29-ea24-4847-b1c5-81d5843899fb-workload-certs\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-5lkrv\" (UID: \"09633e29-ea24-4847-b1c5-81d5843899fb\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-5lkrv" Nov 03 01:16:51.652039 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:51.652007 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09633e29-ea24-4847-b1c5-81d5843899fb-istio-envoy\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-5lkrv\" (UID: \"09633e29-ea24-4847-b1c5-81d5843899fb\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-5lkrv" Nov 03 01:16:51.652080 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:51.652056 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09633e29-ea24-4847-b1c5-81d5843899fb-istio-podinfo\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-5lkrv\" (UID: \"09633e29-ea24-4847-b1c5-81d5843899fb\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-5lkrv" Nov 03 01:16:51.652222 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:51.652210 4421 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eeca2c25-c047-4693-8c4d-def02cfabfa8-istio-envoy\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:16:51.652262 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:51.652227 4421 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eeca2c25-c047-4693-8c4d-def02cfabfa8-istio-podinfo\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:16:51.652262 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:51.652238 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rzn5\" (UniqueName: \"kubernetes.io/projected/eeca2c25-c047-4693-8c4d-def02cfabfa8-kube-api-access-4rzn5\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:16:51.652262 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:51.652250 4421 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eeca2c25-c047-4693-8c4d-def02cfabfa8-istio-token\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:16:51.754043 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:51.754008 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09633e29-ea24-4847-b1c5-81d5843899fb-workload-socket\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-5lkrv\" (UID: \"09633e29-ea24-4847-b1c5-81d5843899fb\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-5lkrv" Nov 03 01:16:51.754043 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:51.754044 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09633e29-ea24-4847-b1c5-81d5843899fb-istio-token\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-5lkrv\" (UID: \"09633e29-ea24-4847-b1c5-81d5843899fb\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-5lkrv" Nov 03 01:16:51.754372 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:51.754091 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09633e29-ea24-4847-b1c5-81d5843899fb-istio-data\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-5lkrv\" (UID: \"09633e29-ea24-4847-b1c5-81d5843899fb\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-5lkrv" Nov 03 01:16:51.754372 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:51.754136 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mc9x2\" (UniqueName: \"kubernetes.io/projected/09633e29-ea24-4847-b1c5-81d5843899fb-kube-api-access-mc9x2\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-5lkrv\" (UID: \"09633e29-ea24-4847-b1c5-81d5843899fb\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-5lkrv" Nov 03 01:16:51.754372 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:51.754320 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09633e29-ea24-4847-b1c5-81d5843899fb-credential-socket\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-5lkrv\" (UID: \"09633e29-ea24-4847-b1c5-81d5843899fb\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-5lkrv" Nov 03 01:16:51.754536 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:51.754450 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09633e29-ea24-4847-b1c5-81d5843899fb-istiod-ca-cert\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-5lkrv\" (UID: \"09633e29-ea24-4847-b1c5-81d5843899fb\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-5lkrv" Nov 03 01:16:51.754536 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:51.754524 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09633e29-ea24-4847-b1c5-81d5843899fb-workload-socket\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-5lkrv\" (UID: \"09633e29-ea24-4847-b1c5-81d5843899fb\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-5lkrv" Nov 03 01:16:51.754644 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:51.754525 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09633e29-ea24-4847-b1c5-81d5843899fb-workload-certs\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-5lkrv\" (UID: \"09633e29-ea24-4847-b1c5-81d5843899fb\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-5lkrv" Nov 03 01:16:51.754644 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:51.754595 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09633e29-ea24-4847-b1c5-81d5843899fb-istio-envoy\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-5lkrv\" (UID: \"09633e29-ea24-4847-b1c5-81d5843899fb\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-5lkrv" Nov 03 01:16:51.754644 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:51.754622 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09633e29-ea24-4847-b1c5-81d5843899fb-istio-podinfo\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-5lkrv\" (UID: \"09633e29-ea24-4847-b1c5-81d5843899fb\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-5lkrv" Nov 03 01:16:51.754809 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:51.754589 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09633e29-ea24-4847-b1c5-81d5843899fb-istio-data\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-5lkrv\" (UID: \"09633e29-ea24-4847-b1c5-81d5843899fb\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-5lkrv" Nov 03 01:16:51.754809 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:51.754685 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09633e29-ea24-4847-b1c5-81d5843899fb-credential-socket\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-5lkrv\" (UID: \"09633e29-ea24-4847-b1c5-81d5843899fb\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-5lkrv" Nov 03 01:16:51.754809 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:51.754769 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09633e29-ea24-4847-b1c5-81d5843899fb-workload-certs\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-5lkrv\" (UID: \"09633e29-ea24-4847-b1c5-81d5843899fb\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-5lkrv" Nov 03 01:16:51.755208 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:51.755182 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09633e29-ea24-4847-b1c5-81d5843899fb-istiod-ca-cert\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-5lkrv\" (UID: \"09633e29-ea24-4847-b1c5-81d5843899fb\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-5lkrv" Nov 03 01:16:51.756875 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:51.756855 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09633e29-ea24-4847-b1c5-81d5843899fb-istio-envoy\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-5lkrv\" (UID: \"09633e29-ea24-4847-b1c5-81d5843899fb\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-5lkrv" Nov 03 01:16:51.757085 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:51.757067 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09633e29-ea24-4847-b1c5-81d5843899fb-istio-podinfo\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-5lkrv\" (UID: \"09633e29-ea24-4847-b1c5-81d5843899fb\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-5lkrv" Nov 03 01:16:51.761880 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:51.761856 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09633e29-ea24-4847-b1c5-81d5843899fb-istio-token\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-5lkrv\" (UID: \"09633e29-ea24-4847-b1c5-81d5843899fb\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-5lkrv" Nov 03 01:16:51.762140 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:51.762120 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mc9x2\" (UniqueName: \"kubernetes.io/projected/09633e29-ea24-4847-b1c5-81d5843899fb-kube-api-access-mc9x2\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-5lkrv\" (UID: \"09633e29-ea24-4847-b1c5-81d5843899fb\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-5lkrv" Nov 03 01:16:51.824552 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:51.824513 4421 generic.go:356] "Generic (PLEG): container finished" podID="eeca2c25-c047-4693-8c4d-def02cfabfa8" containerID="8f392df94677232b2a092579d3a645a15c11416e9ffd51cb37fdaa431a76347e" exitCode=0 Nov 03 01:16:51.824738 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:51.824567 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-2dshc" Nov 03 01:16:51.824738 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:51.824593 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-2dshc" event={"ID":"eeca2c25-c047-4693-8c4d-def02cfabfa8","Type":"ContainerDied","Data":"8f392df94677232b2a092579d3a645a15c11416e9ffd51cb37fdaa431a76347e"} Nov 03 01:16:51.824738 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:51.824655 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-2dshc" event={"ID":"eeca2c25-c047-4693-8c4d-def02cfabfa8","Type":"ContainerDied","Data":"3782690ef640fd4a6e96c80f18381371464d1deea166af781f233a6f8476a646"} Nov 03 01:16:51.824738 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:51.824688 4421 scope.go:122] "RemoveContainer" containerID="8f392df94677232b2a092579d3a645a15c11416e9ffd51cb37fdaa431a76347e" Nov 03 01:16:51.837370 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:51.837350 4421 scope.go:122] "RemoveContainer" containerID="8f392df94677232b2a092579d3a645a15c11416e9ffd51cb37fdaa431a76347e" Nov 03 01:16:51.837664 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:16:51.837641 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f392df94677232b2a092579d3a645a15c11416e9ffd51cb37fdaa431a76347e\": container with ID starting with 8f392df94677232b2a092579d3a645a15c11416e9ffd51cb37fdaa431a76347e not found: ID does not exist" containerID="8f392df94677232b2a092579d3a645a15c11416e9ffd51cb37fdaa431a76347e" Nov 03 01:16:51.837740 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:51.837676 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f392df94677232b2a092579d3a645a15c11416e9ffd51cb37fdaa431a76347e"} err="failed to get container status \"8f392df94677232b2a092579d3a645a15c11416e9ffd51cb37fdaa431a76347e\": rpc error: code = NotFound desc = could not find container \"8f392df94677232b2a092579d3a645a15c11416e9ffd51cb37fdaa431a76347e\": container with ID starting with 8f392df94677232b2a092579d3a645a15c11416e9ffd51cb37fdaa431a76347e not found: ID does not exist" Nov 03 01:16:51.853456 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:51.853377 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-2dshc"] Nov 03 01:16:51.859160 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:51.859134 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-2dshc"] Nov 03 01:16:51.887761 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:51.887721 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-5lkrv" Nov 03 01:16:51.986627 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:51.986595 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eeca2c25-c047-4693-8c4d-def02cfabfa8" path="/var/lib/kubelet/pods/eeca2c25-c047-4693-8c4d-def02cfabfa8/volumes" Nov 03 01:16:52.207490 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:52.207457 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-5lkrv"] Nov 03 01:16:52.208965 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:16:52.208924 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09633e29_ea24_4847_b1c5_81d5843899fb.slice/crio-37fc4672e6f48baeb6341947a95f77ec5de169452acd2380c5a44ccefcea59bb WatchSource:0}: Error finding container 37fc4672e6f48baeb6341947a95f77ec5de169452acd2380c5a44ccefcea59bb: Status 404 returned error can't find the container with id 37fc4672e6f48baeb6341947a95f77ec5de169452acd2380c5a44ccefcea59bb Nov 03 01:16:52.212543 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:52.212506 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:16:52.212641 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:52.212578 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:16:52.212641 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:52.212634 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:16:52.893173 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:52.893119 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-5lkrv" event={"ID":"09633e29-ea24-4847-b1c5-81d5843899fb","Type":"ContainerStarted","Data":"0c9caf837809fb9c889cfa01cddb4f580bf3f63edca9945464ab3bba06eeea11"} Nov 03 01:16:52.893721 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:52.893200 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-5lkrv" event={"ID":"09633e29-ea24-4847-b1c5-81d5843899fb","Type":"ContainerStarted","Data":"37fc4672e6f48baeb6341947a95f77ec5de169452acd2380c5a44ccefcea59bb"} Nov 03 01:16:52.922694 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:52.922642 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-5lkrv" podStartSLOduration=235.922627168 podStartE2EDuration="3m55.922627168s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:12:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:16:52.918174606 +0000 UTC m=+2155.089177706" watchObservedRunningTime="2026-11-03 01:16:52.922627168 +0000 UTC m=+2155.093630265" Nov 03 01:16:53.737369 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:53.736832 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-821f5a/payload-processing-external-model-e2e-extmod-a-821f5a-6f64zw8pg" Nov 03 01:16:53.888876 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:53.888841 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-5lkrv" Nov 03 01:16:53.890465 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:53.890433 4421 patch_prober.go:29] interesting pod/e2e-extmod-a-d25759-openshift-default-d47fd9785-5lkrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.26:15021/healthz/ready\": dial tcp 10.134.1.26:15021: connect: connection refused" start-of-body= Nov 03 01:16:53.890620 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:53.890506 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-5lkrv" podUID="09633e29-ea24-4847-b1c5-81d5843899fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.26:15021/healthz/ready\": dial tcp 10.134.1.26:15021: connect: connection refused" Nov 03 01:16:54.891298 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:54.891246 4421 patch_prober.go:29] interesting pod/e2e-extmod-a-d25759-openshift-default-d47fd9785-5lkrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.26:15021/healthz/ready\": dial tcp 10.134.1.26:15021: connect: connection refused" start-of-body= Nov 03 01:16:54.909335 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:54.891865 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-5lkrv" podUID="09633e29-ea24-4847-b1c5-81d5843899fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.26:15021/healthz/ready\": dial tcp 10.134.1.26:15021: connect: connection refused" Nov 03 01:16:55.888476 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:55.888441 4421 patch_prober.go:29] interesting pod/e2e-extmod-a-d25759-openshift-default-d47fd9785-5lkrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.26:15021/healthz/ready\": dial tcp 10.134.1.26:15021: connect: connection refused" start-of-body= Nov 03 01:16:55.888692 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:55.888500 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-5lkrv" podUID="09633e29-ea24-4847-b1c5-81d5843899fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.26:15021/healthz/ready\": dial tcp 10.134.1.26:15021: connect: connection refused" Nov 03 01:16:56.914418 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:56.898826 4421 patch_prober.go:29] interesting pod/e2e-extmod-a-d25759-openshift-default-d47fd9785-5lkrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.26:15021/healthz/ready\": dial tcp 10.134.1.26:15021: connect: connection refused" start-of-body= Nov 03 01:16:56.914418 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:56.898920 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-5lkrv" podUID="09633e29-ea24-4847-b1c5-81d5843899fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.26:15021/healthz/ready\": dial tcp 10.134.1.26:15021: connect: connection refused" Nov 03 01:16:57.888983 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:57.888946 4421 patch_prober.go:29] interesting pod/e2e-extmod-a-d25759-openshift-default-d47fd9785-5lkrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.26:15021/healthz/ready\": dial tcp 10.134.1.26:15021: connect: connection refused" start-of-body= Nov 03 01:16:57.889161 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:57.889005 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-5lkrv" podUID="09633e29-ea24-4847-b1c5-81d5843899fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.26:15021/healthz/ready\": dial tcp 10.134.1.26:15021: connect: connection refused" Nov 03 01:16:58.893618 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:58.889912 4421 patch_prober.go:29] interesting pod/e2e-extmod-a-d25759-openshift-default-d47fd9785-5lkrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.26:15021/healthz/ready\": dial tcp 10.134.1.26:15021: connect: connection refused" start-of-body= Nov 03 01:16:58.893618 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:58.890107 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-5lkrv" podUID="09633e29-ea24-4847-b1c5-81d5843899fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.26:15021/healthz/ready\": dial tcp 10.134.1.26:15021: connect: connection refused" Nov 03 01:16:59.985951 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:59.985920 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-5lkrv" Nov 03 01:16:59.986872 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:59.986844 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-5lkrv" Nov 03 01:16:59.988630 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:16:59.988601 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-5lkrv" Nov 03 01:17:00.190567 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:00.190522 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-576422vmz"] Nov 03 01:17:00.196011 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:00.195981 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-576422vmz" Nov 03 01:17:00.205380 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:00.205354 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-d25759\"/\"agc-cred-213e03dcdac3\"" Nov 03 01:17:00.207739 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:00.207706 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-576422vmz"] Nov 03 01:17:00.265815 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:00.265774 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07fbdd79-179a-4a9a-bd69-6541c8afff44-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d25759-576422vmz\" (UID: \"07fbdd79-179a-4a9a-bd69-6541c8afff44\") " pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-576422vmz" Nov 03 01:17:00.266043 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:00.265965 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjznm\" (UniqueName: \"kubernetes.io/projected/07fbdd79-179a-4a9a-bd69-6541c8afff44-kube-api-access-mjznm\") pod \"payload-processing-external-model-e2e-extmod-a-d25759-576422vmz\" (UID: \"07fbdd79-179a-4a9a-bd69-6541c8afff44\") " pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-576422vmz" Nov 03 01:17:00.266269 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:00.266248 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/07fbdd79-179a-4a9a-bd69-6541c8afff44-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-d25759-576422vmz\" (UID: \"07fbdd79-179a-4a9a-bd69-6541c8afff44\") " pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-576422vmz" Nov 03 01:17:00.266367 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:00.266322 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/07fbdd79-179a-4a9a-bd69-6541c8afff44-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d25759-576422vmz\" (UID: \"07fbdd79-179a-4a9a-bd69-6541c8afff44\") " pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-576422vmz" Nov 03 01:17:00.368090 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:00.368053 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/07fbdd79-179a-4a9a-bd69-6541c8afff44-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-d25759-576422vmz\" (UID: \"07fbdd79-179a-4a9a-bd69-6541c8afff44\") " pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-576422vmz" Nov 03 01:17:00.368325 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:00.368131 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/07fbdd79-179a-4a9a-bd69-6541c8afff44-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d25759-576422vmz\" (UID: \"07fbdd79-179a-4a9a-bd69-6541c8afff44\") " pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-576422vmz" Nov 03 01:17:00.368325 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:00.368295 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07fbdd79-179a-4a9a-bd69-6541c8afff44-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d25759-576422vmz\" (UID: \"07fbdd79-179a-4a9a-bd69-6541c8afff44\") " pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-576422vmz" Nov 03 01:17:00.368439 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:00.368424 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjznm\" (UniqueName: \"kubernetes.io/projected/07fbdd79-179a-4a9a-bd69-6541c8afff44-kube-api-access-mjznm\") pod \"payload-processing-external-model-e2e-extmod-a-d25759-576422vmz\" (UID: \"07fbdd79-179a-4a9a-bd69-6541c8afff44\") " pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-576422vmz" Nov 03 01:17:00.368981 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:00.368950 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/07fbdd79-179a-4a9a-bd69-6541c8afff44-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d25759-576422vmz\" (UID: \"07fbdd79-179a-4a9a-bd69-6541c8afff44\") " pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-576422vmz" Nov 03 01:17:00.369087 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:00.369065 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07fbdd79-179a-4a9a-bd69-6541c8afff44-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d25759-576422vmz\" (UID: \"07fbdd79-179a-4a9a-bd69-6541c8afff44\") " pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-576422vmz" Nov 03 01:17:00.370866 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:00.370848 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/07fbdd79-179a-4a9a-bd69-6541c8afff44-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-d25759-576422vmz\" (UID: \"07fbdd79-179a-4a9a-bd69-6541c8afff44\") " pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-576422vmz" Nov 03 01:17:00.377555 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:00.377529 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjznm\" (UniqueName: \"kubernetes.io/projected/07fbdd79-179a-4a9a-bd69-6541c8afff44-kube-api-access-mjznm\") pod \"payload-processing-external-model-e2e-extmod-a-d25759-576422vmz\" (UID: \"07fbdd79-179a-4a9a-bd69-6541c8afff44\") " pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-576422vmz" Nov 03 01:17:00.522094 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:00.522008 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-576422vmz" Nov 03 01:17:00.808202 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:17:00.808164 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07fbdd79_179a_4a9a_bd69_6541c8afff44.slice/crio-a1954e33d10fa7432916bb59ef609de2c637cf5f16e932b547789701dc2bb910 WatchSource:0}: Error finding container a1954e33d10fa7432916bb59ef609de2c637cf5f16e932b547789701dc2bb910: Status 404 returned error can't find the container with id a1954e33d10fa7432916bb59ef609de2c637cf5f16e932b547789701dc2bb910 Nov 03 01:17:00.808400 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:00.808380 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-576422vmz"] Nov 03 01:17:01.258934 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:01.258897 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-576422vmz" event={"ID":"07fbdd79-179a-4a9a-bd69-6541c8afff44","Type":"ContainerStarted","Data":"7ca85e2986c4435a5fc7f9fe077f23a09517000bc82ccc776c8c1689f289889b"} Nov 03 01:17:01.259458 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:01.258958 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-576422vmz" event={"ID":"07fbdd79-179a-4a9a-bd69-6541c8afff44","Type":"ContainerStarted","Data":"a1954e33d10fa7432916bb59ef609de2c637cf5f16e932b547789701dc2bb910"} Nov 03 01:17:01.259458 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:01.259211 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-576422vmz" Nov 03 01:17:02.594679 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:02.594604 4421 scope.go:122] "RemoveContainer" containerID="e75355bc23ff18d818c4b0ec2e307288b309a3f58aa7f5c65c3cf77ae688c60e" Nov 03 01:17:02.610425 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:02.610396 4421 scope.go:122] "RemoveContainer" containerID="ee4dbcd3e39c067fbe13dfe1b63aedc2e2e7891bb3c5a330e26dab0d05f747b7" Nov 03 01:17:02.621942 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:02.621912 4421 scope.go:122] "RemoveContainer" containerID="3c2f36f081b4a7265cfe781151e9c74decae9e970c3025789b5e68c257c7cb52" Nov 03 01:17:02.633745 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:02.633722 4421 scope.go:122] "RemoveContainer" containerID="122ef740c552b4ec3e5a77b97ab7baf1ae76f81d46f619e00261610ee9a3f342" Nov 03 01:17:12.270718 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:12.270686 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-576422vmz" Nov 03 01:17:12.292448 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:12.292392 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-576422vmz" podStartSLOduration=12.292376484 podStartE2EDuration="12.292376484s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:17:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:17:01.28639603 +0000 UTC m=+2163.457399127" watchObservedRunningTime="2026-11-03 01:17:12.292376484 +0000 UTC m=+2174.463379580" Nov 03 01:17:12.320143 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:12.320105 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-b894qjmvh"] Nov 03 01:17:14.132865 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:14.132819 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rgsmg"] Nov 03 01:17:14.133853 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:14.133796 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rgsmg" podUID="145e8805-e1b4-4f1f-80d0-86c7da396aee" containerName="istio-proxy" containerID="cri-o://27b6da461720fd54d135f5a8e49e7dd07c9761a700f3e88ac4150daa2116a785" gracePeriod=30 Nov 03 01:17:14.674185 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:17:14.674148 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 01:17:17.321674 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:17.321619 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-b894qjmvh" podUID="cdc5358a-3f4b-43e8-b889-38fa85711fbd" containerName="payload-processing" containerID="cri-o://92f5951089164ddf89736b0835e7932b8d849a9d88c9ba381365e1e11c6fc873" gracePeriod=25 Nov 03 01:17:17.717881 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:17.717849 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-b894qjmvh" Nov 03 01:17:17.800585 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:17.800549 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cdc5358a-3f4b-43e8-b889-38fa85711fbd-kube-api-access-x8vp6\" (UniqueName: \"kubernetes.io/projected/cdc5358a-3f4b-43e8-b889-38fa85711fbd-kube-api-access-x8vp6\") pod \"cdc5358a-3f4b-43e8-b889-38fa85711fbd\" (UID: \"cdc5358a-3f4b-43e8-b889-38fa85711fbd\") " Nov 03 01:17:17.800791 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:17.800598 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cdc5358a-3f4b-43e8-b889-38fa85711fbd-provider-credentials\" (UniqueName: \"kubernetes.io/projected/cdc5358a-3f4b-43e8-b889-38fa85711fbd-provider-credentials\") pod \"cdc5358a-3f4b-43e8-b889-38fa85711fbd\" (UID: \"cdc5358a-3f4b-43e8-b889-38fa85711fbd\") " Nov 03 01:17:17.800791 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:17.800693 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cdc5358a-3f4b-43e8-b889-38fa85711fbd-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cdc5358a-3f4b-43e8-b889-38fa85711fbd-routing-overlay-volume\") pod \"cdc5358a-3f4b-43e8-b889-38fa85711fbd\" (UID: \"cdc5358a-3f4b-43e8-b889-38fa85711fbd\") " Nov 03 01:17:17.800927 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:17.800828 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cdc5358a-3f4b-43e8-b889-38fa85711fbd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdc5358a-3f4b-43e8-b889-38fa85711fbd-plugins-config-volume\") pod \"cdc5358a-3f4b-43e8-b889-38fa85711fbd\" (UID: \"cdc5358a-3f4b-43e8-b889-38fa85711fbd\") " Nov 03 01:17:17.801152 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:17.801126 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cdc5358a-3f4b-43e8-b889-38fa85711fbd-routing-overlay-volume" pod "cdc5358a-3f4b-43e8-b889-38fa85711fbd" (UID: "cdc5358a-3f4b-43e8-b889-38fa85711fbd"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:17:17.801311 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:17.801266 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cdc5358a-3f4b-43e8-b889-38fa85711fbd-plugins-config-volume" pod "cdc5358a-3f4b-43e8-b889-38fa85711fbd" (UID: "cdc5358a-3f4b-43e8-b889-38fa85711fbd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:17:17.801639 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:17.801613 4421 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cdc5358a-3f4b-43e8-b889-38fa85711fbd-routing-overlay-volume\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:17:17.801639 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:17.801632 4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdc5358a-3f4b-43e8-b889-38fa85711fbd-plugins-config-volume\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:17:17.803077 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:17.803056 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdc5358a-3f4b-43e8-b889-38fa85711fbd-provider-credentials" pod "cdc5358a-3f4b-43e8-b889-38fa85711fbd" (UID: "cdc5358a-3f4b-43e8-b889-38fa85711fbd"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:17:17.803178 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:17.803147 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdc5358a-3f4b-43e8-b889-38fa85711fbd-kube-api-access-x8vp6" pod "cdc5358a-3f4b-43e8-b889-38fa85711fbd" (UID: "cdc5358a-3f4b-43e8-b889-38fa85711fbd"). InnerVolumeSpecName "kube-api-access-x8vp6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:17:17.902942 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:17.902905 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8vp6\" (UniqueName: \"kubernetes.io/projected/cdc5358a-3f4b-43e8-b889-38fa85711fbd-kube-api-access-x8vp6\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:17:17.902942 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:17.902939 4421 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cdc5358a-3f4b-43e8-b889-38fa85711fbd-provider-credentials\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:17:18.439265 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:18.439223 4421 generic.go:356] "Generic (PLEG): container finished" podID="cdc5358a-3f4b-43e8-b889-38fa85711fbd" containerID="92f5951089164ddf89736b0835e7932b8d849a9d88c9ba381365e1e11c6fc873" exitCode=0 Nov 03 01:17:18.439764 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:18.439321 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-b894qjmvh" Nov 03 01:17:18.439764 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:18.439315 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-b894qjmvh" event={"ID":"cdc5358a-3f4b-43e8-b889-38fa85711fbd","Type":"ContainerDied","Data":"92f5951089164ddf89736b0835e7932b8d849a9d88c9ba381365e1e11c6fc873"} Nov 03 01:17:18.439764 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:18.439456 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-b894qjmvh" event={"ID":"cdc5358a-3f4b-43e8-b889-38fa85711fbd","Type":"ContainerDied","Data":"bd593175aa1002d05064fc4e0e48fac329e182c7beb5c0f41328add6cd243dfe"} Nov 03 01:17:18.439764 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:18.439489 4421 scope.go:122] "RemoveContainer" containerID="92f5951089164ddf89736b0835e7932b8d849a9d88c9ba381365e1e11c6fc873" Nov 03 01:17:18.455783 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:18.455751 4421 scope.go:122] "RemoveContainer" containerID="92f5951089164ddf89736b0835e7932b8d849a9d88c9ba381365e1e11c6fc873" Nov 03 01:17:18.456373 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:17:18.456258 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92f5951089164ddf89736b0835e7932b8d849a9d88c9ba381365e1e11c6fc873\": container with ID starting with 92f5951089164ddf89736b0835e7932b8d849a9d88c9ba381365e1e11c6fc873 not found: ID does not exist" containerID="92f5951089164ddf89736b0835e7932b8d849a9d88c9ba381365e1e11c6fc873" Nov 03 01:17:18.456496 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:18.456386 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92f5951089164ddf89736b0835e7932b8d849a9d88c9ba381365e1e11c6fc873"} err="failed to get container status \"92f5951089164ddf89736b0835e7932b8d849a9d88c9ba381365e1e11c6fc873\": rpc error: code = NotFound desc = could not find container \"92f5951089164ddf89736b0835e7932b8d849a9d88c9ba381365e1e11c6fc873\": container with ID starting with 92f5951089164ddf89736b0835e7932b8d849a9d88c9ba381365e1e11c6fc873 not found: ID does not exist" Nov 03 01:17:18.462363 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:18.462339 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-b894qjmvh"] Nov 03 01:17:18.467174 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:18.467151 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-b894qjmvh"] Nov 03 01:17:19.450503 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:19.450422 4421 generic.go:356] "Generic (PLEG): container finished" podID="145e8805-e1b4-4f1f-80d0-86c7da396aee" containerID="27b6da461720fd54d135f5a8e49e7dd07c9761a700f3e88ac4150daa2116a785" exitCode=0 Nov 03 01:17:19.450503 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:19.450455 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rgsmg" event={"ID":"145e8805-e1b4-4f1f-80d0-86c7da396aee","Type":"ContainerDied","Data":"27b6da461720fd54d135f5a8e49e7dd07c9761a700f3e88ac4150daa2116a785"} Nov 03 01:17:19.901338 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:19.901308 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rgsmg" Nov 03 01:17:19.947773 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:19.947736 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/145e8805-e1b4-4f1f-80d0-86c7da396aee-kube-api-access-n2chs\" (UniqueName: \"kubernetes.io/projected/145e8805-e1b4-4f1f-80d0-86c7da396aee-kube-api-access-n2chs\") pod \"145e8805-e1b4-4f1f-80d0-86c7da396aee\" (UID: \"145e8805-e1b4-4f1f-80d0-86c7da396aee\") " Nov 03 01:17:19.948047 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:19.947808 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/145e8805-e1b4-4f1f-80d0-86c7da396aee-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/145e8805-e1b4-4f1f-80d0-86c7da396aee-istio-data\") pod \"145e8805-e1b4-4f1f-80d0-86c7da396aee\" (UID: \"145e8805-e1b4-4f1f-80d0-86c7da396aee\") " Nov 03 01:17:19.948047 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:19.947956 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/145e8805-e1b4-4f1f-80d0-86c7da396aee-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/145e8805-e1b4-4f1f-80d0-86c7da396aee-workload-socket\") pod \"145e8805-e1b4-4f1f-80d0-86c7da396aee\" (UID: \"145e8805-e1b4-4f1f-80d0-86c7da396aee\") " Nov 03 01:17:19.948047 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:19.947998 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/145e8805-e1b4-4f1f-80d0-86c7da396aee-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/145e8805-e1b4-4f1f-80d0-86c7da396aee-istio-envoy\") pod \"145e8805-e1b4-4f1f-80d0-86c7da396aee\" (UID: \"145e8805-e1b4-4f1f-80d0-86c7da396aee\") " Nov 03 01:17:19.948208 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:19.948077 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/145e8805-e1b4-4f1f-80d0-86c7da396aee-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/145e8805-e1b4-4f1f-80d0-86c7da396aee-credential-socket\") pod \"145e8805-e1b4-4f1f-80d0-86c7da396aee\" (UID: \"145e8805-e1b4-4f1f-80d0-86c7da396aee\") " Nov 03 01:17:19.948208 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:19.948158 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-be59df-7499f75b58-xgcq7"] Nov 03 01:17:19.948208 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:19.948171 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/145e8805-e1b4-4f1f-80d0-86c7da396aee-istio-token\" (UniqueName: \"kubernetes.io/projected/145e8805-e1b4-4f1f-80d0-86c7da396aee-istio-token\") pod \"145e8805-e1b4-4f1f-80d0-86c7da396aee\" (UID: \"145e8805-e1b4-4f1f-80d0-86c7da396aee\") " Nov 03 01:17:19.948378 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:19.948230 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/145e8805-e1b4-4f1f-80d0-86c7da396aee-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/145e8805-e1b4-4f1f-80d0-86c7da396aee-istiod-ca-cert\") pod \"145e8805-e1b4-4f1f-80d0-86c7da396aee\" (UID: \"145e8805-e1b4-4f1f-80d0-86c7da396aee\") " Nov 03 01:17:19.948378 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:19.948354 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/145e8805-e1b4-4f1f-80d0-86c7da396aee-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/145e8805-e1b4-4f1f-80d0-86c7da396aee-istio-podinfo\") pod \"145e8805-e1b4-4f1f-80d0-86c7da396aee\" (UID: \"145e8805-e1b4-4f1f-80d0-86c7da396aee\") " Nov 03 01:17:19.950248 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:19.948761 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/145e8805-e1b4-4f1f-80d0-86c7da396aee-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/145e8805-e1b4-4f1f-80d0-86c7da396aee-workload-certs\") pod \"145e8805-e1b4-4f1f-80d0-86c7da396aee\" (UID: \"145e8805-e1b4-4f1f-80d0-86c7da396aee\") " Nov 03 01:17:19.950248 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:19.949169 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/145e8805-e1b4-4f1f-80d0-86c7da396aee-credential-socket" pod "145e8805-e1b4-4f1f-80d0-86c7da396aee" (UID: "145e8805-e1b4-4f1f-80d0-86c7da396aee"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:17:19.950248 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:19.949327 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/145e8805-e1b4-4f1f-80d0-86c7da396aee-istio-data" pod "145e8805-e1b4-4f1f-80d0-86c7da396aee" (UID: "145e8805-e1b4-4f1f-80d0-86c7da396aee"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:17:19.950248 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:19.949420 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/145e8805-e1b4-4f1f-80d0-86c7da396aee-istiod-ca-cert" pod "145e8805-e1b4-4f1f-80d0-86c7da396aee" (UID: "145e8805-e1b4-4f1f-80d0-86c7da396aee"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:17:19.950248 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:19.949626 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/145e8805-e1b4-4f1f-80d0-86c7da396aee-workload-certs" pod "145e8805-e1b4-4f1f-80d0-86c7da396aee" (UID: "145e8805-e1b4-4f1f-80d0-86c7da396aee"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:17:19.950248 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:19.949730 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/145e8805-e1b4-4f1f-80d0-86c7da396aee-workload-socket" pod "145e8805-e1b4-4f1f-80d0-86c7da396aee" (UID: "145e8805-e1b4-4f1f-80d0-86c7da396aee"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:17:19.950248 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:19.950166 4421 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/145e8805-e1b4-4f1f-80d0-86c7da396aee-workload-certs\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:17:19.950248 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:19.950185 4421 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/145e8805-e1b4-4f1f-80d0-86c7da396aee-istio-data\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:17:19.950248 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:19.950197 4421 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/145e8805-e1b4-4f1f-80d0-86c7da396aee-workload-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:17:19.950248 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:19.950209 4421 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/145e8805-e1b4-4f1f-80d0-86c7da396aee-credential-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:17:19.950248 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:19.950221 4421 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/145e8805-e1b4-4f1f-80d0-86c7da396aee-istiod-ca-cert\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:17:19.950966 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:19.950458 4421 cpu_manager.go:414] "containerMap: removing container" podUID="145e8805-e1b4-4f1f-80d0-86c7da396aee" containerName="istio-proxy" Nov 03 01:17:19.950966 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:19.950474 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="145e8805-e1b4-4f1f-80d0-86c7da396aee" containerName="istio-proxy" Nov 03 01:17:19.950966 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:19.950499 4421 cpu_manager.go:414] "containerMap: removing container" podUID="cdc5358a-3f4b-43e8-b889-38fa85711fbd" containerName="payload-processing" Nov 03 01:17:19.950966 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:19.950507 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cdc5358a-3f4b-43e8-b889-38fa85711fbd" containerName="payload-processing" Nov 03 01:17:19.950966 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:19.950696 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="145e8805-e1b4-4f1f-80d0-86c7da396aee" containerName="istio-proxy" Nov 03 01:17:19.950966 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:19.950708 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="cdc5358a-3f4b-43e8-b889-38fa85711fbd" containerName="payload-processing" Nov 03 01:17:19.953958 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:19.953749 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/145e8805-e1b4-4f1f-80d0-86c7da396aee-istio-token" pod "145e8805-e1b4-4f1f-80d0-86c7da396aee" (UID: "145e8805-e1b4-4f1f-80d0-86c7da396aee"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:17:19.954600 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:19.954566 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/145e8805-e1b4-4f1f-80d0-86c7da396aee-istio-podinfo" pod "145e8805-e1b4-4f1f-80d0-86c7da396aee" (UID: "145e8805-e1b4-4f1f-80d0-86c7da396aee"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:17:19.954600 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:19.954578 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/145e8805-e1b4-4f1f-80d0-86c7da396aee-istio-envoy" pod "145e8805-e1b4-4f1f-80d0-86c7da396aee" (UID: "145e8805-e1b4-4f1f-80d0-86c7da396aee"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:17:19.955615 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:19.955580 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/145e8805-e1b4-4f1f-80d0-86c7da396aee-kube-api-access-n2chs" pod "145e8805-e1b4-4f1f-80d0-86c7da396aee" (UID: "145e8805-e1b4-4f1f-80d0-86c7da396aee"). InnerVolumeSpecName "kube-api-access-n2chs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:17:19.956556 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:19.956502 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-be59df-7499f75b58-xgcq7" Nov 03 01:17:19.962321 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:19.962169 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-be59df-serving-cert\"" Nov 03 01:17:19.966405 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:19.966380 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-be59df-7499f75b58-xgcq7"] Nov 03 01:17:19.987215 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:19.987180 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cdc5358a-3f4b-43e8-b889-38fa85711fbd" path="/var/lib/kubelet/pods/cdc5358a-3f4b-43e8-b889-38fa85711fbd/volumes" Nov 03 01:17:20.064026 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:20.063910 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8fxd\" (UniqueName: \"kubernetes.io/projected/5c6b86d5-52d4-449c-90c4-3d5bad417c00-kube-api-access-p8fxd\") pod \"maas-api-e2e-extmod-b-be59df-7499f75b58-xgcq7\" (UID: \"5c6b86d5-52d4-449c-90c4-3d5bad417c00\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-be59df-7499f75b58-xgcq7" Nov 03 01:17:20.064309 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:20.064264 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c6b86d5-52d4-449c-90c4-3d5bad417c00-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-be59df-7499f75b58-xgcq7\" (UID: \"5c6b86d5-52d4-449c-90c4-3d5bad417c00\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-be59df-7499f75b58-xgcq7" Nov 03 01:17:20.064447 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:20.064406 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c6b86d5-52d4-449c-90c4-3d5bad417c00-metrics-certs\") pod \"maas-api-e2e-extmod-b-be59df-7499f75b58-xgcq7\" (UID: \"5c6b86d5-52d4-449c-90c4-3d5bad417c00\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-be59df-7499f75b58-xgcq7" Nov 03 01:17:20.064620 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:20.064592 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c6b86d5-52d4-449c-90c4-3d5bad417c00-maas-api-tls\") pod \"maas-api-e2e-extmod-b-be59df-7499f75b58-xgcq7\" (UID: \"5c6b86d5-52d4-449c-90c4-3d5bad417c00\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-be59df-7499f75b58-xgcq7" Nov 03 01:17:20.064825 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:20.064799 4421 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/145e8805-e1b4-4f1f-80d0-86c7da396aee-istio-token\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:17:20.064825 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:20.064820 4421 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/145e8805-e1b4-4f1f-80d0-86c7da396aee-istio-podinfo\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:17:20.064962 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:20.064831 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2chs\" (UniqueName: \"kubernetes.io/projected/145e8805-e1b4-4f1f-80d0-86c7da396aee-kube-api-access-n2chs\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:17:20.064962 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:20.064841 4421 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/145e8805-e1b4-4f1f-80d0-86c7da396aee-istio-envoy\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:17:20.166071 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:20.166030 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c6b86d5-52d4-449c-90c4-3d5bad417c00-metrics-certs\") pod \"maas-api-e2e-extmod-b-be59df-7499f75b58-xgcq7\" (UID: \"5c6b86d5-52d4-449c-90c4-3d5bad417c00\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-be59df-7499f75b58-xgcq7" Nov 03 01:17:20.166352 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:20.166108 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c6b86d5-52d4-449c-90c4-3d5bad417c00-maas-api-tls\") pod \"maas-api-e2e-extmod-b-be59df-7499f75b58-xgcq7\" (UID: \"5c6b86d5-52d4-449c-90c4-3d5bad417c00\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-be59df-7499f75b58-xgcq7" Nov 03 01:17:20.166352 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:20.166231 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8fxd\" (UniqueName: \"kubernetes.io/projected/5c6b86d5-52d4-449c-90c4-3d5bad417c00-kube-api-access-p8fxd\") pod \"maas-api-e2e-extmod-b-be59df-7499f75b58-xgcq7\" (UID: \"5c6b86d5-52d4-449c-90c4-3d5bad417c00\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-be59df-7499f75b58-xgcq7" Nov 03 01:17:20.166541 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:20.166515 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c6b86d5-52d4-449c-90c4-3d5bad417c00-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-be59df-7499f75b58-xgcq7\" (UID: \"5c6b86d5-52d4-449c-90c4-3d5bad417c00\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-be59df-7499f75b58-xgcq7" Nov 03 01:17:20.167213 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:20.167187 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c6b86d5-52d4-449c-90c4-3d5bad417c00-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-be59df-7499f75b58-xgcq7\" (UID: \"5c6b86d5-52d4-449c-90c4-3d5bad417c00\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-be59df-7499f75b58-xgcq7" Nov 03 01:17:20.168692 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:20.168665 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c6b86d5-52d4-449c-90c4-3d5bad417c00-metrics-certs\") pod \"maas-api-e2e-extmod-b-be59df-7499f75b58-xgcq7\" (UID: \"5c6b86d5-52d4-449c-90c4-3d5bad417c00\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-be59df-7499f75b58-xgcq7" Nov 03 01:17:20.168824 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:20.168789 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c6b86d5-52d4-449c-90c4-3d5bad417c00-maas-api-tls\") pod \"maas-api-e2e-extmod-b-be59df-7499f75b58-xgcq7\" (UID: \"5c6b86d5-52d4-449c-90c4-3d5bad417c00\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-be59df-7499f75b58-xgcq7" Nov 03 01:17:20.174939 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:20.174911 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8fxd\" (UniqueName: \"kubernetes.io/projected/5c6b86d5-52d4-449c-90c4-3d5bad417c00-kube-api-access-p8fxd\") pod \"maas-api-e2e-extmod-b-be59df-7499f75b58-xgcq7\" (UID: \"5c6b86d5-52d4-449c-90c4-3d5bad417c00\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-be59df-7499f75b58-xgcq7" Nov 03 01:17:20.351352 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:20.350765 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-be59df-7499f75b58-xgcq7" Nov 03 01:17:20.469477 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:20.468771 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rgsmg" event={"ID":"145e8805-e1b4-4f1f-80d0-86c7da396aee","Type":"ContainerDied","Data":"a525f410061b551b1160cd77bd493426ae16fccb6b5e9616b99aa5a29af838e0"} Nov 03 01:17:20.469477 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:20.468884 4421 scope.go:122] "RemoveContainer" containerID="27b6da461720fd54d135f5a8e49e7dd07c9761a700f3e88ac4150daa2116a785" Nov 03 01:17:20.469477 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:20.469071 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rgsmg" Nov 03 01:17:20.495412 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:20.495379 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rgsmg"] Nov 03 01:17:20.502390 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:20.502352 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rgsmg"] Nov 03 01:17:20.669807 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:17:20.669775 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c6b86d5_52d4_449c_90c4_3d5bad417c00.slice/crio-8eb1e483271feb22c2a9e2c3854c5fed00c5f77b6b34863c9b755a8ba65e6db1 WatchSource:0}: Error finding container 8eb1e483271feb22c2a9e2c3854c5fed00c5f77b6b34863c9b755a8ba65e6db1: Status 404 returned error can't find the container with id 8eb1e483271feb22c2a9e2c3854c5fed00c5f77b6b34863c9b755a8ba65e6db1 Nov 03 01:17:20.670106 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:20.670084 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-be59df-7499f75b58-xgcq7"] Nov 03 01:17:20.671734 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:20.671713 4421 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 01:17:21.494172 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:21.494065 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-be59df-7499f75b58-xgcq7" event={"ID":"5c6b86d5-52d4-449c-90c4-3d5bad417c00","Type":"ContainerStarted","Data":"b91771ea32f82017c0e920b30d0a5ba8dbd437c3211196c47149504a52d5597e"} Nov 03 01:17:21.494172 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:21.494158 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-be59df-7499f75b58-xgcq7" event={"ID":"5c6b86d5-52d4-449c-90c4-3d5bad417c00","Type":"ContainerStarted","Data":"8eb1e483271feb22c2a9e2c3854c5fed00c5f77b6b34863c9b755a8ba65e6db1"} Nov 03 01:17:21.495584 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:21.495555 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-be59df-7499f75b58-xgcq7" Nov 03 01:17:21.518257 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:21.518180 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-be59df-7499f75b58-xgcq7" podStartSLOduration=40.978360048 podStartE2EDuration="41.518162819s" totalImagesPullingTime="539.802771ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:16:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:17:21.517212811 +0000 UTC m=+2183.688215906" watchObservedRunningTime="2026-11-03 01:17:21.518162819 +0000 UTC m=+2183.689165914" Nov 03 01:17:22.001002 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:22.000950 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="145e8805-e1b4-4f1f-80d0-86c7da396aee" path="/var/lib/kubelet/pods/145e8805-e1b4-4f1f-80d0-86c7da396aee/volumes" Nov 03 01:17:28.558296 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:28.553710 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-be59df-7499f75b58-xgcq7" Nov 03 01:17:31.085997 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:31.085960 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-wzfts"] Nov 03 01:17:31.086504 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:31.086172 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-wzfts" podUID="a5d698b3-22ca-43c5-9d37-37e38d418444" containerName="istio-proxy" containerID="cri-o://37a2ddb3407dde0c226d00c827faa3b7e19edc83e8b14a160e1bec20cdccceb3" gracePeriod=30 Nov 03 01:17:36.470832 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:36.470804 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-wzfts" Nov 03 01:17:36.509932 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:36.509892 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5d698b3-22ca-43c5-9d37-37e38d418444-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5d698b3-22ca-43c5-9d37-37e38d418444-workload-socket\") pod \"a5d698b3-22ca-43c5-9d37-37e38d418444\" (UID: \"a5d698b3-22ca-43c5-9d37-37e38d418444\") " Nov 03 01:17:36.510145 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:36.510127 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5d698b3-22ca-43c5-9d37-37e38d418444-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5d698b3-22ca-43c5-9d37-37e38d418444-istio-envoy\") pod \"a5d698b3-22ca-43c5-9d37-37e38d418444\" (UID: \"a5d698b3-22ca-43c5-9d37-37e38d418444\") " Nov 03 01:17:36.510228 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:36.510168 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5d698b3-22ca-43c5-9d37-37e38d418444-workload-socket" pod "a5d698b3-22ca-43c5-9d37-37e38d418444" (UID: "a5d698b3-22ca-43c5-9d37-37e38d418444"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:17:36.510324 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:36.510241 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a5d698b3-22ca-43c5-9d37-37e38d418444-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5d698b3-22ca-43c5-9d37-37e38d418444-istio-podinfo\") pod \"a5d698b3-22ca-43c5-9d37-37e38d418444\" (UID: \"a5d698b3-22ca-43c5-9d37-37e38d418444\") " Nov 03 01:17:36.510324 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:36.510315 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5d698b3-22ca-43c5-9d37-37e38d418444-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5d698b3-22ca-43c5-9d37-37e38d418444-credential-socket\") pod \"a5d698b3-22ca-43c5-9d37-37e38d418444\" (UID: \"a5d698b3-22ca-43c5-9d37-37e38d418444\") " Nov 03 01:17:36.510458 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:36.510360 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5d698b3-22ca-43c5-9d37-37e38d418444-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5d698b3-22ca-43c5-9d37-37e38d418444-istio-data\") pod \"a5d698b3-22ca-43c5-9d37-37e38d418444\" (UID: \"a5d698b3-22ca-43c5-9d37-37e38d418444\") " Nov 03 01:17:36.510458 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:36.510394 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5d698b3-22ca-43c5-9d37-37e38d418444-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5d698b3-22ca-43c5-9d37-37e38d418444-workload-certs\") pod \"a5d698b3-22ca-43c5-9d37-37e38d418444\" (UID: \"a5d698b3-22ca-43c5-9d37-37e38d418444\") " Nov 03 01:17:36.510458 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:36.510434 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5d698b3-22ca-43c5-9d37-37e38d418444-istio-token\" (UniqueName: \"kubernetes.io/projected/a5d698b3-22ca-43c5-9d37-37e38d418444-istio-token\") pod \"a5d698b3-22ca-43c5-9d37-37e38d418444\" (UID: \"a5d698b3-22ca-43c5-9d37-37e38d418444\") " Nov 03 01:17:36.510638 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:36.510481 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5d698b3-22ca-43c5-9d37-37e38d418444-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5d698b3-22ca-43c5-9d37-37e38d418444-istiod-ca-cert\") pod \"a5d698b3-22ca-43c5-9d37-37e38d418444\" (UID: \"a5d698b3-22ca-43c5-9d37-37e38d418444\") " Nov 03 01:17:36.510638 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:36.510590 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5d698b3-22ca-43c5-9d37-37e38d418444-kube-api-access-gfl7p\" (UniqueName: \"kubernetes.io/projected/a5d698b3-22ca-43c5-9d37-37e38d418444-kube-api-access-gfl7p\") pod \"a5d698b3-22ca-43c5-9d37-37e38d418444\" (UID: \"a5d698b3-22ca-43c5-9d37-37e38d418444\") " Nov 03 01:17:36.511079 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:36.510816 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5d698b3-22ca-43c5-9d37-37e38d418444-workload-certs" pod "a5d698b3-22ca-43c5-9d37-37e38d418444" (UID: "a5d698b3-22ca-43c5-9d37-37e38d418444"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:17:36.511079 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:36.511044 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5d698b3-22ca-43c5-9d37-37e38d418444-credential-socket" pod "a5d698b3-22ca-43c5-9d37-37e38d418444" (UID: "a5d698b3-22ca-43c5-9d37-37e38d418444"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:17:36.511406 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:36.511378 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5d698b3-22ca-43c5-9d37-37e38d418444-istio-data" pod "a5d698b3-22ca-43c5-9d37-37e38d418444" (UID: "a5d698b3-22ca-43c5-9d37-37e38d418444"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:17:36.511690 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:36.511668 4421 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5d698b3-22ca-43c5-9d37-37e38d418444-workload-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:17:36.511790 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:36.511695 4421 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5d698b3-22ca-43c5-9d37-37e38d418444-credential-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:17:36.511790 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:36.511711 4421 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5d698b3-22ca-43c5-9d37-37e38d418444-istio-data\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:17:36.511790 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:36.511724 4421 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5d698b3-22ca-43c5-9d37-37e38d418444-workload-certs\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:17:36.511973 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:36.511886 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5d698b3-22ca-43c5-9d37-37e38d418444-istiod-ca-cert" pod "a5d698b3-22ca-43c5-9d37-37e38d418444" (UID: "a5d698b3-22ca-43c5-9d37-37e38d418444"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:17:36.515153 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:36.515093 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a5d698b3-22ca-43c5-9d37-37e38d418444-istio-podinfo" pod "a5d698b3-22ca-43c5-9d37-37e38d418444" (UID: "a5d698b3-22ca-43c5-9d37-37e38d418444"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:17:36.515321 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:36.515217 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5d698b3-22ca-43c5-9d37-37e38d418444-istio-envoy" pod "a5d698b3-22ca-43c5-9d37-37e38d418444" (UID: "a5d698b3-22ca-43c5-9d37-37e38d418444"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:17:36.515473 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:36.515438 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5d698b3-22ca-43c5-9d37-37e38d418444-istio-token" pod "a5d698b3-22ca-43c5-9d37-37e38d418444" (UID: "a5d698b3-22ca-43c5-9d37-37e38d418444"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:17:36.516441 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:36.516415 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5d698b3-22ca-43c5-9d37-37e38d418444-kube-api-access-gfl7p" pod "a5d698b3-22ca-43c5-9d37-37e38d418444" (UID: "a5d698b3-22ca-43c5-9d37-37e38d418444"). InnerVolumeSpecName "kube-api-access-gfl7p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:17:36.613414 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:36.613376 4421 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5d698b3-22ca-43c5-9d37-37e38d418444-istio-envoy\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:17:36.613414 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:36.613412 4421 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5d698b3-22ca-43c5-9d37-37e38d418444-istio-podinfo\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:17:36.613679 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:36.613428 4421 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5d698b3-22ca-43c5-9d37-37e38d418444-istio-token\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:17:36.613679 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:36.613442 4421 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5d698b3-22ca-43c5-9d37-37e38d418444-istiod-ca-cert\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:17:36.613679 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:36.613456 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gfl7p\" (UniqueName: \"kubernetes.io/projected/a5d698b3-22ca-43c5-9d37-37e38d418444-kube-api-access-gfl7p\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:17:36.698607 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:36.698514 4421 generic.go:356] "Generic (PLEG): container finished" podID="a5d698b3-22ca-43c5-9d37-37e38d418444" containerID="37a2ddb3407dde0c226d00c827faa3b7e19edc83e8b14a160e1bec20cdccceb3" exitCode=0 Nov 03 01:17:36.698607 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:36.698567 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-wzfts" Nov 03 01:17:36.698607 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:36.698576 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-wzfts" event={"ID":"a5d698b3-22ca-43c5-9d37-37e38d418444","Type":"ContainerDied","Data":"37a2ddb3407dde0c226d00c827faa3b7e19edc83e8b14a160e1bec20cdccceb3"} Nov 03 01:17:36.698884 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:36.698631 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-wzfts" event={"ID":"a5d698b3-22ca-43c5-9d37-37e38d418444","Type":"ContainerDied","Data":"2119ba706eebb10d5fd0f9fe2f2325fad0cfa26f325d4c00ae4ae6814ff0d1ec"} Nov 03 01:17:36.698884 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:36.698656 4421 scope.go:122] "RemoveContainer" containerID="37a2ddb3407dde0c226d00c827faa3b7e19edc83e8b14a160e1bec20cdccceb3" Nov 03 01:17:36.712703 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:36.712682 4421 scope.go:122] "RemoveContainer" containerID="37a2ddb3407dde0c226d00c827faa3b7e19edc83e8b14a160e1bec20cdccceb3" Nov 03 01:17:36.713078 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:17:36.713058 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37a2ddb3407dde0c226d00c827faa3b7e19edc83e8b14a160e1bec20cdccceb3\": container with ID starting with 37a2ddb3407dde0c226d00c827faa3b7e19edc83e8b14a160e1bec20cdccceb3 not found: ID does not exist" containerID="37a2ddb3407dde0c226d00c827faa3b7e19edc83e8b14a160e1bec20cdccceb3" Nov 03 01:17:36.713134 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:36.713091 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37a2ddb3407dde0c226d00c827faa3b7e19edc83e8b14a160e1bec20cdccceb3"} err="failed to get container status \"37a2ddb3407dde0c226d00c827faa3b7e19edc83e8b14a160e1bec20cdccceb3\": rpc error: code = NotFound desc = could not find container \"37a2ddb3407dde0c226d00c827faa3b7e19edc83e8b14a160e1bec20cdccceb3\": container with ID starting with 37a2ddb3407dde0c226d00c827faa3b7e19edc83e8b14a160e1bec20cdccceb3 not found: ID does not exist" Nov 03 01:17:36.727094 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:36.727059 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-wzfts"] Nov 03 01:17:36.731984 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:36.731955 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-wzfts"] Nov 03 01:17:37.981652 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:37.981616 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5d698b3-22ca-43c5-9d37-37e38d418444" path="/var/lib/kubelet/pods/a5d698b3-22ca-43c5-9d37-37e38d418444/volumes" Nov 03 01:17:44.015963 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:44.015875 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-576422vmz"] Nov 03 01:17:44.672254 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:17:44.672217 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 01:17:48.906863 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:48.906831 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37d9f0bb-k7qkd"] Nov 03 01:17:48.907821 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:48.907802 4421 cpu_manager.go:414] "containerMap: removing container" podUID="a5d698b3-22ca-43c5-9d37-37e38d418444" containerName="istio-proxy" Nov 03 01:17:48.907913 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:48.907824 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5d698b3-22ca-43c5-9d37-37e38d418444" containerName="istio-proxy" Nov 03 01:17:48.907958 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:48.907949 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5d698b3-22ca-43c5-9d37-37e38d418444" containerName="istio-proxy" Nov 03 01:17:48.913883 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:48.913849 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37d9f0bb-k7qkd" Nov 03 01:17:48.922979 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:48.922938 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:17:48.923318 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:48.923289 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6q4xp\"" Nov 03 01:17:48.931737 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:48.931704 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37d9f0bb-k7qkd"] Nov 03 01:17:49.016959 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:49.016913 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-576422vmz" podUID="07fbdd79-179a-4a9a-bd69-6541c8afff44" containerName="payload-processing" containerID="cri-o://7ca85e2986c4435a5fc7f9fe077f23a09517000bc82ccc776c8c1689f289889b" gracePeriod=25 Nov 03 01:17:49.018140 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:49.018112 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwvtg\" (UniqueName: \"kubernetes.io/projected/f01b6791-5118-430d-9034-8f9760f9d963-kube-api-access-vwvtg\") pod \"maas-api-revoke-sub-37d9f0bb-k7qkd\" (UID: \"f01b6791-5118-430d-9034-8f9760f9d963\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37d9f0bb-k7qkd" Nov 03 01:17:49.018239 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:49.018228 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f01b6791-5118-430d-9034-8f9760f9d963-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-37d9f0bb-k7qkd\" (UID: \"f01b6791-5118-430d-9034-8f9760f9d963\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37d9f0bb-k7qkd" Nov 03 01:17:49.018359 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:49.018338 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f01b6791-5118-430d-9034-8f9760f9d963-maas-api-xks-ca\") pod \"maas-api-revoke-sub-37d9f0bb-k7qkd\" (UID: \"f01b6791-5118-430d-9034-8f9760f9d963\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37d9f0bb-k7qkd" Nov 03 01:17:49.146704 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:49.146666 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwvtg\" (UniqueName: \"kubernetes.io/projected/f01b6791-5118-430d-9034-8f9760f9d963-kube-api-access-vwvtg\") pod \"maas-api-revoke-sub-37d9f0bb-k7qkd\" (UID: \"f01b6791-5118-430d-9034-8f9760f9d963\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37d9f0bb-k7qkd" Nov 03 01:17:49.147093 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:49.147061 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f01b6791-5118-430d-9034-8f9760f9d963-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-37d9f0bb-k7qkd\" (UID: \"f01b6791-5118-430d-9034-8f9760f9d963\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37d9f0bb-k7qkd" Nov 03 01:17:49.148357 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:49.148267 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f01b6791-5118-430d-9034-8f9760f9d963-maas-api-xks-ca\") pod \"maas-api-revoke-sub-37d9f0bb-k7qkd\" (UID: \"f01b6791-5118-430d-9034-8f9760f9d963\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37d9f0bb-k7qkd" Nov 03 01:17:49.149440 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:49.149410 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f01b6791-5118-430d-9034-8f9760f9d963-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-37d9f0bb-k7qkd\" (UID: \"f01b6791-5118-430d-9034-8f9760f9d963\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37d9f0bb-k7qkd" Nov 03 01:17:49.194375 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:49.175097 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwvtg\" (UniqueName: \"kubernetes.io/projected/f01b6791-5118-430d-9034-8f9760f9d963-kube-api-access-vwvtg\") pod \"maas-api-revoke-sub-37d9f0bb-k7qkd\" (UID: \"f01b6791-5118-430d-9034-8f9760f9d963\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37d9f0bb-k7qkd" Nov 03 01:17:49.229895 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:49.229818 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f01b6791-5118-430d-9034-8f9760f9d963-maas-api-xks-ca\") pod \"maas-api-revoke-sub-37d9f0bb-k7qkd\" (UID: \"f01b6791-5118-430d-9034-8f9760f9d963\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37d9f0bb-k7qkd" Nov 03 01:17:49.259300 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:49.254543 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37d9f0bb-k7qkd" Nov 03 01:17:49.721004 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:49.720978 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-576422vmz" Nov 03 01:17:49.770120 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:49.770088 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07fbdd79-179a-4a9a-bd69-6541c8afff44-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07fbdd79-179a-4a9a-bd69-6541c8afff44-plugins-config-volume\") pod \"07fbdd79-179a-4a9a-bd69-6541c8afff44\" (UID: \"07fbdd79-179a-4a9a-bd69-6541c8afff44\") " Nov 03 01:17:49.770348 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:49.770319 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07fbdd79-179a-4a9a-bd69-6541c8afff44-kube-api-access-mjznm\" (UniqueName: \"kubernetes.io/projected/07fbdd79-179a-4a9a-bd69-6541c8afff44-kube-api-access-mjznm\") pod \"07fbdd79-179a-4a9a-bd69-6541c8afff44\" (UID: \"07fbdd79-179a-4a9a-bd69-6541c8afff44\") " Nov 03 01:17:49.772132 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:49.770379 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07fbdd79-179a-4a9a-bd69-6541c8afff44-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/07fbdd79-179a-4a9a-bd69-6541c8afff44-routing-overlay-volume\") pod \"07fbdd79-179a-4a9a-bd69-6541c8afff44\" (UID: \"07fbdd79-179a-4a9a-bd69-6541c8afff44\") " Nov 03 01:17:49.772132 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:49.770571 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07fbdd79-179a-4a9a-bd69-6541c8afff44-provider-credentials\" (UniqueName: \"kubernetes.io/projected/07fbdd79-179a-4a9a-bd69-6541c8afff44-provider-credentials\") pod \"07fbdd79-179a-4a9a-bd69-6541c8afff44\" (UID: \"07fbdd79-179a-4a9a-bd69-6541c8afff44\") " Nov 03 01:17:49.772132 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:49.770675 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07fbdd79-179a-4a9a-bd69-6541c8afff44-plugins-config-volume" pod "07fbdd79-179a-4a9a-bd69-6541c8afff44" (UID: "07fbdd79-179a-4a9a-bd69-6541c8afff44"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:17:49.772132 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:49.771758 4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07fbdd79-179a-4a9a-bd69-6541c8afff44-plugins-config-volume\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:17:49.772132 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:49.772076 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07fbdd79-179a-4a9a-bd69-6541c8afff44-routing-overlay-volume" pod "07fbdd79-179a-4a9a-bd69-6541c8afff44" (UID: "07fbdd79-179a-4a9a-bd69-6541c8afff44"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:17:49.777241 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:17:49.777208 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf01b6791_5118_430d_9034_8f9760f9d963.slice/crio-2d0c9b5c12e857fe745ed1dc3d1a7bff850ebe6458c77d4eede9cdacb78ea400 WatchSource:0}: Error finding container 2d0c9b5c12e857fe745ed1dc3d1a7bff850ebe6458c77d4eede9cdacb78ea400: Status 404 returned error can't find the container with id 2d0c9b5c12e857fe745ed1dc3d1a7bff850ebe6458c77d4eede9cdacb78ea400 Nov 03 01:17:49.780609 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:49.780577 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37d9f0bb-k7qkd"] Nov 03 01:17:49.790582 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:49.790517 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07fbdd79-179a-4a9a-bd69-6541c8afff44-provider-credentials" pod "07fbdd79-179a-4a9a-bd69-6541c8afff44" (UID: "07fbdd79-179a-4a9a-bd69-6541c8afff44"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:17:49.790677 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:49.790625 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07fbdd79-179a-4a9a-bd69-6541c8afff44-kube-api-access-mjznm" pod "07fbdd79-179a-4a9a-bd69-6541c8afff44" (UID: "07fbdd79-179a-4a9a-bd69-6541c8afff44"). InnerVolumeSpecName "kube-api-access-mjznm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:17:49.873056 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:49.873016 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mjznm\" (UniqueName: \"kubernetes.io/projected/07fbdd79-179a-4a9a-bd69-6541c8afff44-kube-api-access-mjznm\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:17:49.873056 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:49.873054 4421 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/07fbdd79-179a-4a9a-bd69-6541c8afff44-routing-overlay-volume\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:17:49.873056 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:49.873064 4421 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/07fbdd79-179a-4a9a-bd69-6541c8afff44-provider-credentials\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:17:49.904515 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:49.904472 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37d9f0bb-k7qkd" event={"ID":"f01b6791-5118-430d-9034-8f9760f9d963","Type":"ContainerStarted","Data":"2d0c9b5c12e857fe745ed1dc3d1a7bff850ebe6458c77d4eede9cdacb78ea400"} Nov 03 01:17:49.905986 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:49.905962 4421 generic.go:356] "Generic (PLEG): container finished" podID="07fbdd79-179a-4a9a-bd69-6541c8afff44" containerID="7ca85e2986c4435a5fc7f9fe077f23a09517000bc82ccc776c8c1689f289889b" exitCode=0 Nov 03 01:17:49.906140 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:49.906013 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-576422vmz" Nov 03 01:17:49.906140 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:49.906033 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-576422vmz" event={"ID":"07fbdd79-179a-4a9a-bd69-6541c8afff44","Type":"ContainerDied","Data":"7ca85e2986c4435a5fc7f9fe077f23a09517000bc82ccc776c8c1689f289889b"} Nov 03 01:17:49.906140 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:49.906093 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-576422vmz" event={"ID":"07fbdd79-179a-4a9a-bd69-6541c8afff44","Type":"ContainerDied","Data":"a1954e33d10fa7432916bb59ef609de2c637cf5f16e932b547789701dc2bb910"} Nov 03 01:17:49.906140 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:49.906125 4421 scope.go:122] "RemoveContainer" containerID="7ca85e2986c4435a5fc7f9fe077f23a09517000bc82ccc776c8c1689f289889b" Nov 03 01:17:49.919110 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:49.919089 4421 scope.go:122] "RemoveContainer" containerID="7ca85e2986c4435a5fc7f9fe077f23a09517000bc82ccc776c8c1689f289889b" Nov 03 01:17:49.920028 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:17:49.920000 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ca85e2986c4435a5fc7f9fe077f23a09517000bc82ccc776c8c1689f289889b\": container with ID starting with 7ca85e2986c4435a5fc7f9fe077f23a09517000bc82ccc776c8c1689f289889b not found: ID does not exist" containerID="7ca85e2986c4435a5fc7f9fe077f23a09517000bc82ccc776c8c1689f289889b" Nov 03 01:17:49.920226 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:49.920201 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ca85e2986c4435a5fc7f9fe077f23a09517000bc82ccc776c8c1689f289889b"} err="failed to get container status \"7ca85e2986c4435a5fc7f9fe077f23a09517000bc82ccc776c8c1689f289889b\": rpc error: code = NotFound desc = could not find container \"7ca85e2986c4435a5fc7f9fe077f23a09517000bc82ccc776c8c1689f289889b\": container with ID starting with 7ca85e2986c4435a5fc7f9fe077f23a09517000bc82ccc776c8c1689f289889b not found: ID does not exist" Nov 03 01:17:49.932918 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:49.932883 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-576422vmz"] Nov 03 01:17:49.943914 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:49.943877 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-576422vmz"] Nov 03 01:17:49.981868 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:49.981836 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07fbdd79-179a-4a9a-bd69-6541c8afff44" path="/var/lib/kubelet/pods/07fbdd79-179a-4a9a-bd69-6541c8afff44/volumes" Nov 03 01:17:50.917065 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:50.917026 4421 generic.go:356] "Generic (PLEG): container finished" podID="f01b6791-5118-430d-9034-8f9760f9d963" containerID="e79421dc26462b4bb3132405330f374608c4a66faac68167ae47e570c3bd40bd" exitCode=0 Nov 03 01:17:50.917284 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:50.917116 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37d9f0bb-k7qkd" event={"ID":"f01b6791-5118-430d-9034-8f9760f9d963","Type":"ContainerDied","Data":"e79421dc26462b4bb3132405330f374608c4a66faac68167ae47e570c3bd40bd"} Nov 03 01:17:52.203216 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:52.203189 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37d9f0bb-k7qkd" Nov 03 01:17:52.253984 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:52.253938 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f01b6791-5118-430d-9034-8f9760f9d963-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f01b6791-5118-430d-9034-8f9760f9d963-maas-api-openshift-ca\") pod \"f01b6791-5118-430d-9034-8f9760f9d963\" (UID: \"f01b6791-5118-430d-9034-8f9760f9d963\") " Nov 03 01:17:52.254205 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:52.253996 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f01b6791-5118-430d-9034-8f9760f9d963-kube-api-access-vwvtg\" (UniqueName: \"kubernetes.io/projected/f01b6791-5118-430d-9034-8f9760f9d963-kube-api-access-vwvtg\") pod \"f01b6791-5118-430d-9034-8f9760f9d963\" (UID: \"f01b6791-5118-430d-9034-8f9760f9d963\") " Nov 03 01:17:52.254205 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:52.254077 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f01b6791-5118-430d-9034-8f9760f9d963-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f01b6791-5118-430d-9034-8f9760f9d963-maas-api-xks-ca\") pod \"f01b6791-5118-430d-9034-8f9760f9d963\" (UID: \"f01b6791-5118-430d-9034-8f9760f9d963\") " Nov 03 01:17:52.254511 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:52.254479 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f01b6791-5118-430d-9034-8f9760f9d963-maas-api-openshift-ca" pod "f01b6791-5118-430d-9034-8f9760f9d963" (UID: "f01b6791-5118-430d-9034-8f9760f9d963"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:17:52.255134 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:52.255105 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f01b6791-5118-430d-9034-8f9760f9d963-maas-api-openshift-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:17:52.256509 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:52.256480 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f01b6791-5118-430d-9034-8f9760f9d963-maas-api-xks-ca" pod "f01b6791-5118-430d-9034-8f9760f9d963" (UID: "f01b6791-5118-430d-9034-8f9760f9d963"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:17:52.256916 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:52.256877 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f01b6791-5118-430d-9034-8f9760f9d963-kube-api-access-vwvtg" pod "f01b6791-5118-430d-9034-8f9760f9d963" (UID: "f01b6791-5118-430d-9034-8f9760f9d963"). InnerVolumeSpecName "kube-api-access-vwvtg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:17:52.357106 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:52.357067 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwvtg\" (UniqueName: \"kubernetes.io/projected/f01b6791-5118-430d-9034-8f9760f9d963-kube-api-access-vwvtg\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:17:52.357106 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:52.357104 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f01b6791-5118-430d-9034-8f9760f9d963-maas-api-xks-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:17:52.937851 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:52.937815 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37d9f0bb-k7qkd" event={"ID":"f01b6791-5118-430d-9034-8f9760f9d963","Type":"ContainerDied","Data":"2d0c9b5c12e857fe745ed1dc3d1a7bff850ebe6458c77d4eede9cdacb78ea400"} Nov 03 01:17:52.938025 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:52.937868 4421 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2d0c9b5c12e857fe745ed1dc3d1a7bff850ebe6458c77d4eede9cdacb78ea400" Nov 03 01:17:52.938025 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:52.937866 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37d9f0bb-k7qkd" Nov 03 01:17:53.138507 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:53.138468 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-679574xvq"] Nov 03 01:17:58.138869 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:58.138824 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-679574xvq" podUID="a8667bca-ac8c-4f83-8d87-71c67bb3bf66" containerName="payload-processing" containerID="cri-o://573418613b9ccea8128a348f70efcca45594781ece2a4438bb9dd192a23531a2" gracePeriod=25 Nov 03 01:17:58.533766 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:58.533725 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-679574xvq" Nov 03 01:17:58.676515 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:58.673765 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8667bca-ac8c-4f83-8d87-71c67bb3bf66-provider-credentials\" (UniqueName: \"kubernetes.io/projected/a8667bca-ac8c-4f83-8d87-71c67bb3bf66-provider-credentials\") pod \"a8667bca-ac8c-4f83-8d87-71c67bb3bf66\" (UID: \"a8667bca-ac8c-4f83-8d87-71c67bb3bf66\") " Nov 03 01:17:58.676515 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:58.675786 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8667bca-ac8c-4f83-8d87-71c67bb3bf66-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8667bca-ac8c-4f83-8d87-71c67bb3bf66-plugins-config-volume\") pod \"a8667bca-ac8c-4f83-8d87-71c67bb3bf66\" (UID: \"a8667bca-ac8c-4f83-8d87-71c67bb3bf66\") " Nov 03 01:17:58.676515 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:58.675947 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8667bca-ac8c-4f83-8d87-71c67bb3bf66-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a8667bca-ac8c-4f83-8d87-71c67bb3bf66-routing-overlay-volume\") pod \"a8667bca-ac8c-4f83-8d87-71c67bb3bf66\" (UID: \"a8667bca-ac8c-4f83-8d87-71c67bb3bf66\") " Nov 03 01:17:58.676515 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:58.676045 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8667bca-ac8c-4f83-8d87-71c67bb3bf66-kube-api-access-52h8n\" (UniqueName: \"kubernetes.io/projected/a8667bca-ac8c-4f83-8d87-71c67bb3bf66-kube-api-access-52h8n\") pod \"a8667bca-ac8c-4f83-8d87-71c67bb3bf66\" (UID: \"a8667bca-ac8c-4f83-8d87-71c67bb3bf66\") " Nov 03 01:17:58.677588 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:58.677554 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8667bca-ac8c-4f83-8d87-71c67bb3bf66-plugins-config-volume" pod "a8667bca-ac8c-4f83-8d87-71c67bb3bf66" (UID: "a8667bca-ac8c-4f83-8d87-71c67bb3bf66"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:17:58.677928 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:58.677752 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8667bca-ac8c-4f83-8d87-71c67bb3bf66-routing-overlay-volume" pod "a8667bca-ac8c-4f83-8d87-71c67bb3bf66" (UID: "a8667bca-ac8c-4f83-8d87-71c67bb3bf66"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:17:58.678674 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:58.678647 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8667bca-ac8c-4f83-8d87-71c67bb3bf66-provider-credentials" pod "a8667bca-ac8c-4f83-8d87-71c67bb3bf66" (UID: "a8667bca-ac8c-4f83-8d87-71c67bb3bf66"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:17:58.679869 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:58.679842 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8667bca-ac8c-4f83-8d87-71c67bb3bf66-kube-api-access-52h8n" pod "a8667bca-ac8c-4f83-8d87-71c67bb3bf66" (UID: "a8667bca-ac8c-4f83-8d87-71c67bb3bf66"). InnerVolumeSpecName "kube-api-access-52h8n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:17:58.779024 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:58.778986 4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8667bca-ac8c-4f83-8d87-71c67bb3bf66-plugins-config-volume\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:17:58.779024 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:58.779019 4421 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a8667bca-ac8c-4f83-8d87-71c67bb3bf66-routing-overlay-volume\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:17:58.779024 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:58.779030 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52h8n\" (UniqueName: \"kubernetes.io/projected/a8667bca-ac8c-4f83-8d87-71c67bb3bf66-kube-api-access-52h8n\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:17:58.779318 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:58.779041 4421 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a8667bca-ac8c-4f83-8d87-71c67bb3bf66-provider-credentials\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:17:58.990996 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:58.990960 4421 generic.go:356] "Generic (PLEG): container finished" podID="a8667bca-ac8c-4f83-8d87-71c67bb3bf66" containerID="573418613b9ccea8128a348f70efcca45594781ece2a4438bb9dd192a23531a2" exitCode=0 Nov 03 01:17:58.991205 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:58.991009 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-679574xvq" Nov 03 01:17:58.991205 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:58.991036 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-679574xvq" event={"ID":"a8667bca-ac8c-4f83-8d87-71c67bb3bf66","Type":"ContainerDied","Data":"573418613b9ccea8128a348f70efcca45594781ece2a4438bb9dd192a23531a2"} Nov 03 01:17:58.991205 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:58.991092 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-679574xvq" event={"ID":"a8667bca-ac8c-4f83-8d87-71c67bb3bf66","Type":"ContainerDied","Data":"6368b296693656fb7e13cd582de0e1cb7afdee58da216d9a8b9c1c387d269842"} Nov 03 01:17:58.991205 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:58.991114 4421 scope.go:122] "RemoveContainer" containerID="573418613b9ccea8128a348f70efcca45594781ece2a4438bb9dd192a23531a2" Nov 03 01:17:59.002927 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:59.002910 4421 scope.go:122] "RemoveContainer" containerID="573418613b9ccea8128a348f70efcca45594781ece2a4438bb9dd192a23531a2" Nov 03 01:17:59.003232 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:17:59.003208 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"573418613b9ccea8128a348f70efcca45594781ece2a4438bb9dd192a23531a2\": container with ID starting with 573418613b9ccea8128a348f70efcca45594781ece2a4438bb9dd192a23531a2 not found: ID does not exist" containerID="573418613b9ccea8128a348f70efcca45594781ece2a4438bb9dd192a23531a2" Nov 03 01:17:59.003308 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:59.003243 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"573418613b9ccea8128a348f70efcca45594781ece2a4438bb9dd192a23531a2"} err="failed to get container status \"573418613b9ccea8128a348f70efcca45594781ece2a4438bb9dd192a23531a2\": rpc error: code = NotFound desc = could not find container \"573418613b9ccea8128a348f70efcca45594781ece2a4438bb9dd192a23531a2\": container with ID starting with 573418613b9ccea8128a348f70efcca45594781ece2a4438bb9dd192a23531a2 not found: ID does not exist" Nov 03 01:17:59.017659 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:59.017619 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-679574xvq"] Nov 03 01:17:59.022888 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:59.022857 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-679574xvq"] Nov 03 01:17:59.848077 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:59.848034 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-5d6bztwql"] Nov 03 01:17:59.981877 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:17:59.981838 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8667bca-ac8c-4f83-8d87-71c67bb3bf66" path="/var/lib/kubelet/pods/a8667bca-ac8c-4f83-8d87-71c67bb3bf66/volumes" Nov 03 01:18:00.946301 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:00.943192 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-089a375e-nbjdz"] Nov 03 01:18:00.955118 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:00.953928 4421 cpu_manager.go:414] "containerMap: removing container" podUID="f01b6791-5118-430d-9034-8f9760f9d963" containerName="revoke-keys" Nov 03 01:18:00.955118 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:00.953967 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f01b6791-5118-430d-9034-8f9760f9d963" containerName="revoke-keys" Nov 03 01:18:00.955118 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:00.954007 4421 cpu_manager.go:414] "containerMap: removing container" podUID="07fbdd79-179a-4a9a-bd69-6541c8afff44" containerName="payload-processing" Nov 03 01:18:00.955118 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:00.954016 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07fbdd79-179a-4a9a-bd69-6541c8afff44" containerName="payload-processing" Nov 03 01:18:00.955118 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:00.954043 4421 cpu_manager.go:414] "containerMap: removing container" podUID="a8667bca-ac8c-4f83-8d87-71c67bb3bf66" containerName="payload-processing" Nov 03 01:18:00.955118 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:00.954056 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8667bca-ac8c-4f83-8d87-71c67bb3bf66" containerName="payload-processing" Nov 03 01:18:00.955118 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:00.954362 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="f01b6791-5118-430d-9034-8f9760f9d963" containerName="revoke-keys" Nov 03 01:18:00.955118 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:00.954385 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="07fbdd79-179a-4a9a-bd69-6541c8afff44" containerName="payload-processing" Nov 03 01:18:00.955118 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:00.954396 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8667bca-ac8c-4f83-8d87-71c67bb3bf66" containerName="payload-processing" Nov 03 01:18:00.980484 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:00.980441 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-089a375e-nbjdz" Nov 03 01:18:00.984812 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:00.983593 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-089a375e-nbjdz"] Nov 03 01:18:00.990600 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:00.989039 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:18:00.990600 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:00.989067 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6q4xp\"" Nov 03 01:18:01.073108 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:01.073066 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzdb5\" (UniqueName: \"kubernetes.io/projected/4630e268-c1f1-4954-9e8c-f07c2dc28d6c-kube-api-access-lzdb5\") pod \"maas-api-revoke-sub-089a375e-nbjdz\" (UID: \"4630e268-c1f1-4954-9e8c-f07c2dc28d6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-089a375e-nbjdz" Nov 03 01:18:01.073537 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:01.073510 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4630e268-c1f1-4954-9e8c-f07c2dc28d6c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-089a375e-nbjdz\" (UID: \"4630e268-c1f1-4954-9e8c-f07c2dc28d6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-089a375e-nbjdz" Nov 03 01:18:01.073672 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:01.073647 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4630e268-c1f1-4954-9e8c-f07c2dc28d6c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-089a375e-nbjdz\" (UID: \"4630e268-c1f1-4954-9e8c-f07c2dc28d6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-089a375e-nbjdz" Nov 03 01:18:01.175497 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:01.175459 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzdb5\" (UniqueName: \"kubernetes.io/projected/4630e268-c1f1-4954-9e8c-f07c2dc28d6c-kube-api-access-lzdb5\") pod \"maas-api-revoke-sub-089a375e-nbjdz\" (UID: \"4630e268-c1f1-4954-9e8c-f07c2dc28d6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-089a375e-nbjdz" Nov 03 01:18:01.175704 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:01.175681 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4630e268-c1f1-4954-9e8c-f07c2dc28d6c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-089a375e-nbjdz\" (UID: \"4630e268-c1f1-4954-9e8c-f07c2dc28d6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-089a375e-nbjdz" Nov 03 01:18:01.175792 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:01.175774 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4630e268-c1f1-4954-9e8c-f07c2dc28d6c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-089a375e-nbjdz\" (UID: \"4630e268-c1f1-4954-9e8c-f07c2dc28d6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-089a375e-nbjdz" Nov 03 01:18:01.176471 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:01.176447 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4630e268-c1f1-4954-9e8c-f07c2dc28d6c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-089a375e-nbjdz\" (UID: \"4630e268-c1f1-4954-9e8c-f07c2dc28d6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-089a375e-nbjdz" Nov 03 01:18:01.178495 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:01.178473 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4630e268-c1f1-4954-9e8c-f07c2dc28d6c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-089a375e-nbjdz\" (UID: \"4630e268-c1f1-4954-9e8c-f07c2dc28d6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-089a375e-nbjdz" Nov 03 01:18:01.183297 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:01.183262 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzdb5\" (UniqueName: \"kubernetes.io/projected/4630e268-c1f1-4954-9e8c-f07c2dc28d6c-kube-api-access-lzdb5\") pod \"maas-api-revoke-sub-089a375e-nbjdz\" (UID: \"4630e268-c1f1-4954-9e8c-f07c2dc28d6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-089a375e-nbjdz" Nov 03 01:18:01.299355 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:01.299323 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-089a375e-nbjdz" Nov 03 01:18:01.650487 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:18:01.650454 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4630e268_c1f1_4954_9e8c_f07c2dc28d6c.slice/crio-6b3b178e97595567c126b99876c5a8b3ef07654775e7a3c7cea3883cb8ab8371 WatchSource:0}: Error finding container 6b3b178e97595567c126b99876c5a8b3ef07654775e7a3c7cea3883cb8ab8371: Status 404 returned error can't find the container with id 6b3b178e97595567c126b99876c5a8b3ef07654775e7a3c7cea3883cb8ab8371 Nov 03 01:18:01.650634 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:01.650612 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-089a375e-nbjdz"] Nov 03 01:18:02.322552 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:02.322508 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-089a375e-nbjdz" event={"ID":"4630e268-c1f1-4954-9e8c-f07c2dc28d6c","Type":"ContainerStarted","Data":"6b3b178e97595567c126b99876c5a8b3ef07654775e7a3c7cea3883cb8ab8371"} Nov 03 01:18:03.334406 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:03.334343 4421 generic.go:356] "Generic (PLEG): container finished" podID="4630e268-c1f1-4954-9e8c-f07c2dc28d6c" containerID="3715cc1b061d768269a6191d9d5c682fb4a7c9de3c68e8869be46de64f246228" exitCode=0 Nov 03 01:18:03.335224 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:03.334562 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-089a375e-nbjdz" event={"ID":"4630e268-c1f1-4954-9e8c-f07c2dc28d6c","Type":"ContainerDied","Data":"3715cc1b061d768269a6191d9d5c682fb4a7c9de3c68e8869be46de64f246228"} Nov 03 01:18:04.537497 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:04.537454 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-089a375e-nbjdz" Nov 03 01:18:04.612289 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:04.612184 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4630e268-c1f1-4954-9e8c-f07c2dc28d6c-kube-api-access-lzdb5\" (UniqueName: \"kubernetes.io/projected/4630e268-c1f1-4954-9e8c-f07c2dc28d6c-kube-api-access-lzdb5\") pod \"4630e268-c1f1-4954-9e8c-f07c2dc28d6c\" (UID: \"4630e268-c1f1-4954-9e8c-f07c2dc28d6c\") " Nov 03 01:18:04.612289 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:04.612268 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4630e268-c1f1-4954-9e8c-f07c2dc28d6c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4630e268-c1f1-4954-9e8c-f07c2dc28d6c-maas-api-xks-ca\") pod \"4630e268-c1f1-4954-9e8c-f07c2dc28d6c\" (UID: \"4630e268-c1f1-4954-9e8c-f07c2dc28d6c\") " Nov 03 01:18:04.612516 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:04.612431 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4630e268-c1f1-4954-9e8c-f07c2dc28d6c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4630e268-c1f1-4954-9e8c-f07c2dc28d6c-maas-api-openshift-ca\") pod \"4630e268-c1f1-4954-9e8c-f07c2dc28d6c\" (UID: \"4630e268-c1f1-4954-9e8c-f07c2dc28d6c\") " Nov 03 01:18:04.612932 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:04.612900 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4630e268-c1f1-4954-9e8c-f07c2dc28d6c-maas-api-openshift-ca" pod "4630e268-c1f1-4954-9e8c-f07c2dc28d6c" (UID: "4630e268-c1f1-4954-9e8c-f07c2dc28d6c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:18:04.613478 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:04.613448 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4630e268-c1f1-4954-9e8c-f07c2dc28d6c-maas-api-openshift-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:18:04.614672 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:04.614638 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4630e268-c1f1-4954-9e8c-f07c2dc28d6c-maas-api-xks-ca" pod "4630e268-c1f1-4954-9e8c-f07c2dc28d6c" (UID: "4630e268-c1f1-4954-9e8c-f07c2dc28d6c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:18:04.614809 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:04.614788 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4630e268-c1f1-4954-9e8c-f07c2dc28d6c-kube-api-access-lzdb5" pod "4630e268-c1f1-4954-9e8c-f07c2dc28d6c" (UID: "4630e268-c1f1-4954-9e8c-f07c2dc28d6c"). InnerVolumeSpecName "kube-api-access-lzdb5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:18:04.714743 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:04.714705 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lzdb5\" (UniqueName: \"kubernetes.io/projected/4630e268-c1f1-4954-9e8c-f07c2dc28d6c-kube-api-access-lzdb5\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:18:04.714743 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:04.714737 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4630e268-c1f1-4954-9e8c-f07c2dc28d6c-maas-api-xks-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:18:04.851292 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:04.850352 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-5d6bztwql" podUID="477f510e-003c-4491-bc5d-a3a313f6af13" containerName="payload-processing" containerID="cri-o://bed238877949d81d49d67a64b2b9529010a7418da8863010860f97645a6e3b70" gracePeriod=25 Nov 03 01:18:05.167476 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:05.167449 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-5d6bztwql" Nov 03 01:18:05.225743 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:05.225698 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/477f510e-003c-4491-bc5d-a3a313f6af13-kube-api-access-7vcb5\" (UniqueName: \"kubernetes.io/projected/477f510e-003c-4491-bc5d-a3a313f6af13-kube-api-access-7vcb5\") pod \"477f510e-003c-4491-bc5d-a3a313f6af13\" (UID: \"477f510e-003c-4491-bc5d-a3a313f6af13\") " Nov 03 01:18:05.226006 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:05.225979 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/477f510e-003c-4491-bc5d-a3a313f6af13-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/477f510e-003c-4491-bc5d-a3a313f6af13-routing-overlay-volume\") pod \"477f510e-003c-4491-bc5d-a3a313f6af13\" (UID: \"477f510e-003c-4491-bc5d-a3a313f6af13\") " Nov 03 01:18:05.226238 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:05.226217 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/477f510e-003c-4491-bc5d-a3a313f6af13-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/477f510e-003c-4491-bc5d-a3a313f6af13-plugins-config-volume\") pod \"477f510e-003c-4491-bc5d-a3a313f6af13\" (UID: \"477f510e-003c-4491-bc5d-a3a313f6af13\") " Nov 03 01:18:05.226482 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:05.226338 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/477f510e-003c-4491-bc5d-a3a313f6af13-provider-credentials\" (UniqueName: \"kubernetes.io/projected/477f510e-003c-4491-bc5d-a3a313f6af13-provider-credentials\") pod \"477f510e-003c-4491-bc5d-a3a313f6af13\" (UID: \"477f510e-003c-4491-bc5d-a3a313f6af13\") " Nov 03 01:18:05.226674 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:05.226645 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/477f510e-003c-4491-bc5d-a3a313f6af13-routing-overlay-volume" pod "477f510e-003c-4491-bc5d-a3a313f6af13" (UID: "477f510e-003c-4491-bc5d-a3a313f6af13"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:18:05.227105 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:05.227073 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/477f510e-003c-4491-bc5d-a3a313f6af13-plugins-config-volume" pod "477f510e-003c-4491-bc5d-a3a313f6af13" (UID: "477f510e-003c-4491-bc5d-a3a313f6af13"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:18:05.227563 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:05.227526 4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/477f510e-003c-4491-bc5d-a3a313f6af13-plugins-config-volume\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:18:05.227563 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:05.227560 4421 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/477f510e-003c-4491-bc5d-a3a313f6af13-routing-overlay-volume\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:18:05.228900 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:05.228867 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/477f510e-003c-4491-bc5d-a3a313f6af13-kube-api-access-7vcb5" pod "477f510e-003c-4491-bc5d-a3a313f6af13" (UID: "477f510e-003c-4491-bc5d-a3a313f6af13"). InnerVolumeSpecName "kube-api-access-7vcb5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:18:05.229014 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:05.228992 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/477f510e-003c-4491-bc5d-a3a313f6af13-provider-credentials" pod "477f510e-003c-4491-bc5d-a3a313f6af13" (UID: "477f510e-003c-4491-bc5d-a3a313f6af13"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:18:05.329764 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:05.329731 4421 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/477f510e-003c-4491-bc5d-a3a313f6af13-provider-credentials\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:18:05.329764 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:05.329761 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7vcb5\" (UniqueName: \"kubernetes.io/projected/477f510e-003c-4491-bc5d-a3a313f6af13-kube-api-access-7vcb5\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:18:05.352457 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:05.352418 4421 generic.go:356] "Generic (PLEG): container finished" podID="477f510e-003c-4491-bc5d-a3a313f6af13" containerID="bed238877949d81d49d67a64b2b9529010a7418da8863010860f97645a6e3b70" exitCode=0 Nov 03 01:18:05.352652 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:05.352483 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-5d6bztwql" Nov 03 01:18:05.352652 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:05.352468 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-5d6bztwql" event={"ID":"477f510e-003c-4491-bc5d-a3a313f6af13","Type":"ContainerDied","Data":"bed238877949d81d49d67a64b2b9529010a7418da8863010860f97645a6e3b70"} Nov 03 01:18:05.352652 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:05.352613 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-5d6bztwql" event={"ID":"477f510e-003c-4491-bc5d-a3a313f6af13","Type":"ContainerDied","Data":"15715cdd65e896f3af9bcbcdaa9e6249f523e864501d6740544ba72e8d56b2bd"} Nov 03 01:18:05.352652 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:05.352647 4421 scope.go:122] "RemoveContainer" containerID="bed238877949d81d49d67a64b2b9529010a7418da8863010860f97645a6e3b70" Nov 03 01:18:05.354678 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:05.354650 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-089a375e-nbjdz" event={"ID":"4630e268-c1f1-4954-9e8c-f07c2dc28d6c","Type":"ContainerDied","Data":"6b3b178e97595567c126b99876c5a8b3ef07654775e7a3c7cea3883cb8ab8371"} Nov 03 01:18:05.354810 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:05.354683 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-089a375e-nbjdz" Nov 03 01:18:05.354810 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:05.354710 4421 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6b3b178e97595567c126b99876c5a8b3ef07654775e7a3c7cea3883cb8ab8371" Nov 03 01:18:05.366268 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:05.366241 4421 scope.go:122] "RemoveContainer" containerID="bed238877949d81d49d67a64b2b9529010a7418da8863010860f97645a6e3b70" Nov 03 01:18:05.366702 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:18:05.366670 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bed238877949d81d49d67a64b2b9529010a7418da8863010860f97645a6e3b70\": container with ID starting with bed238877949d81d49d67a64b2b9529010a7418da8863010860f97645a6e3b70 not found: ID does not exist" containerID="bed238877949d81d49d67a64b2b9529010a7418da8863010860f97645a6e3b70" Nov 03 01:18:05.366830 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:05.366717 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bed238877949d81d49d67a64b2b9529010a7418da8863010860f97645a6e3b70"} err="failed to get container status \"bed238877949d81d49d67a64b2b9529010a7418da8863010860f97645a6e3b70\": rpc error: code = NotFound desc = could not find container \"bed238877949d81d49d67a64b2b9529010a7418da8863010860f97645a6e3b70\": container with ID starting with bed238877949d81d49d67a64b2b9529010a7418da8863010860f97645a6e3b70 not found: ID does not exist" Nov 03 01:18:05.384523 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:05.384490 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-5d6bztwql"] Nov 03 01:18:05.389195 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:05.389170 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-5d6bztwql"] Nov 03 01:18:05.982985 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:05.982899 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="477f510e-003c-4491-bc5d-a3a313f6af13" path="/var/lib/kubelet/pods/477f510e-003c-4491-bc5d-a3a313f6af13/volumes" Nov 03 01:18:11.382559 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:11.382510 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dab68fea-lhhm4"] Nov 03 01:18:11.383616 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:11.383590 4421 cpu_manager.go:414] "containerMap: removing container" podUID="477f510e-003c-4491-bc5d-a3a313f6af13" containerName="payload-processing" Nov 03 01:18:11.383616 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:11.383616 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="477f510e-003c-4491-bc5d-a3a313f6af13" containerName="payload-processing" Nov 03 01:18:11.383832 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:11.383639 4421 cpu_manager.go:414] "containerMap: removing container" podUID="4630e268-c1f1-4954-9e8c-f07c2dc28d6c" containerName="revoke-keys" Nov 03 01:18:11.383832 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:11.383647 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4630e268-c1f1-4954-9e8c-f07c2dc28d6c" containerName="revoke-keys" Nov 03 01:18:11.383939 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:11.383917 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="477f510e-003c-4491-bc5d-a3a313f6af13" containerName="payload-processing" Nov 03 01:18:11.383939 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:11.383933 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="4630e268-c1f1-4954-9e8c-f07c2dc28d6c" containerName="revoke-keys" Nov 03 01:18:11.389498 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:11.389471 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dab68fea-lhhm4" Nov 03 01:18:11.393185 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:11.393163 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:18:11.394402 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:11.394384 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6q4xp\"" Nov 03 01:18:11.395041 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:11.395011 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dab68fea-lhhm4"] Nov 03 01:18:11.455830 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:11.455788 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6e3e8f1-3867-4381-83c9-64a97998853e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dab68fea-lhhm4\" (UID: \"f6e3e8f1-3867-4381-83c9-64a97998853e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dab68fea-lhhm4" Nov 03 01:18:11.456108 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:11.456078 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbt6w\" (UniqueName: \"kubernetes.io/projected/f6e3e8f1-3867-4381-83c9-64a97998853e-kube-api-access-jbt6w\") pod \"maas-api-revoke-sub-dab68fea-lhhm4\" (UID: \"f6e3e8f1-3867-4381-83c9-64a97998853e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dab68fea-lhhm4" Nov 03 01:18:11.456339 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:11.456268 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6e3e8f1-3867-4381-83c9-64a97998853e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dab68fea-lhhm4\" (UID: \"f6e3e8f1-3867-4381-83c9-64a97998853e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dab68fea-lhhm4" Nov 03 01:18:11.571330 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:11.562371 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6e3e8f1-3867-4381-83c9-64a97998853e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dab68fea-lhhm4\" (UID: \"f6e3e8f1-3867-4381-83c9-64a97998853e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dab68fea-lhhm4" Nov 03 01:18:11.571330 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:11.562648 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbt6w\" (UniqueName: \"kubernetes.io/projected/f6e3e8f1-3867-4381-83c9-64a97998853e-kube-api-access-jbt6w\") pod \"maas-api-revoke-sub-dab68fea-lhhm4\" (UID: \"f6e3e8f1-3867-4381-83c9-64a97998853e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dab68fea-lhhm4" Nov 03 01:18:11.571330 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:11.562809 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6e3e8f1-3867-4381-83c9-64a97998853e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dab68fea-lhhm4\" (UID: \"f6e3e8f1-3867-4381-83c9-64a97998853e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dab68fea-lhhm4" Nov 03 01:18:11.571687 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:11.571384 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6e3e8f1-3867-4381-83c9-64a97998853e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dab68fea-lhhm4\" (UID: \"f6e3e8f1-3867-4381-83c9-64a97998853e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dab68fea-lhhm4" Nov 03 01:18:11.600897 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:11.590410 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6e3e8f1-3867-4381-83c9-64a97998853e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dab68fea-lhhm4\" (UID: \"f6e3e8f1-3867-4381-83c9-64a97998853e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dab68fea-lhhm4" Nov 03 01:18:11.610757 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:11.610723 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbt6w\" (UniqueName: \"kubernetes.io/projected/f6e3e8f1-3867-4381-83c9-64a97998853e-kube-api-access-jbt6w\") pod \"maas-api-revoke-sub-dab68fea-lhhm4\" (UID: \"f6e3e8f1-3867-4381-83c9-64a97998853e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dab68fea-lhhm4" Nov 03 01:18:11.709763 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:11.709675 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dab68fea-lhhm4" Nov 03 01:18:11.818893 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:11.818478 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-821f5a/payload-processing-external-model-e2e-extmod-a-821f5a-6f64zw8pg"] Nov 03 01:18:12.550298 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:12.549116 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dab68fea-lhhm4"] Nov 03 01:18:12.605241 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:12.605202 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dab68fea-lhhm4" event={"ID":"f6e3e8f1-3867-4381-83c9-64a97998853e","Type":"ContainerStarted","Data":"c34ddf371704b94cda9acd1dcbb032280921c8fb6b372a135fa835dd5bfccd44"} Nov 03 01:18:13.616695 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:13.616643 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dab68fea-lhhm4" event={"ID":"f6e3e8f1-3867-4381-83c9-64a97998853e","Type":"ContainerStarted","Data":"1701130cdd478393bbabc5d863adf88b2562a56c8bb16e67ae3cc46d2af995b3"} Nov 03 01:18:13.638595 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:13.638538 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dab68fea-lhhm4" podStartSLOduration=2.6385134040000002 podStartE2EDuration="2.638513404s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:18:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:18:13.633240847 +0000 UTC m=+2235.804243953" watchObservedRunningTime="2026-11-03 01:18:13.638513404 +0000 UTC m=+2235.809516502" Nov 03 01:18:14.629798 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:14.629758 4421 generic.go:356] "Generic (PLEG): container finished" podID="f6e3e8f1-3867-4381-83c9-64a97998853e" containerID="1701130cdd478393bbabc5d863adf88b2562a56c8bb16e67ae3cc46d2af995b3" exitCode=0 Nov 03 01:18:14.630369 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:14.629965 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dab68fea-lhhm4" event={"ID":"f6e3e8f1-3867-4381-83c9-64a97998853e","Type":"ContainerDied","Data":"1701130cdd478393bbabc5d863adf88b2562a56c8bb16e67ae3cc46d2af995b3"} Nov 03 01:18:14.675430 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:18:14.675390 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 01:18:15.868666 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:15.868639 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dab68fea-lhhm4" Nov 03 01:18:16.036253 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:16.036211 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f6e3e8f1-3867-4381-83c9-64a97998853e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6e3e8f1-3867-4381-83c9-64a97998853e-maas-api-xks-ca\") pod \"f6e3e8f1-3867-4381-83c9-64a97998853e\" (UID: \"f6e3e8f1-3867-4381-83c9-64a97998853e\") " Nov 03 01:18:16.036512 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:16.036439 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6e3e8f1-3867-4381-83c9-64a97998853e-kube-api-access-jbt6w\" (UniqueName: \"kubernetes.io/projected/f6e3e8f1-3867-4381-83c9-64a97998853e-kube-api-access-jbt6w\") pod \"f6e3e8f1-3867-4381-83c9-64a97998853e\" (UID: \"f6e3e8f1-3867-4381-83c9-64a97998853e\") " Nov 03 01:18:16.036512 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:16.036486 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6e3e8f1-3867-4381-83c9-64a97998853e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6e3e8f1-3867-4381-83c9-64a97998853e-maas-api-openshift-ca\") pod \"f6e3e8f1-3867-4381-83c9-64a97998853e\" (UID: \"f6e3e8f1-3867-4381-83c9-64a97998853e\") " Nov 03 01:18:16.036876 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:16.036845 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6e3e8f1-3867-4381-83c9-64a97998853e-maas-api-openshift-ca" pod "f6e3e8f1-3867-4381-83c9-64a97998853e" (UID: "f6e3e8f1-3867-4381-83c9-64a97998853e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:18:16.037119 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:16.037102 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6e3e8f1-3867-4381-83c9-64a97998853e-maas-api-openshift-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:18:16.038559 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:16.038521 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6e3e8f1-3867-4381-83c9-64a97998853e-maas-api-xks-ca" pod "f6e3e8f1-3867-4381-83c9-64a97998853e" (UID: "f6e3e8f1-3867-4381-83c9-64a97998853e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:18:16.038702 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:16.038680 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6e3e8f1-3867-4381-83c9-64a97998853e-kube-api-access-jbt6w" pod "f6e3e8f1-3867-4381-83c9-64a97998853e" (UID: "f6e3e8f1-3867-4381-83c9-64a97998853e"). InnerVolumeSpecName "kube-api-access-jbt6w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:18:16.139627 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:16.139597 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6e3e8f1-3867-4381-83c9-64a97998853e-maas-api-xks-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:18:16.139627 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:16.139624 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jbt6w\" (UniqueName: \"kubernetes.io/projected/f6e3e8f1-3867-4381-83c9-64a97998853e-kube-api-access-jbt6w\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:18:16.679754 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:16.679710 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dab68fea-lhhm4" event={"ID":"f6e3e8f1-3867-4381-83c9-64a97998853e","Type":"ContainerDied","Data":"c34ddf371704b94cda9acd1dcbb032280921c8fb6b372a135fa835dd5bfccd44"} Nov 03 01:18:16.679754 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:16.679754 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dab68fea-lhhm4" Nov 03 01:18:16.679979 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:16.679769 4421 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c34ddf371704b94cda9acd1dcbb032280921c8fb6b372a135fa835dd5bfccd44" Nov 03 01:18:16.819203 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:16.819139 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-821f5a/payload-processing-external-model-e2e-extmod-a-821f5a-6f64zw8pg" podUID="ad8efe73-1c3e-4086-b626-212210ac2d0d" containerName="payload-processing" containerID="cri-o://095437fd34efd6562aabc9a4c25a05dca49ea18ece22e6fef4669d419ddec627" gracePeriod=25 Nov 03 01:18:17.205953 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:17.205875 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-821f5a/payload-processing-external-model-e2e-extmod-a-821f5a-6f64zw8pg" Nov 03 01:18:17.297486 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:17.297443 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad8efe73-1c3e-4086-b626-212210ac2d0d-provider-credentials\" (UniqueName: \"kubernetes.io/projected/ad8efe73-1c3e-4086-b626-212210ac2d0d-provider-credentials\") pod \"ad8efe73-1c3e-4086-b626-212210ac2d0d\" (UID: \"ad8efe73-1c3e-4086-b626-212210ac2d0d\") " Nov 03 01:18:17.297730 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:17.297704 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad8efe73-1c3e-4086-b626-212210ac2d0d-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ad8efe73-1c3e-4086-b626-212210ac2d0d-routing-overlay-volume\") pod \"ad8efe73-1c3e-4086-b626-212210ac2d0d\" (UID: \"ad8efe73-1c3e-4086-b626-212210ac2d0d\") " Nov 03 01:18:17.297823 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:17.297773 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad8efe73-1c3e-4086-b626-212210ac2d0d-kube-api-access-msh9d\" (UniqueName: \"kubernetes.io/projected/ad8efe73-1c3e-4086-b626-212210ac2d0d-kube-api-access-msh9d\") pod \"ad8efe73-1c3e-4086-b626-212210ac2d0d\" (UID: \"ad8efe73-1c3e-4086-b626-212210ac2d0d\") " Nov 03 01:18:17.298086 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:17.298060 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad8efe73-1c3e-4086-b626-212210ac2d0d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad8efe73-1c3e-4086-b626-212210ac2d0d-plugins-config-volume\") pod \"ad8efe73-1c3e-4086-b626-212210ac2d0d\" (UID: \"ad8efe73-1c3e-4086-b626-212210ac2d0d\") " Nov 03 01:18:17.298221 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:17.298168 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad8efe73-1c3e-4086-b626-212210ac2d0d-routing-overlay-volume" pod "ad8efe73-1c3e-4086-b626-212210ac2d0d" (UID: "ad8efe73-1c3e-4086-b626-212210ac2d0d"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:18:17.298477 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:17.298453 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad8efe73-1c3e-4086-b626-212210ac2d0d-plugins-config-volume" pod "ad8efe73-1c3e-4086-b626-212210ac2d0d" (UID: "ad8efe73-1c3e-4086-b626-212210ac2d0d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:18:17.298831 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:17.298815 4421 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ad8efe73-1c3e-4086-b626-212210ac2d0d-routing-overlay-volume\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:18:17.298876 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:17.298834 4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad8efe73-1c3e-4086-b626-212210ac2d0d-plugins-config-volume\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:18:17.300046 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:17.300024 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad8efe73-1c3e-4086-b626-212210ac2d0d-provider-credentials" pod "ad8efe73-1c3e-4086-b626-212210ac2d0d" (UID: "ad8efe73-1c3e-4086-b626-212210ac2d0d"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:18:17.300123 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:17.300057 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad8efe73-1c3e-4086-b626-212210ac2d0d-kube-api-access-msh9d" pod "ad8efe73-1c3e-4086-b626-212210ac2d0d" (UID: "ad8efe73-1c3e-4086-b626-212210ac2d0d"). InnerVolumeSpecName "kube-api-access-msh9d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:18:17.400064 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:17.400027 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-msh9d\" (UniqueName: \"kubernetes.io/projected/ad8efe73-1c3e-4086-b626-212210ac2d0d-kube-api-access-msh9d\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:18:17.400064 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:17.400059 4421 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ad8efe73-1c3e-4086-b626-212210ac2d0d-provider-credentials\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:18:17.689490 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:17.689455 4421 generic.go:356] "Generic (PLEG): container finished" podID="ad8efe73-1c3e-4086-b626-212210ac2d0d" containerID="095437fd34efd6562aabc9a4c25a05dca49ea18ece22e6fef4669d419ddec627" exitCode=0 Nov 03 01:18:17.689714 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:17.689509 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-821f5a/payload-processing-external-model-e2e-extmod-a-821f5a-6f64zw8pg" Nov 03 01:18:17.689714 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:17.689516 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-821f5a/payload-processing-external-model-e2e-extmod-a-821f5a-6f64zw8pg" event={"ID":"ad8efe73-1c3e-4086-b626-212210ac2d0d","Type":"ContainerDied","Data":"095437fd34efd6562aabc9a4c25a05dca49ea18ece22e6fef4669d419ddec627"} Nov 03 01:18:17.689714 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:17.689630 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-821f5a/payload-processing-external-model-e2e-extmod-a-821f5a-6f64zw8pg" event={"ID":"ad8efe73-1c3e-4086-b626-212210ac2d0d","Type":"ContainerDied","Data":"6ec60dfb51ef1e1336b94ac806f7999cf9c090e9f45c5170d568c39737c482c7"} Nov 03 01:18:17.689714 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:17.689651 4421 scope.go:122] "RemoveContainer" containerID="095437fd34efd6562aabc9a4c25a05dca49ea18ece22e6fef4669d419ddec627" Nov 03 01:18:17.702741 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:17.702716 4421 scope.go:122] "RemoveContainer" containerID="095437fd34efd6562aabc9a4c25a05dca49ea18ece22e6fef4669d419ddec627" Nov 03 01:18:17.703036 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:18:17.703017 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"095437fd34efd6562aabc9a4c25a05dca49ea18ece22e6fef4669d419ddec627\": container with ID starting with 095437fd34efd6562aabc9a4c25a05dca49ea18ece22e6fef4669d419ddec627 not found: ID does not exist" containerID="095437fd34efd6562aabc9a4c25a05dca49ea18ece22e6fef4669d419ddec627" Nov 03 01:18:17.703079 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:17.703049 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"095437fd34efd6562aabc9a4c25a05dca49ea18ece22e6fef4669d419ddec627"} err="failed to get container status \"095437fd34efd6562aabc9a4c25a05dca49ea18ece22e6fef4669d419ddec627\": rpc error: code = NotFound desc = could not find container \"095437fd34efd6562aabc9a4c25a05dca49ea18ece22e6fef4669d419ddec627\": container with ID starting with 095437fd34efd6562aabc9a4c25a05dca49ea18ece22e6fef4669d419ddec627 not found: ID does not exist" Nov 03 01:18:17.716543 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:17.716516 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-821f5a/payload-processing-external-model-e2e-extmod-a-821f5a-6f64zw8pg"] Nov 03 01:18:17.721087 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:17.721063 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-821f5a/payload-processing-external-model-e2e-extmod-a-821f5a-6f64zw8pg"] Nov 03 01:18:17.983028 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:17.982943 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad8efe73-1c3e-4086-b626-212210ac2d0d" path="/var/lib/kubelet/pods/ad8efe73-1c3e-4086-b626-212210ac2d0d/volumes" Nov 03 01:18:23.582619 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:23.582582 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7484bf8-6tzhn"] Nov 03 01:18:23.583640 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:23.583615 4421 cpu_manager.go:414] "containerMap: removing container" podUID="f6e3e8f1-3867-4381-83c9-64a97998853e" containerName="revoke-keys" Nov 03 01:18:23.583640 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:23.583639 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6e3e8f1-3867-4381-83c9-64a97998853e" containerName="revoke-keys" Nov 03 01:18:23.583909 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:23.583738 4421 cpu_manager.go:414] "containerMap: removing container" podUID="ad8efe73-1c3e-4086-b626-212210ac2d0d" containerName="payload-processing" Nov 03 01:18:23.583909 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:23.583747 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad8efe73-1c3e-4086-b626-212210ac2d0d" containerName="payload-processing" Nov 03 01:18:23.584031 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:23.583944 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad8efe73-1c3e-4086-b626-212210ac2d0d" containerName="payload-processing" Nov 03 01:18:23.584031 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:23.583963 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6e3e8f1-3867-4381-83c9-64a97998853e" containerName="revoke-keys" Nov 03 01:18:23.587921 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:23.587896 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7484bf8-6tzhn" Nov 03 01:18:23.591256 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:23.591232 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:18:23.592577 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:23.592556 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6q4xp\"" Nov 03 01:18:23.594443 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:23.594422 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7484bf8-6tzhn"] Nov 03 01:18:23.653380 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:23.653338 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b9a0bcb-5c16-4177-a05d-cd5faf208925-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b7484bf8-6tzhn\" (UID: \"7b9a0bcb-5c16-4177-a05d-cd5faf208925\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7484bf8-6tzhn" Nov 03 01:18:23.653740 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:23.653713 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56fkq\" (UniqueName: \"kubernetes.io/projected/7b9a0bcb-5c16-4177-a05d-cd5faf208925-kube-api-access-56fkq\") pod \"maas-api-revoke-sub-b7484bf8-6tzhn\" (UID: \"7b9a0bcb-5c16-4177-a05d-cd5faf208925\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7484bf8-6tzhn" Nov 03 01:18:23.653801 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:23.653783 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b9a0bcb-5c16-4177-a05d-cd5faf208925-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b7484bf8-6tzhn\" (UID: \"7b9a0bcb-5c16-4177-a05d-cd5faf208925\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7484bf8-6tzhn" Nov 03 01:18:23.755189 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:23.755139 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56fkq\" (UniqueName: \"kubernetes.io/projected/7b9a0bcb-5c16-4177-a05d-cd5faf208925-kube-api-access-56fkq\") pod \"maas-api-revoke-sub-b7484bf8-6tzhn\" (UID: \"7b9a0bcb-5c16-4177-a05d-cd5faf208925\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7484bf8-6tzhn" Nov 03 01:18:23.755386 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:23.755207 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b9a0bcb-5c16-4177-a05d-cd5faf208925-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b7484bf8-6tzhn\" (UID: \"7b9a0bcb-5c16-4177-a05d-cd5faf208925\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7484bf8-6tzhn" Nov 03 01:18:23.755386 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:23.755352 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b9a0bcb-5c16-4177-a05d-cd5faf208925-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b7484bf8-6tzhn\" (UID: \"7b9a0bcb-5c16-4177-a05d-cd5faf208925\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7484bf8-6tzhn" Nov 03 01:18:23.755937 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:23.755910 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b9a0bcb-5c16-4177-a05d-cd5faf208925-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b7484bf8-6tzhn\" (UID: \"7b9a0bcb-5c16-4177-a05d-cd5faf208925\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7484bf8-6tzhn" Nov 03 01:18:23.757840 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:23.757815 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b9a0bcb-5c16-4177-a05d-cd5faf208925-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b7484bf8-6tzhn\" (UID: \"7b9a0bcb-5c16-4177-a05d-cd5faf208925\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7484bf8-6tzhn" Nov 03 01:18:23.767833 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:23.767809 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56fkq\" (UniqueName: \"kubernetes.io/projected/7b9a0bcb-5c16-4177-a05d-cd5faf208925-kube-api-access-56fkq\") pod \"maas-api-revoke-sub-b7484bf8-6tzhn\" (UID: \"7b9a0bcb-5c16-4177-a05d-cd5faf208925\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7484bf8-6tzhn" Nov 03 01:18:23.913398 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:23.912007 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7484bf8-6tzhn" Nov 03 01:18:24.227946 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:18:24.227908 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b9a0bcb_5c16_4177_a05d_cd5faf208925.slice/crio-110f81c544deb1ec8781255c391c6bd00520e09718d527d597ba8c30a5afec04 WatchSource:0}: Error finding container 110f81c544deb1ec8781255c391c6bd00520e09718d527d597ba8c30a5afec04: Status 404 returned error can't find the container with id 110f81c544deb1ec8781255c391c6bd00520e09718d527d597ba8c30a5afec04 Nov 03 01:18:24.228590 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:24.228555 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7484bf8-6tzhn"] Nov 03 01:18:24.792620 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:24.792576 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7484bf8-6tzhn" event={"ID":"7b9a0bcb-5c16-4177-a05d-cd5faf208925","Type":"ContainerStarted","Data":"516ee361c55d3d360f60cd1e3eff9d62d54f9712257b89a59f61da78abb8efb7"} Nov 03 01:18:24.793100 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:24.792636 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7484bf8-6tzhn" event={"ID":"7b9a0bcb-5c16-4177-a05d-cd5faf208925","Type":"ContainerStarted","Data":"110f81c544deb1ec8781255c391c6bd00520e09718d527d597ba8c30a5afec04"} Nov 03 01:18:24.813157 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:24.813101 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7484bf8-6tzhn" podStartSLOduration=1.813085939 podStartE2EDuration="1.813085939s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:18:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:18:24.808045761 +0000 UTC m=+2246.979048857" watchObservedRunningTime="2026-11-03 01:18:24.813085939 +0000 UTC m=+2246.984089035" Nov 03 01:18:25.288495 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:18:25.288454 4421 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b9a0bcb_5c16_4177_a05d_cd5faf208925.slice/crio-516ee361c55d3d360f60cd1e3eff9d62d54f9712257b89a59f61da78abb8efb7.scope\": RecentStats: unable to find data in memory cache]" Nov 03 01:18:25.803456 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:25.803423 4421 generic.go:356] "Generic (PLEG): container finished" podID="7b9a0bcb-5c16-4177-a05d-cd5faf208925" containerID="516ee361c55d3d360f60cd1e3eff9d62d54f9712257b89a59f61da78abb8efb7" exitCode=0 Nov 03 01:18:25.803890 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:25.803502 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7484bf8-6tzhn" event={"ID":"7b9a0bcb-5c16-4177-a05d-cd5faf208925","Type":"ContainerDied","Data":"516ee361c55d3d360f60cd1e3eff9d62d54f9712257b89a59f61da78abb8efb7"} Nov 03 01:18:27.001307 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:27.001263 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7484bf8-6tzhn" Nov 03 01:18:27.151436 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:27.142436 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b9a0bcb-5c16-4177-a05d-cd5faf208925-kube-api-access-56fkq\" (UniqueName: \"kubernetes.io/projected/7b9a0bcb-5c16-4177-a05d-cd5faf208925-kube-api-access-56fkq\") pod \"7b9a0bcb-5c16-4177-a05d-cd5faf208925\" (UID: \"7b9a0bcb-5c16-4177-a05d-cd5faf208925\") " Nov 03 01:18:27.151883 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:27.151854 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b9a0bcb-5c16-4177-a05d-cd5faf208925-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b9a0bcb-5c16-4177-a05d-cd5faf208925-maas-api-openshift-ca\") pod \"7b9a0bcb-5c16-4177-a05d-cd5faf208925\" (UID: \"7b9a0bcb-5c16-4177-a05d-cd5faf208925\") " Nov 03 01:18:27.160395 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:27.152198 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7b9a0bcb-5c16-4177-a05d-cd5faf208925-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b9a0bcb-5c16-4177-a05d-cd5faf208925-maas-api-xks-ca\") pod \"7b9a0bcb-5c16-4177-a05d-cd5faf208925\" (UID: \"7b9a0bcb-5c16-4177-a05d-cd5faf208925\") " Nov 03 01:18:27.192810 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:27.153916 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b9a0bcb-5c16-4177-a05d-cd5faf208925-maas-api-openshift-ca" pod "7b9a0bcb-5c16-4177-a05d-cd5faf208925" (UID: "7b9a0bcb-5c16-4177-a05d-cd5faf208925"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:18:27.192810 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:27.189210 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b9a0bcb-5c16-4177-a05d-cd5faf208925-kube-api-access-56fkq" pod "7b9a0bcb-5c16-4177-a05d-cd5faf208925" (UID: "7b9a0bcb-5c16-4177-a05d-cd5faf208925"). InnerVolumeSpecName "kube-api-access-56fkq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:18:27.192810 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:27.189301 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b9a0bcb-5c16-4177-a05d-cd5faf208925-maas-api-xks-ca" pod "7b9a0bcb-5c16-4177-a05d-cd5faf208925" (UID: "7b9a0bcb-5c16-4177-a05d-cd5faf208925"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:18:27.272046 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:27.272011 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b9a0bcb-5c16-4177-a05d-cd5faf208925-maas-api-xks-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:18:27.272199 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:27.272058 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56fkq\" (UniqueName: \"kubernetes.io/projected/7b9a0bcb-5c16-4177-a05d-cd5faf208925-kube-api-access-56fkq\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:18:27.272199 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:27.272070 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b9a0bcb-5c16-4177-a05d-cd5faf208925-maas-api-openshift-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:18:27.822419 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:27.822385 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7484bf8-6tzhn" Nov 03 01:18:27.822629 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:27.822405 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7484bf8-6tzhn" event={"ID":"7b9a0bcb-5c16-4177-a05d-cd5faf208925","Type":"ContainerDied","Data":"110f81c544deb1ec8781255c391c6bd00520e09718d527d597ba8c30a5afec04"} Nov 03 01:18:27.822629 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:27.822489 4421 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="110f81c544deb1ec8781255c391c6bd00520e09718d527d597ba8c30a5afec04" Nov 03 01:18:44.719397 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:18:44.719354 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 01:18:55.493682 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:55.493649 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8602411c-pnhkh"] Nov 03 01:18:55.494555 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:55.494533 4421 cpu_manager.go:414] "containerMap: removing container" podUID="7b9a0bcb-5c16-4177-a05d-cd5faf208925" containerName="revoke-keys" Nov 03 01:18:55.494637 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:55.494556 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b9a0bcb-5c16-4177-a05d-cd5faf208925" containerName="revoke-keys" Nov 03 01:18:55.494762 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:55.494750 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b9a0bcb-5c16-4177-a05d-cd5faf208925" containerName="revoke-keys" Nov 03 01:18:55.499206 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:55.499180 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8602411c-pnhkh" Nov 03 01:18:55.504040 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:55.504014 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:18:55.504984 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:55.504964 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6q4xp\"" Nov 03 01:18:55.507437 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:55.507413 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8602411c-pnhkh"] Nov 03 01:18:55.577706 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:55.577661 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdsr9\" (UniqueName: \"kubernetes.io/projected/a113b7da-2cf3-415a-b6cd-6f38a8bd988f-kube-api-access-xdsr9\") pod \"maas-api-revoke-sub-8602411c-pnhkh\" (UID: \"a113b7da-2cf3-415a-b6cd-6f38a8bd988f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8602411c-pnhkh" Nov 03 01:18:55.577921 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:55.577801 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a113b7da-2cf3-415a-b6cd-6f38a8bd988f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8602411c-pnhkh\" (UID: \"a113b7da-2cf3-415a-b6cd-6f38a8bd988f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8602411c-pnhkh" Nov 03 01:18:55.577921 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:55.577846 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a113b7da-2cf3-415a-b6cd-6f38a8bd988f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8602411c-pnhkh\" (UID: \"a113b7da-2cf3-415a-b6cd-6f38a8bd988f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8602411c-pnhkh" Nov 03 01:18:55.679731 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:55.679688 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdsr9\" (UniqueName: \"kubernetes.io/projected/a113b7da-2cf3-415a-b6cd-6f38a8bd988f-kube-api-access-xdsr9\") pod \"maas-api-revoke-sub-8602411c-pnhkh\" (UID: \"a113b7da-2cf3-415a-b6cd-6f38a8bd988f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8602411c-pnhkh" Nov 03 01:18:55.679934 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:55.679823 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a113b7da-2cf3-415a-b6cd-6f38a8bd988f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8602411c-pnhkh\" (UID: \"a113b7da-2cf3-415a-b6cd-6f38a8bd988f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8602411c-pnhkh" Nov 03 01:18:55.679934 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:55.679869 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a113b7da-2cf3-415a-b6cd-6f38a8bd988f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8602411c-pnhkh\" (UID: \"a113b7da-2cf3-415a-b6cd-6f38a8bd988f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8602411c-pnhkh" Nov 03 01:18:55.680856 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:55.680824 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a113b7da-2cf3-415a-b6cd-6f38a8bd988f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8602411c-pnhkh\" (UID: \"a113b7da-2cf3-415a-b6cd-6f38a8bd988f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8602411c-pnhkh" Nov 03 01:18:55.682550 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:55.682510 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a113b7da-2cf3-415a-b6cd-6f38a8bd988f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8602411c-pnhkh\" (UID: \"a113b7da-2cf3-415a-b6cd-6f38a8bd988f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8602411c-pnhkh" Nov 03 01:18:55.687914 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:55.687883 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdsr9\" (UniqueName: \"kubernetes.io/projected/a113b7da-2cf3-415a-b6cd-6f38a8bd988f-kube-api-access-xdsr9\") pod \"maas-api-revoke-sub-8602411c-pnhkh\" (UID: \"a113b7da-2cf3-415a-b6cd-6f38a8bd988f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8602411c-pnhkh" Nov 03 01:18:55.816122 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:55.816089 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8602411c-pnhkh" Nov 03 01:18:56.099101 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:18:56.099070 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda113b7da_2cf3_415a_b6cd_6f38a8bd988f.slice/crio-af34313c058fa6bb18636d602d58542af3573ec66b236172d9120efe0fc2f4e5 WatchSource:0}: Error finding container af34313c058fa6bb18636d602d58542af3573ec66b236172d9120efe0fc2f4e5: Status 404 returned error can't find the container with id af34313c058fa6bb18636d602d58542af3573ec66b236172d9120efe0fc2f4e5 Nov 03 01:18:56.099371 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:56.099344 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8602411c-pnhkh"] Nov 03 01:18:56.757578 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:56.757535 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8602411c-pnhkh" event={"ID":"a113b7da-2cf3-415a-b6cd-6f38a8bd988f","Type":"ContainerStarted","Data":"b3c6f2ab19a1c51fe26d1906df75c6cbf8e026a06a27808d89b40efe84bcca3e"} Nov 03 01:18:56.758067 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:56.757599 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8602411c-pnhkh" event={"ID":"a113b7da-2cf3-415a-b6cd-6f38a8bd988f","Type":"ContainerStarted","Data":"af34313c058fa6bb18636d602d58542af3573ec66b236172d9120efe0fc2f4e5"} Nov 03 01:18:56.775426 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:56.775371 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8602411c-pnhkh" podStartSLOduration=1.775355995 podStartE2EDuration="1.775355995s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:18:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:18:56.773526247 +0000 UTC m=+2278.944529346" watchObservedRunningTime="2026-11-03 01:18:56.775355995 +0000 UTC m=+2278.946359127" Nov 03 01:18:57.778563 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:57.778411 4421 generic.go:356] "Generic (PLEG): container finished" podID="a113b7da-2cf3-415a-b6cd-6f38a8bd988f" containerID="b3c6f2ab19a1c51fe26d1906df75c6cbf8e026a06a27808d89b40efe84bcca3e" exitCode=0 Nov 03 01:18:57.779666 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:57.779626 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8602411c-pnhkh" event={"ID":"a113b7da-2cf3-415a-b6cd-6f38a8bd988f","Type":"ContainerDied","Data":"b3c6f2ab19a1c51fe26d1906df75c6cbf8e026a06a27808d89b40efe84bcca3e"} Nov 03 01:18:58.964576 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:58.964545 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8602411c-pnhkh" Nov 03 01:18:59.061307 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:59.061169 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a113b7da-2cf3-415a-b6cd-6f38a8bd988f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a113b7da-2cf3-415a-b6cd-6f38a8bd988f-maas-api-openshift-ca\") pod \"a113b7da-2cf3-415a-b6cd-6f38a8bd988f\" (UID: \"a113b7da-2cf3-415a-b6cd-6f38a8bd988f\") " Nov 03 01:18:59.061570 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:59.061543 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a113b7da-2cf3-415a-b6cd-6f38a8bd988f-maas-api-openshift-ca" pod "a113b7da-2cf3-415a-b6cd-6f38a8bd988f" (UID: "a113b7da-2cf3-415a-b6cd-6f38a8bd988f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:18:59.061667 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:59.061572 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a113b7da-2cf3-415a-b6cd-6f38a8bd988f-kube-api-access-xdsr9\" (UniqueName: \"kubernetes.io/projected/a113b7da-2cf3-415a-b6cd-6f38a8bd988f-kube-api-access-xdsr9\") pod \"a113b7da-2cf3-415a-b6cd-6f38a8bd988f\" (UID: \"a113b7da-2cf3-415a-b6cd-6f38a8bd988f\") " Nov 03 01:18:59.061765 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:59.061745 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a113b7da-2cf3-415a-b6cd-6f38a8bd988f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a113b7da-2cf3-415a-b6cd-6f38a8bd988f-maas-api-xks-ca\") pod \"a113b7da-2cf3-415a-b6cd-6f38a8bd988f\" (UID: \"a113b7da-2cf3-415a-b6cd-6f38a8bd988f\") " Nov 03 01:18:59.062867 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:59.062841 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a113b7da-2cf3-415a-b6cd-6f38a8bd988f-maas-api-openshift-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:18:59.064167 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:59.064144 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a113b7da-2cf3-415a-b6cd-6f38a8bd988f-maas-api-xks-ca" pod "a113b7da-2cf3-415a-b6cd-6f38a8bd988f" (UID: "a113b7da-2cf3-415a-b6cd-6f38a8bd988f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:18:59.064362 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:59.064342 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a113b7da-2cf3-415a-b6cd-6f38a8bd988f-kube-api-access-xdsr9" pod "a113b7da-2cf3-415a-b6cd-6f38a8bd988f" (UID: "a113b7da-2cf3-415a-b6cd-6f38a8bd988f"). InnerVolumeSpecName "kube-api-access-xdsr9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:18:59.187305 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:59.184433 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdsr9\" (UniqueName: \"kubernetes.io/projected/a113b7da-2cf3-415a-b6cd-6f38a8bd988f-kube-api-access-xdsr9\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:18:59.187305 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:59.184469 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a113b7da-2cf3-415a-b6cd-6f38a8bd988f-maas-api-xks-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:18:59.850298 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:59.838799 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8602411c-pnhkh" event={"ID":"a113b7da-2cf3-415a-b6cd-6f38a8bd988f","Type":"ContainerDied","Data":"af34313c058fa6bb18636d602d58542af3573ec66b236172d9120efe0fc2f4e5"} Nov 03 01:18:59.850298 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:59.838900 4421 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="af34313c058fa6bb18636d602d58542af3573ec66b236172d9120efe0fc2f4e5" Nov 03 01:18:59.850298 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:18:59.838995 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8602411c-pnhkh" Nov 03 01:19:07.563549 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:07.563507 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af085979-vgdgv"] Nov 03 01:19:07.564621 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:07.564593 4421 cpu_manager.go:414] "containerMap: removing container" podUID="a113b7da-2cf3-415a-b6cd-6f38a8bd988f" containerName="revoke-keys" Nov 03 01:19:07.564621 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:07.564621 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a113b7da-2cf3-415a-b6cd-6f38a8bd988f" containerName="revoke-keys" Nov 03 01:19:07.564935 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:07.564919 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="a113b7da-2cf3-415a-b6cd-6f38a8bd988f" containerName="revoke-keys" Nov 03 01:19:07.570579 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:07.570540 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af085979-vgdgv" Nov 03 01:19:07.574686 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:07.574654 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af085979-vgdgv"] Nov 03 01:19:07.574906 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:07.574734 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:19:07.575521 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:07.575499 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6q4xp\"" Nov 03 01:19:07.684499 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:07.684457 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e89093f3-1b11-4b5c-928b-017480d620b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-af085979-vgdgv\" (UID: \"e89093f3-1b11-4b5c-928b-017480d620b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af085979-vgdgv" Nov 03 01:19:07.684778 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:07.684748 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e89093f3-1b11-4b5c-928b-017480d620b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-af085979-vgdgv\" (UID: \"e89093f3-1b11-4b5c-928b-017480d620b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af085979-vgdgv" Nov 03 01:19:07.684997 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:07.684969 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9czf6\" (UniqueName: \"kubernetes.io/projected/e89093f3-1b11-4b5c-928b-017480d620b0-kube-api-access-9czf6\") pod \"maas-api-revoke-sub-af085979-vgdgv\" (UID: \"e89093f3-1b11-4b5c-928b-017480d620b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af085979-vgdgv" Nov 03 01:19:07.786565 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:07.786526 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e89093f3-1b11-4b5c-928b-017480d620b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-af085979-vgdgv\" (UID: \"e89093f3-1b11-4b5c-928b-017480d620b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af085979-vgdgv" Nov 03 01:19:07.786763 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:07.786672 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e89093f3-1b11-4b5c-928b-017480d620b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-af085979-vgdgv\" (UID: \"e89093f3-1b11-4b5c-928b-017480d620b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af085979-vgdgv" Nov 03 01:19:07.786809 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:07.786766 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9czf6\" (UniqueName: \"kubernetes.io/projected/e89093f3-1b11-4b5c-928b-017480d620b0-kube-api-access-9czf6\") pod \"maas-api-revoke-sub-af085979-vgdgv\" (UID: \"e89093f3-1b11-4b5c-928b-017480d620b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af085979-vgdgv" Nov 03 01:19:07.787228 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:07.787203 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e89093f3-1b11-4b5c-928b-017480d620b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-af085979-vgdgv\" (UID: \"e89093f3-1b11-4b5c-928b-017480d620b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af085979-vgdgv" Nov 03 01:19:07.789195 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:07.789152 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e89093f3-1b11-4b5c-928b-017480d620b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-af085979-vgdgv\" (UID: \"e89093f3-1b11-4b5c-928b-017480d620b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af085979-vgdgv" Nov 03 01:19:07.794811 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:07.794783 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9czf6\" (UniqueName: \"kubernetes.io/projected/e89093f3-1b11-4b5c-928b-017480d620b0-kube-api-access-9czf6\") pod \"maas-api-revoke-sub-af085979-vgdgv\" (UID: \"e89093f3-1b11-4b5c-928b-017480d620b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af085979-vgdgv" Nov 03 01:19:07.891755 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:07.891660 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af085979-vgdgv" Nov 03 01:19:08.201074 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:19:08.201035 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode89093f3_1b11_4b5c_928b_017480d620b0.slice/crio-84e9f224267675d5cc519ea8631eca36626a69f9e779362d33feba21f87ad9c7 WatchSource:0}: Error finding container 84e9f224267675d5cc519ea8631eca36626a69f9e779362d33feba21f87ad9c7: Status 404 returned error can't find the container with id 84e9f224267675d5cc519ea8631eca36626a69f9e779362d33feba21f87ad9c7 Nov 03 01:19:08.201474 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:08.201449 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af085979-vgdgv"] Nov 03 01:19:09.002745 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:09.002704 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af085979-vgdgv" event={"ID":"e89093f3-1b11-4b5c-928b-017480d620b0","Type":"ContainerStarted","Data":"928b6c0cc2980667c0d140643d205f2fcdfea528349d3984a2c62d3b0c8dc1f9"} Nov 03 01:19:09.003154 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:09.002760 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af085979-vgdgv" event={"ID":"e89093f3-1b11-4b5c-928b-017480d620b0","Type":"ContainerStarted","Data":"84e9f224267675d5cc519ea8631eca36626a69f9e779362d33feba21f87ad9c7"} Nov 03 01:19:09.019808 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:09.019756 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af085979-vgdgv" podStartSLOduration=2.019742238 podStartE2EDuration="2.019742238s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:19:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:19:09.017899531 +0000 UTC m=+2291.188902629" watchObservedRunningTime="2026-11-03 01:19:09.019742238 +0000 UTC m=+2291.190745390" Nov 03 01:19:10.013155 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:10.013117 4421 generic.go:356] "Generic (PLEG): container finished" podID="e89093f3-1b11-4b5c-928b-017480d620b0" containerID="928b6c0cc2980667c0d140643d205f2fcdfea528349d3984a2c62d3b0c8dc1f9" exitCode=0 Nov 03 01:19:10.013598 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:10.013182 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af085979-vgdgv" event={"ID":"e89093f3-1b11-4b5c-928b-017480d620b0","Type":"ContainerDied","Data":"928b6c0cc2980667c0d140643d205f2fcdfea528349d3984a2c62d3b0c8dc1f9"} Nov 03 01:19:11.203527 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:11.203497 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af085979-vgdgv" Nov 03 01:19:11.263620 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:11.263578 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e89093f3-1b11-4b5c-928b-017480d620b0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e89093f3-1b11-4b5c-928b-017480d620b0-maas-api-openshift-ca\") pod \"e89093f3-1b11-4b5c-928b-017480d620b0\" (UID: \"e89093f3-1b11-4b5c-928b-017480d620b0\") " Nov 03 01:19:11.263835 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:11.263738 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e89093f3-1b11-4b5c-928b-017480d620b0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e89093f3-1b11-4b5c-928b-017480d620b0-maas-api-xks-ca\") pod \"e89093f3-1b11-4b5c-928b-017480d620b0\" (UID: \"e89093f3-1b11-4b5c-928b-017480d620b0\") " Nov 03 01:19:11.264006 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:11.263985 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e89093f3-1b11-4b5c-928b-017480d620b0-kube-api-access-9czf6\" (UniqueName: \"kubernetes.io/projected/e89093f3-1b11-4b5c-928b-017480d620b0-kube-api-access-9czf6\") pod \"e89093f3-1b11-4b5c-928b-017480d620b0\" (UID: \"e89093f3-1b11-4b5c-928b-017480d620b0\") " Nov 03 01:19:11.264099 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:11.264031 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e89093f3-1b11-4b5c-928b-017480d620b0-maas-api-openshift-ca" pod "e89093f3-1b11-4b5c-928b-017480d620b0" (UID: "e89093f3-1b11-4b5c-928b-017480d620b0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:19:11.264621 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:11.264597 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e89093f3-1b11-4b5c-928b-017480d620b0-maas-api-openshift-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:19:11.266684 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:11.266656 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e89093f3-1b11-4b5c-928b-017480d620b0-maas-api-xks-ca" pod "e89093f3-1b11-4b5c-928b-017480d620b0" (UID: "e89093f3-1b11-4b5c-928b-017480d620b0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:19:11.266837 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:11.266799 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e89093f3-1b11-4b5c-928b-017480d620b0-kube-api-access-9czf6" pod "e89093f3-1b11-4b5c-928b-017480d620b0" (UID: "e89093f3-1b11-4b5c-928b-017480d620b0"). InnerVolumeSpecName "kube-api-access-9czf6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:19:11.366384 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:11.366347 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9czf6\" (UniqueName: \"kubernetes.io/projected/e89093f3-1b11-4b5c-928b-017480d620b0-kube-api-access-9czf6\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:19:11.366384 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:11.366378 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e89093f3-1b11-4b5c-928b-017480d620b0-maas-api-xks-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:19:12.030848 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:12.030802 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af085979-vgdgv" event={"ID":"e89093f3-1b11-4b5c-928b-017480d620b0","Type":"ContainerDied","Data":"84e9f224267675d5cc519ea8631eca36626a69f9e779362d33feba21f87ad9c7"} Nov 03 01:19:12.031029 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:12.030868 4421 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="84e9f224267675d5cc519ea8631eca36626a69f9e779362d33feba21f87ad9c7" Nov 03 01:19:12.031029 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:12.030821 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af085979-vgdgv" Nov 03 01:19:14.672984 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:19:14.672939 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 01:19:20.036777 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:20.036741 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51528194-jjdjb"] Nov 03 01:19:20.037691 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:20.037662 4421 cpu_manager.go:414] "containerMap: removing container" podUID="e89093f3-1b11-4b5c-928b-017480d620b0" containerName="revoke-keys" Nov 03 01:19:20.037691 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:20.037688 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e89093f3-1b11-4b5c-928b-017480d620b0" containerName="revoke-keys" Nov 03 01:19:20.037902 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:20.037859 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="e89093f3-1b11-4b5c-928b-017480d620b0" containerName="revoke-keys" Nov 03 01:19:20.042587 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:20.042559 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51528194-jjdjb" Nov 03 01:19:20.047757 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:20.047683 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:19:20.047757 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:20.047726 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6q4xp\"" Nov 03 01:19:20.049166 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:20.049143 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51528194-jjdjb"] Nov 03 01:19:20.144862 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:20.144828 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xq4t\" (UniqueName: \"kubernetes.io/projected/fa352e8c-0c7d-47d0-84df-8c1ee8c51503-kube-api-access-9xq4t\") pod \"maas-api-revoke-sub-51528194-jjdjb\" (UID: \"fa352e8c-0c7d-47d0-84df-8c1ee8c51503\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51528194-jjdjb" Nov 03 01:19:20.145055 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:20.145032 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa352e8c-0c7d-47d0-84df-8c1ee8c51503-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-51528194-jjdjb\" (UID: \"fa352e8c-0c7d-47d0-84df-8c1ee8c51503\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51528194-jjdjb" Nov 03 01:19:20.145147 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:20.145133 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa352e8c-0c7d-47d0-84df-8c1ee8c51503-maas-api-xks-ca\") pod \"maas-api-revoke-sub-51528194-jjdjb\" (UID: \"fa352e8c-0c7d-47d0-84df-8c1ee8c51503\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51528194-jjdjb" Nov 03 01:19:20.246770 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:20.246730 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa352e8c-0c7d-47d0-84df-8c1ee8c51503-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-51528194-jjdjb\" (UID: \"fa352e8c-0c7d-47d0-84df-8c1ee8c51503\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51528194-jjdjb" Nov 03 01:19:20.246996 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:20.246853 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa352e8c-0c7d-47d0-84df-8c1ee8c51503-maas-api-xks-ca\") pod \"maas-api-revoke-sub-51528194-jjdjb\" (UID: \"fa352e8c-0c7d-47d0-84df-8c1ee8c51503\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51528194-jjdjb" Nov 03 01:19:20.247131 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:20.247110 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xq4t\" (UniqueName: \"kubernetes.io/projected/fa352e8c-0c7d-47d0-84df-8c1ee8c51503-kube-api-access-9xq4t\") pod \"maas-api-revoke-sub-51528194-jjdjb\" (UID: \"fa352e8c-0c7d-47d0-84df-8c1ee8c51503\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51528194-jjdjb" Nov 03 01:19:20.247541 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:20.247514 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa352e8c-0c7d-47d0-84df-8c1ee8c51503-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-51528194-jjdjb\" (UID: \"fa352e8c-0c7d-47d0-84df-8c1ee8c51503\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51528194-jjdjb" Nov 03 01:19:20.249304 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:20.249260 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa352e8c-0c7d-47d0-84df-8c1ee8c51503-maas-api-xks-ca\") pod \"maas-api-revoke-sub-51528194-jjdjb\" (UID: \"fa352e8c-0c7d-47d0-84df-8c1ee8c51503\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51528194-jjdjb" Nov 03 01:19:20.255758 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:20.255733 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xq4t\" (UniqueName: \"kubernetes.io/projected/fa352e8c-0c7d-47d0-84df-8c1ee8c51503-kube-api-access-9xq4t\") pod \"maas-api-revoke-sub-51528194-jjdjb\" (UID: \"fa352e8c-0c7d-47d0-84df-8c1ee8c51503\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51528194-jjdjb" Nov 03 01:19:20.367819 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:20.367729 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51528194-jjdjb" Nov 03 01:19:20.690781 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:20.688393 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51528194-jjdjb"] Nov 03 01:19:21.146347 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:21.146264 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51528194-jjdjb" event={"ID":"fa352e8c-0c7d-47d0-84df-8c1ee8c51503","Type":"ContainerStarted","Data":"d4e8f14d3402493973f719942c3fa4556c6ef4197eafed02aadbe611474040cc"} Nov 03 01:19:22.522389 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:22.522103 4421 generic.go:356] "Generic (PLEG): container finished" podID="fa352e8c-0c7d-47d0-84df-8c1ee8c51503" containerID="3a03bc684b33e68e9cbfc1a634adfd73822d91f7fc3695ec5d90225f56c1b6fd" exitCode=0 Nov 03 01:19:22.522910 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:22.522501 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51528194-jjdjb" event={"ID":"fa352e8c-0c7d-47d0-84df-8c1ee8c51503","Type":"ContainerDied","Data":"3a03bc684b33e68e9cbfc1a634adfd73822d91f7fc3695ec5d90225f56c1b6fd"} Nov 03 01:19:22.696299 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:22.677407 4421 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-83755a/simulator-w3-83755a-kserve-77d6664559-psqtg" podUID="b0a4c234-af58-4da0-a1a4-e1f167953f6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": context deadline exceeded" Nov 03 01:19:22.855590 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:22.855493 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8gm8n" podUID="2d367e12-e46f-4a68-9825-9b1c934d4d12" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": context deadline exceeded" Nov 03 01:19:23.718967 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:23.718940 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51528194-jjdjb" Nov 03 01:19:23.828175 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:23.828085 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa352e8c-0c7d-47d0-84df-8c1ee8c51503-kube-api-access-9xq4t\" (UniqueName: \"kubernetes.io/projected/fa352e8c-0c7d-47d0-84df-8c1ee8c51503-kube-api-access-9xq4t\") pod \"fa352e8c-0c7d-47d0-84df-8c1ee8c51503\" (UID: \"fa352e8c-0c7d-47d0-84df-8c1ee8c51503\") " Nov 03 01:19:23.828441 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:23.828389 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fa352e8c-0c7d-47d0-84df-8c1ee8c51503-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa352e8c-0c7d-47d0-84df-8c1ee8c51503-maas-api-xks-ca\") pod \"fa352e8c-0c7d-47d0-84df-8c1ee8c51503\" (UID: \"fa352e8c-0c7d-47d0-84df-8c1ee8c51503\") " Nov 03 01:19:23.828505 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:23.828447 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa352e8c-0c7d-47d0-84df-8c1ee8c51503-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa352e8c-0c7d-47d0-84df-8c1ee8c51503-maas-api-openshift-ca\") pod \"fa352e8c-0c7d-47d0-84df-8c1ee8c51503\" (UID: \"fa352e8c-0c7d-47d0-84df-8c1ee8c51503\") " Nov 03 01:19:23.829466 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:23.828901 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa352e8c-0c7d-47d0-84df-8c1ee8c51503-maas-api-openshift-ca" pod "fa352e8c-0c7d-47d0-84df-8c1ee8c51503" (UID: "fa352e8c-0c7d-47d0-84df-8c1ee8c51503"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:19:23.829466 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:23.829427 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa352e8c-0c7d-47d0-84df-8c1ee8c51503-maas-api-openshift-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:19:23.830898 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:23.830868 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa352e8c-0c7d-47d0-84df-8c1ee8c51503-kube-api-access-9xq4t" pod "fa352e8c-0c7d-47d0-84df-8c1ee8c51503" (UID: "fa352e8c-0c7d-47d0-84df-8c1ee8c51503"). InnerVolumeSpecName "kube-api-access-9xq4t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:19:23.830898 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:23.830883 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa352e8c-0c7d-47d0-84df-8c1ee8c51503-maas-api-xks-ca" pod "fa352e8c-0c7d-47d0-84df-8c1ee8c51503" (UID: "fa352e8c-0c7d-47d0-84df-8c1ee8c51503"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:19:23.932032 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:23.931986 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa352e8c-0c7d-47d0-84df-8c1ee8c51503-maas-api-xks-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:19:23.932032 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:23.932030 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9xq4t\" (UniqueName: \"kubernetes.io/projected/fa352e8c-0c7d-47d0-84df-8c1ee8c51503-kube-api-access-9xq4t\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:19:24.545920 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:24.545890 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51528194-jjdjb" Nov 03 01:19:24.546100 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:24.545923 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51528194-jjdjb" event={"ID":"fa352e8c-0c7d-47d0-84df-8c1ee8c51503","Type":"ContainerDied","Data":"d4e8f14d3402493973f719942c3fa4556c6ef4197eafed02aadbe611474040cc"} Nov 03 01:19:24.546100 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:24.545977 4421 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d4e8f14d3402493973f719942c3fa4556c6ef4197eafed02aadbe611474040cc" Nov 03 01:19:32.468467 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:32.468425 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12bf75e6-x7t6r"] Nov 03 01:19:32.470302 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:32.470256 4421 cpu_manager.go:414] "containerMap: removing container" podUID="fa352e8c-0c7d-47d0-84df-8c1ee8c51503" containerName="revoke-keys" Nov 03 01:19:32.470473 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:32.470454 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa352e8c-0c7d-47d0-84df-8c1ee8c51503" containerName="revoke-keys" Nov 03 01:19:32.470971 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:32.470945 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa352e8c-0c7d-47d0-84df-8c1ee8c51503" containerName="revoke-keys" Nov 03 01:19:32.479060 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:32.479031 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12bf75e6-x7t6r"] Nov 03 01:19:32.479183 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:32.479148 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12bf75e6-x7t6r" Nov 03 01:19:32.483405 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:32.483379 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:19:32.485817 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:32.485795 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6q4xp\"" Nov 03 01:19:32.586547 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:32.586498 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jps2j\" (UniqueName: \"kubernetes.io/projected/26afee51-c188-4d90-8989-32150bffde59-kube-api-access-jps2j\") pod \"maas-api-revoke-sub-12bf75e6-x7t6r\" (UID: \"26afee51-c188-4d90-8989-32150bffde59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12bf75e6-x7t6r" Nov 03 01:19:32.586750 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:32.586673 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26afee51-c188-4d90-8989-32150bffde59-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-12bf75e6-x7t6r\" (UID: \"26afee51-c188-4d90-8989-32150bffde59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12bf75e6-x7t6r" Nov 03 01:19:32.586895 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:32.586879 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26afee51-c188-4d90-8989-32150bffde59-maas-api-xks-ca\") pod \"maas-api-revoke-sub-12bf75e6-x7t6r\" (UID: \"26afee51-c188-4d90-8989-32150bffde59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12bf75e6-x7t6r" Nov 03 01:19:32.689587 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:32.689545 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26afee51-c188-4d90-8989-32150bffde59-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-12bf75e6-x7t6r\" (UID: \"26afee51-c188-4d90-8989-32150bffde59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12bf75e6-x7t6r" Nov 03 01:19:32.690063 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:32.690040 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26afee51-c188-4d90-8989-32150bffde59-maas-api-xks-ca\") pod \"maas-api-revoke-sub-12bf75e6-x7t6r\" (UID: \"26afee51-c188-4d90-8989-32150bffde59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12bf75e6-x7t6r" Nov 03 01:19:32.694510 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:32.694478 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jps2j\" (UniqueName: \"kubernetes.io/projected/26afee51-c188-4d90-8989-32150bffde59-kube-api-access-jps2j\") pod \"maas-api-revoke-sub-12bf75e6-x7t6r\" (UID: \"26afee51-c188-4d90-8989-32150bffde59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12bf75e6-x7t6r" Nov 03 01:19:32.709068 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:32.709034 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26afee51-c188-4d90-8989-32150bffde59-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-12bf75e6-x7t6r\" (UID: \"26afee51-c188-4d90-8989-32150bffde59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12bf75e6-x7t6r" Nov 03 01:19:32.757296 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:32.756832 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26afee51-c188-4d90-8989-32150bffde59-maas-api-xks-ca\") pod \"maas-api-revoke-sub-12bf75e6-x7t6r\" (UID: \"26afee51-c188-4d90-8989-32150bffde59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12bf75e6-x7t6r" Nov 03 01:19:32.762298 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:32.757846 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jps2j\" (UniqueName: \"kubernetes.io/projected/26afee51-c188-4d90-8989-32150bffde59-kube-api-access-jps2j\") pod \"maas-api-revoke-sub-12bf75e6-x7t6r\" (UID: \"26afee51-c188-4d90-8989-32150bffde59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12bf75e6-x7t6r" Nov 03 01:19:32.809824 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:32.809544 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12bf75e6-x7t6r" Nov 03 01:19:33.264820 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:19:33.264786 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26afee51_c188_4d90_8989_32150bffde59.slice/crio-37c030c53a1754830d707d8f375a4f84d2a01b48728f30954655dd92a231f6e5 WatchSource:0}: Error finding container 37c030c53a1754830d707d8f375a4f84d2a01b48728f30954655dd92a231f6e5: Status 404 returned error can't find the container with id 37c030c53a1754830d707d8f375a4f84d2a01b48728f30954655dd92a231f6e5 Nov 03 01:19:33.264969 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:33.264904 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12bf75e6-x7t6r"] Nov 03 01:19:33.815526 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:33.815492 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12bf75e6-x7t6r" event={"ID":"26afee51-c188-4d90-8989-32150bffde59","Type":"ContainerStarted","Data":"37c030c53a1754830d707d8f375a4f84d2a01b48728f30954655dd92a231f6e5"} Nov 03 01:19:34.825811 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:34.825774 4421 generic.go:356] "Generic (PLEG): container finished" podID="26afee51-c188-4d90-8989-32150bffde59" containerID="b05376c40dc877e2dddbc9e4ec3e2d8b106a096d28ff734c8746aba8a7da17be" exitCode=0 Nov 03 01:19:34.826203 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:34.825851 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12bf75e6-x7t6r" event={"ID":"26afee51-c188-4d90-8989-32150bffde59","Type":"ContainerDied","Data":"b05376c40dc877e2dddbc9e4ec3e2d8b106a096d28ff734c8746aba8a7da17be"} Nov 03 01:19:36.029823 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:36.029796 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12bf75e6-x7t6r" Nov 03 01:19:36.172789 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:36.172697 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26afee51-c188-4d90-8989-32150bffde59-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26afee51-c188-4d90-8989-32150bffde59-maas-api-openshift-ca\") pod \"26afee51-c188-4d90-8989-32150bffde59\" (UID: \"26afee51-c188-4d90-8989-32150bffde59\") " Nov 03 01:19:36.172955 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:36.172901 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/26afee51-c188-4d90-8989-32150bffde59-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26afee51-c188-4d90-8989-32150bffde59-maas-api-xks-ca\") pod \"26afee51-c188-4d90-8989-32150bffde59\" (UID: \"26afee51-c188-4d90-8989-32150bffde59\") " Nov 03 01:19:36.173010 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:36.172973 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26afee51-c188-4d90-8989-32150bffde59-kube-api-access-jps2j\" (UniqueName: \"kubernetes.io/projected/26afee51-c188-4d90-8989-32150bffde59-kube-api-access-jps2j\") pod \"26afee51-c188-4d90-8989-32150bffde59\" (UID: \"26afee51-c188-4d90-8989-32150bffde59\") " Nov 03 01:19:36.173069 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:36.173042 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26afee51-c188-4d90-8989-32150bffde59-maas-api-openshift-ca" pod "26afee51-c188-4d90-8989-32150bffde59" (UID: "26afee51-c188-4d90-8989-32150bffde59"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:19:36.173768 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:36.173745 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26afee51-c188-4d90-8989-32150bffde59-maas-api-openshift-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:19:36.175144 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:36.175119 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26afee51-c188-4d90-8989-32150bffde59-maas-api-xks-ca" pod "26afee51-c188-4d90-8989-32150bffde59" (UID: "26afee51-c188-4d90-8989-32150bffde59"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:19:36.175247 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:36.175192 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26afee51-c188-4d90-8989-32150bffde59-kube-api-access-jps2j" pod "26afee51-c188-4d90-8989-32150bffde59" (UID: "26afee51-c188-4d90-8989-32150bffde59"). InnerVolumeSpecName "kube-api-access-jps2j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:19:36.275529 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:36.275494 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26afee51-c188-4d90-8989-32150bffde59-maas-api-xks-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:19:36.275529 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:36.275527 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jps2j\" (UniqueName: \"kubernetes.io/projected/26afee51-c188-4d90-8989-32150bffde59-kube-api-access-jps2j\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:19:36.843646 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:36.843609 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12bf75e6-x7t6r" Nov 03 01:19:36.843826 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:36.843646 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12bf75e6-x7t6r" event={"ID":"26afee51-c188-4d90-8989-32150bffde59","Type":"ContainerDied","Data":"37c030c53a1754830d707d8f375a4f84d2a01b48728f30954655dd92a231f6e5"} Nov 03 01:19:36.843826 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:36.843693 4421 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="37c030c53a1754830d707d8f375a4f84d2a01b48728f30954655dd92a231f6e5" Nov 03 01:19:44.671915 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:19:44.671876 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 01:19:49.142476 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:49.142438 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-ddd653-openshift-default-779ff6465-6hnrl"] Nov 03 01:19:49.174806 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:49.174711 4421 cpu_manager.go:414] "containerMap: removing container" podUID="26afee51-c188-4d90-8989-32150bffde59" containerName="revoke-keys" Nov 03 01:19:49.175018 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:49.175001 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26afee51-c188-4d90-8989-32150bffde59" containerName="revoke-keys" Nov 03 01:19:49.175653 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:49.175630 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="26afee51-c188-4d90-8989-32150bffde59" containerName="revoke-keys" Nov 03 01:19:49.191856 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:49.191822 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-ddd653-openshift-default-779ff6465-6hnrl" Nov 03 01:19:49.193178 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:49.193151 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-ddd653-openshift-default-779ff6465-6hnrl"] Nov 03 01:19:49.196040 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:49.196010 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-ddd653-openshift-default-dockercfg-zx7f4\"" Nov 03 01:19:49.349666 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:49.349629 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b337af9-fa55-447d-9555-7852367061f5-workload-socket\") pod \"e2e-extmod-a-ddd653-openshift-default-779ff6465-6hnrl\" (UID: \"8b337af9-fa55-447d-9555-7852367061f5\") " pod="openshift-ingress/e2e-extmod-a-ddd653-openshift-default-779ff6465-6hnrl" Nov 03 01:19:49.349881 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:49.349753 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b337af9-fa55-447d-9555-7852367061f5-istio-token\") pod \"e2e-extmod-a-ddd653-openshift-default-779ff6465-6hnrl\" (UID: \"8b337af9-fa55-447d-9555-7852367061f5\") " pod="openshift-ingress/e2e-extmod-a-ddd653-openshift-default-779ff6465-6hnrl" Nov 03 01:19:49.349881 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:49.349792 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b337af9-fa55-447d-9555-7852367061f5-credential-socket\") pod \"e2e-extmod-a-ddd653-openshift-default-779ff6465-6hnrl\" (UID: \"8b337af9-fa55-447d-9555-7852367061f5\") " pod="openshift-ingress/e2e-extmod-a-ddd653-openshift-default-779ff6465-6hnrl" Nov 03 01:19:49.349881 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:49.349822 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b337af9-fa55-447d-9555-7852367061f5-istio-data\") pod \"e2e-extmod-a-ddd653-openshift-default-779ff6465-6hnrl\" (UID: \"8b337af9-fa55-447d-9555-7852367061f5\") " pod="openshift-ingress/e2e-extmod-a-ddd653-openshift-default-779ff6465-6hnrl" Nov 03 01:19:49.350048 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:49.349892 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b337af9-fa55-447d-9555-7852367061f5-istio-envoy\") pod \"e2e-extmod-a-ddd653-openshift-default-779ff6465-6hnrl\" (UID: \"8b337af9-fa55-447d-9555-7852367061f5\") " pod="openshift-ingress/e2e-extmod-a-ddd653-openshift-default-779ff6465-6hnrl" Nov 03 01:19:49.350092 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:49.350041 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b337af9-fa55-447d-9555-7852367061f5-istio-podinfo\") pod \"e2e-extmod-a-ddd653-openshift-default-779ff6465-6hnrl\" (UID: \"8b337af9-fa55-447d-9555-7852367061f5\") " pod="openshift-ingress/e2e-extmod-a-ddd653-openshift-default-779ff6465-6hnrl" Nov 03 01:19:49.350092 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:49.350086 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zfnm\" (UniqueName: \"kubernetes.io/projected/8b337af9-fa55-447d-9555-7852367061f5-kube-api-access-4zfnm\") pod \"e2e-extmod-a-ddd653-openshift-default-779ff6465-6hnrl\" (UID: \"8b337af9-fa55-447d-9555-7852367061f5\") " pod="openshift-ingress/e2e-extmod-a-ddd653-openshift-default-779ff6465-6hnrl" Nov 03 01:19:49.350268 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:49.350255 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b337af9-fa55-447d-9555-7852367061f5-workload-certs\") pod \"e2e-extmod-a-ddd653-openshift-default-779ff6465-6hnrl\" (UID: \"8b337af9-fa55-447d-9555-7852367061f5\") " pod="openshift-ingress/e2e-extmod-a-ddd653-openshift-default-779ff6465-6hnrl" Nov 03 01:19:49.350386 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:49.350374 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b337af9-fa55-447d-9555-7852367061f5-istiod-ca-cert\") pod \"e2e-extmod-a-ddd653-openshift-default-779ff6465-6hnrl\" (UID: \"8b337af9-fa55-447d-9555-7852367061f5\") " pod="openshift-ingress/e2e-extmod-a-ddd653-openshift-default-779ff6465-6hnrl" Nov 03 01:19:49.451930 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:49.451834 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zfnm\" (UniqueName: \"kubernetes.io/projected/8b337af9-fa55-447d-9555-7852367061f5-kube-api-access-4zfnm\") pod \"e2e-extmod-a-ddd653-openshift-default-779ff6465-6hnrl\" (UID: \"8b337af9-fa55-447d-9555-7852367061f5\") " pod="openshift-ingress/e2e-extmod-a-ddd653-openshift-default-779ff6465-6hnrl" Nov 03 01:19:49.452107 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:49.451986 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b337af9-fa55-447d-9555-7852367061f5-workload-certs\") pod \"e2e-extmod-a-ddd653-openshift-default-779ff6465-6hnrl\" (UID: \"8b337af9-fa55-447d-9555-7852367061f5\") " pod="openshift-ingress/e2e-extmod-a-ddd653-openshift-default-779ff6465-6hnrl" Nov 03 01:19:49.452107 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:49.452042 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b337af9-fa55-447d-9555-7852367061f5-istiod-ca-cert\") pod \"e2e-extmod-a-ddd653-openshift-default-779ff6465-6hnrl\" (UID: \"8b337af9-fa55-447d-9555-7852367061f5\") " pod="openshift-ingress/e2e-extmod-a-ddd653-openshift-default-779ff6465-6hnrl" Nov 03 01:19:49.452181 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:49.452132 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b337af9-fa55-447d-9555-7852367061f5-workload-socket\") pod \"e2e-extmod-a-ddd653-openshift-default-779ff6465-6hnrl\" (UID: \"8b337af9-fa55-447d-9555-7852367061f5\") " pod="openshift-ingress/e2e-extmod-a-ddd653-openshift-default-779ff6465-6hnrl" Nov 03 01:19:49.452219 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:49.452186 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b337af9-fa55-447d-9555-7852367061f5-istio-token\") pod \"e2e-extmod-a-ddd653-openshift-default-779ff6465-6hnrl\" (UID: \"8b337af9-fa55-447d-9555-7852367061f5\") " pod="openshift-ingress/e2e-extmod-a-ddd653-openshift-default-779ff6465-6hnrl" Nov 03 01:19:49.452219 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:49.452211 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b337af9-fa55-447d-9555-7852367061f5-credential-socket\") pod \"e2e-extmod-a-ddd653-openshift-default-779ff6465-6hnrl\" (UID: \"8b337af9-fa55-447d-9555-7852367061f5\") " pod="openshift-ingress/e2e-extmod-a-ddd653-openshift-default-779ff6465-6hnrl" Nov 03 01:19:49.452390 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:49.452237 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b337af9-fa55-447d-9555-7852367061f5-istio-data\") pod \"e2e-extmod-a-ddd653-openshift-default-779ff6465-6hnrl\" (UID: \"8b337af9-fa55-447d-9555-7852367061f5\") " pod="openshift-ingress/e2e-extmod-a-ddd653-openshift-default-779ff6465-6hnrl" Nov 03 01:19:49.452390 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:49.452328 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b337af9-fa55-447d-9555-7852367061f5-istio-envoy\") pod \"e2e-extmod-a-ddd653-openshift-default-779ff6465-6hnrl\" (UID: \"8b337af9-fa55-447d-9555-7852367061f5\") " pod="openshift-ingress/e2e-extmod-a-ddd653-openshift-default-779ff6465-6hnrl" Nov 03 01:19:49.452390 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:49.452364 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b337af9-fa55-447d-9555-7852367061f5-istio-podinfo\") pod \"e2e-extmod-a-ddd653-openshift-default-779ff6465-6hnrl\" (UID: \"8b337af9-fa55-447d-9555-7852367061f5\") " pod="openshift-ingress/e2e-extmod-a-ddd653-openshift-default-779ff6465-6hnrl" Nov 03 01:19:49.452567 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:49.452538 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b337af9-fa55-447d-9555-7852367061f5-workload-certs\") pod \"e2e-extmod-a-ddd653-openshift-default-779ff6465-6hnrl\" (UID: \"8b337af9-fa55-447d-9555-7852367061f5\") " pod="openshift-ingress/e2e-extmod-a-ddd653-openshift-default-779ff6465-6hnrl" Nov 03 01:19:49.452567 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:49.452562 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b337af9-fa55-447d-9555-7852367061f5-workload-socket\") pod \"e2e-extmod-a-ddd653-openshift-default-779ff6465-6hnrl\" (UID: \"8b337af9-fa55-447d-9555-7852367061f5\") " pod="openshift-ingress/e2e-extmod-a-ddd653-openshift-default-779ff6465-6hnrl" Nov 03 01:19:49.452702 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:49.452678 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b337af9-fa55-447d-9555-7852367061f5-istio-data\") pod \"e2e-extmod-a-ddd653-openshift-default-779ff6465-6hnrl\" (UID: \"8b337af9-fa55-447d-9555-7852367061f5\") " pod="openshift-ingress/e2e-extmod-a-ddd653-openshift-default-779ff6465-6hnrl" Nov 03 01:19:49.452812 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:49.452787 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b337af9-fa55-447d-9555-7852367061f5-credential-socket\") pod \"e2e-extmod-a-ddd653-openshift-default-779ff6465-6hnrl\" (UID: \"8b337af9-fa55-447d-9555-7852367061f5\") " pod="openshift-ingress/e2e-extmod-a-ddd653-openshift-default-779ff6465-6hnrl" Nov 03 01:19:49.453054 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:49.453035 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b337af9-fa55-447d-9555-7852367061f5-istiod-ca-cert\") pod \"e2e-extmod-a-ddd653-openshift-default-779ff6465-6hnrl\" (UID: \"8b337af9-fa55-447d-9555-7852367061f5\") " pod="openshift-ingress/e2e-extmod-a-ddd653-openshift-default-779ff6465-6hnrl" Nov 03 01:19:49.454785 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:49.454753 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b337af9-fa55-447d-9555-7852367061f5-istio-envoy\") pod \"e2e-extmod-a-ddd653-openshift-default-779ff6465-6hnrl\" (UID: \"8b337af9-fa55-447d-9555-7852367061f5\") " pod="openshift-ingress/e2e-extmod-a-ddd653-openshift-default-779ff6465-6hnrl" Nov 03 01:19:49.454980 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:49.454961 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b337af9-fa55-447d-9555-7852367061f5-istio-podinfo\") pod \"e2e-extmod-a-ddd653-openshift-default-779ff6465-6hnrl\" (UID: \"8b337af9-fa55-447d-9555-7852367061f5\") " pod="openshift-ingress/e2e-extmod-a-ddd653-openshift-default-779ff6465-6hnrl" Nov 03 01:19:49.459888 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:49.459863 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b337af9-fa55-447d-9555-7852367061f5-istio-token\") pod \"e2e-extmod-a-ddd653-openshift-default-779ff6465-6hnrl\" (UID: \"8b337af9-fa55-447d-9555-7852367061f5\") " pod="openshift-ingress/e2e-extmod-a-ddd653-openshift-default-779ff6465-6hnrl" Nov 03 01:19:49.460005 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:49.459966 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zfnm\" (UniqueName: \"kubernetes.io/projected/8b337af9-fa55-447d-9555-7852367061f5-kube-api-access-4zfnm\") pod \"e2e-extmod-a-ddd653-openshift-default-779ff6465-6hnrl\" (UID: \"8b337af9-fa55-447d-9555-7852367061f5\") " pod="openshift-ingress/e2e-extmod-a-ddd653-openshift-default-779ff6465-6hnrl" Nov 03 01:19:49.519852 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:49.519812 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-ddd653-openshift-default-779ff6465-6hnrl" Nov 03 01:19:49.959555 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:49.959517 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-ddd653-openshift-default-779ff6465-6hnrl"] Nov 03 01:19:50.003576 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:49.995141 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:19:50.003576 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:49.995240 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:19:50.003576 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:49.995402 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:19:50.199113 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:50.199072 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-ddd653-openshift-default-779ff6465-6hnrl" event={"ID":"8b337af9-fa55-447d-9555-7852367061f5","Type":"ContainerStarted","Data":"9c8617669bb3974e29df388e4cd520abd543b06add22c069b572607542e2224e"} Nov 03 01:19:50.200385 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:50.199141 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-ddd653-openshift-default-779ff6465-6hnrl" event={"ID":"8b337af9-fa55-447d-9555-7852367061f5","Type":"ContainerStarted","Data":"ccf98034c97b8e6b040bd68fccdb1d7d675256b070a34c40342e286b0b2472c5"} Nov 03 01:19:50.519970 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:50.519935 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-ddd653-openshift-default-779ff6465-6hnrl" Nov 03 01:19:51.532993 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:51.530413 4421 patch_prober.go:29] interesting pod/e2e-extmod-a-ddd653-openshift-default-779ff6465-6hnrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.37:15021/healthz/ready\": dial tcp 10.134.1.37:15021: connect: connection refused" start-of-body= Nov 03 01:19:51.532993 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:51.530498 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ddd653-openshift-default-779ff6465-6hnrl" podUID="8b337af9-fa55-447d-9555-7852367061f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.37:15021/healthz/ready\": dial tcp 10.134.1.37:15021: connect: connection refused" Nov 03 01:19:52.529300 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:52.529192 4421 patch_prober.go:29] interesting pod/e2e-extmod-a-ddd653-openshift-default-779ff6465-6hnrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.37:15021/healthz/ready\": dial tcp 10.134.1.37:15021: connect: connection refused" start-of-body= Nov 03 01:19:52.529300 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:52.529254 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ddd653-openshift-default-779ff6465-6hnrl" podUID="8b337af9-fa55-447d-9555-7852367061f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.37:15021/healthz/ready\": dial tcp 10.134.1.37:15021: connect: connection refused" Nov 03 01:19:53.550321 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:53.542877 4421 patch_prober.go:29] interesting pod/e2e-extmod-a-ddd653-openshift-default-779ff6465-6hnrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.37:15021/healthz/ready\": dial tcp 10.134.1.37:15021: connect: connection refused" start-of-body= Nov 03 01:19:53.550321 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:53.542948 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ddd653-openshift-default-779ff6465-6hnrl" podUID="8b337af9-fa55-447d-9555-7852367061f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.37:15021/healthz/ready\": dial tcp 10.134.1.37:15021: connect: connection refused" Nov 03 01:19:54.532299 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:54.523884 4421 patch_prober.go:29] interesting pod/e2e-extmod-a-ddd653-openshift-default-779ff6465-6hnrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.37:15021/healthz/ready\": dial tcp 10.134.1.37:15021: connect: connection refused" start-of-body= Nov 03 01:19:54.532299 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:54.523947 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ddd653-openshift-default-779ff6465-6hnrl" podUID="8b337af9-fa55-447d-9555-7852367061f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.37:15021/healthz/ready\": dial tcp 10.134.1.37:15021: connect: connection refused" Nov 03 01:19:55.532315 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:55.530951 4421 patch_prober.go:29] interesting pod/e2e-extmod-a-ddd653-openshift-default-779ff6465-6hnrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.37:15021/healthz/ready\": dial tcp 10.134.1.37:15021: connect: connection refused" start-of-body= Nov 03 01:19:55.532315 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:55.531013 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ddd653-openshift-default-779ff6465-6hnrl" podUID="8b337af9-fa55-447d-9555-7852367061f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.37:15021/healthz/ready\": dial tcp 10.134.1.37:15021: connect: connection refused" Nov 03 01:19:56.530737 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:56.530694 4421 patch_prober.go:29] interesting pod/e2e-extmod-a-ddd653-openshift-default-779ff6465-6hnrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.37:15021/healthz/ready\": dial tcp 10.134.1.37:15021: connect: connection refused" start-of-body= Nov 03 01:19:56.530955 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:56.530761 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ddd653-openshift-default-779ff6465-6hnrl" podUID="8b337af9-fa55-447d-9555-7852367061f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.37:15021/healthz/ready\": dial tcp 10.134.1.37:15021: connect: connection refused" Nov 03 01:19:57.525869 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:57.525840 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-ddd653-openshift-default-779ff6465-6hnrl" Nov 03 01:19:57.526265 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:57.526216 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-ddd653-openshift-default-779ff6465-6hnrl" Nov 03 01:19:57.526927 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:57.526907 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-ddd653-openshift-default-779ff6465-6hnrl" Nov 03 01:19:57.552864 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:57.552806 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-ddd653-openshift-default-779ff6465-6hnrl" podStartSLOduration=8.552788845 podStartE2EDuration="8.552788845s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:19:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:19:50.250744179 +0000 UTC m=+2332.421747276" watchObservedRunningTime="2026-11-03 01:19:57.552788845 +0000 UTC m=+2339.723791942" Nov 03 01:19:57.588222 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:57.588189 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5b879d4-kr8sh"] Nov 03 01:19:57.614351 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:57.593712 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5b879d4-kr8sh" Nov 03 01:19:57.614351 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:57.597260 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6q4xp\"" Nov 03 01:19:57.614351 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:57.597346 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:19:57.614351 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:57.612320 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5b879d4-kr8sh"] Nov 03 01:19:57.672393 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:57.672354 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdbng\" (UniqueName: \"kubernetes.io/projected/2e200b2a-46e2-4597-985a-7fc94eb47e42-kube-api-access-sdbng\") pod \"maas-api-revoke-sub-a5b879d4-kr8sh\" (UID: \"2e200b2a-46e2-4597-985a-7fc94eb47e42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5b879d4-kr8sh" Nov 03 01:19:57.672719 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:57.672696 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e200b2a-46e2-4597-985a-7fc94eb47e42-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a5b879d4-kr8sh\" (UID: \"2e200b2a-46e2-4597-985a-7fc94eb47e42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5b879d4-kr8sh" Nov 03 01:19:57.672835 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:57.672815 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e200b2a-46e2-4597-985a-7fc94eb47e42-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a5b879d4-kr8sh\" (UID: \"2e200b2a-46e2-4597-985a-7fc94eb47e42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5b879d4-kr8sh" Nov 03 01:19:57.775338 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:57.775250 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e200b2a-46e2-4597-985a-7fc94eb47e42-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a5b879d4-kr8sh\" (UID: \"2e200b2a-46e2-4597-985a-7fc94eb47e42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5b879d4-kr8sh" Nov 03 01:19:57.775550 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:57.775466 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdbng\" (UniqueName: \"kubernetes.io/projected/2e200b2a-46e2-4597-985a-7fc94eb47e42-kube-api-access-sdbng\") pod \"maas-api-revoke-sub-a5b879d4-kr8sh\" (UID: \"2e200b2a-46e2-4597-985a-7fc94eb47e42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5b879d4-kr8sh" Nov 03 01:19:57.775630 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:57.775609 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e200b2a-46e2-4597-985a-7fc94eb47e42-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a5b879d4-kr8sh\" (UID: \"2e200b2a-46e2-4597-985a-7fc94eb47e42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5b879d4-kr8sh" Nov 03 01:19:57.776028 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:57.775967 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e200b2a-46e2-4597-985a-7fc94eb47e42-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a5b879d4-kr8sh\" (UID: \"2e200b2a-46e2-4597-985a-7fc94eb47e42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5b879d4-kr8sh" Nov 03 01:19:57.778052 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:57.778019 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e200b2a-46e2-4597-985a-7fc94eb47e42-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a5b879d4-kr8sh\" (UID: \"2e200b2a-46e2-4597-985a-7fc94eb47e42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5b879d4-kr8sh" Nov 03 01:19:57.784251 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:57.784221 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdbng\" (UniqueName: \"kubernetes.io/projected/2e200b2a-46e2-4597-985a-7fc94eb47e42-kube-api-access-sdbng\") pod \"maas-api-revoke-sub-a5b879d4-kr8sh\" (UID: \"2e200b2a-46e2-4597-985a-7fc94eb47e42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5b879d4-kr8sh" Nov 03 01:19:57.923163 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:57.923126 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5b879d4-kr8sh" Nov 03 01:19:58.549839 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:19:58.549770 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e200b2a_46e2_4597_985a_7fc94eb47e42.slice/crio-bd1879deb6eb91188d38df6866f15032121b25f23954535d344b61360e28dc97 WatchSource:0}: Error finding container bd1879deb6eb91188d38df6866f15032121b25f23954535d344b61360e28dc97: Status 404 returned error can't find the container with id bd1879deb6eb91188d38df6866f15032121b25f23954535d344b61360e28dc97 Nov 03 01:19:58.550722 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:58.550699 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5b879d4-kr8sh"] Nov 03 01:19:59.438395 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:59.438341 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5b879d4-kr8sh" event={"ID":"2e200b2a-46e2-4597-985a-7fc94eb47e42","Type":"ContainerStarted","Data":"8f265833c1cd6a6f82dfd8ff2360f264f3e4d37868577439d95fc5000544ff1b"} Nov 03 01:19:59.438395 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:59.438402 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5b879d4-kr8sh" event={"ID":"2e200b2a-46e2-4597-985a-7fc94eb47e42","Type":"ContainerStarted","Data":"bd1879deb6eb91188d38df6866f15032121b25f23954535d344b61360e28dc97"} Nov 03 01:19:59.457231 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:19:59.457172 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5b879d4-kr8sh" podStartSLOduration=2.457157841 podStartE2EDuration="2.457157841s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:19:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:19:59.455297429 +0000 UTC m=+2341.626300525" watchObservedRunningTime="2026-11-03 01:19:59.457157841 +0000 UTC m=+2341.628160949" Nov 03 01:20:00.463333 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:00.463291 4421 generic.go:356] "Generic (PLEG): container finished" podID="2e200b2a-46e2-4597-985a-7fc94eb47e42" containerID="8f265833c1cd6a6f82dfd8ff2360f264f3e4d37868577439d95fc5000544ff1b" exitCode=0 Nov 03 01:20:00.463819 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:00.463364 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5b879d4-kr8sh" event={"ID":"2e200b2a-46e2-4597-985a-7fc94eb47e42","Type":"ContainerDied","Data":"8f265833c1cd6a6f82dfd8ff2360f264f3e4d37868577439d95fc5000544ff1b"} Nov 03 01:20:01.685990 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:01.685957 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5b879d4-kr8sh" Nov 03 01:20:01.841027 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:01.840991 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e200b2a-46e2-4597-985a-7fc94eb47e42-kube-api-access-sdbng\" (UniqueName: \"kubernetes.io/projected/2e200b2a-46e2-4597-985a-7fc94eb47e42-kube-api-access-sdbng\") pod \"2e200b2a-46e2-4597-985a-7fc94eb47e42\" (UID: \"2e200b2a-46e2-4597-985a-7fc94eb47e42\") " Nov 03 01:20:01.841373 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:01.841198 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e200b2a-46e2-4597-985a-7fc94eb47e42-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e200b2a-46e2-4597-985a-7fc94eb47e42-maas-api-openshift-ca\") pod \"2e200b2a-46e2-4597-985a-7fc94eb47e42\" (UID: \"2e200b2a-46e2-4597-985a-7fc94eb47e42\") " Nov 03 01:20:01.841373 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:01.841353 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e200b2a-46e2-4597-985a-7fc94eb47e42-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e200b2a-46e2-4597-985a-7fc94eb47e42-maas-api-xks-ca\") pod \"2e200b2a-46e2-4597-985a-7fc94eb47e42\" (UID: \"2e200b2a-46e2-4597-985a-7fc94eb47e42\") " Nov 03 01:20:01.841604 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:01.841574 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e200b2a-46e2-4597-985a-7fc94eb47e42-maas-api-openshift-ca" pod "2e200b2a-46e2-4597-985a-7fc94eb47e42" (UID: "2e200b2a-46e2-4597-985a-7fc94eb47e42"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:20:01.842112 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:01.842090 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e200b2a-46e2-4597-985a-7fc94eb47e42-maas-api-openshift-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:20:01.843400 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:01.843374 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e200b2a-46e2-4597-985a-7fc94eb47e42-maas-api-xks-ca" pod "2e200b2a-46e2-4597-985a-7fc94eb47e42" (UID: "2e200b2a-46e2-4597-985a-7fc94eb47e42"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:20:01.843558 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:01.843537 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e200b2a-46e2-4597-985a-7fc94eb47e42-kube-api-access-sdbng" pod "2e200b2a-46e2-4597-985a-7fc94eb47e42" (UID: "2e200b2a-46e2-4597-985a-7fc94eb47e42"). InnerVolumeSpecName "kube-api-access-sdbng". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:20:01.944143 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:01.944112 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e200b2a-46e2-4597-985a-7fc94eb47e42-maas-api-xks-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:20:01.944143 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:01.944141 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdbng\" (UniqueName: \"kubernetes.io/projected/2e200b2a-46e2-4597-985a-7fc94eb47e42-kube-api-access-sdbng\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:20:02.481161 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:02.481127 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5b879d4-kr8sh" Nov 03 01:20:02.481418 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:02.481163 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5b879d4-kr8sh" event={"ID":"2e200b2a-46e2-4597-985a-7fc94eb47e42","Type":"ContainerDied","Data":"bd1879deb6eb91188d38df6866f15032121b25f23954535d344b61360e28dc97"} Nov 03 01:20:02.481418 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:02.481218 4421 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bd1879deb6eb91188d38df6866f15032121b25f23954535d344b61360e28dc97" Nov 03 01:20:08.352665 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:08.352629 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93bc19da-pjg99"] Nov 03 01:20:08.353609 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:08.353589 4421 cpu_manager.go:414] "containerMap: removing container" podUID="2e200b2a-46e2-4597-985a-7fc94eb47e42" containerName="revoke-keys" Nov 03 01:20:08.353609 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:08.353609 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e200b2a-46e2-4597-985a-7fc94eb47e42" containerName="revoke-keys" Nov 03 01:20:08.353774 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:08.353761 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e200b2a-46e2-4597-985a-7fc94eb47e42" containerName="revoke-keys" Nov 03 01:20:08.358574 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:08.358543 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93bc19da-pjg99" Nov 03 01:20:08.362755 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:08.362731 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:20:08.364091 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:08.364064 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6q4xp\"" Nov 03 01:20:08.369291 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:08.369250 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93bc19da-pjg99"] Nov 03 01:20:08.462000 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:08.461960 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wjt6\" (UniqueName: \"kubernetes.io/projected/752f8f24-ac91-4fb6-96ea-963146efe4c9-kube-api-access-6wjt6\") pod \"maas-api-revoke-sub-93bc19da-pjg99\" (UID: \"752f8f24-ac91-4fb6-96ea-963146efe4c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93bc19da-pjg99" Nov 03 01:20:08.462288 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:08.462258 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/752f8f24-ac91-4fb6-96ea-963146efe4c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-93bc19da-pjg99\" (UID: \"752f8f24-ac91-4fb6-96ea-963146efe4c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93bc19da-pjg99" Nov 03 01:20:08.462367 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:08.462331 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/752f8f24-ac91-4fb6-96ea-963146efe4c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-93bc19da-pjg99\" (UID: \"752f8f24-ac91-4fb6-96ea-963146efe4c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93bc19da-pjg99" Nov 03 01:20:08.563525 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:08.563487 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wjt6\" (UniqueName: \"kubernetes.io/projected/752f8f24-ac91-4fb6-96ea-963146efe4c9-kube-api-access-6wjt6\") pod \"maas-api-revoke-sub-93bc19da-pjg99\" (UID: \"752f8f24-ac91-4fb6-96ea-963146efe4c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93bc19da-pjg99" Nov 03 01:20:08.563740 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:08.563726 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/752f8f24-ac91-4fb6-96ea-963146efe4c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-93bc19da-pjg99\" (UID: \"752f8f24-ac91-4fb6-96ea-963146efe4c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93bc19da-pjg99" Nov 03 01:20:08.563789 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:08.563760 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/752f8f24-ac91-4fb6-96ea-963146efe4c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-93bc19da-pjg99\" (UID: \"752f8f24-ac91-4fb6-96ea-963146efe4c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93bc19da-pjg99" Nov 03 01:20:08.564572 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:08.564550 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/752f8f24-ac91-4fb6-96ea-963146efe4c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-93bc19da-pjg99\" (UID: \"752f8f24-ac91-4fb6-96ea-963146efe4c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93bc19da-pjg99" Nov 03 01:20:08.566192 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:08.566172 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/752f8f24-ac91-4fb6-96ea-963146efe4c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-93bc19da-pjg99\" (UID: \"752f8f24-ac91-4fb6-96ea-963146efe4c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93bc19da-pjg99" Nov 03 01:20:08.572690 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:08.572668 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wjt6\" (UniqueName: \"kubernetes.io/projected/752f8f24-ac91-4fb6-96ea-963146efe4c9-kube-api-access-6wjt6\") pod \"maas-api-revoke-sub-93bc19da-pjg99\" (UID: \"752f8f24-ac91-4fb6-96ea-963146efe4c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93bc19da-pjg99" Nov 03 01:20:08.678209 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:08.678121 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93bc19da-pjg99" Nov 03 01:20:09.098699 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:09.077366 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93bc19da-pjg99"] Nov 03 01:20:09.545527 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:09.545491 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93bc19da-pjg99" event={"ID":"752f8f24-ac91-4fb6-96ea-963146efe4c9","Type":"ContainerStarted","Data":"b8e058573007562eaa6b498412317df8afc18a2ffd0f67cd8524e24c8d2d08d8"} Nov 03 01:20:10.613236 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:10.613185 4421 generic.go:356] "Generic (PLEG): container finished" podID="752f8f24-ac91-4fb6-96ea-963146efe4c9" containerID="7294a8fb4ac1c42480286882a52f096192e4890a9287517d06e90b32b4807eca" exitCode=0 Nov 03 01:20:10.613677 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:10.613250 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93bc19da-pjg99" event={"ID":"752f8f24-ac91-4fb6-96ea-963146efe4c9","Type":"ContainerDied","Data":"7294a8fb4ac1c42480286882a52f096192e4890a9287517d06e90b32b4807eca"} Nov 03 01:20:11.811376 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:11.811346 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93bc19da-pjg99" Nov 03 01:20:11.920560 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:11.904744 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/752f8f24-ac91-4fb6-96ea-963146efe4c9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/752f8f24-ac91-4fb6-96ea-963146efe4c9-maas-api-openshift-ca\") pod \"752f8f24-ac91-4fb6-96ea-963146efe4c9\" (UID: \"752f8f24-ac91-4fb6-96ea-963146efe4c9\") " Nov 03 01:20:11.920560 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:11.904901 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/752f8f24-ac91-4fb6-96ea-963146efe4c9-kube-api-access-6wjt6\" (UniqueName: \"kubernetes.io/projected/752f8f24-ac91-4fb6-96ea-963146efe4c9-kube-api-access-6wjt6\") pod \"752f8f24-ac91-4fb6-96ea-963146efe4c9\" (UID: \"752f8f24-ac91-4fb6-96ea-963146efe4c9\") " Nov 03 01:20:11.920560 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:11.905100 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/752f8f24-ac91-4fb6-96ea-963146efe4c9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/752f8f24-ac91-4fb6-96ea-963146efe4c9-maas-api-xks-ca\") pod \"752f8f24-ac91-4fb6-96ea-963146efe4c9\" (UID: \"752f8f24-ac91-4fb6-96ea-963146efe4c9\") " Nov 03 01:20:11.922415 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:11.922375 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/752f8f24-ac91-4fb6-96ea-963146efe4c9-maas-api-openshift-ca" pod "752f8f24-ac91-4fb6-96ea-963146efe4c9" (UID: "752f8f24-ac91-4fb6-96ea-963146efe4c9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:20:11.939921 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:11.939877 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/752f8f24-ac91-4fb6-96ea-963146efe4c9-kube-api-access-6wjt6" pod "752f8f24-ac91-4fb6-96ea-963146efe4c9" (UID: "752f8f24-ac91-4fb6-96ea-963146efe4c9"). InnerVolumeSpecName "kube-api-access-6wjt6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:20:11.947914 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:11.947855 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/752f8f24-ac91-4fb6-96ea-963146efe4c9-maas-api-xks-ca" pod "752f8f24-ac91-4fb6-96ea-963146efe4c9" (UID: "752f8f24-ac91-4fb6-96ea-963146efe4c9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:20:12.008592 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:12.008558 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/752f8f24-ac91-4fb6-96ea-963146efe4c9-maas-api-xks-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:20:12.008761 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:12.008597 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/752f8f24-ac91-4fb6-96ea-963146efe4c9-maas-api-openshift-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:20:12.008761 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:12.008615 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6wjt6\" (UniqueName: \"kubernetes.io/projected/752f8f24-ac91-4fb6-96ea-963146efe4c9-kube-api-access-6wjt6\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:20:12.647643 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:12.647591 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93bc19da-pjg99" event={"ID":"752f8f24-ac91-4fb6-96ea-963146efe4c9","Type":"ContainerDied","Data":"b8e058573007562eaa6b498412317df8afc18a2ffd0f67cd8524e24c8d2d08d8"} Nov 03 01:20:12.647978 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:12.647659 4421 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b8e058573007562eaa6b498412317df8afc18a2ffd0f67cd8524e24c8d2d08d8" Nov 03 01:20:12.647978 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:12.647731 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93bc19da-pjg99" Nov 03 01:20:14.686056 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:20:14.686003 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 01:20:18.361977 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:18.361933 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-deb4582f-qv4p4"] Nov 03 01:20:18.362930 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:18.362891 4421 cpu_manager.go:414] "containerMap: removing container" podUID="752f8f24-ac91-4fb6-96ea-963146efe4c9" containerName="revoke-keys" Nov 03 01:20:18.362930 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:18.362913 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="752f8f24-ac91-4fb6-96ea-963146efe4c9" containerName="revoke-keys" Nov 03 01:20:18.363138 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:18.363058 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="752f8f24-ac91-4fb6-96ea-963146efe4c9" containerName="revoke-keys" Nov 03 01:20:18.367918 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:18.367894 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-deb4582f-qv4p4" Nov 03 01:20:18.373072 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:18.373044 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:20:18.373215 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:18.373109 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6q4xp\"" Nov 03 01:20:18.374872 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:18.374846 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-deb4582f-qv4p4"] Nov 03 01:20:18.467300 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:18.455890 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrlj2\" (UniqueName: \"kubernetes.io/projected/a4a328ec-02f5-4564-a262-97a89dc7dd85-kube-api-access-mrlj2\") pod \"maas-api-revoke-sub-deb4582f-qv4p4\" (UID: \"a4a328ec-02f5-4564-a262-97a89dc7dd85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-deb4582f-qv4p4" Nov 03 01:20:18.467300 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:18.456173 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4a328ec-02f5-4564-a262-97a89dc7dd85-maas-api-xks-ca\") pod \"maas-api-revoke-sub-deb4582f-qv4p4\" (UID: \"a4a328ec-02f5-4564-a262-97a89dc7dd85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-deb4582f-qv4p4" Nov 03 01:20:18.467300 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:18.456216 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4a328ec-02f5-4564-a262-97a89dc7dd85-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-deb4582f-qv4p4\" (UID: \"a4a328ec-02f5-4564-a262-97a89dc7dd85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-deb4582f-qv4p4" Nov 03 01:20:18.558797 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:18.558746 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4a328ec-02f5-4564-a262-97a89dc7dd85-maas-api-xks-ca\") pod \"maas-api-revoke-sub-deb4582f-qv4p4\" (UID: \"a4a328ec-02f5-4564-a262-97a89dc7dd85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-deb4582f-qv4p4" Nov 03 01:20:18.560709 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:18.558808 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4a328ec-02f5-4564-a262-97a89dc7dd85-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-deb4582f-qv4p4\" (UID: \"a4a328ec-02f5-4564-a262-97a89dc7dd85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-deb4582f-qv4p4" Nov 03 01:20:18.560709 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:18.559069 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrlj2\" (UniqueName: \"kubernetes.io/projected/a4a328ec-02f5-4564-a262-97a89dc7dd85-kube-api-access-mrlj2\") pod \"maas-api-revoke-sub-deb4582f-qv4p4\" (UID: \"a4a328ec-02f5-4564-a262-97a89dc7dd85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-deb4582f-qv4p4" Nov 03 01:20:18.562848 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:18.562821 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4a328ec-02f5-4564-a262-97a89dc7dd85-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-deb4582f-qv4p4\" (UID: \"a4a328ec-02f5-4564-a262-97a89dc7dd85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-deb4582f-qv4p4" Nov 03 01:20:18.563923 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:18.563892 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4a328ec-02f5-4564-a262-97a89dc7dd85-maas-api-xks-ca\") pod \"maas-api-revoke-sub-deb4582f-qv4p4\" (UID: \"a4a328ec-02f5-4564-a262-97a89dc7dd85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-deb4582f-qv4p4" Nov 03 01:20:18.570578 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:18.570545 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrlj2\" (UniqueName: \"kubernetes.io/projected/a4a328ec-02f5-4564-a262-97a89dc7dd85-kube-api-access-mrlj2\") pod \"maas-api-revoke-sub-deb4582f-qv4p4\" (UID: \"a4a328ec-02f5-4564-a262-97a89dc7dd85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-deb4582f-qv4p4" Nov 03 01:20:18.696487 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:18.696393 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-deb4582f-qv4p4" Nov 03 01:20:18.985906 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:20:18.985877 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4a328ec_02f5_4564_a262_97a89dc7dd85.slice/crio-c64bf2718f8621fa679bce14f0da68e931a77aeea256e69851304bad47d8e892 WatchSource:0}: Error finding container c64bf2718f8621fa679bce14f0da68e931a77aeea256e69851304bad47d8e892: Status 404 returned error can't find the container with id c64bf2718f8621fa679bce14f0da68e931a77aeea256e69851304bad47d8e892 Nov 03 01:20:18.986067 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:18.985939 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-deb4582f-qv4p4"] Nov 03 01:20:19.808996 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:19.808944 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-deb4582f-qv4p4" event={"ID":"a4a328ec-02f5-4564-a262-97a89dc7dd85","Type":"ContainerStarted","Data":"624bbe5e7f548037fd864506e64b16716c24acb5ec5ca8cf95559220f85bab39"} Nov 03 01:20:19.809469 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:19.809068 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-deb4582f-qv4p4" event={"ID":"a4a328ec-02f5-4564-a262-97a89dc7dd85","Type":"ContainerStarted","Data":"c64bf2718f8621fa679bce14f0da68e931a77aeea256e69851304bad47d8e892"} Nov 03 01:20:19.829765 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:19.829711 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-deb4582f-qv4p4" podStartSLOduration=1.829695462 podStartE2EDuration="1.829695462s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:20:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:20:19.825909716 +0000 UTC m=+2361.996912813" watchObservedRunningTime="2026-11-03 01:20:19.829695462 +0000 UTC m=+2362.000698557" Nov 03 01:20:20.905896 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:20.905857 4421 generic.go:356] "Generic (PLEG): container finished" podID="a4a328ec-02f5-4564-a262-97a89dc7dd85" containerID="624bbe5e7f548037fd864506e64b16716c24acb5ec5ca8cf95559220f85bab39" exitCode=0 Nov 03 01:20:20.907006 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:20.906887 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-deb4582f-qv4p4" event={"ID":"a4a328ec-02f5-4564-a262-97a89dc7dd85","Type":"ContainerDied","Data":"624bbe5e7f548037fd864506e64b16716c24acb5ec5ca8cf95559220f85bab39"} Nov 03 01:20:22.102736 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:22.102709 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-deb4582f-qv4p4" Nov 03 01:20:22.153862 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:22.153828 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4a328ec-02f5-4564-a262-97a89dc7dd85-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4a328ec-02f5-4564-a262-97a89dc7dd85-maas-api-openshift-ca\") pod \"a4a328ec-02f5-4564-a262-97a89dc7dd85\" (UID: \"a4a328ec-02f5-4564-a262-97a89dc7dd85\") " Nov 03 01:20:22.154147 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:22.154121 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a4a328ec-02f5-4564-a262-97a89dc7dd85-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4a328ec-02f5-4564-a262-97a89dc7dd85-maas-api-xks-ca\") pod \"a4a328ec-02f5-4564-a262-97a89dc7dd85\" (UID: \"a4a328ec-02f5-4564-a262-97a89dc7dd85\") " Nov 03 01:20:22.154258 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:22.154177 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4a328ec-02f5-4564-a262-97a89dc7dd85-kube-api-access-mrlj2\" (UniqueName: \"kubernetes.io/projected/a4a328ec-02f5-4564-a262-97a89dc7dd85-kube-api-access-mrlj2\") pod \"a4a328ec-02f5-4564-a262-97a89dc7dd85\" (UID: \"a4a328ec-02f5-4564-a262-97a89dc7dd85\") " Nov 03 01:20:22.154258 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:22.154235 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4a328ec-02f5-4564-a262-97a89dc7dd85-maas-api-openshift-ca" pod "a4a328ec-02f5-4564-a262-97a89dc7dd85" (UID: "a4a328ec-02f5-4564-a262-97a89dc7dd85"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:20:22.154999 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:22.154971 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4a328ec-02f5-4564-a262-97a89dc7dd85-maas-api-openshift-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:20:22.156997 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:22.156963 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4a328ec-02f5-4564-a262-97a89dc7dd85-maas-api-xks-ca" pod "a4a328ec-02f5-4564-a262-97a89dc7dd85" (UID: "a4a328ec-02f5-4564-a262-97a89dc7dd85"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:20:22.157169 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:22.157148 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4a328ec-02f5-4564-a262-97a89dc7dd85-kube-api-access-mrlj2" pod "a4a328ec-02f5-4564-a262-97a89dc7dd85" (UID: "a4a328ec-02f5-4564-a262-97a89dc7dd85"). InnerVolumeSpecName "kube-api-access-mrlj2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:20:22.256771 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:22.256735 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4a328ec-02f5-4564-a262-97a89dc7dd85-maas-api-xks-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:20:22.256771 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:22.256771 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mrlj2\" (UniqueName: \"kubernetes.io/projected/a4a328ec-02f5-4564-a262-97a89dc7dd85-kube-api-access-mrlj2\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:20:22.926876 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:22.926831 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-deb4582f-qv4p4" event={"ID":"a4a328ec-02f5-4564-a262-97a89dc7dd85","Type":"ContainerDied","Data":"c64bf2718f8621fa679bce14f0da68e931a77aeea256e69851304bad47d8e892"} Nov 03 01:20:22.926876 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:22.926871 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-deb4582f-qv4p4" Nov 03 01:20:22.927113 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:22.926889 4421 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c64bf2718f8621fa679bce14f0da68e931a77aeea256e69851304bad47d8e892" Nov 03 01:20:32.012380 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:32.012338 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-kr275"] Nov 03 01:20:32.014115 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:32.013598 4421 cpu_manager.go:414] "containerMap: removing container" podUID="a4a328ec-02f5-4564-a262-97a89dc7dd85" containerName="revoke-keys" Nov 03 01:20:32.014115 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:32.013622 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4a328ec-02f5-4564-a262-97a89dc7dd85" containerName="revoke-keys" Nov 03 01:20:32.014115 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:32.013783 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4a328ec-02f5-4564-a262-97a89dc7dd85" containerName="revoke-keys" Nov 03 01:20:32.018088 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:32.018065 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-kr275" Nov 03 01:20:32.032974 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:32.032938 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-kr275"] Nov 03 01:20:32.104770 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:32.104735 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e73ad9e-cd24-4d53-82f0-2e521cae7445-istio-podinfo\") pod \"maas-default-gateway-openshift-default-596dc9545f-kr275\" (UID: \"3e73ad9e-cd24-4d53-82f0-2e521cae7445\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-kr275" Nov 03 01:20:32.104997 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:32.104896 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e73ad9e-cd24-4d53-82f0-2e521cae7445-istio-token\") pod \"maas-default-gateway-openshift-default-596dc9545f-kr275\" (UID: \"3e73ad9e-cd24-4d53-82f0-2e521cae7445\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-kr275" Nov 03 01:20:32.105135 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:32.105115 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e73ad9e-cd24-4d53-82f0-2e521cae7445-workload-certs\") pod \"maas-default-gateway-openshift-default-596dc9545f-kr275\" (UID: \"3e73ad9e-cd24-4d53-82f0-2e521cae7445\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-kr275" Nov 03 01:20:32.105307 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:32.105162 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e73ad9e-cd24-4d53-82f0-2e521cae7445-istio-data\") pod \"maas-default-gateway-openshift-default-596dc9545f-kr275\" (UID: \"3e73ad9e-cd24-4d53-82f0-2e521cae7445\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-kr275" Nov 03 01:20:32.105403 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:32.105381 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b28kt\" (UniqueName: \"kubernetes.io/projected/3e73ad9e-cd24-4d53-82f0-2e521cae7445-kube-api-access-b28kt\") pod \"maas-default-gateway-openshift-default-596dc9545f-kr275\" (UID: \"3e73ad9e-cd24-4d53-82f0-2e521cae7445\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-kr275" Nov 03 01:20:32.105526 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:32.105510 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e73ad9e-cd24-4d53-82f0-2e521cae7445-istio-envoy\") pod \"maas-default-gateway-openshift-default-596dc9545f-kr275\" (UID: \"3e73ad9e-cd24-4d53-82f0-2e521cae7445\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-kr275" Nov 03 01:20:32.105603 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:32.105589 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e73ad9e-cd24-4d53-82f0-2e521cae7445-credential-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-kr275\" (UID: \"3e73ad9e-cd24-4d53-82f0-2e521cae7445\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-kr275" Nov 03 01:20:32.105694 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:32.105679 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e73ad9e-cd24-4d53-82f0-2e521cae7445-workload-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-kr275\" (UID: \"3e73ad9e-cd24-4d53-82f0-2e521cae7445\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-kr275" Nov 03 01:20:32.105802 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:32.105781 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e73ad9e-cd24-4d53-82f0-2e521cae7445-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-596dc9545f-kr275\" (UID: \"3e73ad9e-cd24-4d53-82f0-2e521cae7445\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-kr275" Nov 03 01:20:32.207887 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:32.207839 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e73ad9e-cd24-4d53-82f0-2e521cae7445-istio-podinfo\") pod \"maas-default-gateway-openshift-default-596dc9545f-kr275\" (UID: \"3e73ad9e-cd24-4d53-82f0-2e521cae7445\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-kr275" Nov 03 01:20:32.208080 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:32.207944 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e73ad9e-cd24-4d53-82f0-2e521cae7445-istio-token\") pod \"maas-default-gateway-openshift-default-596dc9545f-kr275\" (UID: \"3e73ad9e-cd24-4d53-82f0-2e521cae7445\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-kr275" Nov 03 01:20:32.208080 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:32.208007 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e73ad9e-cd24-4d53-82f0-2e521cae7445-workload-certs\") pod \"maas-default-gateway-openshift-default-596dc9545f-kr275\" (UID: \"3e73ad9e-cd24-4d53-82f0-2e521cae7445\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-kr275" Nov 03 01:20:32.208080 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:32.208037 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e73ad9e-cd24-4d53-82f0-2e521cae7445-istio-data\") pod \"maas-default-gateway-openshift-default-596dc9545f-kr275\" (UID: \"3e73ad9e-cd24-4d53-82f0-2e521cae7445\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-kr275" Nov 03 01:20:32.209018 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:32.208101 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b28kt\" (UniqueName: \"kubernetes.io/projected/3e73ad9e-cd24-4d53-82f0-2e521cae7445-kube-api-access-b28kt\") pod \"maas-default-gateway-openshift-default-596dc9545f-kr275\" (UID: \"3e73ad9e-cd24-4d53-82f0-2e521cae7445\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-kr275" Nov 03 01:20:32.209018 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:32.208216 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e73ad9e-cd24-4d53-82f0-2e521cae7445-istio-envoy\") pod \"maas-default-gateway-openshift-default-596dc9545f-kr275\" (UID: \"3e73ad9e-cd24-4d53-82f0-2e521cae7445\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-kr275" Nov 03 01:20:32.209018 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:32.208339 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e73ad9e-cd24-4d53-82f0-2e521cae7445-credential-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-kr275\" (UID: \"3e73ad9e-cd24-4d53-82f0-2e521cae7445\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-kr275" Nov 03 01:20:32.209018 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:32.208463 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e73ad9e-cd24-4d53-82f0-2e521cae7445-workload-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-kr275\" (UID: \"3e73ad9e-cd24-4d53-82f0-2e521cae7445\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-kr275" Nov 03 01:20:32.209018 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:32.208499 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e73ad9e-cd24-4d53-82f0-2e521cae7445-workload-certs\") pod \"maas-default-gateway-openshift-default-596dc9545f-kr275\" (UID: \"3e73ad9e-cd24-4d53-82f0-2e521cae7445\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-kr275" Nov 03 01:20:32.209018 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:32.208518 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e73ad9e-cd24-4d53-82f0-2e521cae7445-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-596dc9545f-kr275\" (UID: \"3e73ad9e-cd24-4d53-82f0-2e521cae7445\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-kr275" Nov 03 01:20:32.209018 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:32.208528 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e73ad9e-cd24-4d53-82f0-2e521cae7445-istio-data\") pod \"maas-default-gateway-openshift-default-596dc9545f-kr275\" (UID: \"3e73ad9e-cd24-4d53-82f0-2e521cae7445\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-kr275" Nov 03 01:20:32.209018 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:32.208709 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e73ad9e-cd24-4d53-82f0-2e521cae7445-credential-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-kr275\" (UID: \"3e73ad9e-cd24-4d53-82f0-2e521cae7445\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-kr275" Nov 03 01:20:32.209018 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:32.208872 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e73ad9e-cd24-4d53-82f0-2e521cae7445-workload-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-kr275\" (UID: \"3e73ad9e-cd24-4d53-82f0-2e521cae7445\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-kr275" Nov 03 01:20:32.209376 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:32.209127 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e73ad9e-cd24-4d53-82f0-2e521cae7445-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-596dc9545f-kr275\" (UID: \"3e73ad9e-cd24-4d53-82f0-2e521cae7445\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-kr275" Nov 03 01:20:32.210929 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:32.210901 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e73ad9e-cd24-4d53-82f0-2e521cae7445-istio-envoy\") pod \"maas-default-gateway-openshift-default-596dc9545f-kr275\" (UID: \"3e73ad9e-cd24-4d53-82f0-2e521cae7445\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-kr275" Nov 03 01:20:32.211138 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:32.211118 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e73ad9e-cd24-4d53-82f0-2e521cae7445-istio-podinfo\") pod \"maas-default-gateway-openshift-default-596dc9545f-kr275\" (UID: \"3e73ad9e-cd24-4d53-82f0-2e521cae7445\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-kr275" Nov 03 01:20:32.215678 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:32.215648 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e73ad9e-cd24-4d53-82f0-2e521cae7445-istio-token\") pod \"maas-default-gateway-openshift-default-596dc9545f-kr275\" (UID: \"3e73ad9e-cd24-4d53-82f0-2e521cae7445\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-kr275" Nov 03 01:20:32.215966 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:32.215947 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b28kt\" (UniqueName: \"kubernetes.io/projected/3e73ad9e-cd24-4d53-82f0-2e521cae7445-kube-api-access-b28kt\") pod \"maas-default-gateway-openshift-default-596dc9545f-kr275\" (UID: \"3e73ad9e-cd24-4d53-82f0-2e521cae7445\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-kr275" Nov 03 01:20:32.357176 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:32.357080 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-kr275" Nov 03 01:20:32.648073 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:20:32.648040 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e73ad9e_cd24_4d53_82f0_2e521cae7445.slice/crio-b4fe10a7a4903f84438af123d7b7a0e3683358999e271afb49aa168d9963bde5 WatchSource:0}: Error finding container b4fe10a7a4903f84438af123d7b7a0e3683358999e271afb49aa168d9963bde5: Status 404 returned error can't find the container with id b4fe10a7a4903f84438af123d7b7a0e3683358999e271afb49aa168d9963bde5 Nov 03 01:20:32.650501 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:32.650470 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-kr275"] Nov 03 01:20:32.651478 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:32.651440 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:20:32.651538 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:32.651517 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:20:32.651580 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:32.651548 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:20:33.205152 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:33.205110 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-kr275" event={"ID":"3e73ad9e-cd24-4d53-82f0-2e521cae7445","Type":"ContainerStarted","Data":"68db61c64e6bf7544ccda78e99bcbe9861c7d45f26b4665ea4f79e8396b160bd"} Nov 03 01:20:33.205803 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:33.205210 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-kr275" event={"ID":"3e73ad9e-cd24-4d53-82f0-2e521cae7445","Type":"ContainerStarted","Data":"b4fe10a7a4903f84438af123d7b7a0e3683358999e271afb49aa168d9963bde5"} Nov 03 01:20:33.232152 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:33.231047 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-kr275" podStartSLOduration=2.231027503 podStartE2EDuration="2.231027503s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:20:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:20:33.228361111 +0000 UTC m=+2375.399364207" watchObservedRunningTime="2026-11-03 01:20:33.231027503 +0000 UTC m=+2375.402030600" Nov 03 01:20:33.375673 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:33.362897 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-kr275" Nov 03 01:20:33.381024 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:33.380984 4421 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-kr275 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.41:15021/healthz/ready\": dial tcp 10.134.1.41:15021: connect: connection refused" start-of-body= Nov 03 01:20:33.381227 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:33.381043 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-kr275" podUID="3e73ad9e-cd24-4d53-82f0-2e521cae7445" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.41:15021/healthz/ready\": dial tcp 10.134.1.41:15021: connect: connection refused" Nov 03 01:20:34.357599 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:34.357564 4421 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-kr275 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.41:15021/healthz/ready\": dial tcp 10.134.1.41:15021: connect: connection refused" start-of-body= Nov 03 01:20:34.358111 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:34.357624 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-kr275" podUID="3e73ad9e-cd24-4d53-82f0-2e521cae7445" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.41:15021/healthz/ready\": dial tcp 10.134.1.41:15021: connect: connection refused" Nov 03 01:20:35.357428 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:35.357391 4421 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-kr275 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.41:15021/healthz/ready\": dial tcp 10.134.1.41:15021: connect: connection refused" start-of-body= Nov 03 01:20:35.373622 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:35.373582 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-kr275" podUID="3e73ad9e-cd24-4d53-82f0-2e521cae7445" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.41:15021/healthz/ready\": dial tcp 10.134.1.41:15021: connect: connection refused" Nov 03 01:20:36.357845 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:36.357807 4421 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-kr275 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.41:15021/healthz/ready\": dial tcp 10.134.1.41:15021: connect: connection refused" start-of-body= Nov 03 01:20:36.358078 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:36.357874 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-kr275" podUID="3e73ad9e-cd24-4d53-82f0-2e521cae7445" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.41:15021/healthz/ready\": dial tcp 10.134.1.41:15021: connect: connection refused" Nov 03 01:20:37.357699 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:37.357661 4421 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-kr275 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.41:15021/healthz/ready\": dial tcp 10.134.1.41:15021: connect: connection refused" start-of-body= Nov 03 01:20:37.358113 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:37.357720 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-kr275" podUID="3e73ad9e-cd24-4d53-82f0-2e521cae7445" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.41:15021/healthz/ready\": dial tcp 10.134.1.41:15021: connect: connection refused" Nov 03 01:20:38.357515 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:38.357482 4421 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-kr275 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.41:15021/healthz/ready\": dial tcp 10.134.1.41:15021: connect: connection refused" start-of-body= Nov 03 01:20:38.357723 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:38.357545 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-kr275" podUID="3e73ad9e-cd24-4d53-82f0-2e521cae7445" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.41:15021/healthz/ready\": dial tcp 10.134.1.41:15021: connect: connection refused" Nov 03 01:20:39.340223 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:39.340185 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-5cb6nvdxn"] Nov 03 01:20:39.346556 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:39.346533 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-5cb6nvdxn" Nov 03 01:20:39.349371 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:39.349345 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-5cb6nvdxn"] Nov 03 01:20:39.352705 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:39.352680 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-57079b\"/\"agc-cred-a6687acf616c\"" Nov 03 01:20:39.353114 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:39.353091 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-57079b\"/\"routing-overlay\"" Nov 03 01:20:39.353265 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:39.353128 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-57079b\"/\"payload-processing-external-model-plugins-e2e-extmod-b-57079b\"" Nov 03 01:20:39.353265 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:39.353149 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-57079b\"/\"payload-processing-external-model-e2e--4a3ed774-dockercfg-5lwqz\"" Nov 03 01:20:39.353453 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:39.353141 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-57079b\"/\"openshift-service-ca.crt\"" Nov 03 01:20:39.353453 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:39.353185 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-57079b\"/\"kube-root-ca.crt\"" Nov 03 01:20:39.358883 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:39.358854 4421 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-kr275 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.41:15021/healthz/ready\": dial tcp 10.134.1.41:15021: connect: connection refused" start-of-body= Nov 03 01:20:39.359357 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:39.358905 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-kr275" podUID="3e73ad9e-cd24-4d53-82f0-2e521cae7445" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.41:15021/healthz/ready\": dial tcp 10.134.1.41:15021: connect: connection refused" Nov 03 01:20:39.435874 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:39.435829 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cd2d7685-32b0-4bb5-bc90-bbe0d0844c26-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-57079b-5cb6nvdxn\" (UID: \"cd2d7685-32b0-4bb5-bc90-bbe0d0844c26\") " pod="ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-5cb6nvdxn" Nov 03 01:20:39.436250 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:39.436227 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pttdc\" (UniqueName: \"kubernetes.io/projected/cd2d7685-32b0-4bb5-bc90-bbe0d0844c26-kube-api-access-pttdc\") pod \"payload-processing-external-model-e2e-extmod-b-57079b-5cb6nvdxn\" (UID: \"cd2d7685-32b0-4bb5-bc90-bbe0d0844c26\") " pod="ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-5cb6nvdxn" Nov 03 01:20:39.436445 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:39.436331 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd2d7685-32b0-4bb5-bc90-bbe0d0844c26-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-57079b-5cb6nvdxn\" (UID: \"cd2d7685-32b0-4bb5-bc90-bbe0d0844c26\") " pod="ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-5cb6nvdxn" Nov 03 01:20:39.436524 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:39.436469 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cd2d7685-32b0-4bb5-bc90-bbe0d0844c26-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-57079b-5cb6nvdxn\" (UID: \"cd2d7685-32b0-4bb5-bc90-bbe0d0844c26\") " pod="ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-5cb6nvdxn" Nov 03 01:20:39.542667 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:39.542630 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pttdc\" (UniqueName: \"kubernetes.io/projected/cd2d7685-32b0-4bb5-bc90-bbe0d0844c26-kube-api-access-pttdc\") pod \"payload-processing-external-model-e2e-extmod-b-57079b-5cb6nvdxn\" (UID: \"cd2d7685-32b0-4bb5-bc90-bbe0d0844c26\") " pod="ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-5cb6nvdxn" Nov 03 01:20:39.543417 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:39.543385 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd2d7685-32b0-4bb5-bc90-bbe0d0844c26-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-57079b-5cb6nvdxn\" (UID: \"cd2d7685-32b0-4bb5-bc90-bbe0d0844c26\") " pod="ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-5cb6nvdxn" Nov 03 01:20:39.544870 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:39.544848 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cd2d7685-32b0-4bb5-bc90-bbe0d0844c26-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-57079b-5cb6nvdxn\" (UID: \"cd2d7685-32b0-4bb5-bc90-bbe0d0844c26\") " pod="ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-5cb6nvdxn" Nov 03 01:20:39.545413 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:39.545390 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cd2d7685-32b0-4bb5-bc90-bbe0d0844c26-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-57079b-5cb6nvdxn\" (UID: \"cd2d7685-32b0-4bb5-bc90-bbe0d0844c26\") " pod="ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-5cb6nvdxn" Nov 03 01:20:39.546090 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:39.544692 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd2d7685-32b0-4bb5-bc90-bbe0d0844c26-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-57079b-5cb6nvdxn\" (UID: \"cd2d7685-32b0-4bb5-bc90-bbe0d0844c26\") " pod="ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-5cb6nvdxn" Nov 03 01:20:39.546666 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:39.546610 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cd2d7685-32b0-4bb5-bc90-bbe0d0844c26-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-57079b-5cb6nvdxn\" (UID: \"cd2d7685-32b0-4bb5-bc90-bbe0d0844c26\") " pod="ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-5cb6nvdxn" Nov 03 01:20:39.651382 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:39.584822 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cd2d7685-32b0-4bb5-bc90-bbe0d0844c26-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-57079b-5cb6nvdxn\" (UID: \"cd2d7685-32b0-4bb5-bc90-bbe0d0844c26\") " pod="ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-5cb6nvdxn" Nov 03 01:20:39.696297 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:39.672992 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pttdc\" (UniqueName: \"kubernetes.io/projected/cd2d7685-32b0-4bb5-bc90-bbe0d0844c26-kube-api-access-pttdc\") pod \"payload-processing-external-model-e2e-extmod-b-57079b-5cb6nvdxn\" (UID: \"cd2d7685-32b0-4bb5-bc90-bbe0d0844c26\") " pod="ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-5cb6nvdxn" Nov 03 01:20:39.716312 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:39.709630 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-5cb6nvdxn" Nov 03 01:20:40.039256 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:40.039227 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e85f3c00-prm5z"] Nov 03 01:20:40.088092 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:40.088053 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e85f3c00-prm5z"] Nov 03 01:20:40.259169 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:40.259130 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-5cb6nvdxn"] Nov 03 01:20:40.259971 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:20:40.259937 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd2d7685_32b0_4bb5_bc90_bbe0d0844c26.slice/crio-38ebb60dbf69355b058754d1b53710c34176a2f11221971e1b44f7fc9ea47e1d WatchSource:0}: Error finding container 38ebb60dbf69355b058754d1b53710c34176a2f11221971e1b44f7fc9ea47e1d: Status 404 returned error can't find the container with id 38ebb60dbf69355b058754d1b53710c34176a2f11221971e1b44f7fc9ea47e1d Nov 03 01:20:40.300613 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:40.300568 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-5cb6nvdxn" event={"ID":"cd2d7685-32b0-4bb5-bc90-bbe0d0844c26","Type":"ContainerStarted","Data":"38ebb60dbf69355b058754d1b53710c34176a2f11221971e1b44f7fc9ea47e1d"} Nov 03 01:20:40.359548 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:40.359207 4421 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-kr275 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.41:15021/healthz/ready\": dial tcp 10.134.1.41:15021: connect: connection refused" start-of-body= Nov 03 01:20:40.359548 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:40.359306 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-kr275" podUID="3e73ad9e-cd24-4d53-82f0-2e521cae7445" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.41:15021/healthz/ready\": dial tcp 10.134.1.41:15021: connect: connection refused" Nov 03 01:20:41.312347 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:41.312311 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-5cb6nvdxn" event={"ID":"cd2d7685-32b0-4bb5-bc90-bbe0d0844c26","Type":"ContainerStarted","Data":"1941358fef6cceea1e8f8bd50103e9b0fdbfb0b3d4d740814e29807968dcca88"} Nov 03 01:20:41.312554 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:41.312387 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-5cb6nvdxn" Nov 03 01:20:41.337154 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:41.337087 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-5cb6nvdxn" podStartSLOduration=2.3370699200000002 podStartE2EDuration="2.33706992s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:20:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:20:41.328585928 +0000 UTC m=+2383.499589025" watchObservedRunningTime="2026-11-03 01:20:41.33706992 +0000 UTC m=+2383.508073017" Nov 03 01:20:41.358139 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:41.358099 4421 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-kr275 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.41:15021/healthz/ready\": dial tcp 10.134.1.41:15021: connect: connection refused" start-of-body= Nov 03 01:20:41.358385 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:41.358164 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-kr275" podUID="3e73ad9e-cd24-4d53-82f0-2e521cae7445" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.41:15021/healthz/ready\": dial tcp 10.134.1.41:15021: connect: connection refused" Nov 03 01:20:42.073518 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:42.073461 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="613397a3-4bde-4b90-ad05-e73fff87a5fa" path="/var/lib/kubelet/pods/613397a3-4bde-4b90-ad05-e73fff87a5fa/volumes" Nov 03 01:20:43.381300 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:43.380402 4421 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-kr275 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 01:20:43.381300 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:43.380521 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-kr275" podUID="3e73ad9e-cd24-4d53-82f0-2e521cae7445" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 01:20:43.705347 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:43.705247 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-kr275" Nov 03 01:20:43.705808 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:43.705790 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-kr275" Nov 03 01:20:43.706708 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:43.706689 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-kr275" Nov 03 01:20:44.023612 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:44.023580 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f933cbe-wz7pk"] Nov 03 01:20:44.028636 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:44.028612 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f933cbe-wz7pk"] Nov 03 01:20:44.673684 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:20:44.673642 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 01:20:46.025350 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:46.011895 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b673d7f4-7e8e-4573-8208-c9ca7fe1b3f6" path="/var/lib/kubelet/pods/b673d7f4-7e8e-4573-8208-c9ca7fe1b3f6/volumes" Nov 03 01:20:51.026152 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:51.026113 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf8dfc0e-z8qwx"] Nov 03 01:20:51.031583 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:51.031548 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf8dfc0e-z8qwx"] Nov 03 01:20:51.982925 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:51.982892 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="adda682e-735e-42a1-ac06-80e472ae394a" path="/var/lib/kubelet/pods/adda682e-735e-42a1-ac06-80e472ae394a/volumes" Nov 03 01:20:52.361125 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:52.361095 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-5cb6nvdxn" Nov 03 01:20:52.430642 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:52.430604 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-027c19-openshift-default-58d5bc7645-6jphv"] Nov 03 01:20:52.440393 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:52.440349 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-027c19-openshift-default-58d5bc7645-6jphv" Nov 03 01:20:52.443898 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:52.443863 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-027c19-openshift-default-58d5bc7645-6jphv"] Nov 03 01:20:52.444482 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:52.444454 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-027c19-openshift-default-dockercfg-6n7ft\"" Nov 03 01:20:52.585192 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:52.585153 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4fa33da-32e1-40cb-b89d-8b25ab268fb0-workload-certs\") pod \"e2e-extmod-a-027c19-openshift-default-58d5bc7645-6jphv\" (UID: \"c4fa33da-32e1-40cb-b89d-8b25ab268fb0\") " pod="openshift-ingress/e2e-extmod-a-027c19-openshift-default-58d5bc7645-6jphv" Nov 03 01:20:52.585420 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:52.585338 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4fa33da-32e1-40cb-b89d-8b25ab268fb0-istio-podinfo\") pod \"e2e-extmod-a-027c19-openshift-default-58d5bc7645-6jphv\" (UID: \"c4fa33da-32e1-40cb-b89d-8b25ab268fb0\") " pod="openshift-ingress/e2e-extmod-a-027c19-openshift-default-58d5bc7645-6jphv" Nov 03 01:20:52.585420 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:52.585390 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4fa33da-32e1-40cb-b89d-8b25ab268fb0-workload-socket\") pod \"e2e-extmod-a-027c19-openshift-default-58d5bc7645-6jphv\" (UID: \"c4fa33da-32e1-40cb-b89d-8b25ab268fb0\") " pod="openshift-ingress/e2e-extmod-a-027c19-openshift-default-58d5bc7645-6jphv" Nov 03 01:20:52.585601 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:52.585450 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4fa33da-32e1-40cb-b89d-8b25ab268fb0-istio-token\") pod \"e2e-extmod-a-027c19-openshift-default-58d5bc7645-6jphv\" (UID: \"c4fa33da-32e1-40cb-b89d-8b25ab268fb0\") " pod="openshift-ingress/e2e-extmod-a-027c19-openshift-default-58d5bc7645-6jphv" Nov 03 01:20:52.585709 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:52.585597 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwpn5\" (UniqueName: \"kubernetes.io/projected/c4fa33da-32e1-40cb-b89d-8b25ab268fb0-kube-api-access-pwpn5\") pod \"e2e-extmod-a-027c19-openshift-default-58d5bc7645-6jphv\" (UID: \"c4fa33da-32e1-40cb-b89d-8b25ab268fb0\") " pod="openshift-ingress/e2e-extmod-a-027c19-openshift-default-58d5bc7645-6jphv" Nov 03 01:20:52.585709 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:52.585633 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4fa33da-32e1-40cb-b89d-8b25ab268fb0-credential-socket\") pod \"e2e-extmod-a-027c19-openshift-default-58d5bc7645-6jphv\" (UID: \"c4fa33da-32e1-40cb-b89d-8b25ab268fb0\") " pod="openshift-ingress/e2e-extmod-a-027c19-openshift-default-58d5bc7645-6jphv" Nov 03 01:20:52.585841 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:52.585704 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4fa33da-32e1-40cb-b89d-8b25ab268fb0-istio-data\") pod \"e2e-extmod-a-027c19-openshift-default-58d5bc7645-6jphv\" (UID: \"c4fa33da-32e1-40cb-b89d-8b25ab268fb0\") " pod="openshift-ingress/e2e-extmod-a-027c19-openshift-default-58d5bc7645-6jphv" Nov 03 01:20:52.585941 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:52.585914 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4fa33da-32e1-40cb-b89d-8b25ab268fb0-istiod-ca-cert\") pod \"e2e-extmod-a-027c19-openshift-default-58d5bc7645-6jphv\" (UID: \"c4fa33da-32e1-40cb-b89d-8b25ab268fb0\") " pod="openshift-ingress/e2e-extmod-a-027c19-openshift-default-58d5bc7645-6jphv" Nov 03 01:20:52.586120 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:52.586101 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4fa33da-32e1-40cb-b89d-8b25ab268fb0-istio-envoy\") pod \"e2e-extmod-a-027c19-openshift-default-58d5bc7645-6jphv\" (UID: \"c4fa33da-32e1-40cb-b89d-8b25ab268fb0\") " pod="openshift-ingress/e2e-extmod-a-027c19-openshift-default-58d5bc7645-6jphv" Nov 03 01:20:52.701547 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:52.691716 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4fa33da-32e1-40cb-b89d-8b25ab268fb0-istio-envoy\") pod \"e2e-extmod-a-027c19-openshift-default-58d5bc7645-6jphv\" (UID: \"c4fa33da-32e1-40cb-b89d-8b25ab268fb0\") " pod="openshift-ingress/e2e-extmod-a-027c19-openshift-default-58d5bc7645-6jphv" Nov 03 01:20:52.703086 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:52.702268 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4fa33da-32e1-40cb-b89d-8b25ab268fb0-workload-certs\") pod \"e2e-extmod-a-027c19-openshift-default-58d5bc7645-6jphv\" (UID: \"c4fa33da-32e1-40cb-b89d-8b25ab268fb0\") " pod="openshift-ingress/e2e-extmod-a-027c19-openshift-default-58d5bc7645-6jphv" Nov 03 01:20:52.703086 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:52.702464 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4fa33da-32e1-40cb-b89d-8b25ab268fb0-istio-podinfo\") pod \"e2e-extmod-a-027c19-openshift-default-58d5bc7645-6jphv\" (UID: \"c4fa33da-32e1-40cb-b89d-8b25ab268fb0\") " pod="openshift-ingress/e2e-extmod-a-027c19-openshift-default-58d5bc7645-6jphv" Nov 03 01:20:52.703086 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:52.702535 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4fa33da-32e1-40cb-b89d-8b25ab268fb0-workload-socket\") pod \"e2e-extmod-a-027c19-openshift-default-58d5bc7645-6jphv\" (UID: \"c4fa33da-32e1-40cb-b89d-8b25ab268fb0\") " pod="openshift-ingress/e2e-extmod-a-027c19-openshift-default-58d5bc7645-6jphv" Nov 03 01:20:52.703086 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:52.702562 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4fa33da-32e1-40cb-b89d-8b25ab268fb0-istio-token\") pod \"e2e-extmod-a-027c19-openshift-default-58d5bc7645-6jphv\" (UID: \"c4fa33da-32e1-40cb-b89d-8b25ab268fb0\") " pod="openshift-ingress/e2e-extmod-a-027c19-openshift-default-58d5bc7645-6jphv" Nov 03 01:20:52.703086 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:52.702626 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwpn5\" (UniqueName: \"kubernetes.io/projected/c4fa33da-32e1-40cb-b89d-8b25ab268fb0-kube-api-access-pwpn5\") pod \"e2e-extmod-a-027c19-openshift-default-58d5bc7645-6jphv\" (UID: \"c4fa33da-32e1-40cb-b89d-8b25ab268fb0\") " pod="openshift-ingress/e2e-extmod-a-027c19-openshift-default-58d5bc7645-6jphv" Nov 03 01:20:52.703086 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:52.702653 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4fa33da-32e1-40cb-b89d-8b25ab268fb0-credential-socket\") pod \"e2e-extmod-a-027c19-openshift-default-58d5bc7645-6jphv\" (UID: \"c4fa33da-32e1-40cb-b89d-8b25ab268fb0\") " pod="openshift-ingress/e2e-extmod-a-027c19-openshift-default-58d5bc7645-6jphv" Nov 03 01:20:52.703086 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:52.702729 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4fa33da-32e1-40cb-b89d-8b25ab268fb0-istio-data\") pod \"e2e-extmod-a-027c19-openshift-default-58d5bc7645-6jphv\" (UID: \"c4fa33da-32e1-40cb-b89d-8b25ab268fb0\") " pod="openshift-ingress/e2e-extmod-a-027c19-openshift-default-58d5bc7645-6jphv" Nov 03 01:20:52.703086 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:52.702956 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4fa33da-32e1-40cb-b89d-8b25ab268fb0-workload-socket\") pod \"e2e-extmod-a-027c19-openshift-default-58d5bc7645-6jphv\" (UID: \"c4fa33da-32e1-40cb-b89d-8b25ab268fb0\") " pod="openshift-ingress/e2e-extmod-a-027c19-openshift-default-58d5bc7645-6jphv" Nov 03 01:20:52.703086 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:52.703077 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4fa33da-32e1-40cb-b89d-8b25ab268fb0-istiod-ca-cert\") pod \"e2e-extmod-a-027c19-openshift-default-58d5bc7645-6jphv\" (UID: \"c4fa33da-32e1-40cb-b89d-8b25ab268fb0\") " pod="openshift-ingress/e2e-extmod-a-027c19-openshift-default-58d5bc7645-6jphv" Nov 03 01:20:52.704031 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:52.703224 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4fa33da-32e1-40cb-b89d-8b25ab268fb0-workload-certs\") pod \"e2e-extmod-a-027c19-openshift-default-58d5bc7645-6jphv\" (UID: \"c4fa33da-32e1-40cb-b89d-8b25ab268fb0\") " pod="openshift-ingress/e2e-extmod-a-027c19-openshift-default-58d5bc7645-6jphv" Nov 03 01:20:52.704031 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:52.703646 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4fa33da-32e1-40cb-b89d-8b25ab268fb0-istio-data\") pod \"e2e-extmod-a-027c19-openshift-default-58d5bc7645-6jphv\" (UID: \"c4fa33da-32e1-40cb-b89d-8b25ab268fb0\") " pod="openshift-ingress/e2e-extmod-a-027c19-openshift-default-58d5bc7645-6jphv" Nov 03 01:20:52.704031 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:52.703681 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4fa33da-32e1-40cb-b89d-8b25ab268fb0-credential-socket\") pod \"e2e-extmod-a-027c19-openshift-default-58d5bc7645-6jphv\" (UID: \"c4fa33da-32e1-40cb-b89d-8b25ab268fb0\") " pod="openshift-ingress/e2e-extmod-a-027c19-openshift-default-58d5bc7645-6jphv" Nov 03 01:20:52.704158 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:52.704064 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4fa33da-32e1-40cb-b89d-8b25ab268fb0-istiod-ca-cert\") pod \"e2e-extmod-a-027c19-openshift-default-58d5bc7645-6jphv\" (UID: \"c4fa33da-32e1-40cb-b89d-8b25ab268fb0\") " pod="openshift-ingress/e2e-extmod-a-027c19-openshift-default-58d5bc7645-6jphv" Nov 03 01:20:52.705268 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:52.705246 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4fa33da-32e1-40cb-b89d-8b25ab268fb0-istio-envoy\") pod \"e2e-extmod-a-027c19-openshift-default-58d5bc7645-6jphv\" (UID: \"c4fa33da-32e1-40cb-b89d-8b25ab268fb0\") " pod="openshift-ingress/e2e-extmod-a-027c19-openshift-default-58d5bc7645-6jphv" Nov 03 01:20:52.705844 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:52.705826 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4fa33da-32e1-40cb-b89d-8b25ab268fb0-istio-podinfo\") pod \"e2e-extmod-a-027c19-openshift-default-58d5bc7645-6jphv\" (UID: \"c4fa33da-32e1-40cb-b89d-8b25ab268fb0\") " pod="openshift-ingress/e2e-extmod-a-027c19-openshift-default-58d5bc7645-6jphv" Nov 03 01:20:52.712092 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:52.712073 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4fa33da-32e1-40cb-b89d-8b25ab268fb0-istio-token\") pod \"e2e-extmod-a-027c19-openshift-default-58d5bc7645-6jphv\" (UID: \"c4fa33da-32e1-40cb-b89d-8b25ab268fb0\") " pod="openshift-ingress/e2e-extmod-a-027c19-openshift-default-58d5bc7645-6jphv" Nov 03 01:20:52.712184 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:52.712117 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwpn5\" (UniqueName: \"kubernetes.io/projected/c4fa33da-32e1-40cb-b89d-8b25ab268fb0-kube-api-access-pwpn5\") pod \"e2e-extmod-a-027c19-openshift-default-58d5bc7645-6jphv\" (UID: \"c4fa33da-32e1-40cb-b89d-8b25ab268fb0\") " pod="openshift-ingress/e2e-extmod-a-027c19-openshift-default-58d5bc7645-6jphv" Nov 03 01:20:52.768017 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:52.767976 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-027c19-openshift-default-58d5bc7645-6jphv" Nov 03 01:20:53.074545 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:53.074508 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-027c19-openshift-default-58d5bc7645-6jphv"] Nov 03 01:20:53.089893 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:53.089849 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:20:53.090004 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:53.089950 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:20:53.090004 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:53.089994 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:20:53.721340 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:53.721302 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-027c19-openshift-default-58d5bc7645-6jphv" event={"ID":"c4fa33da-32e1-40cb-b89d-8b25ab268fb0","Type":"ContainerStarted","Data":"c390525ebabf64e8a9a245c797763de55fe02a5985c830758a5a35f4d35f7340"} Nov 03 01:20:53.721956 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:53.721918 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-027c19-openshift-default-58d5bc7645-6jphv" event={"ID":"c4fa33da-32e1-40cb-b89d-8b25ab268fb0","Type":"ContainerStarted","Data":"6ff510f4d83e1826980b83e4351383b3b05bf7c7503fd5f9773eaae8f75e331c"} Nov 03 01:20:53.748525 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:53.748466 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-027c19-openshift-default-58d5bc7645-6jphv" podStartSLOduration=1.748451498 podStartE2EDuration="1.748451498s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:20:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:20:53.742366484 +0000 UTC m=+2395.913369583" watchObservedRunningTime="2026-11-03 01:20:53.748451498 +0000 UTC m=+2395.919454594" Nov 03 01:20:53.768782 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:53.768744 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-027c19-openshift-default-58d5bc7645-6jphv" Nov 03 01:20:54.771153 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:54.771117 4421 patch_prober.go:29] interesting pod/e2e-extmod-a-027c19-openshift-default-58d5bc7645-6jphv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.43:15021/healthz/ready\": dial tcp 10.134.1.43:15021: connect: connection refused" start-of-body= Nov 03 01:20:54.771578 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:54.771172 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-027c19-openshift-default-58d5bc7645-6jphv" podUID="c4fa33da-32e1-40cb-b89d-8b25ab268fb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.43:15021/healthz/ready\": dial tcp 10.134.1.43:15021: connect: connection refused" Nov 03 01:20:55.786850 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:55.786812 4421 patch_prober.go:29] interesting pod/e2e-extmod-a-027c19-openshift-default-58d5bc7645-6jphv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.43:15021/healthz/ready\": dial tcp 10.134.1.43:15021: connect: connection refused" start-of-body= Nov 03 01:20:55.787567 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:55.787531 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-027c19-openshift-default-58d5bc7645-6jphv" podUID="c4fa33da-32e1-40cb-b89d-8b25ab268fb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.43:15021/healthz/ready\": dial tcp 10.134.1.43:15021: connect: connection refused" Nov 03 01:20:56.118421 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:56.101319 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a395bce9-flxhm"] Nov 03 01:20:56.383082 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:56.366521 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a395bce9-flxhm"] Nov 03 01:20:56.805693 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:56.805646 4421 patch_prober.go:29] interesting pod/e2e-extmod-a-027c19-openshift-default-58d5bc7645-6jphv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.43:15021/healthz/ready\": dial tcp 10.134.1.43:15021: connect: connection refused" start-of-body= Nov 03 01:20:56.806356 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:56.806317 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-027c19-openshift-default-58d5bc7645-6jphv" podUID="c4fa33da-32e1-40cb-b89d-8b25ab268fb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.43:15021/healthz/ready\": dial tcp 10.134.1.43:15021: connect: connection refused" Nov 03 01:20:56.996454 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:56.996402 4421 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-83755a/distinct-w3-83755a-kserve-86bdbbdd85-2cnnl" podUID="f4ae9624-d127-4a55-837d-7c9ff0267b37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:20:57.321600 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:57.321555 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/distinct-w3-83755a-kserve-86bdbbdd85-2cnnl" podUID="f4ae9624-d127-4a55-837d-7c9ff0267b37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:20:57.485463 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:57.485411 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/simulator-w3-83755a-kserve-77d6664559-psqtg" podUID="b0a4c234-af58-4da0-a1a4-e1f167953f6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:20:57.770001 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:57.769963 4421 patch_prober.go:29] interesting pod/e2e-extmod-a-027c19-openshift-default-58d5bc7645-6jphv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.43:15021/healthz/ready\": dial tcp 10.134.1.43:15021: connect: connection refused" start-of-body= Nov 03 01:20:57.770344 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:57.770307 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-027c19-openshift-default-58d5bc7645-6jphv" podUID="c4fa33da-32e1-40cb-b89d-8b25ab268fb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.43:15021/healthz/ready\": dial tcp 10.134.1.43:15021: connect: connection refused" Nov 03 01:20:57.990302 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:57.990254 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0d13eee-83f0-4cd5-80d0-0a7e6d60db18" path="/var/lib/kubelet/pods/e0d13eee-83f0-4cd5-80d0-0a7e6d60db18/volumes" Nov 03 01:20:58.783662 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:58.783620 4421 patch_prober.go:29] interesting pod/e2e-extmod-a-027c19-openshift-default-58d5bc7645-6jphv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.43:15021/healthz/ready\": dial tcp 10.134.1.43:15021: connect: connection refused" start-of-body= Nov 03 01:20:58.783988 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:58.783952 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-027c19-openshift-default-58d5bc7645-6jphv" podUID="c4fa33da-32e1-40cb-b89d-8b25ab268fb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.43:15021/healthz/ready\": dial tcp 10.134.1.43:15021: connect: connection refused" Nov 03 01:20:59.810220 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:59.810165 4421 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-kr275 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 01:20:59.811458 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:59.811408 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-kr275" podUID="3e73ad9e-cd24-4d53-82f0-2e521cae7445" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 01:20:59.812232 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:59.812211 4421 patch_prober.go:29] interesting pod/e2e-extmod-a-027c19-openshift-default-58d5bc7645-6jphv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.43:15021/healthz/ready\": dial tcp 10.134.1.43:15021: connect: connection refused" start-of-body= Nov 03 01:20:59.812423 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:20:59.812400 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-027c19-openshift-default-58d5bc7645-6jphv" podUID="c4fa33da-32e1-40cb-b89d-8b25ab268fb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.43:15021/healthz/ready\": dial tcp 10.134.1.43:15021: connect: connection refused" Nov 03 01:21:00.791901 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:21:00.791854 4421 patch_prober.go:29] interesting pod/e2e-extmod-a-027c19-openshift-default-58d5bc7645-6jphv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.43:15021/healthz/ready\": dial tcp 10.134.1.43:15021: connect: connection refused" start-of-body= Nov 03 01:21:00.792211 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:21:00.792181 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-027c19-openshift-default-58d5bc7645-6jphv" podUID="c4fa33da-32e1-40cb-b89d-8b25ab268fb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.43:15021/healthz/ready\": dial tcp 10.134.1.43:15021: connect: connection refused" Nov 03 01:21:01.927745 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:21:01.927708 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f51d1a9-5zl5h"] Nov 03 01:21:01.936824 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:21:01.936797 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f51d1a9-5zl5h" Nov 03 01:21:01.941540 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:21:01.941513 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:21:01.941886 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:21:01.941866 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6q4xp\"" Nov 03 01:21:01.944099 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:21:01.944061 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d33fa6fe-3a61-4cfc-ab8a-05186c69266f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4f51d1a9-5zl5h\" (UID: \"d33fa6fe-3a61-4cfc-ab8a-05186c69266f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f51d1a9-5zl5h" Nov 03 01:21:01.944234 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:21:01.944214 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d33fa6fe-3a61-4cfc-ab8a-05186c69266f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4f51d1a9-5zl5h\" (UID: \"d33fa6fe-3a61-4cfc-ab8a-05186c69266f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f51d1a9-5zl5h" Nov 03 01:21:01.944434 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:21:01.944342 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vz5bd\" (UniqueName: \"kubernetes.io/projected/d33fa6fe-3a61-4cfc-ab8a-05186c69266f-kube-api-access-vz5bd\") pod \"maas-api-revoke-sub-4f51d1a9-5zl5h\" (UID: \"d33fa6fe-3a61-4cfc-ab8a-05186c69266f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f51d1a9-5zl5h" Nov 03 01:21:01.945448 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:21:01.945425 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f51d1a9-5zl5h"] Nov 03 01:21:02.047206 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:21:02.046541 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d33fa6fe-3a61-4cfc-ab8a-05186c69266f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4f51d1a9-5zl5h\" (UID: \"d33fa6fe-3a61-4cfc-ab8a-05186c69266f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f51d1a9-5zl5h" Nov 03 01:21:02.047206 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:21:02.046795 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d33fa6fe-3a61-4cfc-ab8a-05186c69266f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4f51d1a9-5zl5h\" (UID: \"d33fa6fe-3a61-4cfc-ab8a-05186c69266f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f51d1a9-5zl5h" Nov 03 01:21:02.047206 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:21:02.046867 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vz5bd\" (UniqueName: \"kubernetes.io/projected/d33fa6fe-3a61-4cfc-ab8a-05186c69266f-kube-api-access-vz5bd\") pod \"maas-api-revoke-sub-4f51d1a9-5zl5h\" (UID: \"d33fa6fe-3a61-4cfc-ab8a-05186c69266f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f51d1a9-5zl5h" Nov 03 01:21:02.047582 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:21:02.047418 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d33fa6fe-3a61-4cfc-ab8a-05186c69266f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4f51d1a9-5zl5h\" (UID: \"d33fa6fe-3a61-4cfc-ab8a-05186c69266f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f51d1a9-5zl5h" Nov 03 01:21:02.049600 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:21:02.049574 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d33fa6fe-3a61-4cfc-ab8a-05186c69266f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4f51d1a9-5zl5h\" (UID: \"d33fa6fe-3a61-4cfc-ab8a-05186c69266f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f51d1a9-5zl5h" Nov 03 01:21:02.058109 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:21:02.058077 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vz5bd\" (UniqueName: \"kubernetes.io/projected/d33fa6fe-3a61-4cfc-ab8a-05186c69266f-kube-api-access-vz5bd\") pod \"maas-api-revoke-sub-4f51d1a9-5zl5h\" (UID: \"d33fa6fe-3a61-4cfc-ab8a-05186c69266f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f51d1a9-5zl5h" Nov 03 01:21:02.094681 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:21:02.094649 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-027c19-openshift-default-58d5bc7645-6jphv" Nov 03 01:21:02.096668 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:21:02.095499 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-027c19-openshift-default-58d5bc7645-6jphv" Nov 03 01:21:02.096668 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:21:02.096644 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-027c19-openshift-default-58d5bc7645-6jphv" Nov 03 01:21:02.281571 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:21:02.281536 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f51d1a9-5zl5h" Nov 03 01:21:02.639373 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:21:02.639336 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd33fa6fe_3a61_4cfc_ab8a_05186c69266f.slice/crio-5d1baeba39ad47777ee6946394602b3af1a80f7010331c438e2dba1bb1a1f3aa WatchSource:0}: Error finding container 5d1baeba39ad47777ee6946394602b3af1a80f7010331c438e2dba1bb1a1f3aa: Status 404 returned error can't find the container with id 5d1baeba39ad47777ee6946394602b3af1a80f7010331c438e2dba1bb1a1f3aa Nov 03 01:21:02.640149 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:21:02.640119 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f51d1a9-5zl5h"] Nov 03 01:21:03.029172 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:21:03.029129 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-66662d95-t4k4k"] Nov 03 01:21:03.036330 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:21:03.036298 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-66662d95-t4k4k"] Nov 03 01:21:03.456202 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:21:03.456089 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f51d1a9-5zl5h" event={"ID":"d33fa6fe-3a61-4cfc-ab8a-05186c69266f","Type":"ContainerStarted","Data":"18da91d8d86b842975bbe0edb31f2c2cd6b649f489fcf2ccf24494dc4738ad9c"} Nov 03 01:21:03.456202 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:21:03.456169 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f51d1a9-5zl5h" event={"ID":"d33fa6fe-3a61-4cfc-ab8a-05186c69266f","Type":"ContainerStarted","Data":"5d1baeba39ad47777ee6946394602b3af1a80f7010331c438e2dba1bb1a1f3aa"} Nov 03 01:21:03.473749 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:21:03.473698 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f51d1a9-5zl5h" podStartSLOduration=2.473685026 podStartE2EDuration="2.473685026s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:21:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:21:03.471350951 +0000 UTC m=+2405.642354048" watchObservedRunningTime="2026-11-03 01:21:03.473685026 +0000 UTC m=+2405.644688121" Nov 03 01:21:03.984949 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:21:03.984857 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a36f95f-aa93-4151-893b-fcc702a5fb9a" path="/var/lib/kubelet/pods/3a36f95f-aa93-4151-893b-fcc702a5fb9a/volumes" Nov 03 01:21:04.550017 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:21:04.549983 4421 generic.go:356] "Generic (PLEG): container finished" podID="d33fa6fe-3a61-4cfc-ab8a-05186c69266f" containerID="18da91d8d86b842975bbe0edb31f2c2cd6b649f489fcf2ccf24494dc4738ad9c" exitCode=0 Nov 03 01:21:04.550583 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:21:04.550050 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f51d1a9-5zl5h" event={"ID":"d33fa6fe-3a61-4cfc-ab8a-05186c69266f","Type":"ContainerDied","Data":"18da91d8d86b842975bbe0edb31f2c2cd6b649f489fcf2ccf24494dc4738ad9c"} Nov 03 01:21:05.764882 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:21:05.764789 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f51d1a9-5zl5h" Nov 03 01:21:05.879406 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:21:05.873729 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d33fa6fe-3a61-4cfc-ab8a-05186c69266f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d33fa6fe-3a61-4cfc-ab8a-05186c69266f-maas-api-openshift-ca\") pod \"d33fa6fe-3a61-4cfc-ab8a-05186c69266f\" (UID: \"d33fa6fe-3a61-4cfc-ab8a-05186c69266f\") " Nov 03 01:21:05.879406 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:21:05.873835 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d33fa6fe-3a61-4cfc-ab8a-05186c69266f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d33fa6fe-3a61-4cfc-ab8a-05186c69266f-maas-api-xks-ca\") pod \"d33fa6fe-3a61-4cfc-ab8a-05186c69266f\" (UID: \"d33fa6fe-3a61-4cfc-ab8a-05186c69266f\") " Nov 03 01:21:05.879406 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:21:05.874092 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d33fa6fe-3a61-4cfc-ab8a-05186c69266f-kube-api-access-vz5bd\" (UniqueName: \"kubernetes.io/projected/d33fa6fe-3a61-4cfc-ab8a-05186c69266f-kube-api-access-vz5bd\") pod \"d33fa6fe-3a61-4cfc-ab8a-05186c69266f\" (UID: \"d33fa6fe-3a61-4cfc-ab8a-05186c69266f\") " Nov 03 01:21:05.900871 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:21:05.900817 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d33fa6fe-3a61-4cfc-ab8a-05186c69266f-maas-api-openshift-ca" pod "d33fa6fe-3a61-4cfc-ab8a-05186c69266f" (UID: "d33fa6fe-3a61-4cfc-ab8a-05186c69266f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:21:05.904692 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:21:05.904661 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d33fa6fe-3a61-4cfc-ab8a-05186c69266f-kube-api-access-vz5bd" pod "d33fa6fe-3a61-4cfc-ab8a-05186c69266f" (UID: "d33fa6fe-3a61-4cfc-ab8a-05186c69266f"). InnerVolumeSpecName "kube-api-access-vz5bd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:21:05.908691 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:21:05.908659 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d33fa6fe-3a61-4cfc-ab8a-05186c69266f-maas-api-xks-ca" pod "d33fa6fe-3a61-4cfc-ab8a-05186c69266f" (UID: "d33fa6fe-3a61-4cfc-ab8a-05186c69266f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:21:05.976859 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:21:05.976827 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vz5bd\" (UniqueName: \"kubernetes.io/projected/d33fa6fe-3a61-4cfc-ab8a-05186c69266f-kube-api-access-vz5bd\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:21:05.976859 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:21:05.976857 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d33fa6fe-3a61-4cfc-ab8a-05186c69266f-maas-api-openshift-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:21:05.976859 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:21:05.976867 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d33fa6fe-3a61-4cfc-ab8a-05186c69266f-maas-api-xks-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:21:06.569455 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:21:06.569422 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f51d1a9-5zl5h" Nov 03 01:21:06.569615 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:21:06.569461 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f51d1a9-5zl5h" event={"ID":"d33fa6fe-3a61-4cfc-ab8a-05186c69266f","Type":"ContainerDied","Data":"5d1baeba39ad47777ee6946394602b3af1a80f7010331c438e2dba1bb1a1f3aa"} Nov 03 01:21:06.569615 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:21:06.569515 4421 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5d1baeba39ad47777ee6946394602b3af1a80f7010331c438e2dba1bb1a1f3aa" Nov 03 01:21:12.397266 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:21:12.397228 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a762f455-v88l9"] Nov 03 01:21:12.398358 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:21:12.398335 4421 cpu_manager.go:414] "containerMap: removing container" podUID="d33fa6fe-3a61-4cfc-ab8a-05186c69266f" containerName="revoke-keys" Nov 03 01:21:12.398358 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:21:12.398356 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d33fa6fe-3a61-4cfc-ab8a-05186c69266f" containerName="revoke-keys" Nov 03 01:21:12.398628 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:21:12.398613 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="d33fa6fe-3a61-4cfc-ab8a-05186c69266f" containerName="revoke-keys" Nov 03 01:21:12.404504 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:21:12.404476 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a762f455-v88l9" Nov 03 01:21:12.408472 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:21:12.408448 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:21:12.409530 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:21:12.409502 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a762f455-v88l9"] Nov 03 01:21:12.409652 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:21:12.409548 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6q4xp\"" Nov 03 01:21:12.496299 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:21:12.484094 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf375ec3-3e92-40ac-a2c6-6ec506ce301d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a762f455-v88l9\" (UID: \"cf375ec3-3e92-40ac-a2c6-6ec506ce301d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a762f455-v88l9" Nov 03 01:21:12.496299 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:21:12.484441 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf375ec3-3e92-40ac-a2c6-6ec506ce301d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a762f455-v88l9\" (UID: \"cf375ec3-3e92-40ac-a2c6-6ec506ce301d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a762f455-v88l9" Nov 03 01:21:12.496299 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:21:12.484559 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mf7p\" (UniqueName: \"kubernetes.io/projected/cf375ec3-3e92-40ac-a2c6-6ec506ce301d-kube-api-access-7mf7p\") pod \"maas-api-revoke-sub-a762f455-v88l9\" (UID: \"cf375ec3-3e92-40ac-a2c6-6ec506ce301d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a762f455-v88l9" Nov 03 01:21:12.586477 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:21:12.586444 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mf7p\" (UniqueName: \"kubernetes.io/projected/cf375ec3-3e92-40ac-a2c6-6ec506ce301d-kube-api-access-7mf7p\") pod \"maas-api-revoke-sub-a762f455-v88l9\" (UID: \"cf375ec3-3e92-40ac-a2c6-6ec506ce301d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a762f455-v88l9" Nov 03 01:21:12.586723 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:21:12.586704 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf375ec3-3e92-40ac-a2c6-6ec506ce301d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a762f455-v88l9\" (UID: \"cf375ec3-3e92-40ac-a2c6-6ec506ce301d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a762f455-v88l9" Nov 03 01:21:12.586984 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:21:12.586928 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf375ec3-3e92-40ac-a2c6-6ec506ce301d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a762f455-v88l9\" (UID: \"cf375ec3-3e92-40ac-a2c6-6ec506ce301d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a762f455-v88l9" Nov 03 01:21:12.587636 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:21:12.587611 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf375ec3-3e92-40ac-a2c6-6ec506ce301d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a762f455-v88l9\" (UID: \"cf375ec3-3e92-40ac-a2c6-6ec506ce301d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a762f455-v88l9" Nov 03 01:21:12.589430 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:21:12.589403 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf375ec3-3e92-40ac-a2c6-6ec506ce301d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a762f455-v88l9\" (UID: \"cf375ec3-3e92-40ac-a2c6-6ec506ce301d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a762f455-v88l9" Nov 03 01:21:12.594745 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:21:12.594716 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mf7p\" (UniqueName: \"kubernetes.io/projected/cf375ec3-3e92-40ac-a2c6-6ec506ce301d-kube-api-access-7mf7p\") pod \"maas-api-revoke-sub-a762f455-v88l9\" (UID: \"cf375ec3-3e92-40ac-a2c6-6ec506ce301d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a762f455-v88l9" Nov 03 01:21:12.732124 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:21:12.732041 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a762f455-v88l9" Nov 03 01:21:13.023501 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:21:13.023472 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf375ec3_3e92_40ac_a2c6_6ec506ce301d.slice/crio-c6b0c88111e87feffbcc3376013d8aab6a18277b349accdeda4943b3d3d20e10 WatchSource:0}: Error finding container c6b0c88111e87feffbcc3376013d8aab6a18277b349accdeda4943b3d3d20e10: Status 404 returned error can't find the container with id c6b0c88111e87feffbcc3376013d8aab6a18277b349accdeda4943b3d3d20e10 Nov 03 01:21:13.025813 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:21:13.025787 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a762f455-v88l9"] Nov 03 01:21:13.633197 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:21:13.633112 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a762f455-v88l9" event={"ID":"cf375ec3-3e92-40ac-a2c6-6ec506ce301d","Type":"ContainerStarted","Data":"694483be4e7539343ac52f3e7be2f8e5b9aba726ebdab3819e576fac6d4698cf"} Nov 03 01:21:13.633601 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:21:13.633207 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a762f455-v88l9" event={"ID":"cf375ec3-3e92-40ac-a2c6-6ec506ce301d","Type":"ContainerStarted","Data":"c6b0c88111e87feffbcc3376013d8aab6a18277b349accdeda4943b3d3d20e10"} Nov 03 01:21:13.649881 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:21:13.649783 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a762f455-v88l9" podStartSLOduration=1.649762881 podStartE2EDuration="1.649762881s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:21:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:21:13.647944807 +0000 UTC m=+2415.818947907" watchObservedRunningTime="2026-11-03 01:21:13.649762881 +0000 UTC m=+2415.820765978" Nov 03 01:21:14.645684 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:21:14.645648 4421 generic.go:356] "Generic (PLEG): container finished" podID="cf375ec3-3e92-40ac-a2c6-6ec506ce301d" containerID="694483be4e7539343ac52f3e7be2f8e5b9aba726ebdab3819e576fac6d4698cf" exitCode=0 Nov 03 01:21:14.646229 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:21:14.645749 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a762f455-v88l9" event={"ID":"cf375ec3-3e92-40ac-a2c6-6ec506ce301d","Type":"ContainerDied","Data":"694483be4e7539343ac52f3e7be2f8e5b9aba726ebdab3819e576fac6d4698cf"} Nov 03 01:21:14.674385 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:21:14.674351 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 01:21:15.958028 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:21:15.957997 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a762f455-v88l9" Nov 03 01:21:16.106293 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:21:16.106242 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf375ec3-3e92-40ac-a2c6-6ec506ce301d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf375ec3-3e92-40ac-a2c6-6ec506ce301d-maas-api-xks-ca\") pod \"cf375ec3-3e92-40ac-a2c6-6ec506ce301d\" (UID: \"cf375ec3-3e92-40ac-a2c6-6ec506ce301d\") " Nov 03 01:21:16.106592 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:21:16.106573 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf375ec3-3e92-40ac-a2c6-6ec506ce301d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf375ec3-3e92-40ac-a2c6-6ec506ce301d-maas-api-openshift-ca\") pod \"cf375ec3-3e92-40ac-a2c6-6ec506ce301d\" (UID: \"cf375ec3-3e92-40ac-a2c6-6ec506ce301d\") " Nov 03 01:21:16.106670 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:21:16.106642 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf375ec3-3e92-40ac-a2c6-6ec506ce301d-kube-api-access-7mf7p\" (UniqueName: \"kubernetes.io/projected/cf375ec3-3e92-40ac-a2c6-6ec506ce301d-kube-api-access-7mf7p\") pod \"cf375ec3-3e92-40ac-a2c6-6ec506ce301d\" (UID: \"cf375ec3-3e92-40ac-a2c6-6ec506ce301d\") " Nov 03 01:21:16.107129 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:21:16.107093 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf375ec3-3e92-40ac-a2c6-6ec506ce301d-maas-api-openshift-ca" pod "cf375ec3-3e92-40ac-a2c6-6ec506ce301d" (UID: "cf375ec3-3e92-40ac-a2c6-6ec506ce301d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:21:16.108247 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:21:16.108142 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf375ec3-3e92-40ac-a2c6-6ec506ce301d-maas-api-openshift-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:21:16.108957 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:21:16.108921 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf375ec3-3e92-40ac-a2c6-6ec506ce301d-maas-api-xks-ca" pod "cf375ec3-3e92-40ac-a2c6-6ec506ce301d" (UID: "cf375ec3-3e92-40ac-a2c6-6ec506ce301d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:21:16.109244 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:21:16.109216 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf375ec3-3e92-40ac-a2c6-6ec506ce301d-kube-api-access-7mf7p" pod "cf375ec3-3e92-40ac-a2c6-6ec506ce301d" (UID: "cf375ec3-3e92-40ac-a2c6-6ec506ce301d"). InnerVolumeSpecName "kube-api-access-7mf7p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:21:16.209872 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:21:16.209779 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf375ec3-3e92-40ac-a2c6-6ec506ce301d-maas-api-xks-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:21:16.209872 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:21:16.209824 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mf7p\" (UniqueName: \"kubernetes.io/projected/cf375ec3-3e92-40ac-a2c6-6ec506ce301d-kube-api-access-7mf7p\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:21:16.665781 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:21:16.665749 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a762f455-v88l9" Nov 03 01:21:16.665981 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:21:16.665786 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a762f455-v88l9" event={"ID":"cf375ec3-3e92-40ac-a2c6-6ec506ce301d","Type":"ContainerDied","Data":"c6b0c88111e87feffbcc3376013d8aab6a18277b349accdeda4943b3d3d20e10"} Nov 03 01:21:16.665981 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:21:16.665841 4421 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c6b0c88111e87feffbcc3376013d8aab6a18277b349accdeda4943b3d3d20e10" Nov 03 01:21:32.198310 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:21:32.197089 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/simulator-w2-f7680d-kserve-85bdc86cd6-8gxrg" podUID="2bdcf521-cebd-4d6b-8554-1b60db8b0f3c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 03 01:21:32.354299 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:21:32.354043 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-650be0/simulator-w4-650be0-kserve-7bc588774f-gvz2k" podUID="a827fa88-f078-4312-8f12-2c799c11b461" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:21:32.456569 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:21:32.374847 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/distinct-w3-83755a-kserve-86bdbbdd85-2cnnl" podUID="f4ae9624-d127-4a55-837d-7c9ff0267b37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded" Nov 03 01:21:32.456569 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:21:32.380496 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/premium-w3-83755a-kserve-7c9d66988b-lt4xd" podUID="c0c6b02e-2ecb-428e-a13d-be6d7cd1ba39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Nov 03 01:21:32.551743 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:21:32.551366 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/simulator-w3-83755a-kserve-77d6664559-psqtg" podUID="b0a4c234-af58-4da0-a1a4-e1f167953f6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 01:21:44.672572 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:21:44.672525 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 01:21:50.248261 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:21:50.248226 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-tc5dx"] Nov 03 01:21:50.248790 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:21:50.248453 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-tc5dx" podUID="3c57e761-5812-43b8-a25f-f983c0dea9b0" containerName="istio-proxy" containerID="cri-o://c229c496285e24181e14e79befb1ee74dc2c0d1ce42f2f3b077fad7442c800ad" gracePeriod=30 Nov 03 01:21:55.922311 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:21:55.917793 4421 generic.go:356] "Generic (PLEG): container finished" podID="3c57e761-5812-43b8-a25f-f983c0dea9b0" containerID="c229c496285e24181e14e79befb1ee74dc2c0d1ce42f2f3b077fad7442c800ad" exitCode=0 Nov 03 01:21:55.922311 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:21:55.918079 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-tc5dx" event={"ID":"3c57e761-5812-43b8-a25f-f983c0dea9b0","Type":"ContainerDied","Data":"c229c496285e24181e14e79befb1ee74dc2c0d1ce42f2f3b077fad7442c800ad"} Nov 03 01:21:56.429740 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:21:56.429708 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-tc5dx" Nov 03 01:21:56.551326 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:21:56.549308 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c57e761-5812-43b8-a25f-f983c0dea9b0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c57e761-5812-43b8-a25f-f983c0dea9b0-istio-data\") pod \"3c57e761-5812-43b8-a25f-f983c0dea9b0\" (UID: \"3c57e761-5812-43b8-a25f-f983c0dea9b0\") " Nov 03 01:21:56.551326 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:21:56.549582 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c57e761-5812-43b8-a25f-f983c0dea9b0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c57e761-5812-43b8-a25f-f983c0dea9b0-istio-envoy\") pod \"3c57e761-5812-43b8-a25f-f983c0dea9b0\" (UID: \"3c57e761-5812-43b8-a25f-f983c0dea9b0\") " Nov 03 01:21:56.551326 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:21:56.549614 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c57e761-5812-43b8-a25f-f983c0dea9b0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c57e761-5812-43b8-a25f-f983c0dea9b0-workload-certs\") pod \"3c57e761-5812-43b8-a25f-f983c0dea9b0\" (UID: \"3c57e761-5812-43b8-a25f-f983c0dea9b0\") " Nov 03 01:21:56.551326 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:21:56.549690 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c57e761-5812-43b8-a25f-f983c0dea9b0-kube-api-access-sr427\" (UniqueName: \"kubernetes.io/projected/3c57e761-5812-43b8-a25f-f983c0dea9b0-kube-api-access-sr427\") pod \"3c57e761-5812-43b8-a25f-f983c0dea9b0\" (UID: \"3c57e761-5812-43b8-a25f-f983c0dea9b0\") " Nov 03 01:21:56.551326 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:21:56.549739 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c57e761-5812-43b8-a25f-f983c0dea9b0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c57e761-5812-43b8-a25f-f983c0dea9b0-workload-socket\") pod \"3c57e761-5812-43b8-a25f-f983c0dea9b0\" (UID: \"3c57e761-5812-43b8-a25f-f983c0dea9b0\") " Nov 03 01:21:56.551326 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:21:56.549781 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c57e761-5812-43b8-a25f-f983c0dea9b0-istio-token\" (UniqueName: \"kubernetes.io/projected/3c57e761-5812-43b8-a25f-f983c0dea9b0-istio-token\") pod \"3c57e761-5812-43b8-a25f-f983c0dea9b0\" (UID: \"3c57e761-5812-43b8-a25f-f983c0dea9b0\") " Nov 03 01:21:56.551326 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:21:56.549812 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c57e761-5812-43b8-a25f-f983c0dea9b0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c57e761-5812-43b8-a25f-f983c0dea9b0-credential-socket\") pod \"3c57e761-5812-43b8-a25f-f983c0dea9b0\" (UID: \"3c57e761-5812-43b8-a25f-f983c0dea9b0\") " Nov 03 01:21:56.551326 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:21:56.549931 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c57e761-5812-43b8-a25f-f983c0dea9b0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c57e761-5812-43b8-a25f-f983c0dea9b0-istiod-ca-cert\") pod \"3c57e761-5812-43b8-a25f-f983c0dea9b0\" (UID: \"3c57e761-5812-43b8-a25f-f983c0dea9b0\") " Nov 03 01:21:56.551326 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:21:56.550015 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3c57e761-5812-43b8-a25f-f983c0dea9b0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c57e761-5812-43b8-a25f-f983c0dea9b0-istio-podinfo\") pod \"3c57e761-5812-43b8-a25f-f983c0dea9b0\" (UID: \"3c57e761-5812-43b8-a25f-f983c0dea9b0\") " Nov 03 01:21:56.551861 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:21:56.551586 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c57e761-5812-43b8-a25f-f983c0dea9b0-istio-data" pod "3c57e761-5812-43b8-a25f-f983c0dea9b0" (UID: "3c57e761-5812-43b8-a25f-f983c0dea9b0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:21:56.552719 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:21:56.552119 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c57e761-5812-43b8-a25f-f983c0dea9b0-workload-socket" pod "3c57e761-5812-43b8-a25f-f983c0dea9b0" (UID: "3c57e761-5812-43b8-a25f-f983c0dea9b0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:21:56.552719 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:21:56.552361 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c57e761-5812-43b8-a25f-f983c0dea9b0-credential-socket" pod "3c57e761-5812-43b8-a25f-f983c0dea9b0" (UID: "3c57e761-5812-43b8-a25f-f983c0dea9b0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:21:56.552719 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:21:56.552671 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c57e761-5812-43b8-a25f-f983c0dea9b0-istiod-ca-cert" pod "3c57e761-5812-43b8-a25f-f983c0dea9b0" (UID: "3c57e761-5812-43b8-a25f-f983c0dea9b0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:21:56.553670 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:21:56.553636 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c57e761-5812-43b8-a25f-f983c0dea9b0-kube-api-access-sr427" pod "3c57e761-5812-43b8-a25f-f983c0dea9b0" (UID: "3c57e761-5812-43b8-a25f-f983c0dea9b0"). InnerVolumeSpecName "kube-api-access-sr427". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:21:56.555017 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:21:56.554974 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c57e761-5812-43b8-a25f-f983c0dea9b0-workload-certs" pod "3c57e761-5812-43b8-a25f-f983c0dea9b0" (UID: "3c57e761-5812-43b8-a25f-f983c0dea9b0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:21:56.556170 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:21:56.556129 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c57e761-5812-43b8-a25f-f983c0dea9b0-istio-token" pod "3c57e761-5812-43b8-a25f-f983c0dea9b0" (UID: "3c57e761-5812-43b8-a25f-f983c0dea9b0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:21:56.557735 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:21:56.557694 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c57e761-5812-43b8-a25f-f983c0dea9b0-istio-envoy" pod "3c57e761-5812-43b8-a25f-f983c0dea9b0" (UID: "3c57e761-5812-43b8-a25f-f983c0dea9b0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:21:56.558716 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:21:56.558694 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3c57e761-5812-43b8-a25f-f983c0dea9b0-istio-podinfo" pod "3c57e761-5812-43b8-a25f-f983c0dea9b0" (UID: "3c57e761-5812-43b8-a25f-f983c0dea9b0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:21:56.653500 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:21:56.653454 4421 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c57e761-5812-43b8-a25f-f983c0dea9b0-istio-envoy\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:21:56.653500 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:21:56.653491 4421 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c57e761-5812-43b8-a25f-f983c0dea9b0-workload-certs\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:21:56.653500 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:21:56.653507 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sr427\" (UniqueName: \"kubernetes.io/projected/3c57e761-5812-43b8-a25f-f983c0dea9b0-kube-api-access-sr427\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:21:56.653840 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:21:56.653523 4421 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c57e761-5812-43b8-a25f-f983c0dea9b0-workload-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:21:56.653840 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:21:56.653537 4421 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c57e761-5812-43b8-a25f-f983c0dea9b0-istio-token\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:21:56.653840 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:21:56.653551 4421 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c57e761-5812-43b8-a25f-f983c0dea9b0-credential-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:21:56.653840 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:21:56.653564 4421 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c57e761-5812-43b8-a25f-f983c0dea9b0-istiod-ca-cert\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:21:56.653840 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:21:56.653577 4421 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c57e761-5812-43b8-a25f-f983c0dea9b0-istio-podinfo\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:21:56.653840 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:21:56.653585 4421 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c57e761-5812-43b8-a25f-f983c0dea9b0-istio-data\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:21:56.967345 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:21:56.965467 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-tc5dx" event={"ID":"3c57e761-5812-43b8-a25f-f983c0dea9b0","Type":"ContainerDied","Data":"b5fa9a1ce2abf23e50e78d9db3809c58ae6edd7f7fc8d0424053f407d027ca62"} Nov 03 01:21:56.967345 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:21:56.965550 4421 scope.go:122] "RemoveContainer" containerID="c229c496285e24181e14e79befb1ee74dc2c0d1ce42f2f3b077fad7442c800ad" Nov 03 01:21:56.967345 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:21:56.965713 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-tc5dx" Nov 03 01:21:57.253087 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:21:57.253046 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-tc5dx"] Nov 03 01:21:57.260974 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:21:57.260938 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-tc5dx"] Nov 03 01:21:58.003162 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:21:58.002998 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c57e761-5812-43b8-a25f-f983c0dea9b0" path="/var/lib/kubelet/pods/3c57e761-5812-43b8-a25f-f983c0dea9b0/volumes" Nov 03 01:22:02.878010 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:22:02.877970 4421 scope.go:122] "RemoveContainer" containerID="d370c6f01369231bb6918b93173892863d5f868e89f92258affd730f451b8f45" Nov 03 01:22:02.899485 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:22:02.899461 4421 scope.go:122] "RemoveContainer" containerID="b7edec4b4715df42d415edaa293dffb409339ef52b7aedee62d053817135a056" Nov 03 01:22:02.910923 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:22:02.910899 4421 scope.go:122] "RemoveContainer" containerID="16c5a0a4f80f28c7b6d31df3a91689ef3b72a917d930e31fa2739d3b51a4fb36" Nov 03 01:22:02.923367 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:22:02.923258 4421 scope.go:122] "RemoveContainer" containerID="72f41cea4e90ff8b1b7a4d6d0af68b4b02b38c4baaf881c8069941248efa7a4f" Nov 03 01:22:02.936412 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:22:02.936381 4421 scope.go:122] "RemoveContainer" containerID="487c72a1af88c682515c46c7a05334f46a99bbe8e738ce4ff6f60c427a364ad5" Nov 03 01:22:06.759786 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:22:06.759747 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bda16dd6-j7lmk"] Nov 03 01:22:06.760742 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:22:06.760721 4421 cpu_manager.go:414] "containerMap: removing container" podUID="cf375ec3-3e92-40ac-a2c6-6ec506ce301d" containerName="revoke-keys" Nov 03 01:22:06.760806 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:22:06.760743 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf375ec3-3e92-40ac-a2c6-6ec506ce301d" containerName="revoke-keys" Nov 03 01:22:06.760806 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:22:06.760786 4421 cpu_manager.go:414] "containerMap: removing container" podUID="3c57e761-5812-43b8-a25f-f983c0dea9b0" containerName="istio-proxy" Nov 03 01:22:06.760806 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:22:06.760791 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c57e761-5812-43b8-a25f-f983c0dea9b0" containerName="istio-proxy" Nov 03 01:22:06.760952 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:22:06.760938 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c57e761-5812-43b8-a25f-f983c0dea9b0" containerName="istio-proxy" Nov 03 01:22:06.760993 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:22:06.760961 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf375ec3-3e92-40ac-a2c6-6ec506ce301d" containerName="revoke-keys" Nov 03 01:22:06.768554 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:22:06.768506 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bda16dd6-j7lmk" Nov 03 01:22:06.769575 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:22:06.769543 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bda16dd6-j7lmk"] Nov 03 01:22:06.773319 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:22:06.773241 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:22:06.773319 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:22:06.773319 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6q4xp\"" Nov 03 01:22:06.924303 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:22:06.921025 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kntf\" (UniqueName: \"kubernetes.io/projected/2749d4f6-690f-4c75-aed8-79b8b3ee832e-kube-api-access-5kntf\") pod \"maas-api-revoke-sub-bda16dd6-j7lmk\" (UID: \"2749d4f6-690f-4c75-aed8-79b8b3ee832e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bda16dd6-j7lmk" Nov 03 01:22:06.924303 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:22:06.921330 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2749d4f6-690f-4c75-aed8-79b8b3ee832e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bda16dd6-j7lmk\" (UID: \"2749d4f6-690f-4c75-aed8-79b8b3ee832e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bda16dd6-j7lmk" Nov 03 01:22:06.924303 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:22:06.923185 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2749d4f6-690f-4c75-aed8-79b8b3ee832e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bda16dd6-j7lmk\" (UID: \"2749d4f6-690f-4c75-aed8-79b8b3ee832e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bda16dd6-j7lmk" Nov 03 01:22:07.025857 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:22:07.025755 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kntf\" (UniqueName: \"kubernetes.io/projected/2749d4f6-690f-4c75-aed8-79b8b3ee832e-kube-api-access-5kntf\") pod \"maas-api-revoke-sub-bda16dd6-j7lmk\" (UID: \"2749d4f6-690f-4c75-aed8-79b8b3ee832e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bda16dd6-j7lmk" Nov 03 01:22:07.026050 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:22:07.026036 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2749d4f6-690f-4c75-aed8-79b8b3ee832e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bda16dd6-j7lmk\" (UID: \"2749d4f6-690f-4c75-aed8-79b8b3ee832e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bda16dd6-j7lmk" Nov 03 01:22:07.026205 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:22:07.026178 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2749d4f6-690f-4c75-aed8-79b8b3ee832e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bda16dd6-j7lmk\" (UID: \"2749d4f6-690f-4c75-aed8-79b8b3ee832e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bda16dd6-j7lmk" Nov 03 01:22:07.027135 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:22:07.027100 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2749d4f6-690f-4c75-aed8-79b8b3ee832e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bda16dd6-j7lmk\" (UID: \"2749d4f6-690f-4c75-aed8-79b8b3ee832e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bda16dd6-j7lmk" Nov 03 01:22:07.031392 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:22:07.031362 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2749d4f6-690f-4c75-aed8-79b8b3ee832e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bda16dd6-j7lmk\" (UID: \"2749d4f6-690f-4c75-aed8-79b8b3ee832e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bda16dd6-j7lmk" Nov 03 01:22:07.040068 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:22:07.040033 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kntf\" (UniqueName: \"kubernetes.io/projected/2749d4f6-690f-4c75-aed8-79b8b3ee832e-kube-api-access-5kntf\") pod \"maas-api-revoke-sub-bda16dd6-j7lmk\" (UID: \"2749d4f6-690f-4c75-aed8-79b8b3ee832e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bda16dd6-j7lmk" Nov 03 01:22:07.090663 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:22:07.090627 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bda16dd6-j7lmk" Nov 03 01:22:07.427626 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:22:07.427588 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bda16dd6-j7lmk"] Nov 03 01:22:07.428031 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:22:07.427988 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2749d4f6_690f_4c75_aed8_79b8b3ee832e.slice/crio-a34a93e486e173059e79b5cbf5d30a511ba70bd0dbdb29c039a1d500d0d6f505 WatchSource:0}: Error finding container a34a93e486e173059e79b5cbf5d30a511ba70bd0dbdb29c039a1d500d0d6f505: Status 404 returned error can't find the container with id a34a93e486e173059e79b5cbf5d30a511ba70bd0dbdb29c039a1d500d0d6f505 Nov 03 01:22:08.283359 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:22:08.283256 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bda16dd6-j7lmk" event={"ID":"2749d4f6-690f-4c75-aed8-79b8b3ee832e","Type":"ContainerStarted","Data":"d90180e72b36cf902d5a0e10aa43c45af2bda464c701b540431c67ad18d09cd0"} Nov 03 01:22:08.283906 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:22:08.283390 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bda16dd6-j7lmk" event={"ID":"2749d4f6-690f-4c75-aed8-79b8b3ee832e","Type":"ContainerStarted","Data":"a34a93e486e173059e79b5cbf5d30a511ba70bd0dbdb29c039a1d500d0d6f505"} Nov 03 01:22:08.303098 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:22:08.303041 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bda16dd6-j7lmk" podStartSLOduration=2.303015703 podStartE2EDuration="2.303015703s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:22:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:22:08.299752037 +0000 UTC m=+2470.470755134" watchObservedRunningTime="2026-11-03 01:22:08.303015703 +0000 UTC m=+2470.474019198" Nov 03 01:22:09.295014 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:22:09.294973 4421 generic.go:356] "Generic (PLEG): container finished" podID="2749d4f6-690f-4c75-aed8-79b8b3ee832e" containerID="d90180e72b36cf902d5a0e10aa43c45af2bda464c701b540431c67ad18d09cd0" exitCode=0 Nov 03 01:22:09.295531 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:22:09.295062 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bda16dd6-j7lmk" event={"ID":"2749d4f6-690f-4c75-aed8-79b8b3ee832e","Type":"ContainerDied","Data":"d90180e72b36cf902d5a0e10aa43c45af2bda464c701b540431c67ad18d09cd0"} Nov 03 01:22:10.523357 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:22:10.523324 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bda16dd6-j7lmk" Nov 03 01:22:10.604828 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:22:10.604738 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2749d4f6-690f-4c75-aed8-79b8b3ee832e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2749d4f6-690f-4c75-aed8-79b8b3ee832e-maas-api-openshift-ca\") pod \"2749d4f6-690f-4c75-aed8-79b8b3ee832e\" (UID: \"2749d4f6-690f-4c75-aed8-79b8b3ee832e\") " Nov 03 01:22:10.605007 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:22:10.604828 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2749d4f6-690f-4c75-aed8-79b8b3ee832e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2749d4f6-690f-4c75-aed8-79b8b3ee832e-maas-api-xks-ca\") pod \"2749d4f6-690f-4c75-aed8-79b8b3ee832e\" (UID: \"2749d4f6-690f-4c75-aed8-79b8b3ee832e\") " Nov 03 01:22:10.605007 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:22:10.604877 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2749d4f6-690f-4c75-aed8-79b8b3ee832e-kube-api-access-5kntf\" (UniqueName: \"kubernetes.io/projected/2749d4f6-690f-4c75-aed8-79b8b3ee832e-kube-api-access-5kntf\") pod \"2749d4f6-690f-4c75-aed8-79b8b3ee832e\" (UID: \"2749d4f6-690f-4c75-aed8-79b8b3ee832e\") " Nov 03 01:22:10.605251 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:22:10.605219 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2749d4f6-690f-4c75-aed8-79b8b3ee832e-maas-api-openshift-ca" pod "2749d4f6-690f-4c75-aed8-79b8b3ee832e" (UID: "2749d4f6-690f-4c75-aed8-79b8b3ee832e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:22:10.605755 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:22:10.605728 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2749d4f6-690f-4c75-aed8-79b8b3ee832e-maas-api-openshift-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:22:10.608001 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:22:10.607980 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2749d4f6-690f-4c75-aed8-79b8b3ee832e-maas-api-xks-ca" pod "2749d4f6-690f-4c75-aed8-79b8b3ee832e" (UID: "2749d4f6-690f-4c75-aed8-79b8b3ee832e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:22:10.608074 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:22:10.608037 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2749d4f6-690f-4c75-aed8-79b8b3ee832e-kube-api-access-5kntf" pod "2749d4f6-690f-4c75-aed8-79b8b3ee832e" (UID: "2749d4f6-690f-4c75-aed8-79b8b3ee832e"). InnerVolumeSpecName "kube-api-access-5kntf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:22:10.707778 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:22:10.707739 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2749d4f6-690f-4c75-aed8-79b8b3ee832e-maas-api-xks-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:22:10.707778 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:22:10.707771 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5kntf\" (UniqueName: \"kubernetes.io/projected/2749d4f6-690f-4c75-aed8-79b8b3ee832e-kube-api-access-5kntf\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:22:11.313412 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:22:11.313382 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bda16dd6-j7lmk" Nov 03 01:22:11.313623 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:22:11.313424 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bda16dd6-j7lmk" event={"ID":"2749d4f6-690f-4c75-aed8-79b8b3ee832e","Type":"ContainerDied","Data":"a34a93e486e173059e79b5cbf5d30a511ba70bd0dbdb29c039a1d500d0d6f505"} Nov 03 01:22:11.313623 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:22:11.313477 4421 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a34a93e486e173059e79b5cbf5d30a511ba70bd0dbdb29c039a1d500d0d6f505" Nov 03 01:22:14.690886 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:22:14.690849 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 01:22:24.983355 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:22:24.983319 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a88862-57bd8764b7-vsv9f"] Nov 03 01:22:24.984701 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:22:24.984678 4421 cpu_manager.go:414] "containerMap: removing container" podUID="2749d4f6-690f-4c75-aed8-79b8b3ee832e" containerName="revoke-keys" Nov 03 01:22:24.984806 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:22:24.984704 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2749d4f6-690f-4c75-aed8-79b8b3ee832e" containerName="revoke-keys" Nov 03 01:22:24.984981 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:22:24.984964 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="2749d4f6-690f-4c75-aed8-79b8b3ee832e" containerName="revoke-keys" Nov 03 01:22:24.990163 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:22:24.990138 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a88862-57bd8764b7-vsv9f" Nov 03 01:22:24.996893 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:22:24.996853 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-a88862-dockercfg-7z2q8\"" Nov 03 01:22:25.004009 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:22:25.003984 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a88862-57bd8764b7-vsv9f"] Nov 03 01:22:25.056479 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:22:25.056433 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whpv8\" (UniqueName: \"kubernetes.io/projected/867b2cbf-fc7e-4fc0-816b-53cee0b424f8-kube-api-access-whpv8\") pod \"payload-pre-processing-e2e-inf-a-a88862-57bd8764b7-vsv9f\" (UID: \"867b2cbf-fc7e-4fc0-816b-53cee0b424f8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a88862-57bd8764b7-vsv9f" Nov 03 01:22:25.056678 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:22:25.056597 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/867b2cbf-fc7e-4fc0-816b-53cee0b424f8-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-a88862-57bd8764b7-vsv9f\" (UID: \"867b2cbf-fc7e-4fc0-816b-53cee0b424f8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a88862-57bd8764b7-vsv9f" Nov 03 01:22:25.081431 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:22:25.081387 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-a88862\"" Nov 03 01:22:25.164338 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:22:25.164300 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/867b2cbf-fc7e-4fc0-816b-53cee0b424f8-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-a88862-57bd8764b7-vsv9f\" (UID: \"867b2cbf-fc7e-4fc0-816b-53cee0b424f8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a88862-57bd8764b7-vsv9f" Nov 03 01:22:25.178532 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:22:25.178484 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whpv8\" (UniqueName: \"kubernetes.io/projected/867b2cbf-fc7e-4fc0-816b-53cee0b424f8-kube-api-access-whpv8\") pod \"payload-pre-processing-e2e-inf-a-a88862-57bd8764b7-vsv9f\" (UID: \"867b2cbf-fc7e-4fc0-816b-53cee0b424f8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a88862-57bd8764b7-vsv9f" Nov 03 01:22:25.179314 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:22:25.169547 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/867b2cbf-fc7e-4fc0-816b-53cee0b424f8-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-a88862-57bd8764b7-vsv9f\" (UID: \"867b2cbf-fc7e-4fc0-816b-53cee0b424f8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a88862-57bd8764b7-vsv9f" Nov 03 01:22:25.211529 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:22:25.206079 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whpv8\" (UniqueName: \"kubernetes.io/projected/867b2cbf-fc7e-4fc0-816b-53cee0b424f8-kube-api-access-whpv8\") pod \"payload-pre-processing-e2e-inf-a-a88862-57bd8764b7-vsv9f\" (UID: \"867b2cbf-fc7e-4fc0-816b-53cee0b424f8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a88862-57bd8764b7-vsv9f" Nov 03 01:22:25.326002 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:22:25.325969 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a88862-57bd8764b7-vsv9f" Nov 03 01:22:25.760092 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:22:25.760057 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod867b2cbf_fc7e_4fc0_816b_53cee0b424f8.slice/crio-ccf8dcfe86fb69d38ad2b46a8a7d0f8ec4e917316c54704a22de795e67883c91 WatchSource:0}: Error finding container ccf8dcfe86fb69d38ad2b46a8a7d0f8ec4e917316c54704a22de795e67883c91: Status 404 returned error can't find the container with id ccf8dcfe86fb69d38ad2b46a8a7d0f8ec4e917316c54704a22de795e67883c91 Nov 03 01:22:25.760836 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:22:25.760804 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a88862-57bd8764b7-vsv9f"] Nov 03 01:22:26.512787 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:22:26.512748 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a88862-57bd8764b7-vsv9f" event={"ID":"867b2cbf-fc7e-4fc0-816b-53cee0b424f8","Type":"ContainerStarted","Data":"fe21c6cab8d4b8ef2165d437a570c2861d159905c4f18047116e3004cb23c12c"} Nov 03 01:22:26.513359 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:22:26.512806 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a88862-57bd8764b7-vsv9f" event={"ID":"867b2cbf-fc7e-4fc0-816b-53cee0b424f8","Type":"ContainerStarted","Data":"ccf8dcfe86fb69d38ad2b46a8a7d0f8ec4e917316c54704a22de795e67883c91"} Nov 03 01:22:26.513359 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:22:26.512857 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a88862-57bd8764b7-vsv9f" Nov 03 01:22:26.529197 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:22:26.529136 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a88862-57bd8764b7-vsv9f" podStartSLOduration=2.5291219050000002 podStartE2EDuration="2.529121905s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:22:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:22:26.527722866 +0000 UTC m=+2488.698725963" watchObservedRunningTime="2026-11-03 01:22:26.529121905 +0000 UTC m=+2488.700125030" Nov 03 01:22:36.166317 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:22:36.165480 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/simulator-w2-f7680d-kserve-85bdc86cd6-8gxrg" podUID="2bdcf521-cebd-4d6b-8554-1b60db8b0f3c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:22:36.321473 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:22:36.321424 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/distinct-w3-83755a-kserve-86bdbbdd85-2cnnl" podUID="f4ae9624-d127-4a55-837d-7c9ff0267b37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:22:36.323000 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:22:36.322964 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-650be0/simulator-w4-650be0-kserve-7bc588774f-gvz2k" podUID="a827fa88-f078-4312-8f12-2c799c11b461" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:22:36.341427 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:22:36.341385 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-650be0/premium-w4-650be0-kserve-7bf8555dfb-z97n5" podUID="40601dc5-808f-456f-826f-6e0c35966ada" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Nov 03 01:22:36.354634 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:22:36.354587 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/premium-w3-83755a-kserve-7c9d66988b-lt4xd" podUID="c0c6b02e-2ecb-428e-a13d-be6d7cd1ba39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Nov 03 01:22:36.505747 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:22:36.505704 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/simulator-w3-83755a-kserve-77d6664559-psqtg" podUID="b0a4c234-af58-4da0-a1a4-e1f167953f6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Nov 03 01:22:36.859494 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:22:36.859375 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8gm8n" podUID="2d367e12-e46f-4a68-9825-9b1c934d4d12" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:22:37.013426 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:22:37.013373 4421 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-83755a/distinct-w3-83755a-kserve-86bdbbdd85-2cnnl" podUID="f4ae9624-d127-4a55-837d-7c9ff0267b37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:22:37.169039 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:22:37.168928 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/simulator-w2-f7680d-kserve-85bdc86cd6-8gxrg" podUID="2bdcf521-cebd-4d6b-8554-1b60db8b0f3c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:22:37.183177 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:22:37.182881 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f7680d/simulator-w2-f7680d-kserve-85bdc86cd6-8gxrg" Nov 03 01:22:37.325637 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:22:37.325600 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-650be0/simulator-w4-650be0-kserve-7bc588774f-gvz2k" podUID="a827fa88-f078-4312-8f12-2c799c11b461" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Nov 03 01:22:37.339297 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:22:37.327145 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-650be0/simulator-w4-650be0-kserve-7bc588774f-gvz2k" Nov 03 01:22:37.354329 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:22:37.344462 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/distinct-w3-83755a-kserve-86bdbbdd85-2cnnl" podUID="f4ae9624-d127-4a55-837d-7c9ff0267b37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:22:37.354329 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:22:37.345129 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-83755a/distinct-w3-83755a-kserve-86bdbbdd85-2cnnl" Nov 03 01:22:37.354329 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:22:37.349769 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-650be0/premium-w4-650be0-kserve-7bf8555dfb-z97n5" podUID="40601dc5-808f-456f-826f-6e0c35966ada" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:22:37.354329 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:22:37.351808 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-650be0/premium-w4-650be0-kserve-7bf8555dfb-z97n5" Nov 03 01:22:37.357101 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:22:37.357065 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/premium-w3-83755a-kserve-7c9d66988b-lt4xd" podUID="c0c6b02e-2ecb-428e-a13d-be6d7cd1ba39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:22:37.361435 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:22:37.357497 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-83755a/premium-w3-83755a-kserve-7c9d66988b-lt4xd" Nov 03 01:22:37.603882 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:22:37.603849 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-650be0/simulator-w4-650be0-kserve-7bc588774f-gvz2k" Nov 03 01:22:37.631484 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:22:37.627717 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f7680d/simulator-w2-f7680d-kserve-85bdc86cd6-8gxrg" Nov 03 01:22:37.631484 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:22:37.627783 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a88862-57bd8764b7-vsv9f" Nov 03 01:22:37.638019 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:22:37.637987 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-83755a/premium-w3-83755a-kserve-7c9d66988b-lt4xd" Nov 03 01:22:37.680939 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:22:37.680906 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-83755a/distinct-w3-83755a-kserve-86bdbbdd85-2cnnl" Nov 03 01:22:37.763299 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:22:37.722810 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-650be0/premium-w4-650be0-kserve-7bf8555dfb-z97n5" Nov 03 01:22:44.673629 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:22:44.673588 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 01:22:53.029921 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:22:53.029873 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37d9f0bb-k7qkd"] Nov 03 01:22:53.035497 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:22:53.035468 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37d9f0bb-k7qkd"] Nov 03 01:22:53.982739 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:22:53.982701 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f01b6791-5118-430d-9034-8f9760f9d963" path="/var/lib/kubelet/pods/f01b6791-5118-430d-9034-8f9760f9d963/volumes" Nov 03 01:22:58.729214 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:22:58.729174 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-5lkrv"] Nov 03 01:22:58.729779 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:22:58.729463 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-5lkrv" podUID="09633e29-ea24-4847-b1c5-81d5843899fb" containerName="istio-proxy" containerID="cri-o://0c9caf837809fb9c889cfa01cddb4f580bf3f63edca9945464ab3bba06eeea11" gracePeriod=30 Nov 03 01:22:59.989165 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:22:59.989132 4421 patch_prober.go:29] interesting pod/e2e-extmod-a-d25759-openshift-default-d47fd9785-5lkrv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 01:22:59.989575 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:22:59.989183 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-5lkrv" podUID="09633e29-ea24-4847-b1c5-81d5843899fb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 01:23:03.993402 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:23:03.993360 4421 generic.go:356] "Generic (PLEG): container finished" podID="09633e29-ea24-4847-b1c5-81d5843899fb" containerID="0c9caf837809fb9c889cfa01cddb4f580bf3f63edca9945464ab3bba06eeea11" exitCode=0 Nov 03 01:23:03.993820 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:23:03.993430 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-5lkrv" event={"ID":"09633e29-ea24-4847-b1c5-81d5843899fb","Type":"ContainerDied","Data":"0c9caf837809fb9c889cfa01cddb4f580bf3f63edca9945464ab3bba06eeea11"} Nov 03 01:23:04.179740 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:23:04.179702 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-5lkrv" Nov 03 01:23:04.342193 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:23:04.342155 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09633e29-ea24-4847-b1c5-81d5843899fb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09633e29-ea24-4847-b1c5-81d5843899fb-istiod-ca-cert\") pod \"09633e29-ea24-4847-b1c5-81d5843899fb\" (UID: \"09633e29-ea24-4847-b1c5-81d5843899fb\") " Nov 03 01:23:04.342429 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:23:04.342263 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09633e29-ea24-4847-b1c5-81d5843899fb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09633e29-ea24-4847-b1c5-81d5843899fb-credential-socket\") pod \"09633e29-ea24-4847-b1c5-81d5843899fb\" (UID: \"09633e29-ea24-4847-b1c5-81d5843899fb\") " Nov 03 01:23:04.342429 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:23:04.342352 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09633e29-ea24-4847-b1c5-81d5843899fb-kube-api-access-mc9x2\" (UniqueName: \"kubernetes.io/projected/09633e29-ea24-4847-b1c5-81d5843899fb-kube-api-access-mc9x2\") pod \"09633e29-ea24-4847-b1c5-81d5843899fb\" (UID: \"09633e29-ea24-4847-b1c5-81d5843899fb\") " Nov 03 01:23:04.342595 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:23:04.342578 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09633e29-ea24-4847-b1c5-81d5843899fb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09633e29-ea24-4847-b1c5-81d5843899fb-istio-envoy\") pod \"09633e29-ea24-4847-b1c5-81d5843899fb\" (UID: \"09633e29-ea24-4847-b1c5-81d5843899fb\") " Nov 03 01:23:04.342666 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:23:04.342629 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09633e29-ea24-4847-b1c5-81d5843899fb-istio-token\" (UniqueName: \"kubernetes.io/projected/09633e29-ea24-4847-b1c5-81d5843899fb-istio-token\") pod \"09633e29-ea24-4847-b1c5-81d5843899fb\" (UID: \"09633e29-ea24-4847-b1c5-81d5843899fb\") " Nov 03 01:23:04.342723 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:23:04.342662 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/09633e29-ea24-4847-b1c5-81d5843899fb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09633e29-ea24-4847-b1c5-81d5843899fb-istio-podinfo\") pod \"09633e29-ea24-4847-b1c5-81d5843899fb\" (UID: \"09633e29-ea24-4847-b1c5-81d5843899fb\") " Nov 03 01:23:04.342785 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:23:04.342769 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09633e29-ea24-4847-b1c5-81d5843899fb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09633e29-ea24-4847-b1c5-81d5843899fb-workload-socket\") pod \"09633e29-ea24-4847-b1c5-81d5843899fb\" (UID: \"09633e29-ea24-4847-b1c5-81d5843899fb\") " Nov 03 01:23:04.342841 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:23:04.342659 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09633e29-ea24-4847-b1c5-81d5843899fb-istiod-ca-cert" pod "09633e29-ea24-4847-b1c5-81d5843899fb" (UID: "09633e29-ea24-4847-b1c5-81d5843899fb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:23:04.343011 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:23:04.342986 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09633e29-ea24-4847-b1c5-81d5843899fb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09633e29-ea24-4847-b1c5-81d5843899fb-workload-certs\") pod \"09633e29-ea24-4847-b1c5-81d5843899fb\" (UID: \"09633e29-ea24-4847-b1c5-81d5843899fb\") " Nov 03 01:23:04.343097 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:23:04.343057 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09633e29-ea24-4847-b1c5-81d5843899fb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09633e29-ea24-4847-b1c5-81d5843899fb-istio-data\") pod \"09633e29-ea24-4847-b1c5-81d5843899fb\" (UID: \"09633e29-ea24-4847-b1c5-81d5843899fb\") " Nov 03 01:23:04.343097 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:23:04.343058 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09633e29-ea24-4847-b1c5-81d5843899fb-workload-socket" pod "09633e29-ea24-4847-b1c5-81d5843899fb" (UID: "09633e29-ea24-4847-b1c5-81d5843899fb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:23:04.343097 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:23:04.343077 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09633e29-ea24-4847-b1c5-81d5843899fb-credential-socket" pod "09633e29-ea24-4847-b1c5-81d5843899fb" (UID: "09633e29-ea24-4847-b1c5-81d5843899fb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:23:04.343314 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:23:04.343270 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09633e29-ea24-4847-b1c5-81d5843899fb-workload-certs" pod "09633e29-ea24-4847-b1c5-81d5843899fb" (UID: "09633e29-ea24-4847-b1c5-81d5843899fb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:23:04.343712 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:23:04.343690 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09633e29-ea24-4847-b1c5-81d5843899fb-istio-data" pod "09633e29-ea24-4847-b1c5-81d5843899fb" (UID: "09633e29-ea24-4847-b1c5-81d5843899fb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:23:04.343835 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:23:04.343819 4421 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09633e29-ea24-4847-b1c5-81d5843899fb-credential-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:23:04.343913 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:23:04.343838 4421 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09633e29-ea24-4847-b1c5-81d5843899fb-workload-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:23:04.343913 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:23:04.343848 4421 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09633e29-ea24-4847-b1c5-81d5843899fb-workload-certs\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:23:04.343913 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:23:04.343858 4421 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09633e29-ea24-4847-b1c5-81d5843899fb-istio-data\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:23:04.343913 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:23:04.343867 4421 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09633e29-ea24-4847-b1c5-81d5843899fb-istiod-ca-cert\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:23:04.346013 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:23:04.345989 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09633e29-ea24-4847-b1c5-81d5843899fb-istio-token" pod "09633e29-ea24-4847-b1c5-81d5843899fb" (UID: "09633e29-ea24-4847-b1c5-81d5843899fb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:23:04.420486 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:23:04.420372 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/09633e29-ea24-4847-b1c5-81d5843899fb-istio-podinfo" pod "09633e29-ea24-4847-b1c5-81d5843899fb" (UID: "09633e29-ea24-4847-b1c5-81d5843899fb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:23:04.420486 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:23:04.420398 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09633e29-ea24-4847-b1c5-81d5843899fb-istio-envoy" pod "09633e29-ea24-4847-b1c5-81d5843899fb" (UID: "09633e29-ea24-4847-b1c5-81d5843899fb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:23:04.420486 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:23:04.420439 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09633e29-ea24-4847-b1c5-81d5843899fb-kube-api-access-mc9x2" pod "09633e29-ea24-4847-b1c5-81d5843899fb" (UID: "09633e29-ea24-4847-b1c5-81d5843899fb"). InnerVolumeSpecName "kube-api-access-mc9x2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:23:04.445288 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:23:04.445247 4421 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09633e29-ea24-4847-b1c5-81d5843899fb-istio-envoy\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:23:04.445484 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:23:04.445333 4421 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09633e29-ea24-4847-b1c5-81d5843899fb-istio-token\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:23:04.445484 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:23:04.445348 4421 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09633e29-ea24-4847-b1c5-81d5843899fb-istio-podinfo\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:23:04.445484 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:23:04.445363 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mc9x2\" (UniqueName: \"kubernetes.io/projected/09633e29-ea24-4847-b1c5-81d5843899fb-kube-api-access-mc9x2\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:23:05.004216 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:23:05.004165 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-5lkrv" event={"ID":"09633e29-ea24-4847-b1c5-81d5843899fb","Type":"ContainerDied","Data":"37fc4672e6f48baeb6341947a95f77ec5de169452acd2380c5a44ccefcea59bb"} Nov 03 01:23:05.004813 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:23:05.004242 4421 scope.go:122] "RemoveContainer" containerID="0c9caf837809fb9c889cfa01cddb4f580bf3f63edca9945464ab3bba06eeea11" Nov 03 01:23:05.004813 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:23:05.004176 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-5lkrv" Nov 03 01:23:05.027038 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:23:05.027013 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-089a375e-nbjdz"] Nov 03 01:23:05.032812 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:23:05.032781 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-089a375e-nbjdz"] Nov 03 01:23:05.038434 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:23:05.038410 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-5lkrv"] Nov 03 01:23:05.043646 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:23:05.043620 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-5lkrv"] Nov 03 01:23:05.982336 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:23:05.982262 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09633e29-ea24-4847-b1c5-81d5843899fb" path="/var/lib/kubelet/pods/09633e29-ea24-4847-b1c5-81d5843899fb/volumes" Nov 03 01:23:05.983471 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:23:05.983448 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4630e268-c1f1-4954-9e8c-f07c2dc28d6c" path="/var/lib/kubelet/pods/4630e268-c1f1-4954-9e8c-f07c2dc28d6c/volumes" Nov 03 01:23:10.459354 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:23:10.459318 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-fcw4z"] Nov 03 01:23:10.460504 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:23:10.460480 4421 cpu_manager.go:414] "containerMap: removing container" podUID="09633e29-ea24-4847-b1c5-81d5843899fb" containerName="istio-proxy" Nov 03 01:23:10.460504 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:23:10.460504 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09633e29-ea24-4847-b1c5-81d5843899fb" containerName="istio-proxy" Nov 03 01:23:10.460764 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:23:10.460747 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="09633e29-ea24-4847-b1c5-81d5843899fb" containerName="istio-proxy" Nov 03 01:23:10.468688 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:23:10.468660 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-fcw4z" Nov 03 01:23:10.474817 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:23:10.474789 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-ce04af-openshift-default-dockercfg-rdq4s\"" Nov 03 01:23:10.475796 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:23:10.475758 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-fcw4z"] Nov 03 01:23:10.598542 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:23:10.598477 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37f37c99-0b3d-45d2-ba18-a4ec39043f67-istiod-ca-cert\") pod \"e2e-inf-b-ce04af-openshift-default-7dccd45f47-fcw4z\" (UID: \"37f37c99-0b3d-45d2-ba18-a4ec39043f67\") " pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-fcw4z" Nov 03 01:23:10.598943 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:23:10.598904 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37f37c99-0b3d-45d2-ba18-a4ec39043f67-workload-certs\") pod \"e2e-inf-b-ce04af-openshift-default-7dccd45f47-fcw4z\" (UID: \"37f37c99-0b3d-45d2-ba18-a4ec39043f67\") " pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-fcw4z" Nov 03 01:23:10.602296 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:23:10.599117 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37f37c99-0b3d-45d2-ba18-a4ec39043f67-istio-envoy\") pod \"e2e-inf-b-ce04af-openshift-default-7dccd45f47-fcw4z\" (UID: \"37f37c99-0b3d-45d2-ba18-a4ec39043f67\") " pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-fcw4z" Nov 03 01:23:10.630298 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:23:10.603426 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37f37c99-0b3d-45d2-ba18-a4ec39043f67-credential-socket\") pod \"e2e-inf-b-ce04af-openshift-default-7dccd45f47-fcw4z\" (UID: \"37f37c99-0b3d-45d2-ba18-a4ec39043f67\") " pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-fcw4z" Nov 03 01:23:10.630639 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:23:10.630607 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tczf6\" (UniqueName: \"kubernetes.io/projected/37f37c99-0b3d-45d2-ba18-a4ec39043f67-kube-api-access-tczf6\") pod \"e2e-inf-b-ce04af-openshift-default-7dccd45f47-fcw4z\" (UID: \"37f37c99-0b3d-45d2-ba18-a4ec39043f67\") " pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-fcw4z" Nov 03 01:23:10.630882 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:23:10.630859 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37f37c99-0b3d-45d2-ba18-a4ec39043f67-istio-podinfo\") pod \"e2e-inf-b-ce04af-openshift-default-7dccd45f47-fcw4z\" (UID: \"37f37c99-0b3d-45d2-ba18-a4ec39043f67\") " pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-fcw4z" Nov 03 01:23:10.631016 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:23:10.631000 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37f37c99-0b3d-45d2-ba18-a4ec39043f67-istio-token\") pod \"e2e-inf-b-ce04af-openshift-default-7dccd45f47-fcw4z\" (UID: \"37f37c99-0b3d-45d2-ba18-a4ec39043f67\") " pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-fcw4z" Nov 03 01:23:10.638296 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:23:10.631191 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37f37c99-0b3d-45d2-ba18-a4ec39043f67-istio-data\") pod \"e2e-inf-b-ce04af-openshift-default-7dccd45f47-fcw4z\" (UID: \"37f37c99-0b3d-45d2-ba18-a4ec39043f67\") " pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-fcw4z" Nov 03 01:23:10.638604 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:23:10.638577 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37f37c99-0b3d-45d2-ba18-a4ec39043f67-workload-socket\") pod \"e2e-inf-b-ce04af-openshift-default-7dccd45f47-fcw4z\" (UID: \"37f37c99-0b3d-45d2-ba18-a4ec39043f67\") " pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-fcw4z" Nov 03 01:23:10.757296 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:23:10.756720 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37f37c99-0b3d-45d2-ba18-a4ec39043f67-workload-certs\") pod \"e2e-inf-b-ce04af-openshift-default-7dccd45f47-fcw4z\" (UID: \"37f37c99-0b3d-45d2-ba18-a4ec39043f67\") " pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-fcw4z" Nov 03 01:23:10.757296 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:23:10.756791 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37f37c99-0b3d-45d2-ba18-a4ec39043f67-istio-envoy\") pod \"e2e-inf-b-ce04af-openshift-default-7dccd45f47-fcw4z\" (UID: \"37f37c99-0b3d-45d2-ba18-a4ec39043f67\") " pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-fcw4z" Nov 03 01:23:10.757296 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:23:10.756959 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37f37c99-0b3d-45d2-ba18-a4ec39043f67-credential-socket\") pod \"e2e-inf-b-ce04af-openshift-default-7dccd45f47-fcw4z\" (UID: \"37f37c99-0b3d-45d2-ba18-a4ec39043f67\") " pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-fcw4z" Nov 03 01:23:10.757296 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:23:10.757000 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tczf6\" (UniqueName: \"kubernetes.io/projected/37f37c99-0b3d-45d2-ba18-a4ec39043f67-kube-api-access-tczf6\") pod \"e2e-inf-b-ce04af-openshift-default-7dccd45f47-fcw4z\" (UID: \"37f37c99-0b3d-45d2-ba18-a4ec39043f67\") " pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-fcw4z" Nov 03 01:23:10.757296 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:23:10.757091 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37f37c99-0b3d-45d2-ba18-a4ec39043f67-istio-podinfo\") pod \"e2e-inf-b-ce04af-openshift-default-7dccd45f47-fcw4z\" (UID: \"37f37c99-0b3d-45d2-ba18-a4ec39043f67\") " pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-fcw4z" Nov 03 01:23:10.757296 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:23:10.757119 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37f37c99-0b3d-45d2-ba18-a4ec39043f67-istio-token\") pod \"e2e-inf-b-ce04af-openshift-default-7dccd45f47-fcw4z\" (UID: \"37f37c99-0b3d-45d2-ba18-a4ec39043f67\") " pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-fcw4z" Nov 03 01:23:10.757296 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:23:10.757209 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37f37c99-0b3d-45d2-ba18-a4ec39043f67-istio-data\") pod \"e2e-inf-b-ce04af-openshift-default-7dccd45f47-fcw4z\" (UID: \"37f37c99-0b3d-45d2-ba18-a4ec39043f67\") " pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-fcw4z" Nov 03 01:23:10.757795 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:23:10.757730 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37f37c99-0b3d-45d2-ba18-a4ec39043f67-workload-socket\") pod \"e2e-inf-b-ce04af-openshift-default-7dccd45f47-fcw4z\" (UID: \"37f37c99-0b3d-45d2-ba18-a4ec39043f67\") " pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-fcw4z" Nov 03 01:23:10.759306 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:23:10.757994 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37f37c99-0b3d-45d2-ba18-a4ec39043f67-workload-certs\") pod \"e2e-inf-b-ce04af-openshift-default-7dccd45f47-fcw4z\" (UID: \"37f37c99-0b3d-45d2-ba18-a4ec39043f67\") " pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-fcw4z" Nov 03 01:23:10.759306 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:23:10.758356 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37f37c99-0b3d-45d2-ba18-a4ec39043f67-workload-socket\") pod \"e2e-inf-b-ce04af-openshift-default-7dccd45f47-fcw4z\" (UID: \"37f37c99-0b3d-45d2-ba18-a4ec39043f67\") " pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-fcw4z" Nov 03 01:23:10.759306 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:23:10.758536 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37f37c99-0b3d-45d2-ba18-a4ec39043f67-istiod-ca-cert\") pod \"e2e-inf-b-ce04af-openshift-default-7dccd45f47-fcw4z\" (UID: \"37f37c99-0b3d-45d2-ba18-a4ec39043f67\") " pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-fcw4z" Nov 03 01:23:10.759882 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:23:10.759852 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37f37c99-0b3d-45d2-ba18-a4ec39043f67-istiod-ca-cert\") pod \"e2e-inf-b-ce04af-openshift-default-7dccd45f47-fcw4z\" (UID: \"37f37c99-0b3d-45d2-ba18-a4ec39043f67\") " pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-fcw4z" Nov 03 01:23:10.760149 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:23:10.760125 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37f37c99-0b3d-45d2-ba18-a4ec39043f67-credential-socket\") pod \"e2e-inf-b-ce04af-openshift-default-7dccd45f47-fcw4z\" (UID: \"37f37c99-0b3d-45d2-ba18-a4ec39043f67\") " pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-fcw4z" Nov 03 01:23:10.784299 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:23:10.779566 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37f37c99-0b3d-45d2-ba18-a4ec39043f67-istio-data\") pod \"e2e-inf-b-ce04af-openshift-default-7dccd45f47-fcw4z\" (UID: \"37f37c99-0b3d-45d2-ba18-a4ec39043f67\") " pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-fcw4z" Nov 03 01:23:10.784299 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:23:10.781451 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37f37c99-0b3d-45d2-ba18-a4ec39043f67-istio-envoy\") pod \"e2e-inf-b-ce04af-openshift-default-7dccd45f47-fcw4z\" (UID: \"37f37c99-0b3d-45d2-ba18-a4ec39043f67\") " pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-fcw4z" Nov 03 01:23:10.792580 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:23:10.786713 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37f37c99-0b3d-45d2-ba18-a4ec39043f67-istio-token\") pod \"e2e-inf-b-ce04af-openshift-default-7dccd45f47-fcw4z\" (UID: \"37f37c99-0b3d-45d2-ba18-a4ec39043f67\") " pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-fcw4z" Nov 03 01:23:10.800221 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:23:10.800180 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tczf6\" (UniqueName: \"kubernetes.io/projected/37f37c99-0b3d-45d2-ba18-a4ec39043f67-kube-api-access-tczf6\") pod \"e2e-inf-b-ce04af-openshift-default-7dccd45f47-fcw4z\" (UID: \"37f37c99-0b3d-45d2-ba18-a4ec39043f67\") " pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-fcw4z" Nov 03 01:23:10.805710 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:23:10.805671 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37f37c99-0b3d-45d2-ba18-a4ec39043f67-istio-podinfo\") pod \"e2e-inf-b-ce04af-openshift-default-7dccd45f47-fcw4z\" (UID: \"37f37c99-0b3d-45d2-ba18-a4ec39043f67\") " pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-fcw4z" Nov 03 01:23:11.095257 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:23:11.094895 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-fcw4z" Nov 03 01:23:11.440630 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:23:11.440581 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37f37c99_0b3d_45d2_ba18_a4ec39043f67.slice/crio-149dcfe320145bc60419ad9b487294d4a88a083d369794664f199913008797d0 WatchSource:0}: Error finding container 149dcfe320145bc60419ad9b487294d4a88a083d369794664f199913008797d0: Status 404 returned error can't find the container with id 149dcfe320145bc60419ad9b487294d4a88a083d369794664f199913008797d0 Nov 03 01:23:11.440786 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:23:11.440639 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-fcw4z"] Nov 03 01:23:11.444201 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:23:11.444164 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:23:11.444336 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:23:11.444242 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:23:11.444336 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:23:11.444310 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:23:12.171148 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:23:12.171104 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-fcw4z" event={"ID":"37f37c99-0b3d-45d2-ba18-a4ec39043f67","Type":"ContainerStarted","Data":"cb413e1300f544506216d31332bc15a5a0787f3e218711849e2e819ef25dba88"} Nov 03 01:23:12.171705 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:23:12.171487 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-fcw4z" event={"ID":"37f37c99-0b3d-45d2-ba18-a4ec39043f67","Type":"ContainerStarted","Data":"149dcfe320145bc60419ad9b487294d4a88a083d369794664f199913008797d0"} Nov 03 01:23:12.195611 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:23:12.195556 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-fcw4z" podStartSLOduration=2.195542595 podStartE2EDuration="2.195542595s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:23:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:23:12.190064765 +0000 UTC m=+2534.361067860" watchObservedRunningTime="2026-11-03 01:23:12.195542595 +0000 UTC m=+2534.366545692" Nov 03 01:23:13.105304 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:23:13.102679 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-fcw4z" Nov 03 01:23:13.167042 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:23:13.166633 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-fcw4z" Nov 03 01:23:13.262470 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:23:13.262393 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-fcw4z" Nov 03 01:23:13.322879 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:23:13.299173 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-fcw4z" Nov 03 01:23:14.687837 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:23:14.687791 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 01:23:15.944932 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:23:15.944898 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ce04af-6b9b884869-vwgxv"] Nov 03 01:23:15.951840 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:23:15.951808 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-ce04af-6b9b884869-vwgxv" Nov 03 01:23:15.957374 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:23:15.957345 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ce04af-6b9b884869-vwgxv"] Nov 03 01:23:15.961582 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:23:15.961332 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-ce04af\"" Nov 03 01:23:15.967687 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:23:15.967163 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ce04af-ffcb7d45f-zdv8k"] Nov 03 01:23:15.977756 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:23:15.977149 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ce04af-ffcb7d45f-zdv8k"] Nov 03 01:23:15.977756 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:23:15.977358 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ce04af-ffcb7d45f-zdv8k" Nov 03 01:23:15.981957 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:23:15.981930 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-ce04af-dockercfg-lftdn\"" Nov 03 01:23:16.030064 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:23:16.030024 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dab68fea-lhhm4"] Nov 03 01:23:16.032785 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:23:16.032747 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee89f011-059f-493a-becf-4816c5550dbf-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-ce04af-ffcb7d45f-zdv8k\" (UID: \"ee89f011-059f-493a-becf-4816c5550dbf\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ce04af-ffcb7d45f-zdv8k" Nov 03 01:23:16.033000 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:23:16.032981 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6tbp\" (UniqueName: \"kubernetes.io/projected/dbd53b13-13c5-449a-b204-1fc94ac8afad-kube-api-access-m6tbp\") pod \"payload-processing-e2e-inf-b-ce04af-6b9b884869-vwgxv\" (UID: \"dbd53b13-13c5-449a-b204-1fc94ac8afad\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ce04af-6b9b884869-vwgxv" Nov 03 01:23:16.033207 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:23:16.033190 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7f74\" (UniqueName: \"kubernetes.io/projected/ee89f011-059f-493a-becf-4816c5550dbf-kube-api-access-c7f74\") pod \"payload-pre-processing-e2e-inf-b-ce04af-ffcb7d45f-zdv8k\" (UID: \"ee89f011-059f-493a-becf-4816c5550dbf\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ce04af-ffcb7d45f-zdv8k" Nov 03 01:23:16.033328 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:23:16.033264 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbd53b13-13c5-449a-b204-1fc94ac8afad-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-ce04af-6b9b884869-vwgxv\" (UID: \"dbd53b13-13c5-449a-b204-1fc94ac8afad\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ce04af-6b9b884869-vwgxv" Nov 03 01:23:16.036059 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:23:16.036035 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dab68fea-lhhm4"] Nov 03 01:23:16.160297 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:23:16.145004 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee89f011-059f-493a-becf-4816c5550dbf-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-ce04af-ffcb7d45f-zdv8k\" (UID: \"ee89f011-059f-493a-becf-4816c5550dbf\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ce04af-ffcb7d45f-zdv8k" Nov 03 01:23:16.160297 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:23:16.145191 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6tbp\" (UniqueName: \"kubernetes.io/projected/dbd53b13-13c5-449a-b204-1fc94ac8afad-kube-api-access-m6tbp\") pod \"payload-processing-e2e-inf-b-ce04af-6b9b884869-vwgxv\" (UID: \"dbd53b13-13c5-449a-b204-1fc94ac8afad\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ce04af-6b9b884869-vwgxv" Nov 03 01:23:16.160297 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:23:16.145395 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7f74\" (UniqueName: \"kubernetes.io/projected/ee89f011-059f-493a-becf-4816c5550dbf-kube-api-access-c7f74\") pod \"payload-pre-processing-e2e-inf-b-ce04af-ffcb7d45f-zdv8k\" (UID: \"ee89f011-059f-493a-becf-4816c5550dbf\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ce04af-ffcb7d45f-zdv8k" Nov 03 01:23:16.160297 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:23:16.145481 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbd53b13-13c5-449a-b204-1fc94ac8afad-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-ce04af-6b9b884869-vwgxv\" (UID: \"dbd53b13-13c5-449a-b204-1fc94ac8afad\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ce04af-6b9b884869-vwgxv" Nov 03 01:23:16.217343 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:23:16.213947 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee89f011-059f-493a-becf-4816c5550dbf-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-ce04af-ffcb7d45f-zdv8k\" (UID: \"ee89f011-059f-493a-becf-4816c5550dbf\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ce04af-ffcb7d45f-zdv8k" Nov 03 01:23:16.217343 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:23:16.214125 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbd53b13-13c5-449a-b204-1fc94ac8afad-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-ce04af-6b9b884869-vwgxv\" (UID: \"dbd53b13-13c5-449a-b204-1fc94ac8afad\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ce04af-6b9b884869-vwgxv" Nov 03 01:23:16.303325 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:23:16.296078 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7f74\" (UniqueName: \"kubernetes.io/projected/ee89f011-059f-493a-becf-4816c5550dbf-kube-api-access-c7f74\") pod \"payload-pre-processing-e2e-inf-b-ce04af-ffcb7d45f-zdv8k\" (UID: \"ee89f011-059f-493a-becf-4816c5550dbf\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ce04af-ffcb7d45f-zdv8k" Nov 03 01:23:16.313300 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:23:16.309677 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ce04af-ffcb7d45f-zdv8k" Nov 03 01:23:16.313300 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:23:16.312863 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6tbp\" (UniqueName: \"kubernetes.io/projected/dbd53b13-13c5-449a-b204-1fc94ac8afad-kube-api-access-m6tbp\") pod \"payload-processing-e2e-inf-b-ce04af-6b9b884869-vwgxv\" (UID: \"dbd53b13-13c5-449a-b204-1fc94ac8afad\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ce04af-6b9b884869-vwgxv" Nov 03 01:23:16.577981 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:23:16.577932 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-ce04af-6b9b884869-vwgxv" Nov 03 01:23:16.924470 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:23:16.924438 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ce04af-6b9b884869-vwgxv"] Nov 03 01:23:16.929427 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:23:16.929398 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ce04af-ffcb7d45f-zdv8k"] Nov 03 01:23:17.530001 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:23:17.529961 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ce04af-ffcb7d45f-zdv8k" event={"ID":"ee89f011-059f-493a-becf-4816c5550dbf","Type":"ContainerStarted","Data":"9013783a056208fe3d0b1d218151a61a9299c9219cb05a7de4b8a6f7d8662b42"} Nov 03 01:23:17.530491 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:23:17.530032 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ce04af-ffcb7d45f-zdv8k" Nov 03 01:23:17.530491 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:23:17.530060 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ce04af-ffcb7d45f-zdv8k" event={"ID":"ee89f011-059f-493a-becf-4816c5550dbf","Type":"ContainerStarted","Data":"b78fd32c77270378e02f6a0b19577313472363491f0ecba527e8f2e207336412"} Nov 03 01:23:17.531890 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:23:17.531860 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-ce04af-6b9b884869-vwgxv" event={"ID":"dbd53b13-13c5-449a-b204-1fc94ac8afad","Type":"ContainerStarted","Data":"b6e88418322adb40448b9fb23445d8522c2e6a7a794c295be254f5877f4e5eec"} Nov 03 01:23:17.532034 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:23:17.531916 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-ce04af-6b9b884869-vwgxv" event={"ID":"dbd53b13-13c5-449a-b204-1fc94ac8afad","Type":"ContainerStarted","Data":"916011a87c819ec239be470783aa70ed6620da9ca932023ad1598d66423bbdb8"} Nov 03 01:23:17.532034 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:23:17.531946 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-ce04af-6b9b884869-vwgxv" Nov 03 01:23:17.601688 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:23:17.600011 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ce04af-ffcb7d45f-zdv8k" podStartSLOduration=2.599992575 podStartE2EDuration="2.599992575s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:23:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:23:17.551718414 +0000 UTC m=+2539.722721508" watchObservedRunningTime="2026-11-03 01:23:17.599992575 +0000 UTC m=+2539.770995672" Nov 03 01:23:17.605304 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:23:17.605187 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-ce04af-6b9b884869-vwgxv" podStartSLOduration=2.605173663 podStartE2EDuration="2.605173663s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:23:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:23:17.599352465 +0000 UTC m=+2539.770355562" watchObservedRunningTime="2026-11-03 01:23:17.605173663 +0000 UTC m=+2539.776176762" Nov 03 01:23:17.982625 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:23:17.982545 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6e3e8f1-3867-4381-83c9-64a97998853e" path="/var/lib/kubelet/pods/f6e3e8f1-3867-4381-83c9-64a97998853e/volumes" Nov 03 01:23:27.023991 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:23:27.023950 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7484bf8-6tzhn"] Nov 03 01:23:27.029486 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:23:27.029456 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7484bf8-6tzhn"] Nov 03 01:23:27.982454 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:23:27.982420 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b9a0bcb-5c16-4177-a05d-cd5faf208925" path="/var/lib/kubelet/pods/7b9a0bcb-5c16-4177-a05d-cd5faf208925/volumes" Nov 03 01:23:28.545725 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:23:28.545691 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-ce04af-6b9b884869-vwgxv" Nov 03 01:23:28.547133 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:23:28.545793 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ce04af-ffcb7d45f-zdv8k" Nov 03 01:23:44.713080 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:23:44.712989 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 01:23:59.027757 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:23:59.027718 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8602411c-pnhkh"] Nov 03 01:23:59.032717 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:23:59.032693 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8602411c-pnhkh"] Nov 03 01:23:59.982646 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:23:59.982612 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a113b7da-2cf3-415a-b6cd-6f38a8bd988f" path="/var/lib/kubelet/pods/a113b7da-2cf3-415a-b6cd-6f38a8bd988f/volumes" Nov 03 01:24:03.062465 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:03.062428 4421 scope.go:122] "RemoveContainer" containerID="3715cc1b061d768269a6191d9d5c682fb4a7c9de3c68e8869be46de64f246228" Nov 03 01:24:03.122384 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:03.122358 4421 scope.go:122] "RemoveContainer" containerID="e79421dc26462b4bb3132405330f374608c4a66faac68167ae47e570c3bd40bd" Nov 03 01:24:12.180377 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:12.180338 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af085979-vgdgv"] Nov 03 01:24:12.207770 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:12.207735 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af085979-vgdgv"] Nov 03 01:24:13.994297 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:13.994241 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e89093f3-1b11-4b5c-928b-017480d620b0" path="/var/lib/kubelet/pods/e89093f3-1b11-4b5c-928b-017480d620b0/volumes" Nov 03 01:24:14.088685 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:14.088477 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc950da5-pttqt"] Nov 03 01:24:14.098092 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:14.098063 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc950da5-pttqt" Nov 03 01:24:14.102620 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:14.102598 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:24:14.103850 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:14.103828 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6q4xp\"" Nov 03 01:24:14.143899 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:14.143860 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc950da5-pttqt"] Nov 03 01:24:14.202725 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:14.202683 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34c91155-3ee2-4729-bef4-fbff39a5a300-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cc950da5-pttqt\" (UID: \"34c91155-3ee2-4729-bef4-fbff39a5a300\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc950da5-pttqt" Nov 03 01:24:14.202935 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:14.202909 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34c91155-3ee2-4729-bef4-fbff39a5a300-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cc950da5-pttqt\" (UID: \"34c91155-3ee2-4729-bef4-fbff39a5a300\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc950da5-pttqt" Nov 03 01:24:14.203162 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:14.203149 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cvql\" (UniqueName: \"kubernetes.io/projected/34c91155-3ee2-4729-bef4-fbff39a5a300-kube-api-access-4cvql\") pod \"maas-api-revoke-sub-cc950da5-pttqt\" (UID: \"34c91155-3ee2-4729-bef4-fbff39a5a300\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc950da5-pttqt" Nov 03 01:24:14.352301 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:14.322608 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34c91155-3ee2-4729-bef4-fbff39a5a300-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cc950da5-pttqt\" (UID: \"34c91155-3ee2-4729-bef4-fbff39a5a300\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc950da5-pttqt" Nov 03 01:24:14.352906 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:14.352868 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34c91155-3ee2-4729-bef4-fbff39a5a300-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cc950da5-pttqt\" (UID: \"34c91155-3ee2-4729-bef4-fbff39a5a300\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc950da5-pttqt" Nov 03 01:24:14.353222 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:14.353202 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cvql\" (UniqueName: \"kubernetes.io/projected/34c91155-3ee2-4729-bef4-fbff39a5a300-kube-api-access-4cvql\") pod \"maas-api-revoke-sub-cc950da5-pttqt\" (UID: \"34c91155-3ee2-4729-bef4-fbff39a5a300\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc950da5-pttqt" Nov 03 01:24:14.359433 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:14.339289 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34c91155-3ee2-4729-bef4-fbff39a5a300-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cc950da5-pttqt\" (UID: \"34c91155-3ee2-4729-bef4-fbff39a5a300\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc950da5-pttqt" Nov 03 01:24:14.650350 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:14.644163 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34c91155-3ee2-4729-bef4-fbff39a5a300-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cc950da5-pttqt\" (UID: \"34c91155-3ee2-4729-bef4-fbff39a5a300\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc950da5-pttqt" Nov 03 01:24:14.699301 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:14.675594 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cvql\" (UniqueName: \"kubernetes.io/projected/34c91155-3ee2-4729-bef4-fbff39a5a300-kube-api-access-4cvql\") pod \"maas-api-revoke-sub-cc950da5-pttqt\" (UID: \"34c91155-3ee2-4729-bef4-fbff39a5a300\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc950da5-pttqt" Nov 03 01:24:14.791297 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:14.782192 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc950da5-pttqt" Nov 03 01:24:14.831334 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:24:14.831248 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 01:24:15.193832 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:15.193790 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc950da5-pttqt"] Nov 03 01:24:15.696660 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:15.696625 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc950da5-pttqt" event={"ID":"34c91155-3ee2-4729-bef4-fbff39a5a300","Type":"ContainerStarted","Data":"ffde5e88bb9f2fa88683130a38ad049a0ed67bda615262661c88b6d2ba6ac91b"} Nov 03 01:24:16.820322 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:16.815439 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc950da5-pttqt" event={"ID":"34c91155-3ee2-4729-bef4-fbff39a5a300","Type":"ContainerStarted","Data":"a815d1e2aaf9f68e0b75cd8024b9cb7911738a8edf96b9728cb50bb0b988e611"} Nov 03 01:24:16.960448 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:16.953389 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc950da5-pttqt" podStartSLOduration=2.9533680650000003 podStartE2EDuration="2.953368065s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:24:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:24:16.848704932 +0000 UTC m=+2599.019708029" watchObservedRunningTime="2026-11-03 01:24:16.953368065 +0000 UTC m=+2599.124371163" Nov 03 01:24:18.039095 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:18.039003 4421 generic.go:356] "Generic (PLEG): container finished" podID="34c91155-3ee2-4729-bef4-fbff39a5a300" containerID="a815d1e2aaf9f68e0b75cd8024b9cb7911738a8edf96b9728cb50bb0b988e611" exitCode=0 Nov 03 01:24:18.039707 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:18.039316 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc950da5-pttqt" event={"ID":"34c91155-3ee2-4729-bef4-fbff39a5a300","Type":"ContainerDied","Data":"a815d1e2aaf9f68e0b75cd8024b9cb7911738a8edf96b9728cb50bb0b988e611"} Nov 03 01:24:19.256499 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:19.256461 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc950da5-pttqt" Nov 03 01:24:19.437495 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:19.437396 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34c91155-3ee2-4729-bef4-fbff39a5a300-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34c91155-3ee2-4729-bef4-fbff39a5a300-maas-api-openshift-ca\") pod \"34c91155-3ee2-4729-bef4-fbff39a5a300\" (UID: \"34c91155-3ee2-4729-bef4-fbff39a5a300\") " Nov 03 01:24:19.437665 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:19.437532 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34c91155-3ee2-4729-bef4-fbff39a5a300-kube-api-access-4cvql\" (UniqueName: \"kubernetes.io/projected/34c91155-3ee2-4729-bef4-fbff39a5a300-kube-api-access-4cvql\") pod \"34c91155-3ee2-4729-bef4-fbff39a5a300\" (UID: \"34c91155-3ee2-4729-bef4-fbff39a5a300\") " Nov 03 01:24:19.438240 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:19.437768 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34c91155-3ee2-4729-bef4-fbff39a5a300-maas-api-openshift-ca" pod "34c91155-3ee2-4729-bef4-fbff39a5a300" (UID: "34c91155-3ee2-4729-bef4-fbff39a5a300"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:24:19.438240 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:19.437778 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/34c91155-3ee2-4729-bef4-fbff39a5a300-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34c91155-3ee2-4729-bef4-fbff39a5a300-maas-api-xks-ca\") pod \"34c91155-3ee2-4729-bef4-fbff39a5a300\" (UID: \"34c91155-3ee2-4729-bef4-fbff39a5a300\") " Nov 03 01:24:19.438772 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:19.438753 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34c91155-3ee2-4729-bef4-fbff39a5a300-maas-api-openshift-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:24:19.440674 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:19.440648 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34c91155-3ee2-4729-bef4-fbff39a5a300-maas-api-xks-ca" pod "34c91155-3ee2-4729-bef4-fbff39a5a300" (UID: "34c91155-3ee2-4729-bef4-fbff39a5a300"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:24:19.440784 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:19.440707 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34c91155-3ee2-4729-bef4-fbff39a5a300-kube-api-access-4cvql" pod "34c91155-3ee2-4729-bef4-fbff39a5a300" (UID: "34c91155-3ee2-4729-bef4-fbff39a5a300"). InnerVolumeSpecName "kube-api-access-4cvql". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:24:19.547304 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:19.544464 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4cvql\" (UniqueName: \"kubernetes.io/projected/34c91155-3ee2-4729-bef4-fbff39a5a300-kube-api-access-4cvql\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:24:19.547304 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:19.544604 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34c91155-3ee2-4729-bef4-fbff39a5a300-maas-api-xks-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:24:20.060161 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:20.060124 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc950da5-pttqt" Nov 03 01:24:20.060161 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:20.060141 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc950da5-pttqt" event={"ID":"34c91155-3ee2-4729-bef4-fbff39a5a300","Type":"ContainerDied","Data":"ffde5e88bb9f2fa88683130a38ad049a0ed67bda615262661c88b6d2ba6ac91b"} Nov 03 01:24:20.060970 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:20.060207 4421 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ffde5e88bb9f2fa88683130a38ad049a0ed67bda615262661c88b6d2ba6ac91b" Nov 03 01:24:24.265062 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:24.265008 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51528194-jjdjb"] Nov 03 01:24:24.266539 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:24.265101 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51528194-jjdjb"] Nov 03 01:24:25.901323 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:25.899133 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc9bc07e-rgvk5"] Nov 03 01:24:25.901323 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:25.900554 4421 cpu_manager.go:414] "containerMap: removing container" podUID="34c91155-3ee2-4729-bef4-fbff39a5a300" containerName="revoke-keys" Nov 03 01:24:25.901323 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:25.900579 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34c91155-3ee2-4729-bef4-fbff39a5a300" containerName="revoke-keys" Nov 03 01:24:25.901323 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:25.900814 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="34c91155-3ee2-4729-bef4-fbff39a5a300" containerName="revoke-keys" Nov 03 01:24:25.915587 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:25.915556 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc9bc07e-rgvk5"] Nov 03 01:24:25.915940 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:25.915918 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc9bc07e-rgvk5" Nov 03 01:24:25.928827 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:25.922458 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6q4xp\"" Nov 03 01:24:25.928827 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:25.922506 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:24:25.989575 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:25.987552 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa352e8c-0c7d-47d0-84df-8c1ee8c51503" path="/var/lib/kubelet/pods/fa352e8c-0c7d-47d0-84df-8c1ee8c51503/volumes" Nov 03 01:24:26.027581 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:26.027531 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f866c13c-8629-4da7-b7af-6af1e9932157-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fc9bc07e-rgvk5\" (UID: \"f866c13c-8629-4da7-b7af-6af1e9932157\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc9bc07e-rgvk5" Nov 03 01:24:26.027581 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:26.027582 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6dvk\" (UniqueName: \"kubernetes.io/projected/f866c13c-8629-4da7-b7af-6af1e9932157-kube-api-access-q6dvk\") pod \"maas-api-revoke-sub-fc9bc07e-rgvk5\" (UID: \"f866c13c-8629-4da7-b7af-6af1e9932157\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc9bc07e-rgvk5" Nov 03 01:24:26.030758 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:26.027803 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f866c13c-8629-4da7-b7af-6af1e9932157-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fc9bc07e-rgvk5\" (UID: \"f866c13c-8629-4da7-b7af-6af1e9932157\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc9bc07e-rgvk5" Nov 03 01:24:26.129594 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:26.129552 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f866c13c-8629-4da7-b7af-6af1e9932157-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fc9bc07e-rgvk5\" (UID: \"f866c13c-8629-4da7-b7af-6af1e9932157\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc9bc07e-rgvk5" Nov 03 01:24:26.129802 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:26.129734 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f866c13c-8629-4da7-b7af-6af1e9932157-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fc9bc07e-rgvk5\" (UID: \"f866c13c-8629-4da7-b7af-6af1e9932157\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc9bc07e-rgvk5" Nov 03 01:24:26.129802 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:26.129756 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6dvk\" (UniqueName: \"kubernetes.io/projected/f866c13c-8629-4da7-b7af-6af1e9932157-kube-api-access-q6dvk\") pod \"maas-api-revoke-sub-fc9bc07e-rgvk5\" (UID: \"f866c13c-8629-4da7-b7af-6af1e9932157\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc9bc07e-rgvk5" Nov 03 01:24:26.130579 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:26.130554 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f866c13c-8629-4da7-b7af-6af1e9932157-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fc9bc07e-rgvk5\" (UID: \"f866c13c-8629-4da7-b7af-6af1e9932157\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc9bc07e-rgvk5" Nov 03 01:24:26.277362 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:26.277320 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f866c13c-8629-4da7-b7af-6af1e9932157-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fc9bc07e-rgvk5\" (UID: \"f866c13c-8629-4da7-b7af-6af1e9932157\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc9bc07e-rgvk5" Nov 03 01:24:26.277565 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:26.277544 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6dvk\" (UniqueName: \"kubernetes.io/projected/f866c13c-8629-4da7-b7af-6af1e9932157-kube-api-access-q6dvk\") pod \"maas-api-revoke-sub-fc9bc07e-rgvk5\" (UID: \"f866c13c-8629-4da7-b7af-6af1e9932157\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc9bc07e-rgvk5" Nov 03 01:24:26.284592 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:26.284563 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc9bc07e-rgvk5" Nov 03 01:24:30.077586 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:29.318239 4421 patch_prober.go:29] interesting pod/e2e-inf-b-ce04af-openshift-default-7dccd45f47-fcw4z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 01:24:30.077586 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:29.318384 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-fcw4z" podUID="37f37c99-0b3d-45d2-ba18-a4ec39043f67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 01:24:30.401056 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:30.240388 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc9bc07e-rgvk5"] Nov 03 01:24:31.240557 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:31.240520 4421 patch_prober.go:29] interesting pod/prometheus-operator-admission-webhook-6489d8bf89-2496n container/prometheus-operator-admission-webhook namespace/openshift-monitoring: Liveness probe status=failure output="Get \"https://10.134.0.10:8443/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 01:24:31.240995 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:31.240579 4421 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-2496n" podUID="e906bfeb-04c0-4465-9ec6-6c18aad596ae" containerName="prometheus-operator-admission-webhook" probeResult="failure" output="Get \"https://10.134.0.10:8443/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 01:24:31.312873 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:31.248897 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc9bc07e-rgvk5" event={"ID":"f866c13c-8629-4da7-b7af-6af1e9932157","Type":"ContainerStarted","Data":"7781ba8494c0e2b57247a149f2b6bd42cf461557c27e551d20446806b32ba756"} Nov 03 01:24:31.597475 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:31.597438 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-5cb6nvdxn"] Nov 03 01:24:33.267577 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:33.267537 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc9bc07e-rgvk5" event={"ID":"f866c13c-8629-4da7-b7af-6af1e9932157","Type":"ContainerStarted","Data":"0b344147dabf5a9c489eca52ff337bf3730d7d1ccc55b93ca73f00185e244c4d"} Nov 03 01:24:33.289075 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:33.288979 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc9bc07e-rgvk5" podStartSLOduration=8.288965205 podStartE2EDuration="8.288965205s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:24:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:24:33.288523086 +0000 UTC m=+2615.459526184" watchObservedRunningTime="2026-11-03 01:24:33.288965205 +0000 UTC m=+2615.459968300" Nov 03 01:24:36.297247 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:36.297203 4421 generic.go:356] "Generic (PLEG): container finished" podID="f866c13c-8629-4da7-b7af-6af1e9932157" containerID="0b344147dabf5a9c489eca52ff337bf3730d7d1ccc55b93ca73f00185e244c4d" exitCode=0 Nov 03 01:24:36.301486 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:36.297368 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc9bc07e-rgvk5" event={"ID":"f866c13c-8629-4da7-b7af-6af1e9932157","Type":"ContainerDied","Data":"0b344147dabf5a9c489eca52ff337bf3730d7d1ccc55b93ca73f00185e244c4d"} Nov 03 01:24:37.217060 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:37.213421 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-5cb6nvdxn" podUID="cd2d7685-32b0-4bb5-bc90-bbe0d0844c26" containerName="payload-processing" containerID="cri-o://1941358fef6cceea1e8f8bd50103e9b0fdbfb0b3d4d740814e29807968dcca88" gracePeriod=25 Nov 03 01:24:37.294387 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:37.289325 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12bf75e6-x7t6r"] Nov 03 01:24:37.434318 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:37.433934 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12bf75e6-x7t6r"] Nov 03 01:24:37.496903 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:37.493389 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/simulator-w3-83755a-kserve-77d6664559-psqtg" podUID="b0a4c234-af58-4da0-a1a4-e1f167953f6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Nov 03 01:24:37.707645 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:37.707594 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/distinct-w3-83755a-kserve-86bdbbdd85-2cnnl" podUID="f4ae9624-d127-4a55-837d-7c9ff0267b37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 01:24:37.987806 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:37.987765 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26afee51-c188-4d90-8989-32150bffde59" path="/var/lib/kubelet/pods/26afee51-c188-4d90-8989-32150bffde59/volumes" Nov 03 01:24:38.476764 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:38.476728 4421 generic.go:356] "Generic (PLEG): container finished" podID="cd2d7685-32b0-4bb5-bc90-bbe0d0844c26" containerID="1941358fef6cceea1e8f8bd50103e9b0fdbfb0b3d4d740814e29807968dcca88" exitCode=0 Nov 03 01:24:38.477204 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:38.476815 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-5cb6nvdxn" event={"ID":"cd2d7685-32b0-4bb5-bc90-bbe0d0844c26","Type":"ContainerDied","Data":"1941358fef6cceea1e8f8bd50103e9b0fdbfb0b3d4d740814e29807968dcca88"} Nov 03 01:24:39.086680 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:39.086646 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-5cb6nvdxn" Nov 03 01:24:39.093432 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:39.092646 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc9bc07e-rgvk5" Nov 03 01:24:39.135757 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:39.130673 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f866c13c-8629-4da7-b7af-6af1e9932157-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f866c13c-8629-4da7-b7af-6af1e9932157-maas-api-xks-ca\") pod \"f866c13c-8629-4da7-b7af-6af1e9932157\" (UID: \"f866c13c-8629-4da7-b7af-6af1e9932157\") " Nov 03 01:24:39.135757 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:39.130783 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd2d7685-32b0-4bb5-bc90-bbe0d0844c26-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cd2d7685-32b0-4bb5-bc90-bbe0d0844c26-routing-overlay-volume\") pod \"cd2d7685-32b0-4bb5-bc90-bbe0d0844c26\" (UID: \"cd2d7685-32b0-4bb5-bc90-bbe0d0844c26\") " Nov 03 01:24:39.135757 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:39.130895 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd2d7685-32b0-4bb5-bc90-bbe0d0844c26-kube-api-access-pttdc\" (UniqueName: \"kubernetes.io/projected/cd2d7685-32b0-4bb5-bc90-bbe0d0844c26-kube-api-access-pttdc\") pod \"cd2d7685-32b0-4bb5-bc90-bbe0d0844c26\" (UID: \"cd2d7685-32b0-4bb5-bc90-bbe0d0844c26\") " Nov 03 01:24:39.135757 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:39.130963 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f866c13c-8629-4da7-b7af-6af1e9932157-kube-api-access-q6dvk\" (UniqueName: \"kubernetes.io/projected/f866c13c-8629-4da7-b7af-6af1e9932157-kube-api-access-q6dvk\") pod \"f866c13c-8629-4da7-b7af-6af1e9932157\" (UID: \"f866c13c-8629-4da7-b7af-6af1e9932157\") " Nov 03 01:24:39.135757 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:39.131033 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f866c13c-8629-4da7-b7af-6af1e9932157-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f866c13c-8629-4da7-b7af-6af1e9932157-maas-api-openshift-ca\") pod \"f866c13c-8629-4da7-b7af-6af1e9932157\" (UID: \"f866c13c-8629-4da7-b7af-6af1e9932157\") " Nov 03 01:24:39.135757 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:39.131207 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd2d7685-32b0-4bb5-bc90-bbe0d0844c26-provider-credentials\" (UniqueName: \"kubernetes.io/projected/cd2d7685-32b0-4bb5-bc90-bbe0d0844c26-provider-credentials\") pod \"cd2d7685-32b0-4bb5-bc90-bbe0d0844c26\" (UID: \"cd2d7685-32b0-4bb5-bc90-bbe0d0844c26\") " Nov 03 01:24:39.135757 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:39.131654 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd2d7685-32b0-4bb5-bc90-bbe0d0844c26-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd2d7685-32b0-4bb5-bc90-bbe0d0844c26-plugins-config-volume\") pod \"cd2d7685-32b0-4bb5-bc90-bbe0d0844c26\" (UID: \"cd2d7685-32b0-4bb5-bc90-bbe0d0844c26\") " Nov 03 01:24:39.157506 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:39.136350 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f866c13c-8629-4da7-b7af-6af1e9932157-maas-api-openshift-ca" pod "f866c13c-8629-4da7-b7af-6af1e9932157" (UID: "f866c13c-8629-4da7-b7af-6af1e9932157"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:24:39.172255 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:39.172204 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd2d7685-32b0-4bb5-bc90-bbe0d0844c26-routing-overlay-volume" pod "cd2d7685-32b0-4bb5-bc90-bbe0d0844c26" (UID: "cd2d7685-32b0-4bb5-bc90-bbe0d0844c26"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:24:39.172255 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:39.172234 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd2d7685-32b0-4bb5-bc90-bbe0d0844c26-plugins-config-volume" pod "cd2d7685-32b0-4bb5-bc90-bbe0d0844c26" (UID: "cd2d7685-32b0-4bb5-bc90-bbe0d0844c26"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:24:39.235681 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:39.235643 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f866c13c-8629-4da7-b7af-6af1e9932157-maas-api-openshift-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:24:39.240707 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:39.235683 4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd2d7685-32b0-4bb5-bc90-bbe0d0844c26-plugins-config-volume\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:24:39.240707 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:39.235703 4421 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cd2d7685-32b0-4bb5-bc90-bbe0d0844c26-routing-overlay-volume\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:24:39.240873 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:39.240733 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f866c13c-8629-4da7-b7af-6af1e9932157-maas-api-xks-ca" pod "f866c13c-8629-4da7-b7af-6af1e9932157" (UID: "f866c13c-8629-4da7-b7af-6af1e9932157"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:24:39.240873 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:39.240765 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd2d7685-32b0-4bb5-bc90-bbe0d0844c26-kube-api-access-pttdc" pod "cd2d7685-32b0-4bb5-bc90-bbe0d0844c26" (UID: "cd2d7685-32b0-4bb5-bc90-bbe0d0844c26"). InnerVolumeSpecName "kube-api-access-pttdc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:24:39.240873 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:39.240802 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f866c13c-8629-4da7-b7af-6af1e9932157-kube-api-access-q6dvk" pod "f866c13c-8629-4da7-b7af-6af1e9932157" (UID: "f866c13c-8629-4da7-b7af-6af1e9932157"). InnerVolumeSpecName "kube-api-access-q6dvk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:24:39.250229 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:39.249539 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd2d7685-32b0-4bb5-bc90-bbe0d0844c26-provider-credentials" pod "cd2d7685-32b0-4bb5-bc90-bbe0d0844c26" (UID: "cd2d7685-32b0-4bb5-bc90-bbe0d0844c26"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:24:39.337692 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:39.337652 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pttdc\" (UniqueName: \"kubernetes.io/projected/cd2d7685-32b0-4bb5-bc90-bbe0d0844c26-kube-api-access-pttdc\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:24:39.337692 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:39.337687 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6dvk\" (UniqueName: \"kubernetes.io/projected/f866c13c-8629-4da7-b7af-6af1e9932157-kube-api-access-q6dvk\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:24:39.337692 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:39.337697 4421 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cd2d7685-32b0-4bb5-bc90-bbe0d0844c26-provider-credentials\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:24:39.342633 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:39.337707 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f866c13c-8629-4da7-b7af-6af1e9932157-maas-api-xks-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:24:39.492375 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:39.492267 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-5cb6nvdxn" Nov 03 01:24:39.492375 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:39.492322 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-5cb6nvdxn" event={"ID":"cd2d7685-32b0-4bb5-bc90-bbe0d0844c26","Type":"ContainerDied","Data":"38ebb60dbf69355b058754d1b53710c34176a2f11221971e1b44f7fc9ea47e1d"} Nov 03 01:24:39.492916 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:39.492409 4421 scope.go:122] "RemoveContainer" containerID="1941358fef6cceea1e8f8bd50103e9b0fdbfb0b3d4d740814e29807968dcca88" Nov 03 01:24:39.494854 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:39.494830 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc9bc07e-rgvk5" event={"ID":"f866c13c-8629-4da7-b7af-6af1e9932157","Type":"ContainerDied","Data":"7781ba8494c0e2b57247a149f2b6bd42cf461557c27e551d20446806b32ba756"} Nov 03 01:24:39.494991 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:39.494873 4421 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7781ba8494c0e2b57247a149f2b6bd42cf461557c27e551d20446806b32ba756" Nov 03 01:24:39.494991 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:39.494918 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc9bc07e-rgvk5" Nov 03 01:24:39.540431 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:39.538303 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-5cb6nvdxn"] Nov 03 01:24:39.544106 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:39.544078 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-5cb6nvdxn"] Nov 03 01:24:39.998448 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:39.998403 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd2d7685-32b0-4bb5-bc90-bbe0d0844c26" path="/var/lib/kubelet/pods/cd2d7685-32b0-4bb5-bc90-bbe0d0844c26/volumes" Nov 03 01:24:40.459496 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:40.362374 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-579b4mp4l"] Nov 03 01:24:40.459496 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:40.367425 4421 cpu_manager.go:414] "containerMap: removing container" podUID="cd2d7685-32b0-4bb5-bc90-bbe0d0844c26" containerName="payload-processing" Nov 03 01:24:40.459496 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:40.367455 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd2d7685-32b0-4bb5-bc90-bbe0d0844c26" containerName="payload-processing" Nov 03 01:24:40.459496 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:40.367558 4421 cpu_manager.go:414] "containerMap: removing container" podUID="f866c13c-8629-4da7-b7af-6af1e9932157" containerName="revoke-keys" Nov 03 01:24:40.459496 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:40.367567 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f866c13c-8629-4da7-b7af-6af1e9932157" containerName="revoke-keys" Nov 03 01:24:40.459496 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:40.368407 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd2d7685-32b0-4bb5-bc90-bbe0d0844c26" containerName="payload-processing" Nov 03 01:24:40.459496 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:40.368449 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="f866c13c-8629-4da7-b7af-6af1e9932157" containerName="revoke-keys" Nov 03 01:24:40.578150 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:40.462517 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-579b4mp4l"] Nov 03 01:24:40.578150 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:40.464266 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-579b4mp4l" Nov 03 01:24:40.907510 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:40.611846 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ea9ef1e6-5453-414c-a759-d70493addc5c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d25759-579b4mp4l\" (UID: \"ea9ef1e6-5453-414c-a759-d70493addc5c\") " pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-579b4mp4l" Nov 03 01:24:40.907510 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:40.612098 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea9ef1e6-5453-414c-a759-d70493addc5c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d25759-579b4mp4l\" (UID: \"ea9ef1e6-5453-414c-a759-d70493addc5c\") " pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-579b4mp4l" Nov 03 01:24:40.907510 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:40.612250 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxl7t\" (UniqueName: \"kubernetes.io/projected/ea9ef1e6-5453-414c-a759-d70493addc5c-kube-api-access-pxl7t\") pod \"payload-processing-external-model-e2e-extmod-a-d25759-579b4mp4l\" (UID: \"ea9ef1e6-5453-414c-a759-d70493addc5c\") " pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-579b4mp4l" Nov 03 01:24:40.907510 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:40.612333 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ea9ef1e6-5453-414c-a759-d70493addc5c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-d25759-579b4mp4l\" (UID: \"ea9ef1e6-5453-414c-a759-d70493addc5c\") " pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-579b4mp4l" Nov 03 01:24:40.907510 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:40.662403 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-d25759\"/\"payload-processing-external-model-plugins-e2e-extmod-a-d25759\"" Nov 03 01:24:40.907510 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:40.662639 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-d25759\"/\"agc-cred-22a51f875964\"" Nov 03 01:24:40.907510 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:40.662925 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-d25759\"/\"kube-root-ca.crt\"" Nov 03 01:24:40.907510 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:40.662981 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-d25759\"/\"routing-overlay\"" Nov 03 01:24:40.907510 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:40.663058 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-d25759\"/\"payload-processing-external-model-e2e--4d491765-dockercfg-56tgf\"" Nov 03 01:24:40.907510 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:40.663145 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-d25759\"/\"openshift-service-ca.crt\"" Nov 03 01:24:40.907510 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:40.713914 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea9ef1e6-5453-414c-a759-d70493addc5c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d25759-579b4mp4l\" (UID: \"ea9ef1e6-5453-414c-a759-d70493addc5c\") " pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-579b4mp4l" Nov 03 01:24:40.907510 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:40.714049 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxl7t\" (UniqueName: \"kubernetes.io/projected/ea9ef1e6-5453-414c-a759-d70493addc5c-kube-api-access-pxl7t\") pod \"payload-processing-external-model-e2e-extmod-a-d25759-579b4mp4l\" (UID: \"ea9ef1e6-5453-414c-a759-d70493addc5c\") " pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-579b4mp4l" Nov 03 01:24:40.907510 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:40.714100 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ea9ef1e6-5453-414c-a759-d70493addc5c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-d25759-579b4mp4l\" (UID: \"ea9ef1e6-5453-414c-a759-d70493addc5c\") " pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-579b4mp4l" Nov 03 01:24:40.907510 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:40.714237 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ea9ef1e6-5453-414c-a759-d70493addc5c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d25759-579b4mp4l\" (UID: \"ea9ef1e6-5453-414c-a759-d70493addc5c\") " pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-579b4mp4l" Nov 03 01:24:40.907510 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:40.846894 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ea9ef1e6-5453-414c-a759-d70493addc5c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-d25759-579b4mp4l\" (UID: \"ea9ef1e6-5453-414c-a759-d70493addc5c\") " pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-579b4mp4l" Nov 03 01:24:40.907510 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:40.846924 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ea9ef1e6-5453-414c-a759-d70493addc5c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d25759-579b4mp4l\" (UID: \"ea9ef1e6-5453-414c-a759-d70493addc5c\") " pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-579b4mp4l" Nov 03 01:24:40.907510 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:40.846965 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea9ef1e6-5453-414c-a759-d70493addc5c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d25759-579b4mp4l\" (UID: \"ea9ef1e6-5453-414c-a759-d70493addc5c\") " pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-579b4mp4l" Nov 03 01:24:40.907510 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:40.846980 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxl7t\" (UniqueName: \"kubernetes.io/projected/ea9ef1e6-5453-414c-a759-d70493addc5c-kube-api-access-pxl7t\") pod \"payload-processing-external-model-e2e-extmod-a-d25759-579b4mp4l\" (UID: \"ea9ef1e6-5453-414c-a759-d70493addc5c\") " pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-579b4mp4l" Nov 03 01:24:41.018299 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:41.014824 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-579b4mp4l" Nov 03 01:24:41.691010 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:41.690973 4421 generic.go:356] "Generic (PLEG): container finished" podID="37f37c99-0b3d-45d2-ba18-a4ec39043f67" containerID="cb413e1300f544506216d31332bc15a5a0787f3e218711849e2e819ef25dba88" exitCode=0 Nov 03 01:24:41.692343 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:41.691100 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-fcw4z" event={"ID":"37f37c99-0b3d-45d2-ba18-a4ec39043f67","Type":"ContainerDied","Data":"cb413e1300f544506216d31332bc15a5a0787f3e218711849e2e819ef25dba88"} Nov 03 01:24:41.692343 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:41.691611 4421 scope.go:122] "RemoveContainer" containerID="cb413e1300f544506216d31332bc15a5a0787f3e218711849e2e819ef25dba88" Nov 03 01:24:41.693813 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:41.693779 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:24:41.693923 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:41.693879 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:24:41.693999 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:41.693927 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:24:41.855469 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:41.855402 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-579b4mp4l"] Nov 03 01:24:41.861009 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:24:41.860972 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea9ef1e6_5453_414c_a759_d70493addc5c.slice/crio-fd1683498aa7bfabd844a10a91df564ff1073ebba844dc4dac9ce792f324fd93 WatchSource:0}: Error finding container fd1683498aa7bfabd844a10a91df564ff1073ebba844dc4dac9ce792f324fd93: Status 404 returned error can't find the container with id fd1683498aa7bfabd844a10a91df564ff1073ebba844dc4dac9ce792f324fd93 Nov 03 01:24:42.513309 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:42.511999 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-579b4mp4l"] Nov 03 01:24:42.702174 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:42.702130 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-579b4mp4l" event={"ID":"ea9ef1e6-5453-414c-a759-d70493addc5c","Type":"ContainerStarted","Data":"40433d0cb96cd0730ccac952fdef6a5f064f6df85e85c509b34691be479d8a57"} Nov 03 01:24:42.702735 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:42.702203 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-579b4mp4l" event={"ID":"ea9ef1e6-5453-414c-a759-d70493addc5c","Type":"ContainerStarted","Data":"fd1683498aa7bfabd844a10a91df564ff1073ebba844dc4dac9ce792f324fd93"} Nov 03 01:24:42.702735 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:42.702263 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-579b4mp4l" Nov 03 01:24:42.702735 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:42.702446 4421 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-579b4mp4l" secret="" err="secret \"payload-processing-external-model-e2e--4d491765-dockercfg-56tgf\" not found" Nov 03 01:24:42.704588 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:42.704568 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-fcw4z" event={"ID":"37f37c99-0b3d-45d2-ba18-a4ec39043f67","Type":"ContainerStarted","Data":"788021faebbc3370b311a69120db115daab85538b593570e04d4d813cc0ceb6d"} Nov 03 01:24:42.723386 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:42.723335 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-579b4mp4l" podStartSLOduration=2.72331684 podStartE2EDuration="2.72331684s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:24:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:24:42.71764438 +0000 UTC m=+2624.888647499" watchObservedRunningTime="2026-11-03 01:24:42.72331684 +0000 UTC m=+2624.894319917" Nov 03 01:24:42.823347 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:24:42.819756 4421 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-plugins-e2e-extmod-a-d25759: configmap "payload-processing-external-model-plugins-e2e-extmod-a-d25759" not found Nov 03 01:24:42.823347 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:24:42.819859 4421 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ea9ef1e6-5453-414c-a759-d70493addc5c-plugins-config-volume podName:ea9ef1e6-5453-414c-a759-d70493addc5c nodeName:}" failed. No retries permitted until 2026-11-03 01:24:43.31983425 +0000 UTC m=+2625.490837337 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/ea9ef1e6-5453-414c-a759-d70493addc5c-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-d25759-579b4mp4l" (UID: "ea9ef1e6-5453-414c-a759-d70493addc5c") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-d25759" not found Nov 03 01:24:42.823347 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:24:42.821560 4421 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-d25759/agc-cred-22a51f875964: secret "agc-cred-22a51f875964" not found Nov 03 01:24:42.823347 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:24:42.821579 4421 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-579b4mp4l: secret "agc-cred-22a51f875964" not found Nov 03 01:24:42.823347 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:24:42.821625 4421 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ea9ef1e6-5453-414c-a759-d70493addc5c-provider-credentials podName:ea9ef1e6-5453-414c-a759-d70493addc5c nodeName:}" failed. No retries permitted until 2026-11-03 01:24:43.321609787 +0000 UTC m=+2625.492612876 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/ea9ef1e6-5453-414c-a759-d70493addc5c-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-d25759-579b4mp4l" (UID: "ea9ef1e6-5453-414c-a759-d70493addc5c") : secret "agc-cred-22a51f875964" not found Nov 03 01:24:43.110348 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:43.108488 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-fcw4z" Nov 03 01:24:43.127297 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:43.124333 4421 patch_prober.go:29] interesting pod/e2e-inf-b-ce04af-openshift-default-7dccd45f47-fcw4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.48:15021/healthz/ready\": dial tcp 10.134.1.48:15021: connect: connection refused" start-of-body= Nov 03 01:24:43.127297 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:43.124393 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-fcw4z" podUID="37f37c99-0b3d-45d2-ba18-a4ec39043f67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.48:15021/healthz/ready\": dial tcp 10.134.1.48:15021: connect: connection refused" Nov 03 01:24:43.261639 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:43.261588 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-fcw4z" Nov 03 01:24:43.366513 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:24:43.366422 4421 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-plugins-e2e-extmod-a-d25759: configmap "payload-processing-external-model-plugins-e2e-extmod-a-d25759" not found Nov 03 01:24:43.366513 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:24:43.366502 4421 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ea9ef1e6-5453-414c-a759-d70493addc5c-plugins-config-volume podName:ea9ef1e6-5453-414c-a759-d70493addc5c nodeName:}" failed. No retries permitted until 2026-11-03 01:24:44.366482198 +0000 UTC m=+2626.537485275 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/ea9ef1e6-5453-414c-a759-d70493addc5c-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-d25759-579b4mp4l" (UID: "ea9ef1e6-5453-414c-a759-d70493addc5c") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-d25759" not found Nov 03 01:24:43.366719 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:24:43.366569 4421 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-d25759/agc-cred-22a51f875964: secret "agc-cred-22a51f875964" not found Nov 03 01:24:43.366719 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:24:43.366588 4421 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-579b4mp4l: secret "agc-cred-22a51f875964" not found Nov 03 01:24:43.366719 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:24:43.366647 4421 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ea9ef1e6-5453-414c-a759-d70493addc5c-provider-credentials podName:ea9ef1e6-5453-414c-a759-d70493addc5c nodeName:}" failed. No retries permitted until 2026-11-03 01:24:44.366635809 +0000 UTC m=+2626.537638897 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/ea9ef1e6-5453-414c-a759-d70493addc5c-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-d25759-579b4mp4l" (UID: "ea9ef1e6-5453-414c-a759-d70493addc5c") : secret "agc-cred-22a51f875964" not found Nov 03 01:24:44.098633 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:44.098595 4421 patch_prober.go:29] interesting pod/e2e-inf-b-ce04af-openshift-default-7dccd45f47-fcw4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.48:15021/healthz/ready\": dial tcp 10.134.1.48:15021: connect: connection refused" start-of-body= Nov 03 01:24:44.099046 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:44.098650 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-fcw4z" podUID="37f37c99-0b3d-45d2-ba18-a4ec39043f67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.48:15021/healthz/ready\": dial tcp 10.134.1.48:15021: connect: connection refused" Nov 03 01:24:44.449828 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:24:44.449723 4421 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-d25759/agc-cred-22a51f875964: secret "agc-cred-22a51f875964" not found Nov 03 01:24:44.449828 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:24:44.449761 4421 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-579b4mp4l: secret "agc-cred-22a51f875964" not found Nov 03 01:24:44.449828 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:24:44.449831 4421 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ea9ef1e6-5453-414c-a759-d70493addc5c-provider-credentials podName:ea9ef1e6-5453-414c-a759-d70493addc5c nodeName:}" failed. No retries permitted until 2026-11-03 01:24:46.449812421 +0000 UTC m=+2628.620815516 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/ea9ef1e6-5453-414c-a759-d70493addc5c-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-d25759-579b4mp4l" (UID: "ea9ef1e6-5453-414c-a759-d70493addc5c") : secret "agc-cred-22a51f875964" not found Nov 03 01:24:44.451846 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:24:44.449853 4421 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-plugins-e2e-extmod-a-d25759: configmap "payload-processing-external-model-plugins-e2e-extmod-a-d25759" not found Nov 03 01:24:44.451846 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:24:44.449894 4421 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ea9ef1e6-5453-414c-a759-d70493addc5c-plugins-config-volume podName:ea9ef1e6-5453-414c-a759-d70493addc5c nodeName:}" failed. No retries permitted until 2026-11-03 01:24:46.449884104 +0000 UTC m=+2628.620887199 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/ea9ef1e6-5453-414c-a759-d70493addc5c-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-d25759-579b4mp4l" (UID: "ea9ef1e6-5453-414c-a759-d70493addc5c") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-d25759" not found Nov 03 01:24:44.742646 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:24:44.742563 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 01:24:45.098777 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:45.098741 4421 patch_prober.go:29] interesting pod/e2e-inf-b-ce04af-openshift-default-7dccd45f47-fcw4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.48:15021/healthz/ready\": dial tcp 10.134.1.48:15021: connect: connection refused" start-of-body= Nov 03 01:24:45.100025 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:45.098808 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-fcw4z" podUID="37f37c99-0b3d-45d2-ba18-a4ec39043f67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.48:15021/healthz/ready\": dial tcp 10.134.1.48:15021: connect: connection refused" Nov 03 01:24:46.098020 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:46.097985 4421 patch_prober.go:29] interesting pod/e2e-inf-b-ce04af-openshift-default-7dccd45f47-fcw4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.48:15021/healthz/ready\": dial tcp 10.134.1.48:15021: connect: connection refused" start-of-body= Nov 03 01:24:46.098978 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:46.098048 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-fcw4z" podUID="37f37c99-0b3d-45d2-ba18-a4ec39043f67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.48:15021/healthz/ready\": dial tcp 10.134.1.48:15021: connect: connection refused" Nov 03 01:24:46.523350 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:24:46.520021 4421 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-d25759/agc-cred-22a51f875964: secret "agc-cred-22a51f875964" not found Nov 03 01:24:46.523350 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:24:46.520056 4421 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-579b4mp4l: secret "agc-cred-22a51f875964" not found Nov 03 01:24:46.523350 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:24:46.520117 4421 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ea9ef1e6-5453-414c-a759-d70493addc5c-provider-credentials podName:ea9ef1e6-5453-414c-a759-d70493addc5c nodeName:}" failed. No retries permitted until 2026-11-03 01:24:50.520098671 +0000 UTC m=+2632.691101767 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/ea9ef1e6-5453-414c-a759-d70493addc5c-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-d25759-579b4mp4l" (UID: "ea9ef1e6-5453-414c-a759-d70493addc5c") : secret "agc-cred-22a51f875964" not found Nov 03 01:24:46.535317 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:24:46.534850 4421 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-plugins-e2e-extmod-a-d25759: configmap "payload-processing-external-model-plugins-e2e-extmod-a-d25759" not found Nov 03 01:24:46.535317 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:24:46.534937 4421 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ea9ef1e6-5453-414c-a759-d70493addc5c-plugins-config-volume podName:ea9ef1e6-5453-414c-a759-d70493addc5c nodeName:}" failed. No retries permitted until 2026-11-03 01:24:50.534914283 +0000 UTC m=+2632.705917374 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/ea9ef1e6-5453-414c-a759-d70493addc5c-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-d25759-579b4mp4l" (UID: "ea9ef1e6-5453-414c-a759-d70493addc5c") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-d25759" not found Nov 03 01:24:46.781380 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:46.781301 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-gwl6k"] Nov 03 01:24:46.781910 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:46.781576 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-gwl6k" podUID="e6c36e3a-89e3-4f54-8266-c7dcfc0ca19b" containerName="istio-proxy" containerID="cri-o://25c5f74ca2414a88339beef2035ed59bc07f8a1ba6fdb15614b0c100590ec788" gracePeriod=30 Nov 03 01:24:47.099242 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:47.099144 4421 patch_prober.go:29] interesting pod/e2e-inf-b-ce04af-openshift-default-7dccd45f47-fcw4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.48:15021/healthz/ready\": dial tcp 10.134.1.48:15021: connect: connection refused" start-of-body= Nov 03 01:24:47.103562 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:47.099763 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-fcw4z" podUID="37f37c99-0b3d-45d2-ba18-a4ec39043f67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.48:15021/healthz/ready\": dial tcp 10.134.1.48:15021: connect: connection refused" Nov 03 01:24:48.098301 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:48.098243 4421 patch_prober.go:29] interesting pod/e2e-inf-b-ce04af-openshift-default-7dccd45f47-fcw4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.48:15021/healthz/ready\": dial tcp 10.134.1.48:15021: connect: connection refused" start-of-body= Nov 03 01:24:48.099431 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:48.098332 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-fcw4z" podUID="37f37c99-0b3d-45d2-ba18-a4ec39043f67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.48:15021/healthz/ready\": dial tcp 10.134.1.48:15021: connect: connection refused" Nov 03 01:24:48.780698 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:48.780650 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-579b4mp4l" podUID="ea9ef1e6-5453-414c-a759-d70493addc5c" containerName="payload-processing" containerID="cri-o://40433d0cb96cd0730ccac952fdef6a5f064f6df85e85c509b34691be479d8a57" gracePeriod=25 Nov 03 01:24:49.098483 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:49.098362 4421 patch_prober.go:29] interesting pod/e2e-inf-b-ce04af-openshift-default-7dccd45f47-fcw4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.48:15021/healthz/ready\": dial tcp 10.134.1.48:15021: connect: connection refused" start-of-body= Nov 03 01:24:49.098761 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:49.098720 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-fcw4z" podUID="37f37c99-0b3d-45d2-ba18-a4ec39043f67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.48:15021/healthz/ready\": dial tcp 10.134.1.48:15021: connect: connection refused" Nov 03 01:24:49.283084 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:49.283053 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-579b4mp4l" Nov 03 01:24:49.481430 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:49.481345 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea9ef1e6-5453-414c-a759-d70493addc5c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea9ef1e6-5453-414c-a759-d70493addc5c-plugins-config-volume\") pod \"ea9ef1e6-5453-414c-a759-d70493addc5c\" (UID: \"ea9ef1e6-5453-414c-a759-d70493addc5c\") " Nov 03 01:24:49.481586 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:49.481524 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea9ef1e6-5453-414c-a759-d70493addc5c-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ea9ef1e6-5453-414c-a759-d70493addc5c-routing-overlay-volume\") pod \"ea9ef1e6-5453-414c-a759-d70493addc5c\" (UID: \"ea9ef1e6-5453-414c-a759-d70493addc5c\") " Nov 03 01:24:49.482634 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:49.481612 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea9ef1e6-5453-414c-a759-d70493addc5c-kube-api-access-pxl7t\" (UniqueName: \"kubernetes.io/projected/ea9ef1e6-5453-414c-a759-d70493addc5c-kube-api-access-pxl7t\") pod \"ea9ef1e6-5453-414c-a759-d70493addc5c\" (UID: \"ea9ef1e6-5453-414c-a759-d70493addc5c\") " Nov 03 01:24:49.482634 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:49.481775 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea9ef1e6-5453-414c-a759-d70493addc5c-plugins-config-volume" pod "ea9ef1e6-5453-414c-a759-d70493addc5c" (UID: "ea9ef1e6-5453-414c-a759-d70493addc5c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:24:49.482634 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:49.481803 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea9ef1e6-5453-414c-a759-d70493addc5c-provider-credentials\" (UniqueName: \"kubernetes.io/projected/ea9ef1e6-5453-414c-a759-d70493addc5c-provider-credentials\") pod \"ea9ef1e6-5453-414c-a759-d70493addc5c\" (UID: \"ea9ef1e6-5453-414c-a759-d70493addc5c\") " Nov 03 01:24:49.482634 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:49.481861 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea9ef1e6-5453-414c-a759-d70493addc5c-routing-overlay-volume" pod "ea9ef1e6-5453-414c-a759-d70493addc5c" (UID: "ea9ef1e6-5453-414c-a759-d70493addc5c"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:24:49.482634 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:49.482348 4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea9ef1e6-5453-414c-a759-d70493addc5c-plugins-config-volume\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:24:49.482634 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:49.482365 4421 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ea9ef1e6-5453-414c-a759-d70493addc5c-routing-overlay-volume\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:24:49.485376 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:49.483852 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea9ef1e6-5453-414c-a759-d70493addc5c-kube-api-access-pxl7t" pod "ea9ef1e6-5453-414c-a759-d70493addc5c" (UID: "ea9ef1e6-5453-414c-a759-d70493addc5c"). InnerVolumeSpecName "kube-api-access-pxl7t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:24:49.485376 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:49.484009 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea9ef1e6-5453-414c-a759-d70493addc5c-provider-credentials" pod "ea9ef1e6-5453-414c-a759-d70493addc5c" (UID: "ea9ef1e6-5453-414c-a759-d70493addc5c"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:24:49.583897 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:49.583859 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxl7t\" (UniqueName: \"kubernetes.io/projected/ea9ef1e6-5453-414c-a759-d70493addc5c-kube-api-access-pxl7t\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:24:49.583897 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:49.583891 4421 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ea9ef1e6-5453-414c-a759-d70493addc5c-provider-credentials\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:24:49.873142 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:49.873094 4421 generic.go:356] "Generic (PLEG): container finished" podID="ea9ef1e6-5453-414c-a759-d70493addc5c" containerID="40433d0cb96cd0730ccac952fdef6a5f064f6df85e85c509b34691be479d8a57" exitCode=0 Nov 03 01:24:49.873385 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:49.873208 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-579b4mp4l" Nov 03 01:24:49.873385 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:49.873318 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-579b4mp4l" event={"ID":"ea9ef1e6-5453-414c-a759-d70493addc5c","Type":"ContainerDied","Data":"40433d0cb96cd0730ccac952fdef6a5f064f6df85e85c509b34691be479d8a57"} Nov 03 01:24:49.873560 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:49.873387 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-579b4mp4l" event={"ID":"ea9ef1e6-5453-414c-a759-d70493addc5c","Type":"ContainerDied","Data":"fd1683498aa7bfabd844a10a91df564ff1073ebba844dc4dac9ce792f324fd93"} Nov 03 01:24:49.873560 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:49.873417 4421 scope.go:122] "RemoveContainer" containerID="40433d0cb96cd0730ccac952fdef6a5f064f6df85e85c509b34691be479d8a57" Nov 03 01:24:49.892302 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:49.892281 4421 scope.go:122] "RemoveContainer" containerID="40433d0cb96cd0730ccac952fdef6a5f064f6df85e85c509b34691be479d8a57" Nov 03 01:24:49.893312 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:24:49.893286 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40433d0cb96cd0730ccac952fdef6a5f064f6df85e85c509b34691be479d8a57\": container with ID starting with 40433d0cb96cd0730ccac952fdef6a5f064f6df85e85c509b34691be479d8a57 not found: ID does not exist" containerID="40433d0cb96cd0730ccac952fdef6a5f064f6df85e85c509b34691be479d8a57" Nov 03 01:24:49.893437 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:49.893317 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40433d0cb96cd0730ccac952fdef6a5f064f6df85e85c509b34691be479d8a57"} err="failed to get container status \"40433d0cb96cd0730ccac952fdef6a5f064f6df85e85c509b34691be479d8a57\": rpc error: code = NotFound desc = could not find container \"40433d0cb96cd0730ccac952fdef6a5f064f6df85e85c509b34691be479d8a57\": container with ID starting with 40433d0cb96cd0730ccac952fdef6a5f064f6df85e85c509b34691be479d8a57 not found: ID does not exist" Nov 03 01:24:49.905912 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:49.905877 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-579b4mp4l"] Nov 03 01:24:49.922206 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:49.921165 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-579b4mp4l"] Nov 03 01:24:50.067296 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:50.064182 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea9ef1e6-5453-414c-a759-d70493addc5c" path="/var/lib/kubelet/pods/ea9ef1e6-5453-414c-a759-d70493addc5c/volumes" Nov 03 01:24:50.534533 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:50.534486 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-fcw4z" Nov 03 01:24:50.535732 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:50.535712 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-fcw4z" Nov 03 01:24:51.944109 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:51.944071 4421 generic.go:356] "Generic (PLEG): container finished" podID="e6c36e3a-89e3-4f54-8266-c7dcfc0ca19b" containerID="25c5f74ca2414a88339beef2035ed59bc07f8a1ba6fdb15614b0c100590ec788" exitCode=0 Nov 03 01:24:51.944558 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:51.944145 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-gwl6k" event={"ID":"e6c36e3a-89e3-4f54-8266-c7dcfc0ca19b","Type":"ContainerDied","Data":"25c5f74ca2414a88339beef2035ed59bc07f8a1ba6fdb15614b0c100590ec788"} Nov 03 01:24:51.944558 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:51.944218 4421 scope.go:122] "RemoveContainer" containerID="35d6c0da9bf4ae0002ba6cde6d7fcb78cc529f65db86aca46c4b5fe17840a493" Nov 03 01:24:52.104463 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:52.104432 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-gwl6k" Nov 03 01:24:52.153600 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:52.153544 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-vhjlx"] Nov 03 01:24:52.158143 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:52.158106 4421 cpu_manager.go:414] "containerMap: removing container" podUID="ea9ef1e6-5453-414c-a759-d70493addc5c" containerName="payload-processing" Nov 03 01:24:52.159850 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:52.159819 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea9ef1e6-5453-414c-a759-d70493addc5c" containerName="payload-processing" Nov 03 01:24:52.160040 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:52.159892 4421 cpu_manager.go:414] "containerMap: removing container" podUID="e6c36e3a-89e3-4f54-8266-c7dcfc0ca19b" containerName="istio-proxy" Nov 03 01:24:52.160040 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:52.159903 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6c36e3a-89e3-4f54-8266-c7dcfc0ca19b" containerName="istio-proxy" Nov 03 01:24:52.160040 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:52.159963 4421 cpu_manager.go:414] "containerMap: removing container" podUID="e6c36e3a-89e3-4f54-8266-c7dcfc0ca19b" containerName="istio-proxy" Nov 03 01:24:52.160040 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:52.159972 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6c36e3a-89e3-4f54-8266-c7dcfc0ca19b" containerName="istio-proxy" Nov 03 01:24:52.161113 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:52.161083 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea9ef1e6-5453-414c-a759-d70493addc5c" containerName="payload-processing" Nov 03 01:24:52.161241 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:52.161117 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6c36e3a-89e3-4f54-8266-c7dcfc0ca19b" containerName="istio-proxy" Nov 03 01:24:52.161241 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:52.161143 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6c36e3a-89e3-4f54-8266-c7dcfc0ca19b" containerName="istio-proxy" Nov 03 01:24:52.169379 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:52.169352 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-vhjlx" Nov 03 01:24:52.170056 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:52.170028 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-vhjlx"] Nov 03 01:24:52.173214 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:52.173180 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e2345c1-2a99-4b03-b078-eebe1f198589-istio-token\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-vhjlx\" (UID: \"1e2345c1-2a99-4b03-b078-eebe1f198589\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-vhjlx" Nov 03 01:24:52.173418 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:52.173396 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e2345c1-2a99-4b03-b078-eebe1f198589-istio-podinfo\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-vhjlx\" (UID: \"1e2345c1-2a99-4b03-b078-eebe1f198589\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-vhjlx" Nov 03 01:24:52.173509 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:52.173433 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e2345c1-2a99-4b03-b078-eebe1f198589-istio-envoy\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-vhjlx\" (UID: \"1e2345c1-2a99-4b03-b078-eebe1f198589\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-vhjlx" Nov 03 01:24:52.173509 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:52.173490 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45hxf\" (UniqueName: \"kubernetes.io/projected/1e2345c1-2a99-4b03-b078-eebe1f198589-kube-api-access-45hxf\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-vhjlx\" (UID: \"1e2345c1-2a99-4b03-b078-eebe1f198589\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-vhjlx" Nov 03 01:24:52.173995 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:52.173635 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e2345c1-2a99-4b03-b078-eebe1f198589-workload-socket\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-vhjlx\" (UID: \"1e2345c1-2a99-4b03-b078-eebe1f198589\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-vhjlx" Nov 03 01:24:52.173995 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:52.173673 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e2345c1-2a99-4b03-b078-eebe1f198589-credential-socket\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-vhjlx\" (UID: \"1e2345c1-2a99-4b03-b078-eebe1f198589\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-vhjlx" Nov 03 01:24:52.173995 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:52.173774 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e2345c1-2a99-4b03-b078-eebe1f198589-istio-data\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-vhjlx\" (UID: \"1e2345c1-2a99-4b03-b078-eebe1f198589\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-vhjlx" Nov 03 01:24:52.173995 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:52.173826 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e2345c1-2a99-4b03-b078-eebe1f198589-workload-certs\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-vhjlx\" (UID: \"1e2345c1-2a99-4b03-b078-eebe1f198589\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-vhjlx" Nov 03 01:24:52.173995 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:52.173929 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e2345c1-2a99-4b03-b078-eebe1f198589-istiod-ca-cert\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-vhjlx\" (UID: \"1e2345c1-2a99-4b03-b078-eebe1f198589\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-vhjlx" Nov 03 01:24:52.175318 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:52.175261 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-ee618c-openshift-default-dockercfg-d6ps5\"" Nov 03 01:24:52.274781 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:52.274740 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6c36e3a-89e3-4f54-8266-c7dcfc0ca19b-istio-token\" (UniqueName: \"kubernetes.io/projected/e6c36e3a-89e3-4f54-8266-c7dcfc0ca19b-istio-token\") pod \"e6c36e3a-89e3-4f54-8266-c7dcfc0ca19b\" (UID: \"e6c36e3a-89e3-4f54-8266-c7dcfc0ca19b\") " Nov 03 01:24:52.275231 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:52.275206 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6c36e3a-89e3-4f54-8266-c7dcfc0ca19b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6c36e3a-89e3-4f54-8266-c7dcfc0ca19b-istiod-ca-cert\") pod \"e6c36e3a-89e3-4f54-8266-c7dcfc0ca19b\" (UID: \"e6c36e3a-89e3-4f54-8266-c7dcfc0ca19b\") " Nov 03 01:24:52.275311 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:52.275287 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6c36e3a-89e3-4f54-8266-c7dcfc0ca19b-kube-api-access-bhlzz\" (UniqueName: \"kubernetes.io/projected/e6c36e3a-89e3-4f54-8266-c7dcfc0ca19b-kube-api-access-bhlzz\") pod \"e6c36e3a-89e3-4f54-8266-c7dcfc0ca19b\" (UID: \"e6c36e3a-89e3-4f54-8266-c7dcfc0ca19b\") " Nov 03 01:24:52.275363 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:52.275338 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e6c36e3a-89e3-4f54-8266-c7dcfc0ca19b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6c36e3a-89e3-4f54-8266-c7dcfc0ca19b-istio-podinfo\") pod \"e6c36e3a-89e3-4f54-8266-c7dcfc0ca19b\" (UID: \"e6c36e3a-89e3-4f54-8266-c7dcfc0ca19b\") " Nov 03 01:24:52.275455 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:52.275439 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6c36e3a-89e3-4f54-8266-c7dcfc0ca19b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6c36e3a-89e3-4f54-8266-c7dcfc0ca19b-workload-socket\") pod \"e6c36e3a-89e3-4f54-8266-c7dcfc0ca19b\" (UID: \"e6c36e3a-89e3-4f54-8266-c7dcfc0ca19b\") " Nov 03 01:24:52.275532 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:52.275485 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6c36e3a-89e3-4f54-8266-c7dcfc0ca19b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6c36e3a-89e3-4f54-8266-c7dcfc0ca19b-workload-certs\") pod \"e6c36e3a-89e3-4f54-8266-c7dcfc0ca19b\" (UID: \"e6c36e3a-89e3-4f54-8266-c7dcfc0ca19b\") " Nov 03 01:24:52.276408 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:52.275645 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6c36e3a-89e3-4f54-8266-c7dcfc0ca19b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6c36e3a-89e3-4f54-8266-c7dcfc0ca19b-credential-socket\") pod \"e6c36e3a-89e3-4f54-8266-c7dcfc0ca19b\" (UID: \"e6c36e3a-89e3-4f54-8266-c7dcfc0ca19b\") " Nov 03 01:24:52.276408 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:52.275756 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6c36e3a-89e3-4f54-8266-c7dcfc0ca19b-workload-socket" pod "e6c36e3a-89e3-4f54-8266-c7dcfc0ca19b" (UID: "e6c36e3a-89e3-4f54-8266-c7dcfc0ca19b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:24:52.276408 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:52.275811 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6c36e3a-89e3-4f54-8266-c7dcfc0ca19b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6c36e3a-89e3-4f54-8266-c7dcfc0ca19b-istio-envoy\") pod \"e6c36e3a-89e3-4f54-8266-c7dcfc0ca19b\" (UID: \"e6c36e3a-89e3-4f54-8266-c7dcfc0ca19b\") " Nov 03 01:24:52.276408 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:52.275848 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6c36e3a-89e3-4f54-8266-c7dcfc0ca19b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6c36e3a-89e3-4f54-8266-c7dcfc0ca19b-istio-data\") pod \"e6c36e3a-89e3-4f54-8266-c7dcfc0ca19b\" (UID: \"e6c36e3a-89e3-4f54-8266-c7dcfc0ca19b\") " Nov 03 01:24:52.276408 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:52.276113 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e2345c1-2a99-4b03-b078-eebe1f198589-istiod-ca-cert\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-vhjlx\" (UID: \"1e2345c1-2a99-4b03-b078-eebe1f198589\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-vhjlx" Nov 03 01:24:52.276408 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:52.276221 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e2345c1-2a99-4b03-b078-eebe1f198589-istio-token\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-vhjlx\" (UID: \"1e2345c1-2a99-4b03-b078-eebe1f198589\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-vhjlx" Nov 03 01:24:52.278054 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:52.276424 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e2345c1-2a99-4b03-b078-eebe1f198589-istio-podinfo\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-vhjlx\" (UID: \"1e2345c1-2a99-4b03-b078-eebe1f198589\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-vhjlx" Nov 03 01:24:52.278054 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:52.276452 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e2345c1-2a99-4b03-b078-eebe1f198589-istio-envoy\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-vhjlx\" (UID: \"1e2345c1-2a99-4b03-b078-eebe1f198589\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-vhjlx" Nov 03 01:24:52.278054 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:52.276647 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45hxf\" (UniqueName: \"kubernetes.io/projected/1e2345c1-2a99-4b03-b078-eebe1f198589-kube-api-access-45hxf\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-vhjlx\" (UID: \"1e2345c1-2a99-4b03-b078-eebe1f198589\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-vhjlx" Nov 03 01:24:52.278054 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:52.276838 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e2345c1-2a99-4b03-b078-eebe1f198589-workload-socket\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-vhjlx\" (UID: \"1e2345c1-2a99-4b03-b078-eebe1f198589\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-vhjlx" Nov 03 01:24:52.278054 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:52.276872 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e2345c1-2a99-4b03-b078-eebe1f198589-credential-socket\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-vhjlx\" (UID: \"1e2345c1-2a99-4b03-b078-eebe1f198589\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-vhjlx" Nov 03 01:24:52.278054 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:52.276929 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e2345c1-2a99-4b03-b078-eebe1f198589-istiod-ca-cert\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-vhjlx\" (UID: \"1e2345c1-2a99-4b03-b078-eebe1f198589\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-vhjlx" Nov 03 01:24:52.278054 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:52.276986 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e2345c1-2a99-4b03-b078-eebe1f198589-istio-data\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-vhjlx\" (UID: \"1e2345c1-2a99-4b03-b078-eebe1f198589\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-vhjlx" Nov 03 01:24:52.278054 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:52.277034 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e2345c1-2a99-4b03-b078-eebe1f198589-workload-certs\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-vhjlx\" (UID: \"1e2345c1-2a99-4b03-b078-eebe1f198589\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-vhjlx" Nov 03 01:24:52.278054 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:52.277175 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e2345c1-2a99-4b03-b078-eebe1f198589-credential-socket\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-vhjlx\" (UID: \"1e2345c1-2a99-4b03-b078-eebe1f198589\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-vhjlx" Nov 03 01:24:52.278054 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:52.277192 4421 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6c36e3a-89e3-4f54-8266-c7dcfc0ca19b-workload-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:24:52.278054 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:52.277854 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e2345c1-2a99-4b03-b078-eebe1f198589-workload-socket\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-vhjlx\" (UID: \"1e2345c1-2a99-4b03-b078-eebe1f198589\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-vhjlx" Nov 03 01:24:52.278054 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:52.277992 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e2345c1-2a99-4b03-b078-eebe1f198589-istio-data\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-vhjlx\" (UID: \"1e2345c1-2a99-4b03-b078-eebe1f198589\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-vhjlx" Nov 03 01:24:52.278611 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:52.278135 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e2345c1-2a99-4b03-b078-eebe1f198589-workload-certs\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-vhjlx\" (UID: \"1e2345c1-2a99-4b03-b078-eebe1f198589\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-vhjlx" Nov 03 01:24:52.278611 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:52.278235 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e6c36e3a-89e3-4f54-8266-c7dcfc0ca19b-istio-podinfo" pod "e6c36e3a-89e3-4f54-8266-c7dcfc0ca19b" (UID: "e6c36e3a-89e3-4f54-8266-c7dcfc0ca19b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:24:52.278941 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:52.278913 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6c36e3a-89e3-4f54-8266-c7dcfc0ca19b-workload-certs" pod "e6c36e3a-89e3-4f54-8266-c7dcfc0ca19b" (UID: "e6c36e3a-89e3-4f54-8266-c7dcfc0ca19b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:24:52.280299 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:52.279481 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6c36e3a-89e3-4f54-8266-c7dcfc0ca19b-istio-data" pod "e6c36e3a-89e3-4f54-8266-c7dcfc0ca19b" (UID: "e6c36e3a-89e3-4f54-8266-c7dcfc0ca19b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:24:52.280299 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:52.279867 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6c36e3a-89e3-4f54-8266-c7dcfc0ca19b-credential-socket" pod "e6c36e3a-89e3-4f54-8266-c7dcfc0ca19b" (UID: "e6c36e3a-89e3-4f54-8266-c7dcfc0ca19b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:24:52.281170 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:52.280391 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6c36e3a-89e3-4f54-8266-c7dcfc0ca19b-istio-token" pod "e6c36e3a-89e3-4f54-8266-c7dcfc0ca19b" (UID: "e6c36e3a-89e3-4f54-8266-c7dcfc0ca19b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:24:52.281170 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:52.280400 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6c36e3a-89e3-4f54-8266-c7dcfc0ca19b-istiod-ca-cert" pod "e6c36e3a-89e3-4f54-8266-c7dcfc0ca19b" (UID: "e6c36e3a-89e3-4f54-8266-c7dcfc0ca19b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:24:52.281170 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:52.280473 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e2345c1-2a99-4b03-b078-eebe1f198589-istio-envoy\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-vhjlx\" (UID: \"1e2345c1-2a99-4b03-b078-eebe1f198589\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-vhjlx" Nov 03 01:24:52.281170 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:52.280674 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e2345c1-2a99-4b03-b078-eebe1f198589-istio-podinfo\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-vhjlx\" (UID: \"1e2345c1-2a99-4b03-b078-eebe1f198589\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-vhjlx" Nov 03 01:24:52.281170 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:52.280751 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6c36e3a-89e3-4f54-8266-c7dcfc0ca19b-istio-envoy" pod "e6c36e3a-89e3-4f54-8266-c7dcfc0ca19b" (UID: "e6c36e3a-89e3-4f54-8266-c7dcfc0ca19b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:24:52.283490 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:52.283469 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6c36e3a-89e3-4f54-8266-c7dcfc0ca19b-kube-api-access-bhlzz" pod "e6c36e3a-89e3-4f54-8266-c7dcfc0ca19b" (UID: "e6c36e3a-89e3-4f54-8266-c7dcfc0ca19b"). InnerVolumeSpecName "kube-api-access-bhlzz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:24:52.287567 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:52.287548 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e2345c1-2a99-4b03-b078-eebe1f198589-istio-token\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-vhjlx\" (UID: \"1e2345c1-2a99-4b03-b078-eebe1f198589\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-vhjlx" Nov 03 01:24:52.288104 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:52.287734 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45hxf\" (UniqueName: \"kubernetes.io/projected/1e2345c1-2a99-4b03-b078-eebe1f198589-kube-api-access-45hxf\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-vhjlx\" (UID: \"1e2345c1-2a99-4b03-b078-eebe1f198589\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-vhjlx" Nov 03 01:24:52.401349 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:52.397507 4421 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6c36e3a-89e3-4f54-8266-c7dcfc0ca19b-credential-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:24:52.401349 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:52.397551 4421 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6c36e3a-89e3-4f54-8266-c7dcfc0ca19b-istio-envoy\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:24:52.401349 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:52.397569 4421 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6c36e3a-89e3-4f54-8266-c7dcfc0ca19b-istio-data\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:24:52.401349 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:52.397583 4421 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6c36e3a-89e3-4f54-8266-c7dcfc0ca19b-istio-token\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:24:52.401349 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:52.397600 4421 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6c36e3a-89e3-4f54-8266-c7dcfc0ca19b-istiod-ca-cert\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:24:52.401349 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:52.397616 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhlzz\" (UniqueName: \"kubernetes.io/projected/e6c36e3a-89e3-4f54-8266-c7dcfc0ca19b-kube-api-access-bhlzz\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:24:52.401349 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:52.397632 4421 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6c36e3a-89e3-4f54-8266-c7dcfc0ca19b-istio-podinfo\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:24:52.401349 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:52.397646 4421 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6c36e3a-89e3-4f54-8266-c7dcfc0ca19b-workload-certs\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:24:52.490228 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:52.490192 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-vhjlx" Nov 03 01:24:52.783520 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:24:52.783490 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e2345c1_2a99_4b03_b078_eebe1f198589.slice/crio-b4255f57b4864f07879f04d0762b47722fc529ae128fdc08adb488343ab6ef86 WatchSource:0}: Error finding container b4255f57b4864f07879f04d0762b47722fc529ae128fdc08adb488343ab6ef86: Status 404 returned error can't find the container with id b4255f57b4864f07879f04d0762b47722fc529ae128fdc08adb488343ab6ef86 Nov 03 01:24:52.784942 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:52.784914 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-vhjlx"] Nov 03 01:24:52.787003 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:52.786972 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:24:52.787070 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:52.787035 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:24:52.787070 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:52.787065 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:24:52.955652 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:52.955611 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-gwl6k" event={"ID":"e6c36e3a-89e3-4f54-8266-c7dcfc0ca19b","Type":"ContainerDied","Data":"b1e181a1e651c1a44f62794edcbab0f86277704334d2ba45f7789760491d7b87"} Nov 03 01:24:52.956143 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:52.955689 4421 scope.go:122] "RemoveContainer" containerID="25c5f74ca2414a88339beef2035ed59bc07f8a1ba6fdb15614b0c100590ec788" Nov 03 01:24:52.956143 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:52.955625 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-gwl6k" Nov 03 01:24:52.957726 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:52.957691 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-vhjlx" event={"ID":"1e2345c1-2a99-4b03-b078-eebe1f198589","Type":"ContainerStarted","Data":"751bf328470f6f5ab404fb70bf67aa7c9c20ff91cd278e623b26099e33f00960"} Nov 03 01:24:52.958692 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:52.957755 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-vhjlx" event={"ID":"1e2345c1-2a99-4b03-b078-eebe1f198589","Type":"ContainerStarted","Data":"b4255f57b4864f07879f04d0762b47722fc529ae128fdc08adb488343ab6ef86"} Nov 03 01:24:52.984545 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:52.984488 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-vhjlx" podStartSLOduration=3.984467554 podStartE2EDuration="3.984467554s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:24:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:24:52.979378257 +0000 UTC m=+2635.150381354" watchObservedRunningTime="2026-11-03 01:24:52.984467554 +0000 UTC m=+2635.155470656" Nov 03 01:24:53.004133 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:53.004044 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-gwl6k"] Nov 03 01:24:53.008192 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:53.008162 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-gwl6k"] Nov 03 01:24:53.490715 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:53.490663 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-vhjlx" Nov 03 01:24:53.492594 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:53.492543 4421 patch_prober.go:29] interesting pod/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-vhjlx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.54:15021/healthz/ready\": dial tcp 10.134.1.54:15021: connect: connection refused" start-of-body= Nov 03 01:24:53.492719 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:53.492607 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-vhjlx" podUID="1e2345c1-2a99-4b03-b078-eebe1f198589" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.54:15021/healthz/ready\": dial tcp 10.134.1.54:15021: connect: connection refused" Nov 03 01:24:53.982189 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:53.982151 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6c36e3a-89e3-4f54-8266-c7dcfc0ca19b" path="/var/lib/kubelet/pods/e6c36e3a-89e3-4f54-8266-c7dcfc0ca19b/volumes" Nov 03 01:24:54.497347 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:54.497319 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-vhjlx" Nov 03 01:24:54.990373 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:54.990288 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-vhjlx" Nov 03 01:24:54.991438 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:54.991413 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-vhjlx" Nov 03 01:24:56.788996 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:56.788963 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-7d466hz7g"] Nov 03 01:24:56.794067 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:56.794049 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-7d466hz7g" Nov 03 01:24:56.799697 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:56.799670 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-af8244\"/\"agc-cred-9eb5259cb578\"" Nov 03 01:24:56.801024 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:56.800996 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-af8244\"/\"openshift-service-ca.crt\"" Nov 03 01:24:56.801024 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:56.801012 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-af8244\"/\"payload-processing-external-model-plugins-e2e-extmod-b-af8244\"" Nov 03 01:24:56.801024 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:56.801006 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-af8244\"/\"kube-root-ca.crt\"" Nov 03 01:24:56.802078 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:56.801308 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-af8244\"/\"routing-overlay\"" Nov 03 01:24:56.802078 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:56.801415 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-af8244\"/\"payload-processing-external-model-e2e--fa7ecc9b-dockercfg-hm65p\"" Nov 03 01:24:56.803788 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:56.803763 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-7d466hz7g"] Nov 03 01:24:56.998347 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:56.962041 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fc73f2b8-379e-44cb-945f-5f91f05fa833-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-af8244-7d466hz7g\" (UID: \"fc73f2b8-379e-44cb-945f-5f91f05fa833\") " pod="ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-7d466hz7g" Nov 03 01:24:57.093493 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:57.093395 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/fc73f2b8-379e-44cb-945f-5f91f05fa833-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-af8244-7d466hz7g\" (UID: \"fc73f2b8-379e-44cb-945f-5f91f05fa833\") " pod="ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-7d466hz7g" Nov 03 01:24:57.174760 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:57.174723 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rktvh\" (UniqueName: \"kubernetes.io/projected/fc73f2b8-379e-44cb-945f-5f91f05fa833-kube-api-access-rktvh\") pod \"payload-processing-external-model-e2e-extmod-b-af8244-7d466hz7g\" (UID: \"fc73f2b8-379e-44cb-945f-5f91f05fa833\") " pod="ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-7d466hz7g" Nov 03 01:24:57.175088 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:57.175067 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc73f2b8-379e-44cb-945f-5f91f05fa833-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-af8244-7d466hz7g\" (UID: \"fc73f2b8-379e-44cb-945f-5f91f05fa833\") " pod="ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-7d466hz7g" Nov 03 01:24:57.362368 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:57.339122 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fc73f2b8-379e-44cb-945f-5f91f05fa833-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-af8244-7d466hz7g\" (UID: \"fc73f2b8-379e-44cb-945f-5f91f05fa833\") " pod="ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-7d466hz7g" Nov 03 01:24:57.384304 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:57.362756 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/fc73f2b8-379e-44cb-945f-5f91f05fa833-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-af8244-7d466hz7g\" (UID: \"fc73f2b8-379e-44cb-945f-5f91f05fa833\") " pod="ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-7d466hz7g" Nov 03 01:24:57.417890 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:57.345226 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fc73f2b8-379e-44cb-945f-5f91f05fa833-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-af8244-7d466hz7g\" (UID: \"fc73f2b8-379e-44cb-945f-5f91f05fa833\") " pod="ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-7d466hz7g" Nov 03 01:24:57.418170 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:57.389433 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/fc73f2b8-379e-44cb-945f-5f91f05fa833-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-af8244-7d466hz7g\" (UID: \"fc73f2b8-379e-44cb-945f-5f91f05fa833\") " pod="ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-7d466hz7g" Nov 03 01:24:57.418327 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:57.418062 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rktvh\" (UniqueName: \"kubernetes.io/projected/fc73f2b8-379e-44cb-945f-5f91f05fa833-kube-api-access-rktvh\") pod \"payload-processing-external-model-e2e-extmod-b-af8244-7d466hz7g\" (UID: \"fc73f2b8-379e-44cb-945f-5f91f05fa833\") " pod="ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-7d466hz7g" Nov 03 01:24:57.418645 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:57.418624 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc73f2b8-379e-44cb-945f-5f91f05fa833-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-af8244-7d466hz7g\" (UID: \"fc73f2b8-379e-44cb-945f-5f91f05fa833\") " pod="ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-7d466hz7g" Nov 03 01:24:57.420099 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:57.420073 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc73f2b8-379e-44cb-945f-5f91f05fa833-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-af8244-7d466hz7g\" (UID: \"fc73f2b8-379e-44cb-945f-5f91f05fa833\") " pod="ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-7d466hz7g" Nov 03 01:24:57.434996 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:57.432800 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rktvh\" (UniqueName: \"kubernetes.io/projected/fc73f2b8-379e-44cb-945f-5f91f05fa833-kube-api-access-rktvh\") pod \"payload-processing-external-model-e2e-extmod-b-af8244-7d466hz7g\" (UID: \"fc73f2b8-379e-44cb-945f-5f91f05fa833\") " pod="ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-7d466hz7g" Nov 03 01:24:57.713407 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:57.713061 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-7d466hz7g" Nov 03 01:24:58.242220 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:24:58.242183 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc73f2b8_379e_44cb_945f_5f91f05fa833.slice/crio-325fea208f8d0684e7651b6d62e767f6e9d07b468baa7a4f782e5d681d351f9c WatchSource:0}: Error finding container 325fea208f8d0684e7651b6d62e767f6e9d07b468baa7a4f782e5d681d351f9c: Status 404 returned error can't find the container with id 325fea208f8d0684e7651b6d62e767f6e9d07b468baa7a4f782e5d681d351f9c Nov 03 01:24:58.242818 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:58.242791 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-7d466hz7g"] Nov 03 01:24:58.500086 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:58.500043 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-7d466hz7g" event={"ID":"fc73f2b8-379e-44cb-945f-5f91f05fa833","Type":"ContainerStarted","Data":"77f2b9fa12e4a79f6cf4126fde10dbad574fd801760163f8d4e7b0a67de74e88"} Nov 03 01:24:58.500268 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:58.500148 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-7d466hz7g" event={"ID":"fc73f2b8-379e-44cb-945f-5f91f05fa833","Type":"ContainerStarted","Data":"325fea208f8d0684e7651b6d62e767f6e9d07b468baa7a4f782e5d681d351f9c"} Nov 03 01:24:58.501335 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:58.501314 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-7d466hz7g" Nov 03 01:24:58.522818 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:24:58.522702 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-7d466hz7g" podStartSLOduration=2.522684929 podStartE2EDuration="2.522684929s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:24:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:24:58.522251753 +0000 UTC m=+2640.693254850" watchObservedRunningTime="2026-11-03 01:24:58.522684929 +0000 UTC m=+2640.693688026" Nov 03 01:25:02.094995 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:25:02.094957 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5b879d4-kr8sh"] Nov 03 01:25:02.112261 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:25:02.112222 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5b879d4-kr8sh"] Nov 03 01:25:02.713842 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:25:02.713810 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-7d466hz7g"] Nov 03 01:25:03.208579 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:25:03.208546 4421 scope.go:122] "RemoveContainer" containerID="516ee361c55d3d360f60cd1e3eff9d62d54f9712257b89a59f61da78abb8efb7" Nov 03 01:25:03.280994 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:25:03.280355 4421 scope.go:122] "RemoveContainer" containerID="b3c6f2ab19a1c51fe26d1906df75c6cbf8e026a06a27808d89b40efe84bcca3e" Nov 03 01:25:03.300845 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:25:03.300820 4421 scope.go:122] "RemoveContainer" containerID="1701130cdd478393bbabc5d863adf88b2562a56c8bb16e67ae3cc46d2af995b3" Nov 03 01:25:03.981678 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:25:03.981640 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e200b2a-46e2-4597-985a-7fc94eb47e42" path="/var/lib/kubelet/pods/2e200b2a-46e2-4597-985a-7fc94eb47e42/volumes" Nov 03 01:25:07.714584 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:25:07.714503 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-7d466hz7g" podUID="fc73f2b8-379e-44cb-945f-5f91f05fa833" containerName="payload-processing" containerID="cri-o://77f2b9fa12e4a79f6cf4126fde10dbad574fd801760163f8d4e7b0a67de74e88" gracePeriod=25 Nov 03 01:25:08.015834 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:25:08.015801 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-7d466hz7g" Nov 03 01:25:08.045426 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:25:08.045393 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ee618c-7bc658cb5-2kgml"] Nov 03 01:25:08.046761 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:25:08.046732 4421 cpu_manager.go:414] "containerMap: removing container" podUID="fc73f2b8-379e-44cb-945f-5f91f05fa833" containerName="payload-processing" Nov 03 01:25:08.046878 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:25:08.046762 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc73f2b8-379e-44cb-945f-5f91f05fa833" containerName="payload-processing" Nov 03 01:25:08.047673 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:25:08.047031 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc73f2b8-379e-44cb-945f-5f91f05fa833" containerName="payload-processing" Nov 03 01:25:08.052302 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:25:08.052264 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-ee618c-7bc658cb5-2kgml" Nov 03 01:25:08.058906 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:25:08.058854 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-ee618c\"" Nov 03 01:25:08.088581 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:25:08.061095 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-ee618c-dockercfg-nmqxx\"" Nov 03 01:25:08.090027 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:25:08.090002 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc73f2b8-379e-44cb-945f-5f91f05fa833-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc73f2b8-379e-44cb-945f-5f91f05fa833-plugins-config-volume\") pod \"fc73f2b8-379e-44cb-945f-5f91f05fa833\" (UID: \"fc73f2b8-379e-44cb-945f-5f91f05fa833\") " Nov 03 01:25:08.090732 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:25:08.090710 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc73f2b8-379e-44cb-945f-5f91f05fa833-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fc73f2b8-379e-44cb-945f-5f91f05fa833-routing-overlay-volume\") pod \"fc73f2b8-379e-44cb-945f-5f91f05fa833\" (UID: \"fc73f2b8-379e-44cb-945f-5f91f05fa833\") " Nov 03 01:25:08.091669 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:25:08.091646 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc73f2b8-379e-44cb-945f-5f91f05fa833-provider-credentials\" (UniqueName: \"kubernetes.io/projected/fc73f2b8-379e-44cb-945f-5f91f05fa833-provider-credentials\") pod \"fc73f2b8-379e-44cb-945f-5f91f05fa833\" (UID: \"fc73f2b8-379e-44cb-945f-5f91f05fa833\") " Nov 03 01:25:08.092533 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:25:08.092514 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc73f2b8-379e-44cb-945f-5f91f05fa833-kube-api-access-rktvh\" (UniqueName: \"kubernetes.io/projected/fc73f2b8-379e-44cb-945f-5f91f05fa833-kube-api-access-rktvh\") pod \"fc73f2b8-379e-44cb-945f-5f91f05fa833\" (UID: \"fc73f2b8-379e-44cb-945f-5f91f05fa833\") " Nov 03 01:25:08.094891 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:25:08.090626 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc73f2b8-379e-44cb-945f-5f91f05fa833-plugins-config-volume" pod "fc73f2b8-379e-44cb-945f-5f91f05fa833" (UID: "fc73f2b8-379e-44cb-945f-5f91f05fa833"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:25:08.095938 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:25:08.091203 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc73f2b8-379e-44cb-945f-5f91f05fa833-routing-overlay-volume" pod "fc73f2b8-379e-44cb-945f-5f91f05fa833" (UID: "fc73f2b8-379e-44cb-945f-5f91f05fa833"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:25:08.103163 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:25:08.103120 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ee618c-7bc658cb5-2kgml"] Nov 03 01:25:08.133543 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:25:08.127505 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc73f2b8-379e-44cb-945f-5f91f05fa833-kube-api-access-rktvh" pod "fc73f2b8-379e-44cb-945f-5f91f05fa833" (UID: "fc73f2b8-379e-44cb-945f-5f91f05fa833"). InnerVolumeSpecName "kube-api-access-rktvh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:25:08.212152 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:25:08.203007 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psx52\" (UniqueName: \"kubernetes.io/projected/157f33b5-87a2-4d03-8288-a8e0cc90de31-kube-api-access-psx52\") pod \"payload-processing-e2e-inf-a-ee618c-7bc658cb5-2kgml\" (UID: \"157f33b5-87a2-4d03-8288-a8e0cc90de31\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ee618c-7bc658cb5-2kgml" Nov 03 01:25:08.212152 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:25:08.203430 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/157f33b5-87a2-4d03-8288-a8e0cc90de31-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-ee618c-7bc658cb5-2kgml\" (UID: \"157f33b5-87a2-4d03-8288-a8e0cc90de31\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ee618c-7bc658cb5-2kgml" Nov 03 01:25:08.212152 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:25:08.203945 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rktvh\" (UniqueName: \"kubernetes.io/projected/fc73f2b8-379e-44cb-945f-5f91f05fa833-kube-api-access-rktvh\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:25:08.212152 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:25:08.203975 4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc73f2b8-379e-44cb-945f-5f91f05fa833-plugins-config-volume\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:25:08.212152 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:25:08.204032 4421 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fc73f2b8-379e-44cb-945f-5f91f05fa833-routing-overlay-volume\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:25:08.242053 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:25:08.241937 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc73f2b8-379e-44cb-945f-5f91f05fa833-provider-credentials" pod "fc73f2b8-379e-44cb-945f-5f91f05fa833" (UID: "fc73f2b8-379e-44cb-945f-5f91f05fa833"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:25:08.305602 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:25:08.305557 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/157f33b5-87a2-4d03-8288-a8e0cc90de31-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-ee618c-7bc658cb5-2kgml\" (UID: \"157f33b5-87a2-4d03-8288-a8e0cc90de31\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ee618c-7bc658cb5-2kgml" Nov 03 01:25:08.305830 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:25:08.305818 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psx52\" (UniqueName: \"kubernetes.io/projected/157f33b5-87a2-4d03-8288-a8e0cc90de31-kube-api-access-psx52\") pod \"payload-processing-e2e-inf-a-ee618c-7bc658cb5-2kgml\" (UID: \"157f33b5-87a2-4d03-8288-a8e0cc90de31\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ee618c-7bc658cb5-2kgml" Nov 03 01:25:08.305983 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:25:08.305972 4421 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/fc73f2b8-379e-44cb-945f-5f91f05fa833-provider-credentials\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:25:08.306410 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:25:08.306389 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/157f33b5-87a2-4d03-8288-a8e0cc90de31-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-ee618c-7bc658cb5-2kgml\" (UID: \"157f33b5-87a2-4d03-8288-a8e0cc90de31\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ee618c-7bc658cb5-2kgml" Nov 03 01:25:08.314217 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:25:08.314184 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-psx52\" (UniqueName: \"kubernetes.io/projected/157f33b5-87a2-4d03-8288-a8e0cc90de31-kube-api-access-psx52\") pod \"payload-processing-e2e-inf-a-ee618c-7bc658cb5-2kgml\" (UID: \"157f33b5-87a2-4d03-8288-a8e0cc90de31\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ee618c-7bc658cb5-2kgml" Nov 03 01:25:08.407708 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:25:08.407668 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-ee618c-7bc658cb5-2kgml" Nov 03 01:25:08.699528 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:25:08.699440 4421 generic.go:356] "Generic (PLEG): container finished" podID="fc73f2b8-379e-44cb-945f-5f91f05fa833" containerID="77f2b9fa12e4a79f6cf4126fde10dbad574fd801760163f8d4e7b0a67de74e88" exitCode=0 Nov 03 01:25:08.699528 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:25:08.699512 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-7d466hz7g" Nov 03 01:25:08.699816 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:25:08.699574 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-7d466hz7g" event={"ID":"fc73f2b8-379e-44cb-945f-5f91f05fa833","Type":"ContainerDied","Data":"77f2b9fa12e4a79f6cf4126fde10dbad574fd801760163f8d4e7b0a67de74e88"} Nov 03 01:25:08.699816 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:25:08.699644 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-7d466hz7g" event={"ID":"fc73f2b8-379e-44cb-945f-5f91f05fa833","Type":"ContainerDied","Data":"325fea208f8d0684e7651b6d62e767f6e9d07b468baa7a4f782e5d681d351f9c"} Nov 03 01:25:08.699816 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:25:08.699678 4421 scope.go:122] "RemoveContainer" containerID="77f2b9fa12e4a79f6cf4126fde10dbad574fd801760163f8d4e7b0a67de74e88" Nov 03 01:25:08.703863 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:25:08.702494 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ee618c-7bc658cb5-2kgml"] Nov 03 01:25:08.709872 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:25:08.709774 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod157f33b5_87a2_4d03_8288_a8e0cc90de31.slice/crio-07db8329cc3207078b1e161481ccd1135832982914900d3942504c92a16c4793 WatchSource:0}: Error finding container 07db8329cc3207078b1e161481ccd1135832982914900d3942504c92a16c4793: Status 404 returned error can't find the container with id 07db8329cc3207078b1e161481ccd1135832982914900d3942504c92a16c4793 Nov 03 01:25:08.714373 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:25:08.714353 4421 scope.go:122] "RemoveContainer" containerID="77f2b9fa12e4a79f6cf4126fde10dbad574fd801760163f8d4e7b0a67de74e88" Nov 03 01:25:08.714708 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:25:08.714677 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77f2b9fa12e4a79f6cf4126fde10dbad574fd801760163f8d4e7b0a67de74e88\": container with ID starting with 77f2b9fa12e4a79f6cf4126fde10dbad574fd801760163f8d4e7b0a67de74e88 not found: ID does not exist" containerID="77f2b9fa12e4a79f6cf4126fde10dbad574fd801760163f8d4e7b0a67de74e88" Nov 03 01:25:08.715093 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:25:08.714719 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77f2b9fa12e4a79f6cf4126fde10dbad574fd801760163f8d4e7b0a67de74e88"} err="failed to get container status \"77f2b9fa12e4a79f6cf4126fde10dbad574fd801760163f8d4e7b0a67de74e88\": rpc error: code = NotFound desc = could not find container \"77f2b9fa12e4a79f6cf4126fde10dbad574fd801760163f8d4e7b0a67de74e88\": container with ID starting with 77f2b9fa12e4a79f6cf4126fde10dbad574fd801760163f8d4e7b0a67de74e88 not found: ID does not exist" Nov 03 01:25:08.732402 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:25:08.732372 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-7d466hz7g"] Nov 03 01:25:08.737807 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:25:08.737782 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-7d466hz7g"] Nov 03 01:25:09.714591 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:25:09.714535 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-ee618c-7bc658cb5-2kgml" event={"ID":"157f33b5-87a2-4d03-8288-a8e0cc90de31","Type":"ContainerStarted","Data":"dcd8edadc6ada4b6f7df7691368602f193a6de408eeaf39e65ff806f7aff509e"} Nov 03 01:25:09.714787 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:25:09.714617 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-ee618c-7bc658cb5-2kgml" event={"ID":"157f33b5-87a2-4d03-8288-a8e0cc90de31","Type":"ContainerStarted","Data":"07db8329cc3207078b1e161481ccd1135832982914900d3942504c92a16c4793"} Nov 03 01:25:09.714787 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:25:09.714671 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-ee618c-7bc658cb5-2kgml" Nov 03 01:25:09.736527 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:25:09.736474 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-ee618c-7bc658cb5-2kgml" podStartSLOduration=12.736460724 podStartE2EDuration="12.736460724s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:24:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:25:09.736176222 +0000 UTC m=+2651.907179317" watchObservedRunningTime="2026-11-03 01:25:09.736460724 +0000 UTC m=+2651.907463819" Nov 03 01:25:10.085305 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:25:10.080457 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc73f2b8-379e-44cb-945f-5f91f05fa833" path="/var/lib/kubelet/pods/fc73f2b8-379e-44cb-945f-5f91f05fa833/volumes" Nov 03 01:25:12.028833 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:25:12.028799 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93bc19da-pjg99"] Nov 03 01:25:12.033999 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:25:12.033972 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93bc19da-pjg99"] Nov 03 01:25:13.982041 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:25:13.982000 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="752f8f24-ac91-4fb6-96ea-963146efe4c9" path="/var/lib/kubelet/pods/752f8f24-ac91-4fb6-96ea-963146efe4c9/volumes" Nov 03 01:25:14.672513 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:25:14.672472 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 01:25:20.795337 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:25:20.795302 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-ee618c-7bc658cb5-2kgml" Nov 03 01:25:22.029931 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:25:22.029892 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-deb4582f-qv4p4"] Nov 03 01:25:22.034571 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:25:22.034545 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-deb4582f-qv4p4"] Nov 03 01:25:23.763113 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:25:23.763074 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-84e20ed8-crbln"] Nov 03 01:25:23.769856 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:25:23.769823 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84e20ed8-crbln" Nov 03 01:25:23.773645 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:25:23.773618 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:25:23.774870 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:25:23.774848 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6q4xp\"" Nov 03 01:25:23.775456 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:25:23.775083 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-84e20ed8-crbln"] Nov 03 01:25:23.875070 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:25:23.875029 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f15340f2-6e04-4537-96d0-6b10fab3a34f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-84e20ed8-crbln\" (UID: \"f15340f2-6e04-4537-96d0-6b10fab3a34f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84e20ed8-crbln" Nov 03 01:25:23.875261 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:25:23.875084 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f15340f2-6e04-4537-96d0-6b10fab3a34f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-84e20ed8-crbln\" (UID: \"f15340f2-6e04-4537-96d0-6b10fab3a34f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84e20ed8-crbln" Nov 03 01:25:23.875343 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:25:23.875317 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfxkr\" (UniqueName: \"kubernetes.io/projected/f15340f2-6e04-4537-96d0-6b10fab3a34f-kube-api-access-wfxkr\") pod \"maas-api-revoke-sub-84e20ed8-crbln\" (UID: \"f15340f2-6e04-4537-96d0-6b10fab3a34f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84e20ed8-crbln" Nov 03 01:25:23.977852 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:25:23.977803 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f15340f2-6e04-4537-96d0-6b10fab3a34f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-84e20ed8-crbln\" (UID: \"f15340f2-6e04-4537-96d0-6b10fab3a34f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84e20ed8-crbln" Nov 03 01:25:23.978051 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:25:23.977876 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f15340f2-6e04-4537-96d0-6b10fab3a34f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-84e20ed8-crbln\" (UID: \"f15340f2-6e04-4537-96d0-6b10fab3a34f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84e20ed8-crbln" Nov 03 01:25:23.978051 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:25:23.978037 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfxkr\" (UniqueName: \"kubernetes.io/projected/f15340f2-6e04-4537-96d0-6b10fab3a34f-kube-api-access-wfxkr\") pod \"maas-api-revoke-sub-84e20ed8-crbln\" (UID: \"f15340f2-6e04-4537-96d0-6b10fab3a34f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84e20ed8-crbln" Nov 03 01:25:23.978483 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:25:23.978460 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f15340f2-6e04-4537-96d0-6b10fab3a34f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-84e20ed8-crbln\" (UID: \"f15340f2-6e04-4537-96d0-6b10fab3a34f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84e20ed8-crbln" Nov 03 01:25:23.980317 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:25:23.980291 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f15340f2-6e04-4537-96d0-6b10fab3a34f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-84e20ed8-crbln\" (UID: \"f15340f2-6e04-4537-96d0-6b10fab3a34f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84e20ed8-crbln" Nov 03 01:25:23.983724 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:25:23.983699 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4a328ec-02f5-4564-a262-97a89dc7dd85" path="/var/lib/kubelet/pods/a4a328ec-02f5-4564-a262-97a89dc7dd85/volumes" Nov 03 01:25:23.986360 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:25:23.986338 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfxkr\" (UniqueName: \"kubernetes.io/projected/f15340f2-6e04-4537-96d0-6b10fab3a34f-kube-api-access-wfxkr\") pod \"maas-api-revoke-sub-84e20ed8-crbln\" (UID: \"f15340f2-6e04-4537-96d0-6b10fab3a34f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84e20ed8-crbln" Nov 03 01:25:24.098564 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:25:24.098464 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84e20ed8-crbln" Nov 03 01:25:24.393296 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:25:24.393252 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf15340f2_6e04_4537_96d0_6b10fab3a34f.slice/crio-13ecee02751260516460dc0713cff27ab65e7cdc8f1c0096d107be4c197b16d3 WatchSource:0}: Error finding container 13ecee02751260516460dc0713cff27ab65e7cdc8f1c0096d107be4c197b16d3: Status 404 returned error can't find the container with id 13ecee02751260516460dc0713cff27ab65e7cdc8f1c0096d107be4c197b16d3 Nov 03 01:25:24.393707 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:25:24.393680 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-84e20ed8-crbln"] Nov 03 01:25:24.999236 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:25:24.999208 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84e20ed8-crbln" event={"ID":"f15340f2-6e04-4537-96d0-6b10fab3a34f","Type":"ContainerStarted","Data":"b31ea700cceb57b7be39745ae51066822042d8df5dcc05e34b55723af0e5a5a3"} Nov 03 01:25:24.999741 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:25:24.999262 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84e20ed8-crbln" event={"ID":"f15340f2-6e04-4537-96d0-6b10fab3a34f","Type":"ContainerStarted","Data":"13ecee02751260516460dc0713cff27ab65e7cdc8f1c0096d107be4c197b16d3"} Nov 03 01:25:25.016444 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:25:25.016376 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84e20ed8-crbln" podStartSLOduration=2.016357513 podStartE2EDuration="2.016357513s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:25:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:25:25.014981572 +0000 UTC m=+2667.185984671" watchObservedRunningTime="2026-11-03 01:25:25.016357513 +0000 UTC m=+2667.187360609" Nov 03 01:25:26.008725 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:25:26.008694 4421 generic.go:356] "Generic (PLEG): container finished" podID="f15340f2-6e04-4537-96d0-6b10fab3a34f" containerID="b31ea700cceb57b7be39745ae51066822042d8df5dcc05e34b55723af0e5a5a3" exitCode=0 Nov 03 01:25:26.009118 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:25:26.008733 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84e20ed8-crbln" event={"ID":"f15340f2-6e04-4537-96d0-6b10fab3a34f","Type":"ContainerDied","Data":"b31ea700cceb57b7be39745ae51066822042d8df5dcc05e34b55723af0e5a5a3"} Nov 03 01:25:27.304894 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:25:27.304869 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84e20ed8-crbln" Nov 03 01:25:27.399340 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:25:27.399247 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f15340f2-6e04-4537-96d0-6b10fab3a34f-kube-api-access-wfxkr\" (UniqueName: \"kubernetes.io/projected/f15340f2-6e04-4537-96d0-6b10fab3a34f-kube-api-access-wfxkr\") pod \"f15340f2-6e04-4537-96d0-6b10fab3a34f\" (UID: \"f15340f2-6e04-4537-96d0-6b10fab3a34f\") " Nov 03 01:25:27.399537 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:25:27.399458 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f15340f2-6e04-4537-96d0-6b10fab3a34f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f15340f2-6e04-4537-96d0-6b10fab3a34f-maas-api-openshift-ca\") pod \"f15340f2-6e04-4537-96d0-6b10fab3a34f\" (UID: \"f15340f2-6e04-4537-96d0-6b10fab3a34f\") " Nov 03 01:25:27.399611 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:25:27.399564 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f15340f2-6e04-4537-96d0-6b10fab3a34f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f15340f2-6e04-4537-96d0-6b10fab3a34f-maas-api-xks-ca\") pod \"f15340f2-6e04-4537-96d0-6b10fab3a34f\" (UID: \"f15340f2-6e04-4537-96d0-6b10fab3a34f\") " Nov 03 01:25:27.401110 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:25:27.401079 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f15340f2-6e04-4537-96d0-6b10fab3a34f-maas-api-openshift-ca" pod "f15340f2-6e04-4537-96d0-6b10fab3a34f" (UID: "f15340f2-6e04-4537-96d0-6b10fab3a34f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:25:27.403861 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:25:27.403824 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f15340f2-6e04-4537-96d0-6b10fab3a34f-kube-api-access-wfxkr" pod "f15340f2-6e04-4537-96d0-6b10fab3a34f" (UID: "f15340f2-6e04-4537-96d0-6b10fab3a34f"). InnerVolumeSpecName "kube-api-access-wfxkr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:25:27.404572 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:25:27.404544 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f15340f2-6e04-4537-96d0-6b10fab3a34f-maas-api-xks-ca" pod "f15340f2-6e04-4537-96d0-6b10fab3a34f" (UID: "f15340f2-6e04-4537-96d0-6b10fab3a34f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:25:27.502346 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:25:27.502314 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f15340f2-6e04-4537-96d0-6b10fab3a34f-maas-api-xks-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:25:27.502346 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:25:27.502341 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wfxkr\" (UniqueName: \"kubernetes.io/projected/f15340f2-6e04-4537-96d0-6b10fab3a34f-kube-api-access-wfxkr\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:25:27.502346 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:25:27.502354 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f15340f2-6e04-4537-96d0-6b10fab3a34f-maas-api-openshift-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:25:28.026402 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:25:28.026369 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84e20ed8-crbln" event={"ID":"f15340f2-6e04-4537-96d0-6b10fab3a34f","Type":"ContainerDied","Data":"13ecee02751260516460dc0713cff27ab65e7cdc8f1c0096d107be4c197b16d3"} Nov 03 01:25:28.026402 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:25:28.026406 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84e20ed8-crbln" Nov 03 01:25:28.026634 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:25:28.026426 4421 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="13ecee02751260516460dc0713cff27ab65e7cdc8f1c0096d107be4c197b16d3" Nov 03 01:25:34.381160 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:25:34.381124 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-73646312-dbwcm"] Nov 03 01:25:34.382326 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:25:34.382042 4421 cpu_manager.go:414] "containerMap: removing container" podUID="f15340f2-6e04-4537-96d0-6b10fab3a34f" containerName="revoke-keys" Nov 03 01:25:34.382326 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:25:34.382057 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f15340f2-6e04-4537-96d0-6b10fab3a34f" containerName="revoke-keys" Nov 03 01:25:34.382326 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:25:34.382206 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="f15340f2-6e04-4537-96d0-6b10fab3a34f" containerName="revoke-keys" Nov 03 01:25:34.387133 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:25:34.387107 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73646312-dbwcm" Nov 03 01:25:34.390797 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:25:34.390766 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:25:34.391933 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:25:34.391915 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6q4xp\"" Nov 03 01:25:34.392240 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:25:34.392216 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-73646312-dbwcm"] Nov 03 01:25:34.489720 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:25:34.489680 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ld8mr\" (UniqueName: \"kubernetes.io/projected/587b5308-6d39-4f44-9cb1-28da425a481e-kube-api-access-ld8mr\") pod \"maas-api-revoke-sub-73646312-dbwcm\" (UID: \"587b5308-6d39-4f44-9cb1-28da425a481e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73646312-dbwcm" Nov 03 01:25:34.489913 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:25:34.489869 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/587b5308-6d39-4f44-9cb1-28da425a481e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-73646312-dbwcm\" (UID: \"587b5308-6d39-4f44-9cb1-28da425a481e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73646312-dbwcm" Nov 03 01:25:34.489991 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:25:34.489948 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/587b5308-6d39-4f44-9cb1-28da425a481e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-73646312-dbwcm\" (UID: \"587b5308-6d39-4f44-9cb1-28da425a481e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73646312-dbwcm" Nov 03 01:25:34.599373 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:25:34.597642 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/587b5308-6d39-4f44-9cb1-28da425a481e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-73646312-dbwcm\" (UID: \"587b5308-6d39-4f44-9cb1-28da425a481e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73646312-dbwcm" Nov 03 01:25:34.599373 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:25:34.597771 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/587b5308-6d39-4f44-9cb1-28da425a481e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-73646312-dbwcm\" (UID: \"587b5308-6d39-4f44-9cb1-28da425a481e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73646312-dbwcm" Nov 03 01:25:34.601314 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:25:34.600123 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ld8mr\" (UniqueName: \"kubernetes.io/projected/587b5308-6d39-4f44-9cb1-28da425a481e-kube-api-access-ld8mr\") pod \"maas-api-revoke-sub-73646312-dbwcm\" (UID: \"587b5308-6d39-4f44-9cb1-28da425a481e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73646312-dbwcm" Nov 03 01:25:34.612300 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:25:34.608835 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/587b5308-6d39-4f44-9cb1-28da425a481e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-73646312-dbwcm\" (UID: \"587b5308-6d39-4f44-9cb1-28da425a481e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73646312-dbwcm" Nov 03 01:25:34.650359 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:25:34.642903 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/587b5308-6d39-4f44-9cb1-28da425a481e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-73646312-dbwcm\" (UID: \"587b5308-6d39-4f44-9cb1-28da425a481e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73646312-dbwcm" Nov 03 01:25:34.656294 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:25:34.654395 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ld8mr\" (UniqueName: \"kubernetes.io/projected/587b5308-6d39-4f44-9cb1-28da425a481e-kube-api-access-ld8mr\") pod \"maas-api-revoke-sub-73646312-dbwcm\" (UID: \"587b5308-6d39-4f44-9cb1-28da425a481e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73646312-dbwcm" Nov 03 01:25:34.712599 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:25:34.712567 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73646312-dbwcm" Nov 03 01:25:34.999764 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:25:34.999734 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod587b5308_6d39_4f44_9cb1_28da425a481e.slice/crio-35315edcd989e7e974f7dd688df748423ecd3d6f918790ea9f2ba487be7ed8a5 WatchSource:0}: Error finding container 35315edcd989e7e974f7dd688df748423ecd3d6f918790ea9f2ba487be7ed8a5: Status 404 returned error can't find the container with id 35315edcd989e7e974f7dd688df748423ecd3d6f918790ea9f2ba487be7ed8a5 Nov 03 01:25:35.000208 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:25:35.000185 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-73646312-dbwcm"] Nov 03 01:25:35.148120 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:25:35.148085 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73646312-dbwcm" event={"ID":"587b5308-6d39-4f44-9cb1-28da425a481e","Type":"ContainerStarted","Data":"35315edcd989e7e974f7dd688df748423ecd3d6f918790ea9f2ba487be7ed8a5"} Nov 03 01:25:36.159626 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:25:36.159588 4421 generic.go:356] "Generic (PLEG): container finished" podID="587b5308-6d39-4f44-9cb1-28da425a481e" containerID="67c897df89aad8918d042dec6841eff2be6de9a96585fe2d0e69e07b16429332" exitCode=0 Nov 03 01:25:36.160020 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:25:36.159676 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73646312-dbwcm" event={"ID":"587b5308-6d39-4f44-9cb1-28da425a481e","Type":"ContainerDied","Data":"67c897df89aad8918d042dec6841eff2be6de9a96585fe2d0e69e07b16429332"} Nov 03 01:25:37.453972 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:25:37.453945 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73646312-dbwcm" Nov 03 01:25:37.498204 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:25:37.498158 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/587b5308-6d39-4f44-9cb1-28da425a481e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/587b5308-6d39-4f44-9cb1-28da425a481e-maas-api-openshift-ca\") pod \"587b5308-6d39-4f44-9cb1-28da425a481e\" (UID: \"587b5308-6d39-4f44-9cb1-28da425a481e\") " Nov 03 01:25:37.498441 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:25:37.498255 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/587b5308-6d39-4f44-9cb1-28da425a481e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/587b5308-6d39-4f44-9cb1-28da425a481e-maas-api-xks-ca\") pod \"587b5308-6d39-4f44-9cb1-28da425a481e\" (UID: \"587b5308-6d39-4f44-9cb1-28da425a481e\") " Nov 03 01:25:37.498441 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:25:37.498349 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/587b5308-6d39-4f44-9cb1-28da425a481e-kube-api-access-ld8mr\" (UniqueName: \"kubernetes.io/projected/587b5308-6d39-4f44-9cb1-28da425a481e-kube-api-access-ld8mr\") pod \"587b5308-6d39-4f44-9cb1-28da425a481e\" (UID: \"587b5308-6d39-4f44-9cb1-28da425a481e\") " Nov 03 01:25:37.499210 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:25:37.499170 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/587b5308-6d39-4f44-9cb1-28da425a481e-maas-api-openshift-ca" pod "587b5308-6d39-4f44-9cb1-28da425a481e" (UID: "587b5308-6d39-4f44-9cb1-28da425a481e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:25:37.501320 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:25:37.501287 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/587b5308-6d39-4f44-9cb1-28da425a481e-maas-api-xks-ca" pod "587b5308-6d39-4f44-9cb1-28da425a481e" (UID: "587b5308-6d39-4f44-9cb1-28da425a481e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:25:37.501540 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:25:37.501512 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/587b5308-6d39-4f44-9cb1-28da425a481e-kube-api-access-ld8mr" pod "587b5308-6d39-4f44-9cb1-28da425a481e" (UID: "587b5308-6d39-4f44-9cb1-28da425a481e"). InnerVolumeSpecName "kube-api-access-ld8mr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:25:37.600636 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:25:37.600599 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/587b5308-6d39-4f44-9cb1-28da425a481e-maas-api-openshift-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:25:37.600636 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:25:37.600631 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/587b5308-6d39-4f44-9cb1-28da425a481e-maas-api-xks-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:25:37.600636 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:25:37.600642 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ld8mr\" (UniqueName: \"kubernetes.io/projected/587b5308-6d39-4f44-9cb1-28da425a481e-kube-api-access-ld8mr\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:25:38.178140 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:25:38.178113 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73646312-dbwcm" Nov 03 01:25:38.178363 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:25:38.178141 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73646312-dbwcm" event={"ID":"587b5308-6d39-4f44-9cb1-28da425a481e","Type":"ContainerDied","Data":"35315edcd989e7e974f7dd688df748423ecd3d6f918790ea9f2ba487be7ed8a5"} Nov 03 01:25:38.178363 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:25:38.178189 4421 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="35315edcd989e7e974f7dd688df748423ecd3d6f918790ea9f2ba487be7ed8a5" Nov 03 01:25:44.672304 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:25:44.672246 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 01:26:01.186603 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:01.186562 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dbbdccc9-lp88t"] Nov 03 01:26:01.376092 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:01.371765 4421 cpu_manager.go:414] "containerMap: removing container" podUID="587b5308-6d39-4f44-9cb1-28da425a481e" containerName="revoke-keys" Nov 03 01:26:01.376092 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:01.371799 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="587b5308-6d39-4f44-9cb1-28da425a481e" containerName="revoke-keys" Nov 03 01:26:01.376092 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:01.372142 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="587b5308-6d39-4f44-9cb1-28da425a481e" containerName="revoke-keys" Nov 03 01:26:01.377503 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:01.377483 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbbdccc9-lp88t" Nov 03 01:26:01.384426 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:01.383102 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:26:01.387698 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:01.387675 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6q4xp\"" Nov 03 01:26:01.389393 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:01.389367 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dbbdccc9-lp88t"] Nov 03 01:26:01.531717 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:01.531675 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kl2rr\" (UniqueName: \"kubernetes.io/projected/3f7644c6-d7bd-4e28-ab78-ed80a99428dc-kube-api-access-kl2rr\") pod \"maas-api-revoke-sub-dbbdccc9-lp88t\" (UID: \"3f7644c6-d7bd-4e28-ab78-ed80a99428dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbbdccc9-lp88t" Nov 03 01:26:01.532381 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:01.531810 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f7644c6-d7bd-4e28-ab78-ed80a99428dc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dbbdccc9-lp88t\" (UID: \"3f7644c6-d7bd-4e28-ab78-ed80a99428dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbbdccc9-lp88t" Nov 03 01:26:01.532381 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:01.531850 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f7644c6-d7bd-4e28-ab78-ed80a99428dc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dbbdccc9-lp88t\" (UID: \"3f7644c6-d7bd-4e28-ab78-ed80a99428dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbbdccc9-lp88t" Nov 03 01:26:01.633707 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:01.633669 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f7644c6-d7bd-4e28-ab78-ed80a99428dc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dbbdccc9-lp88t\" (UID: \"3f7644c6-d7bd-4e28-ab78-ed80a99428dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbbdccc9-lp88t" Nov 03 01:26:01.633938 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:01.633728 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f7644c6-d7bd-4e28-ab78-ed80a99428dc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dbbdccc9-lp88t\" (UID: \"3f7644c6-d7bd-4e28-ab78-ed80a99428dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbbdccc9-lp88t" Nov 03 01:26:01.634068 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:01.634051 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kl2rr\" (UniqueName: \"kubernetes.io/projected/3f7644c6-d7bd-4e28-ab78-ed80a99428dc-kube-api-access-kl2rr\") pod \"maas-api-revoke-sub-dbbdccc9-lp88t\" (UID: \"3f7644c6-d7bd-4e28-ab78-ed80a99428dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbbdccc9-lp88t" Nov 03 01:26:01.634855 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:01.634820 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f7644c6-d7bd-4e28-ab78-ed80a99428dc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dbbdccc9-lp88t\" (UID: \"3f7644c6-d7bd-4e28-ab78-ed80a99428dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbbdccc9-lp88t" Nov 03 01:26:01.636403 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:01.636373 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f7644c6-d7bd-4e28-ab78-ed80a99428dc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dbbdccc9-lp88t\" (UID: \"3f7644c6-d7bd-4e28-ab78-ed80a99428dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbbdccc9-lp88t" Nov 03 01:26:01.646589 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:01.646562 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kl2rr\" (UniqueName: \"kubernetes.io/projected/3f7644c6-d7bd-4e28-ab78-ed80a99428dc-kube-api-access-kl2rr\") pod \"maas-api-revoke-sub-dbbdccc9-lp88t\" (UID: \"3f7644c6-d7bd-4e28-ab78-ed80a99428dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbbdccc9-lp88t" Nov 03 01:26:01.746008 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:01.745971 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbbdccc9-lp88t" Nov 03 01:26:01.795101 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:01.795003 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j52qt"] Nov 03 01:26:01.796034 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:01.795263 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j52qt" podUID="9417c149-c243-4ba7-9f66-7ad0688a9f78" containerName="istio-proxy" containerID="cri-o://c39c010b7c84d7a414eb844b763fa6e4134edc385464d389a6b7425498aca746" gracePeriod=30 Nov 03 01:26:02.102167 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:02.102132 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dbbdccc9-lp88t"] Nov 03 01:26:02.108793 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:26:02.108762 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f7644c6_d7bd_4e28_ab78_ed80a99428dc.slice/crio-42f2c3545aac0baddce8eb8978655866c70d1d1598ae5901b8ff2f632200c14e WatchSource:0}: Error finding container 42f2c3545aac0baddce8eb8978655866c70d1d1598ae5901b8ff2f632200c14e: Status 404 returned error can't find the container with id 42f2c3545aac0baddce8eb8978655866c70d1d1598ae5901b8ff2f632200c14e Nov 03 01:26:03.105172 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:03.105071 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbbdccc9-lp88t" event={"ID":"3f7644c6-d7bd-4e28-ab78-ed80a99428dc","Type":"ContainerStarted","Data":"443d16093546a0bf0378f12426effa19e7504067d5adeeaa4ead18d0e0e76459"} Nov 03 01:26:03.105172 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:03.105147 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbbdccc9-lp88t" event={"ID":"3f7644c6-d7bd-4e28-ab78-ed80a99428dc","Type":"ContainerStarted","Data":"42f2c3545aac0baddce8eb8978655866c70d1d1598ae5901b8ff2f632200c14e"} Nov 03 01:26:03.145357 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:03.145288 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbbdccc9-lp88t" podStartSLOduration=2.145256248 podStartE2EDuration="2.145256248s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:26:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:26:03.136620402 +0000 UTC m=+2705.307623499" watchObservedRunningTime="2026-11-03 01:26:03.145256248 +0000 UTC m=+2705.316259343" Nov 03 01:26:03.436756 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:03.436658 4421 scope.go:122] "RemoveContainer" containerID="b05376c40dc877e2dddbc9e4ec3e2d8b106a096d28ff734c8746aba8a7da17be" Nov 03 01:26:03.466811 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:03.466783 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-bktsz"] Nov 03 01:26:03.467303 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:03.467254 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-bktsz" podUID="85cec606-e8b7-4ce1-8af2-4afb303268f9" containerName="istio-proxy" containerID="cri-o://37f978db4958aa10297ee8efda2770ebcf37c14ad3910ad9e10f356af46ea6d3" gracePeriod=30 Nov 03 01:26:03.555984 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:03.555944 4421 scope.go:122] "RemoveContainer" containerID="3a03bc684b33e68e9cbfc1a634adfd73822d91f7fc3695ec5d90225f56c1b6fd" Nov 03 01:26:03.594491 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:03.585674 4421 scope.go:122] "RemoveContainer" containerID="8f265833c1cd6a6f82dfd8ff2360f264f3e4d37868577439d95fc5000544ff1b" Nov 03 01:26:03.638213 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:03.638145 4421 scope.go:122] "RemoveContainer" containerID="928b6c0cc2980667c0d140643d205f2fcdfea528349d3984a2c62d3b0c8dc1f9" Nov 03 01:26:04.130299 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:04.129754 4421 generic.go:356] "Generic (PLEG): container finished" podID="3f7644c6-d7bd-4e28-ab78-ed80a99428dc" containerID="443d16093546a0bf0378f12426effa19e7504067d5adeeaa4ead18d0e0e76459" exitCode=0 Nov 03 01:26:04.130299 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:04.129969 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbbdccc9-lp88t" event={"ID":"3f7644c6-d7bd-4e28-ab78-ed80a99428dc","Type":"ContainerDied","Data":"443d16093546a0bf0378f12426effa19e7504067d5adeeaa4ead18d0e0e76459"} Nov 03 01:26:05.541217 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:05.541181 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbbdccc9-lp88t" Nov 03 01:26:05.633115 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:05.633066 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f7644c6-d7bd-4e28-ab78-ed80a99428dc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f7644c6-d7bd-4e28-ab78-ed80a99428dc-maas-api-openshift-ca\") pod \"3f7644c6-d7bd-4e28-ab78-ed80a99428dc\" (UID: \"3f7644c6-d7bd-4e28-ab78-ed80a99428dc\") " Nov 03 01:26:05.633406 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:05.633379 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f7644c6-d7bd-4e28-ab78-ed80a99428dc-kube-api-access-kl2rr\" (UniqueName: \"kubernetes.io/projected/3f7644c6-d7bd-4e28-ab78-ed80a99428dc-kube-api-access-kl2rr\") pod \"3f7644c6-d7bd-4e28-ab78-ed80a99428dc\" (UID: \"3f7644c6-d7bd-4e28-ab78-ed80a99428dc\") " Nov 03 01:26:05.634034 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:05.633867 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f7644c6-d7bd-4e28-ab78-ed80a99428dc-maas-api-openshift-ca" pod "3f7644c6-d7bd-4e28-ab78-ed80a99428dc" (UID: "3f7644c6-d7bd-4e28-ab78-ed80a99428dc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:26:05.634161 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:05.634138 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f7644c6-d7bd-4e28-ab78-ed80a99428dc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f7644c6-d7bd-4e28-ab78-ed80a99428dc-maas-api-xks-ca\") pod \"3f7644c6-d7bd-4e28-ab78-ed80a99428dc\" (UID: \"3f7644c6-d7bd-4e28-ab78-ed80a99428dc\") " Nov 03 01:26:05.635097 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:05.634981 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f7644c6-d7bd-4e28-ab78-ed80a99428dc-maas-api-openshift-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:26:05.641533 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:05.641444 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f7644c6-d7bd-4e28-ab78-ed80a99428dc-maas-api-xks-ca" pod "3f7644c6-d7bd-4e28-ab78-ed80a99428dc" (UID: "3f7644c6-d7bd-4e28-ab78-ed80a99428dc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:26:05.644127 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:05.644099 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f7644c6-d7bd-4e28-ab78-ed80a99428dc-kube-api-access-kl2rr" pod "3f7644c6-d7bd-4e28-ab78-ed80a99428dc" (UID: "3f7644c6-d7bd-4e28-ab78-ed80a99428dc"). InnerVolumeSpecName "kube-api-access-kl2rr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:26:05.736381 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:05.736340 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f7644c6-d7bd-4e28-ab78-ed80a99428dc-maas-api-xks-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:26:05.736381 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:05.736374 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kl2rr\" (UniqueName: \"kubernetes.io/projected/3f7644c6-d7bd-4e28-ab78-ed80a99428dc-kube-api-access-kl2rr\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:26:06.029187 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:06.029150 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f51d1a9-5zl5h"] Nov 03 01:26:06.037238 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:06.037196 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f51d1a9-5zl5h"] Nov 03 01:26:06.153796 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:06.153768 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbbdccc9-lp88t" Nov 03 01:26:06.153990 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:06.153805 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbbdccc9-lp88t" event={"ID":"3f7644c6-d7bd-4e28-ab78-ed80a99428dc","Type":"ContainerDied","Data":"42f2c3545aac0baddce8eb8978655866c70d1d1598ae5901b8ff2f632200c14e"} Nov 03 01:26:06.153990 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:06.153854 4421 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="42f2c3545aac0baddce8eb8978655866c70d1d1598ae5901b8ff2f632200c14e" Nov 03 01:26:07.179803 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:07.179770 4421 generic.go:356] "Generic (PLEG): container finished" podID="9417c149-c243-4ba7-9f66-7ad0688a9f78" containerID="c39c010b7c84d7a414eb844b763fa6e4134edc385464d389a6b7425498aca746" exitCode=0 Nov 03 01:26:07.183411 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:07.179849 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j52qt" event={"ID":"9417c149-c243-4ba7-9f66-7ad0688a9f78","Type":"ContainerDied","Data":"c39c010b7c84d7a414eb844b763fa6e4134edc385464d389a6b7425498aca746"} Nov 03 01:26:07.245224 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:07.245186 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j52qt" Nov 03 01:26:07.278424 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:07.278371 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9417c149-c243-4ba7-9f66-7ad0688a9f78-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9417c149-c243-4ba7-9f66-7ad0688a9f78-workload-certs\") pod \"9417c149-c243-4ba7-9f66-7ad0688a9f78\" (UID: \"9417c149-c243-4ba7-9f66-7ad0688a9f78\") " Nov 03 01:26:07.278662 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:07.278454 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9417c149-c243-4ba7-9f66-7ad0688a9f78-istio-token\" (UniqueName: \"kubernetes.io/projected/9417c149-c243-4ba7-9f66-7ad0688a9f78-istio-token\") pod \"9417c149-c243-4ba7-9f66-7ad0688a9f78\" (UID: \"9417c149-c243-4ba7-9f66-7ad0688a9f78\") " Nov 03 01:26:07.278662 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:07.278495 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9417c149-c243-4ba7-9f66-7ad0688a9f78-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9417c149-c243-4ba7-9f66-7ad0688a9f78-istio-data\") pod \"9417c149-c243-4ba7-9f66-7ad0688a9f78\" (UID: \"9417c149-c243-4ba7-9f66-7ad0688a9f78\") " Nov 03 01:26:07.278662 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:07.278582 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9417c149-c243-4ba7-9f66-7ad0688a9f78-kube-api-access-hjsw9\" (UniqueName: \"kubernetes.io/projected/9417c149-c243-4ba7-9f66-7ad0688a9f78-kube-api-access-hjsw9\") pod \"9417c149-c243-4ba7-9f66-7ad0688a9f78\" (UID: \"9417c149-c243-4ba7-9f66-7ad0688a9f78\") " Nov 03 01:26:07.278878 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:07.278749 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9417c149-c243-4ba7-9f66-7ad0688a9f78-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9417c149-c243-4ba7-9f66-7ad0688a9f78-istiod-ca-cert\") pod \"9417c149-c243-4ba7-9f66-7ad0688a9f78\" (UID: \"9417c149-c243-4ba7-9f66-7ad0688a9f78\") " Nov 03 01:26:07.278878 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:07.278811 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9417c149-c243-4ba7-9f66-7ad0688a9f78-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9417c149-c243-4ba7-9f66-7ad0688a9f78-workload-socket\") pod \"9417c149-c243-4ba7-9f66-7ad0688a9f78\" (UID: \"9417c149-c243-4ba7-9f66-7ad0688a9f78\") " Nov 03 01:26:07.278878 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:07.278870 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9417c149-c243-4ba7-9f66-7ad0688a9f78-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9417c149-c243-4ba7-9f66-7ad0688a9f78-credential-socket\") pod \"9417c149-c243-4ba7-9f66-7ad0688a9f78\" (UID: \"9417c149-c243-4ba7-9f66-7ad0688a9f78\") " Nov 03 01:26:07.279046 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:07.278939 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9417c149-c243-4ba7-9f66-7ad0688a9f78-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9417c149-c243-4ba7-9f66-7ad0688a9f78-istio-envoy\") pod \"9417c149-c243-4ba7-9f66-7ad0688a9f78\" (UID: \"9417c149-c243-4ba7-9f66-7ad0688a9f78\") " Nov 03 01:26:07.279046 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:07.278978 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9417c149-c243-4ba7-9f66-7ad0688a9f78-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9417c149-c243-4ba7-9f66-7ad0688a9f78-istio-podinfo\") pod \"9417c149-c243-4ba7-9f66-7ad0688a9f78\" (UID: \"9417c149-c243-4ba7-9f66-7ad0688a9f78\") " Nov 03 01:26:07.280030 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:07.280000 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-b6afd6-openshift-default-78d46d7dcc-dg79z"] Nov 03 01:26:07.281074 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:07.281003 4421 cpu_manager.go:414] "containerMap: removing container" podUID="9417c149-c243-4ba7-9f66-7ad0688a9f78" containerName="istio-proxy" Nov 03 01:26:07.281074 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:07.281022 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9417c149-c243-4ba7-9f66-7ad0688a9f78" containerName="istio-proxy" Nov 03 01:26:07.281074 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:07.281063 4421 cpu_manager.go:414] "containerMap: removing container" podUID="3f7644c6-d7bd-4e28-ab78-ed80a99428dc" containerName="revoke-keys" Nov 03 01:26:07.281074 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:07.281071 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f7644c6-d7bd-4e28-ab78-ed80a99428dc" containerName="revoke-keys" Nov 03 01:26:07.281324 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:07.281293 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f7644c6-d7bd-4e28-ab78-ed80a99428dc" containerName="revoke-keys" Nov 03 01:26:07.281324 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:07.281322 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="9417c149-c243-4ba7-9f66-7ad0688a9f78" containerName="istio-proxy" Nov 03 01:26:07.295934 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:07.293426 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b6afd6-openshift-default-78d46d7dcc-dg79z" Nov 03 01:26:07.295934 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:07.295449 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9417c149-c243-4ba7-9f66-7ad0688a9f78-workload-socket" pod "9417c149-c243-4ba7-9f66-7ad0688a9f78" (UID: "9417c149-c243-4ba7-9f66-7ad0688a9f78"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:26:07.295934 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:07.295672 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9417c149-c243-4ba7-9f66-7ad0688a9f78-istio-data" pod "9417c149-c243-4ba7-9f66-7ad0688a9f78" (UID: "9417c149-c243-4ba7-9f66-7ad0688a9f78"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:26:07.299824 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:07.298682 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-b6afd6-openshift-default-dockercfg-gsntz\"" Nov 03 01:26:07.316516 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:07.312487 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9417c149-c243-4ba7-9f66-7ad0688a9f78-istiod-ca-cert" pod "9417c149-c243-4ba7-9f66-7ad0688a9f78" (UID: "9417c149-c243-4ba7-9f66-7ad0688a9f78"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:26:07.325739 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:07.325710 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-b6afd6-openshift-default-78d46d7dcc-dg79z"] Nov 03 01:26:07.329444 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:07.328571 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9417c149-c243-4ba7-9f66-7ad0688a9f78-credential-socket" pod "9417c149-c243-4ba7-9f66-7ad0688a9f78" (UID: "9417c149-c243-4ba7-9f66-7ad0688a9f78"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:26:07.329444 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:07.328586 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9417c149-c243-4ba7-9f66-7ad0688a9f78-workload-certs" pod "9417c149-c243-4ba7-9f66-7ad0688a9f78" (UID: "9417c149-c243-4ba7-9f66-7ad0688a9f78"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:26:07.337709 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:07.337650 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9417c149-c243-4ba7-9f66-7ad0688a9f78-kube-api-access-hjsw9" pod "9417c149-c243-4ba7-9f66-7ad0688a9f78" (UID: "9417c149-c243-4ba7-9f66-7ad0688a9f78"). InnerVolumeSpecName "kube-api-access-hjsw9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:26:07.337901 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:07.337871 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9417c149-c243-4ba7-9f66-7ad0688a9f78-istio-podinfo" pod "9417c149-c243-4ba7-9f66-7ad0688a9f78" (UID: "9417c149-c243-4ba7-9f66-7ad0688a9f78"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:26:07.338154 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:07.338115 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9417c149-c243-4ba7-9f66-7ad0688a9f78-istio-token" pod "9417c149-c243-4ba7-9f66-7ad0688a9f78" (UID: "9417c149-c243-4ba7-9f66-7ad0688a9f78"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:26:07.342799 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:07.342774 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9417c149-c243-4ba7-9f66-7ad0688a9f78-istio-envoy" pod "9417c149-c243-4ba7-9f66-7ad0688a9f78" (UID: "9417c149-c243-4ba7-9f66-7ad0688a9f78"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:26:07.449595 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:07.448995 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50b59bdf-0949-473f-8849-bb5648c6cba1-workload-socket\") pod \"e2e-inf-a-b6afd6-openshift-default-78d46d7dcc-dg79z\" (UID: \"50b59bdf-0949-473f-8849-bb5648c6cba1\") " pod="openshift-ingress/e2e-inf-a-b6afd6-openshift-default-78d46d7dcc-dg79z" Nov 03 01:26:07.449595 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:07.449059 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqzq2\" (UniqueName: \"kubernetes.io/projected/50b59bdf-0949-473f-8849-bb5648c6cba1-kube-api-access-gqzq2\") pod \"e2e-inf-a-b6afd6-openshift-default-78d46d7dcc-dg79z\" (UID: \"50b59bdf-0949-473f-8849-bb5648c6cba1\") " pod="openshift-ingress/e2e-inf-a-b6afd6-openshift-default-78d46d7dcc-dg79z" Nov 03 01:26:07.449595 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:07.449151 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50b59bdf-0949-473f-8849-bb5648c6cba1-workload-certs\") pod \"e2e-inf-a-b6afd6-openshift-default-78d46d7dcc-dg79z\" (UID: \"50b59bdf-0949-473f-8849-bb5648c6cba1\") " pod="openshift-ingress/e2e-inf-a-b6afd6-openshift-default-78d46d7dcc-dg79z" Nov 03 01:26:07.449595 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:07.449247 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50b59bdf-0949-473f-8849-bb5648c6cba1-istiod-ca-cert\") pod \"e2e-inf-a-b6afd6-openshift-default-78d46d7dcc-dg79z\" (UID: \"50b59bdf-0949-473f-8849-bb5648c6cba1\") " pod="openshift-ingress/e2e-inf-a-b6afd6-openshift-default-78d46d7dcc-dg79z" Nov 03 01:26:07.451907 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:07.451858 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50b59bdf-0949-473f-8849-bb5648c6cba1-istio-envoy\") pod \"e2e-inf-a-b6afd6-openshift-default-78d46d7dcc-dg79z\" (UID: \"50b59bdf-0949-473f-8849-bb5648c6cba1\") " pod="openshift-ingress/e2e-inf-a-b6afd6-openshift-default-78d46d7dcc-dg79z" Nov 03 01:26:07.456462 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:07.452166 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50b59bdf-0949-473f-8849-bb5648c6cba1-istio-data\") pod \"e2e-inf-a-b6afd6-openshift-default-78d46d7dcc-dg79z\" (UID: \"50b59bdf-0949-473f-8849-bb5648c6cba1\") " pod="openshift-ingress/e2e-inf-a-b6afd6-openshift-default-78d46d7dcc-dg79z" Nov 03 01:26:07.456462 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:07.452297 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50b59bdf-0949-473f-8849-bb5648c6cba1-credential-socket\") pod \"e2e-inf-a-b6afd6-openshift-default-78d46d7dcc-dg79z\" (UID: \"50b59bdf-0949-473f-8849-bb5648c6cba1\") " pod="openshift-ingress/e2e-inf-a-b6afd6-openshift-default-78d46d7dcc-dg79z" Nov 03 01:26:07.468700 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:07.452380 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50b59bdf-0949-473f-8849-bb5648c6cba1-istio-podinfo\") pod \"e2e-inf-a-b6afd6-openshift-default-78d46d7dcc-dg79z\" (UID: \"50b59bdf-0949-473f-8849-bb5648c6cba1\") " pod="openshift-ingress/e2e-inf-a-b6afd6-openshift-default-78d46d7dcc-dg79z" Nov 03 01:26:07.468700 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:07.456731 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50b59bdf-0949-473f-8849-bb5648c6cba1-istio-token\") pod \"e2e-inf-a-b6afd6-openshift-default-78d46d7dcc-dg79z\" (UID: \"50b59bdf-0949-473f-8849-bb5648c6cba1\") " pod="openshift-ingress/e2e-inf-a-b6afd6-openshift-default-78d46d7dcc-dg79z" Nov 03 01:26:07.468700 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:07.456984 4421 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9417c149-c243-4ba7-9f66-7ad0688a9f78-workload-certs\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:26:07.468700 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:07.457010 4421 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9417c149-c243-4ba7-9f66-7ad0688a9f78-istio-token\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:26:07.468700 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:07.457027 4421 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9417c149-c243-4ba7-9f66-7ad0688a9f78-istio-data\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:26:07.468700 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:07.457046 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hjsw9\" (UniqueName: \"kubernetes.io/projected/9417c149-c243-4ba7-9f66-7ad0688a9f78-kube-api-access-hjsw9\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:26:07.468700 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:07.457068 4421 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9417c149-c243-4ba7-9f66-7ad0688a9f78-istiod-ca-cert\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:26:07.468700 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:07.457087 4421 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9417c149-c243-4ba7-9f66-7ad0688a9f78-workload-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:26:07.468700 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:07.457100 4421 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9417c149-c243-4ba7-9f66-7ad0688a9f78-credential-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:26:07.468700 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:07.457116 4421 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9417c149-c243-4ba7-9f66-7ad0688a9f78-istio-envoy\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:26:07.468700 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:07.457130 4421 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9417c149-c243-4ba7-9f66-7ad0688a9f78-istio-podinfo\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:26:07.558922 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:07.558884 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50b59bdf-0949-473f-8849-bb5648c6cba1-workload-certs\") pod \"e2e-inf-a-b6afd6-openshift-default-78d46d7dcc-dg79z\" (UID: \"50b59bdf-0949-473f-8849-bb5648c6cba1\") " pod="openshift-ingress/e2e-inf-a-b6afd6-openshift-default-78d46d7dcc-dg79z" Nov 03 01:26:07.559105 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:07.558979 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50b59bdf-0949-473f-8849-bb5648c6cba1-istiod-ca-cert\") pod \"e2e-inf-a-b6afd6-openshift-default-78d46d7dcc-dg79z\" (UID: \"50b59bdf-0949-473f-8849-bb5648c6cba1\") " pod="openshift-ingress/e2e-inf-a-b6afd6-openshift-default-78d46d7dcc-dg79z" Nov 03 01:26:07.559180 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:07.559163 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50b59bdf-0949-473f-8849-bb5648c6cba1-istio-envoy\") pod \"e2e-inf-a-b6afd6-openshift-default-78d46d7dcc-dg79z\" (UID: \"50b59bdf-0949-473f-8849-bb5648c6cba1\") " pod="openshift-ingress/e2e-inf-a-b6afd6-openshift-default-78d46d7dcc-dg79z" Nov 03 01:26:07.559239 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:07.559222 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50b59bdf-0949-473f-8849-bb5648c6cba1-istio-data\") pod \"e2e-inf-a-b6afd6-openshift-default-78d46d7dcc-dg79z\" (UID: \"50b59bdf-0949-473f-8849-bb5648c6cba1\") " pod="openshift-ingress/e2e-inf-a-b6afd6-openshift-default-78d46d7dcc-dg79z" Nov 03 01:26:07.559342 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:07.559323 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50b59bdf-0949-473f-8849-bb5648c6cba1-credential-socket\") pod \"e2e-inf-a-b6afd6-openshift-default-78d46d7dcc-dg79z\" (UID: \"50b59bdf-0949-473f-8849-bb5648c6cba1\") " pod="openshift-ingress/e2e-inf-a-b6afd6-openshift-default-78d46d7dcc-dg79z" Nov 03 01:26:07.559408 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:07.559344 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50b59bdf-0949-473f-8849-bb5648c6cba1-workload-certs\") pod \"e2e-inf-a-b6afd6-openshift-default-78d46d7dcc-dg79z\" (UID: \"50b59bdf-0949-473f-8849-bb5648c6cba1\") " pod="openshift-ingress/e2e-inf-a-b6afd6-openshift-default-78d46d7dcc-dg79z" Nov 03 01:26:07.559478 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:07.559407 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50b59bdf-0949-473f-8849-bb5648c6cba1-istio-podinfo\") pod \"e2e-inf-a-b6afd6-openshift-default-78d46d7dcc-dg79z\" (UID: \"50b59bdf-0949-473f-8849-bb5648c6cba1\") " pod="openshift-ingress/e2e-inf-a-b6afd6-openshift-default-78d46d7dcc-dg79z" Nov 03 01:26:07.559478 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:07.559453 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50b59bdf-0949-473f-8849-bb5648c6cba1-istio-token\") pod \"e2e-inf-a-b6afd6-openshift-default-78d46d7dcc-dg79z\" (UID: \"50b59bdf-0949-473f-8849-bb5648c6cba1\") " pod="openshift-ingress/e2e-inf-a-b6afd6-openshift-default-78d46d7dcc-dg79z" Nov 03 01:26:07.559592 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:07.559526 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50b59bdf-0949-473f-8849-bb5648c6cba1-workload-socket\") pod \"e2e-inf-a-b6afd6-openshift-default-78d46d7dcc-dg79z\" (UID: \"50b59bdf-0949-473f-8849-bb5648c6cba1\") " pod="openshift-ingress/e2e-inf-a-b6afd6-openshift-default-78d46d7dcc-dg79z" Nov 03 01:26:07.559592 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:07.559548 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50b59bdf-0949-473f-8849-bb5648c6cba1-istio-data\") pod \"e2e-inf-a-b6afd6-openshift-default-78d46d7dcc-dg79z\" (UID: \"50b59bdf-0949-473f-8849-bb5648c6cba1\") " pod="openshift-ingress/e2e-inf-a-b6afd6-openshift-default-78d46d7dcc-dg79z" Nov 03 01:26:07.559592 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:07.559567 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqzq2\" (UniqueName: \"kubernetes.io/projected/50b59bdf-0949-473f-8849-bb5648c6cba1-kube-api-access-gqzq2\") pod \"e2e-inf-a-b6afd6-openshift-default-78d46d7dcc-dg79z\" (UID: \"50b59bdf-0949-473f-8849-bb5648c6cba1\") " pod="openshift-ingress/e2e-inf-a-b6afd6-openshift-default-78d46d7dcc-dg79z" Nov 03 01:26:07.559807 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:07.559634 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50b59bdf-0949-473f-8849-bb5648c6cba1-credential-socket\") pod \"e2e-inf-a-b6afd6-openshift-default-78d46d7dcc-dg79z\" (UID: \"50b59bdf-0949-473f-8849-bb5648c6cba1\") " pod="openshift-ingress/e2e-inf-a-b6afd6-openshift-default-78d46d7dcc-dg79z" Nov 03 01:26:07.559807 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:07.559674 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50b59bdf-0949-473f-8849-bb5648c6cba1-istiod-ca-cert\") pod \"e2e-inf-a-b6afd6-openshift-default-78d46d7dcc-dg79z\" (UID: \"50b59bdf-0949-473f-8849-bb5648c6cba1\") " pod="openshift-ingress/e2e-inf-a-b6afd6-openshift-default-78d46d7dcc-dg79z" Nov 03 01:26:07.559988 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:07.559964 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50b59bdf-0949-473f-8849-bb5648c6cba1-workload-socket\") pod \"e2e-inf-a-b6afd6-openshift-default-78d46d7dcc-dg79z\" (UID: \"50b59bdf-0949-473f-8849-bb5648c6cba1\") " pod="openshift-ingress/e2e-inf-a-b6afd6-openshift-default-78d46d7dcc-dg79z" Nov 03 01:26:07.561525 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:07.561504 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50b59bdf-0949-473f-8849-bb5648c6cba1-istio-envoy\") pod \"e2e-inf-a-b6afd6-openshift-default-78d46d7dcc-dg79z\" (UID: \"50b59bdf-0949-473f-8849-bb5648c6cba1\") " pod="openshift-ingress/e2e-inf-a-b6afd6-openshift-default-78d46d7dcc-dg79z" Nov 03 01:26:07.561872 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:07.561854 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50b59bdf-0949-473f-8849-bb5648c6cba1-istio-podinfo\") pod \"e2e-inf-a-b6afd6-openshift-default-78d46d7dcc-dg79z\" (UID: \"50b59bdf-0949-473f-8849-bb5648c6cba1\") " pod="openshift-ingress/e2e-inf-a-b6afd6-openshift-default-78d46d7dcc-dg79z" Nov 03 01:26:07.567564 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:07.567535 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50b59bdf-0949-473f-8849-bb5648c6cba1-istio-token\") pod \"e2e-inf-a-b6afd6-openshift-default-78d46d7dcc-dg79z\" (UID: \"50b59bdf-0949-473f-8849-bb5648c6cba1\") " pod="openshift-ingress/e2e-inf-a-b6afd6-openshift-default-78d46d7dcc-dg79z" Nov 03 01:26:07.567679 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:07.567663 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqzq2\" (UniqueName: \"kubernetes.io/projected/50b59bdf-0949-473f-8849-bb5648c6cba1-kube-api-access-gqzq2\") pod \"e2e-inf-a-b6afd6-openshift-default-78d46d7dcc-dg79z\" (UID: \"50b59bdf-0949-473f-8849-bb5648c6cba1\") " pod="openshift-ingress/e2e-inf-a-b6afd6-openshift-default-78d46d7dcc-dg79z" Nov 03 01:26:07.633294 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:07.633246 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b6afd6-openshift-default-78d46d7dcc-dg79z" Nov 03 01:26:07.982585 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:07.982496 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d33fa6fe-3a61-4cfc-ab8a-05186c69266f" path="/var/lib/kubelet/pods/d33fa6fe-3a61-4cfc-ab8a-05186c69266f/volumes" Nov 03 01:26:08.004799 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:08.002862 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-b6afd6-openshift-default-78d46d7dcc-dg79z"] Nov 03 01:26:08.036520 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:08.006168 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:26:08.036520 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:08.006235 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:26:08.036520 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:08.006289 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:26:08.190224 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:08.190195 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b6afd6-openshift-default-78d46d7dcc-dg79z" event={"ID":"50b59bdf-0949-473f-8849-bb5648c6cba1","Type":"ContainerStarted","Data":"fc5052938d5a2a6ef4a1e69afadee14bd0057a16400deb40a8d42320f8784725"} Nov 03 01:26:08.191890 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:08.191866 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j52qt" Nov 03 01:26:08.191890 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:08.191881 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j52qt" event={"ID":"9417c149-c243-4ba7-9f66-7ad0688a9f78","Type":"ContainerDied","Data":"659390c7b7b414dd071cc1dd1d4293746d608cd2a0a6ebed598b119342b9a7af"} Nov 03 01:26:08.214092 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:08.191933 4421 scope.go:122] "RemoveContainer" containerID="c39c010b7c84d7a414eb844b763fa6e4134edc385464d389a6b7425498aca746" Nov 03 01:26:08.216609 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:08.216585 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j52qt"] Nov 03 01:26:08.230117 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:08.230091 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j52qt"] Nov 03 01:26:08.907881 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:08.900071 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-bktsz" Nov 03 01:26:08.951420 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:08.951364 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-4b17ac-5867b8956f-tlz2b"] Nov 03 01:26:08.952783 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:08.952758 4421 cpu_manager.go:414] "containerMap: removing container" podUID="85cec606-e8b7-4ce1-8af2-4afb303268f9" containerName="istio-proxy" Nov 03 01:26:08.952865 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:08.952785 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85cec606-e8b7-4ce1-8af2-4afb303268f9" containerName="istio-proxy" Nov 03 01:26:08.953626 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:08.953039 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="85cec606-e8b7-4ce1-8af2-4afb303268f9" containerName="istio-proxy" Nov 03 01:26:08.967927 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:08.967877 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-4b17ac-5867b8956f-tlz2b"] Nov 03 01:26:08.974431 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:08.967946 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-4b17ac-5867b8956f-tlz2b" Nov 03 01:26:08.974431 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:08.972094 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-4b17ac-dockercfg-ckrnv\"" Nov 03 01:26:08.974431 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:08.972251 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-4b17ac\"" Nov 03 01:26:09.015886 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:09.015850 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85cec606-e8b7-4ce1-8af2-4afb303268f9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85cec606-e8b7-4ce1-8af2-4afb303268f9-istio-data\") pod \"85cec606-e8b7-4ce1-8af2-4afb303268f9\" (UID: \"85cec606-e8b7-4ce1-8af2-4afb303268f9\") " Nov 03 01:26:09.016100 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:09.015929 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85cec606-e8b7-4ce1-8af2-4afb303268f9-kube-api-access-hr8mk\" (UniqueName: \"kubernetes.io/projected/85cec606-e8b7-4ce1-8af2-4afb303268f9-kube-api-access-hr8mk\") pod \"85cec606-e8b7-4ce1-8af2-4afb303268f9\" (UID: \"85cec606-e8b7-4ce1-8af2-4afb303268f9\") " Nov 03 01:26:09.016152 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:09.016112 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85cec606-e8b7-4ce1-8af2-4afb303268f9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85cec606-e8b7-4ce1-8af2-4afb303268f9-workload-socket\") pod \"85cec606-e8b7-4ce1-8af2-4afb303268f9\" (UID: \"85cec606-e8b7-4ce1-8af2-4afb303268f9\") " Nov 03 01:26:09.016354 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:09.016334 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85cec606-e8b7-4ce1-8af2-4afb303268f9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85cec606-e8b7-4ce1-8af2-4afb303268f9-istio-envoy\") pod \"85cec606-e8b7-4ce1-8af2-4afb303268f9\" (UID: \"85cec606-e8b7-4ce1-8af2-4afb303268f9\") " Nov 03 01:26:09.016434 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:09.016380 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85cec606-e8b7-4ce1-8af2-4afb303268f9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85cec606-e8b7-4ce1-8af2-4afb303268f9-credential-socket\") pod \"85cec606-e8b7-4ce1-8af2-4afb303268f9\" (UID: \"85cec606-e8b7-4ce1-8af2-4afb303268f9\") " Nov 03 01:26:09.016525 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:09.016509 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85cec606-e8b7-4ce1-8af2-4afb303268f9-istio-token\" (UniqueName: \"kubernetes.io/projected/85cec606-e8b7-4ce1-8af2-4afb303268f9-istio-token\") pod \"85cec606-e8b7-4ce1-8af2-4afb303268f9\" (UID: \"85cec606-e8b7-4ce1-8af2-4afb303268f9\") " Nov 03 01:26:09.016602 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:09.016586 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85cec606-e8b7-4ce1-8af2-4afb303268f9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85cec606-e8b7-4ce1-8af2-4afb303268f9-workload-certs\") pod \"85cec606-e8b7-4ce1-8af2-4afb303268f9\" (UID: \"85cec606-e8b7-4ce1-8af2-4afb303268f9\") " Nov 03 01:26:09.016738 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:09.016722 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85cec606-e8b7-4ce1-8af2-4afb303268f9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85cec606-e8b7-4ce1-8af2-4afb303268f9-istiod-ca-cert\") pod \"85cec606-e8b7-4ce1-8af2-4afb303268f9\" (UID: \"85cec606-e8b7-4ce1-8af2-4afb303268f9\") " Nov 03 01:26:09.016792 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:09.016771 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/85cec606-e8b7-4ce1-8af2-4afb303268f9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85cec606-e8b7-4ce1-8af2-4afb303268f9-istio-podinfo\") pod \"85cec606-e8b7-4ce1-8af2-4afb303268f9\" (UID: \"85cec606-e8b7-4ce1-8af2-4afb303268f9\") " Nov 03 01:26:09.017367 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:09.017334 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddc7cbfa-9cfd-469e-a53e-ef8175080492-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-4b17ac-5867b8956f-tlz2b\" (UID: \"ddc7cbfa-9cfd-469e-a53e-ef8175080492\") " pod="openshift-ingress/payload-processing-e2e-inf-a-4b17ac-5867b8956f-tlz2b" Nov 03 01:26:09.017667 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:09.017644 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-br84n\" (UniqueName: \"kubernetes.io/projected/ddc7cbfa-9cfd-469e-a53e-ef8175080492-kube-api-access-br84n\") pod \"payload-processing-e2e-inf-a-4b17ac-5867b8956f-tlz2b\" (UID: \"ddc7cbfa-9cfd-469e-a53e-ef8175080492\") " pod="openshift-ingress/payload-processing-e2e-inf-a-4b17ac-5867b8956f-tlz2b" Nov 03 01:26:09.119412 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:09.119329 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddc7cbfa-9cfd-469e-a53e-ef8175080492-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-4b17ac-5867b8956f-tlz2b\" (UID: \"ddc7cbfa-9cfd-469e-a53e-ef8175080492\") " pod="openshift-ingress/payload-processing-e2e-inf-a-4b17ac-5867b8956f-tlz2b" Nov 03 01:26:09.119647 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:09.119490 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-br84n\" (UniqueName: \"kubernetes.io/projected/ddc7cbfa-9cfd-469e-a53e-ef8175080492-kube-api-access-br84n\") pod \"payload-processing-e2e-inf-a-4b17ac-5867b8956f-tlz2b\" (UID: \"ddc7cbfa-9cfd-469e-a53e-ef8175080492\") " pod="openshift-ingress/payload-processing-e2e-inf-a-4b17ac-5867b8956f-tlz2b" Nov 03 01:26:09.120064 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:09.120038 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddc7cbfa-9cfd-469e-a53e-ef8175080492-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-4b17ac-5867b8956f-tlz2b\" (UID: \"ddc7cbfa-9cfd-469e-a53e-ef8175080492\") " pod="openshift-ingress/payload-processing-e2e-inf-a-4b17ac-5867b8956f-tlz2b" Nov 03 01:26:09.127970 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:09.127945 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-br84n\" (UniqueName: \"kubernetes.io/projected/ddc7cbfa-9cfd-469e-a53e-ef8175080492-kube-api-access-br84n\") pod \"payload-processing-e2e-inf-a-4b17ac-5867b8956f-tlz2b\" (UID: \"ddc7cbfa-9cfd-469e-a53e-ef8175080492\") " pod="openshift-ingress/payload-processing-e2e-inf-a-4b17ac-5867b8956f-tlz2b" Nov 03 01:26:09.203481 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:09.203397 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85cec606-e8b7-4ce1-8af2-4afb303268f9-credential-socket" pod "85cec606-e8b7-4ce1-8af2-4afb303268f9" (UID: "85cec606-e8b7-4ce1-8af2-4afb303268f9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:26:09.203960 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:09.203514 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85cec606-e8b7-4ce1-8af2-4afb303268f9-istio-data" pod "85cec606-e8b7-4ce1-8af2-4afb303268f9" (UID: "85cec606-e8b7-4ce1-8af2-4afb303268f9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:26:09.205429 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:09.205402 4421 generic.go:356] "Generic (PLEG): container finished" podID="85cec606-e8b7-4ce1-8af2-4afb303268f9" containerID="37f978db4958aa10297ee8efda2770ebcf37c14ad3910ad9e10f356af46ea6d3" exitCode=0 Nov 03 01:26:09.205560 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:09.205452 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-bktsz" Nov 03 01:26:09.205560 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:09.205491 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-bktsz" event={"ID":"85cec606-e8b7-4ce1-8af2-4afb303268f9","Type":"ContainerDied","Data":"37f978db4958aa10297ee8efda2770ebcf37c14ad3910ad9e10f356af46ea6d3"} Nov 03 01:26:09.205560 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:09.205551 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-bktsz" event={"ID":"85cec606-e8b7-4ce1-8af2-4afb303268f9","Type":"ContainerDied","Data":"63527a301ff23e236d638e8f1ddc83318675abd5ed169032f09326d220be516c"} Nov 03 01:26:09.215907 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:09.205583 4421 scope.go:122] "RemoveContainer" containerID="37f978db4958aa10297ee8efda2770ebcf37c14ad3910ad9e10f356af46ea6d3" Nov 03 01:26:09.215907 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:09.207766 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b6afd6-openshift-default-78d46d7dcc-dg79z" event={"ID":"50b59bdf-0949-473f-8849-bb5648c6cba1","Type":"ContainerStarted","Data":"6a0daece2fecdfdfc172298e4260507ca77c25680ec24beb166c78a92f218ce7"} Nov 03 01:26:09.220533 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:09.220471 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85cec606-e8b7-4ce1-8af2-4afb303268f9-workload-certs" pod "85cec606-e8b7-4ce1-8af2-4afb303268f9" (UID: "85cec606-e8b7-4ce1-8af2-4afb303268f9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:26:09.256782 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:09.221992 4421 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85cec606-e8b7-4ce1-8af2-4afb303268f9-istio-data\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:26:09.256782 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:09.222015 4421 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85cec606-e8b7-4ce1-8af2-4afb303268f9-credential-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:26:09.256782 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:09.222025 4421 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85cec606-e8b7-4ce1-8af2-4afb303268f9-workload-certs\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:26:09.256782 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:09.238379 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-b6afd6-openshift-default-78d46d7dcc-dg79z" podStartSLOduration=14.238358295 podStartE2EDuration="14.238358295s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:25:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:26:09.233548708 +0000 UTC m=+2711.404551807" watchObservedRunningTime="2026-11-03 01:26:09.238358295 +0000 UTC m=+2711.409361392" Nov 03 01:26:09.262259 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:09.261952 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85cec606-e8b7-4ce1-8af2-4afb303268f9-workload-socket" pod "85cec606-e8b7-4ce1-8af2-4afb303268f9" (UID: "85cec606-e8b7-4ce1-8af2-4afb303268f9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:26:09.262259 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:09.261971 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85cec606-e8b7-4ce1-8af2-4afb303268f9-istio-token" pod "85cec606-e8b7-4ce1-8af2-4afb303268f9" (UID: "85cec606-e8b7-4ce1-8af2-4afb303268f9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:26:09.262259 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:09.261980 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/85cec606-e8b7-4ce1-8af2-4afb303268f9-istio-podinfo" pod "85cec606-e8b7-4ce1-8af2-4afb303268f9" (UID: "85cec606-e8b7-4ce1-8af2-4afb303268f9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:26:09.275095 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:09.275056 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85cec606-e8b7-4ce1-8af2-4afb303268f9-istio-envoy" pod "85cec606-e8b7-4ce1-8af2-4afb303268f9" (UID: "85cec606-e8b7-4ce1-8af2-4afb303268f9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:26:09.275673 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:09.275650 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85cec606-e8b7-4ce1-8af2-4afb303268f9-kube-api-access-hr8mk" pod "85cec606-e8b7-4ce1-8af2-4afb303268f9" (UID: "85cec606-e8b7-4ce1-8af2-4afb303268f9"). InnerVolumeSpecName "kube-api-access-hr8mk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:26:09.283786 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:09.283753 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85cec606-e8b7-4ce1-8af2-4afb303268f9-istiod-ca-cert" pod "85cec606-e8b7-4ce1-8af2-4afb303268f9" (UID: "85cec606-e8b7-4ce1-8af2-4afb303268f9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:26:09.324304 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:09.324242 4421 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85cec606-e8b7-4ce1-8af2-4afb303268f9-istio-envoy\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:26:09.324304 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:09.324268 4421 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85cec606-e8b7-4ce1-8af2-4afb303268f9-istio-token\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:26:09.324304 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:09.324296 4421 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85cec606-e8b7-4ce1-8af2-4afb303268f9-istiod-ca-cert\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:26:09.324304 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:09.324307 4421 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85cec606-e8b7-4ce1-8af2-4afb303268f9-istio-podinfo\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:26:09.325011 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:09.324319 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hr8mk\" (UniqueName: \"kubernetes.io/projected/85cec606-e8b7-4ce1-8af2-4afb303268f9-kube-api-access-hr8mk\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:26:09.325011 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:09.324334 4421 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85cec606-e8b7-4ce1-8af2-4afb303268f9-workload-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:26:09.328164 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:09.328133 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-4b17ac-5867b8956f-tlz2b" Nov 03 01:26:09.386562 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:09.385738 4421 scope.go:122] "RemoveContainer" containerID="37f978db4958aa10297ee8efda2770ebcf37c14ad3910ad9e10f356af46ea6d3" Nov 03 01:26:09.386562 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:26:09.386151 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37f978db4958aa10297ee8efda2770ebcf37c14ad3910ad9e10f356af46ea6d3\": container with ID starting with 37f978db4958aa10297ee8efda2770ebcf37c14ad3910ad9e10f356af46ea6d3 not found: ID does not exist" containerID="37f978db4958aa10297ee8efda2770ebcf37c14ad3910ad9e10f356af46ea6d3" Nov 03 01:26:09.386562 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:09.386186 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37f978db4958aa10297ee8efda2770ebcf37c14ad3910ad9e10f356af46ea6d3"} err="failed to get container status \"37f978db4958aa10297ee8efda2770ebcf37c14ad3910ad9e10f356af46ea6d3\": rpc error: code = NotFound desc = could not find container \"37f978db4958aa10297ee8efda2770ebcf37c14ad3910ad9e10f356af46ea6d3\": container with ID starting with 37f978db4958aa10297ee8efda2770ebcf37c14ad3910ad9e10f356af46ea6d3 not found: ID does not exist" Nov 03 01:26:09.575164 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:09.575124 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-bktsz"] Nov 03 01:26:09.585332 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:09.580545 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-bktsz"] Nov 03 01:26:09.650825 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:09.633751 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-b6afd6-openshift-default-78d46d7dcc-dg79z" Nov 03 01:26:09.650825 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:09.641567 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-b6afd6-openshift-default-78d46d7dcc-dg79z" Nov 03 01:26:09.844638 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:09.843186 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-4b17ac-5867b8956f-tlz2b"] Nov 03 01:26:09.900395 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:26:09.900325 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddc7cbfa_9cfd_469e_a53e_ef8175080492.slice/crio-471f2da2ab7f071a2b791aae58c3d70ec1d2962b501974d0b571a6117b766b5a WatchSource:0}: Error finding container 471f2da2ab7f071a2b791aae58c3d70ec1d2962b501974d0b571a6117b766b5a: Status 404 returned error can't find the container with id 471f2da2ab7f071a2b791aae58c3d70ec1d2962b501974d0b571a6117b766b5a Nov 03 01:26:10.077822 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:10.077772 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85cec606-e8b7-4ce1-8af2-4afb303268f9" path="/var/lib/kubelet/pods/85cec606-e8b7-4ce1-8af2-4afb303268f9/volumes" Nov 03 01:26:10.082027 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:10.081991 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9417c149-c243-4ba7-9f66-7ad0688a9f78" path="/var/lib/kubelet/pods/9417c149-c243-4ba7-9f66-7ad0688a9f78/volumes" Nov 03 01:26:10.219433 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:10.219327 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-4b17ac-5867b8956f-tlz2b" event={"ID":"ddc7cbfa-9cfd-469e-a53e-ef8175080492","Type":"ContainerStarted","Data":"e5bb3932102fd1b9db0ec800b138a9620307e3534b26325245bf37c48c5c6065"} Nov 03 01:26:10.275093 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:10.219623 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-b6afd6-openshift-default-78d46d7dcc-dg79z" Nov 03 01:26:10.275093 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:10.219667 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-4b17ac-5867b8956f-tlz2b" Nov 03 01:26:10.275093 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:10.219685 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-4b17ac-5867b8956f-tlz2b" event={"ID":"ddc7cbfa-9cfd-469e-a53e-ef8175080492","Type":"ContainerStarted","Data":"471f2da2ab7f071a2b791aae58c3d70ec1d2962b501974d0b571a6117b766b5a"} Nov 03 01:26:10.275093 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:10.220568 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-b6afd6-openshift-default-78d46d7dcc-dg79z" Nov 03 01:26:10.275093 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:10.241820 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-4b17ac-5867b8956f-tlz2b" podStartSLOduration=11.241806708 podStartE2EDuration="11.241806708s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:25:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:26:10.235925447 +0000 UTC m=+2712.406928544" watchObservedRunningTime="2026-11-03 01:26:10.241806708 +0000 UTC m=+2712.412809804" Nov 03 01:26:11.032443 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:11.032405 4421 patch_prober.go:29] interesting pod/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-vhjlx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 01:26:11.032713 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:11.032474 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-vhjlx" podUID="1e2345c1-2a99-4b03-b078-eebe1f198589" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 01:26:11.815289 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:11.815244 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5c9e521-qx24h"] Nov 03 01:26:12.035429 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:12.035388 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5c9e521-qx24h" Nov 03 01:26:12.043296 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:12.043252 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:26:12.043749 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:12.043729 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6q4xp\"" Nov 03 01:26:12.046913 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:12.046888 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5c9e521-qx24h"] Nov 03 01:26:12.131887 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:12.131562 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2907a10d-55cc-4321-b4ec-169e4d8d0de2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d5c9e521-qx24h\" (UID: \"2907a10d-55cc-4321-b4ec-169e4d8d0de2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5c9e521-qx24h" Nov 03 01:26:12.131887 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:12.131768 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c24rm\" (UniqueName: \"kubernetes.io/projected/2907a10d-55cc-4321-b4ec-169e4d8d0de2-kube-api-access-c24rm\") pod \"maas-api-revoke-sub-d5c9e521-qx24h\" (UID: \"2907a10d-55cc-4321-b4ec-169e4d8d0de2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5c9e521-qx24h" Nov 03 01:26:12.132089 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:12.131915 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2907a10d-55cc-4321-b4ec-169e4d8d0de2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d5c9e521-qx24h\" (UID: \"2907a10d-55cc-4321-b4ec-169e4d8d0de2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5c9e521-qx24h" Nov 03 01:26:12.274592 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:12.235055 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c24rm\" (UniqueName: \"kubernetes.io/projected/2907a10d-55cc-4321-b4ec-169e4d8d0de2-kube-api-access-c24rm\") pod \"maas-api-revoke-sub-d5c9e521-qx24h\" (UID: \"2907a10d-55cc-4321-b4ec-169e4d8d0de2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5c9e521-qx24h" Nov 03 01:26:12.274592 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:12.235258 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2907a10d-55cc-4321-b4ec-169e4d8d0de2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d5c9e521-qx24h\" (UID: \"2907a10d-55cc-4321-b4ec-169e4d8d0de2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5c9e521-qx24h" Nov 03 01:26:12.274592 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:12.235624 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2907a10d-55cc-4321-b4ec-169e4d8d0de2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d5c9e521-qx24h\" (UID: \"2907a10d-55cc-4321-b4ec-169e4d8d0de2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5c9e521-qx24h" Nov 03 01:26:12.274592 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:12.248781 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2907a10d-55cc-4321-b4ec-169e4d8d0de2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d5c9e521-qx24h\" (UID: \"2907a10d-55cc-4321-b4ec-169e4d8d0de2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5c9e521-qx24h" Nov 03 01:26:12.323265 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:12.317743 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2907a10d-55cc-4321-b4ec-169e4d8d0de2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d5c9e521-qx24h\" (UID: \"2907a10d-55cc-4321-b4ec-169e4d8d0de2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5c9e521-qx24h" Nov 03 01:26:12.325429 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:12.325388 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c24rm\" (UniqueName: \"kubernetes.io/projected/2907a10d-55cc-4321-b4ec-169e4d8d0de2-kube-api-access-c24rm\") pod \"maas-api-revoke-sub-d5c9e521-qx24h\" (UID: \"2907a10d-55cc-4321-b4ec-169e4d8d0de2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5c9e521-qx24h" Nov 03 01:26:12.562080 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:12.562047 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5c9e521-qx24h" Nov 03 01:26:12.861061 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:26:12.861022 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2907a10d_55cc_4321_b4ec_169e4d8d0de2.slice/crio-265b842f14e88141328e539c4e5c86f8b3bde358b61d54e91dec7cdcb81500fe WatchSource:0}: Error finding container 265b842f14e88141328e539c4e5c86f8b3bde358b61d54e91dec7cdcb81500fe: Status 404 returned error can't find the container with id 265b842f14e88141328e539c4e5c86f8b3bde358b61d54e91dec7cdcb81500fe Nov 03 01:26:12.862049 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:12.862018 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5c9e521-qx24h"] Nov 03 01:26:13.052354 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:13.052317 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-b6afd6-7db7d566cf-rkxmr"] Nov 03 01:26:13.218619 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:13.218572 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-b6afd6-7db7d566cf-rkxmr"] Nov 03 01:26:13.219042 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:13.219023 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b6afd6-7db7d566cf-rkxmr" Nov 03 01:26:13.237790 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:13.237762 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-b6afd6\"" Nov 03 01:26:13.407352 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:13.244900 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-b6afd6-dockercfg-bprnb\"" Nov 03 01:26:13.407352 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:13.283093 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18de1ce4-36e1-4a3c-90cc-ecbac5dcfc80-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-b6afd6-7db7d566cf-rkxmr\" (UID: \"18de1ce4-36e1-4a3c-90cc-ecbac5dcfc80\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b6afd6-7db7d566cf-rkxmr" Nov 03 01:26:13.407352 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:13.396634 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlmlw\" (UniqueName: \"kubernetes.io/projected/18de1ce4-36e1-4a3c-90cc-ecbac5dcfc80-kube-api-access-wlmlw\") pod \"payload-pre-processing-e2e-inf-a-b6afd6-7db7d566cf-rkxmr\" (UID: \"18de1ce4-36e1-4a3c-90cc-ecbac5dcfc80\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b6afd6-7db7d566cf-rkxmr" Nov 03 01:26:13.442716 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:13.442627 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5c9e521-qx24h" event={"ID":"2907a10d-55cc-4321-b4ec-169e4d8d0de2","Type":"ContainerStarted","Data":"265b842f14e88141328e539c4e5c86f8b3bde358b61d54e91dec7cdcb81500fe"} Nov 03 01:26:13.523297 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:13.515761 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlmlw\" (UniqueName: \"kubernetes.io/projected/18de1ce4-36e1-4a3c-90cc-ecbac5dcfc80-kube-api-access-wlmlw\") pod \"payload-pre-processing-e2e-inf-a-b6afd6-7db7d566cf-rkxmr\" (UID: \"18de1ce4-36e1-4a3c-90cc-ecbac5dcfc80\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b6afd6-7db7d566cf-rkxmr" Nov 03 01:26:13.523297 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:13.516327 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18de1ce4-36e1-4a3c-90cc-ecbac5dcfc80-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-b6afd6-7db7d566cf-rkxmr\" (UID: \"18de1ce4-36e1-4a3c-90cc-ecbac5dcfc80\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b6afd6-7db7d566cf-rkxmr" Nov 03 01:26:13.523297 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:13.517413 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18de1ce4-36e1-4a3c-90cc-ecbac5dcfc80-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-b6afd6-7db7d566cf-rkxmr\" (UID: \"18de1ce4-36e1-4a3c-90cc-ecbac5dcfc80\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b6afd6-7db7d566cf-rkxmr" Nov 03 01:26:13.565850 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:13.540645 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlmlw\" (UniqueName: \"kubernetes.io/projected/18de1ce4-36e1-4a3c-90cc-ecbac5dcfc80-kube-api-access-wlmlw\") pod \"payload-pre-processing-e2e-inf-a-b6afd6-7db7d566cf-rkxmr\" (UID: \"18de1ce4-36e1-4a3c-90cc-ecbac5dcfc80\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b6afd6-7db7d566cf-rkxmr" Nov 03 01:26:13.565850 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:13.541147 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b6afd6-7db7d566cf-rkxmr" Nov 03 01:26:14.065807 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:14.065761 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-b6afd6-7db7d566cf-rkxmr"] Nov 03 01:26:14.454571 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:14.454461 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5c9e521-qx24h" event={"ID":"2907a10d-55cc-4321-b4ec-169e4d8d0de2","Type":"ContainerStarted","Data":"dbd348316291fbaa1f21a8d30694af3833bd10a53501410d0953973375476206"} Nov 03 01:26:14.456355 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:14.456323 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b6afd6-7db7d566cf-rkxmr" event={"ID":"18de1ce4-36e1-4a3c-90cc-ecbac5dcfc80","Type":"ContainerStarted","Data":"8d6251233781c5d4701a829135afab2c6ca67341b7bd3e30d11ea3aa253a8f44"} Nov 03 01:26:14.473563 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:14.473503 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5c9e521-qx24h" podStartSLOduration=3.473488003 podStartE2EDuration="3.473488003s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:26:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:26:14.470766592 +0000 UTC m=+2716.641769689" watchObservedRunningTime="2026-11-03 01:26:14.473488003 +0000 UTC m=+2716.644491098" Nov 03 01:26:14.688019 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:26:14.687978 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 01:26:15.467377 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:15.467338 4421 generic.go:356] "Generic (PLEG): container finished" podID="2907a10d-55cc-4321-b4ec-169e4d8d0de2" containerID="dbd348316291fbaa1f21a8d30694af3833bd10a53501410d0953973375476206" exitCode=0 Nov 03 01:26:15.467377 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:15.467373 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5c9e521-qx24h" event={"ID":"2907a10d-55cc-4321-b4ec-169e4d8d0de2","Type":"ContainerDied","Data":"dbd348316291fbaa1f21a8d30694af3833bd10a53501410d0953973375476206"} Nov 03 01:26:15.469322 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:15.469295 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b6afd6-7db7d566cf-rkxmr" event={"ID":"18de1ce4-36e1-4a3c-90cc-ecbac5dcfc80","Type":"ContainerStarted","Data":"0b370d615a26cca3f7c32796de54eb12c2a3fb3fee464e67cc8c0c5acb0379a8"} Nov 03 01:26:15.469447 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:15.469360 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b6afd6-7db7d566cf-rkxmr" Nov 03 01:26:15.502884 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:15.502823 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b6afd6-7db7d566cf-rkxmr" podStartSLOduration=2.50280494 podStartE2EDuration="2.50280494s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:26:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:26:15.498744557 +0000 UTC m=+2717.669747655" watchObservedRunningTime="2026-11-03 01:26:15.50280494 +0000 UTC m=+2717.673808037" Nov 03 01:26:16.027709 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:16.027669 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a762f455-v88l9"] Nov 03 01:26:16.032987 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:16.032954 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a762f455-v88l9"] Nov 03 01:26:16.699519 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:16.699493 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5c9e521-qx24h" Nov 03 01:26:16.874436 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:16.874395 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2907a10d-55cc-4321-b4ec-169e4d8d0de2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2907a10d-55cc-4321-b4ec-169e4d8d0de2-maas-api-openshift-ca\") pod \"2907a10d-55cc-4321-b4ec-169e4d8d0de2\" (UID: \"2907a10d-55cc-4321-b4ec-169e4d8d0de2\") " Nov 03 01:26:16.874710 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:16.874655 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2907a10d-55cc-4321-b4ec-169e4d8d0de2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2907a10d-55cc-4321-b4ec-169e4d8d0de2-maas-api-xks-ca\") pod \"2907a10d-55cc-4321-b4ec-169e4d8d0de2\" (UID: \"2907a10d-55cc-4321-b4ec-169e4d8d0de2\") " Nov 03 01:26:16.874823 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:16.874803 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2907a10d-55cc-4321-b4ec-169e4d8d0de2-maas-api-openshift-ca" pod "2907a10d-55cc-4321-b4ec-169e4d8d0de2" (UID: "2907a10d-55cc-4321-b4ec-169e4d8d0de2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:26:16.874887 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:16.874869 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2907a10d-55cc-4321-b4ec-169e4d8d0de2-kube-api-access-c24rm\" (UniqueName: \"kubernetes.io/projected/2907a10d-55cc-4321-b4ec-169e4d8d0de2-kube-api-access-c24rm\") pod \"2907a10d-55cc-4321-b4ec-169e4d8d0de2\" (UID: \"2907a10d-55cc-4321-b4ec-169e4d8d0de2\") " Nov 03 01:26:16.875661 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:16.875633 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2907a10d-55cc-4321-b4ec-169e4d8d0de2-maas-api-openshift-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:26:16.876982 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:16.876952 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2907a10d-55cc-4321-b4ec-169e4d8d0de2-maas-api-xks-ca" pod "2907a10d-55cc-4321-b4ec-169e4d8d0de2" (UID: "2907a10d-55cc-4321-b4ec-169e4d8d0de2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:26:16.877495 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:16.877462 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2907a10d-55cc-4321-b4ec-169e4d8d0de2-kube-api-access-c24rm" pod "2907a10d-55cc-4321-b4ec-169e4d8d0de2" (UID: "2907a10d-55cc-4321-b4ec-169e4d8d0de2"). InnerVolumeSpecName "kube-api-access-c24rm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:26:16.985794 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:16.984554 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2907a10d-55cc-4321-b4ec-169e4d8d0de2-maas-api-xks-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:26:16.985794 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:16.984596 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c24rm\" (UniqueName: \"kubernetes.io/projected/2907a10d-55cc-4321-b4ec-169e4d8d0de2-kube-api-access-c24rm\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:26:17.498361 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:17.495582 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5c9e521-qx24h" event={"ID":"2907a10d-55cc-4321-b4ec-169e4d8d0de2","Type":"ContainerDied","Data":"265b842f14e88141328e539c4e5c86f8b3bde358b61d54e91dec7cdcb81500fe"} Nov 03 01:26:17.498361 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:17.495653 4421 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="265b842f14e88141328e539c4e5c86f8b3bde358b61d54e91dec7cdcb81500fe" Nov 03 01:26:17.498361 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:17.495729 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5c9e521-qx24h" Nov 03 01:26:18.226311 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:18.226252 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf375ec3-3e92-40ac-a2c6-6ec506ce301d" path="/var/lib/kubelet/pods/cf375ec3-3e92-40ac-a2c6-6ec506ce301d/volumes" Nov 03 01:26:20.613829 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:20.613792 4421 generic.go:356] "Generic (PLEG): container finished" podID="1e2345c1-2a99-4b03-b078-eebe1f198589" containerID="751bf328470f6f5ab404fb70bf67aa7c9c20ff91cd278e623b26099e33f00960" exitCode=0 Nov 03 01:26:20.614330 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:20.613912 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-vhjlx" event={"ID":"1e2345c1-2a99-4b03-b078-eebe1f198589","Type":"ContainerDied","Data":"751bf328470f6f5ab404fb70bf67aa7c9c20ff91cd278e623b26099e33f00960"} Nov 03 01:26:20.614937 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:20.614422 4421 scope.go:122] "RemoveContainer" containerID="751bf328470f6f5ab404fb70bf67aa7c9c20ff91cd278e623b26099e33f00960" Nov 03 01:26:20.616726 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:20.616692 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:26:20.617531 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:20.616780 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:26:20.617531 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:20.616829 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:26:21.262294 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:21.258639 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-4b17ac-5867b8956f-tlz2b" Nov 03 01:26:21.642468 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:21.642385 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-vhjlx" event={"ID":"1e2345c1-2a99-4b03-b078-eebe1f198589","Type":"ContainerStarted","Data":"69702dff6eed0fd1f37433181868ea6b682951c24343e3e04dc3116e146c03c3"} Nov 03 01:26:22.490920 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:22.490878 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-vhjlx" Nov 03 01:26:22.492262 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:22.492236 4421 patch_prober.go:29] interesting pod/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-vhjlx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.54:15021/healthz/ready\": dial tcp 10.134.1.54:15021: connect: connection refused" start-of-body= Nov 03 01:26:22.492380 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:22.492335 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-vhjlx" podUID="1e2345c1-2a99-4b03-b078-eebe1f198589" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.54:15021/healthz/ready\": dial tcp 10.134.1.54:15021: connect: connection refused" Nov 03 01:26:23.494296 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:23.493648 4421 patch_prober.go:29] interesting pod/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-vhjlx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.54:15021/healthz/ready\": dial tcp 10.134.1.54:15021: connect: connection refused" start-of-body= Nov 03 01:26:23.494296 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:23.493726 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-vhjlx" podUID="1e2345c1-2a99-4b03-b078-eebe1f198589" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.54:15021/healthz/ready\": dial tcp 10.134.1.54:15021: connect: connection refused" Nov 03 01:26:24.051311 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:24.029327 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-048e6381-wwgl5"] Nov 03 01:26:24.051311 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:24.030722 4421 cpu_manager.go:414] "containerMap: removing container" podUID="2907a10d-55cc-4321-b4ec-169e4d8d0de2" containerName="revoke-keys" Nov 03 01:26:24.051311 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:24.030744 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2907a10d-55cc-4321-b4ec-169e4d8d0de2" containerName="revoke-keys" Nov 03 01:26:24.051311 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:24.031005 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="2907a10d-55cc-4321-b4ec-169e4d8d0de2" containerName="revoke-keys" Nov 03 01:26:24.084109 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:24.084075 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-048e6381-wwgl5"] Nov 03 01:26:24.084361 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:24.084178 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-048e6381-wwgl5" Nov 03 01:26:24.087896 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:24.087854 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:26:24.088051 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:24.087995 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6q4xp\"" Nov 03 01:26:24.231409 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:24.231308 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb88a728-a121-47b2-84a0-a5ef8ba91de6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-048e6381-wwgl5\" (UID: \"fb88a728-a121-47b2-84a0-a5ef8ba91de6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-048e6381-wwgl5" Nov 03 01:26:24.231629 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:24.231493 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb88a728-a121-47b2-84a0-a5ef8ba91de6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-048e6381-wwgl5\" (UID: \"fb88a728-a121-47b2-84a0-a5ef8ba91de6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-048e6381-wwgl5" Nov 03 01:26:24.231700 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:24.231626 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x98qg\" (UniqueName: \"kubernetes.io/projected/fb88a728-a121-47b2-84a0-a5ef8ba91de6-kube-api-access-x98qg\") pod \"maas-api-revoke-sub-048e6381-wwgl5\" (UID: \"fb88a728-a121-47b2-84a0-a5ef8ba91de6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-048e6381-wwgl5" Nov 03 01:26:24.356354 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:24.355578 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb88a728-a121-47b2-84a0-a5ef8ba91de6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-048e6381-wwgl5\" (UID: \"fb88a728-a121-47b2-84a0-a5ef8ba91de6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-048e6381-wwgl5" Nov 03 01:26:24.356354 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:24.355746 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb88a728-a121-47b2-84a0-a5ef8ba91de6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-048e6381-wwgl5\" (UID: \"fb88a728-a121-47b2-84a0-a5ef8ba91de6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-048e6381-wwgl5" Nov 03 01:26:24.356354 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:24.355925 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x98qg\" (UniqueName: \"kubernetes.io/projected/fb88a728-a121-47b2-84a0-a5ef8ba91de6-kube-api-access-x98qg\") pod \"maas-api-revoke-sub-048e6381-wwgl5\" (UID: \"fb88a728-a121-47b2-84a0-a5ef8ba91de6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-048e6381-wwgl5" Nov 03 01:26:24.370295 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:24.367690 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb88a728-a121-47b2-84a0-a5ef8ba91de6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-048e6381-wwgl5\" (UID: \"fb88a728-a121-47b2-84a0-a5ef8ba91de6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-048e6381-wwgl5" Nov 03 01:26:24.436296 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:24.428161 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb88a728-a121-47b2-84a0-a5ef8ba91de6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-048e6381-wwgl5\" (UID: \"fb88a728-a121-47b2-84a0-a5ef8ba91de6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-048e6381-wwgl5" Nov 03 01:26:24.445297 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:24.442139 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x98qg\" (UniqueName: \"kubernetes.io/projected/fb88a728-a121-47b2-84a0-a5ef8ba91de6-kube-api-access-x98qg\") pod \"maas-api-revoke-sub-048e6381-wwgl5\" (UID: \"fb88a728-a121-47b2-84a0-a5ef8ba91de6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-048e6381-wwgl5" Nov 03 01:26:24.491296 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:24.490655 4421 patch_prober.go:29] interesting pod/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-vhjlx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.54:15021/healthz/ready\": dial tcp 10.134.1.54:15021: connect: connection refused" start-of-body= Nov 03 01:26:24.491296 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:24.490716 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-vhjlx" podUID="1e2345c1-2a99-4b03-b078-eebe1f198589" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.54:15021/healthz/ready\": dial tcp 10.134.1.54:15021: connect: connection refused" Nov 03 01:26:24.734388 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:24.724120 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-048e6381-wwgl5" Nov 03 01:26:24.990924 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:24.990837 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-vhjlx" Nov 03 01:26:25.053101 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:25.053065 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-048e6381-wwgl5"] Nov 03 01:26:25.053101 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:26:25.053085 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb88a728_a121_47b2_84a0_a5ef8ba91de6.slice/crio-170b2cf7948356e996c834fc88507e2d413c900bec8946f8df92c0ba34eb880a WatchSource:0}: Error finding container 170b2cf7948356e996c834fc88507e2d413c900bec8946f8df92c0ba34eb880a: Status 404 returned error can't find the container with id 170b2cf7948356e996c834fc88507e2d413c900bec8946f8df92c0ba34eb880a Nov 03 01:26:25.491881 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:25.491841 4421 patch_prober.go:29] interesting pod/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-vhjlx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.54:15021/healthz/ready\": dial tcp 10.134.1.54:15021: connect: connection refused" start-of-body= Nov 03 01:26:25.492035 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:25.491909 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-vhjlx" podUID="1e2345c1-2a99-4b03-b078-eebe1f198589" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.54:15021/healthz/ready\": dial tcp 10.134.1.54:15021: connect: connection refused" Nov 03 01:26:25.756974 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:25.756937 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-048e6381-wwgl5" event={"ID":"fb88a728-a121-47b2-84a0-a5ef8ba91de6","Type":"ContainerStarted","Data":"d92b00f9bb0ec3513d94e150a12836f44e0fd2aadf5635980f4bf6f7b7117ec8"} Nov 03 01:26:25.757548 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:25.757032 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-048e6381-wwgl5" event={"ID":"fb88a728-a121-47b2-84a0-a5ef8ba91de6","Type":"ContainerStarted","Data":"170b2cf7948356e996c834fc88507e2d413c900bec8946f8df92c0ba34eb880a"} Nov 03 01:26:25.774222 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:25.774158 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-048e6381-wwgl5" podStartSLOduration=2.7741395129999997 podStartE2EDuration="2.774139513s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:26:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:26:25.772779531 +0000 UTC m=+2727.943782626" watchObservedRunningTime="2026-11-03 01:26:25.774139513 +0000 UTC m=+2727.945142610" Nov 03 01:26:26.518755 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:26.518660 4421 patch_prober.go:29] interesting pod/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-vhjlx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.54:15021/healthz/ready\": dial tcp 10.134.1.54:15021: connect: connection refused" start-of-body= Nov 03 01:26:26.520893 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:26.520856 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-vhjlx" podUID="1e2345c1-2a99-4b03-b078-eebe1f198589" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.54:15021/healthz/ready\": dial tcp 10.134.1.54:15021: connect: connection refused" Nov 03 01:26:26.542019 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:26.540566 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b6afd6-7db7d566cf-rkxmr" Nov 03 01:26:26.766806 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:26.766770 4421 generic.go:356] "Generic (PLEG): container finished" podID="fb88a728-a121-47b2-84a0-a5ef8ba91de6" containerID="d92b00f9bb0ec3513d94e150a12836f44e0fd2aadf5635980f4bf6f7b7117ec8" exitCode=0 Nov 03 01:26:26.767223 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:26.767014 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-048e6381-wwgl5" event={"ID":"fb88a728-a121-47b2-84a0-a5ef8ba91de6","Type":"ContainerDied","Data":"d92b00f9bb0ec3513d94e150a12836f44e0fd2aadf5635980f4bf6f7b7117ec8"} Nov 03 01:26:27.491105 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:27.491070 4421 patch_prober.go:29] interesting pod/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-vhjlx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.54:15021/healthz/ready\": dial tcp 10.134.1.54:15021: connect: connection refused" start-of-body= Nov 03 01:26:27.491340 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:27.491131 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-vhjlx" podUID="1e2345c1-2a99-4b03-b078-eebe1f198589" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.54:15021/healthz/ready\": dial tcp 10.134.1.54:15021: connect: connection refused" Nov 03 01:26:28.383604 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:28.383571 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-048e6381-wwgl5" Nov 03 01:26:28.425498 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:28.424975 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb88a728-a121-47b2-84a0-a5ef8ba91de6-kube-api-access-x98qg\" (UniqueName: \"kubernetes.io/projected/fb88a728-a121-47b2-84a0-a5ef8ba91de6-kube-api-access-x98qg\") pod \"fb88a728-a121-47b2-84a0-a5ef8ba91de6\" (UID: \"fb88a728-a121-47b2-84a0-a5ef8ba91de6\") " Nov 03 01:26:28.425905 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:28.425881 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb88a728-a121-47b2-84a0-a5ef8ba91de6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb88a728-a121-47b2-84a0-a5ef8ba91de6-maas-api-xks-ca\") pod \"fb88a728-a121-47b2-84a0-a5ef8ba91de6\" (UID: \"fb88a728-a121-47b2-84a0-a5ef8ba91de6\") " Nov 03 01:26:28.426542 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:28.426511 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb88a728-a121-47b2-84a0-a5ef8ba91de6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb88a728-a121-47b2-84a0-a5ef8ba91de6-maas-api-openshift-ca\") pod \"fb88a728-a121-47b2-84a0-a5ef8ba91de6\" (UID: \"fb88a728-a121-47b2-84a0-a5ef8ba91de6\") " Nov 03 01:26:28.428137 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:28.426999 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb88a728-a121-47b2-84a0-a5ef8ba91de6-maas-api-openshift-ca" pod "fb88a728-a121-47b2-84a0-a5ef8ba91de6" (UID: "fb88a728-a121-47b2-84a0-a5ef8ba91de6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:26:28.469086 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:28.469024 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb88a728-a121-47b2-84a0-a5ef8ba91de6-maas-api-xks-ca" pod "fb88a728-a121-47b2-84a0-a5ef8ba91de6" (UID: "fb88a728-a121-47b2-84a0-a5ef8ba91de6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:26:28.506632 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:28.506569 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb88a728-a121-47b2-84a0-a5ef8ba91de6-kube-api-access-x98qg" pod "fb88a728-a121-47b2-84a0-a5ef8ba91de6" (UID: "fb88a728-a121-47b2-84a0-a5ef8ba91de6"). InnerVolumeSpecName "kube-api-access-x98qg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:26:28.522895 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:28.520656 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-vhjlx" Nov 03 01:26:28.522895 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:28.522006 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-vhjlx" Nov 03 01:26:28.530307 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:28.530263 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x98qg\" (UniqueName: \"kubernetes.io/projected/fb88a728-a121-47b2-84a0-a5ef8ba91de6-kube-api-access-x98qg\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:26:28.530307 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:28.530301 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb88a728-a121-47b2-84a0-a5ef8ba91de6-maas-api-xks-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:26:28.530307 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:28.530312 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb88a728-a121-47b2-84a0-a5ef8ba91de6-maas-api-openshift-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:26:28.786134 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:28.786108 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-048e6381-wwgl5" Nov 03 01:26:28.786325 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:28.786140 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-048e6381-wwgl5" event={"ID":"fb88a728-a121-47b2-84a0-a5ef8ba91de6","Type":"ContainerDied","Data":"170b2cf7948356e996c834fc88507e2d413c900bec8946f8df92c0ba34eb880a"} Nov 03 01:26:28.786325 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:28.786215 4421 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="170b2cf7948356e996c834fc88507e2d413c900bec8946f8df92c0ba34eb880a" Nov 03 01:26:34.668334 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:34.668289 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c67e94fe-m5ldg"] Nov 03 01:26:34.669444 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:34.669412 4421 cpu_manager.go:414] "containerMap: removing container" podUID="fb88a728-a121-47b2-84a0-a5ef8ba91de6" containerName="revoke-keys" Nov 03 01:26:34.669444 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:34.669432 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb88a728-a121-47b2-84a0-a5ef8ba91de6" containerName="revoke-keys" Nov 03 01:26:34.669650 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:34.669602 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb88a728-a121-47b2-84a0-a5ef8ba91de6" containerName="revoke-keys" Nov 03 01:26:34.673804 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:34.673783 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c67e94fe-m5ldg" Nov 03 01:26:34.684224 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:34.684196 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:26:34.691605 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:34.691573 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6q4xp\"" Nov 03 01:26:34.692051 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:34.692024 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c67e94fe-m5ldg"] Nov 03 01:26:34.749401 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:34.749357 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqfkg\" (UniqueName: \"kubernetes.io/projected/6e1fd99f-0a01-4ae3-8d80-1788ae33e332-kube-api-access-gqfkg\") pod \"maas-api-revoke-sub-c67e94fe-m5ldg\" (UID: \"6e1fd99f-0a01-4ae3-8d80-1788ae33e332\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c67e94fe-m5ldg" Nov 03 01:26:34.749649 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:34.749437 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e1fd99f-0a01-4ae3-8d80-1788ae33e332-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c67e94fe-m5ldg\" (UID: \"6e1fd99f-0a01-4ae3-8d80-1788ae33e332\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c67e94fe-m5ldg" Nov 03 01:26:34.749739 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:34.749707 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e1fd99f-0a01-4ae3-8d80-1788ae33e332-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c67e94fe-m5ldg\" (UID: \"6e1fd99f-0a01-4ae3-8d80-1788ae33e332\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c67e94fe-m5ldg" Nov 03 01:26:34.851622 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:34.851577 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqfkg\" (UniqueName: \"kubernetes.io/projected/6e1fd99f-0a01-4ae3-8d80-1788ae33e332-kube-api-access-gqfkg\") pod \"maas-api-revoke-sub-c67e94fe-m5ldg\" (UID: \"6e1fd99f-0a01-4ae3-8d80-1788ae33e332\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c67e94fe-m5ldg" Nov 03 01:26:34.851816 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:34.851651 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e1fd99f-0a01-4ae3-8d80-1788ae33e332-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c67e94fe-m5ldg\" (UID: \"6e1fd99f-0a01-4ae3-8d80-1788ae33e332\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c67e94fe-m5ldg" Nov 03 01:26:34.852651 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:34.851889 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e1fd99f-0a01-4ae3-8d80-1788ae33e332-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c67e94fe-m5ldg\" (UID: \"6e1fd99f-0a01-4ae3-8d80-1788ae33e332\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c67e94fe-m5ldg" Nov 03 01:26:34.852715 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:34.852684 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e1fd99f-0a01-4ae3-8d80-1788ae33e332-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c67e94fe-m5ldg\" (UID: \"6e1fd99f-0a01-4ae3-8d80-1788ae33e332\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c67e94fe-m5ldg" Nov 03 01:26:34.854194 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:34.854172 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e1fd99f-0a01-4ae3-8d80-1788ae33e332-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c67e94fe-m5ldg\" (UID: \"6e1fd99f-0a01-4ae3-8d80-1788ae33e332\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c67e94fe-m5ldg" Nov 03 01:26:34.860754 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:34.860727 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqfkg\" (UniqueName: \"kubernetes.io/projected/6e1fd99f-0a01-4ae3-8d80-1788ae33e332-kube-api-access-gqfkg\") pod \"maas-api-revoke-sub-c67e94fe-m5ldg\" (UID: \"6e1fd99f-0a01-4ae3-8d80-1788ae33e332\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c67e94fe-m5ldg" Nov 03 01:26:34.996104 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:34.996015 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c67e94fe-m5ldg" Nov 03 01:26:35.289864 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:35.289830 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c67e94fe-m5ldg"] Nov 03 01:26:35.291528 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:26:35.291495 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e1fd99f_0a01_4ae3_8d80_1788ae33e332.slice/crio-613f9d0f65470f2301e3d9d188100e64604c00bc30d4cff5a9a855e2bed9ed34 WatchSource:0}: Error finding container 613f9d0f65470f2301e3d9d188100e64604c00bc30d4cff5a9a855e2bed9ed34: Status 404 returned error can't find the container with id 613f9d0f65470f2301e3d9d188100e64604c00bc30d4cff5a9a855e2bed9ed34 Nov 03 01:26:35.927181 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:35.927136 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c67e94fe-m5ldg" event={"ID":"6e1fd99f-0a01-4ae3-8d80-1788ae33e332","Type":"ContainerStarted","Data":"d2be94e4b4d1becf36ddae56b1a30f6c113cbaeb864532e9203eb964bd48da05"} Nov 03 01:26:35.927743 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:35.927209 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c67e94fe-m5ldg" event={"ID":"6e1fd99f-0a01-4ae3-8d80-1788ae33e332","Type":"ContainerStarted","Data":"613f9d0f65470f2301e3d9d188100e64604c00bc30d4cff5a9a855e2bed9ed34"} Nov 03 01:26:35.947578 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:35.947526 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c67e94fe-m5ldg" podStartSLOduration=1.9475112559999999 podStartE2EDuration="1.947511256s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:26:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:26:35.945433663 +0000 UTC m=+2738.116436771" watchObservedRunningTime="2026-11-03 01:26:35.947511256 +0000 UTC m=+2738.118514351" Nov 03 01:26:36.943192 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:36.943152 4421 generic.go:356] "Generic (PLEG): container finished" podID="6e1fd99f-0a01-4ae3-8d80-1788ae33e332" containerID="d2be94e4b4d1becf36ddae56b1a30f6c113cbaeb864532e9203eb964bd48da05" exitCode=0 Nov 03 01:26:36.943678 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:36.943196 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c67e94fe-m5ldg" event={"ID":"6e1fd99f-0a01-4ae3-8d80-1788ae33e332","Type":"ContainerDied","Data":"d2be94e4b4d1becf36ddae56b1a30f6c113cbaeb864532e9203eb964bd48da05"} Nov 03 01:26:38.135962 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:38.135934 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c67e94fe-m5ldg" Nov 03 01:26:38.212321 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:38.212202 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6e1fd99f-0a01-4ae3-8d80-1788ae33e332-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e1fd99f-0a01-4ae3-8d80-1788ae33e332-maas-api-xks-ca\") pod \"6e1fd99f-0a01-4ae3-8d80-1788ae33e332\" (UID: \"6e1fd99f-0a01-4ae3-8d80-1788ae33e332\") " Nov 03 01:26:38.212608 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:38.212586 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e1fd99f-0a01-4ae3-8d80-1788ae33e332-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e1fd99f-0a01-4ae3-8d80-1788ae33e332-maas-api-openshift-ca\") pod \"6e1fd99f-0a01-4ae3-8d80-1788ae33e332\" (UID: \"6e1fd99f-0a01-4ae3-8d80-1788ae33e332\") " Nov 03 01:26:38.212693 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:38.212639 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e1fd99f-0a01-4ae3-8d80-1788ae33e332-kube-api-access-gqfkg\" (UniqueName: \"kubernetes.io/projected/6e1fd99f-0a01-4ae3-8d80-1788ae33e332-kube-api-access-gqfkg\") pod \"6e1fd99f-0a01-4ae3-8d80-1788ae33e332\" (UID: \"6e1fd99f-0a01-4ae3-8d80-1788ae33e332\") " Nov 03 01:26:38.213022 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:38.212998 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e1fd99f-0a01-4ae3-8d80-1788ae33e332-maas-api-openshift-ca" pod "6e1fd99f-0a01-4ae3-8d80-1788ae33e332" (UID: "6e1fd99f-0a01-4ae3-8d80-1788ae33e332"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:26:38.213504 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:38.213485 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e1fd99f-0a01-4ae3-8d80-1788ae33e332-maas-api-openshift-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:26:38.214504 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:38.214477 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e1fd99f-0a01-4ae3-8d80-1788ae33e332-maas-api-xks-ca" pod "6e1fd99f-0a01-4ae3-8d80-1788ae33e332" (UID: "6e1fd99f-0a01-4ae3-8d80-1788ae33e332"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:26:38.215353 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:38.215321 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e1fd99f-0a01-4ae3-8d80-1788ae33e332-kube-api-access-gqfkg" pod "6e1fd99f-0a01-4ae3-8d80-1788ae33e332" (UID: "6e1fd99f-0a01-4ae3-8d80-1788ae33e332"). InnerVolumeSpecName "kube-api-access-gqfkg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:26:38.333294 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:38.326920 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e1fd99f-0a01-4ae3-8d80-1788ae33e332-maas-api-xks-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:26:38.333294 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:38.326964 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqfkg\" (UniqueName: \"kubernetes.io/projected/6e1fd99f-0a01-4ae3-8d80-1788ae33e332-kube-api-access-gqfkg\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:26:38.962453 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:38.962421 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c67e94fe-m5ldg" Nov 03 01:26:38.962667 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:38.962445 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c67e94fe-m5ldg" event={"ID":"6e1fd99f-0a01-4ae3-8d80-1788ae33e332","Type":"ContainerDied","Data":"613f9d0f65470f2301e3d9d188100e64604c00bc30d4cff5a9a855e2bed9ed34"} Nov 03 01:26:38.962667 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:38.962502 4421 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="613f9d0f65470f2301e3d9d188100e64604c00bc30d4cff5a9a855e2bed9ed34" Nov 03 01:26:44.672256 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:26:44.672181 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 01:26:59.012714 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:59.012681 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-hbjwh"] Nov 03 01:26:59.013814 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:26:59.012873 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-hbjwh" podUID="2f17e4c5-9bbd-4c90-bc08-c20d373da97d" containerName="istio-proxy" containerID="cri-o://c737a942d8498e9c0f5f94824e2a69553bb3e6ad5d8cc508dd534eea3b00525f" gracePeriod=30 Nov 03 01:27:03.785742 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:27:03.785709 4421 scope.go:122] "RemoveContainer" containerID="18da91d8d86b842975bbe0edb31f2c2cd6b649f489fcf2ccf24494dc4738ad9c" Nov 03 01:27:03.825847 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:27:03.821851 4421 scope.go:122] "RemoveContainer" containerID="624bbe5e7f548037fd864506e64b16716c24acb5ec5ca8cf95559220f85bab39" Nov 03 01:27:03.847316 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:27:03.836424 4421 patch_prober.go:29] interesting pod/e2e-extmod-a-d25759-openshift-default-d47fd9785-hbjwh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 01:27:03.847316 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:27:03.836498 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-hbjwh" podUID="2f17e4c5-9bbd-4c90-bc08-c20d373da97d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 01:27:03.914207 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:27:03.914181 4421 scope.go:122] "RemoveContainer" containerID="7294a8fb4ac1c42480286882a52f096192e4890a9287517d06e90b32b4807eca" Nov 03 01:27:04.346100 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:27:04.346070 4421 generic.go:356] "Generic (PLEG): container finished" podID="2f17e4c5-9bbd-4c90-bc08-c20d373da97d" containerID="c737a942d8498e9c0f5f94824e2a69553bb3e6ad5d8cc508dd534eea3b00525f" exitCode=0 Nov 03 01:27:04.346235 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:27:04.346152 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-hbjwh" event={"ID":"2f17e4c5-9bbd-4c90-bc08-c20d373da97d","Type":"ContainerDied","Data":"c737a942d8498e9c0f5f94824e2a69553bb3e6ad5d8cc508dd534eea3b00525f"} Nov 03 01:27:04.346235 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:27:04.346218 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-hbjwh" event={"ID":"2f17e4c5-9bbd-4c90-bc08-c20d373da97d","Type":"ContainerDied","Data":"ca3ea94c6b990eaf04ab7004baf2f8c2a883bc63de501168e6f641d9b55ebb12"} Nov 03 01:27:04.346346 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:27:04.346242 4421 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ca3ea94c6b990eaf04ab7004baf2f8c2a883bc63de501168e6f641d9b55ebb12" Nov 03 01:27:04.353218 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:27:04.353190 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-hbjwh" Nov 03 01:27:04.393412 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:27:04.393375 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-5bfc51-openshift-default-54d8846889-sc6sj"] Nov 03 01:27:04.394379 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:27:04.394358 4421 cpu_manager.go:414] "containerMap: removing container" podUID="6e1fd99f-0a01-4ae3-8d80-1788ae33e332" containerName="revoke-keys" Nov 03 01:27:04.394379 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:27:04.394378 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e1fd99f-0a01-4ae3-8d80-1788ae33e332" containerName="revoke-keys" Nov 03 01:27:04.394562 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:27:04.394396 4421 cpu_manager.go:414] "containerMap: removing container" podUID="2f17e4c5-9bbd-4c90-bc08-c20d373da97d" containerName="istio-proxy" Nov 03 01:27:04.394562 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:27:04.394403 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f17e4c5-9bbd-4c90-bc08-c20d373da97d" containerName="istio-proxy" Nov 03 01:27:04.394562 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:27:04.394541 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e1fd99f-0a01-4ae3-8d80-1788ae33e332" containerName="revoke-keys" Nov 03 01:27:04.394562 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:27:04.394551 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f17e4c5-9bbd-4c90-bc08-c20d373da97d" containerName="istio-proxy" Nov 03 01:27:04.399634 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:27:04.399252 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5bfc51-openshift-default-54d8846889-sc6sj" Nov 03 01:27:04.404098 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:27:04.404068 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-5bfc51-openshift-default-dockercfg-mjtdq\"" Nov 03 01:27:04.407629 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:27:04.407595 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-5bfc51-openshift-default-54d8846889-sc6sj"] Nov 03 01:27:04.449131 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:27:04.449091 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f17e4c5-9bbd-4c90-bc08-c20d373da97d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f17e4c5-9bbd-4c90-bc08-c20d373da97d-istiod-ca-cert\") pod \"2f17e4c5-9bbd-4c90-bc08-c20d373da97d\" (UID: \"2f17e4c5-9bbd-4c90-bc08-c20d373da97d\") " Nov 03 01:27:04.449410 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:27:04.449367 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f17e4c5-9bbd-4c90-bc08-c20d373da97d-istio-token\" (UniqueName: \"kubernetes.io/projected/2f17e4c5-9bbd-4c90-bc08-c20d373da97d-istio-token\") pod \"2f17e4c5-9bbd-4c90-bc08-c20d373da97d\" (UID: \"2f17e4c5-9bbd-4c90-bc08-c20d373da97d\") " Nov 03 01:27:04.449604 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:27:04.449574 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f17e4c5-9bbd-4c90-bc08-c20d373da97d-istiod-ca-cert" pod "2f17e4c5-9bbd-4c90-bc08-c20d373da97d" (UID: "2f17e4c5-9bbd-4c90-bc08-c20d373da97d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:27:04.450422 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:27:04.449576 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f17e4c5-9bbd-4c90-bc08-c20d373da97d-kube-api-access-ns47l\" (UniqueName: \"kubernetes.io/projected/2f17e4c5-9bbd-4c90-bc08-c20d373da97d-kube-api-access-ns47l\") pod \"2f17e4c5-9bbd-4c90-bc08-c20d373da97d\" (UID: \"2f17e4c5-9bbd-4c90-bc08-c20d373da97d\") " Nov 03 01:27:04.450422 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:27:04.449668 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f17e4c5-9bbd-4c90-bc08-c20d373da97d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f17e4c5-9bbd-4c90-bc08-c20d373da97d-workload-socket\") pod \"2f17e4c5-9bbd-4c90-bc08-c20d373da97d\" (UID: \"2f17e4c5-9bbd-4c90-bc08-c20d373da97d\") " Nov 03 01:27:04.450422 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:27:04.449759 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f17e4c5-9bbd-4c90-bc08-c20d373da97d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f17e4c5-9bbd-4c90-bc08-c20d373da97d-istio-envoy\") pod \"2f17e4c5-9bbd-4c90-bc08-c20d373da97d\" (UID: \"2f17e4c5-9bbd-4c90-bc08-c20d373da97d\") " Nov 03 01:27:04.450422 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:27:04.449843 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2f17e4c5-9bbd-4c90-bc08-c20d373da97d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f17e4c5-9bbd-4c90-bc08-c20d373da97d-istio-podinfo\") pod \"2f17e4c5-9bbd-4c90-bc08-c20d373da97d\" (UID: \"2f17e4c5-9bbd-4c90-bc08-c20d373da97d\") " Nov 03 01:27:04.450422 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:27:04.449936 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f17e4c5-9bbd-4c90-bc08-c20d373da97d-workload-socket" pod "2f17e4c5-9bbd-4c90-bc08-c20d373da97d" (UID: "2f17e4c5-9bbd-4c90-bc08-c20d373da97d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:27:04.450422 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:27:04.449994 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f17e4c5-9bbd-4c90-bc08-c20d373da97d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f17e4c5-9bbd-4c90-bc08-c20d373da97d-workload-certs\") pod \"2f17e4c5-9bbd-4c90-bc08-c20d373da97d\" (UID: \"2f17e4c5-9bbd-4c90-bc08-c20d373da97d\") " Nov 03 01:27:04.450422 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:27:04.450057 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f17e4c5-9bbd-4c90-bc08-c20d373da97d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f17e4c5-9bbd-4c90-bc08-c20d373da97d-istio-data\") pod \"2f17e4c5-9bbd-4c90-bc08-c20d373da97d\" (UID: \"2f17e4c5-9bbd-4c90-bc08-c20d373da97d\") " Nov 03 01:27:04.450422 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:27:04.450107 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f17e4c5-9bbd-4c90-bc08-c20d373da97d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f17e4c5-9bbd-4c90-bc08-c20d373da97d-credential-socket\") pod \"2f17e4c5-9bbd-4c90-bc08-c20d373da97d\" (UID: \"2f17e4c5-9bbd-4c90-bc08-c20d373da97d\") " Nov 03 01:27:04.453021 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:27:04.450423 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f17e4c5-9bbd-4c90-bc08-c20d373da97d-istio-data" pod "2f17e4c5-9bbd-4c90-bc08-c20d373da97d" (UID: "2f17e4c5-9bbd-4c90-bc08-c20d373da97d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:27:04.453021 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:27:04.450612 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f17e4c5-9bbd-4c90-bc08-c20d373da97d-credential-socket" pod "2f17e4c5-9bbd-4c90-bc08-c20d373da97d" (UID: "2f17e4c5-9bbd-4c90-bc08-c20d373da97d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:27:04.453021 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:27:04.451135 4421 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f17e4c5-9bbd-4c90-bc08-c20d373da97d-workload-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:27:04.453021 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:27:04.451160 4421 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f17e4c5-9bbd-4c90-bc08-c20d373da97d-istio-data\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:27:04.453021 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:27:04.451176 4421 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f17e4c5-9bbd-4c90-bc08-c20d373da97d-credential-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:27:04.453021 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:27:04.451189 4421 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f17e4c5-9bbd-4c90-bc08-c20d373da97d-istiod-ca-cert\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:27:04.453021 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:27:04.451314 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f17e4c5-9bbd-4c90-bc08-c20d373da97d-workload-certs" pod "2f17e4c5-9bbd-4c90-bc08-c20d373da97d" (UID: "2f17e4c5-9bbd-4c90-bc08-c20d373da97d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:27:04.453021 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:27:04.452431 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f17e4c5-9bbd-4c90-bc08-c20d373da97d-istio-token" pod "2f17e4c5-9bbd-4c90-bc08-c20d373da97d" (UID: "2f17e4c5-9bbd-4c90-bc08-c20d373da97d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:27:04.453021 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:27:04.452443 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2f17e4c5-9bbd-4c90-bc08-c20d373da97d-istio-podinfo" pod "2f17e4c5-9bbd-4c90-bc08-c20d373da97d" (UID: "2f17e4c5-9bbd-4c90-bc08-c20d373da97d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:27:04.453021 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:27:04.452472 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f17e4c5-9bbd-4c90-bc08-c20d373da97d-kube-api-access-ns47l" pod "2f17e4c5-9bbd-4c90-bc08-c20d373da97d" (UID: "2f17e4c5-9bbd-4c90-bc08-c20d373da97d"). InnerVolumeSpecName "kube-api-access-ns47l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:27:04.454057 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:27:04.453909 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f17e4c5-9bbd-4c90-bc08-c20d373da97d-istio-envoy" pod "2f17e4c5-9bbd-4c90-bc08-c20d373da97d" (UID: "2f17e4c5-9bbd-4c90-bc08-c20d373da97d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:27:04.557966 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:27:04.556849 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/781e1acc-4bc9-489f-8df6-6b5f8c10ba1b-workload-certs\") pod \"e2e-inf-b-5bfc51-openshift-default-54d8846889-sc6sj\" (UID: \"781e1acc-4bc9-489f-8df6-6b5f8c10ba1b\") " pod="openshift-ingress/e2e-inf-b-5bfc51-openshift-default-54d8846889-sc6sj" Nov 03 01:27:04.557966 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:27:04.557023 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ssrq5\" (UniqueName: \"kubernetes.io/projected/781e1acc-4bc9-489f-8df6-6b5f8c10ba1b-kube-api-access-ssrq5\") pod \"e2e-inf-b-5bfc51-openshift-default-54d8846889-sc6sj\" (UID: \"781e1acc-4bc9-489f-8df6-6b5f8c10ba1b\") " pod="openshift-ingress/e2e-inf-b-5bfc51-openshift-default-54d8846889-sc6sj" Nov 03 01:27:04.557966 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:27:04.557186 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/781e1acc-4bc9-489f-8df6-6b5f8c10ba1b-workload-socket\") pod \"e2e-inf-b-5bfc51-openshift-default-54d8846889-sc6sj\" (UID: \"781e1acc-4bc9-489f-8df6-6b5f8c10ba1b\") " pod="openshift-ingress/e2e-inf-b-5bfc51-openshift-default-54d8846889-sc6sj" Nov 03 01:27:04.557966 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:27:04.557215 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/781e1acc-4bc9-489f-8df6-6b5f8c10ba1b-istio-podinfo\") pod \"e2e-inf-b-5bfc51-openshift-default-54d8846889-sc6sj\" (UID: \"781e1acc-4bc9-489f-8df6-6b5f8c10ba1b\") " pod="openshift-ingress/e2e-inf-b-5bfc51-openshift-default-54d8846889-sc6sj" Nov 03 01:27:04.557966 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:27:04.557358 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/781e1acc-4bc9-489f-8df6-6b5f8c10ba1b-istio-envoy\") pod \"e2e-inf-b-5bfc51-openshift-default-54d8846889-sc6sj\" (UID: \"781e1acc-4bc9-489f-8df6-6b5f8c10ba1b\") " pod="openshift-ingress/e2e-inf-b-5bfc51-openshift-default-54d8846889-sc6sj" Nov 03 01:27:04.557966 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:27:04.557438 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/781e1acc-4bc9-489f-8df6-6b5f8c10ba1b-istio-token\") pod \"e2e-inf-b-5bfc51-openshift-default-54d8846889-sc6sj\" (UID: \"781e1acc-4bc9-489f-8df6-6b5f8c10ba1b\") " pod="openshift-ingress/e2e-inf-b-5bfc51-openshift-default-54d8846889-sc6sj" Nov 03 01:27:04.557966 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:27:04.557507 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/781e1acc-4bc9-489f-8df6-6b5f8c10ba1b-istio-data\") pod \"e2e-inf-b-5bfc51-openshift-default-54d8846889-sc6sj\" (UID: \"781e1acc-4bc9-489f-8df6-6b5f8c10ba1b\") " pod="openshift-ingress/e2e-inf-b-5bfc51-openshift-default-54d8846889-sc6sj" Nov 03 01:27:04.557966 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:27:04.557559 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/781e1acc-4bc9-489f-8df6-6b5f8c10ba1b-istiod-ca-cert\") pod \"e2e-inf-b-5bfc51-openshift-default-54d8846889-sc6sj\" (UID: \"781e1acc-4bc9-489f-8df6-6b5f8c10ba1b\") " pod="openshift-ingress/e2e-inf-b-5bfc51-openshift-default-54d8846889-sc6sj" Nov 03 01:27:04.557966 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:27:04.557611 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/781e1acc-4bc9-489f-8df6-6b5f8c10ba1b-credential-socket\") pod \"e2e-inf-b-5bfc51-openshift-default-54d8846889-sc6sj\" (UID: \"781e1acc-4bc9-489f-8df6-6b5f8c10ba1b\") " pod="openshift-ingress/e2e-inf-b-5bfc51-openshift-default-54d8846889-sc6sj" Nov 03 01:27:04.557966 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:27:04.557739 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ns47l\" (UniqueName: \"kubernetes.io/projected/2f17e4c5-9bbd-4c90-bc08-c20d373da97d-kube-api-access-ns47l\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:27:04.557966 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:27:04.557756 4421 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f17e4c5-9bbd-4c90-bc08-c20d373da97d-istio-envoy\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:27:04.557966 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:27:04.557771 4421 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f17e4c5-9bbd-4c90-bc08-c20d373da97d-istio-podinfo\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:27:04.557966 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:27:04.557784 4421 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f17e4c5-9bbd-4c90-bc08-c20d373da97d-workload-certs\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:27:04.557966 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:27:04.557798 4421 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f17e4c5-9bbd-4c90-bc08-c20d373da97d-istio-token\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:27:04.659556 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:27:04.659465 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/781e1acc-4bc9-489f-8df6-6b5f8c10ba1b-workload-certs\") pod \"e2e-inf-b-5bfc51-openshift-default-54d8846889-sc6sj\" (UID: \"781e1acc-4bc9-489f-8df6-6b5f8c10ba1b\") " pod="openshift-ingress/e2e-inf-b-5bfc51-openshift-default-54d8846889-sc6sj" Nov 03 01:27:04.659725 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:27:04.659599 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ssrq5\" (UniqueName: \"kubernetes.io/projected/781e1acc-4bc9-489f-8df6-6b5f8c10ba1b-kube-api-access-ssrq5\") pod \"e2e-inf-b-5bfc51-openshift-default-54d8846889-sc6sj\" (UID: \"781e1acc-4bc9-489f-8df6-6b5f8c10ba1b\") " pod="openshift-ingress/e2e-inf-b-5bfc51-openshift-default-54d8846889-sc6sj" Nov 03 01:27:04.659725 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:27:04.659686 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/781e1acc-4bc9-489f-8df6-6b5f8c10ba1b-workload-socket\") pod \"e2e-inf-b-5bfc51-openshift-default-54d8846889-sc6sj\" (UID: \"781e1acc-4bc9-489f-8df6-6b5f8c10ba1b\") " pod="openshift-ingress/e2e-inf-b-5bfc51-openshift-default-54d8846889-sc6sj" Nov 03 01:27:04.659725 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:27:04.659710 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/781e1acc-4bc9-489f-8df6-6b5f8c10ba1b-istio-podinfo\") pod \"e2e-inf-b-5bfc51-openshift-default-54d8846889-sc6sj\" (UID: \"781e1acc-4bc9-489f-8df6-6b5f8c10ba1b\") " pod="openshift-ingress/e2e-inf-b-5bfc51-openshift-default-54d8846889-sc6sj" Nov 03 01:27:04.659911 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:27:04.659825 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/781e1acc-4bc9-489f-8df6-6b5f8c10ba1b-istio-envoy\") pod \"e2e-inf-b-5bfc51-openshift-default-54d8846889-sc6sj\" (UID: \"781e1acc-4bc9-489f-8df6-6b5f8c10ba1b\") " pod="openshift-ingress/e2e-inf-b-5bfc51-openshift-default-54d8846889-sc6sj" Nov 03 01:27:04.659911 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:27:04.659892 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/781e1acc-4bc9-489f-8df6-6b5f8c10ba1b-istio-token\") pod \"e2e-inf-b-5bfc51-openshift-default-54d8846889-sc6sj\" (UID: \"781e1acc-4bc9-489f-8df6-6b5f8c10ba1b\") " pod="openshift-ingress/e2e-inf-b-5bfc51-openshift-default-54d8846889-sc6sj" Nov 03 01:27:04.660043 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:27:04.659902 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/781e1acc-4bc9-489f-8df6-6b5f8c10ba1b-workload-certs\") pod \"e2e-inf-b-5bfc51-openshift-default-54d8846889-sc6sj\" (UID: \"781e1acc-4bc9-489f-8df6-6b5f8c10ba1b\") " pod="openshift-ingress/e2e-inf-b-5bfc51-openshift-default-54d8846889-sc6sj" Nov 03 01:27:04.660043 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:27:04.659945 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/781e1acc-4bc9-489f-8df6-6b5f8c10ba1b-istio-data\") pod \"e2e-inf-b-5bfc51-openshift-default-54d8846889-sc6sj\" (UID: \"781e1acc-4bc9-489f-8df6-6b5f8c10ba1b\") " pod="openshift-ingress/e2e-inf-b-5bfc51-openshift-default-54d8846889-sc6sj" Nov 03 01:27:04.660043 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:27:04.659986 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/781e1acc-4bc9-489f-8df6-6b5f8c10ba1b-istiod-ca-cert\") pod \"e2e-inf-b-5bfc51-openshift-default-54d8846889-sc6sj\" (UID: \"781e1acc-4bc9-489f-8df6-6b5f8c10ba1b\") " pod="openshift-ingress/e2e-inf-b-5bfc51-openshift-default-54d8846889-sc6sj" Nov 03 01:27:04.660043 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:27:04.660027 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/781e1acc-4bc9-489f-8df6-6b5f8c10ba1b-credential-socket\") pod \"e2e-inf-b-5bfc51-openshift-default-54d8846889-sc6sj\" (UID: \"781e1acc-4bc9-489f-8df6-6b5f8c10ba1b\") " pod="openshift-ingress/e2e-inf-b-5bfc51-openshift-default-54d8846889-sc6sj" Nov 03 01:27:04.660950 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:27:04.660078 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/781e1acc-4bc9-489f-8df6-6b5f8c10ba1b-workload-socket\") pod \"e2e-inf-b-5bfc51-openshift-default-54d8846889-sc6sj\" (UID: \"781e1acc-4bc9-489f-8df6-6b5f8c10ba1b\") " pod="openshift-ingress/e2e-inf-b-5bfc51-openshift-default-54d8846889-sc6sj" Nov 03 01:27:04.660950 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:27:04.660263 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/781e1acc-4bc9-489f-8df6-6b5f8c10ba1b-istio-data\") pod \"e2e-inf-b-5bfc51-openshift-default-54d8846889-sc6sj\" (UID: \"781e1acc-4bc9-489f-8df6-6b5f8c10ba1b\") " pod="openshift-ingress/e2e-inf-b-5bfc51-openshift-default-54d8846889-sc6sj" Nov 03 01:27:04.660950 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:27:04.660428 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/781e1acc-4bc9-489f-8df6-6b5f8c10ba1b-credential-socket\") pod \"e2e-inf-b-5bfc51-openshift-default-54d8846889-sc6sj\" (UID: \"781e1acc-4bc9-489f-8df6-6b5f8c10ba1b\") " pod="openshift-ingress/e2e-inf-b-5bfc51-openshift-default-54d8846889-sc6sj" Nov 03 01:27:04.660950 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:27:04.660674 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/781e1acc-4bc9-489f-8df6-6b5f8c10ba1b-istiod-ca-cert\") pod \"e2e-inf-b-5bfc51-openshift-default-54d8846889-sc6sj\" (UID: \"781e1acc-4bc9-489f-8df6-6b5f8c10ba1b\") " pod="openshift-ingress/e2e-inf-b-5bfc51-openshift-default-54d8846889-sc6sj" Nov 03 01:27:04.662145 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:27:04.662130 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/781e1acc-4bc9-489f-8df6-6b5f8c10ba1b-istio-envoy\") pod \"e2e-inf-b-5bfc51-openshift-default-54d8846889-sc6sj\" (UID: \"781e1acc-4bc9-489f-8df6-6b5f8c10ba1b\") " pod="openshift-ingress/e2e-inf-b-5bfc51-openshift-default-54d8846889-sc6sj" Nov 03 01:27:04.662445 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:27:04.662428 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/781e1acc-4bc9-489f-8df6-6b5f8c10ba1b-istio-podinfo\") pod \"e2e-inf-b-5bfc51-openshift-default-54d8846889-sc6sj\" (UID: \"781e1acc-4bc9-489f-8df6-6b5f8c10ba1b\") " pod="openshift-ingress/e2e-inf-b-5bfc51-openshift-default-54d8846889-sc6sj" Nov 03 01:27:04.668135 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:27:04.668114 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/781e1acc-4bc9-489f-8df6-6b5f8c10ba1b-istio-token\") pod \"e2e-inf-b-5bfc51-openshift-default-54d8846889-sc6sj\" (UID: \"781e1acc-4bc9-489f-8df6-6b5f8c10ba1b\") " pod="openshift-ingress/e2e-inf-b-5bfc51-openshift-default-54d8846889-sc6sj" Nov 03 01:27:04.668259 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:27:04.668161 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssrq5\" (UniqueName: \"kubernetes.io/projected/781e1acc-4bc9-489f-8df6-6b5f8c10ba1b-kube-api-access-ssrq5\") pod \"e2e-inf-b-5bfc51-openshift-default-54d8846889-sc6sj\" (UID: \"781e1acc-4bc9-489f-8df6-6b5f8c10ba1b\") " pod="openshift-ingress/e2e-inf-b-5bfc51-openshift-default-54d8846889-sc6sj" Nov 03 01:27:04.723197 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:27:04.723165 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5bfc51-openshift-default-54d8846889-sc6sj" Nov 03 01:27:05.015131 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:27:05.015097 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod781e1acc_4bc9_489f_8df6_6b5f8c10ba1b.slice/crio-7857e624a87614454db50b098f5dc7732cedbcaf34f73e08697a5f19a26e3b4a WatchSource:0}: Error finding container 7857e624a87614454db50b098f5dc7732cedbcaf34f73e08697a5f19a26e3b4a: Status 404 returned error can't find the container with id 7857e624a87614454db50b098f5dc7732cedbcaf34f73e08697a5f19a26e3b4a Nov 03 01:27:05.015538 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:27:05.015517 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-5bfc51-openshift-default-54d8846889-sc6sj"] Nov 03 01:27:05.018395 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:27:05.018358 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:27:05.018519 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:27:05.018451 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:27:05.018519 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:27:05.018480 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:27:05.359414 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:27:05.359307 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5bfc51-openshift-default-54d8846889-sc6sj" event={"ID":"781e1acc-4bc9-489f-8df6-6b5f8c10ba1b","Type":"ContainerStarted","Data":"39421a5a8e7703794bb42341d5ced6151daf742935511581e9300f6e52d11b57"} Nov 03 01:27:05.359414 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:27:05.359369 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5bfc51-openshift-default-54d8846889-sc6sj" event={"ID":"781e1acc-4bc9-489f-8df6-6b5f8c10ba1b","Type":"ContainerStarted","Data":"7857e624a87614454db50b098f5dc7732cedbcaf34f73e08697a5f19a26e3b4a"} Nov 03 01:27:05.359414 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:27:05.359384 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-hbjwh" Nov 03 01:27:05.382132 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:27:05.382066 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-5bfc51-openshift-default-54d8846889-sc6sj" podStartSLOduration=32.382046093 podStartE2EDuration="32.382046093s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:26:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:27:05.377484816 +0000 UTC m=+2767.548487912" watchObservedRunningTime="2026-11-03 01:27:05.382046093 +0000 UTC m=+2767.553049190" Nov 03 01:27:05.407041 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:27:05.401676 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-hbjwh"] Nov 03 01:27:05.408571 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:27:05.408541 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-hbjwh"] Nov 03 01:27:05.723702 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:27:05.723619 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-5bfc51-openshift-default-54d8846889-sc6sj" Nov 03 01:27:06.005927 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:27:06.005899 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f17e4c5-9bbd-4c90-bc08-c20d373da97d" path="/var/lib/kubelet/pods/2f17e4c5-9bbd-4c90-bc08-c20d373da97d/volumes" Nov 03 01:27:06.759300 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:27:06.758710 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-5bfc51-openshift-default-54d8846889-sc6sj" Nov 03 01:27:07.376563 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:27:07.376535 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-5bfc51-openshift-default-54d8846889-sc6sj" Nov 03 01:27:07.377739 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:27:07.377715 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-5bfc51-openshift-default-54d8846889-sc6sj" Nov 03 01:27:09.021576 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:27:09.021531 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-5bfc51-6c7fc6b99f-kdhcn"] Nov 03 01:27:09.030784 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:27:09.030734 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5bfc51-6c7fc6b99f-kdhcn" Nov 03 01:27:09.038070 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:27:09.035594 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-5bfc51-6c7fc6b99f-kdhcn"] Nov 03 01:27:09.038070 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:27:09.037835 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-5bfc51-dockercfg-dhwp2\"" Nov 03 01:27:09.126312 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:27:09.124552 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-5bfc51\"" Nov 03 01:27:09.202396 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:27:09.202100 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1758fd0f-1c79-41fe-be74-474ba88962ca-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-5bfc51-6c7fc6b99f-kdhcn\" (UID: \"1758fd0f-1c79-41fe-be74-474ba88962ca\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5bfc51-6c7fc6b99f-kdhcn" Nov 03 01:27:09.207400 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:27:09.202979 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnxmw\" (UniqueName: \"kubernetes.io/projected/1758fd0f-1c79-41fe-be74-474ba88962ca-kube-api-access-cnxmw\") pod \"payload-pre-processing-e2e-inf-b-5bfc51-6c7fc6b99f-kdhcn\" (UID: \"1758fd0f-1c79-41fe-be74-474ba88962ca\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5bfc51-6c7fc6b99f-kdhcn" Nov 03 01:27:09.318352 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:27:09.311797 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnxmw\" (UniqueName: \"kubernetes.io/projected/1758fd0f-1c79-41fe-be74-474ba88962ca-kube-api-access-cnxmw\") pod \"payload-pre-processing-e2e-inf-b-5bfc51-6c7fc6b99f-kdhcn\" (UID: \"1758fd0f-1c79-41fe-be74-474ba88962ca\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5bfc51-6c7fc6b99f-kdhcn" Nov 03 01:27:09.318919 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:27:09.317203 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1758fd0f-1c79-41fe-be74-474ba88962ca-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-5bfc51-6c7fc6b99f-kdhcn\" (UID: \"1758fd0f-1c79-41fe-be74-474ba88962ca\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5bfc51-6c7fc6b99f-kdhcn" Nov 03 01:27:09.318919 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:27:09.318687 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1758fd0f-1c79-41fe-be74-474ba88962ca-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-5bfc51-6c7fc6b99f-kdhcn\" (UID: \"1758fd0f-1c79-41fe-be74-474ba88962ca\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5bfc51-6c7fc6b99f-kdhcn" Nov 03 01:27:09.365664 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:27:09.358553 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnxmw\" (UniqueName: \"kubernetes.io/projected/1758fd0f-1c79-41fe-be74-474ba88962ca-kube-api-access-cnxmw\") pod \"payload-pre-processing-e2e-inf-b-5bfc51-6c7fc6b99f-kdhcn\" (UID: \"1758fd0f-1c79-41fe-be74-474ba88962ca\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5bfc51-6c7fc6b99f-kdhcn" Nov 03 01:27:09.654797 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:27:09.654709 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5bfc51-6c7fc6b99f-kdhcn" Nov 03 01:27:09.948867 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:27:09.948830 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-5bfc51-6c7fc6b99f-kdhcn"] Nov 03 01:27:09.949477 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:27:09.949440 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1758fd0f_1c79_41fe_be74_474ba88962ca.slice/crio-9935c56a019ed82f5a582426dfe6dd1cdce1e64e975c42d0f674b6a6f2a682f0 WatchSource:0}: Error finding container 9935c56a019ed82f5a582426dfe6dd1cdce1e64e975c42d0f674b6a6f2a682f0: Status 404 returned error can't find the container with id 9935c56a019ed82f5a582426dfe6dd1cdce1e64e975c42d0f674b6a6f2a682f0 Nov 03 01:27:10.464774 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:27:10.464723 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5bfc51-6c7fc6b99f-kdhcn" event={"ID":"1758fd0f-1c79-41fe-be74-474ba88962ca","Type":"ContainerStarted","Data":"24254f8efebde82101fba3587d2b4d21e199d080797c90107ca55a1575aa0b90"} Nov 03 01:27:10.465303 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:27:10.465262 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5bfc51-6c7fc6b99f-kdhcn" event={"ID":"1758fd0f-1c79-41fe-be74-474ba88962ca","Type":"ContainerStarted","Data":"9935c56a019ed82f5a582426dfe6dd1cdce1e64e975c42d0f674b6a6f2a682f0"} Nov 03 01:27:10.465569 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:27:10.465537 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5bfc51-6c7fc6b99f-kdhcn" Nov 03 01:27:10.487082 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:27:10.487026 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5bfc51-6c7fc6b99f-kdhcn" podStartSLOduration=2.4870126089999998 podStartE2EDuration="2.487012609s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:27:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:27:10.482551897 +0000 UTC m=+2772.653554994" watchObservedRunningTime="2026-11-03 01:27:10.487012609 +0000 UTC m=+2772.658015705" Nov 03 01:27:11.088393 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:27:11.088344 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bda16dd6-j7lmk"] Nov 03 01:27:11.109323 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:27:11.109290 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bda16dd6-j7lmk"] Nov 03 01:27:11.982489 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:27:11.982454 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2749d4f6-690f-4c75-aed8-79b8b3ee832e" path="/var/lib/kubelet/pods/2749d4f6-690f-4c75-aed8-79b8b3ee832e/volumes" Nov 03 01:27:14.672583 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:27:14.672538 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 01:27:21.502747 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:27:21.502708 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5bfc51-6c7fc6b99f-kdhcn" Nov 03 01:27:44.671743 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:27:44.671704 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 01:27:51.416296 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:27:51.415996 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31ccefb4-n9r9z"] Nov 03 01:27:51.429902 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:27:51.429871 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31ccefb4-n9r9z"] Nov 03 01:27:51.430150 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:27:51.430005 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31ccefb4-n9r9z" Nov 03 01:27:51.434700 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:27:51.434660 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:27:51.435865 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:27:51.435842 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6q4xp\"" Nov 03 01:27:51.460858 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:27:51.460817 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bf7762e-6620-4f61-bd57-a988ed382691-maas-api-xks-ca\") pod \"maas-api-revoke-sub-31ccefb4-n9r9z\" (UID: \"6bf7762e-6620-4f61-bd57-a988ed382691\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31ccefb4-n9r9z" Nov 03 01:27:51.461107 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:27:51.461086 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bf7762e-6620-4f61-bd57-a988ed382691-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-31ccefb4-n9r9z\" (UID: \"6bf7762e-6620-4f61-bd57-a988ed382691\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31ccefb4-n9r9z" Nov 03 01:27:51.461379 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:27:51.461354 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmh6s\" (UniqueName: \"kubernetes.io/projected/6bf7762e-6620-4f61-bd57-a988ed382691-kube-api-access-xmh6s\") pod \"maas-api-revoke-sub-31ccefb4-n9r9z\" (UID: \"6bf7762e-6620-4f61-bd57-a988ed382691\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31ccefb4-n9r9z" Nov 03 01:27:51.563568 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:27:51.563530 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bf7762e-6620-4f61-bd57-a988ed382691-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-31ccefb4-n9r9z\" (UID: \"6bf7762e-6620-4f61-bd57-a988ed382691\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31ccefb4-n9r9z" Nov 03 01:27:51.563817 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:27:51.563798 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmh6s\" (UniqueName: \"kubernetes.io/projected/6bf7762e-6620-4f61-bd57-a988ed382691-kube-api-access-xmh6s\") pod \"maas-api-revoke-sub-31ccefb4-n9r9z\" (UID: \"6bf7762e-6620-4f61-bd57-a988ed382691\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31ccefb4-n9r9z" Nov 03 01:27:51.563921 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:27:51.563902 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bf7762e-6620-4f61-bd57-a988ed382691-maas-api-xks-ca\") pod \"maas-api-revoke-sub-31ccefb4-n9r9z\" (UID: \"6bf7762e-6620-4f61-bd57-a988ed382691\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31ccefb4-n9r9z" Nov 03 01:27:51.564354 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:27:51.564329 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bf7762e-6620-4f61-bd57-a988ed382691-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-31ccefb4-n9r9z\" (UID: \"6bf7762e-6620-4f61-bd57-a988ed382691\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31ccefb4-n9r9z" Nov 03 01:27:51.566465 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:27:51.566437 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bf7762e-6620-4f61-bd57-a988ed382691-maas-api-xks-ca\") pod \"maas-api-revoke-sub-31ccefb4-n9r9z\" (UID: \"6bf7762e-6620-4f61-bd57-a988ed382691\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31ccefb4-n9r9z" Nov 03 01:27:51.574041 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:27:51.574015 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmh6s\" (UniqueName: \"kubernetes.io/projected/6bf7762e-6620-4f61-bd57-a988ed382691-kube-api-access-xmh6s\") pod \"maas-api-revoke-sub-31ccefb4-n9r9z\" (UID: \"6bf7762e-6620-4f61-bd57-a988ed382691\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31ccefb4-n9r9z" Nov 03 01:27:51.753996 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:27:51.753968 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31ccefb4-n9r9z" Nov 03 01:27:52.042269 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:27:52.042233 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bf7762e_6620_4f61_bd57_a988ed382691.slice/crio-8ba2fec1d21403eba2071e6eee28f5e96f612717b99e6247bea00db777446116 WatchSource:0}: Error finding container 8ba2fec1d21403eba2071e6eee28f5e96f612717b99e6247bea00db777446116: Status 404 returned error can't find the container with id 8ba2fec1d21403eba2071e6eee28f5e96f612717b99e6247bea00db777446116 Nov 03 01:27:52.042449 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:27:52.042399 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31ccefb4-n9r9z"] Nov 03 01:27:52.200332 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:27:52.198994 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31ccefb4-n9r9z" event={"ID":"6bf7762e-6620-4f61-bd57-a988ed382691","Type":"ContainerStarted","Data":"8ba2fec1d21403eba2071e6eee28f5e96f612717b99e6247bea00db777446116"} Nov 03 01:27:53.209764 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:27:53.209721 4421 generic.go:356] "Generic (PLEG): container finished" podID="6bf7762e-6620-4f61-bd57-a988ed382691" containerID="82496076a6fe00149ba48e215e7b265b317fc263b9a661dcb61afcb37ed3b110" exitCode=0 Nov 03 01:27:53.210171 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:27:53.209807 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31ccefb4-n9r9z" event={"ID":"6bf7762e-6620-4f61-bd57-a988ed382691","Type":"ContainerDied","Data":"82496076a6fe00149ba48e215e7b265b317fc263b9a661dcb61afcb37ed3b110"} Nov 03 01:27:54.415879 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:27:54.415850 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31ccefb4-n9r9z" Nov 03 01:27:54.547175 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:27:54.547134 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6bf7762e-6620-4f61-bd57-a988ed382691-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bf7762e-6620-4f61-bd57-a988ed382691-maas-api-xks-ca\") pod \"6bf7762e-6620-4f61-bd57-a988ed382691\" (UID: \"6bf7762e-6620-4f61-bd57-a988ed382691\") " Nov 03 01:27:54.547623 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:27:54.547509 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6bf7762e-6620-4f61-bd57-a988ed382691-kube-api-access-xmh6s\" (UniqueName: \"kubernetes.io/projected/6bf7762e-6620-4f61-bd57-a988ed382691-kube-api-access-xmh6s\") pod \"6bf7762e-6620-4f61-bd57-a988ed382691\" (UID: \"6bf7762e-6620-4f61-bd57-a988ed382691\") " Nov 03 01:27:54.547748 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:27:54.547688 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6bf7762e-6620-4f61-bd57-a988ed382691-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bf7762e-6620-4f61-bd57-a988ed382691-maas-api-openshift-ca\") pod \"6bf7762e-6620-4f61-bd57-a988ed382691\" (UID: \"6bf7762e-6620-4f61-bd57-a988ed382691\") " Nov 03 01:27:54.548700 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:27:54.548650 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bf7762e-6620-4f61-bd57-a988ed382691-maas-api-openshift-ca" pod "6bf7762e-6620-4f61-bd57-a988ed382691" (UID: "6bf7762e-6620-4f61-bd57-a988ed382691"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:27:54.550933 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:27:54.550899 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bf7762e-6620-4f61-bd57-a988ed382691-kube-api-access-xmh6s" pod "6bf7762e-6620-4f61-bd57-a988ed382691" (UID: "6bf7762e-6620-4f61-bd57-a988ed382691"). InnerVolumeSpecName "kube-api-access-xmh6s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:27:54.551186 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:27:54.551160 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6bf7762e-6620-4f61-bd57-a988ed382691-maas-api-xks-ca" pod "6bf7762e-6620-4f61-bd57-a988ed382691" (UID: "6bf7762e-6620-4f61-bd57-a988ed382691"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:27:54.667302 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:27:54.656874 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmh6s\" (UniqueName: \"kubernetes.io/projected/6bf7762e-6620-4f61-bd57-a988ed382691-kube-api-access-xmh6s\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:27:54.667302 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:27:54.656916 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bf7762e-6620-4f61-bd57-a988ed382691-maas-api-openshift-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:27:54.667302 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:27:54.656932 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bf7762e-6620-4f61-bd57-a988ed382691-maas-api-xks-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:27:55.237693 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:27:55.237652 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31ccefb4-n9r9z" Nov 03 01:27:55.237915 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:27:55.237651 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31ccefb4-n9r9z" event={"ID":"6bf7762e-6620-4f61-bd57-a988ed382691","Type":"ContainerDied","Data":"8ba2fec1d21403eba2071e6eee28f5e96f612717b99e6247bea00db777446116"} Nov 03 01:27:55.237915 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:27:55.237811 4421 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8ba2fec1d21403eba2071e6eee28f5e96f612717b99e6247bea00db777446116" Nov 03 01:28:02.559296 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:02.555166 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f462289d-9ngww"] Nov 03 01:28:02.559296 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:02.556617 4421 cpu_manager.go:414] "containerMap: removing container" podUID="6bf7762e-6620-4f61-bd57-a988ed382691" containerName="revoke-keys" Nov 03 01:28:02.559296 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:02.556641 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bf7762e-6620-4f61-bd57-a988ed382691" containerName="revoke-keys" Nov 03 01:28:02.559296 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:02.556918 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bf7762e-6620-4f61-bd57-a988ed382691" containerName="revoke-keys" Nov 03 01:28:02.576391 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:02.564470 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f462289d-9ngww" Nov 03 01:28:02.576391 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:02.568721 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:28:02.576391 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:02.572705 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f462289d-9ngww"] Nov 03 01:28:02.576391 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:02.574497 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6q4xp\"" Nov 03 01:28:02.733301 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:02.730704 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hbc2\" (UniqueName: \"kubernetes.io/projected/d06ea1eb-b168-4b5e-be57-cb4e5a45d911-kube-api-access-4hbc2\") pod \"maas-api-revoke-sub-f462289d-9ngww\" (UID: \"d06ea1eb-b168-4b5e-be57-cb4e5a45d911\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f462289d-9ngww" Nov 03 01:28:02.733701 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:02.733658 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d06ea1eb-b168-4b5e-be57-cb4e5a45d911-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f462289d-9ngww\" (UID: \"d06ea1eb-b168-4b5e-be57-cb4e5a45d911\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f462289d-9ngww" Nov 03 01:28:02.733908 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:02.733887 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d06ea1eb-b168-4b5e-be57-cb4e5a45d911-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f462289d-9ngww\" (UID: \"d06ea1eb-b168-4b5e-be57-cb4e5a45d911\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f462289d-9ngww" Nov 03 01:28:02.836735 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:02.836636 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hbc2\" (UniqueName: \"kubernetes.io/projected/d06ea1eb-b168-4b5e-be57-cb4e5a45d911-kube-api-access-4hbc2\") pod \"maas-api-revoke-sub-f462289d-9ngww\" (UID: \"d06ea1eb-b168-4b5e-be57-cb4e5a45d911\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f462289d-9ngww" Nov 03 01:28:02.836927 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:02.836827 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d06ea1eb-b168-4b5e-be57-cb4e5a45d911-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f462289d-9ngww\" (UID: \"d06ea1eb-b168-4b5e-be57-cb4e5a45d911\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f462289d-9ngww" Nov 03 01:28:02.836999 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:02.836963 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d06ea1eb-b168-4b5e-be57-cb4e5a45d911-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f462289d-9ngww\" (UID: \"d06ea1eb-b168-4b5e-be57-cb4e5a45d911\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f462289d-9ngww" Nov 03 01:28:02.837707 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:02.837680 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d06ea1eb-b168-4b5e-be57-cb4e5a45d911-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f462289d-9ngww\" (UID: \"d06ea1eb-b168-4b5e-be57-cb4e5a45d911\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f462289d-9ngww" Nov 03 01:28:02.839445 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:02.839422 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d06ea1eb-b168-4b5e-be57-cb4e5a45d911-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f462289d-9ngww\" (UID: \"d06ea1eb-b168-4b5e-be57-cb4e5a45d911\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f462289d-9ngww" Nov 03 01:28:02.844789 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:02.844761 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hbc2\" (UniqueName: \"kubernetes.io/projected/d06ea1eb-b168-4b5e-be57-cb4e5a45d911-kube-api-access-4hbc2\") pod \"maas-api-revoke-sub-f462289d-9ngww\" (UID: \"d06ea1eb-b168-4b5e-be57-cb4e5a45d911\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f462289d-9ngww" Nov 03 01:28:02.952732 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:02.952698 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f462289d-9ngww" Nov 03 01:28:03.279548 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:03.278902 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f462289d-9ngww"] Nov 03 01:28:03.435775 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:03.435738 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f462289d-9ngww" event={"ID":"d06ea1eb-b168-4b5e-be57-cb4e5a45d911","Type":"ContainerStarted","Data":"1472b24c4f33486bfc7e3b4bc1d05cb75882cddf7c5a976e1c140daae66e11e7"} Nov 03 01:28:03.989624 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:03.989592 4421 scope.go:122] "RemoveContainer" containerID="694483be4e7539343ac52f3e7be2f8e5b9aba726ebdab3819e576fac6d4698cf" Nov 03 01:28:04.030672 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:04.023632 4421 scope.go:122] "RemoveContainer" containerID="c737a942d8498e9c0f5f94824e2a69553bb3e6ad5d8cc508dd534eea3b00525f" Nov 03 01:28:04.455371 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:04.455325 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f462289d-9ngww" event={"ID":"d06ea1eb-b168-4b5e-be57-cb4e5a45d911","Type":"ContainerStarted","Data":"3956e74d8c81b1a31ac85be391532806d5ba2ccc1c5bd7eacec13186b950eacb"} Nov 03 01:28:04.476237 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:04.476170 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f462289d-9ngww" podStartSLOduration=2.476149265 podStartE2EDuration="2.476149265s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:28:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:28:04.47321735 +0000 UTC m=+2826.644220446" watchObservedRunningTime="2026-11-03 01:28:04.476149265 +0000 UTC m=+2826.647152362" Nov 03 01:28:05.566000 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:05.565964 4421 generic.go:356] "Generic (PLEG): container finished" podID="d06ea1eb-b168-4b5e-be57-cb4e5a45d911" containerID="3956e74d8c81b1a31ac85be391532806d5ba2ccc1c5bd7eacec13186b950eacb" exitCode=0 Nov 03 01:28:05.566685 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:05.566039 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f462289d-9ngww" event={"ID":"d06ea1eb-b168-4b5e-be57-cb4e5a45d911","Type":"ContainerDied","Data":"3956e74d8c81b1a31ac85be391532806d5ba2ccc1c5bd7eacec13186b950eacb"} Nov 03 01:28:06.814730 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:06.814702 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f462289d-9ngww" Nov 03 01:28:06.895391 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:06.895240 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d06ea1eb-b168-4b5e-be57-cb4e5a45d911-kube-api-access-4hbc2\" (UniqueName: \"kubernetes.io/projected/d06ea1eb-b168-4b5e-be57-cb4e5a45d911-kube-api-access-4hbc2\") pod \"d06ea1eb-b168-4b5e-be57-cb4e5a45d911\" (UID: \"d06ea1eb-b168-4b5e-be57-cb4e5a45d911\") " Nov 03 01:28:06.895923 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:06.895883 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d06ea1eb-b168-4b5e-be57-cb4e5a45d911-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d06ea1eb-b168-4b5e-be57-cb4e5a45d911-maas-api-openshift-ca\") pod \"d06ea1eb-b168-4b5e-be57-cb4e5a45d911\" (UID: \"d06ea1eb-b168-4b5e-be57-cb4e5a45d911\") " Nov 03 01:28:06.896781 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:06.896443 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d06ea1eb-b168-4b5e-be57-cb4e5a45d911-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d06ea1eb-b168-4b5e-be57-cb4e5a45d911-maas-api-xks-ca\") pod \"d06ea1eb-b168-4b5e-be57-cb4e5a45d911\" (UID: \"d06ea1eb-b168-4b5e-be57-cb4e5a45d911\") " Nov 03 01:28:06.896884 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:06.896811 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d06ea1eb-b168-4b5e-be57-cb4e5a45d911-maas-api-openshift-ca" pod "d06ea1eb-b168-4b5e-be57-cb4e5a45d911" (UID: "d06ea1eb-b168-4b5e-be57-cb4e5a45d911"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:28:06.897832 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:06.897811 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d06ea1eb-b168-4b5e-be57-cb4e5a45d911-maas-api-openshift-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:28:06.901878 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:06.901832 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d06ea1eb-b168-4b5e-be57-cb4e5a45d911-kube-api-access-4hbc2" pod "d06ea1eb-b168-4b5e-be57-cb4e5a45d911" (UID: "d06ea1eb-b168-4b5e-be57-cb4e5a45d911"). InnerVolumeSpecName "kube-api-access-4hbc2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:28:06.904802 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:06.904760 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d06ea1eb-b168-4b5e-be57-cb4e5a45d911-maas-api-xks-ca" pod "d06ea1eb-b168-4b5e-be57-cb4e5a45d911" (UID: "d06ea1eb-b168-4b5e-be57-cb4e5a45d911"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:28:06.999695 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:06.999656 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4hbc2\" (UniqueName: \"kubernetes.io/projected/d06ea1eb-b168-4b5e-be57-cb4e5a45d911-kube-api-access-4hbc2\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:28:06.999695 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:06.999697 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d06ea1eb-b168-4b5e-be57-cb4e5a45d911-maas-api-xks-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:28:07.588186 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:07.588153 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f462289d-9ngww" Nov 03 01:28:07.588186 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:07.588169 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f462289d-9ngww" event={"ID":"d06ea1eb-b168-4b5e-be57-cb4e5a45d911","Type":"ContainerDied","Data":"1472b24c4f33486bfc7e3b4bc1d05cb75882cddf7c5a976e1c140daae66e11e7"} Nov 03 01:28:07.588504 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:07.588223 4421 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1472b24c4f33486bfc7e3b4bc1d05cb75882cddf7c5a976e1c140daae66e11e7" Nov 03 01:28:08.418524 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:08.418468 4421 patch_prober.go:29] interesting pod/e2e-inf-b-5bfc51-openshift-default-54d8846889-sc6sj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 01:28:08.418948 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:08.418560 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-5bfc51-openshift-default-54d8846889-sc6sj" podUID="781e1acc-4bc9-489f-8df6-6b5f8c10ba1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 01:28:13.684541 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:13.684447 4421 generic.go:356] "Generic (PLEG): container finished" podID="781e1acc-4bc9-489f-8df6-6b5f8c10ba1b" containerID="39421a5a8e7703794bb42341d5ced6151daf742935511581e9300f6e52d11b57" exitCode=0 Nov 03 01:28:13.685082 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:13.684669 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5bfc51-openshift-default-54d8846889-sc6sj" event={"ID":"781e1acc-4bc9-489f-8df6-6b5f8c10ba1b","Type":"ContainerDied","Data":"39421a5a8e7703794bb42341d5ced6151daf742935511581e9300f6e52d11b57"} Nov 03 01:28:13.685335 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:13.685284 4421 scope.go:122] "RemoveContainer" containerID="39421a5a8e7703794bb42341d5ced6151daf742935511581e9300f6e52d11b57" Nov 03 01:28:13.692694 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:13.689347 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:28:13.692694 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:13.689447 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:28:13.692694 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:13.689497 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:28:14.677319 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:28:14.677241 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 01:28:14.698074 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:14.698042 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5bfc51-openshift-default-54d8846889-sc6sj" event={"ID":"781e1acc-4bc9-489f-8df6-6b5f8c10ba1b","Type":"ContainerStarted","Data":"c4a61a0ded25ac4ec91681148656a34e9d166be8de77bc84446b246b23a8b181"} Nov 03 01:28:14.723736 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:14.723703 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-5bfc51-openshift-default-54d8846889-sc6sj" Nov 03 01:28:14.723950 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:14.723812 4421 patch_prober.go:29] interesting pod/e2e-inf-b-5bfc51-openshift-default-54d8846889-sc6sj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.66:15021/healthz/ready\": dial tcp 10.134.1.66:15021: connect: connection refused" start-of-body= Nov 03 01:28:14.723950 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:14.723858 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5bfc51-openshift-default-54d8846889-sc6sj" podUID="781e1acc-4bc9-489f-8df6-6b5f8c10ba1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.66:15021/healthz/ready\": dial tcp 10.134.1.66:15021: connect: connection refused" Nov 03 01:28:15.729299 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:15.724570 4421 patch_prober.go:29] interesting pod/e2e-inf-b-5bfc51-openshift-default-54d8846889-sc6sj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.66:15021/healthz/ready\": dial tcp 10.134.1.66:15021: connect: connection refused" start-of-body= Nov 03 01:28:15.729299 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:15.724648 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5bfc51-openshift-default-54d8846889-sc6sj" podUID="781e1acc-4bc9-489f-8df6-6b5f8c10ba1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.66:15021/healthz/ready\": dial tcp 10.134.1.66:15021: connect: connection refused" Nov 03 01:28:16.740296 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:16.729717 4421 patch_prober.go:29] interesting pod/e2e-inf-b-5bfc51-openshift-default-54d8846889-sc6sj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.66:15021/healthz/ready\": dial tcp 10.134.1.66:15021: connect: connection refused" start-of-body= Nov 03 01:28:16.740296 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:16.729811 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5bfc51-openshift-default-54d8846889-sc6sj" podUID="781e1acc-4bc9-489f-8df6-6b5f8c10ba1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.66:15021/healthz/ready\": dial tcp 10.134.1.66:15021: connect: connection refused" Nov 03 01:28:17.727298 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:17.726740 4421 patch_prober.go:29] interesting pod/e2e-inf-b-5bfc51-openshift-default-54d8846889-sc6sj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.66:15021/healthz/ready\": dial tcp 10.134.1.66:15021: connect: connection refused" start-of-body= Nov 03 01:28:17.727298 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:17.726821 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5bfc51-openshift-default-54d8846889-sc6sj" podUID="781e1acc-4bc9-489f-8df6-6b5f8c10ba1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.66:15021/healthz/ready\": dial tcp 10.134.1.66:15021: connect: connection refused" Nov 03 01:28:18.731137 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:18.730685 4421 patch_prober.go:29] interesting pod/e2e-inf-b-5bfc51-openshift-default-54d8846889-sc6sj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.66:15021/healthz/ready\": dial tcp 10.134.1.66:15021: connect: connection refused" start-of-body= Nov 03 01:28:18.731137 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:18.730747 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5bfc51-openshift-default-54d8846889-sc6sj" podUID="781e1acc-4bc9-489f-8df6-6b5f8c10ba1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.66:15021/healthz/ready\": dial tcp 10.134.1.66:15021: connect: connection refused" Nov 03 01:28:18.784506 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:18.777112 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a88862-57bd8764b7-vsv9f"] Nov 03 01:28:19.728298 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:19.727554 4421 patch_prober.go:29] interesting pod/e2e-inf-b-5bfc51-openshift-default-54d8846889-sc6sj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.66:15021/healthz/ready\": dial tcp 10.134.1.66:15021: connect: connection refused" start-of-body= Nov 03 01:28:19.728298 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:19.727630 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5bfc51-openshift-default-54d8846889-sc6sj" podUID="781e1acc-4bc9-489f-8df6-6b5f8c10ba1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.66:15021/healthz/ready\": dial tcp 10.134.1.66:15021: connect: connection refused" Nov 03 01:28:20.730301 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:20.728007 4421 patch_prober.go:29] interesting pod/e2e-inf-b-5bfc51-openshift-default-54d8846889-sc6sj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.66:15021/healthz/ready\": dial tcp 10.134.1.66:15021: connect: connection refused" start-of-body= Nov 03 01:28:20.730301 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:20.728085 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5bfc51-openshift-default-54d8846889-sc6sj" podUID="781e1acc-4bc9-489f-8df6-6b5f8c10ba1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.66:15021/healthz/ready\": dial tcp 10.134.1.66:15021: connect: connection refused" Nov 03 01:28:21.729927 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:21.729896 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-5bfc51-openshift-default-54d8846889-sc6sj" Nov 03 01:28:21.730415 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:21.730387 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-5bfc51-openshift-default-54d8846889-sc6sj" Nov 03 01:28:21.731173 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:21.731152 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-5bfc51-openshift-default-54d8846889-sc6sj" Nov 03 01:28:23.800400 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:23.800173 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a88862-57bd8764b7-vsv9f" podUID="867b2cbf-fc7e-4fc0-816b-53cee0b424f8" containerName="payload-pre-processing" containerID="cri-o://fe21c6cab8d4b8ef2165d437a570c2861d159905c4f18047116e3004cb23c12c" gracePeriod=25 Nov 03 01:28:27.527452 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:27.527415 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a88862-57bd8764b7-vsv9f" podUID="867b2cbf-fc7e-4fc0-816b-53cee0b424f8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:28:33.651657 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:33.651609 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ce04af-6b9b884869-vwgxv"] Nov 03 01:28:33.660221 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:33.660186 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ce04af-ffcb7d45f-zdv8k"] Nov 03 01:28:37.526728 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:37.526692 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a88862-57bd8764b7-vsv9f" podUID="867b2cbf-fc7e-4fc0-816b-53cee0b424f8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:28:38.652179 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:38.652127 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-ce04af-6b9b884869-vwgxv" podUID="dbd53b13-13c5-449a-b204-1fc94ac8afad" containerName="payload-processing" containerID="cri-o://b6e88418322adb40448b9fb23445d8522c2e6a7a794c295be254f5877f4e5eec" gracePeriod=25 Nov 03 01:28:38.660692 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:38.660653 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ce04af-ffcb7d45f-zdv8k" podUID="ee89f011-059f-493a-becf-4816c5550dbf" containerName="payload-pre-processing" containerID="cri-o://9013783a056208fe3d0b1d218151a61a9299c9219cb05a7de4b8a6f7d8662b42" gracePeriod=25 Nov 03 01:28:40.229139 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:40.229096 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-fcw4z"] Nov 03 01:28:40.229993 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:40.229955 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-fcw4z" podUID="37f37c99-0b3d-45d2-ba18-a4ec39043f67" containerName="istio-proxy" containerID="cri-o://788021faebbc3370b311a69120db115daab85538b593570e04d4d813cc0ceb6d" gracePeriod=30 Nov 03 01:28:44.142095 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:44.142065 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a88862-57bd8764b7-vsv9f" Nov 03 01:28:44.232056 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:44.232015 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/867b2cbf-fc7e-4fc0-816b-53cee0b424f8-kube-api-access-whpv8\" (UniqueName: \"kubernetes.io/projected/867b2cbf-fc7e-4fc0-816b-53cee0b424f8-kube-api-access-whpv8\") pod \"867b2cbf-fc7e-4fc0-816b-53cee0b424f8\" (UID: \"867b2cbf-fc7e-4fc0-816b-53cee0b424f8\") " Nov 03 01:28:44.232339 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:44.232314 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/867b2cbf-fc7e-4fc0-816b-53cee0b424f8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/867b2cbf-fc7e-4fc0-816b-53cee0b424f8-plugins-config-volume\") pod \"867b2cbf-fc7e-4fc0-816b-53cee0b424f8\" (UID: \"867b2cbf-fc7e-4fc0-816b-53cee0b424f8\") " Nov 03 01:28:44.232802 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:44.232773 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/867b2cbf-fc7e-4fc0-816b-53cee0b424f8-plugins-config-volume" pod "867b2cbf-fc7e-4fc0-816b-53cee0b424f8" (UID: "867b2cbf-fc7e-4fc0-816b-53cee0b424f8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:28:44.233125 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:44.233102 4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/867b2cbf-fc7e-4fc0-816b-53cee0b424f8-plugins-config-volume\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:28:44.234361 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:44.234331 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/867b2cbf-fc7e-4fc0-816b-53cee0b424f8-kube-api-access-whpv8" pod "867b2cbf-fc7e-4fc0-816b-53cee0b424f8" (UID: "867b2cbf-fc7e-4fc0-816b-53cee0b424f8"). InnerVolumeSpecName "kube-api-access-whpv8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:28:44.309169 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:44.309135 4421 generic.go:356] "Generic (PLEG): container finished" podID="867b2cbf-fc7e-4fc0-816b-53cee0b424f8" containerID="fe21c6cab8d4b8ef2165d437a570c2861d159905c4f18047116e3004cb23c12c" exitCode=0 Nov 03 01:28:44.309391 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:44.309178 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a88862-57bd8764b7-vsv9f" Nov 03 01:28:44.309391 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:44.309220 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a88862-57bd8764b7-vsv9f" event={"ID":"867b2cbf-fc7e-4fc0-816b-53cee0b424f8","Type":"ContainerDied","Data":"fe21c6cab8d4b8ef2165d437a570c2861d159905c4f18047116e3004cb23c12c"} Nov 03 01:28:44.309391 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:44.309288 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a88862-57bd8764b7-vsv9f" event={"ID":"867b2cbf-fc7e-4fc0-816b-53cee0b424f8","Type":"ContainerDied","Data":"ccf8dcfe86fb69d38ad2b46a8a7d0f8ec4e917316c54704a22de795e67883c91"} Nov 03 01:28:44.309391 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:44.309317 4421 scope.go:122] "RemoveContainer" containerID="fe21c6cab8d4b8ef2165d437a570c2861d159905c4f18047116e3004cb23c12c" Nov 03 01:28:44.322322 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:44.322266 4421 scope.go:122] "RemoveContainer" containerID="fe21c6cab8d4b8ef2165d437a570c2861d159905c4f18047116e3004cb23c12c" Nov 03 01:28:44.322649 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:28:44.322630 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe21c6cab8d4b8ef2165d437a570c2861d159905c4f18047116e3004cb23c12c\": container with ID starting with fe21c6cab8d4b8ef2165d437a570c2861d159905c4f18047116e3004cb23c12c not found: ID does not exist" containerID="fe21c6cab8d4b8ef2165d437a570c2861d159905c4f18047116e3004cb23c12c" Nov 03 01:28:44.322716 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:44.322657 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe21c6cab8d4b8ef2165d437a570c2861d159905c4f18047116e3004cb23c12c"} err="failed to get container status \"fe21c6cab8d4b8ef2165d437a570c2861d159905c4f18047116e3004cb23c12c\": rpc error: code = NotFound desc = could not find container \"fe21c6cab8d4b8ef2165d437a570c2861d159905c4f18047116e3004cb23c12c\": container with ID starting with fe21c6cab8d4b8ef2165d437a570c2861d159905c4f18047116e3004cb23c12c not found: ID does not exist" Nov 03 01:28:44.335088 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:44.335053 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whpv8\" (UniqueName: \"kubernetes.io/projected/867b2cbf-fc7e-4fc0-816b-53cee0b424f8-kube-api-access-whpv8\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:28:44.336263 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:44.336240 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a88862-57bd8764b7-vsv9f"] Nov 03 01:28:44.341664 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:44.341644 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a88862-57bd8764b7-vsv9f"] Nov 03 01:28:44.673893 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:28:44.673856 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 01:28:45.548865 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:45.548839 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-fcw4z" Nov 03 01:28:45.561897 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:45.561872 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37f37c99-0b3d-45d2-ba18-a4ec39043f67-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37f37c99-0b3d-45d2-ba18-a4ec39043f67-istio-envoy\") pod \"37f37c99-0b3d-45d2-ba18-a4ec39043f67\" (UID: \"37f37c99-0b3d-45d2-ba18-a4ec39043f67\") " Nov 03 01:28:45.562048 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:45.561916 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37f37c99-0b3d-45d2-ba18-a4ec39043f67-istio-token\" (UniqueName: \"kubernetes.io/projected/37f37c99-0b3d-45d2-ba18-a4ec39043f67-istio-token\") pod \"37f37c99-0b3d-45d2-ba18-a4ec39043f67\" (UID: \"37f37c99-0b3d-45d2-ba18-a4ec39043f67\") " Nov 03 01:28:45.562048 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:45.561943 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37f37c99-0b3d-45d2-ba18-a4ec39043f67-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37f37c99-0b3d-45d2-ba18-a4ec39043f67-istio-data\") pod \"37f37c99-0b3d-45d2-ba18-a4ec39043f67\" (UID: \"37f37c99-0b3d-45d2-ba18-a4ec39043f67\") " Nov 03 01:28:45.562048 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:45.562012 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/37f37c99-0b3d-45d2-ba18-a4ec39043f67-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37f37c99-0b3d-45d2-ba18-a4ec39043f67-istio-podinfo\") pod \"37f37c99-0b3d-45d2-ba18-a4ec39043f67\" (UID: \"37f37c99-0b3d-45d2-ba18-a4ec39043f67\") " Nov 03 01:28:45.562163 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:45.562052 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37f37c99-0b3d-45d2-ba18-a4ec39043f67-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37f37c99-0b3d-45d2-ba18-a4ec39043f67-workload-certs\") pod \"37f37c99-0b3d-45d2-ba18-a4ec39043f67\" (UID: \"37f37c99-0b3d-45d2-ba18-a4ec39043f67\") " Nov 03 01:28:45.562199 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:45.562178 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37f37c99-0b3d-45d2-ba18-a4ec39043f67-kube-api-access-tczf6\" (UniqueName: \"kubernetes.io/projected/37f37c99-0b3d-45d2-ba18-a4ec39043f67-kube-api-access-tczf6\") pod \"37f37c99-0b3d-45d2-ba18-a4ec39043f67\" (UID: \"37f37c99-0b3d-45d2-ba18-a4ec39043f67\") " Nov 03 01:28:45.562238 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:45.562217 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37f37c99-0b3d-45d2-ba18-a4ec39043f67-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37f37c99-0b3d-45d2-ba18-a4ec39043f67-credential-socket\") pod \"37f37c99-0b3d-45d2-ba18-a4ec39043f67\" (UID: \"37f37c99-0b3d-45d2-ba18-a4ec39043f67\") " Nov 03 01:28:45.562364 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:45.562332 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37f37c99-0b3d-45d2-ba18-a4ec39043f67-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37f37c99-0b3d-45d2-ba18-a4ec39043f67-istiod-ca-cert\") pod \"37f37c99-0b3d-45d2-ba18-a4ec39043f67\" (UID: \"37f37c99-0b3d-45d2-ba18-a4ec39043f67\") " Nov 03 01:28:45.562812 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:45.562739 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37f37c99-0b3d-45d2-ba18-a4ec39043f67-istio-data" pod "37f37c99-0b3d-45d2-ba18-a4ec39043f67" (UID: "37f37c99-0b3d-45d2-ba18-a4ec39043f67"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:28:45.562812 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:45.562772 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37f37c99-0b3d-45d2-ba18-a4ec39043f67-workload-certs" pod "37f37c99-0b3d-45d2-ba18-a4ec39043f67" (UID: "37f37c99-0b3d-45d2-ba18-a4ec39043f67"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:28:45.563283 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:45.563244 4421 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37f37c99-0b3d-45d2-ba18-a4ec39043f67-workload-certs\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:28:45.563373 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:45.563323 4421 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37f37c99-0b3d-45d2-ba18-a4ec39043f67-istio-data\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:28:45.563373 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:45.563248 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37f37c99-0b3d-45d2-ba18-a4ec39043f67-credential-socket" pod "37f37c99-0b3d-45d2-ba18-a4ec39043f67" (UID: "37f37c99-0b3d-45d2-ba18-a4ec39043f67"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:28:45.563456 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:45.563401 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37f37c99-0b3d-45d2-ba18-a4ec39043f67-istiod-ca-cert" pod "37f37c99-0b3d-45d2-ba18-a4ec39043f67" (UID: "37f37c99-0b3d-45d2-ba18-a4ec39043f67"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:28:45.564807 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:45.564784 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37f37c99-0b3d-45d2-ba18-a4ec39043f67-kube-api-access-tczf6" pod "37f37c99-0b3d-45d2-ba18-a4ec39043f67" (UID: "37f37c99-0b3d-45d2-ba18-a4ec39043f67"). InnerVolumeSpecName "kube-api-access-tczf6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:28:45.564921 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:45.564843 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37f37c99-0b3d-45d2-ba18-a4ec39043f67-istio-token" pod "37f37c99-0b3d-45d2-ba18-a4ec39043f67" (UID: "37f37c99-0b3d-45d2-ba18-a4ec39043f67"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:28:45.565023 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:45.565005 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37f37c99-0b3d-45d2-ba18-a4ec39043f67-istio-envoy" pod "37f37c99-0b3d-45d2-ba18-a4ec39043f67" (UID: "37f37c99-0b3d-45d2-ba18-a4ec39043f67"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:28:45.565116 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:45.565095 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/37f37c99-0b3d-45d2-ba18-a4ec39043f67-istio-podinfo" pod "37f37c99-0b3d-45d2-ba18-a4ec39043f67" (UID: "37f37c99-0b3d-45d2-ba18-a4ec39043f67"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:28:45.626679 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:45.588878 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-x8tb9"] Nov 03 01:28:45.626679 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:45.591080 4421 cpu_manager.go:414] "containerMap: removing container" podUID="37f37c99-0b3d-45d2-ba18-a4ec39043f67" containerName="istio-proxy" Nov 03 01:28:45.626679 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:45.591103 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37f37c99-0b3d-45d2-ba18-a4ec39043f67" containerName="istio-proxy" Nov 03 01:28:45.626679 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:45.591130 4421 cpu_manager.go:414] "containerMap: removing container" podUID="867b2cbf-fc7e-4fc0-816b-53cee0b424f8" containerName="payload-pre-processing" Nov 03 01:28:45.626679 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:45.591138 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="867b2cbf-fc7e-4fc0-816b-53cee0b424f8" containerName="payload-pre-processing" Nov 03 01:28:45.626679 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:45.591208 4421 cpu_manager.go:414] "containerMap: removing container" podUID="37f37c99-0b3d-45d2-ba18-a4ec39043f67" containerName="istio-proxy" Nov 03 01:28:45.626679 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:45.591216 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37f37c99-0b3d-45d2-ba18-a4ec39043f67" containerName="istio-proxy" Nov 03 01:28:45.626679 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:45.591246 4421 cpu_manager.go:414] "containerMap: removing container" podUID="d06ea1eb-b168-4b5e-be57-cb4e5a45d911" containerName="revoke-keys" Nov 03 01:28:45.626679 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:45.591253 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d06ea1eb-b168-4b5e-be57-cb4e5a45d911" containerName="revoke-keys" Nov 03 01:28:45.626679 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:45.591466 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="37f37c99-0b3d-45d2-ba18-a4ec39043f67" containerName="istio-proxy" Nov 03 01:28:45.626679 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:45.591482 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="867b2cbf-fc7e-4fc0-816b-53cee0b424f8" containerName="payload-pre-processing" Nov 03 01:28:45.626679 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:45.591502 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="37f37c99-0b3d-45d2-ba18-a4ec39043f67" containerName="istio-proxy" Nov 03 01:28:45.626679 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:45.591519 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="d06ea1eb-b168-4b5e-be57-cb4e5a45d911" containerName="revoke-keys" Nov 03 01:28:45.626679 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:45.607539 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-x8tb9"] Nov 03 01:28:45.626679 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:45.607663 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-x8tb9" Nov 03 01:28:45.664130 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:45.664097 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37f37c99-0b3d-45d2-ba18-a4ec39043f67-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37f37c99-0b3d-45d2-ba18-a4ec39043f67-workload-socket\") pod \"37f37c99-0b3d-45d2-ba18-a4ec39043f67\" (UID: \"37f37c99-0b3d-45d2-ba18-a4ec39043f67\") " Nov 03 01:28:45.664820 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:45.664720 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37f37c99-0b3d-45d2-ba18-a4ec39043f67-workload-socket" pod "37f37c99-0b3d-45d2-ba18-a4ec39043f67" (UID: "37f37c99-0b3d-45d2-ba18-a4ec39043f67"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:28:45.666854 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:45.666611 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79c7fff7-62df-44a2-aa3e-f24696d71b03-istio-podinfo\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-x8tb9\" (UID: \"79c7fff7-62df-44a2-aa3e-f24696d71b03\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-x8tb9" Nov 03 01:28:45.666854 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:45.666692 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdbdb\" (UniqueName: \"kubernetes.io/projected/79c7fff7-62df-44a2-aa3e-f24696d71b03-kube-api-access-kdbdb\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-x8tb9\" (UID: \"79c7fff7-62df-44a2-aa3e-f24696d71b03\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-x8tb9" Nov 03 01:28:45.666854 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:45.666733 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79c7fff7-62df-44a2-aa3e-f24696d71b03-workload-certs\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-x8tb9\" (UID: \"79c7fff7-62df-44a2-aa3e-f24696d71b03\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-x8tb9" Nov 03 01:28:45.666854 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:45.666783 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79c7fff7-62df-44a2-aa3e-f24696d71b03-istiod-ca-cert\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-x8tb9\" (UID: \"79c7fff7-62df-44a2-aa3e-f24696d71b03\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-x8tb9" Nov 03 01:28:45.667428 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:45.666899 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79c7fff7-62df-44a2-aa3e-f24696d71b03-istio-data\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-x8tb9\" (UID: \"79c7fff7-62df-44a2-aa3e-f24696d71b03\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-x8tb9" Nov 03 01:28:45.667428 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:45.666941 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79c7fff7-62df-44a2-aa3e-f24696d71b03-istio-token\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-x8tb9\" (UID: \"79c7fff7-62df-44a2-aa3e-f24696d71b03\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-x8tb9" Nov 03 01:28:45.667428 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:45.667024 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79c7fff7-62df-44a2-aa3e-f24696d71b03-credential-socket\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-x8tb9\" (UID: \"79c7fff7-62df-44a2-aa3e-f24696d71b03\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-x8tb9" Nov 03 01:28:45.667428 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:45.667134 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79c7fff7-62df-44a2-aa3e-f24696d71b03-workload-socket\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-x8tb9\" (UID: \"79c7fff7-62df-44a2-aa3e-f24696d71b03\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-x8tb9" Nov 03 01:28:45.667428 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:45.667212 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79c7fff7-62df-44a2-aa3e-f24696d71b03-istio-envoy\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-x8tb9\" (UID: \"79c7fff7-62df-44a2-aa3e-f24696d71b03\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-x8tb9" Nov 03 01:28:45.667428 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:45.667372 4421 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37f37c99-0b3d-45d2-ba18-a4ec39043f67-istio-podinfo\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:28:45.667428 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:45.667394 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tczf6\" (UniqueName: \"kubernetes.io/projected/37f37c99-0b3d-45d2-ba18-a4ec39043f67-kube-api-access-tczf6\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:28:45.667428 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:45.667409 4421 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37f37c99-0b3d-45d2-ba18-a4ec39043f67-credential-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:28:45.667798 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:45.667423 4421 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37f37c99-0b3d-45d2-ba18-a4ec39043f67-istiod-ca-cert\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:28:45.667798 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:45.667456 4421 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37f37c99-0b3d-45d2-ba18-a4ec39043f67-workload-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:28:45.667798 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:45.667472 4421 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37f37c99-0b3d-45d2-ba18-a4ec39043f67-istio-envoy\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:28:45.667798 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:45.667488 4421 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37f37c99-0b3d-45d2-ba18-a4ec39043f67-istio-token\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:28:45.768967 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:45.768859 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79c7fff7-62df-44a2-aa3e-f24696d71b03-credential-socket\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-x8tb9\" (UID: \"79c7fff7-62df-44a2-aa3e-f24696d71b03\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-x8tb9" Nov 03 01:28:45.769151 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:45.768993 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79c7fff7-62df-44a2-aa3e-f24696d71b03-workload-socket\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-x8tb9\" (UID: \"79c7fff7-62df-44a2-aa3e-f24696d71b03\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-x8tb9" Nov 03 01:28:45.769151 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:45.769068 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79c7fff7-62df-44a2-aa3e-f24696d71b03-istio-envoy\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-x8tb9\" (UID: \"79c7fff7-62df-44a2-aa3e-f24696d71b03\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-x8tb9" Nov 03 01:28:45.769334 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:45.769226 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79c7fff7-62df-44a2-aa3e-f24696d71b03-istio-podinfo\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-x8tb9\" (UID: \"79c7fff7-62df-44a2-aa3e-f24696d71b03\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-x8tb9" Nov 03 01:28:45.769421 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:45.769322 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79c7fff7-62df-44a2-aa3e-f24696d71b03-credential-socket\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-x8tb9\" (UID: \"79c7fff7-62df-44a2-aa3e-f24696d71b03\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-x8tb9" Nov 03 01:28:45.769421 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:45.769352 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdbdb\" (UniqueName: \"kubernetes.io/projected/79c7fff7-62df-44a2-aa3e-f24696d71b03-kube-api-access-kdbdb\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-x8tb9\" (UID: \"79c7fff7-62df-44a2-aa3e-f24696d71b03\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-x8tb9" Nov 03 01:28:45.769421 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:45.769395 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79c7fff7-62df-44a2-aa3e-f24696d71b03-workload-certs\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-x8tb9\" (UID: \"79c7fff7-62df-44a2-aa3e-f24696d71b03\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-x8tb9" Nov 03 01:28:45.769605 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:45.769452 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79c7fff7-62df-44a2-aa3e-f24696d71b03-istiod-ca-cert\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-x8tb9\" (UID: \"79c7fff7-62df-44a2-aa3e-f24696d71b03\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-x8tb9" Nov 03 01:28:45.769605 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:45.769574 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79c7fff7-62df-44a2-aa3e-f24696d71b03-istio-data\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-x8tb9\" (UID: \"79c7fff7-62df-44a2-aa3e-f24696d71b03\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-x8tb9" Nov 03 01:28:45.769605 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:45.769593 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79c7fff7-62df-44a2-aa3e-f24696d71b03-workload-socket\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-x8tb9\" (UID: \"79c7fff7-62df-44a2-aa3e-f24696d71b03\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-x8tb9" Nov 03 01:28:45.769781 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:45.769639 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79c7fff7-62df-44a2-aa3e-f24696d71b03-istio-token\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-x8tb9\" (UID: \"79c7fff7-62df-44a2-aa3e-f24696d71b03\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-x8tb9" Nov 03 01:28:45.770027 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:45.769919 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79c7fff7-62df-44a2-aa3e-f24696d71b03-istio-data\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-x8tb9\" (UID: \"79c7fff7-62df-44a2-aa3e-f24696d71b03\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-x8tb9" Nov 03 01:28:45.770382 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:45.770306 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79c7fff7-62df-44a2-aa3e-f24696d71b03-workload-certs\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-x8tb9\" (UID: \"79c7fff7-62df-44a2-aa3e-f24696d71b03\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-x8tb9" Nov 03 01:28:45.772128 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:45.771373 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79c7fff7-62df-44a2-aa3e-f24696d71b03-istiod-ca-cert\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-x8tb9\" (UID: \"79c7fff7-62df-44a2-aa3e-f24696d71b03\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-x8tb9" Nov 03 01:28:45.772757 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:45.772729 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79c7fff7-62df-44a2-aa3e-f24696d71b03-istio-envoy\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-x8tb9\" (UID: \"79c7fff7-62df-44a2-aa3e-f24696d71b03\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-x8tb9" Nov 03 01:28:45.774225 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:45.774200 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79c7fff7-62df-44a2-aa3e-f24696d71b03-istio-podinfo\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-x8tb9\" (UID: \"79c7fff7-62df-44a2-aa3e-f24696d71b03\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-x8tb9" Nov 03 01:28:45.782039 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:45.781657 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79c7fff7-62df-44a2-aa3e-f24696d71b03-istio-token\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-x8tb9\" (UID: \"79c7fff7-62df-44a2-aa3e-f24696d71b03\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-x8tb9" Nov 03 01:28:45.782575 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:45.782549 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdbdb\" (UniqueName: \"kubernetes.io/projected/79c7fff7-62df-44a2-aa3e-f24696d71b03-kube-api-access-kdbdb\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-x8tb9\" (UID: \"79c7fff7-62df-44a2-aa3e-f24696d71b03\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-x8tb9" Nov 03 01:28:45.983300 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:45.979489 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-x8tb9" Nov 03 01:28:46.035452 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:46.035379 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="867b2cbf-fc7e-4fc0-816b-53cee0b424f8" path="/var/lib/kubelet/pods/867b2cbf-fc7e-4fc0-816b-53cee0b424f8/volumes" Nov 03 01:28:46.315168 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:28:46.315133 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79c7fff7_62df_44a2_aa3e_f24696d71b03.slice/crio-209ca632afd1755e2e87cd58903e10b1dc1a7ef524d009cf8a7740a13727b94d WatchSource:0}: Error finding container 209ca632afd1755e2e87cd58903e10b1dc1a7ef524d009cf8a7740a13727b94d: Status 404 returned error can't find the container with id 209ca632afd1755e2e87cd58903e10b1dc1a7ef524d009cf8a7740a13727b94d Nov 03 01:28:46.316341 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:46.316308 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-x8tb9"] Nov 03 01:28:46.318871 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:46.318838 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:28:46.318972 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:46.318908 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:28:46.318972 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:46.318938 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:28:46.331391 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:46.331347 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-x8tb9" event={"ID":"79c7fff7-62df-44a2-aa3e-f24696d71b03","Type":"ContainerStarted","Data":"209ca632afd1755e2e87cd58903e10b1dc1a7ef524d009cf8a7740a13727b94d"} Nov 03 01:28:46.333444 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:46.333415 4421 generic.go:356] "Generic (PLEG): container finished" podID="37f37c99-0b3d-45d2-ba18-a4ec39043f67" containerID="788021faebbc3370b311a69120db115daab85538b593570e04d4d813cc0ceb6d" exitCode=0 Nov 03 01:28:46.333603 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:46.333480 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-fcw4z" Nov 03 01:28:46.333603 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:46.333498 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-fcw4z" event={"ID":"37f37c99-0b3d-45d2-ba18-a4ec39043f67","Type":"ContainerDied","Data":"788021faebbc3370b311a69120db115daab85538b593570e04d4d813cc0ceb6d"} Nov 03 01:28:46.333603 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:46.333552 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-fcw4z" event={"ID":"37f37c99-0b3d-45d2-ba18-a4ec39043f67","Type":"ContainerDied","Data":"149dcfe320145bc60419ad9b487294d4a88a083d369794664f199913008797d0"} Nov 03 01:28:46.333603 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:46.333576 4421 scope.go:122] "RemoveContainer" containerID="788021faebbc3370b311a69120db115daab85538b593570e04d4d813cc0ceb6d" Nov 03 01:28:46.348983 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:46.348924 4421 scope.go:122] "RemoveContainer" containerID="cb413e1300f544506216d31332bc15a5a0787f3e218711849e2e819ef25dba88" Nov 03 01:28:46.358224 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:46.358197 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-fcw4z"] Nov 03 01:28:46.363740 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:46.363718 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-fcw4z"] Nov 03 01:28:46.373512 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:46.373486 4421 scope.go:122] "RemoveContainer" containerID="788021faebbc3370b311a69120db115daab85538b593570e04d4d813cc0ceb6d" Nov 03 01:28:46.373842 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:28:46.373823 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"788021faebbc3370b311a69120db115daab85538b593570e04d4d813cc0ceb6d\": container with ID starting with 788021faebbc3370b311a69120db115daab85538b593570e04d4d813cc0ceb6d not found: ID does not exist" containerID="788021faebbc3370b311a69120db115daab85538b593570e04d4d813cc0ceb6d" Nov 03 01:28:46.373900 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:46.373850 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"788021faebbc3370b311a69120db115daab85538b593570e04d4d813cc0ceb6d"} err="failed to get container status \"788021faebbc3370b311a69120db115daab85538b593570e04d4d813cc0ceb6d\": rpc error: code = NotFound desc = could not find container \"788021faebbc3370b311a69120db115daab85538b593570e04d4d813cc0ceb6d\": container with ID starting with 788021faebbc3370b311a69120db115daab85538b593570e04d4d813cc0ceb6d not found: ID does not exist" Nov 03 01:28:46.373900 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:46.373866 4421 scope.go:122] "RemoveContainer" containerID="cb413e1300f544506216d31332bc15a5a0787f3e218711849e2e819ef25dba88" Nov 03 01:28:46.374143 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:28:46.374125 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb413e1300f544506216d31332bc15a5a0787f3e218711849e2e819ef25dba88\": container with ID starting with cb413e1300f544506216d31332bc15a5a0787f3e218711849e2e819ef25dba88 not found: ID does not exist" containerID="cb413e1300f544506216d31332bc15a5a0787f3e218711849e2e819ef25dba88" Nov 03 01:28:46.374192 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:46.374151 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb413e1300f544506216d31332bc15a5a0787f3e218711849e2e819ef25dba88"} err="failed to get container status \"cb413e1300f544506216d31332bc15a5a0787f3e218711849e2e819ef25dba88\": rpc error: code = NotFound desc = could not find container \"cb413e1300f544506216d31332bc15a5a0787f3e218711849e2e819ef25dba88\": container with ID starting with cb413e1300f544506216d31332bc15a5a0787f3e218711849e2e819ef25dba88 not found: ID does not exist" Nov 03 01:28:47.383563 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:47.383518 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-x8tb9" event={"ID":"79c7fff7-62df-44a2-aa3e-f24696d71b03","Type":"ContainerStarted","Data":"d74595c5bdaade18ab20fc49ed721cc7212db6df5232f69a0d362df3e9687ab6"} Nov 03 01:28:47.437604 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:47.437529 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-x8tb9" podStartSLOduration=103.437506689 podStartE2EDuration="1m43.437506689s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:27:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:28:47.404424911 +0000 UTC m=+2869.575428010" watchObservedRunningTime="2026-11-03 01:28:47.437506689 +0000 UTC m=+2869.608509802" Nov 03 01:28:47.982340 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:47.982308 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37f37c99-0b3d-45d2-ba18-a4ec39043f67" path="/var/lib/kubelet/pods/37f37c99-0b3d-45d2-ba18-a4ec39043f67/volumes" Nov 03 01:28:48.026691 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:48.024897 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-x8tb9" Nov 03 01:28:48.027326 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:48.027012 4421 patch_prober.go:29] interesting pod/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-x8tb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.70:15021/healthz/ready\": dial tcp 10.134.1.70:15021: connect: connection refused" start-of-body= Nov 03 01:28:48.027326 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:48.027073 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-x8tb9" podUID="79c7fff7-62df-44a2-aa3e-f24696d71b03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.70:15021/healthz/ready\": dial tcp 10.134.1.70:15021: connect: connection refused" Nov 03 01:28:48.543595 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:48.543550 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ce04af-ffcb7d45f-zdv8k" podUID="ee89f011-059f-493a-becf-4816c5550dbf" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:28:48.543595 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:48.543589 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-ce04af-6b9b884869-vwgxv" podUID="dbd53b13-13c5-449a-b204-1fc94ac8afad" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:28:49.026784 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:49.026744 4421 patch_prober.go:29] interesting pod/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-x8tb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.70:15021/healthz/ready\": dial tcp 10.134.1.70:15021: connect: connection refused" start-of-body= Nov 03 01:28:49.027079 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:49.027049 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-x8tb9" podUID="79c7fff7-62df-44a2-aa3e-f24696d71b03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.70:15021/healthz/ready\": dial tcp 10.134.1.70:15021: connect: connection refused" Nov 03 01:28:50.029452 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:50.029407 4421 patch_prober.go:29] interesting pod/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-x8tb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.70:15021/healthz/ready\": dial tcp 10.134.1.70:15021: connect: connection refused" start-of-body= Nov 03 01:28:50.030047 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:50.030012 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-x8tb9" podUID="79c7fff7-62df-44a2-aa3e-f24696d71b03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.70:15021/healthz/ready\": dial tcp 10.134.1.70:15021: connect: connection refused" Nov 03 01:28:51.029303 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:51.026837 4421 patch_prober.go:29] interesting pod/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-x8tb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.70:15021/healthz/ready\": dial tcp 10.134.1.70:15021: connect: connection refused" start-of-body= Nov 03 01:28:51.029303 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:51.026982 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-x8tb9" podUID="79c7fff7-62df-44a2-aa3e-f24696d71b03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.70:15021/healthz/ready\": dial tcp 10.134.1.70:15021: connect: connection refused" Nov 03 01:28:52.030893 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:52.030849 4421 patch_prober.go:29] interesting pod/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-x8tb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.70:15021/healthz/ready\": dial tcp 10.134.1.70:15021: connect: connection refused" start-of-body= Nov 03 01:28:52.031342 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:52.030925 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-x8tb9" podUID="79c7fff7-62df-44a2-aa3e-f24696d71b03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.70:15021/healthz/ready\": dial tcp 10.134.1.70:15021: connect: connection refused" Nov 03 01:28:53.031004 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:53.030970 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-x8tb9" Nov 03 01:28:53.032334 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:53.031993 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-x8tb9" Nov 03 01:28:53.032704 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:53.032685 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-x8tb9" Nov 03 01:28:58.571598 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:58.571545 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ce04af-ffcb7d45f-zdv8k" podUID="ee89f011-059f-493a-becf-4816c5550dbf" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:28:58.573081 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:58.571812 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-ce04af-6b9b884869-vwgxv" podUID="dbd53b13-13c5-449a-b204-1fc94ac8afad" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:28:59.227396 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:59.225496 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ce04af-ffcb7d45f-zdv8k" Nov 03 01:28:59.265511 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:59.265481 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-ce04af-6b9b884869-vwgxv" Nov 03 01:28:59.302402 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:59.302364 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-5dab6a-openshift-default-6ff495656c-bxn8s"] Nov 03 01:28:59.303813 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:59.303785 4421 cpu_manager.go:414] "containerMap: removing container" podUID="dbd53b13-13c5-449a-b204-1fc94ac8afad" containerName="payload-processing" Nov 03 01:28:59.303932 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:59.303814 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbd53b13-13c5-449a-b204-1fc94ac8afad" containerName="payload-processing" Nov 03 01:28:59.303932 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:59.303836 4421 cpu_manager.go:414] "containerMap: removing container" podUID="ee89f011-059f-493a-becf-4816c5550dbf" containerName="payload-pre-processing" Nov 03 01:28:59.303932 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:59.303844 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee89f011-059f-493a-becf-4816c5550dbf" containerName="payload-pre-processing" Nov 03 01:28:59.304113 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:59.304098 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbd53b13-13c5-449a-b204-1fc94ac8afad" containerName="payload-processing" Nov 03 01:28:59.304154 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:59.304119 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee89f011-059f-493a-becf-4816c5550dbf" containerName="payload-pre-processing" Nov 03 01:28:59.339303 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:59.332681 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5dab6a-openshift-default-6ff495656c-bxn8s" Nov 03 01:28:59.341150 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:59.341074 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-5dab6a-openshift-default-6ff495656c-bxn8s"] Nov 03 01:28:59.345262 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:59.345226 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dbd53b13-13c5-449a-b204-1fc94ac8afad-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbd53b13-13c5-449a-b204-1fc94ac8afad-plugins-config-volume\") pod \"dbd53b13-13c5-449a-b204-1fc94ac8afad\" (UID: \"dbd53b13-13c5-449a-b204-1fc94ac8afad\") " Nov 03 01:28:59.345953 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:59.345924 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbd53b13-13c5-449a-b204-1fc94ac8afad-plugins-config-volume" pod "dbd53b13-13c5-449a-b204-1fc94ac8afad" (UID: "dbd53b13-13c5-449a-b204-1fc94ac8afad"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:28:59.351473 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:59.351426 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-5dab6a-openshift-default-dockercfg-qngwt\"" Nov 03 01:28:59.352484 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:59.352455 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbd53b13-13c5-449a-b204-1fc94ac8afad-kube-api-access-m6tbp" pod "dbd53b13-13c5-449a-b204-1fc94ac8afad" (UID: "dbd53b13-13c5-449a-b204-1fc94ac8afad"). InnerVolumeSpecName "kube-api-access-m6tbp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:28:59.373632 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:59.368347 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbd53b13-13c5-449a-b204-1fc94ac8afad-kube-api-access-m6tbp\" (UniqueName: \"kubernetes.io/projected/dbd53b13-13c5-449a-b204-1fc94ac8afad-kube-api-access-m6tbp\") pod \"dbd53b13-13c5-449a-b204-1fc94ac8afad\" (UID: \"dbd53b13-13c5-449a-b204-1fc94ac8afad\") " Nov 03 01:28:59.373632 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:59.368552 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee89f011-059f-493a-becf-4816c5550dbf-kube-api-access-c7f74\" (UniqueName: \"kubernetes.io/projected/ee89f011-059f-493a-becf-4816c5550dbf-kube-api-access-c7f74\") pod \"ee89f011-059f-493a-becf-4816c5550dbf\" (UID: \"ee89f011-059f-493a-becf-4816c5550dbf\") " Nov 03 01:28:59.373632 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:59.368677 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee89f011-059f-493a-becf-4816c5550dbf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee89f011-059f-493a-becf-4816c5550dbf-plugins-config-volume\") pod \"ee89f011-059f-493a-becf-4816c5550dbf\" (UID: \"ee89f011-059f-493a-becf-4816c5550dbf\") " Nov 03 01:28:59.373632 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:59.369073 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96mbm\" (UniqueName: \"kubernetes.io/projected/e0c12073-7ff8-413c-9940-978f4b600778-kube-api-access-96mbm\") pod \"e2e-inf-b-5dab6a-openshift-default-6ff495656c-bxn8s\" (UID: \"e0c12073-7ff8-413c-9940-978f4b600778\") " pod="openshift-ingress/e2e-inf-b-5dab6a-openshift-default-6ff495656c-bxn8s" Nov 03 01:28:59.373632 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:59.369338 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0c12073-7ff8-413c-9940-978f4b600778-workload-certs\") pod \"e2e-inf-b-5dab6a-openshift-default-6ff495656c-bxn8s\" (UID: \"e0c12073-7ff8-413c-9940-978f4b600778\") " pod="openshift-ingress/e2e-inf-b-5dab6a-openshift-default-6ff495656c-bxn8s" Nov 03 01:28:59.373632 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:59.369377 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0c12073-7ff8-413c-9940-978f4b600778-istio-podinfo\") pod \"e2e-inf-b-5dab6a-openshift-default-6ff495656c-bxn8s\" (UID: \"e0c12073-7ff8-413c-9940-978f4b600778\") " pod="openshift-ingress/e2e-inf-b-5dab6a-openshift-default-6ff495656c-bxn8s" Nov 03 01:28:59.373632 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:59.369504 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0c12073-7ff8-413c-9940-978f4b600778-workload-socket\") pod \"e2e-inf-b-5dab6a-openshift-default-6ff495656c-bxn8s\" (UID: \"e0c12073-7ff8-413c-9940-978f4b600778\") " pod="openshift-ingress/e2e-inf-b-5dab6a-openshift-default-6ff495656c-bxn8s" Nov 03 01:28:59.373632 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:59.369559 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0c12073-7ff8-413c-9940-978f4b600778-istio-envoy\") pod \"e2e-inf-b-5dab6a-openshift-default-6ff495656c-bxn8s\" (UID: \"e0c12073-7ff8-413c-9940-978f4b600778\") " pod="openshift-ingress/e2e-inf-b-5dab6a-openshift-default-6ff495656c-bxn8s" Nov 03 01:28:59.373632 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:59.369599 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0c12073-7ff8-413c-9940-978f4b600778-istio-data\") pod \"e2e-inf-b-5dab6a-openshift-default-6ff495656c-bxn8s\" (UID: \"e0c12073-7ff8-413c-9940-978f4b600778\") " pod="openshift-ingress/e2e-inf-b-5dab6a-openshift-default-6ff495656c-bxn8s" Nov 03 01:28:59.373632 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:59.369656 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0c12073-7ff8-413c-9940-978f4b600778-credential-socket\") pod \"e2e-inf-b-5dab6a-openshift-default-6ff495656c-bxn8s\" (UID: \"e0c12073-7ff8-413c-9940-978f4b600778\") " pod="openshift-ingress/e2e-inf-b-5dab6a-openshift-default-6ff495656c-bxn8s" Nov 03 01:28:59.373632 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:59.369820 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0c12073-7ff8-413c-9940-978f4b600778-istio-token\") pod \"e2e-inf-b-5dab6a-openshift-default-6ff495656c-bxn8s\" (UID: \"e0c12073-7ff8-413c-9940-978f4b600778\") " pod="openshift-ingress/e2e-inf-b-5dab6a-openshift-default-6ff495656c-bxn8s" Nov 03 01:28:59.373632 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:59.369877 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0c12073-7ff8-413c-9940-978f4b600778-istiod-ca-cert\") pod \"e2e-inf-b-5dab6a-openshift-default-6ff495656c-bxn8s\" (UID: \"e0c12073-7ff8-413c-9940-978f4b600778\") " pod="openshift-ingress/e2e-inf-b-5dab6a-openshift-default-6ff495656c-bxn8s" Nov 03 01:28:59.373632 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:59.370012 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6tbp\" (UniqueName: \"kubernetes.io/projected/dbd53b13-13c5-449a-b204-1fc94ac8afad-kube-api-access-m6tbp\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:28:59.373632 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:59.370028 4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbd53b13-13c5-449a-b204-1fc94ac8afad-plugins-config-volume\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:28:59.415089 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:59.394925 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee89f011-059f-493a-becf-4816c5550dbf-plugins-config-volume" pod "ee89f011-059f-493a-becf-4816c5550dbf" (UID: "ee89f011-059f-493a-becf-4816c5550dbf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:28:59.427326 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:59.425369 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee89f011-059f-493a-becf-4816c5550dbf-kube-api-access-c7f74" pod "ee89f011-059f-493a-becf-4816c5550dbf" (UID: "ee89f011-059f-493a-becf-4816c5550dbf"). InnerVolumeSpecName "kube-api-access-c7f74". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:28:59.476704 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:59.476659 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0c12073-7ff8-413c-9940-978f4b600778-istio-envoy\") pod \"e2e-inf-b-5dab6a-openshift-default-6ff495656c-bxn8s\" (UID: \"e0c12073-7ff8-413c-9940-978f4b600778\") " pod="openshift-ingress/e2e-inf-b-5dab6a-openshift-default-6ff495656c-bxn8s" Nov 03 01:28:59.476996 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:59.476975 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0c12073-7ff8-413c-9940-978f4b600778-istio-data\") pod \"e2e-inf-b-5dab6a-openshift-default-6ff495656c-bxn8s\" (UID: \"e0c12073-7ff8-413c-9940-978f4b600778\") " pod="openshift-ingress/e2e-inf-b-5dab6a-openshift-default-6ff495656c-bxn8s" Nov 03 01:28:59.477163 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:59.477144 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0c12073-7ff8-413c-9940-978f4b600778-credential-socket\") pod \"e2e-inf-b-5dab6a-openshift-default-6ff495656c-bxn8s\" (UID: \"e0c12073-7ff8-413c-9940-978f4b600778\") " pod="openshift-ingress/e2e-inf-b-5dab6a-openshift-default-6ff495656c-bxn8s" Nov 03 01:28:59.477456 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:59.477439 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0c12073-7ff8-413c-9940-978f4b600778-istio-token\") pod \"e2e-inf-b-5dab6a-openshift-default-6ff495656c-bxn8s\" (UID: \"e0c12073-7ff8-413c-9940-978f4b600778\") " pod="openshift-ingress/e2e-inf-b-5dab6a-openshift-default-6ff495656c-bxn8s" Nov 03 01:28:59.477626 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:59.477607 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0c12073-7ff8-413c-9940-978f4b600778-istiod-ca-cert\") pod \"e2e-inf-b-5dab6a-openshift-default-6ff495656c-bxn8s\" (UID: \"e0c12073-7ff8-413c-9940-978f4b600778\") " pod="openshift-ingress/e2e-inf-b-5dab6a-openshift-default-6ff495656c-bxn8s" Nov 03 01:28:59.477822 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:59.477808 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96mbm\" (UniqueName: \"kubernetes.io/projected/e0c12073-7ff8-413c-9940-978f4b600778-kube-api-access-96mbm\") pod \"e2e-inf-b-5dab6a-openshift-default-6ff495656c-bxn8s\" (UID: \"e0c12073-7ff8-413c-9940-978f4b600778\") " pod="openshift-ingress/e2e-inf-b-5dab6a-openshift-default-6ff495656c-bxn8s" Nov 03 01:28:59.478096 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:59.478080 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0c12073-7ff8-413c-9940-978f4b600778-workload-certs\") pod \"e2e-inf-b-5dab6a-openshift-default-6ff495656c-bxn8s\" (UID: \"e0c12073-7ff8-413c-9940-978f4b600778\") " pod="openshift-ingress/e2e-inf-b-5dab6a-openshift-default-6ff495656c-bxn8s" Nov 03 01:28:59.478219 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:59.478204 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0c12073-7ff8-413c-9940-978f4b600778-istio-podinfo\") pod \"e2e-inf-b-5dab6a-openshift-default-6ff495656c-bxn8s\" (UID: \"e0c12073-7ff8-413c-9940-978f4b600778\") " pod="openshift-ingress/e2e-inf-b-5dab6a-openshift-default-6ff495656c-bxn8s" Nov 03 01:28:59.478453 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:59.478437 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0c12073-7ff8-413c-9940-978f4b600778-workload-socket\") pod \"e2e-inf-b-5dab6a-openshift-default-6ff495656c-bxn8s\" (UID: \"e0c12073-7ff8-413c-9940-978f4b600778\") " pod="openshift-ingress/e2e-inf-b-5dab6a-openshift-default-6ff495656c-bxn8s" Nov 03 01:28:59.478698 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:59.478683 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c7f74\" (UniqueName: \"kubernetes.io/projected/ee89f011-059f-493a-becf-4816c5550dbf-kube-api-access-c7f74\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:28:59.478808 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:59.478795 4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee89f011-059f-493a-becf-4816c5550dbf-plugins-config-volume\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:28:59.479339 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:59.479316 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0c12073-7ff8-413c-9940-978f4b600778-workload-socket\") pod \"e2e-inf-b-5dab6a-openshift-default-6ff495656c-bxn8s\" (UID: \"e0c12073-7ff8-413c-9940-978f4b600778\") " pod="openshift-ingress/e2e-inf-b-5dab6a-openshift-default-6ff495656c-bxn8s" Nov 03 01:28:59.503298 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:59.480886 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0c12073-7ff8-413c-9940-978f4b600778-istio-data\") pod \"e2e-inf-b-5dab6a-openshift-default-6ff495656c-bxn8s\" (UID: \"e0c12073-7ff8-413c-9940-978f4b600778\") " pod="openshift-ingress/e2e-inf-b-5dab6a-openshift-default-6ff495656c-bxn8s" Nov 03 01:28:59.505703 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:59.504871 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0c12073-7ff8-413c-9940-978f4b600778-workload-certs\") pod \"e2e-inf-b-5dab6a-openshift-default-6ff495656c-bxn8s\" (UID: \"e0c12073-7ff8-413c-9940-978f4b600778\") " pod="openshift-ingress/e2e-inf-b-5dab6a-openshift-default-6ff495656c-bxn8s" Nov 03 01:28:59.506209 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:59.506182 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0c12073-7ff8-413c-9940-978f4b600778-credential-socket\") pod \"e2e-inf-b-5dab6a-openshift-default-6ff495656c-bxn8s\" (UID: \"e0c12073-7ff8-413c-9940-978f4b600778\") " pod="openshift-ingress/e2e-inf-b-5dab6a-openshift-default-6ff495656c-bxn8s" Nov 03 01:28:59.508899 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:59.508837 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0c12073-7ff8-413c-9940-978f4b600778-istiod-ca-cert\") pod \"e2e-inf-b-5dab6a-openshift-default-6ff495656c-bxn8s\" (UID: \"e0c12073-7ff8-413c-9940-978f4b600778\") " pod="openshift-ingress/e2e-inf-b-5dab6a-openshift-default-6ff495656c-bxn8s" Nov 03 01:28:59.745299 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:59.745183 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-96mbm\" (UniqueName: \"kubernetes.io/projected/e0c12073-7ff8-413c-9940-978f4b600778-kube-api-access-96mbm\") pod \"e2e-inf-b-5dab6a-openshift-default-6ff495656c-bxn8s\" (UID: \"e0c12073-7ff8-413c-9940-978f4b600778\") " pod="openshift-ingress/e2e-inf-b-5dab6a-openshift-default-6ff495656c-bxn8s" Nov 03 01:28:59.745895 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:59.745841 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0c12073-7ff8-413c-9940-978f4b600778-istio-podinfo\") pod \"e2e-inf-b-5dab6a-openshift-default-6ff495656c-bxn8s\" (UID: \"e0c12073-7ff8-413c-9940-978f4b600778\") " pod="openshift-ingress/e2e-inf-b-5dab6a-openshift-default-6ff495656c-bxn8s" Nov 03 01:28:59.820338 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:59.746389 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0c12073-7ff8-413c-9940-978f4b600778-istio-token\") pod \"e2e-inf-b-5dab6a-openshift-default-6ff495656c-bxn8s\" (UID: \"e0c12073-7ff8-413c-9940-978f4b600778\") " pod="openshift-ingress/e2e-inf-b-5dab6a-openshift-default-6ff495656c-bxn8s" Nov 03 01:28:59.820338 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:59.746671 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0c12073-7ff8-413c-9940-978f4b600778-istio-envoy\") pod \"e2e-inf-b-5dab6a-openshift-default-6ff495656c-bxn8s\" (UID: \"e0c12073-7ff8-413c-9940-978f4b600778\") " pod="openshift-ingress/e2e-inf-b-5dab6a-openshift-default-6ff495656c-bxn8s" Nov 03 01:28:59.888257 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:28:59.876590 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5dab6a-openshift-default-6ff495656c-bxn8s" Nov 03 01:29:00.084154 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:29:00.084119 4421 generic.go:356] "Generic (PLEG): container finished" podID="ee89f011-059f-493a-becf-4816c5550dbf" containerID="9013783a056208fe3d0b1d218151a61a9299c9219cb05a7de4b8a6f7d8662b42" exitCode=0 Nov 03 01:29:00.084724 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:29:00.084463 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ce04af-ffcb7d45f-zdv8k" Nov 03 01:29:00.091596 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:29:00.091560 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ce04af-ffcb7d45f-zdv8k" event={"ID":"ee89f011-059f-493a-becf-4816c5550dbf","Type":"ContainerDied","Data":"9013783a056208fe3d0b1d218151a61a9299c9219cb05a7de4b8a6f7d8662b42"} Nov 03 01:29:00.091748 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:29:00.091630 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ce04af-ffcb7d45f-zdv8k" event={"ID":"ee89f011-059f-493a-becf-4816c5550dbf","Type":"ContainerDied","Data":"b78fd32c77270378e02f6a0b19577313472363491f0ecba527e8f2e207336412"} Nov 03 01:29:00.091748 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:29:00.091663 4421 scope.go:122] "RemoveContainer" containerID="9013783a056208fe3d0b1d218151a61a9299c9219cb05a7de4b8a6f7d8662b42" Nov 03 01:29:00.091890 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:29:00.091865 4421 generic.go:356] "Generic (PLEG): container finished" podID="dbd53b13-13c5-449a-b204-1fc94ac8afad" containerID="b6e88418322adb40448b9fb23445d8522c2e6a7a794c295be254f5877f4e5eec" exitCode=0 Nov 03 01:29:00.091954 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:29:00.091942 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-ce04af-6b9b884869-vwgxv" event={"ID":"dbd53b13-13c5-449a-b204-1fc94ac8afad","Type":"ContainerDied","Data":"b6e88418322adb40448b9fb23445d8522c2e6a7a794c295be254f5877f4e5eec"} Nov 03 01:29:00.092018 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:29:00.091980 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-ce04af-6b9b884869-vwgxv" event={"ID":"dbd53b13-13c5-449a-b204-1fc94ac8afad","Type":"ContainerDied","Data":"916011a87c819ec239be470783aa70ed6620da9ca932023ad1598d66423bbdb8"} Nov 03 01:29:00.092123 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:29:00.092106 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-ce04af-6b9b884869-vwgxv" Nov 03 01:29:00.170020 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:29:00.169908 4421 scope.go:122] "RemoveContainer" containerID="9013783a056208fe3d0b1d218151a61a9299c9219cb05a7de4b8a6f7d8662b42" Nov 03 01:29:00.170676 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:29:00.170500 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9013783a056208fe3d0b1d218151a61a9299c9219cb05a7de4b8a6f7d8662b42\": container with ID starting with 9013783a056208fe3d0b1d218151a61a9299c9219cb05a7de4b8a6f7d8662b42 not found: ID does not exist" containerID="9013783a056208fe3d0b1d218151a61a9299c9219cb05a7de4b8a6f7d8662b42" Nov 03 01:29:00.170676 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:29:00.170552 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9013783a056208fe3d0b1d218151a61a9299c9219cb05a7de4b8a6f7d8662b42"} err="failed to get container status \"9013783a056208fe3d0b1d218151a61a9299c9219cb05a7de4b8a6f7d8662b42\": rpc error: code = NotFound desc = could not find container \"9013783a056208fe3d0b1d218151a61a9299c9219cb05a7de4b8a6f7d8662b42\": container with ID starting with 9013783a056208fe3d0b1d218151a61a9299c9219cb05a7de4b8a6f7d8662b42 not found: ID does not exist" Nov 03 01:29:00.170676 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:29:00.170579 4421 scope.go:122] "RemoveContainer" containerID="b6e88418322adb40448b9fb23445d8522c2e6a7a794c295be254f5877f4e5eec" Nov 03 01:29:00.192302 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:29:00.190572 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ce04af-6b9b884869-vwgxv"] Nov 03 01:29:00.197336 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:29:00.195103 4421 scope.go:122] "RemoveContainer" containerID="b6e88418322adb40448b9fb23445d8522c2e6a7a794c295be254f5877f4e5eec" Nov 03 01:29:00.197336 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:29:00.195564 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6e88418322adb40448b9fb23445d8522c2e6a7a794c295be254f5877f4e5eec\": container with ID starting with b6e88418322adb40448b9fb23445d8522c2e6a7a794c295be254f5877f4e5eec not found: ID does not exist" containerID="b6e88418322adb40448b9fb23445d8522c2e6a7a794c295be254f5877f4e5eec" Nov 03 01:29:00.197336 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:29:00.195615 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6e88418322adb40448b9fb23445d8522c2e6a7a794c295be254f5877f4e5eec"} err="failed to get container status \"b6e88418322adb40448b9fb23445d8522c2e6a7a794c295be254f5877f4e5eec\": rpc error: code = NotFound desc = could not find container \"b6e88418322adb40448b9fb23445d8522c2e6a7a794c295be254f5877f4e5eec\": container with ID starting with b6e88418322adb40448b9fb23445d8522c2e6a7a794c295be254f5877f4e5eec not found: ID does not exist" Nov 03 01:29:00.201623 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:29:00.201594 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ce04af-6b9b884869-vwgxv"] Nov 03 01:29:00.226133 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:29:00.206160 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ce04af-ffcb7d45f-zdv8k"] Nov 03 01:29:00.226133 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:29:00.216520 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ce04af-ffcb7d45f-zdv8k"] Nov 03 01:29:00.379160 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:29:00.379131 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0c12073_7ff8_413c_9940_978f4b600778.slice/crio-a10011ecb5c5c1f2e6aa6ec2891d49ecbbd492c9b499e0e072369f4d6990fc7b WatchSource:0}: Error finding container a10011ecb5c5c1f2e6aa6ec2891d49ecbbd492c9b499e0e072369f4d6990fc7b: Status 404 returned error can't find the container with id a10011ecb5c5c1f2e6aa6ec2891d49ecbbd492c9b499e0e072369f4d6990fc7b Nov 03 01:29:00.380441 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:29:00.380416 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-5dab6a-openshift-default-6ff495656c-bxn8s"] Nov 03 01:29:00.382757 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:29:00.382716 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:29:00.382899 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:29:00.382790 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:29:00.382899 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:29:00.382822 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:29:01.118325 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:29:01.116445 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5dab6a-openshift-default-6ff495656c-bxn8s" event={"ID":"e0c12073-7ff8-413c-9940-978f4b600778","Type":"ContainerStarted","Data":"4fd927cf38a5c09578c9fa47907e1dbebcfc4f5b21825dd9733c61fb91f2a3f9"} Nov 03 01:29:01.118325 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:29:01.116529 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5dab6a-openshift-default-6ff495656c-bxn8s" event={"ID":"e0c12073-7ff8-413c-9940-978f4b600778","Type":"ContainerStarted","Data":"a10011ecb5c5c1f2e6aa6ec2891d49ecbbd492c9b499e0e072369f4d6990fc7b"} Nov 03 01:29:01.147208 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:29:01.147154 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-5dab6a-openshift-default-6ff495656c-bxn8s" podStartSLOduration=2.147139432 podStartE2EDuration="2.147139432s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:28:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:29:01.141924768 +0000 UTC m=+2883.312927865" watchObservedRunningTime="2026-11-03 01:29:01.147139432 +0000 UTC m=+2883.318142528" Nov 03 01:29:01.883301 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:29:01.880487 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-5dab6a-openshift-default-6ff495656c-bxn8s" Nov 03 01:29:01.923703 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:29:01.922397 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-5dab6a-openshift-default-6ff495656c-bxn8s" Nov 03 01:29:01.989665 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:29:01.988757 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dbd53b13-13c5-449a-b204-1fc94ac8afad" path="/var/lib/kubelet/pods/dbd53b13-13c5-449a-b204-1fc94ac8afad/volumes" Nov 03 01:29:01.989665 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:29:01.989341 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee89f011-059f-493a-becf-4816c5550dbf" path="/var/lib/kubelet/pods/ee89f011-059f-493a-becf-4816c5550dbf/volumes" Nov 03 01:29:02.210351 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:29:02.202217 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-5dab6a-openshift-default-6ff495656c-bxn8s" Nov 03 01:29:02.227171 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:29:02.218812 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-5dab6a-openshift-default-6ff495656c-bxn8s" Nov 03 01:29:02.948300 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:29:02.944420 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-4b17ac-5867b8956f-tlz2b"] Nov 03 01:29:04.180800 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:29:04.180765 4421 scope.go:122] "RemoveContainer" containerID="d90180e72b36cf902d5a0e10aa43c45af2bda464c701b540431c67ad18d09cd0" Nov 03 01:29:07.949438 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:29:07.949363 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-4b17ac-5867b8956f-tlz2b" podUID="ddc7cbfa-9cfd-469e-a53e-ef8175080492" containerName="payload-processing" containerID="cri-o://e5bb3932102fd1b9db0ec800b138a9620307e3534b26325245bf37c48c5c6065" gracePeriod=25 Nov 03 01:29:08.752358 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:29:08.752331 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-4b17ac-5867b8956f-tlz2b" Nov 03 01:29:08.789117 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:29:08.789027 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-5dab6a-6c4b59c89-txkqj"] Nov 03 01:29:08.790383 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:29:08.790353 4421 cpu_manager.go:414] "containerMap: removing container" podUID="ddc7cbfa-9cfd-469e-a53e-ef8175080492" containerName="payload-processing" Nov 03 01:29:08.790383 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:29:08.790381 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ddc7cbfa-9cfd-469e-a53e-ef8175080492" containerName="payload-processing" Nov 03 01:29:08.790689 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:29:08.790672 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="ddc7cbfa-9cfd-469e-a53e-ef8175080492" containerName="payload-processing" Nov 03 01:29:08.798578 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:29:08.797292 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-5dab6a-6c4b59c89-txkqj" Nov 03 01:29:08.807241 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:29:08.807200 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-5dab6a-6c4b59c89-txkqj"] Nov 03 01:29:08.809738 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:29:08.809652 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-5dab6a-dockercfg-wzwxb\"" Nov 03 01:29:08.810110 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:29:08.810091 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-5dab6a\"" Nov 03 01:29:08.838885 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:29:08.838849 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ddc7cbfa-9cfd-469e-a53e-ef8175080492-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddc7cbfa-9cfd-469e-a53e-ef8175080492-plugins-config-volume\") pod \"ddc7cbfa-9cfd-469e-a53e-ef8175080492\" (UID: \"ddc7cbfa-9cfd-469e-a53e-ef8175080492\") " Nov 03 01:29:08.839125 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:29:08.839104 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ddc7cbfa-9cfd-469e-a53e-ef8175080492-kube-api-access-br84n\" (UniqueName: \"kubernetes.io/projected/ddc7cbfa-9cfd-469e-a53e-ef8175080492-kube-api-access-br84n\") pod \"ddc7cbfa-9cfd-469e-a53e-ef8175080492\" (UID: \"ddc7cbfa-9cfd-469e-a53e-ef8175080492\") " Nov 03 01:29:08.839342 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:29:08.839266 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddc7cbfa-9cfd-469e-a53e-ef8175080492-plugins-config-volume" pod "ddc7cbfa-9cfd-469e-a53e-ef8175080492" (UID: "ddc7cbfa-9cfd-469e-a53e-ef8175080492"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:29:08.839584 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:29:08.839558 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/896b4a5e-a44c-48a3-a5ee-aaef0ddad638-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-5dab6a-6c4b59c89-txkqj\" (UID: \"896b4a5e-a44c-48a3-a5ee-aaef0ddad638\") " pod="openshift-ingress/payload-processing-e2e-inf-b-5dab6a-6c4b59c89-txkqj" Nov 03 01:29:08.839964 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:29:08.839935 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrmsz\" (UniqueName: \"kubernetes.io/projected/896b4a5e-a44c-48a3-a5ee-aaef0ddad638-kube-api-access-nrmsz\") pod \"payload-processing-e2e-inf-b-5dab6a-6c4b59c89-txkqj\" (UID: \"896b4a5e-a44c-48a3-a5ee-aaef0ddad638\") " pod="openshift-ingress/payload-processing-e2e-inf-b-5dab6a-6c4b59c89-txkqj" Nov 03 01:29:08.840096 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:29:08.840077 4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddc7cbfa-9cfd-469e-a53e-ef8175080492-plugins-config-volume\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:29:08.844912 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:29:08.844878 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddc7cbfa-9cfd-469e-a53e-ef8175080492-kube-api-access-br84n" pod "ddc7cbfa-9cfd-469e-a53e-ef8175080492" (UID: "ddc7cbfa-9cfd-469e-a53e-ef8175080492"). InnerVolumeSpecName "kube-api-access-br84n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:29:08.941787 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:29:08.941592 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrmsz\" (UniqueName: \"kubernetes.io/projected/896b4a5e-a44c-48a3-a5ee-aaef0ddad638-kube-api-access-nrmsz\") pod \"payload-processing-e2e-inf-b-5dab6a-6c4b59c89-txkqj\" (UID: \"896b4a5e-a44c-48a3-a5ee-aaef0ddad638\") " pod="openshift-ingress/payload-processing-e2e-inf-b-5dab6a-6c4b59c89-txkqj" Nov 03 01:29:08.941981 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:29:08.941874 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/896b4a5e-a44c-48a3-a5ee-aaef0ddad638-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-5dab6a-6c4b59c89-txkqj\" (UID: \"896b4a5e-a44c-48a3-a5ee-aaef0ddad638\") " pod="openshift-ingress/payload-processing-e2e-inf-b-5dab6a-6c4b59c89-txkqj" Nov 03 01:29:08.942186 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:29:08.942168 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-br84n\" (UniqueName: \"kubernetes.io/projected/ddc7cbfa-9cfd-469e-a53e-ef8175080492-kube-api-access-br84n\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:29:08.942647 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:29:08.942625 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/896b4a5e-a44c-48a3-a5ee-aaef0ddad638-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-5dab6a-6c4b59c89-txkqj\" (UID: \"896b4a5e-a44c-48a3-a5ee-aaef0ddad638\") " pod="openshift-ingress/payload-processing-e2e-inf-b-5dab6a-6c4b59c89-txkqj" Nov 03 01:29:08.951167 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:29:08.951112 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrmsz\" (UniqueName: \"kubernetes.io/projected/896b4a5e-a44c-48a3-a5ee-aaef0ddad638-kube-api-access-nrmsz\") pod \"payload-processing-e2e-inf-b-5dab6a-6c4b59c89-txkqj\" (UID: \"896b4a5e-a44c-48a3-a5ee-aaef0ddad638\") " pod="openshift-ingress/payload-processing-e2e-inf-b-5dab6a-6c4b59c89-txkqj" Nov 03 01:29:09.098533 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:29:09.098434 4421 generic.go:356] "Generic (PLEG): container finished" podID="ddc7cbfa-9cfd-469e-a53e-ef8175080492" containerID="e5bb3932102fd1b9db0ec800b138a9620307e3534b26325245bf37c48c5c6065" exitCode=0 Nov 03 01:29:09.098533 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:29:09.098491 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-4b17ac-5867b8956f-tlz2b" Nov 03 01:29:09.098753 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:29:09.098532 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-4b17ac-5867b8956f-tlz2b" event={"ID":"ddc7cbfa-9cfd-469e-a53e-ef8175080492","Type":"ContainerDied","Data":"e5bb3932102fd1b9db0ec800b138a9620307e3534b26325245bf37c48c5c6065"} Nov 03 01:29:09.098753 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:29:09.098602 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-4b17ac-5867b8956f-tlz2b" event={"ID":"ddc7cbfa-9cfd-469e-a53e-ef8175080492","Type":"ContainerDied","Data":"471f2da2ab7f071a2b791aae58c3d70ec1d2962b501974d0b571a6117b766b5a"} Nov 03 01:29:09.098753 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:29:09.098632 4421 scope.go:122] "RemoveContainer" containerID="e5bb3932102fd1b9db0ec800b138a9620307e3534b26325245bf37c48c5c6065" Nov 03 01:29:09.113201 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:29:09.113172 4421 scope.go:122] "RemoveContainer" containerID="e5bb3932102fd1b9db0ec800b138a9620307e3534b26325245bf37c48c5c6065" Nov 03 01:29:09.113630 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:29:09.113602 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5bb3932102fd1b9db0ec800b138a9620307e3534b26325245bf37c48c5c6065\": container with ID starting with e5bb3932102fd1b9db0ec800b138a9620307e3534b26325245bf37c48c5c6065 not found: ID does not exist" containerID="e5bb3932102fd1b9db0ec800b138a9620307e3534b26325245bf37c48c5c6065" Nov 03 01:29:09.113709 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:29:09.113644 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5bb3932102fd1b9db0ec800b138a9620307e3534b26325245bf37c48c5c6065"} err="failed to get container status \"e5bb3932102fd1b9db0ec800b138a9620307e3534b26325245bf37c48c5c6065\": rpc error: code = NotFound desc = could not find container \"e5bb3932102fd1b9db0ec800b138a9620307e3534b26325245bf37c48c5c6065\": container with ID starting with e5bb3932102fd1b9db0ec800b138a9620307e3534b26325245bf37c48c5c6065 not found: ID does not exist" Nov 03 01:29:09.130564 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:29:09.130532 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-4b17ac-5867b8956f-tlz2b"] Nov 03 01:29:09.135048 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:29:09.135021 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-5dab6a-6c4b59c89-txkqj" Nov 03 01:29:09.139180 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:29:09.139154 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-4b17ac-5867b8956f-tlz2b"] Nov 03 01:29:09.724384 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:29:09.724230 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-5dab6a-6c4b59c89-txkqj"] Nov 03 01:29:09.983528 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:29:09.983431 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ddc7cbfa-9cfd-469e-a53e-ef8175080492" path="/var/lib/kubelet/pods/ddc7cbfa-9cfd-469e-a53e-ef8175080492/volumes" Nov 03 01:29:10.109942 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:29:10.109819 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-5dab6a-6c4b59c89-txkqj" event={"ID":"896b4a5e-a44c-48a3-a5ee-aaef0ddad638","Type":"ContainerStarted","Data":"c8bc28dc919ef1123a5e72cb7b7a8bbf2f798e6cdde506de1c25a1da48f6b937"} Nov 03 01:29:10.109942 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:29:10.109896 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-5dab6a-6c4b59c89-txkqj" event={"ID":"896b4a5e-a44c-48a3-a5ee-aaef0ddad638","Type":"ContainerStarted","Data":"5b61f3b19264594cb01990e007fe5e426059dc436f4f69f6dff5e4e4d195d5bf"} Nov 03 01:29:10.136025 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:29:10.128088 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-5dab6a-6c4b59c89-txkqj" podStartSLOduration=5.128073969 podStartE2EDuration="5.128073969s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:29:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:29:10.127682737 +0000 UTC m=+2892.298685834" watchObservedRunningTime="2026-11-03 01:29:10.128073969 +0000 UTC m=+2892.299077067" Nov 03 01:29:11.142351 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:29:11.142086 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-5dab6a-6c4b59c89-txkqj" Nov 03 01:29:11.261525 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:29:11.261466 4421 patch_prober.go:29] interesting pod/e2e-inf-a-b6afd6-openshift-default-78d46d7dcc-dg79z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 01:29:11.261751 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:29:11.261562 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-b6afd6-openshift-default-78d46d7dcc-dg79z" podUID="50b59bdf-0949-473f-8849-bb5648c6cba1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 01:29:14.673688 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:29:14.673644 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 01:29:15.306382 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:29:15.303781 4421 generic.go:356] "Generic (PLEG): container finished" podID="50b59bdf-0949-473f-8849-bb5648c6cba1" containerID="6a0daece2fecdfdfc172298e4260507ca77c25680ec24beb166c78a92f218ce7" exitCode=0 Nov 03 01:29:15.306382 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:29:15.303822 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b6afd6-openshift-default-78d46d7dcc-dg79z" event={"ID":"50b59bdf-0949-473f-8849-bb5648c6cba1","Type":"ContainerDied","Data":"6a0daece2fecdfdfc172298e4260507ca77c25680ec24beb166c78a92f218ce7"} Nov 03 01:29:15.306382 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:29:15.304357 4421 scope.go:122] "RemoveContainer" containerID="6a0daece2fecdfdfc172298e4260507ca77c25680ec24beb166c78a92f218ce7" Nov 03 01:29:15.310983 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:29:15.310935 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:29:15.311123 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:29:15.311045 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:29:15.311123 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:29:15.311097 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:29:16.422294 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:29:16.413209 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b6afd6-openshift-default-78d46d7dcc-dg79z" event={"ID":"50b59bdf-0949-473f-8849-bb5648c6cba1","Type":"ContainerStarted","Data":"157d6621f49d6e3d9575bc04f56fd034ae3f052d0e4e15f5e2ca92cb14e760dc"} Nov 03 01:29:16.649131 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:29:16.649098 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-b6afd6-openshift-default-78d46d7dcc-dg79z" Nov 03 01:29:16.649405 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:29:16.649212 4421 patch_prober.go:29] interesting pod/e2e-inf-a-b6afd6-openshift-default-78d46d7dcc-dg79z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.60:15021/healthz/ready\": dial tcp 10.134.1.60:15021: connect: connection refused" start-of-body= Nov 03 01:29:16.649405 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:29:16.649298 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b6afd6-openshift-default-78d46d7dcc-dg79z" podUID="50b59bdf-0949-473f-8849-bb5648c6cba1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.60:15021/healthz/ready\": dial tcp 10.134.1.60:15021: connect: connection refused" Nov 03 01:29:17.637309 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:29:17.635499 4421 patch_prober.go:29] interesting pod/e2e-inf-a-b6afd6-openshift-default-78d46d7dcc-dg79z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.60:15021/healthz/ready\": dial tcp 10.134.1.60:15021: connect: connection refused" start-of-body= Nov 03 01:29:17.637309 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:29:17.635573 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b6afd6-openshift-default-78d46d7dcc-dg79z" podUID="50b59bdf-0949-473f-8849-bb5648c6cba1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.60:15021/healthz/ready\": dial tcp 10.134.1.60:15021: connect: connection refused" Nov 03 01:29:18.645311 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:29:18.643786 4421 patch_prober.go:29] interesting pod/e2e-inf-a-b6afd6-openshift-default-78d46d7dcc-dg79z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.60:15021/healthz/ready\": dial tcp 10.134.1.60:15021: connect: connection refused" start-of-body= Nov 03 01:29:18.645311 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:29:18.643900 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b6afd6-openshift-default-78d46d7dcc-dg79z" podUID="50b59bdf-0949-473f-8849-bb5648c6cba1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.60:15021/healthz/ready\": dial tcp 10.134.1.60:15021: connect: connection refused" Nov 03 01:29:19.070370 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:29:19.070329 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc950da5-pttqt"] Nov 03 01:29:19.098554 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:29:19.098509 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc950da5-pttqt"] Nov 03 01:29:19.636295 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:29:19.634876 4421 patch_prober.go:29] interesting pod/e2e-inf-a-b6afd6-openshift-default-78d46d7dcc-dg79z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.60:15021/healthz/ready\": dial tcp 10.134.1.60:15021: connect: connection refused" start-of-body= Nov 03 01:29:19.636295 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:29:19.634942 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b6afd6-openshift-default-78d46d7dcc-dg79z" podUID="50b59bdf-0949-473f-8849-bb5648c6cba1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.60:15021/healthz/ready\": dial tcp 10.134.1.60:15021: connect: connection refused" Nov 03 01:29:19.982703 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:29:19.982623 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34c91155-3ee2-4729-bef4-fbff39a5a300" path="/var/lib/kubelet/pods/34c91155-3ee2-4729-bef4-fbff39a5a300/volumes" Nov 03 01:29:20.637262 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:29:20.637221 4421 patch_prober.go:29] interesting pod/e2e-inf-a-b6afd6-openshift-default-78d46d7dcc-dg79z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.60:15021/healthz/ready\": dial tcp 10.134.1.60:15021: connect: connection refused" start-of-body= Nov 03 01:29:20.637262 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:29:20.637298 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b6afd6-openshift-default-78d46d7dcc-dg79z" podUID="50b59bdf-0949-473f-8849-bb5648c6cba1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.60:15021/healthz/ready\": dial tcp 10.134.1.60:15021: connect: connection refused" Nov 03 01:29:21.634709 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:29:21.634669 4421 patch_prober.go:29] interesting pod/e2e-inf-a-b6afd6-openshift-default-78d46d7dcc-dg79z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.60:15021/healthz/ready\": dial tcp 10.134.1.60:15021: connect: connection refused" start-of-body= Nov 03 01:29:21.635146 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:29:21.634735 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b6afd6-openshift-default-78d46d7dcc-dg79z" podUID="50b59bdf-0949-473f-8849-bb5648c6cba1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.60:15021/healthz/ready\": dial tcp 10.134.1.60:15021: connect: connection refused" Nov 03 01:29:22.269374 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:29:22.269343 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-5dab6a-6c4b59c89-txkqj" Nov 03 01:29:22.873831 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:29:22.873797 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-b6afd6-openshift-default-78d46d7dcc-dg79z" Nov 03 01:29:22.898131 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:29:22.895161 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-b6afd6-openshift-default-78d46d7dcc-dg79z" Nov 03 01:29:22.925579 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:29:22.916660 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-b6afd6-openshift-default-78d46d7dcc-dg79z" Nov 03 01:29:34.220675 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:29:34.220638 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2767ec4a-rg2kd"] Nov 03 01:29:34.226404 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:29:34.226372 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2767ec4a-rg2kd" Nov 03 01:29:34.231060 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:29:34.230996 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:29:34.231060 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:29:34.231014 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6q4xp\"" Nov 03 01:29:34.232011 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:29:34.231932 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2767ec4a-rg2kd"] Nov 03 01:29:34.317752 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:29:34.317710 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de29d9b6-da5a-4b65-99b9-996326dac022-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2767ec4a-rg2kd\" (UID: \"de29d9b6-da5a-4b65-99b9-996326dac022\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2767ec4a-rg2kd" Nov 03 01:29:34.317969 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:29:34.317765 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czd5d\" (UniqueName: \"kubernetes.io/projected/de29d9b6-da5a-4b65-99b9-996326dac022-kube-api-access-czd5d\") pod \"maas-api-revoke-sub-2767ec4a-rg2kd\" (UID: \"de29d9b6-da5a-4b65-99b9-996326dac022\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2767ec4a-rg2kd" Nov 03 01:29:34.318116 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:29:34.318093 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de29d9b6-da5a-4b65-99b9-996326dac022-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2767ec4a-rg2kd\" (UID: \"de29d9b6-da5a-4b65-99b9-996326dac022\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2767ec4a-rg2kd" Nov 03 01:29:34.419688 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:29:34.419645 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de29d9b6-da5a-4b65-99b9-996326dac022-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2767ec4a-rg2kd\" (UID: \"de29d9b6-da5a-4b65-99b9-996326dac022\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2767ec4a-rg2kd" Nov 03 01:29:34.419892 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:29:34.419704 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czd5d\" (UniqueName: \"kubernetes.io/projected/de29d9b6-da5a-4b65-99b9-996326dac022-kube-api-access-czd5d\") pod \"maas-api-revoke-sub-2767ec4a-rg2kd\" (UID: \"de29d9b6-da5a-4b65-99b9-996326dac022\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2767ec4a-rg2kd" Nov 03 01:29:34.419980 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:29:34.419963 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de29d9b6-da5a-4b65-99b9-996326dac022-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2767ec4a-rg2kd\" (UID: \"de29d9b6-da5a-4b65-99b9-996326dac022\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2767ec4a-rg2kd" Nov 03 01:29:34.420498 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:29:34.420462 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de29d9b6-da5a-4b65-99b9-996326dac022-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2767ec4a-rg2kd\" (UID: \"de29d9b6-da5a-4b65-99b9-996326dac022\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2767ec4a-rg2kd" Nov 03 01:29:34.422523 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:29:34.422492 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de29d9b6-da5a-4b65-99b9-996326dac022-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2767ec4a-rg2kd\" (UID: \"de29d9b6-da5a-4b65-99b9-996326dac022\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2767ec4a-rg2kd" Nov 03 01:29:34.429088 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:29:34.429062 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-czd5d\" (UniqueName: \"kubernetes.io/projected/de29d9b6-da5a-4b65-99b9-996326dac022-kube-api-access-czd5d\") pod \"maas-api-revoke-sub-2767ec4a-rg2kd\" (UID: \"de29d9b6-da5a-4b65-99b9-996326dac022\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2767ec4a-rg2kd" Nov 03 01:29:34.545678 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:29:34.545640 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2767ec4a-rg2kd" Nov 03 01:29:34.931035 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:29:34.931004 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde29d9b6_da5a_4b65_99b9_996326dac022.slice/crio-06c6f62fc4bd37d77819b677d651ef507c57d8c2f2a732d80361fd0d9747d1e4 WatchSource:0}: Error finding container 06c6f62fc4bd37d77819b677d651ef507c57d8c2f2a732d80361fd0d9747d1e4: Status 404 returned error can't find the container with id 06c6f62fc4bd37d77819b677d651ef507c57d8c2f2a732d80361fd0d9747d1e4 Nov 03 01:29:34.931283 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:29:34.931237 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2767ec4a-rg2kd"] Nov 03 01:29:35.828890 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:29:35.828854 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2767ec4a-rg2kd" event={"ID":"de29d9b6-da5a-4b65-99b9-996326dac022","Type":"ContainerStarted","Data":"876417d98447d80450b3399706caca23ada482a1d659849487ad6e6ec51d88a0"} Nov 03 01:29:35.829432 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:29:35.828911 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2767ec4a-rg2kd" event={"ID":"de29d9b6-da5a-4b65-99b9-996326dac022","Type":"ContainerStarted","Data":"06c6f62fc4bd37d77819b677d651ef507c57d8c2f2a732d80361fd0d9747d1e4"} Nov 03 01:29:35.850766 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:29:35.850706 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2767ec4a-rg2kd" podStartSLOduration=1.850689182 podStartE2EDuration="1.850689182s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:29:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:29:35.845806939 +0000 UTC m=+2918.016810036" watchObservedRunningTime="2026-11-03 01:29:35.850689182 +0000 UTC m=+2918.021692256" Nov 03 01:29:36.839690 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:29:36.839651 4421 generic.go:356] "Generic (PLEG): container finished" podID="de29d9b6-da5a-4b65-99b9-996326dac022" containerID="876417d98447d80450b3399706caca23ada482a1d659849487ad6e6ec51d88a0" exitCode=0 Nov 03 01:29:36.840197 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:29:36.839697 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2767ec4a-rg2kd" event={"ID":"de29d9b6-da5a-4b65-99b9-996326dac022","Type":"ContainerDied","Data":"876417d98447d80450b3399706caca23ada482a1d659849487ad6e6ec51d88a0"} Nov 03 01:29:38.125873 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:29:38.125842 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2767ec4a-rg2kd" Nov 03 01:29:38.236186 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:29:38.236094 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de29d9b6-da5a-4b65-99b9-996326dac022-kube-api-access-czd5d\" (UniqueName: \"kubernetes.io/projected/de29d9b6-da5a-4b65-99b9-996326dac022-kube-api-access-czd5d\") pod \"de29d9b6-da5a-4b65-99b9-996326dac022\" (UID: \"de29d9b6-da5a-4b65-99b9-996326dac022\") " Nov 03 01:29:38.236186 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:29:38.236133 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de29d9b6-da5a-4b65-99b9-996326dac022-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de29d9b6-da5a-4b65-99b9-996326dac022-maas-api-openshift-ca\") pod \"de29d9b6-da5a-4b65-99b9-996326dac022\" (UID: \"de29d9b6-da5a-4b65-99b9-996326dac022\") " Nov 03 01:29:38.236489 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:29:38.236232 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de29d9b6-da5a-4b65-99b9-996326dac022-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de29d9b6-da5a-4b65-99b9-996326dac022-maas-api-xks-ca\") pod \"de29d9b6-da5a-4b65-99b9-996326dac022\" (UID: \"de29d9b6-da5a-4b65-99b9-996326dac022\") " Nov 03 01:29:38.236657 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:29:38.236631 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de29d9b6-da5a-4b65-99b9-996326dac022-maas-api-openshift-ca" pod "de29d9b6-da5a-4b65-99b9-996326dac022" (UID: "de29d9b6-da5a-4b65-99b9-996326dac022"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:29:38.237175 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:29:38.237150 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de29d9b6-da5a-4b65-99b9-996326dac022-maas-api-openshift-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:29:38.239099 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:29:38.239072 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de29d9b6-da5a-4b65-99b9-996326dac022-maas-api-xks-ca" pod "de29d9b6-da5a-4b65-99b9-996326dac022" (UID: "de29d9b6-da5a-4b65-99b9-996326dac022"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:29:38.239330 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:29:38.239264 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de29d9b6-da5a-4b65-99b9-996326dac022-kube-api-access-czd5d" pod "de29d9b6-da5a-4b65-99b9-996326dac022" (UID: "de29d9b6-da5a-4b65-99b9-996326dac022"). InnerVolumeSpecName "kube-api-access-czd5d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:29:38.338962 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:29:38.338910 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-czd5d\" (UniqueName: \"kubernetes.io/projected/de29d9b6-da5a-4b65-99b9-996326dac022-kube-api-access-czd5d\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:29:38.338962 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:29:38.338958 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de29d9b6-da5a-4b65-99b9-996326dac022-maas-api-xks-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:29:38.933311 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:29:38.933268 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2767ec4a-rg2kd" Nov 03 01:29:38.933311 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:29:38.933297 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2767ec4a-rg2kd" event={"ID":"de29d9b6-da5a-4b65-99b9-996326dac022","Type":"ContainerDied","Data":"06c6f62fc4bd37d77819b677d651ef507c57d8c2f2a732d80361fd0d9747d1e4"} Nov 03 01:29:38.933535 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:29:38.933351 4421 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="06c6f62fc4bd37d77819b677d651ef507c57d8c2f2a732d80361fd0d9747d1e4" Nov 03 01:29:40.029993 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:29:40.029959 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc9bc07e-rgvk5"] Nov 03 01:29:40.035236 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:29:40.035202 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc9bc07e-rgvk5"] Nov 03 01:29:41.981588 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:29:41.981554 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f866c13c-8629-4da7-b7af-6af1e9932157" path="/var/lib/kubelet/pods/f866c13c-8629-4da7-b7af-6af1e9932157/volumes" Nov 03 01:29:44.672500 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:29:44.672465 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 01:29:58.133701 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:29:58.133663 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-7nssr"] Nov 03 01:29:58.134688 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:29:58.134644 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-7nssr" podUID="4a72680d-8a9d-4aed-89c9-50206468ea74" containerName="istio-proxy" containerID="cri-o://313995a8e8d94d5c2a3e4375572a13e07e40738c5db13afca7b4c3b3dca9312f" gracePeriod=30 Nov 03 01:30:00.138074 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:00.138039 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-be59df-29894490-4l7wj"] Nov 03 01:30:00.139216 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:00.139014 4421 cpu_manager.go:414] "containerMap: removing container" podUID="de29d9b6-da5a-4b65-99b9-996326dac022" containerName="revoke-keys" Nov 03 01:30:00.139216 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:00.139032 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de29d9b6-da5a-4b65-99b9-996326dac022" containerName="revoke-keys" Nov 03 01:30:00.139216 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:00.139154 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="de29d9b6-da5a-4b65-99b9-996326dac022" containerName="revoke-keys" Nov 03 01:30:00.143103 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:00.143083 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-be59df-29894490-4l7wj" Nov 03 01:30:00.162828 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:00.156443 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6q4xp\"" Nov 03 01:30:00.162828 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:00.161698 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cp675\" (UniqueName: \"kubernetes.io/projected/8b32bc18-0be3-4f64-811c-ce4b662df3e6-kube-api-access-cp675\") pod \"maas-api-key-cleanup-e2e-extmod-b-be59df-29894490-4l7wj\" (UID: \"8b32bc18-0be3-4f64-811c-ce4b662df3e6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-be59df-29894490-4l7wj" Nov 03 01:30:00.162828 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:00.161891 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b32bc18-0be3-4f64-811c-ce4b662df3e6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-be59df-29894490-4l7wj\" (UID: \"8b32bc18-0be3-4f64-811c-ce4b662df3e6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-be59df-29894490-4l7wj" Nov 03 01:30:00.166051 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:00.166016 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-be59df-29894490-4l7wj"] Nov 03 01:30:00.264494 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:00.264451 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cp675\" (UniqueName: \"kubernetes.io/projected/8b32bc18-0be3-4f64-811c-ce4b662df3e6-kube-api-access-cp675\") pod \"maas-api-key-cleanup-e2e-extmod-b-be59df-29894490-4l7wj\" (UID: \"8b32bc18-0be3-4f64-811c-ce4b662df3e6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-be59df-29894490-4l7wj" Nov 03 01:30:00.264701 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:00.264524 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b32bc18-0be3-4f64-811c-ce4b662df3e6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-be59df-29894490-4l7wj\" (UID: \"8b32bc18-0be3-4f64-811c-ce4b662df3e6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-be59df-29894490-4l7wj" Nov 03 01:30:00.265317 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:00.265262 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b32bc18-0be3-4f64-811c-ce4b662df3e6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-be59df-29894490-4l7wj\" (UID: \"8b32bc18-0be3-4f64-811c-ce4b662df3e6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-be59df-29894490-4l7wj" Nov 03 01:30:00.279997 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:00.279966 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cp675\" (UniqueName: \"kubernetes.io/projected/8b32bc18-0be3-4f64-811c-ce4b662df3e6-kube-api-access-cp675\") pod \"maas-api-key-cleanup-e2e-extmod-b-be59df-29894490-4l7wj\" (UID: \"8b32bc18-0be3-4f64-811c-ce4b662df3e6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-be59df-29894490-4l7wj" Nov 03 01:30:00.463544 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:00.463465 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-be59df-29894490-4l7wj" Nov 03 01:30:00.829245 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:00.829211 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-be59df-29894490-4l7wj"] Nov 03 01:30:01.406561 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:01.406444 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-be59df-29894490-4l7wj" event={"ID":"8b32bc18-0be3-4f64-811c-ce4b662df3e6","Type":"ContainerStarted","Data":"87cd1aa4e93e5b50dafd631b6eeea9424f5fe3415579819d8e95384df84be53c"} Nov 03 01:30:01.406561 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:01.406522 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-be59df-29894490-4l7wj" event={"ID":"8b32bc18-0be3-4f64-811c-ce4b662df3e6","Type":"ContainerStarted","Data":"b2f507300533a71cf9846753a6e686a241bc1053127b9817221452cb236410ad"} Nov 03 01:30:01.437989 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:01.437926 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-be59df-29894490-4l7wj" podStartSLOduration=1.437909244 podStartE2EDuration="1.437909244s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:30:01.433666826 +0000 UTC m=+2943.604669929" watchObservedRunningTime="2026-11-03 01:30:01.437909244 +0000 UTC m=+2943.608912340" Nov 03 01:30:01.615688 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:01.615635 4421 patch_prober.go:29] interesting pod/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-7nssr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 01:30:01.615884 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:01.615711 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-7nssr" podUID="4a72680d-8a9d-4aed-89c9-50206468ea74" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 01:30:02.417538 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:02.417504 4421 generic.go:356] "Generic (PLEG): container finished" podID="8b32bc18-0be3-4f64-811c-ce4b662df3e6" containerID="87cd1aa4e93e5b50dafd631b6eeea9424f5fe3415579819d8e95384df84be53c" exitCode=60 Nov 03 01:30:02.417938 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:02.417585 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-be59df-29894490-4l7wj" event={"ID":"8b32bc18-0be3-4f64-811c-ce4b662df3e6","Type":"ContainerDied","Data":"87cd1aa4e93e5b50dafd631b6eeea9424f5fe3415579819d8e95384df84be53c"} Nov 03 01:30:02.417938 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:02.417912 4421 scope.go:122] "RemoveContainer" containerID="87cd1aa4e93e5b50dafd631b6eeea9424f5fe3415579819d8e95384df84be53c" Nov 03 01:30:03.440906 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:03.440869 4421 generic.go:356] "Generic (PLEG): container finished" podID="8b32bc18-0be3-4f64-811c-ce4b662df3e6" containerID="0b9e95a6280b30c3029e49f427260e6a25e44c2e9dec42514fdb23f4d8701792" exitCode=60 Nov 03 01:30:03.442306 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:03.441049 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-be59df-29894490-4l7wj" event={"ID":"8b32bc18-0be3-4f64-811c-ce4b662df3e6","Type":"ContainerDied","Data":"0b9e95a6280b30c3029e49f427260e6a25e44c2e9dec42514fdb23f4d8701792"} Nov 03 01:30:03.442306 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:03.441109 4421 scope.go:122] "RemoveContainer" containerID="87cd1aa4e93e5b50dafd631b6eeea9424f5fe3415579819d8e95384df84be53c" Nov 03 01:30:03.442306 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:03.441703 4421 scope.go:122] "RemoveContainer" containerID="0b9e95a6280b30c3029e49f427260e6a25e44c2e9dec42514fdb23f4d8701792" Nov 03 01:30:03.442306 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:30:03.442192 4421 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-be59df-29894490-4l7wj_odh-ai-gateway-infra(8b32bc18-0be3-4f64-811c-ce4b662df3e6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-be59df-29894490-4l7wj" podUID="8b32bc18-0be3-4f64-811c-ce4b662df3e6" Nov 03 01:30:03.447344 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:03.447304 4421 generic.go:356] "Generic (PLEG): container finished" podID="4a72680d-8a9d-4aed-89c9-50206468ea74" containerID="313995a8e8d94d5c2a3e4375572a13e07e40738c5db13afca7b4c3b3dca9312f" exitCode=0 Nov 03 01:30:03.447772 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:03.447602 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-7nssr" event={"ID":"4a72680d-8a9d-4aed-89c9-50206468ea74","Type":"ContainerDied","Data":"313995a8e8d94d5c2a3e4375572a13e07e40738c5db13afca7b4c3b3dca9312f"} Nov 03 01:30:03.853631 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:03.853597 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-7nssr" Nov 03 01:30:03.928314 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:03.927591 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zgdl8"] Nov 03 01:30:03.930005 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:03.929047 4421 cpu_manager.go:414] "containerMap: removing container" podUID="4a72680d-8a9d-4aed-89c9-50206468ea74" containerName="istio-proxy" Nov 03 01:30:03.930005 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:03.929073 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a72680d-8a9d-4aed-89c9-50206468ea74" containerName="istio-proxy" Nov 03 01:30:03.930005 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:03.929332 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a72680d-8a9d-4aed-89c9-50206468ea74" containerName="istio-proxy" Nov 03 01:30:03.954301 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:03.934858 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zgdl8" Nov 03 01:30:03.978525 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:03.970534 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a72680d-8a9d-4aed-89c9-50206468ea74-istio-token\" (UniqueName: \"kubernetes.io/projected/4a72680d-8a9d-4aed-89c9-50206468ea74-istio-token\") pod \"4a72680d-8a9d-4aed-89c9-50206468ea74\" (UID: \"4a72680d-8a9d-4aed-89c9-50206468ea74\") " Nov 03 01:30:03.978525 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:03.970594 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a72680d-8a9d-4aed-89c9-50206468ea74-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a72680d-8a9d-4aed-89c9-50206468ea74-workload-socket\") pod \"4a72680d-8a9d-4aed-89c9-50206468ea74\" (UID: \"4a72680d-8a9d-4aed-89c9-50206468ea74\") " Nov 03 01:30:03.978525 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:03.972136 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a72680d-8a9d-4aed-89c9-50206468ea74-workload-socket" pod "4a72680d-8a9d-4aed-89c9-50206468ea74" (UID: "4a72680d-8a9d-4aed-89c9-50206468ea74"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:30:03.978525 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:03.977821 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a72680d-8a9d-4aed-89c9-50206468ea74-istio-token" pod "4a72680d-8a9d-4aed-89c9-50206468ea74" (UID: "4a72680d-8a9d-4aed-89c9-50206468ea74"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:30:04.012427 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:03.979477 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4a72680d-8a9d-4aed-89c9-50206468ea74-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a72680d-8a9d-4aed-89c9-50206468ea74-istio-podinfo\") pod \"4a72680d-8a9d-4aed-89c9-50206468ea74\" (UID: \"4a72680d-8a9d-4aed-89c9-50206468ea74\") " Nov 03 01:30:04.012427 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:03.979969 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a72680d-8a9d-4aed-89c9-50206468ea74-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a72680d-8a9d-4aed-89c9-50206468ea74-istiod-ca-cert\") pod \"4a72680d-8a9d-4aed-89c9-50206468ea74\" (UID: \"4a72680d-8a9d-4aed-89c9-50206468ea74\") " Nov 03 01:30:04.012427 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:03.980331 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a72680d-8a9d-4aed-89c9-50206468ea74-istiod-ca-cert" pod "4a72680d-8a9d-4aed-89c9-50206468ea74" (UID: "4a72680d-8a9d-4aed-89c9-50206468ea74"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:30:04.012427 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:03.980375 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a72680d-8a9d-4aed-89c9-50206468ea74-kube-api-access-lnpt2\" (UniqueName: \"kubernetes.io/projected/4a72680d-8a9d-4aed-89c9-50206468ea74-kube-api-access-lnpt2\") pod \"4a72680d-8a9d-4aed-89c9-50206468ea74\" (UID: \"4a72680d-8a9d-4aed-89c9-50206468ea74\") " Nov 03 01:30:04.012427 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:03.980725 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a72680d-8a9d-4aed-89c9-50206468ea74-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a72680d-8a9d-4aed-89c9-50206468ea74-istio-data\") pod \"4a72680d-8a9d-4aed-89c9-50206468ea74\" (UID: \"4a72680d-8a9d-4aed-89c9-50206468ea74\") " Nov 03 01:30:04.012427 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:03.980809 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a72680d-8a9d-4aed-89c9-50206468ea74-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a72680d-8a9d-4aed-89c9-50206468ea74-workload-certs\") pod \"4a72680d-8a9d-4aed-89c9-50206468ea74\" (UID: \"4a72680d-8a9d-4aed-89c9-50206468ea74\") " Nov 03 01:30:04.012427 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:03.980881 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a72680d-8a9d-4aed-89c9-50206468ea74-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a72680d-8a9d-4aed-89c9-50206468ea74-istio-envoy\") pod \"4a72680d-8a9d-4aed-89c9-50206468ea74\" (UID: \"4a72680d-8a9d-4aed-89c9-50206468ea74\") " Nov 03 01:30:04.012427 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:03.980959 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a72680d-8a9d-4aed-89c9-50206468ea74-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a72680d-8a9d-4aed-89c9-50206468ea74-credential-socket\") pod \"4a72680d-8a9d-4aed-89c9-50206468ea74\" (UID: \"4a72680d-8a9d-4aed-89c9-50206468ea74\") " Nov 03 01:30:04.012427 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:03.981323 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/920eea17-797a-4c19-9f09-36f121782d2e-istiod-ca-cert\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zgdl8\" (UID: \"920eea17-797a-4c19-9f09-36f121782d2e\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zgdl8" Nov 03 01:30:04.012427 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:03.981871 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/920eea17-797a-4c19-9f09-36f121782d2e-istio-token\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zgdl8\" (UID: \"920eea17-797a-4c19-9f09-36f121782d2e\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zgdl8" Nov 03 01:30:04.012427 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:03.981921 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/920eea17-797a-4c19-9f09-36f121782d2e-istio-podinfo\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zgdl8\" (UID: \"920eea17-797a-4c19-9f09-36f121782d2e\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zgdl8" Nov 03 01:30:04.012427 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:03.982049 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/920eea17-797a-4c19-9f09-36f121782d2e-workload-socket\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zgdl8\" (UID: \"920eea17-797a-4c19-9f09-36f121782d2e\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zgdl8" Nov 03 01:30:04.012427 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:03.982122 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qg594\" (UniqueName: \"kubernetes.io/projected/920eea17-797a-4c19-9f09-36f121782d2e-kube-api-access-qg594\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zgdl8\" (UID: \"920eea17-797a-4c19-9f09-36f121782d2e\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zgdl8" Nov 03 01:30:04.012427 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:03.982506 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/920eea17-797a-4c19-9f09-36f121782d2e-credential-socket\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zgdl8\" (UID: \"920eea17-797a-4c19-9f09-36f121782d2e\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zgdl8" Nov 03 01:30:04.012427 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:03.982552 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/920eea17-797a-4c19-9f09-36f121782d2e-workload-certs\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zgdl8\" (UID: \"920eea17-797a-4c19-9f09-36f121782d2e\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zgdl8" Nov 03 01:30:04.012427 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:03.982598 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/920eea17-797a-4c19-9f09-36f121782d2e-istio-data\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zgdl8\" (UID: \"920eea17-797a-4c19-9f09-36f121782d2e\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zgdl8" Nov 03 01:30:04.012427 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:03.982821 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/920eea17-797a-4c19-9f09-36f121782d2e-istio-envoy\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zgdl8\" (UID: \"920eea17-797a-4c19-9f09-36f121782d2e\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zgdl8" Nov 03 01:30:04.012427 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:03.982966 4421 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a72680d-8a9d-4aed-89c9-50206468ea74-istio-token\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:30:04.012427 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:03.982984 4421 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a72680d-8a9d-4aed-89c9-50206468ea74-workload-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:30:04.012427 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:03.983001 4421 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a72680d-8a9d-4aed-89c9-50206468ea74-istiod-ca-cert\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:30:04.012427 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:03.984743 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a72680d-8a9d-4aed-89c9-50206468ea74-istio-data" pod "4a72680d-8a9d-4aed-89c9-50206468ea74" (UID: "4a72680d-8a9d-4aed-89c9-50206468ea74"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:30:04.012427 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:03.985535 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a72680d-8a9d-4aed-89c9-50206468ea74-credential-socket" pod "4a72680d-8a9d-4aed-89c9-50206468ea74" (UID: "4a72680d-8a9d-4aed-89c9-50206468ea74"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:30:04.012427 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:04.010601 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a72680d-8a9d-4aed-89c9-50206468ea74-workload-certs" pod "4a72680d-8a9d-4aed-89c9-50206468ea74" (UID: "4a72680d-8a9d-4aed-89c9-50206468ea74"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:30:04.016826 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:04.016781 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a72680d-8a9d-4aed-89c9-50206468ea74-istio-envoy" pod "4a72680d-8a9d-4aed-89c9-50206468ea74" (UID: "4a72680d-8a9d-4aed-89c9-50206468ea74"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:30:04.017696 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:04.017656 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4a72680d-8a9d-4aed-89c9-50206468ea74-istio-podinfo" pod "4a72680d-8a9d-4aed-89c9-50206468ea74" (UID: "4a72680d-8a9d-4aed-89c9-50206468ea74"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:30:04.020372 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:04.020286 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a72680d-8a9d-4aed-89c9-50206468ea74-kube-api-access-lnpt2" pod "4a72680d-8a9d-4aed-89c9-50206468ea74" (UID: "4a72680d-8a9d-4aed-89c9-50206468ea74"). InnerVolumeSpecName "kube-api-access-lnpt2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:30:04.022913 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:04.022892 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zgdl8"] Nov 03 01:30:04.084259 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:04.084220 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/920eea17-797a-4c19-9f09-36f121782d2e-istio-envoy\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zgdl8\" (UID: \"920eea17-797a-4c19-9f09-36f121782d2e\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zgdl8" Nov 03 01:30:04.084468 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:04.084295 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/920eea17-797a-4c19-9f09-36f121782d2e-istiod-ca-cert\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zgdl8\" (UID: \"920eea17-797a-4c19-9f09-36f121782d2e\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zgdl8" Nov 03 01:30:04.084468 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:04.084435 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/920eea17-797a-4c19-9f09-36f121782d2e-istio-token\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zgdl8\" (UID: \"920eea17-797a-4c19-9f09-36f121782d2e\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zgdl8" Nov 03 01:30:04.084608 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:04.084485 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/920eea17-797a-4c19-9f09-36f121782d2e-istio-podinfo\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zgdl8\" (UID: \"920eea17-797a-4c19-9f09-36f121782d2e\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zgdl8" Nov 03 01:30:04.084608 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:04.084596 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/920eea17-797a-4c19-9f09-36f121782d2e-workload-socket\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zgdl8\" (UID: \"920eea17-797a-4c19-9f09-36f121782d2e\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zgdl8" Nov 03 01:30:04.084723 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:04.084664 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qg594\" (UniqueName: \"kubernetes.io/projected/920eea17-797a-4c19-9f09-36f121782d2e-kube-api-access-qg594\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zgdl8\" (UID: \"920eea17-797a-4c19-9f09-36f121782d2e\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zgdl8" Nov 03 01:30:04.084910 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:04.084845 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/920eea17-797a-4c19-9f09-36f121782d2e-credential-socket\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zgdl8\" (UID: \"920eea17-797a-4c19-9f09-36f121782d2e\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zgdl8" Nov 03 01:30:04.084910 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:04.084884 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/920eea17-797a-4c19-9f09-36f121782d2e-workload-certs\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zgdl8\" (UID: \"920eea17-797a-4c19-9f09-36f121782d2e\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zgdl8" Nov 03 01:30:04.085072 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:04.084929 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/920eea17-797a-4c19-9f09-36f121782d2e-istio-data\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zgdl8\" (UID: \"920eea17-797a-4c19-9f09-36f121782d2e\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zgdl8" Nov 03 01:30:04.085072 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:04.085045 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lnpt2\" (UniqueName: \"kubernetes.io/projected/4a72680d-8a9d-4aed-89c9-50206468ea74-kube-api-access-lnpt2\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:30:04.085386 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:04.085066 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/920eea17-797a-4c19-9f09-36f121782d2e-istiod-ca-cert\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zgdl8\" (UID: \"920eea17-797a-4c19-9f09-36f121782d2e\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zgdl8" Nov 03 01:30:04.085386 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:04.085076 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/920eea17-797a-4c19-9f09-36f121782d2e-workload-socket\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zgdl8\" (UID: \"920eea17-797a-4c19-9f09-36f121782d2e\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zgdl8" Nov 03 01:30:04.085386 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:04.085067 4421 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a72680d-8a9d-4aed-89c9-50206468ea74-istio-data\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:30:04.085386 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:04.085161 4421 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a72680d-8a9d-4aed-89c9-50206468ea74-workload-certs\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:30:04.085386 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:04.085180 4421 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a72680d-8a9d-4aed-89c9-50206468ea74-istio-envoy\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:30:04.085386 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:04.085197 4421 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a72680d-8a9d-4aed-89c9-50206468ea74-credential-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:30:04.085386 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:04.085203 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/920eea17-797a-4c19-9f09-36f121782d2e-credential-socket\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zgdl8\" (UID: \"920eea17-797a-4c19-9f09-36f121782d2e\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zgdl8" Nov 03 01:30:04.085386 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:04.085216 4421 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a72680d-8a9d-4aed-89c9-50206468ea74-istio-podinfo\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:30:04.085386 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:04.085366 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/920eea17-797a-4c19-9f09-36f121782d2e-workload-certs\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zgdl8\" (UID: \"920eea17-797a-4c19-9f09-36f121782d2e\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zgdl8" Nov 03 01:30:04.086118 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:04.086094 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/920eea17-797a-4c19-9f09-36f121782d2e-istio-data\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zgdl8\" (UID: \"920eea17-797a-4c19-9f09-36f121782d2e\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zgdl8" Nov 03 01:30:04.088390 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:04.088360 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/920eea17-797a-4c19-9f09-36f121782d2e-istio-envoy\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zgdl8\" (UID: \"920eea17-797a-4c19-9f09-36f121782d2e\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zgdl8" Nov 03 01:30:04.088513 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:04.088471 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/920eea17-797a-4c19-9f09-36f121782d2e-istio-podinfo\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zgdl8\" (UID: \"920eea17-797a-4c19-9f09-36f121782d2e\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zgdl8" Nov 03 01:30:04.093095 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:04.093066 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qg594\" (UniqueName: \"kubernetes.io/projected/920eea17-797a-4c19-9f09-36f121782d2e-kube-api-access-qg594\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zgdl8\" (UID: \"920eea17-797a-4c19-9f09-36f121782d2e\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zgdl8" Nov 03 01:30:04.093226 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:04.093207 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/920eea17-797a-4c19-9f09-36f121782d2e-istio-token\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zgdl8\" (UID: \"920eea17-797a-4c19-9f09-36f121782d2e\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zgdl8" Nov 03 01:30:04.315939 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:04.315850 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zgdl8" Nov 03 01:30:04.535513 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:04.533913 4421 scope.go:122] "RemoveContainer" containerID="0b9e95a6280b30c3029e49f427260e6a25e44c2e9dec42514fdb23f4d8701792" Nov 03 01:30:04.535513 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:30:04.534555 4421 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-be59df-29894490-4l7wj_odh-ai-gateway-infra(8b32bc18-0be3-4f64-811c-ce4b662df3e6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-be59df-29894490-4l7wj" podUID="8b32bc18-0be3-4f64-811c-ce4b662df3e6" Nov 03 01:30:04.540387 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:04.537642 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-7nssr" event={"ID":"4a72680d-8a9d-4aed-89c9-50206468ea74","Type":"ContainerDied","Data":"cc44f065d5a69a61292301609539afaddff33c6402f64107ef35008d0c4983c4"} Nov 03 01:30:04.540387 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:04.537715 4421 scope.go:122] "RemoveContainer" containerID="313995a8e8d94d5c2a3e4375572a13e07e40738c5db13afca7b4c3b3dca9312f" Nov 03 01:30:04.540387 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:04.537837 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-7nssr" Nov 03 01:30:04.601419 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:04.601333 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-7nssr"] Nov 03 01:30:04.603965 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:04.603940 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-7nssr"] Nov 03 01:30:04.632968 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:30:04.624132 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod920eea17_797a_4c19_9f09_36f121782d2e.slice/crio-72338106d88185586eddfd3755eb712102e2d30a941fe9caba3f1caa0e6a1435 WatchSource:0}: Error finding container 72338106d88185586eddfd3755eb712102e2d30a941fe9caba3f1caa0e6a1435: Status 404 returned error can't find the container with id 72338106d88185586eddfd3755eb712102e2d30a941fe9caba3f1caa0e6a1435 Nov 03 01:30:04.632968 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:04.630701 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:30:04.632968 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:04.630803 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:30:04.632968 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:04.630850 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:30:04.652268 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:04.652226 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zgdl8"] Nov 03 01:30:05.849295 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:05.845558 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zgdl8" event={"ID":"920eea17-797a-4c19-9f09-36f121782d2e","Type":"ContainerStarted","Data":"1cf56a802f20d20a896f34cee26bb9327a975f2779269509d11017a42b7c66c6"} Nov 03 01:30:05.849295 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:05.845635 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zgdl8" event={"ID":"920eea17-797a-4c19-9f09-36f121782d2e","Type":"ContainerStarted","Data":"72338106d88185586eddfd3755eb712102e2d30a941fe9caba3f1caa0e6a1435"} Nov 03 01:30:05.858381 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:05.856547 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8gm8n" podUID="2d367e12-e46f-4a68-9825-9b1c934d4d12" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": context deadline exceeded" Nov 03 01:30:05.885115 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:05.883161 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zgdl8" podStartSLOduration=226.883136186 podStartE2EDuration="3m46.883136186s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:26:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:30:05.873617492 +0000 UTC m=+2948.044620624" watchObservedRunningTime="2026-11-03 01:30:05.883136186 +0000 UTC m=+2948.054139283" Nov 03 01:30:06.343296 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:06.314636 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a72680d-8a9d-4aed-89c9-50206468ea74" path="/var/lib/kubelet/pods/4a72680d-8a9d-4aed-89c9-50206468ea74/volumes" Nov 03 01:30:06.343296 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:06.333537 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zgdl8" Nov 03 01:30:06.345411 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:06.343938 4421 patch_prober.go:29] interesting pod/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zgdl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.75:15021/healthz/ready\": dial tcp 10.134.1.75:15021: connect: connection refused" start-of-body= Nov 03 01:30:06.345411 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:06.344009 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zgdl8" podUID="920eea17-797a-4c19-9f09-36f121782d2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.75:15021/healthz/ready\": dial tcp 10.134.1.75:15021: connect: connection refused" Nov 03 01:30:06.489473 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:06.489422 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/simulator-w3-83755a-kserve-77d6664559-psqtg" podUID="b0a4c234-af58-4da0-a1a4-e1f167953f6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:30:06.660363 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:06.653362 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/premium-w3-83755a-kserve-7c9d66988b-lt4xd" podUID="c0c6b02e-2ecb-428e-a13d-be6d7cd1ba39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:30:06.660363 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:06.654120 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/simulator-w2-f7680d-kserve-85bdc86cd6-8gxrg" podUID="2bdcf521-cebd-4d6b-8554-1b60db8b0f3c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 03 01:30:06.685298 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:06.681688 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/distinct-w3-83755a-kserve-86bdbbdd85-2cnnl" podUID="f4ae9624-d127-4a55-837d-7c9ff0267b37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:30:06.798300 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:06.793619 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-650be0/premium-w4-650be0-kserve-7bf8555dfb-z97n5" podUID="40601dc5-808f-456f-826f-6e0c35966ada" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Nov 03 01:30:06.865105 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:06.865055 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8gm8n" podUID="2d367e12-e46f-4a68-9825-9b1c934d4d12" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": context deadline exceeded" Nov 03 01:30:06.886133 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:06.886090 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8gm8n" Nov 03 01:30:06.989560 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:06.989469 4421 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-83755a/distinct-w3-83755a-kserve-86bdbbdd85-2cnnl" podUID="f4ae9624-d127-4a55-837d-7c9ff0267b37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:30:07.437346 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:07.437306 4421 patch_prober.go:29] interesting pod/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zgdl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.75:15021/healthz/ready\": dial tcp 10.134.1.75:15021: connect: connection refused" start-of-body= Nov 03 01:30:07.437641 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:07.437607 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zgdl8" podUID="920eea17-797a-4c19-9f09-36f121782d2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.75:15021/healthz/ready\": dial tcp 10.134.1.75:15021: connect: connection refused" Nov 03 01:30:07.658298 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:07.655522 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/simulator-w2-f7680d-kserve-85bdc86cd6-8gxrg" podUID="2bdcf521-cebd-4d6b-8554-1b60db8b0f3c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:30:07.659382 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:07.659340 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/premium-w3-83755a-kserve-7c9d66988b-lt4xd" podUID="c0c6b02e-2ecb-428e-a13d-be6d7cd1ba39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Nov 03 01:30:07.681298 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:07.670462 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f7680d/simulator-w2-f7680d-kserve-85bdc86cd6-8gxrg" Nov 03 01:30:07.681298 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:07.670542 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-83755a/premium-w3-83755a-kserve-7c9d66988b-lt4xd" Nov 03 01:30:07.682665 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:07.682620 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/distinct-w3-83755a-kserve-86bdbbdd85-2cnnl" podUID="f4ae9624-d127-4a55-837d-7c9ff0267b37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:30:07.683822 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:07.682902 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-83755a/distinct-w3-83755a-kserve-86bdbbdd85-2cnnl" Nov 03 01:30:07.793206 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:07.793160 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-650be0/simulator-w4-650be0-kserve-7bc588774f-gvz2k" podUID="a827fa88-f078-4312-8f12-2c799c11b461" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 01:30:07.795545 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:07.795508 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-650be0/premium-w4-650be0-kserve-7bf8555dfb-z97n5" podUID="40601dc5-808f-456f-826f-6e0c35966ada" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Nov 03 01:30:07.812550 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:07.812516 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-650be0/premium-w4-650be0-kserve-7bf8555dfb-z97n5" Nov 03 01:30:07.873289 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:07.873225 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8gm8n" podUID="2d367e12-e46f-4a68-9825-9b1c934d4d12" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": context deadline exceeded" Nov 03 01:30:08.337310 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:08.336503 4421 patch_prober.go:29] interesting pod/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zgdl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.75:15021/healthz/ready\": dial tcp 10.134.1.75:15021: connect: connection refused" start-of-body= Nov 03 01:30:08.337310 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:08.336561 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zgdl8" podUID="920eea17-797a-4c19-9f09-36f121782d2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.75:15021/healthz/ready\": dial tcp 10.134.1.75:15021: connect: connection refused" Nov 03 01:30:08.583312 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:08.581861 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/simulator-w3-83755a-kserve-77d6664559-psqtg" podUID="b0a4c234-af58-4da0-a1a4-e1f167953f6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Nov 03 01:30:08.661037 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:08.660693 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/premium-w3-83755a-kserve-7c9d66988b-lt4xd" podUID="c0c6b02e-2ecb-428e-a13d-be6d7cd1ba39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:30:08.663296 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:08.662117 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/simulator-w2-f7680d-kserve-85bdc86cd6-8gxrg" podUID="2bdcf521-cebd-4d6b-8554-1b60db8b0f3c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:30:08.809842 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:08.809782 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-650be0/premium-w4-650be0-kserve-7bf8555dfb-z97n5" podUID="40601dc5-808f-456f-826f-6e0c35966ada" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:30:08.819831 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:08.819775 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/distinct-w3-83755a-kserve-86bdbbdd85-2cnnl" podUID="f4ae9624-d127-4a55-837d-7c9ff0267b37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:30:08.863383 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:08.863337 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-650be0/simulator-w4-650be0-kserve-7bc588774f-gvz2k" podUID="a827fa88-f078-4312-8f12-2c799c11b461" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Nov 03 01:30:08.864264 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:08.864235 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-650be0/simulator-w4-650be0-kserve-7bc588774f-gvz2k" Nov 03 01:30:08.904984 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:08.904947 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8gm8n" podUID="2d367e12-e46f-4a68-9825-9b1c934d4d12" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": context deadline exceeded" Nov 03 01:30:09.241295 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:09.224312 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-650be0/simulator-w4-650be0-kserve-7bc588774f-gvz2k" Nov 03 01:30:09.338832 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:09.338789 4421 patch_prober.go:29] interesting pod/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zgdl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.75:15021/healthz/ready\": dial tcp 10.134.1.75:15021: connect: connection refused" start-of-body= Nov 03 01:30:09.339117 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:09.339089 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zgdl8" podUID="920eea17-797a-4c19-9f09-36f121782d2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.75:15021/healthz/ready\": dial tcp 10.134.1.75:15021: connect: connection refused" Nov 03 01:30:09.373009 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:09.372980 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f7680d/simulator-w2-f7680d-kserve-85bdc86cd6-8gxrg" Nov 03 01:30:09.592372 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:09.575974 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-83755a/premium-w3-83755a-kserve-7c9d66988b-lt4xd" Nov 03 01:30:09.592372 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:09.586070 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/simulator-w3-83755a-kserve-77d6664559-psqtg" podUID="b0a4c234-af58-4da0-a1a4-e1f167953f6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:30:09.611197 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:09.611151 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-83755a/simulator-w3-83755a-kserve-77d6664559-psqtg" Nov 03 01:30:09.812588 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:09.811997 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-650be0/premium-w4-650be0-kserve-7bf8555dfb-z97n5" podUID="40601dc5-808f-456f-826f-6e0c35966ada" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:30:09.825359 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:09.821378 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/distinct-w3-83755a-kserve-86bdbbdd85-2cnnl" podUID="f4ae9624-d127-4a55-837d-7c9ff0267b37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:30:09.969722 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:09.969630 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-83755a/distinct-w3-83755a-kserve-86bdbbdd85-2cnnl" Nov 03 01:30:10.016129 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:10.007239 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8gm8n" Nov 03 01:30:10.029751 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:10.029720 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-650be0/premium-w4-650be0-kserve-7bf8555dfb-z97n5" Nov 03 01:30:10.035414 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:10.035382 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-83755a/simulator-w3-83755a-kserve-77d6664559-psqtg" Nov 03 01:30:10.325293 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:10.319699 4421 patch_prober.go:29] interesting pod/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zgdl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.75:15021/healthz/ready\": dial tcp 10.134.1.75:15021: connect: connection refused" start-of-body= Nov 03 01:30:10.325293 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:10.319765 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zgdl8" podUID="920eea17-797a-4c19-9f09-36f121782d2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.75:15021/healthz/ready\": dial tcp 10.134.1.75:15021: connect: connection refused" Nov 03 01:30:11.323675 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:11.323638 4421 patch_prober.go:29] interesting pod/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zgdl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.75:15021/healthz/ready\": dial tcp 10.134.1.75:15021: connect: connection refused" start-of-body= Nov 03 01:30:11.324322 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:11.324261 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zgdl8" podUID="920eea17-797a-4c19-9f09-36f121782d2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.75:15021/healthz/ready\": dial tcp 10.134.1.75:15021: connect: connection refused" Nov 03 01:30:12.335301 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:12.330743 4421 patch_prober.go:29] interesting pod/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zgdl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.75:15021/healthz/ready\": dial tcp 10.134.1.75:15021: connect: connection refused" start-of-body= Nov 03 01:30:12.335301 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:12.330810 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zgdl8" podUID="920eea17-797a-4c19-9f09-36f121782d2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.75:15021/healthz/ready\": dial tcp 10.134.1.75:15021: connect: connection refused" Nov 03 01:30:13.322830 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:13.317698 4421 patch_prober.go:29] interesting pod/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zgdl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.75:15021/healthz/ready\": dial tcp 10.134.1.75:15021: connect: connection refused" start-of-body= Nov 03 01:30:13.322830 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:13.317764 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zgdl8" podUID="920eea17-797a-4c19-9f09-36f121782d2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.75:15021/healthz/ready\": dial tcp 10.134.1.75:15021: connect: connection refused" Nov 03 01:30:14.332294 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:14.330390 4421 patch_prober.go:29] interesting pod/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zgdl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.75:15021/healthz/ready\": dial tcp 10.134.1.75:15021: connect: connection refused" start-of-body= Nov 03 01:30:14.332294 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:14.330472 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zgdl8" podUID="920eea17-797a-4c19-9f09-36f121782d2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.75:15021/healthz/ready\": dial tcp 10.134.1.75:15021: connect: connection refused" Nov 03 01:30:14.691908 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:30:14.691764 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 01:30:15.321666 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:15.321626 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zgdl8" Nov 03 01:30:15.322137 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:15.322115 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zgdl8" Nov 03 01:30:15.322963 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:15.322944 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zgdl8" Nov 03 01:30:15.972046 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:15.972007 4421 scope.go:122] "RemoveContainer" containerID="0b9e95a6280b30c3029e49f427260e6a25e44c2e9dec42514fdb23f4d8701792" Nov 03 01:30:17.204227 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:17.204190 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-27vg6"] Nov 03 01:30:17.204651 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:17.204403 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-27vg6" podUID="793a3a3a-a1fa-4174-a94a-a8382bc6f412" containerName="istio-proxy" containerID="cri-o://bffed1cf3083920f03a0c871e1ec52375da877d82b7ffdc202146f6b8b9749a3" gracePeriod=30 Nov 03 01:30:17.631423 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:17.631384 4421 generic.go:356] "Generic (PLEG): container finished" podID="8b32bc18-0be3-4f64-811c-ce4b662df3e6" containerID="3fedc0fd5f9a7a2aa580ec9ba47005cee64972153a58fd81435ac7be97715884" exitCode=60 Nov 03 01:30:17.631641 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:17.631455 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-be59df-29894490-4l7wj" event={"ID":"8b32bc18-0be3-4f64-811c-ce4b662df3e6","Type":"ContainerDied","Data":"3fedc0fd5f9a7a2aa580ec9ba47005cee64972153a58fd81435ac7be97715884"} Nov 03 01:30:17.631641 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:17.631558 4421 scope.go:122] "RemoveContainer" containerID="0b9e95a6280b30c3029e49f427260e6a25e44c2e9dec42514fdb23f4d8701792" Nov 03 01:30:17.632066 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:17.632045 4421 scope.go:122] "RemoveContainer" containerID="3fedc0fd5f9a7a2aa580ec9ba47005cee64972153a58fd81435ac7be97715884" Nov 03 01:30:17.632586 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:30:17.632557 4421 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-be59df-29894490-4l7wj_odh-ai-gateway-infra(8b32bc18-0be3-4f64-811c-ce4b662df3e6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-be59df-29894490-4l7wj" podUID="8b32bc18-0be3-4f64-811c-ce4b662df3e6" Nov 03 01:30:18.243522 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:18.243470 4421 patch_prober.go:29] interesting pod/e2e-inf-b-5dab6a-openshift-default-6ff495656c-bxn8s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 01:30:18.244065 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:18.243545 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-5dab6a-openshift-default-6ff495656c-bxn8s" podUID="e0c12073-7ff8-413c-9940-978f4b600778" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 01:30:18.685419 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:18.685375 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-be59df-29894490-4l7wj"] Nov 03 01:30:19.023749 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:19.023716 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-be59df-29894490-4l7wj" Nov 03 01:30:19.055449 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:19.055416 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-50e623-29894490-mgsqg"] Nov 03 01:30:19.056480 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:19.056456 4421 cpu_manager.go:414] "containerMap: removing container" podUID="8b32bc18-0be3-4f64-811c-ce4b662df3e6" containerName="cleanup" Nov 03 01:30:19.056480 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:19.056475 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b32bc18-0be3-4f64-811c-ce4b662df3e6" containerName="cleanup" Nov 03 01:30:19.056738 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:19.056503 4421 cpu_manager.go:414] "containerMap: removing container" podUID="8b32bc18-0be3-4f64-811c-ce4b662df3e6" containerName="cleanup" Nov 03 01:30:19.056738 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:19.056508 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b32bc18-0be3-4f64-811c-ce4b662df3e6" containerName="cleanup" Nov 03 01:30:19.056738 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:19.056530 4421 cpu_manager.go:414] "containerMap: removing container" podUID="8b32bc18-0be3-4f64-811c-ce4b662df3e6" containerName="cleanup" Nov 03 01:30:19.056738 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:19.056535 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b32bc18-0be3-4f64-811c-ce4b662df3e6" containerName="cleanup" Nov 03 01:30:19.056738 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:19.056715 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b32bc18-0be3-4f64-811c-ce4b662df3e6" containerName="cleanup" Nov 03 01:30:19.056738 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:19.056735 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b32bc18-0be3-4f64-811c-ce4b662df3e6" containerName="cleanup" Nov 03 01:30:19.057587 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:19.056751 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b32bc18-0be3-4f64-811c-ce4b662df3e6" containerName="cleanup" Nov 03 01:30:19.127029 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:19.126990 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b32bc18-0be3-4f64-811c-ce4b662df3e6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b32bc18-0be3-4f64-811c-ce4b662df3e6-maas-api-service-ca\") pod \"8b32bc18-0be3-4f64-811c-ce4b662df3e6\" (UID: \"8b32bc18-0be3-4f64-811c-ce4b662df3e6\") " Nov 03 01:30:19.127235 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:19.127111 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b32bc18-0be3-4f64-811c-ce4b662df3e6-kube-api-access-cp675\" (UniqueName: \"kubernetes.io/projected/8b32bc18-0be3-4f64-811c-ce4b662df3e6-kube-api-access-cp675\") pod \"8b32bc18-0be3-4f64-811c-ce4b662df3e6\" (UID: \"8b32bc18-0be3-4f64-811c-ce4b662df3e6\") " Nov 03 01:30:19.127425 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:19.127402 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b32bc18-0be3-4f64-811c-ce4b662df3e6-maas-api-service-ca" pod "8b32bc18-0be3-4f64-811c-ce4b662df3e6" (UID: "8b32bc18-0be3-4f64-811c-ce4b662df3e6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:30:19.127955 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:19.127926 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b32bc18-0be3-4f64-811c-ce4b662df3e6-maas-api-service-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:30:19.129785 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:19.129763 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b32bc18-0be3-4f64-811c-ce4b662df3e6-kube-api-access-cp675" pod "8b32bc18-0be3-4f64-811c-ce4b662df3e6" (UID: "8b32bc18-0be3-4f64-811c-ce4b662df3e6"). InnerVolumeSpecName "kube-api-access-cp675". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:30:19.157865 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:19.157818 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-50e623-29894490-mgsqg"] Nov 03 01:30:19.158054 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:19.157918 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-50e623-29894490-mgsqg" Nov 03 01:30:19.229253 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:19.229218 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/10c1b824-d0d5-4b84-8961-c41c36242725-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-50e623-29894490-mgsqg\" (UID: \"10c1b824-d0d5-4b84-8961-c41c36242725\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-50e623-29894490-mgsqg" Nov 03 01:30:19.229462 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:19.229446 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jfzh\" (UniqueName: \"kubernetes.io/projected/10c1b824-d0d5-4b84-8961-c41c36242725-kube-api-access-4jfzh\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-50e623-29894490-mgsqg\" (UID: \"10c1b824-d0d5-4b84-8961-c41c36242725\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-50e623-29894490-mgsqg" Nov 03 01:30:19.229657 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:19.229637 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cp675\" (UniqueName: \"kubernetes.io/projected/8b32bc18-0be3-4f64-811c-ce4b662df3e6-kube-api-access-cp675\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:30:19.331245 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:19.331149 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/10c1b824-d0d5-4b84-8961-c41c36242725-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-50e623-29894490-mgsqg\" (UID: \"10c1b824-d0d5-4b84-8961-c41c36242725\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-50e623-29894490-mgsqg" Nov 03 01:30:19.331668 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:19.331326 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jfzh\" (UniqueName: \"kubernetes.io/projected/10c1b824-d0d5-4b84-8961-c41c36242725-kube-api-access-4jfzh\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-50e623-29894490-mgsqg\" (UID: \"10c1b824-d0d5-4b84-8961-c41c36242725\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-50e623-29894490-mgsqg" Nov 03 01:30:19.332113 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:19.332082 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/10c1b824-d0d5-4b84-8961-c41c36242725-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-50e623-29894490-mgsqg\" (UID: \"10c1b824-d0d5-4b84-8961-c41c36242725\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-50e623-29894490-mgsqg" Nov 03 01:30:19.339050 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:19.339025 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jfzh\" (UniqueName: \"kubernetes.io/projected/10c1b824-d0d5-4b84-8961-c41c36242725-kube-api-access-4jfzh\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-50e623-29894490-mgsqg\" (UID: \"10c1b824-d0d5-4b84-8961-c41c36242725\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-50e623-29894490-mgsqg" Nov 03 01:30:19.593309 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:19.593198 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-50e623-29894490-mgsqg" Nov 03 01:30:19.662535 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:19.662498 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-be59df-29894490-4l7wj" Nov 03 01:30:19.662743 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:19.662488 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-be59df-29894490-4l7wj" event={"ID":"8b32bc18-0be3-4f64-811c-ce4b662df3e6","Type":"ContainerDied","Data":"b2f507300533a71cf9846753a6e686a241bc1053127b9817221452cb236410ad"} Nov 03 01:30:19.662743 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:19.662666 4421 scope.go:122] "RemoveContainer" containerID="3fedc0fd5f9a7a2aa580ec9ba47005cee64972153a58fd81435ac7be97715884" Nov 03 01:30:19.690789 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:19.690763 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-be59df-29894490-4l7wj"] Nov 03 01:30:19.706937 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:19.706904 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-be59df-29894490-4l7wj"] Nov 03 01:30:19.964689 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:30:19.964654 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10c1b824_d0d5_4b84_8961_c41c36242725.slice/crio-5956c8fe1645f8e60dd1c0846c700acce2fbeefee32a600070e2c5ccae0d60ea WatchSource:0}: Error finding container 5956c8fe1645f8e60dd1c0846c700acce2fbeefee32a600070e2c5ccae0d60ea: Status 404 returned error can't find the container with id 5956c8fe1645f8e60dd1c0846c700acce2fbeefee32a600070e2c5ccae0d60ea Nov 03 01:30:19.964902 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:19.964877 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-50e623-29894490-mgsqg"] Nov 03 01:30:20.006185 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:20.006149 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b32bc18-0be3-4f64-811c-ce4b662df3e6" path="/var/lib/kubelet/pods/8b32bc18-0be3-4f64-811c-ce4b662df3e6/volumes" Nov 03 01:30:20.677821 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:20.677740 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-50e623-29894490-mgsqg" event={"ID":"10c1b824-d0d5-4b84-8961-c41c36242725","Type":"ContainerStarted","Data":"103df8198a7fbc709106d177574a57eb04388282e56e3aff4d296bf6b69a2a94"} Nov 03 01:30:20.677821 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:20.677791 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-50e623-29894490-mgsqg" event={"ID":"10c1b824-d0d5-4b84-8961-c41c36242725","Type":"ContainerStarted","Data":"5956c8fe1645f8e60dd1c0846c700acce2fbeefee32a600070e2c5ccae0d60ea"} Nov 03 01:30:20.707420 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:20.707359 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-50e623-29894490-mgsqg" podStartSLOduration=20.707340491 podStartE2EDuration="20.707340491s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:30:20.704377321 +0000 UTC m=+2962.875380416" watchObservedRunningTime="2026-11-03 01:30:20.707340491 +0000 UTC m=+2962.878343589" Nov 03 01:30:21.689110 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:21.689070 4421 generic.go:356] "Generic (PLEG): container finished" podID="10c1b824-d0d5-4b84-8961-c41c36242725" containerID="103df8198a7fbc709106d177574a57eb04388282e56e3aff4d296bf6b69a2a94" exitCode=60 Nov 03 01:30:21.689641 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:21.689153 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-50e623-29894490-mgsqg" event={"ID":"10c1b824-d0d5-4b84-8961-c41c36242725","Type":"ContainerDied","Data":"103df8198a7fbc709106d177574a57eb04388282e56e3aff4d296bf6b69a2a94"} Nov 03 01:30:21.689641 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:21.689574 4421 scope.go:122] "RemoveContainer" containerID="103df8198a7fbc709106d177574a57eb04388282e56e3aff4d296bf6b69a2a94" Nov 03 01:30:22.560015 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:22.559976 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-27vg6" Nov 03 01:30:22.604990 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:22.601344 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6c0ec0-29894490-5p6qm"] Nov 03 01:30:22.604990 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:22.603312 4421 cpu_manager.go:414] "containerMap: removing container" podUID="793a3a3a-a1fa-4174-a94a-a8382bc6f412" containerName="istio-proxy" Nov 03 01:30:22.604990 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:22.603364 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="793a3a3a-a1fa-4174-a94a-a8382bc6f412" containerName="istio-proxy" Nov 03 01:30:22.604990 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:22.604267 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="793a3a3a-a1fa-4174-a94a-a8382bc6f412" containerName="istio-proxy" Nov 03 01:30:22.654330 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:22.624829 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-af8244-29894490-5dpb4"] Nov 03 01:30:22.654330 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:22.627117 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6c0ec0-29894490-5p6qm" Nov 03 01:30:22.654330 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:22.631729 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/793a3a3a-a1fa-4174-a94a-a8382bc6f412-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/793a3a3a-a1fa-4174-a94a-a8382bc6f412-istio-podinfo\") pod \"793a3a3a-a1fa-4174-a94a-a8382bc6f412\" (UID: \"793a3a3a-a1fa-4174-a94a-a8382bc6f412\") " Nov 03 01:30:22.654330 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:22.631800 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/793a3a3a-a1fa-4174-a94a-a8382bc6f412-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/793a3a3a-a1fa-4174-a94a-a8382bc6f412-workload-socket\") pod \"793a3a3a-a1fa-4174-a94a-a8382bc6f412\" (UID: \"793a3a3a-a1fa-4174-a94a-a8382bc6f412\") " Nov 03 01:30:22.654330 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:22.631880 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/793a3a3a-a1fa-4174-a94a-a8382bc6f412-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/793a3a3a-a1fa-4174-a94a-a8382bc6f412-credential-socket\") pod \"793a3a3a-a1fa-4174-a94a-a8382bc6f412\" (UID: \"793a3a3a-a1fa-4174-a94a-a8382bc6f412\") " Nov 03 01:30:22.654330 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:22.632005 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/793a3a3a-a1fa-4174-a94a-a8382bc6f412-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/793a3a3a-a1fa-4174-a94a-a8382bc6f412-istio-envoy\") pod \"793a3a3a-a1fa-4174-a94a-a8382bc6f412\" (UID: \"793a3a3a-a1fa-4174-a94a-a8382bc6f412\") " Nov 03 01:30:22.654330 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:22.632076 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/793a3a3a-a1fa-4174-a94a-a8382bc6f412-istio-token\" (UniqueName: \"kubernetes.io/projected/793a3a3a-a1fa-4174-a94a-a8382bc6f412-istio-token\") pod \"793a3a3a-a1fa-4174-a94a-a8382bc6f412\" (UID: \"793a3a3a-a1fa-4174-a94a-a8382bc6f412\") " Nov 03 01:30:22.654330 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:22.632189 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/793a3a3a-a1fa-4174-a94a-a8382bc6f412-kube-api-access-ll7v2\" (UniqueName: \"kubernetes.io/projected/793a3a3a-a1fa-4174-a94a-a8382bc6f412-kube-api-access-ll7v2\") pod \"793a3a3a-a1fa-4174-a94a-a8382bc6f412\" (UID: \"793a3a3a-a1fa-4174-a94a-a8382bc6f412\") " Nov 03 01:30:22.654330 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:22.632331 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/793a3a3a-a1fa-4174-a94a-a8382bc6f412-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/793a3a3a-a1fa-4174-a94a-a8382bc6f412-istiod-ca-cert\") pod \"793a3a3a-a1fa-4174-a94a-a8382bc6f412\" (UID: \"793a3a3a-a1fa-4174-a94a-a8382bc6f412\") " Nov 03 01:30:22.654330 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:22.632377 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/793a3a3a-a1fa-4174-a94a-a8382bc6f412-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/793a3a3a-a1fa-4174-a94a-a8382bc6f412-istio-data\") pod \"793a3a3a-a1fa-4174-a94a-a8382bc6f412\" (UID: \"793a3a3a-a1fa-4174-a94a-a8382bc6f412\") " Nov 03 01:30:22.654330 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:22.632456 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/793a3a3a-a1fa-4174-a94a-a8382bc6f412-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/793a3a3a-a1fa-4174-a94a-a8382bc6f412-workload-certs\") pod \"793a3a3a-a1fa-4174-a94a-a8382bc6f412\" (UID: \"793a3a3a-a1fa-4174-a94a-a8382bc6f412\") " Nov 03 01:30:22.654330 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:22.645094 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/793a3a3a-a1fa-4174-a94a-a8382bc6f412-istiod-ca-cert" pod "793a3a3a-a1fa-4174-a94a-a8382bc6f412" (UID: "793a3a3a-a1fa-4174-a94a-a8382bc6f412"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:30:22.654330 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:22.645431 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/793a3a3a-a1fa-4174-a94a-a8382bc6f412-credential-socket" pod "793a3a3a-a1fa-4174-a94a-a8382bc6f412" (UID: "793a3a3a-a1fa-4174-a94a-a8382bc6f412"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:30:22.654330 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:22.645913 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/793a3a3a-a1fa-4174-a94a-a8382bc6f412-workload-socket" pod "793a3a3a-a1fa-4174-a94a-a8382bc6f412" (UID: "793a3a3a-a1fa-4174-a94a-a8382bc6f412"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:30:22.654330 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:22.652981 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/793a3a3a-a1fa-4174-a94a-a8382bc6f412-istio-data" pod "793a3a3a-a1fa-4174-a94a-a8382bc6f412" (UID: "793a3a3a-a1fa-4174-a94a-a8382bc6f412"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:30:22.654330 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:22.653163 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/793a3a3a-a1fa-4174-a94a-a8382bc6f412-workload-certs" pod "793a3a3a-a1fa-4174-a94a-a8382bc6f412" (UID: "793a3a3a-a1fa-4174-a94a-a8382bc6f412"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:30:22.675846 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:22.674243 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b4971a-29894490-f47mr"] Nov 03 01:30:22.675846 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:22.674468 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/793a3a3a-a1fa-4174-a94a-a8382bc6f412-istio-token" pod "793a3a3a-a1fa-4174-a94a-a8382bc6f412" (UID: "793a3a3a-a1fa-4174-a94a-a8382bc6f412"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:30:22.677579 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:22.677552 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-af8244-29894490-5dpb4" Nov 03 01:30:22.687066 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:22.687038 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-83755a-29894490-txc6m"] Nov 03 01:30:22.691775 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:22.691735 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b4971a-29894490-f47mr" Nov 03 01:30:22.725831 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:22.724332 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894490-tqrss"] Nov 03 01:30:22.728184 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:22.728070 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/793a3a3a-a1fa-4174-a94a-a8382bc6f412-istio-podinfo" pod "793a3a3a-a1fa-4174-a94a-a8382bc6f412" (UID: "793a3a3a-a1fa-4174-a94a-a8382bc6f412"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:30:22.728386 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:22.728186 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-83755a-29894490-txc6m" Nov 03 01:30:22.737174 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:22.737139 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b230406-0b7d-4d69-ac8e-b5e57164a5e4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-af8244-29894490-5dpb4\" (UID: \"1b230406-0b7d-4d69-ac8e-b5e57164a5e4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-af8244-29894490-5dpb4" Nov 03 01:30:22.738573 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:22.738542 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/645536c5-e930-48f6-bbf2-f4591ee8c368-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-6c0ec0-29894490-5p6qm\" (UID: \"645536c5-e930-48f6-bbf2-f4591ee8c368\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6c0ec0-29894490-5p6qm" Nov 03 01:30:22.738754 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:22.738657 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/793a3a3a-a1fa-4174-a94a-a8382bc6f412-kube-api-access-ll7v2" pod "793a3a3a-a1fa-4174-a94a-a8382bc6f412" (UID: "793a3a3a-a1fa-4174-a94a-a8382bc6f412"). InnerVolumeSpecName "kube-api-access-ll7v2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:30:22.738849 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:22.738724 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/793a3a3a-a1fa-4174-a94a-a8382bc6f412-istio-envoy" pod "793a3a3a-a1fa-4174-a94a-a8382bc6f412" (UID: "793a3a3a-a1fa-4174-a94a-a8382bc6f412"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:30:22.738909 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:22.738804 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hz872\" (UniqueName: \"kubernetes.io/projected/1b230406-0b7d-4d69-ac8e-b5e57164a5e4-kube-api-access-hz872\") pod \"maas-api-key-cleanup-e2e-extmod-b-af8244-29894490-5dpb4\" (UID: \"1b230406-0b7d-4d69-ac8e-b5e57164a5e4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-af8244-29894490-5dpb4" Nov 03 01:30:22.739057 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:22.739045 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4jtp\" (UniqueName: \"kubernetes.io/projected/645536c5-e930-48f6-bbf2-f4591ee8c368-kube-api-access-w4jtp\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-6c0ec0-29894490-5p6qm\" (UID: \"645536c5-e930-48f6-bbf2-f4591ee8c368\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6c0ec0-29894490-5p6qm" Nov 03 01:30:22.739213 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:22.739204 4421 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/793a3a3a-a1fa-4174-a94a-a8382bc6f412-istio-envoy\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:30:22.739305 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:22.739267 4421 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/793a3a3a-a1fa-4174-a94a-a8382bc6f412-istio-token\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:30:22.739374 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:22.739365 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ll7v2\" (UniqueName: \"kubernetes.io/projected/793a3a3a-a1fa-4174-a94a-a8382bc6f412-kube-api-access-ll7v2\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:30:22.739433 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:22.739426 4421 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/793a3a3a-a1fa-4174-a94a-a8382bc6f412-istiod-ca-cert\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:30:22.739490 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:22.739483 4421 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/793a3a3a-a1fa-4174-a94a-a8382bc6f412-istio-data\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:30:22.739544 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:22.739537 4421 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/793a3a3a-a1fa-4174-a94a-a8382bc6f412-workload-certs\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:30:22.739597 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:22.739590 4421 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/793a3a3a-a1fa-4174-a94a-a8382bc6f412-istio-podinfo\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:30:22.739652 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:22.739645 4421 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/793a3a3a-a1fa-4174-a94a-a8382bc6f412-workload-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:30:22.739723 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:22.739714 4421 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/793a3a3a-a1fa-4174-a94a-a8382bc6f412-credential-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:30:22.750721 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:22.750333 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6fec8a-29894490-r4cj4"] Nov 03 01:30:22.750850 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:22.750790 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894490-tqrss" Nov 03 01:30:22.760607 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:22.760563 4421 generic.go:356] "Generic (PLEG): container finished" podID="793a3a3a-a1fa-4174-a94a-a8382bc6f412" containerID="bffed1cf3083920f03a0c871e1ec52375da877d82b7ffdc202146f6b8b9749a3" exitCode=0 Nov 03 01:30:22.760973 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:22.760950 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-27vg6" Nov 03 01:30:22.767316 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:22.767186 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f7680d-29894490-6dkgx"] Nov 03 01:30:22.767316 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:22.767267 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6fec8a-29894490-r4cj4" Nov 03 01:30:22.783100 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:22.782660 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-27vg6" event={"ID":"793a3a3a-a1fa-4174-a94a-a8382bc6f412","Type":"ContainerDied","Data":"bffed1cf3083920f03a0c871e1ec52375da877d82b7ffdc202146f6b8b9749a3"} Nov 03 01:30:22.783100 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:22.782706 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5dab6a-29894490-4pqxr"] Nov 03 01:30:22.783100 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:22.782782 4421 scope.go:122] "RemoveContainer" containerID="bffed1cf3083920f03a0c871e1ec52375da877d82b7ffdc202146f6b8b9749a3" Nov 03 01:30:22.783100 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:22.782946 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f7680d-29894490-6dkgx" Nov 03 01:30:22.800023 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:22.799985 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-27vg6" event={"ID":"793a3a3a-a1fa-4174-a94a-a8382bc6f412","Type":"ContainerDied","Data":"c6a254689969bbf542bb53903b6b77b023f3429d5db3884073ab3d8b9e80d89c"} Nov 03 01:30:22.800176 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:22.800026 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-50e623-29894490-mgsqg" event={"ID":"10c1b824-d0d5-4b84-8961-c41c36242725","Type":"ContainerStarted","Data":"6bedbda9a52c131d9db6a80bae32ca47ed8242447d9fbf03ec983d4cf93faf25"} Nov 03 01:30:22.800176 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:22.800064 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6c0ec0-29894490-5p6qm"] Nov 03 01:30:22.800176 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:22.800103 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b4971a-29894490-f47mr"] Nov 03 01:30:22.800176 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:22.800132 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-83755a-29894490-txc6m"] Nov 03 01:30:22.800176 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:22.800162 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894490-tqrss"] Nov 03 01:30:22.800526 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:22.800221 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f7680d-29894490-6dkgx"] Nov 03 01:30:22.800526 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:22.800252 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5dab6a-29894490-4pqxr"] Nov 03 01:30:22.800526 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:22.800386 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5dab6a-29894490-4pqxr" Nov 03 01:30:22.805928 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:22.805900 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6fec8a-29894490-r4cj4"] Nov 03 01:30:22.806993 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:22.806973 4421 scope.go:122] "RemoveContainer" containerID="bffed1cf3083920f03a0c871e1ec52375da877d82b7ffdc202146f6b8b9749a3" Nov 03 01:30:22.807384 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:30:22.807347 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bffed1cf3083920f03a0c871e1ec52375da877d82b7ffdc202146f6b8b9749a3\": container with ID starting with bffed1cf3083920f03a0c871e1ec52375da877d82b7ffdc202146f6b8b9749a3 not found: ID does not exist" containerID="bffed1cf3083920f03a0c871e1ec52375da877d82b7ffdc202146f6b8b9749a3" Nov 03 01:30:22.807476 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:22.807383 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bffed1cf3083920f03a0c871e1ec52375da877d82b7ffdc202146f6b8b9749a3"} err="failed to get container status \"bffed1cf3083920f03a0c871e1ec52375da877d82b7ffdc202146f6b8b9749a3\": rpc error: code = NotFound desc = could not find container \"bffed1cf3083920f03a0c871e1ec52375da877d82b7ffdc202146f6b8b9749a3\": container with ID starting with bffed1cf3083920f03a0c871e1ec52375da877d82b7ffdc202146f6b8b9749a3 not found: ID does not exist" Nov 03 01:30:22.820672 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:22.818320 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-af8244-29894490-5dpb4"] Nov 03 01:30:22.841635 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:22.841602 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rv7d2\" (UniqueName: \"kubernetes.io/projected/58e1f190-d19c-43f7-afe2-e3079e8d9814-kube-api-access-rv7d2\") pod \"maas-api-key-cleanup-e2e-inf-b-b4971a-29894490-f47mr\" (UID: \"58e1f190-d19c-43f7-afe2-e3079e8d9814\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b4971a-29894490-f47mr" Nov 03 01:30:22.841814 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:22.841643 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/58e1f190-d19c-43f7-afe2-e3079e8d9814-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-b4971a-29894490-f47mr\" (UID: \"58e1f190-d19c-43f7-afe2-e3079e8d9814\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b4971a-29894490-f47mr" Nov 03 01:30:22.841814 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:22.841708 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bb86426-5f37-4afc-9d67-83399abe9862-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-6fec8a-29894490-r4cj4\" (UID: \"6bb86426-5f37-4afc-9d67-83399abe9862\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6fec8a-29894490-r4cj4" Nov 03 01:30:22.841814 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:22.841736 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmzsd\" (UniqueName: \"kubernetes.io/projected/e9bfb106-b718-4909-a826-79040347a857-kube-api-access-gmzsd\") pod \"maas-api-key-cleanup-e2e-worker-w2-f7680d-29894490-6dkgx\" (UID: \"e9bfb106-b718-4909-a826-79040347a857\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f7680d-29894490-6dkgx" Nov 03 01:30:22.841814 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:22.841805 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1579a23a-e983-4da2-b9be-427fe00d9de3-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894490-tqrss\" (UID: \"1579a23a-e983-4da2-b9be-427fe00d9de3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894490-tqrss" Nov 03 01:30:22.841979 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:22.841914 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91be6fd5-032c-4abe-aedf-e753fe1d7386-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-83755a-29894490-txc6m\" (UID: \"91be6fd5-032c-4abe-aedf-e753fe1d7386\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-83755a-29894490-txc6m" Nov 03 01:30:22.842023 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:22.841991 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l577x\" (UniqueName: \"kubernetes.io/projected/6bb86426-5f37-4afc-9d67-83399abe9862-kube-api-access-l577x\") pod \"maas-api-key-cleanup-e2e-inf-a-6fec8a-29894490-r4cj4\" (UID: \"6bb86426-5f37-4afc-9d67-83399abe9862\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6fec8a-29894490-r4cj4" Nov 03 01:30:22.842064 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:22.842030 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/645536c5-e930-48f6-bbf2-f4591ee8c368-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-6c0ec0-29894490-5p6qm\" (UID: \"645536c5-e930-48f6-bbf2-f4591ee8c368\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6c0ec0-29894490-5p6qm" Nov 03 01:30:22.842137 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:22.842121 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hz872\" (UniqueName: \"kubernetes.io/projected/1b230406-0b7d-4d69-ac8e-b5e57164a5e4-kube-api-access-hz872\") pod \"maas-api-key-cleanup-e2e-extmod-b-af8244-29894490-5dpb4\" (UID: \"1b230406-0b7d-4d69-ac8e-b5e57164a5e4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-af8244-29894490-5dpb4" Nov 03 01:30:22.842185 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:22.842152 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfzfl\" (UniqueName: \"kubernetes.io/projected/1579a23a-e983-4da2-b9be-427fe00d9de3-kube-api-access-xfzfl\") pod \"maas-api-key-cleanup-29894490-tqrss\" (UID: \"1579a23a-e983-4da2-b9be-427fe00d9de3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894490-tqrss" Nov 03 01:30:22.842242 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:22.842180 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9bfb106-b718-4909-a826-79040347a857-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-f7680d-29894490-6dkgx\" (UID: \"e9bfb106-b718-4909-a826-79040347a857\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f7680d-29894490-6dkgx" Nov 03 01:30:22.842482 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:22.842458 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4jtp\" (UniqueName: \"kubernetes.io/projected/645536c5-e930-48f6-bbf2-f4591ee8c368-kube-api-access-w4jtp\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-6c0ec0-29894490-5p6qm\" (UID: \"645536c5-e930-48f6-bbf2-f4591ee8c368\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6c0ec0-29894490-5p6qm" Nov 03 01:30:22.842608 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:22.842498 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6v8l4\" (UniqueName: \"kubernetes.io/projected/91be6fd5-032c-4abe-aedf-e753fe1d7386-kube-api-access-6v8l4\") pod \"maas-api-key-cleanup-e2e-worker-w3-83755a-29894490-txc6m\" (UID: \"91be6fd5-032c-4abe-aedf-e753fe1d7386\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-83755a-29894490-txc6m" Nov 03 01:30:22.842784 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:22.842768 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/645536c5-e930-48f6-bbf2-f4591ee8c368-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-6c0ec0-29894490-5p6qm\" (UID: \"645536c5-e930-48f6-bbf2-f4591ee8c368\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6c0ec0-29894490-5p6qm" Nov 03 01:30:22.842855 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:22.842777 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b230406-0b7d-4d69-ac8e-b5e57164a5e4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-af8244-29894490-5dpb4\" (UID: \"1b230406-0b7d-4d69-ac8e-b5e57164a5e4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-af8244-29894490-5dpb4" Nov 03 01:30:22.843384 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:22.843362 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b230406-0b7d-4d69-ac8e-b5e57164a5e4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-af8244-29894490-5dpb4\" (UID: \"1b230406-0b7d-4d69-ac8e-b5e57164a5e4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-af8244-29894490-5dpb4" Nov 03 01:30:22.855400 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:22.855374 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hz872\" (UniqueName: \"kubernetes.io/projected/1b230406-0b7d-4d69-ac8e-b5e57164a5e4-kube-api-access-hz872\") pod \"maas-api-key-cleanup-e2e-extmod-b-af8244-29894490-5dpb4\" (UID: \"1b230406-0b7d-4d69-ac8e-b5e57164a5e4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-af8244-29894490-5dpb4" Nov 03 01:30:22.855614 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:22.855592 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4jtp\" (UniqueName: \"kubernetes.io/projected/645536c5-e930-48f6-bbf2-f4591ee8c368-kube-api-access-w4jtp\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-6c0ec0-29894490-5p6qm\" (UID: \"645536c5-e930-48f6-bbf2-f4591ee8c368\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6c0ec0-29894490-5p6qm" Nov 03 01:30:22.866942 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:22.866904 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-27vg6"] Nov 03 01:30:22.871389 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:22.871363 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-27vg6"] Nov 03 01:30:22.967098 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:22.948387 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fj5g9\" (UniqueName: \"kubernetes.io/projected/917a9367-6e36-4f93-a9a6-b65bfd868704-kube-api-access-fj5g9\") pod \"maas-api-key-cleanup-e2e-inf-b-5dab6a-29894490-4pqxr\" (UID: \"917a9367-6e36-4f93-a9a6-b65bfd868704\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5dab6a-29894490-4pqxr" Nov 03 01:30:22.967098 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:22.948519 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rv7d2\" (UniqueName: \"kubernetes.io/projected/58e1f190-d19c-43f7-afe2-e3079e8d9814-kube-api-access-rv7d2\") pod \"maas-api-key-cleanup-e2e-inf-b-b4971a-29894490-f47mr\" (UID: \"58e1f190-d19c-43f7-afe2-e3079e8d9814\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b4971a-29894490-f47mr" Nov 03 01:30:22.967098 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:22.948555 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/58e1f190-d19c-43f7-afe2-e3079e8d9814-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-b4971a-29894490-f47mr\" (UID: \"58e1f190-d19c-43f7-afe2-e3079e8d9814\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b4971a-29894490-f47mr" Nov 03 01:30:22.967098 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:22.948590 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bb86426-5f37-4afc-9d67-83399abe9862-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-6fec8a-29894490-r4cj4\" (UID: \"6bb86426-5f37-4afc-9d67-83399abe9862\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6fec8a-29894490-r4cj4" Nov 03 01:30:22.967098 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:22.948619 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmzsd\" (UniqueName: \"kubernetes.io/projected/e9bfb106-b718-4909-a826-79040347a857-kube-api-access-gmzsd\") pod \"maas-api-key-cleanup-e2e-worker-w2-f7680d-29894490-6dkgx\" (UID: \"e9bfb106-b718-4909-a826-79040347a857\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f7680d-29894490-6dkgx" Nov 03 01:30:22.967098 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:22.948659 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1579a23a-e983-4da2-b9be-427fe00d9de3-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894490-tqrss\" (UID: \"1579a23a-e983-4da2-b9be-427fe00d9de3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894490-tqrss" Nov 03 01:30:22.967098 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:22.948775 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91be6fd5-032c-4abe-aedf-e753fe1d7386-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-83755a-29894490-txc6m\" (UID: \"91be6fd5-032c-4abe-aedf-e753fe1d7386\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-83755a-29894490-txc6m" Nov 03 01:30:22.967098 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:22.948853 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l577x\" (UniqueName: \"kubernetes.io/projected/6bb86426-5f37-4afc-9d67-83399abe9862-kube-api-access-l577x\") pod \"maas-api-key-cleanup-e2e-inf-a-6fec8a-29894490-r4cj4\" (UID: \"6bb86426-5f37-4afc-9d67-83399abe9862\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6fec8a-29894490-r4cj4" Nov 03 01:30:22.967098 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:22.948989 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfzfl\" (UniqueName: \"kubernetes.io/projected/1579a23a-e983-4da2-b9be-427fe00d9de3-kube-api-access-xfzfl\") pod \"maas-api-key-cleanup-29894490-tqrss\" (UID: \"1579a23a-e983-4da2-b9be-427fe00d9de3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894490-tqrss" Nov 03 01:30:22.967098 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:22.949017 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9bfb106-b718-4909-a826-79040347a857-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-f7680d-29894490-6dkgx\" (UID: \"e9bfb106-b718-4909-a826-79040347a857\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f7680d-29894490-6dkgx" Nov 03 01:30:22.967098 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:22.949113 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/917a9367-6e36-4f93-a9a6-b65bfd868704-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-5dab6a-29894490-4pqxr\" (UID: \"917a9367-6e36-4f93-a9a6-b65bfd868704\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5dab6a-29894490-4pqxr" Nov 03 01:30:22.967098 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:22.949171 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6v8l4\" (UniqueName: \"kubernetes.io/projected/91be6fd5-032c-4abe-aedf-e753fe1d7386-kube-api-access-6v8l4\") pod \"maas-api-key-cleanup-e2e-worker-w3-83755a-29894490-txc6m\" (UID: \"91be6fd5-032c-4abe-aedf-e753fe1d7386\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-83755a-29894490-txc6m" Nov 03 01:30:22.983955 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:22.978424 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6v8l4\" (UniqueName: \"kubernetes.io/projected/91be6fd5-032c-4abe-aedf-e753fe1d7386-kube-api-access-6v8l4\") pod \"maas-api-key-cleanup-e2e-worker-w3-83755a-29894490-txc6m\" (UID: \"91be6fd5-032c-4abe-aedf-e753fe1d7386\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-83755a-29894490-txc6m" Nov 03 01:30:22.983955 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:22.979149 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bb86426-5f37-4afc-9d67-83399abe9862-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-6fec8a-29894490-r4cj4\" (UID: \"6bb86426-5f37-4afc-9d67-83399abe9862\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6fec8a-29894490-r4cj4" Nov 03 01:30:22.983955 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:22.979878 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/58e1f190-d19c-43f7-afe2-e3079e8d9814-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-b4971a-29894490-f47mr\" (UID: \"58e1f190-d19c-43f7-afe2-e3079e8d9814\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b4971a-29894490-f47mr" Nov 03 01:30:22.983955 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:22.980109 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1579a23a-e983-4da2-b9be-427fe00d9de3-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894490-tqrss\" (UID: \"1579a23a-e983-4da2-b9be-427fe00d9de3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894490-tqrss" Nov 03 01:30:22.983955 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:22.980808 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91be6fd5-032c-4abe-aedf-e753fe1d7386-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-83755a-29894490-txc6m\" (UID: \"91be6fd5-032c-4abe-aedf-e753fe1d7386\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-83755a-29894490-txc6m" Nov 03 01:30:22.990963 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:22.989621 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9bfb106-b718-4909-a826-79040347a857-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-f7680d-29894490-6dkgx\" (UID: \"e9bfb106-b718-4909-a826-79040347a857\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f7680d-29894490-6dkgx" Nov 03 01:30:23.003791 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:23.003692 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfzfl\" (UniqueName: \"kubernetes.io/projected/1579a23a-e983-4da2-b9be-427fe00d9de3-kube-api-access-xfzfl\") pod \"maas-api-key-cleanup-29894490-tqrss\" (UID: \"1579a23a-e983-4da2-b9be-427fe00d9de3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894490-tqrss" Nov 03 01:30:23.004329 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:23.004268 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rv7d2\" (UniqueName: \"kubernetes.io/projected/58e1f190-d19c-43f7-afe2-e3079e8d9814-kube-api-access-rv7d2\") pod \"maas-api-key-cleanup-e2e-inf-b-b4971a-29894490-f47mr\" (UID: \"58e1f190-d19c-43f7-afe2-e3079e8d9814\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b4971a-29894490-f47mr" Nov 03 01:30:23.006375 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:23.006311 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmzsd\" (UniqueName: \"kubernetes.io/projected/e9bfb106-b718-4909-a826-79040347a857-kube-api-access-gmzsd\") pod \"maas-api-key-cleanup-e2e-worker-w2-f7680d-29894490-6dkgx\" (UID: \"e9bfb106-b718-4909-a826-79040347a857\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f7680d-29894490-6dkgx" Nov 03 01:30:23.011044 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:23.010977 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l577x\" (UniqueName: \"kubernetes.io/projected/6bb86426-5f37-4afc-9d67-83399abe9862-kube-api-access-l577x\") pod \"maas-api-key-cleanup-e2e-inf-a-6fec8a-29894490-r4cj4\" (UID: \"6bb86426-5f37-4afc-9d67-83399abe9862\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6fec8a-29894490-r4cj4" Nov 03 01:30:23.039057 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:23.039026 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6c0ec0-29894490-5p6qm" Nov 03 01:30:23.050904 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:23.050873 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/917a9367-6e36-4f93-a9a6-b65bfd868704-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-5dab6a-29894490-4pqxr\" (UID: \"917a9367-6e36-4f93-a9a6-b65bfd868704\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5dab6a-29894490-4pqxr" Nov 03 01:30:23.051055 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:23.051024 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fj5g9\" (UniqueName: \"kubernetes.io/projected/917a9367-6e36-4f93-a9a6-b65bfd868704-kube-api-access-fj5g9\") pod \"maas-api-key-cleanup-e2e-inf-b-5dab6a-29894490-4pqxr\" (UID: \"917a9367-6e36-4f93-a9a6-b65bfd868704\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5dab6a-29894490-4pqxr" Nov 03 01:30:23.051656 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:23.051632 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/917a9367-6e36-4f93-a9a6-b65bfd868704-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-5dab6a-29894490-4pqxr\" (UID: \"917a9367-6e36-4f93-a9a6-b65bfd868704\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5dab6a-29894490-4pqxr" Nov 03 01:30:23.059692 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:23.059637 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fj5g9\" (UniqueName: \"kubernetes.io/projected/917a9367-6e36-4f93-a9a6-b65bfd868704-kube-api-access-fj5g9\") pod \"maas-api-key-cleanup-e2e-inf-b-5dab6a-29894490-4pqxr\" (UID: \"917a9367-6e36-4f93-a9a6-b65bfd868704\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5dab6a-29894490-4pqxr" Nov 03 01:30:23.061303 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:23.061283 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-af8244-29894490-5dpb4" Nov 03 01:30:23.077069 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:23.077037 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b4971a-29894490-f47mr" Nov 03 01:30:23.087542 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:23.087511 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-83755a-29894490-txc6m" Nov 03 01:30:23.099795 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:23.099768 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894490-tqrss" Nov 03 01:30:23.114134 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:23.114097 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6fec8a-29894490-r4cj4" Nov 03 01:30:23.118427 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:23.118236 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f7680d-29894490-6dkgx" Nov 03 01:30:23.136143 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:23.136070 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5dab6a-29894490-4pqxr" Nov 03 01:30:23.379458 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:23.366767 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6c0ec0-29894490-5p6qm"] Nov 03 01:30:23.450188 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:23.450150 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-af8244-29894490-5dpb4"] Nov 03 01:30:23.455793 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:30:23.455375 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b230406_0b7d_4d69_ac8e_b5e57164a5e4.slice/crio-068ee3e7952fc7e4c4c29466a5ebe1e39976600347bb1ab24e8a53be1f658548 WatchSource:0}: Error finding container 068ee3e7952fc7e4c4c29466a5ebe1e39976600347bb1ab24e8a53be1f658548: Status 404 returned error can't find the container with id 068ee3e7952fc7e4c4c29466a5ebe1e39976600347bb1ab24e8a53be1f658548 Nov 03 01:30:23.539338 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:23.520481 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b4971a-29894490-f47mr"] Nov 03 01:30:23.539338 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:30:23.530194 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58e1f190_d19c_43f7_afe2_e3079e8d9814.slice/crio-fc8bcdbec76a5f24638e8f6f4f223a9299f9fdede8cb9affd05e22a6b0eca739 WatchSource:0}: Error finding container fc8bcdbec76a5f24638e8f6f4f223a9299f9fdede8cb9affd05e22a6b0eca739: Status 404 returned error can't find the container with id fc8bcdbec76a5f24638e8f6f4f223a9299f9fdede8cb9affd05e22a6b0eca739 Nov 03 01:30:23.642252 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:23.641755 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5dab6a-29894490-4pqxr"] Nov 03 01:30:23.830013 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:23.829824 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b4971a-29894490-f47mr" event={"ID":"58e1f190-d19c-43f7-afe2-e3079e8d9814","Type":"ContainerStarted","Data":"fc8bcdbec76a5f24638e8f6f4f223a9299f9fdede8cb9affd05e22a6b0eca739"} Nov 03 01:30:23.832853 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:23.832781 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6c0ec0-29894490-5p6qm" event={"ID":"645536c5-e930-48f6-bbf2-f4591ee8c368","Type":"ContainerStarted","Data":"225f8191f984a56b6389838cc8d7b0a502d93e0912088f1146edb524f6f74cb7"} Nov 03 01:30:23.834842 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:23.834738 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-af8244-29894490-5dpb4" event={"ID":"1b230406-0b7d-4d69-ac8e-b5e57164a5e4","Type":"ContainerStarted","Data":"068ee3e7952fc7e4c4c29466a5ebe1e39976600347bb1ab24e8a53be1f658548"} Nov 03 01:30:23.841958 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:23.840757 4421 generic.go:356] "Generic (PLEG): container finished" podID="10c1b824-d0d5-4b84-8961-c41c36242725" containerID="6bedbda9a52c131d9db6a80bae32ca47ed8242447d9fbf03ec983d4cf93faf25" exitCode=60 Nov 03 01:30:23.841958 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:23.840852 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-50e623-29894490-mgsqg" event={"ID":"10c1b824-d0d5-4b84-8961-c41c36242725","Type":"ContainerDied","Data":"6bedbda9a52c131d9db6a80bae32ca47ed8242447d9fbf03ec983d4cf93faf25"} Nov 03 01:30:23.841958 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:23.840886 4421 scope.go:122] "RemoveContainer" containerID="103df8198a7fbc709106d177574a57eb04388282e56e3aff4d296bf6b69a2a94" Nov 03 01:30:23.841958 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:23.841226 4421 scope.go:122] "RemoveContainer" containerID="6bedbda9a52c131d9db6a80bae32ca47ed8242447d9fbf03ec983d4cf93faf25" Nov 03 01:30:23.841958 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:30:23.841658 4421 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-50e623-29894490-mgsqg_odh-ai-gateway-infra(10c1b824-d0d5-4b84-8961-c41c36242725)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-50e623-29894490-mgsqg" podUID="10c1b824-d0d5-4b84-8961-c41c36242725" Nov 03 01:30:23.843129 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:23.843080 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5dab6a-29894490-4pqxr" event={"ID":"917a9367-6e36-4f93-a9a6-b65bfd868704","Type":"ContainerStarted","Data":"e41eb92e53a4eade4ffe7da6c9e47302b2da3fa3e1767e9715189d927a041970"} Nov 03 01:30:23.914324 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:23.914232 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894490-tqrss"] Nov 03 01:30:23.930154 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:30:23.930106 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1579a23a_e983_4da2_b9be_427fe00d9de3.slice/crio-5df1dfe9c6f53724b2b897fb900f750529ed36edd1a24749a941488dc6382dc4 WatchSource:0}: Error finding container 5df1dfe9c6f53724b2b897fb900f750529ed36edd1a24749a941488dc6382dc4: Status 404 returned error can't find the container with id 5df1dfe9c6f53724b2b897fb900f750529ed36edd1a24749a941488dc6382dc4 Nov 03 01:30:24.002744 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:24.002702 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="793a3a3a-a1fa-4174-a94a-a8382bc6f412" path="/var/lib/kubelet/pods/793a3a3a-a1fa-4174-a94a-a8382bc6f412/volumes" Nov 03 01:30:24.003533 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:24.003498 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f7680d-29894490-6dkgx"] Nov 03 01:30:24.004195 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:24.004175 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6fec8a-29894490-r4cj4"] Nov 03 01:30:24.012804 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:30:24.012774 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bb86426_5f37_4afc_9d67_83399abe9862.slice/crio-8f648f87e721607e4305a103b9586634a08fcb619a40e32248d857bac4bc788a WatchSource:0}: Error finding container 8f648f87e721607e4305a103b9586634a08fcb619a40e32248d857bac4bc788a: Status 404 returned error can't find the container with id 8f648f87e721607e4305a103b9586634a08fcb619a40e32248d857bac4bc788a Nov 03 01:30:24.019237 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:30:24.019217 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9bfb106_b718_4909_a826_79040347a857.slice/crio-482cfd7e33be5de99e230341ed3b4038ec28ce07871c720c477f6403c8991acd WatchSource:0}: Error finding container 482cfd7e33be5de99e230341ed3b4038ec28ce07871c720c477f6403c8991acd: Status 404 returned error can't find the container with id 482cfd7e33be5de99e230341ed3b4038ec28ce07871c720c477f6403c8991acd Nov 03 01:30:24.036227 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:24.036200 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-83755a-29894490-txc6m"] Nov 03 01:30:24.124258 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:30:24.124222 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91be6fd5_032c_4abe_aedf_e753fe1d7386.slice/crio-e8a7be46d79042cb048e237851e0ee1d0a1243d7352d29e83af44a615f9f5714 WatchSource:0}: Error finding container e8a7be46d79042cb048e237851e0ee1d0a1243d7352d29e83af44a615f9f5714: Status 404 returned error can't find the container with id e8a7be46d79042cb048e237851e0ee1d0a1243d7352d29e83af44a615f9f5714 Nov 03 01:30:24.855090 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:24.855053 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b4971a-29894490-f47mr" event={"ID":"58e1f190-d19c-43f7-afe2-e3079e8d9814","Type":"ContainerStarted","Data":"1824db924c0daceff23ade37f0d3bf294640a2ce9e172b1f25e179508bed8b8d"} Nov 03 01:30:24.856901 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:24.856868 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6c0ec0-29894490-5p6qm" event={"ID":"645536c5-e930-48f6-bbf2-f4591ee8c368","Type":"ContainerStarted","Data":"2c267a51fab51d3e378d58a4dc21d7ff7fb59faea972069d7f83bdffa21db65b"} Nov 03 01:30:24.858658 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:24.858632 4421 generic.go:356] "Generic (PLEG): container finished" podID="1b230406-0b7d-4d69-ac8e-b5e57164a5e4" containerID="07fc7743b11438c89a163a58785b8c9fb087c043b7b5128565b4f7ccb567e27c" exitCode=60 Nov 03 01:30:24.858786 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:24.858711 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-af8244-29894490-5dpb4" event={"ID":"1b230406-0b7d-4d69-ac8e-b5e57164a5e4","Type":"ContainerDied","Data":"07fc7743b11438c89a163a58785b8c9fb087c043b7b5128565b4f7ccb567e27c"} Nov 03 01:30:24.859407 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:24.858963 4421 scope.go:122] "RemoveContainer" containerID="07fc7743b11438c89a163a58785b8c9fb087c043b7b5128565b4f7ccb567e27c" Nov 03 01:30:24.860500 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:24.860478 4421 generic.go:356] "Generic (PLEG): container finished" podID="e0c12073-7ff8-413c-9940-978f4b600778" containerID="4fd927cf38a5c09578c9fa47907e1dbebcfc4f5b21825dd9733c61fb91f2a3f9" exitCode=0 Nov 03 01:30:24.860627 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:24.860548 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5dab6a-openshift-default-6ff495656c-bxn8s" event={"ID":"e0c12073-7ff8-413c-9940-978f4b600778","Type":"ContainerDied","Data":"4fd927cf38a5c09578c9fa47907e1dbebcfc4f5b21825dd9733c61fb91f2a3f9"} Nov 03 01:30:24.860973 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:24.860958 4421 scope.go:122] "RemoveContainer" containerID="4fd927cf38a5c09578c9fa47907e1dbebcfc4f5b21825dd9733c61fb91f2a3f9" Nov 03 01:30:24.863313 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:24.862937 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894490-tqrss" event={"ID":"1579a23a-e983-4da2-b9be-427fe00d9de3","Type":"ContainerStarted","Data":"544091e808775d9fc2ad3edae2829003092456a7b142e6327ccdcf6bde7d396c"} Nov 03 01:30:24.863313 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:24.862997 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894490-tqrss" event={"ID":"1579a23a-e983-4da2-b9be-427fe00d9de3","Type":"ContainerStarted","Data":"5df1dfe9c6f53724b2b897fb900f750529ed36edd1a24749a941488dc6382dc4"} Nov 03 01:30:24.863632 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:24.863320 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:30:24.863632 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:24.863400 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:30:24.863632 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:24.863440 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:30:24.865678 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:24.865657 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-83755a-29894490-txc6m" event={"ID":"91be6fd5-032c-4abe-aedf-e753fe1d7386","Type":"ContainerStarted","Data":"77884d74e096046edef89113a03345da193a9846973d3803cb894f9f6bdcb763"} Nov 03 01:30:24.865780 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:24.865696 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-83755a-29894490-txc6m" event={"ID":"91be6fd5-032c-4abe-aedf-e753fe1d7386","Type":"ContainerStarted","Data":"e8a7be46d79042cb048e237851e0ee1d0a1243d7352d29e83af44a615f9f5714"} Nov 03 01:30:24.867560 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:24.867537 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6fec8a-29894490-r4cj4" event={"ID":"6bb86426-5f37-4afc-9d67-83399abe9862","Type":"ContainerStarted","Data":"38b4dd0575c5a321fffb612937450ab85cd4a8b44579aaf674d971e1d69df39d"} Nov 03 01:30:24.867657 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:24.867586 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6fec8a-29894490-r4cj4" event={"ID":"6bb86426-5f37-4afc-9d67-83399abe9862","Type":"ContainerStarted","Data":"8f648f87e721607e4305a103b9586634a08fcb619a40e32248d857bac4bc788a"} Nov 03 01:30:24.869549 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:24.869531 4421 scope.go:122] "RemoveContainer" containerID="6bedbda9a52c131d9db6a80bae32ca47ed8242447d9fbf03ec983d4cf93faf25" Nov 03 01:30:24.869831 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:30:24.869808 4421 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-50e623-29894490-mgsqg_odh-ai-gateway-infra(10c1b824-d0d5-4b84-8961-c41c36242725)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-50e623-29894490-mgsqg" podUID="10c1b824-d0d5-4b84-8961-c41c36242725" Nov 03 01:30:24.871572 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:24.871484 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f7680d-29894490-6dkgx" event={"ID":"e9bfb106-b718-4909-a826-79040347a857","Type":"ContainerStarted","Data":"e1844d996f4ca4fec72a88bd2ee8c4798808e3dcb798fe91cad3889710eb953e"} Nov 03 01:30:24.871572 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:24.871533 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f7680d-29894490-6dkgx" event={"ID":"e9bfb106-b718-4909-a826-79040347a857","Type":"ContainerStarted","Data":"482cfd7e33be5de99e230341ed3b4038ec28ce07871c720c477f6403c8991acd"} Nov 03 01:30:24.875535 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:24.875502 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5dab6a-29894490-4pqxr" event={"ID":"917a9367-6e36-4f93-a9a6-b65bfd868704","Type":"ContainerStarted","Data":"667165127ecd1a7c3880a18e301ece0bc0180547d8e05d3b8dce9c037c3ee9e7"} Nov 03 01:30:24.876689 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:24.876650 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b4971a-29894490-f47mr" podStartSLOduration=24.876638538 podStartE2EDuration="24.876638538s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:30:24.871028651 +0000 UTC m=+2967.042031748" watchObservedRunningTime="2026-11-03 01:30:24.876638538 +0000 UTC m=+2967.047641633" Nov 03 01:30:24.906194 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:24.906146 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6c0ec0-29894490-5p6qm" podStartSLOduration=24.906129439 podStartE2EDuration="24.906129439s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:30:24.903553061 +0000 UTC m=+2967.074556157" watchObservedRunningTime="2026-11-03 01:30:24.906129439 +0000 UTC m=+2967.077132537" Nov 03 01:30:24.955956 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:24.955902 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894490-tqrss" podStartSLOduration=24.955888552 podStartE2EDuration="24.955888552s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:30:24.951890009 +0000 UTC m=+2967.122893105" watchObservedRunningTime="2026-11-03 01:30:24.955888552 +0000 UTC m=+2967.126891647" Nov 03 01:30:24.976790 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:24.976674 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6fec8a-29894490-r4cj4" podStartSLOduration=24.976655734 podStartE2EDuration="24.976655734s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:30:24.971609041 +0000 UTC m=+2967.142612150" watchObservedRunningTime="2026-11-03 01:30:24.976655734 +0000 UTC m=+2967.147658831" Nov 03 01:30:24.990588 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:24.990532 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-83755a-29894490-txc6m" podStartSLOduration=24.99051732 podStartE2EDuration="24.99051732s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:30:24.988857678 +0000 UTC m=+2967.159860774" watchObservedRunningTime="2026-11-03 01:30:24.99051732 +0000 UTC m=+2967.161520445" Nov 03 01:30:25.016971 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:25.016905 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f7680d-29894490-6dkgx" podStartSLOduration=25.016883932 podStartE2EDuration="25.016883932s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:30:25.010466621 +0000 UTC m=+2967.181469719" watchObservedRunningTime="2026-11-03 01:30:25.016883932 +0000 UTC m=+2967.187887031" Nov 03 01:30:25.032634 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:25.032579 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5dab6a-29894490-4pqxr" podStartSLOduration=25.032565008 podStartE2EDuration="25.032565008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:30:25.029639574 +0000 UTC m=+2967.200642671" watchObservedRunningTime="2026-11-03 01:30:25.032565008 +0000 UTC m=+2967.203568104" Nov 03 01:30:25.315064 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:30:25.315026 4421 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod917a9367_6e36_4f93_a9a6_b65bfd868704.slice/crio-conmon-667165127ecd1a7c3880a18e301ece0bc0180547d8e05d3b8dce9c037c3ee9e7.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58e1f190_d19c_43f7_afe2_e3079e8d9814.slice/crio-1824db924c0daceff23ade37f0d3bf294640a2ce9e172b1f25e179508bed8b8d.scope\": RecentStats: unable to find data in memory cache]" Nov 03 01:30:25.915735 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:25.914076 4421 generic.go:356] "Generic (PLEG): container finished" podID="58e1f190-d19c-43f7-afe2-e3079e8d9814" containerID="1824db924c0daceff23ade37f0d3bf294640a2ce9e172b1f25e179508bed8b8d" exitCode=60 Nov 03 01:30:25.915735 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:25.914211 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b4971a-29894490-f47mr" event={"ID":"58e1f190-d19c-43f7-afe2-e3079e8d9814","Type":"ContainerDied","Data":"1824db924c0daceff23ade37f0d3bf294640a2ce9e172b1f25e179508bed8b8d"} Nov 03 01:30:25.915735 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:25.914863 4421 scope.go:122] "RemoveContainer" containerID="1824db924c0daceff23ade37f0d3bf294640a2ce9e172b1f25e179508bed8b8d" Nov 03 01:30:25.918245 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:25.917971 4421 scope.go:122] "RemoveContainer" containerID="2c267a51fab51d3e378d58a4dc21d7ff7fb59faea972069d7f83bdffa21db65b" Nov 03 01:30:25.918245 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:25.918047 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6c0ec0-29894490-5p6qm" event={"ID":"645536c5-e930-48f6-bbf2-f4591ee8c368","Type":"ContainerDied","Data":"2c267a51fab51d3e378d58a4dc21d7ff7fb59faea972069d7f83bdffa21db65b"} Nov 03 01:30:25.919309 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:25.918466 4421 generic.go:356] "Generic (PLEG): container finished" podID="645536c5-e930-48f6-bbf2-f4591ee8c368" containerID="2c267a51fab51d3e378d58a4dc21d7ff7fb59faea972069d7f83bdffa21db65b" exitCode=60 Nov 03 01:30:25.932685 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:25.930145 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5dab6a-openshift-default-6ff495656c-bxn8s" event={"ID":"e0c12073-7ff8-413c-9940-978f4b600778","Type":"ContainerStarted","Data":"249bb0fe1738567e86a1979d8fcfbba2130e5107a49c6e70218a77527379e944"} Nov 03 01:30:25.942762 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:25.942732 4421 generic.go:356] "Generic (PLEG): container finished" podID="1579a23a-e983-4da2-b9be-427fe00d9de3" containerID="544091e808775d9fc2ad3edae2829003092456a7b142e6327ccdcf6bde7d396c" exitCode=60 Nov 03 01:30:25.943072 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:25.943046 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894490-tqrss" event={"ID":"1579a23a-e983-4da2-b9be-427fe00d9de3","Type":"ContainerDied","Data":"544091e808775d9fc2ad3edae2829003092456a7b142e6327ccdcf6bde7d396c"} Nov 03 01:30:25.943658 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:25.943638 4421 scope.go:122] "RemoveContainer" containerID="544091e808775d9fc2ad3edae2829003092456a7b142e6327ccdcf6bde7d396c" Nov 03 01:30:25.947778 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:25.947753 4421 generic.go:356] "Generic (PLEG): container finished" podID="91be6fd5-032c-4abe-aedf-e753fe1d7386" containerID="77884d74e096046edef89113a03345da193a9846973d3803cb894f9f6bdcb763" exitCode=60 Nov 03 01:30:25.948046 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:25.948023 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-83755a-29894490-txc6m" event={"ID":"91be6fd5-032c-4abe-aedf-e753fe1d7386","Type":"ContainerDied","Data":"77884d74e096046edef89113a03345da193a9846973d3803cb894f9f6bdcb763"} Nov 03 01:30:25.948832 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:25.948811 4421 scope.go:122] "RemoveContainer" containerID="77884d74e096046edef89113a03345da193a9846973d3803cb894f9f6bdcb763" Nov 03 01:30:25.963837 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:25.961914 4421 generic.go:356] "Generic (PLEG): container finished" podID="6bb86426-5f37-4afc-9d67-83399abe9862" containerID="38b4dd0575c5a321fffb612937450ab85cd4a8b44579aaf674d971e1d69df39d" exitCode=60 Nov 03 01:30:25.963837 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:25.962053 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6fec8a-29894490-r4cj4" event={"ID":"6bb86426-5f37-4afc-9d67-83399abe9862","Type":"ContainerDied","Data":"38b4dd0575c5a321fffb612937450ab85cd4a8b44579aaf674d971e1d69df39d"} Nov 03 01:30:25.963837 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:25.962482 4421 scope.go:122] "RemoveContainer" containerID="38b4dd0575c5a321fffb612937450ab85cd4a8b44579aaf674d971e1d69df39d" Nov 03 01:30:25.970390 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:25.970188 4421 generic.go:356] "Generic (PLEG): container finished" podID="e9bfb106-b718-4909-a826-79040347a857" containerID="e1844d996f4ca4fec72a88bd2ee8c4798808e3dcb798fe91cad3889710eb953e" exitCode=60 Nov 03 01:30:25.970390 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:25.970331 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f7680d-29894490-6dkgx" event={"ID":"e9bfb106-b718-4909-a826-79040347a857","Type":"ContainerDied","Data":"e1844d996f4ca4fec72a88bd2ee8c4798808e3dcb798fe91cad3889710eb953e"} Nov 03 01:30:25.970588 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:25.970567 4421 scope.go:122] "RemoveContainer" containerID="e1844d996f4ca4fec72a88bd2ee8c4798808e3dcb798fe91cad3889710eb953e" Nov 03 01:30:25.985308 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:25.984288 4421 generic.go:356] "Generic (PLEG): container finished" podID="917a9367-6e36-4f93-a9a6-b65bfd868704" containerID="667165127ecd1a7c3880a18e301ece0bc0180547d8e05d3b8dce9c037c3ee9e7" exitCode=7 Nov 03 01:30:25.985308 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:25.984848 4421 scope.go:122] "RemoveContainer" containerID="667165127ecd1a7c3880a18e301ece0bc0180547d8e05d3b8dce9c037c3ee9e7" Nov 03 01:30:26.005208 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:26.005173 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5dab6a-29894490-4pqxr" event={"ID":"917a9367-6e36-4f93-a9a6-b65bfd868704","Type":"ContainerDied","Data":"667165127ecd1a7c3880a18e301ece0bc0180547d8e05d3b8dce9c037c3ee9e7"} Nov 03 01:30:26.877634 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:26.877601 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-5dab6a-openshift-default-6ff495656c-bxn8s" Nov 03 01:30:26.877830 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:26.877805 4421 patch_prober.go:29] interesting pod/e2e-inf-b-5dab6a-openshift-default-6ff495656c-bxn8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.71:15021/healthz/ready\": dial tcp 10.134.1.71:15021: connect: connection refused" start-of-body= Nov 03 01:30:26.877881 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:26.877858 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5dab6a-openshift-default-6ff495656c-bxn8s" podUID="e0c12073-7ff8-413c-9940-978f4b600778" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.71:15021/healthz/ready\": dial tcp 10.134.1.71:15021: connect: connection refused" Nov 03 01:30:26.997081 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:26.997043 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-83755a-29894490-txc6m" event={"ID":"91be6fd5-032c-4abe-aedf-e753fe1d7386","Type":"ContainerStarted","Data":"6a79f8d5e28850288a64dfa04de0c19dc1bee68d3042788e9e37ad2d7b7dd5a0"} Nov 03 01:30:26.999359 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:26.999328 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6fec8a-29894490-r4cj4" event={"ID":"6bb86426-5f37-4afc-9d67-83399abe9862","Type":"ContainerStarted","Data":"6e23d727b4fc44eb70480422e0405a9335258953cc202d9ec9a9d0b16c5498a3"} Nov 03 01:30:27.001561 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:27.001535 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f7680d-29894490-6dkgx" event={"ID":"e9bfb106-b718-4909-a826-79040347a857","Type":"ContainerStarted","Data":"948aed91f12798bb2a2c7ce021967683f8d6fdc18685e32282263144f2ce8604"} Nov 03 01:30:27.003561 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:27.003538 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5dab6a-29894490-4pqxr" event={"ID":"917a9367-6e36-4f93-a9a6-b65bfd868704","Type":"ContainerStarted","Data":"d873316be6b2ffc9663f01932be02aaa3db3d00b6ea315d2258b981f6637858e"} Nov 03 01:30:27.018345 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:27.009810 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b4971a-29894490-f47mr" event={"ID":"58e1f190-d19c-43f7-afe2-e3079e8d9814","Type":"ContainerStarted","Data":"051e25efc59c5ce0dfd6c49b46314c37975afef02b23709765aca9ac7822cf48"} Nov 03 01:30:27.042027 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:27.040583 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6c0ec0-29894490-5p6qm" event={"ID":"645536c5-e930-48f6-bbf2-f4591ee8c368","Type":"ContainerStarted","Data":"72223f128fe1810cd4c12989eca04398e8e8422fc6bc317f4d3da0cc2bc7a51d"} Nov 03 01:30:27.043936 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:27.043899 4421 generic.go:356] "Generic (PLEG): container finished" podID="1b230406-0b7d-4d69-ac8e-b5e57164a5e4" containerID="dabb4c58b30b29a06225852c1866c0be2a87acfcb913b6cc5195754fda043318" exitCode=60 Nov 03 01:30:27.044083 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:27.043975 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-af8244-29894490-5dpb4" event={"ID":"1b230406-0b7d-4d69-ac8e-b5e57164a5e4","Type":"ContainerDied","Data":"dabb4c58b30b29a06225852c1866c0be2a87acfcb913b6cc5195754fda043318"} Nov 03 01:30:27.044083 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:27.044037 4421 scope.go:122] "RemoveContainer" containerID="07fc7743b11438c89a163a58785b8c9fb087c043b7b5128565b4f7ccb567e27c" Nov 03 01:30:27.044529 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:27.044506 4421 scope.go:122] "RemoveContainer" containerID="dabb4c58b30b29a06225852c1866c0be2a87acfcb913b6cc5195754fda043318" Nov 03 01:30:27.045198 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:30:27.045111 4421 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-af8244-29894490-5dpb4_odh-ai-gateway-infra(1b230406-0b7d-4d69-ac8e-b5e57164a5e4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-af8244-29894490-5dpb4" podUID="1b230406-0b7d-4d69-ac8e-b5e57164a5e4" Nov 03 01:30:27.048222 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:27.048193 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894490-tqrss" event={"ID":"1579a23a-e983-4da2-b9be-427fe00d9de3","Type":"ContainerStarted","Data":"a39cf58c07c12da48e8356f4daddb0603d967eaddbf9ae2fe626276e8012fc1a"} Nov 03 01:30:27.877185 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:27.877093 4421 patch_prober.go:29] interesting pod/e2e-inf-b-5dab6a-openshift-default-6ff495656c-bxn8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.71:15021/healthz/ready\": dial tcp 10.134.1.71:15021: connect: connection refused" start-of-body= Nov 03 01:30:27.877185 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:27.877156 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5dab6a-openshift-default-6ff495656c-bxn8s" podUID="e0c12073-7ff8-413c-9940-978f4b600778" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.71:15021/healthz/ready\": dial tcp 10.134.1.71:15021: connect: connection refused" Nov 03 01:30:28.078785 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:28.078751 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-84e20ed8-crbln"] Nov 03 01:30:28.085066 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:28.085037 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-84e20ed8-crbln"] Nov 03 01:30:28.086846 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:28.086827 4421 scope.go:122] "RemoveContainer" containerID="dabb4c58b30b29a06225852c1866c0be2a87acfcb913b6cc5195754fda043318" Nov 03 01:30:28.087387 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:30:28.087364 4421 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-af8244-29894490-5dpb4_odh-ai-gateway-infra(1b230406-0b7d-4d69-ac8e-b5e57164a5e4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-af8244-29894490-5dpb4" podUID="1b230406-0b7d-4d69-ac8e-b5e57164a5e4" Nov 03 01:30:28.088693 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:28.088671 4421 generic.go:356] "Generic (PLEG): container finished" podID="1579a23a-e983-4da2-b9be-427fe00d9de3" containerID="a39cf58c07c12da48e8356f4daddb0603d967eaddbf9ae2fe626276e8012fc1a" exitCode=60 Nov 03 01:30:28.088805 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:28.088791 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894490-tqrss" event={"ID":"1579a23a-e983-4da2-b9be-427fe00d9de3","Type":"ContainerDied","Data":"a39cf58c07c12da48e8356f4daddb0603d967eaddbf9ae2fe626276e8012fc1a"} Nov 03 01:30:28.088902 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:28.088872 4421 scope.go:122] "RemoveContainer" containerID="544091e808775d9fc2ad3edae2829003092456a7b142e6327ccdcf6bde7d396c" Nov 03 01:30:28.088980 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:28.088964 4421 scope.go:122] "RemoveContainer" containerID="a39cf58c07c12da48e8356f4daddb0603d967eaddbf9ae2fe626276e8012fc1a" Nov 03 01:30:28.089392 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:30:28.089370 4421 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894490-tqrss_odh-ai-gateway-infra(1579a23a-e983-4da2-b9be-427fe00d9de3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894490-tqrss" podUID="1579a23a-e983-4da2-b9be-427fe00d9de3" Nov 03 01:30:28.090981 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:28.090961 4421 generic.go:356] "Generic (PLEG): container finished" podID="91be6fd5-032c-4abe-aedf-e753fe1d7386" containerID="6a79f8d5e28850288a64dfa04de0c19dc1bee68d3042788e9e37ad2d7b7dd5a0" exitCode=60 Nov 03 01:30:28.091092 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:28.091076 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-83755a-29894490-txc6m" event={"ID":"91be6fd5-032c-4abe-aedf-e753fe1d7386","Type":"ContainerDied","Data":"6a79f8d5e28850288a64dfa04de0c19dc1bee68d3042788e9e37ad2d7b7dd5a0"} Nov 03 01:30:28.091413 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:28.091399 4421 scope.go:122] "RemoveContainer" containerID="6a79f8d5e28850288a64dfa04de0c19dc1bee68d3042788e9e37ad2d7b7dd5a0" Nov 03 01:30:28.091833 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:30:28.091811 4421 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-83755a-29894490-txc6m_odh-ai-gateway-infra(91be6fd5-032c-4abe-aedf-e753fe1d7386)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-83755a-29894490-txc6m" podUID="91be6fd5-032c-4abe-aedf-e753fe1d7386" Nov 03 01:30:28.093007 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:28.092985 4421 generic.go:356] "Generic (PLEG): container finished" podID="6bb86426-5f37-4afc-9d67-83399abe9862" containerID="6e23d727b4fc44eb70480422e0405a9335258953cc202d9ec9a9d0b16c5498a3" exitCode=60 Nov 03 01:30:28.093107 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:28.093011 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6fec8a-29894490-r4cj4" event={"ID":"6bb86426-5f37-4afc-9d67-83399abe9862","Type":"ContainerDied","Data":"6e23d727b4fc44eb70480422e0405a9335258953cc202d9ec9a9d0b16c5498a3"} Nov 03 01:30:28.093442 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:28.093427 4421 scope.go:122] "RemoveContainer" containerID="6e23d727b4fc44eb70480422e0405a9335258953cc202d9ec9a9d0b16c5498a3" Nov 03 01:30:28.093873 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:30:28.093855 4421 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-6fec8a-29894490-r4cj4_odh-ai-gateway-infra(6bb86426-5f37-4afc-9d67-83399abe9862)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6fec8a-29894490-r4cj4" podUID="6bb86426-5f37-4afc-9d67-83399abe9862" Nov 03 01:30:28.095057 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:28.095038 4421 generic.go:356] "Generic (PLEG): container finished" podID="e9bfb106-b718-4909-a826-79040347a857" containerID="948aed91f12798bb2a2c7ce021967683f8d6fdc18685e32282263144f2ce8604" exitCode=60 Nov 03 01:30:28.095253 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:28.095235 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f7680d-29894490-6dkgx" event={"ID":"e9bfb106-b718-4909-a826-79040347a857","Type":"ContainerDied","Data":"948aed91f12798bb2a2c7ce021967683f8d6fdc18685e32282263144f2ce8604"} Nov 03 01:30:28.095406 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:28.095389 4421 scope.go:122] "RemoveContainer" containerID="948aed91f12798bb2a2c7ce021967683f8d6fdc18685e32282263144f2ce8604" Nov 03 01:30:28.095692 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:30:28.095668 4421 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-f7680d-29894490-6dkgx_odh-ai-gateway-infra(e9bfb106-b718-4909-a826-79040347a857)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f7680d-29894490-6dkgx" podUID="e9bfb106-b718-4909-a826-79040347a857" Nov 03 01:30:28.097055 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:28.097035 4421 generic.go:356] "Generic (PLEG): container finished" podID="917a9367-6e36-4f93-a9a6-b65bfd868704" containerID="d873316be6b2ffc9663f01932be02aaa3db3d00b6ea315d2258b981f6637858e" exitCode=7 Nov 03 01:30:28.101424 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:28.097072 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5dab6a-29894490-4pqxr" event={"ID":"917a9367-6e36-4f93-a9a6-b65bfd868704","Type":"ContainerDied","Data":"d873316be6b2ffc9663f01932be02aaa3db3d00b6ea315d2258b981f6637858e"} Nov 03 01:30:28.101526 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:28.097307 4421 scope.go:122] "RemoveContainer" containerID="d873316be6b2ffc9663f01932be02aaa3db3d00b6ea315d2258b981f6637858e" Nov 03 01:30:28.101746 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:30:28.101727 4421 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-5dab6a-29894490-4pqxr_odh-ai-gateway-infra(917a9367-6e36-4f93-a9a6-b65bfd868704)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5dab6a-29894490-4pqxr" podUID="917a9367-6e36-4f93-a9a6-b65bfd868704" Nov 03 01:30:28.103385 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:28.103365 4421 generic.go:356] "Generic (PLEG): container finished" podID="58e1f190-d19c-43f7-afe2-e3079e8d9814" containerID="051e25efc59c5ce0dfd6c49b46314c37975afef02b23709765aca9ac7822cf48" exitCode=60 Nov 03 01:30:28.103525 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:28.103459 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b4971a-29894490-f47mr" event={"ID":"58e1f190-d19c-43f7-afe2-e3079e8d9814","Type":"ContainerDied","Data":"051e25efc59c5ce0dfd6c49b46314c37975afef02b23709765aca9ac7822cf48"} Nov 03 01:30:28.103849 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:28.103819 4421 scope.go:122] "RemoveContainer" containerID="051e25efc59c5ce0dfd6c49b46314c37975afef02b23709765aca9ac7822cf48" Nov 03 01:30:28.104195 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:30:28.104164 4421 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-b4971a-29894490-f47mr_odh-ai-gateway-infra(58e1f190-d19c-43f7-afe2-e3079e8d9814)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b4971a-29894490-f47mr" podUID="58e1f190-d19c-43f7-afe2-e3079e8d9814" Nov 03 01:30:28.106025 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:28.105979 4421 scope.go:122] "RemoveContainer" containerID="77884d74e096046edef89113a03345da193a9846973d3803cb894f9f6bdcb763" Nov 03 01:30:28.139153 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:28.139118 4421 generic.go:356] "Generic (PLEG): container finished" podID="645536c5-e930-48f6-bbf2-f4591ee8c368" containerID="72223f128fe1810cd4c12989eca04398e8e8422fc6bc317f4d3da0cc2bc7a51d" exitCode=60 Nov 03 01:30:28.139437 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:28.139418 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6c0ec0-29894490-5p6qm" event={"ID":"645536c5-e930-48f6-bbf2-f4591ee8c368","Type":"ContainerDied","Data":"72223f128fe1810cd4c12989eca04398e8e8422fc6bc317f4d3da0cc2bc7a51d"} Nov 03 01:30:28.140109 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:28.140090 4421 scope.go:122] "RemoveContainer" containerID="72223f128fe1810cd4c12989eca04398e8e8422fc6bc317f4d3da0cc2bc7a51d" Nov 03 01:30:28.140906 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:30:28.140881 4421 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-6c0ec0-29894490-5p6qm_odh-ai-gateway-infra(645536c5-e930-48f6-bbf2-f4591ee8c368)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6c0ec0-29894490-5p6qm" podUID="645536c5-e930-48f6-bbf2-f4591ee8c368" Nov 03 01:30:28.157189 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:28.157169 4421 scope.go:122] "RemoveContainer" containerID="38b4dd0575c5a321fffb612937450ab85cd4a8b44579aaf674d971e1d69df39d" Nov 03 01:30:28.174341 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:28.174316 4421 scope.go:122] "RemoveContainer" containerID="e1844d996f4ca4fec72a88bd2ee8c4798808e3dcb798fe91cad3889710eb953e" Nov 03 01:30:28.190228 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:28.190204 4421 scope.go:122] "RemoveContainer" containerID="667165127ecd1a7c3880a18e301ece0bc0180547d8e05d3b8dce9c037c3ee9e7" Nov 03 01:30:28.225683 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:28.225504 4421 scope.go:122] "RemoveContainer" containerID="1824db924c0daceff23ade37f0d3bf294640a2ce9e172b1f25e179508bed8b8d" Nov 03 01:30:28.250393 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:28.250366 4421 scope.go:122] "RemoveContainer" containerID="2c267a51fab51d3e378d58a4dc21d7ff7fb59faea972069d7f83bdffa21db65b" Nov 03 01:30:28.877594 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:28.877556 4421 patch_prober.go:29] interesting pod/e2e-inf-b-5dab6a-openshift-default-6ff495656c-bxn8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.71:15021/healthz/ready\": dial tcp 10.134.1.71:15021: connect: connection refused" start-of-body= Nov 03 01:30:28.877818 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:28.877611 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5dab6a-openshift-default-6ff495656c-bxn8s" podUID="e0c12073-7ff8-413c-9940-978f4b600778" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.71:15021/healthz/ready\": dial tcp 10.134.1.71:15021: connect: connection refused" Nov 03 01:30:29.151548 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:29.151462 4421 scope.go:122] "RemoveContainer" containerID="6a79f8d5e28850288a64dfa04de0c19dc1bee68d3042788e9e37ad2d7b7dd5a0" Nov 03 01:30:29.152017 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:30:29.151910 4421 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-83755a-29894490-txc6m_odh-ai-gateway-infra(91be6fd5-032c-4abe-aedf-e753fe1d7386)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-83755a-29894490-txc6m" podUID="91be6fd5-032c-4abe-aedf-e753fe1d7386" Nov 03 01:30:29.153306 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:29.153247 4421 scope.go:122] "RemoveContainer" containerID="6e23d727b4fc44eb70480422e0405a9335258953cc202d9ec9a9d0b16c5498a3" Nov 03 01:30:29.153747 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:30:29.153725 4421 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-6fec8a-29894490-r4cj4_odh-ai-gateway-infra(6bb86426-5f37-4afc-9d67-83399abe9862)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6fec8a-29894490-r4cj4" podUID="6bb86426-5f37-4afc-9d67-83399abe9862" Nov 03 01:30:29.155307 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:29.155268 4421 scope.go:122] "RemoveContainer" containerID="948aed91f12798bb2a2c7ce021967683f8d6fdc18685e32282263144f2ce8604" Nov 03 01:30:29.155664 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:30:29.155636 4421 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-f7680d-29894490-6dkgx_odh-ai-gateway-infra(e9bfb106-b718-4909-a826-79040347a857)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f7680d-29894490-6dkgx" podUID="e9bfb106-b718-4909-a826-79040347a857" Nov 03 01:30:29.157117 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:29.157092 4421 scope.go:122] "RemoveContainer" containerID="d873316be6b2ffc9663f01932be02aaa3db3d00b6ea315d2258b981f6637858e" Nov 03 01:30:29.157373 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:30:29.157354 4421 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-5dab6a-29894490-4pqxr_odh-ai-gateway-infra(917a9367-6e36-4f93-a9a6-b65bfd868704)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5dab6a-29894490-4pqxr" podUID="917a9367-6e36-4f93-a9a6-b65bfd868704" Nov 03 01:30:29.159156 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:29.159134 4421 scope.go:122] "RemoveContainer" containerID="051e25efc59c5ce0dfd6c49b46314c37975afef02b23709765aca9ac7822cf48" Nov 03 01:30:29.159489 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:30:29.159465 4421 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-b4971a-29894490-f47mr_odh-ai-gateway-infra(58e1f190-d19c-43f7-afe2-e3079e8d9814)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b4971a-29894490-f47mr" podUID="58e1f190-d19c-43f7-afe2-e3079e8d9814" Nov 03 01:30:29.161104 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:29.161084 4421 scope.go:122] "RemoveContainer" containerID="72223f128fe1810cd4c12989eca04398e8e8422fc6bc317f4d3da0cc2bc7a51d" Nov 03 01:30:29.161410 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:30:29.161392 4421 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-6c0ec0-29894490-5p6qm_odh-ai-gateway-infra(645536c5-e930-48f6-bbf2-f4591ee8c368)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6c0ec0-29894490-5p6qm" podUID="645536c5-e930-48f6-bbf2-f4591ee8c368" Nov 03 01:30:29.162867 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:29.162848 4421 scope.go:122] "RemoveContainer" containerID="a39cf58c07c12da48e8356f4daddb0603d967eaddbf9ae2fe626276e8012fc1a" Nov 03 01:30:29.163142 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:30:29.163119 4421 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894490-tqrss_odh-ai-gateway-infra(1579a23a-e983-4da2-b9be-427fe00d9de3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894490-tqrss" podUID="1579a23a-e983-4da2-b9be-427fe00d9de3" Nov 03 01:30:29.877716 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:29.877678 4421 patch_prober.go:29] interesting pod/e2e-inf-b-5dab6a-openshift-default-6ff495656c-bxn8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.71:15021/healthz/ready\": dial tcp 10.134.1.71:15021: connect: connection refused" start-of-body= Nov 03 01:30:29.877931 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:29.877737 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5dab6a-openshift-default-6ff495656c-bxn8s" podUID="e0c12073-7ff8-413c-9940-978f4b600778" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.71:15021/healthz/ready\": dial tcp 10.134.1.71:15021: connect: connection refused" Nov 03 01:30:29.983344 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:29.983300 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f15340f2-6e04-4537-96d0-6b10fab3a34f" path="/var/lib/kubelet/pods/f15340f2-6e04-4537-96d0-6b10fab3a34f/volumes" Nov 03 01:30:30.877154 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:30.877111 4421 patch_prober.go:29] interesting pod/e2e-inf-b-5dab6a-openshift-default-6ff495656c-bxn8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.71:15021/healthz/ready\": dial tcp 10.134.1.71:15021: connect: connection refused" start-of-body= Nov 03 01:30:30.877624 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:30.877170 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5dab6a-openshift-default-6ff495656c-bxn8s" podUID="e0c12073-7ff8-413c-9940-978f4b600778" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.71:15021/healthz/ready\": dial tcp 10.134.1.71:15021: connect: connection refused" Nov 03 01:30:31.877563 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:31.877524 4421 patch_prober.go:29] interesting pod/e2e-inf-b-5dab6a-openshift-default-6ff495656c-bxn8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.71:15021/healthz/ready\": dial tcp 10.134.1.71:15021: connect: connection refused" start-of-body= Nov 03 01:30:31.878004 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:31.877589 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5dab6a-openshift-default-6ff495656c-bxn8s" podUID="e0c12073-7ff8-413c-9940-978f4b600778" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.71:15021/healthz/ready\": dial tcp 10.134.1.71:15021: connect: connection refused" Nov 03 01:30:32.202091 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:32.202004 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-5dab6a-openshift-default-6ff495656c-bxn8s" Nov 03 01:30:32.905300 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:32.894579 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-5dab6a-openshift-default-6ff495656c-bxn8s" Nov 03 01:30:33.217383 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:33.217287 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-5dab6a-openshift-default-6ff495656c-bxn8s" Nov 03 01:30:36.971640 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:36.971603 4421 scope.go:122] "RemoveContainer" containerID="6bedbda9a52c131d9db6a80bae32ca47ed8242447d9fbf03ec983d4cf93faf25" Nov 03 01:30:37.980390 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:37.980360 4421 scope.go:122] "RemoveContainer" containerID="d873316be6b2ffc9663f01932be02aaa3db3d00b6ea315d2258b981f6637858e" Nov 03 01:30:37.980945 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:37.980596 4421 scope.go:122] "RemoveContainer" containerID="948aed91f12798bb2a2c7ce021967683f8d6fdc18685e32282263144f2ce8604" Nov 03 01:30:38.032615 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:38.032579 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-73646312-dbwcm"] Nov 03 01:30:38.038169 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:38.038135 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-73646312-dbwcm"] Nov 03 01:30:38.323467 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:38.323021 4421 generic.go:356] "Generic (PLEG): container finished" podID="10c1b824-d0d5-4b84-8961-c41c36242725" containerID="cdf95c471e1edadf0b4873272bb3c6f8816b278f51baa2baac44ec498af0a5c0" exitCode=60 Nov 03 01:30:38.323467 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:38.323095 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-50e623-29894490-mgsqg" event={"ID":"10c1b824-d0d5-4b84-8961-c41c36242725","Type":"ContainerDied","Data":"cdf95c471e1edadf0b4873272bb3c6f8816b278f51baa2baac44ec498af0a5c0"} Nov 03 01:30:38.323467 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:38.323164 4421 scope.go:122] "RemoveContainer" containerID="6bedbda9a52c131d9db6a80bae32ca47ed8242447d9fbf03ec983d4cf93faf25" Nov 03 01:30:38.323692 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:38.323626 4421 scope.go:122] "RemoveContainer" containerID="cdf95c471e1edadf0b4873272bb3c6f8816b278f51baa2baac44ec498af0a5c0" Nov 03 01:30:38.324034 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:30:38.324010 4421 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-50e623-29894490-mgsqg_odh-ai-gateway-infra(10c1b824-d0d5-4b84-8961-c41c36242725)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-50e623-29894490-mgsqg" podUID="10c1b824-d0d5-4b84-8961-c41c36242725" Nov 03 01:30:38.340291 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:30:38.340183 4421 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10c1b824_d0d5_4b84_8961_c41c36242725.slice/crio-cdf95c471e1edadf0b4873272bb3c6f8816b278f51baa2baac44ec498af0a5c0.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10c1b824_d0d5_4b84_8961_c41c36242725.slice/crio-conmon-cdf95c471e1edadf0b4873272bb3c6f8816b278f51baa2baac44ec498af0a5c0.scope\": RecentStats: unable to find data in memory cache]" Nov 03 01:30:38.971535 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:38.971443 4421 scope.go:122] "RemoveContainer" containerID="6a79f8d5e28850288a64dfa04de0c19dc1bee68d3042788e9e37ad2d7b7dd5a0" Nov 03 01:30:38.971718 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:38.971649 4421 scope.go:122] "RemoveContainer" containerID="72223f128fe1810cd4c12989eca04398e8e8422fc6bc317f4d3da0cc2bc7a51d" Nov 03 01:30:38.971828 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:38.971808 4421 scope.go:122] "RemoveContainer" containerID="a39cf58c07c12da48e8356f4daddb0603d967eaddbf9ae2fe626276e8012fc1a" Nov 03 01:30:39.338456 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:39.338427 4421 generic.go:356] "Generic (PLEG): container finished" podID="e9bfb106-b718-4909-a826-79040347a857" containerID="17fbb29e4ebaf4a07570b2c934d1d03fb883bd9ed4d11d42ec5b5cba48bc2ff6" exitCode=60 Nov 03 01:30:39.338881 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:39.338507 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f7680d-29894490-6dkgx" event={"ID":"e9bfb106-b718-4909-a826-79040347a857","Type":"ContainerDied","Data":"17fbb29e4ebaf4a07570b2c934d1d03fb883bd9ed4d11d42ec5b5cba48bc2ff6"} Nov 03 01:30:39.338881 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:39.338580 4421 scope.go:122] "RemoveContainer" containerID="948aed91f12798bb2a2c7ce021967683f8d6fdc18685e32282263144f2ce8604" Nov 03 01:30:39.339094 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:39.339073 4421 scope.go:122] "RemoveContainer" containerID="17fbb29e4ebaf4a07570b2c934d1d03fb883bd9ed4d11d42ec5b5cba48bc2ff6" Nov 03 01:30:39.339618 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:30:39.339594 4421 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-f7680d-29894490-6dkgx_odh-ai-gateway-infra(e9bfb106-b718-4909-a826-79040347a857)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f7680d-29894490-6dkgx" podUID="e9bfb106-b718-4909-a826-79040347a857" Nov 03 01:30:39.340989 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:39.340967 4421 generic.go:356] "Generic (PLEG): container finished" podID="917a9367-6e36-4f93-a9a6-b65bfd868704" containerID="9f769c068a368cb64dd5bf05787f98545ada69d46505a323b07f342fb8a194fd" exitCode=7 Nov 03 01:30:39.341085 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:39.341044 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5dab6a-29894490-4pqxr" event={"ID":"917a9367-6e36-4f93-a9a6-b65bfd868704","Type":"ContainerDied","Data":"9f769c068a368cb64dd5bf05787f98545ada69d46505a323b07f342fb8a194fd"} Nov 03 01:30:39.341572 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:39.341555 4421 scope.go:122] "RemoveContainer" containerID="9f769c068a368cb64dd5bf05787f98545ada69d46505a323b07f342fb8a194fd" Nov 03 01:30:39.342001 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:30:39.341978 4421 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-5dab6a-29894490-4pqxr_odh-ai-gateway-infra(917a9367-6e36-4f93-a9a6-b65bfd868704)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5dab6a-29894490-4pqxr" podUID="917a9367-6e36-4f93-a9a6-b65bfd868704" Nov 03 01:30:39.399201 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:39.399166 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-50e623-29894490-mgsqg"] Nov 03 01:30:39.418421 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:39.418397 4421 scope.go:122] "RemoveContainer" containerID="d873316be6b2ffc9663f01932be02aaa3db3d00b6ea315d2258b981f6637858e" Nov 03 01:30:39.607780 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:39.607746 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-50e623-29894490-mgsqg" Nov 03 01:30:39.697310 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:39.697243 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10c1b824-d0d5-4b84-8961-c41c36242725-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/10c1b824-d0d5-4b84-8961-c41c36242725-maas-api-service-ca\") pod \"10c1b824-d0d5-4b84-8961-c41c36242725\" (UID: \"10c1b824-d0d5-4b84-8961-c41c36242725\") " Nov 03 01:30:39.699062 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:39.697440 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10c1b824-d0d5-4b84-8961-c41c36242725-kube-api-access-4jfzh\" (UniqueName: \"kubernetes.io/projected/10c1b824-d0d5-4b84-8961-c41c36242725-kube-api-access-4jfzh\") pod \"10c1b824-d0d5-4b84-8961-c41c36242725\" (UID: \"10c1b824-d0d5-4b84-8961-c41c36242725\") " Nov 03 01:30:39.699062 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:39.697819 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10c1b824-d0d5-4b84-8961-c41c36242725-maas-api-service-ca" pod "10c1b824-d0d5-4b84-8961-c41c36242725" (UID: "10c1b824-d0d5-4b84-8961-c41c36242725"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:30:39.699062 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:39.698374 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/10c1b824-d0d5-4b84-8961-c41c36242725-maas-api-service-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:30:39.700743 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:39.700717 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10c1b824-d0d5-4b84-8961-c41c36242725-kube-api-access-4jfzh" pod "10c1b824-d0d5-4b84-8961-c41c36242725" (UID: "10c1b824-d0d5-4b84-8961-c41c36242725"). InnerVolumeSpecName "kube-api-access-4jfzh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:30:39.801116 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:39.801030 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jfzh\" (UniqueName: \"kubernetes.io/projected/10c1b824-d0d5-4b84-8961-c41c36242725-kube-api-access-4jfzh\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:30:39.980185 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:39.980144 4421 scope.go:122] "RemoveContainer" containerID="dabb4c58b30b29a06225852c1866c0be2a87acfcb913b6cc5195754fda043318" Nov 03 01:30:40.028338 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:40.028266 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="587b5308-6d39-4f44-9cb1-28da425a481e" path="/var/lib/kubelet/pods/587b5308-6d39-4f44-9cb1-28da425a481e/volumes" Nov 03 01:30:40.356592 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:40.356502 4421 generic.go:356] "Generic (PLEG): container finished" podID="645536c5-e930-48f6-bbf2-f4591ee8c368" containerID="06396ff9d32b8d9239968e0d95a686e7b0f19fc374e053bb510f0e7c907ac69f" exitCode=60 Nov 03 01:30:40.356592 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:40.356574 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6c0ec0-29894490-5p6qm" event={"ID":"645536c5-e930-48f6-bbf2-f4591ee8c368","Type":"ContainerDied","Data":"06396ff9d32b8d9239968e0d95a686e7b0f19fc374e053bb510f0e7c907ac69f"} Nov 03 01:30:40.357122 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:40.356641 4421 scope.go:122] "RemoveContainer" containerID="72223f128fe1810cd4c12989eca04398e8e8422fc6bc317f4d3da0cc2bc7a51d" Nov 03 01:30:40.357180 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:40.357160 4421 scope.go:122] "RemoveContainer" containerID="06396ff9d32b8d9239968e0d95a686e7b0f19fc374e053bb510f0e7c907ac69f" Nov 03 01:30:40.357791 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:30:40.357670 4421 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-6c0ec0-29894490-5p6qm_odh-ai-gateway-infra(645536c5-e930-48f6-bbf2-f4591ee8c368)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6c0ec0-29894490-5p6qm" podUID="645536c5-e930-48f6-bbf2-f4591ee8c368" Nov 03 01:30:40.359009 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:40.358983 4421 generic.go:356] "Generic (PLEG): container finished" podID="1579a23a-e983-4da2-b9be-427fe00d9de3" containerID="4f7310994ee9acb40e799a703243b6963d66b926d57df513d736c9e7cf3fe476" exitCode=60 Nov 03 01:30:40.359124 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:40.359053 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894490-tqrss" event={"ID":"1579a23a-e983-4da2-b9be-427fe00d9de3","Type":"ContainerDied","Data":"4f7310994ee9acb40e799a703243b6963d66b926d57df513d736c9e7cf3fe476"} Nov 03 01:30:40.359624 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:40.359601 4421 scope.go:122] "RemoveContainer" containerID="4f7310994ee9acb40e799a703243b6963d66b926d57df513d736c9e7cf3fe476" Nov 03 01:30:40.360062 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:30:40.360035 4421 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29894490-tqrss_odh-ai-gateway-infra(1579a23a-e983-4da2-b9be-427fe00d9de3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894490-tqrss" podUID="1579a23a-e983-4da2-b9be-427fe00d9de3" Nov 03 01:30:40.362261 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:40.362237 4421 generic.go:356] "Generic (PLEG): container finished" podID="91be6fd5-032c-4abe-aedf-e753fe1d7386" containerID="9a90275e55ee5c7e5bb0de4d9ea4d2f37db0bb9994c79f7aec682c461a3645ff" exitCode=60 Nov 03 01:30:40.362409 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:40.362383 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-83755a-29894490-txc6m" event={"ID":"91be6fd5-032c-4abe-aedf-e753fe1d7386","Type":"ContainerDied","Data":"9a90275e55ee5c7e5bb0de4d9ea4d2f37db0bb9994c79f7aec682c461a3645ff"} Nov 03 01:30:40.362864 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:40.362847 4421 scope.go:122] "RemoveContainer" containerID="9a90275e55ee5c7e5bb0de4d9ea4d2f37db0bb9994c79f7aec682c461a3645ff" Nov 03 01:30:40.363248 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:30:40.363226 4421 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-83755a-29894490-txc6m_odh-ai-gateway-infra(91be6fd5-032c-4abe-aedf-e753fe1d7386)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-83755a-29894490-txc6m" podUID="91be6fd5-032c-4abe-aedf-e753fe1d7386" Nov 03 01:30:40.364669 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:40.364651 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-50e623-29894490-mgsqg" event={"ID":"10c1b824-d0d5-4b84-8961-c41c36242725","Type":"ContainerDied","Data":"5956c8fe1645f8e60dd1c0846c700acce2fbeefee32a600070e2c5ccae0d60ea"} Nov 03 01:30:40.364799 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:40.364680 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-50e623-29894490-mgsqg" Nov 03 01:30:40.369269 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:40.369245 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f7680d-29894490-6dkgx"] Nov 03 01:30:40.388441 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:40.388360 4421 scope.go:122] "RemoveContainer" containerID="a39cf58c07c12da48e8356f4daddb0603d967eaddbf9ae2fe626276e8012fc1a" Nov 03 01:30:40.392311 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:40.392250 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5dab6a-29894490-4pqxr"] Nov 03 01:30:40.411628 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:40.411563 4421 scope.go:122] "RemoveContainer" containerID="6a79f8d5e28850288a64dfa04de0c19dc1bee68d3042788e9e37ad2d7b7dd5a0" Nov 03 01:30:40.432432 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:40.432319 4421 scope.go:122] "RemoveContainer" containerID="cdf95c471e1edadf0b4873272bb3c6f8816b278f51baa2baac44ec498af0a5c0" Nov 03 01:30:40.490341 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:40.488048 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-50e623-29894490-mgsqg"] Nov 03 01:30:40.502755 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:40.502410 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-50e623-29894490-mgsqg"] Nov 03 01:30:40.575352 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:40.575323 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f7680d-29894490-6dkgx" Nov 03 01:30:40.633552 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:40.633517 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9bfb106-b718-4909-a826-79040347a857-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9bfb106-b718-4909-a826-79040347a857-maas-api-service-ca\") pod \"e9bfb106-b718-4909-a826-79040347a857\" (UID: \"e9bfb106-b718-4909-a826-79040347a857\") " Nov 03 01:30:40.633850 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:40.633831 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9bfb106-b718-4909-a826-79040347a857-kube-api-access-gmzsd\" (UniqueName: \"kubernetes.io/projected/e9bfb106-b718-4909-a826-79040347a857-kube-api-access-gmzsd\") pod \"e9bfb106-b718-4909-a826-79040347a857\" (UID: \"e9bfb106-b718-4909-a826-79040347a857\") " Nov 03 01:30:40.633947 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:40.633919 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9bfb106-b718-4909-a826-79040347a857-maas-api-service-ca" pod "e9bfb106-b718-4909-a826-79040347a857" (UID: "e9bfb106-b718-4909-a826-79040347a857"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:30:40.634805 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:40.634778 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9bfb106-b718-4909-a826-79040347a857-maas-api-service-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:30:40.636063 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:40.636037 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9bfb106-b718-4909-a826-79040347a857-kube-api-access-gmzsd" pod "e9bfb106-b718-4909-a826-79040347a857" (UID: "e9bfb106-b718-4909-a826-79040347a857"). InnerVolumeSpecName "kube-api-access-gmzsd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:30:40.641088 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:40.641062 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5dab6a-29894490-4pqxr" Nov 03 01:30:40.742324 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:40.741399 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/917a9367-6e36-4f93-a9a6-b65bfd868704-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/917a9367-6e36-4f93-a9a6-b65bfd868704-maas-api-service-ca\") pod \"917a9367-6e36-4f93-a9a6-b65bfd868704\" (UID: \"917a9367-6e36-4f93-a9a6-b65bfd868704\") " Nov 03 01:30:40.742324 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:40.741625 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/917a9367-6e36-4f93-a9a6-b65bfd868704-kube-api-access-fj5g9\" (UniqueName: \"kubernetes.io/projected/917a9367-6e36-4f93-a9a6-b65bfd868704-kube-api-access-fj5g9\") pod \"917a9367-6e36-4f93-a9a6-b65bfd868704\" (UID: \"917a9367-6e36-4f93-a9a6-b65bfd868704\") " Nov 03 01:30:40.776337 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:40.774020 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmzsd\" (UniqueName: \"kubernetes.io/projected/e9bfb106-b718-4909-a826-79040347a857-kube-api-access-gmzsd\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:30:40.776337 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:40.775413 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/917a9367-6e36-4f93-a9a6-b65bfd868704-maas-api-service-ca" pod "917a9367-6e36-4f93-a9a6-b65bfd868704" (UID: "917a9367-6e36-4f93-a9a6-b65bfd868704"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:30:40.776337 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:40.776096 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/917a9367-6e36-4f93-a9a6-b65bfd868704-kube-api-access-fj5g9" pod "917a9367-6e36-4f93-a9a6-b65bfd868704" (UID: "917a9367-6e36-4f93-a9a6-b65bfd868704"). InnerVolumeSpecName "kube-api-access-fj5g9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:30:40.876567 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:40.876513 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/917a9367-6e36-4f93-a9a6-b65bfd868704-maas-api-service-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:30:40.876567 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:40.876570 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fj5g9\" (UniqueName: \"kubernetes.io/projected/917a9367-6e36-4f93-a9a6-b65bfd868704-kube-api-access-fj5g9\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:30:40.971950 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:40.971863 4421 scope.go:122] "RemoveContainer" containerID="051e25efc59c5ce0dfd6c49b46314c37975afef02b23709765aca9ac7822cf48" Nov 03 01:30:41.377112 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:41.377077 4421 generic.go:356] "Generic (PLEG): container finished" podID="1b230406-0b7d-4d69-ac8e-b5e57164a5e4" containerID="6878591712a8ab8fa23267a6cc7f107937e5f29416d647e7a91b9bbb16486479" exitCode=60 Nov 03 01:30:41.377618 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:41.377136 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-af8244-29894490-5dpb4" event={"ID":"1b230406-0b7d-4d69-ac8e-b5e57164a5e4","Type":"ContainerDied","Data":"6878591712a8ab8fa23267a6cc7f107937e5f29416d647e7a91b9bbb16486479"} Nov 03 01:30:41.377618 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:41.377200 4421 scope.go:122] "RemoveContainer" containerID="dabb4c58b30b29a06225852c1866c0be2a87acfcb913b6cc5195754fda043318" Nov 03 01:30:41.377778 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:41.377760 4421 scope.go:122] "RemoveContainer" containerID="6878591712a8ab8fa23267a6cc7f107937e5f29416d647e7a91b9bbb16486479" Nov 03 01:30:41.378266 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:30:41.378237 4421 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-af8244-29894490-5dpb4_odh-ai-gateway-infra(1b230406-0b7d-4d69-ac8e-b5e57164a5e4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-af8244-29894490-5dpb4" podUID="1b230406-0b7d-4d69-ac8e-b5e57164a5e4" Nov 03 01:30:41.385029 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:41.384990 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f7680d-29894490-6dkgx" event={"ID":"e9bfb106-b718-4909-a826-79040347a857","Type":"ContainerDied","Data":"482cfd7e33be5de99e230341ed3b4038ec28ce07871c720c477f6403c8991acd"} Nov 03 01:30:41.385136 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:41.385081 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f7680d-29894490-6dkgx" Nov 03 01:30:41.389076 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:41.389054 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6c0ec0-29894490-5p6qm"] Nov 03 01:30:41.394688 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:41.394668 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5dab6a-29894490-4pqxr" Nov 03 01:30:41.394784 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:41.394682 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5dab6a-29894490-4pqxr" event={"ID":"917a9367-6e36-4f93-a9a6-b65bfd868704","Type":"ContainerDied","Data":"e41eb92e53a4eade4ffe7da6c9e47302b2da3fa3e1767e9715189d927a041970"} Nov 03 01:30:41.397369 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:41.397351 4421 scope.go:122] "RemoveContainer" containerID="17fbb29e4ebaf4a07570b2c934d1d03fb883bd9ed4d11d42ec5b5cba48bc2ff6" Nov 03 01:30:41.412684 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:41.412638 4421 scope.go:122] "RemoveContainer" containerID="9f769c068a368cb64dd5bf05787f98545ada69d46505a323b07f342fb8a194fd" Nov 03 01:30:41.449858 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:41.449826 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894490-tqrss"] Nov 03 01:30:41.459461 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:41.459429 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5dab6a-29894490-4pqxr"] Nov 03 01:30:41.488768 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:41.488173 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5dab6a-29894490-4pqxr"] Nov 03 01:30:41.504758 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:41.503525 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-83755a-29894490-txc6m"] Nov 03 01:30:41.692295 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:41.673777 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f7680d-29894490-6dkgx"] Nov 03 01:30:41.734701 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:41.732389 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f7680d-29894490-6dkgx"] Nov 03 01:30:41.885825 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:41.885796 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6c0ec0-29894490-5p6qm" Nov 03 01:30:41.972421 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:41.972387 4421 scope.go:122] "RemoveContainer" containerID="6e23d727b4fc44eb70480422e0405a9335258953cc202d9ec9a9d0b16c5498a3" Nov 03 01:30:41.972962 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:41.972938 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894490-tqrss" Nov 03 01:30:42.005533 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:42.005495 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10c1b824-d0d5-4b84-8961-c41c36242725" path="/var/lib/kubelet/pods/10c1b824-d0d5-4b84-8961-c41c36242725/volumes" Nov 03 01:30:42.006594 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:42.006564 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="917a9367-6e36-4f93-a9a6-b65bfd868704" path="/var/lib/kubelet/pods/917a9367-6e36-4f93-a9a6-b65bfd868704/volumes" Nov 03 01:30:42.007231 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:42.007207 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9bfb106-b718-4909-a826-79040347a857" path="/var/lib/kubelet/pods/e9bfb106-b718-4909-a826-79040347a857/volumes" Nov 03 01:30:42.059995 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:42.059962 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/645536c5-e930-48f6-bbf2-f4591ee8c368-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/645536c5-e930-48f6-bbf2-f4591ee8c368-maas-api-service-ca\") pod \"645536c5-e930-48f6-bbf2-f4591ee8c368\" (UID: \"645536c5-e930-48f6-bbf2-f4591ee8c368\") " Nov 03 01:30:42.060196 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:42.060015 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/645536c5-e930-48f6-bbf2-f4591ee8c368-kube-api-access-w4jtp\" (UniqueName: \"kubernetes.io/projected/645536c5-e930-48f6-bbf2-f4591ee8c368-kube-api-access-w4jtp\") pod \"645536c5-e930-48f6-bbf2-f4591ee8c368\" (UID: \"645536c5-e930-48f6-bbf2-f4591ee8c368\") " Nov 03 01:30:42.061054 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:42.060404 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/645536c5-e930-48f6-bbf2-f4591ee8c368-maas-api-service-ca" pod "645536c5-e930-48f6-bbf2-f4591ee8c368" (UID: "645536c5-e930-48f6-bbf2-f4591ee8c368"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:30:42.061054 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:42.060890 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/645536c5-e930-48f6-bbf2-f4591ee8c368-maas-api-service-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:30:42.062326 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:42.062299 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/645536c5-e930-48f6-bbf2-f4591ee8c368-kube-api-access-w4jtp" pod "645536c5-e930-48f6-bbf2-f4591ee8c368" (UID: "645536c5-e930-48f6-bbf2-f4591ee8c368"). InnerVolumeSpecName "kube-api-access-w4jtp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:30:42.101656 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:42.101627 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-83755a-29894490-txc6m" Nov 03 01:30:42.162196 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:42.162164 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1579a23a-e983-4da2-b9be-427fe00d9de3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1579a23a-e983-4da2-b9be-427fe00d9de3-maas-api-service-ca\") pod \"1579a23a-e983-4da2-b9be-427fe00d9de3\" (UID: \"1579a23a-e983-4da2-b9be-427fe00d9de3\") " Nov 03 01:30:42.162396 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:42.162334 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1579a23a-e983-4da2-b9be-427fe00d9de3-kube-api-access-xfzfl\" (UniqueName: \"kubernetes.io/projected/1579a23a-e983-4da2-b9be-427fe00d9de3-kube-api-access-xfzfl\") pod \"1579a23a-e983-4da2-b9be-427fe00d9de3\" (UID: \"1579a23a-e983-4da2-b9be-427fe00d9de3\") " Nov 03 01:30:42.162645 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:42.162621 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1579a23a-e983-4da2-b9be-427fe00d9de3-maas-api-service-ca" pod "1579a23a-e983-4da2-b9be-427fe00d9de3" (UID: "1579a23a-e983-4da2-b9be-427fe00d9de3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:30:42.163389 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:42.163361 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4jtp\" (UniqueName: \"kubernetes.io/projected/645536c5-e930-48f6-bbf2-f4591ee8c368-kube-api-access-w4jtp\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:30:42.163496 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:42.163394 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1579a23a-e983-4da2-b9be-427fe00d9de3-maas-api-service-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:30:42.165033 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:42.165007 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1579a23a-e983-4da2-b9be-427fe00d9de3-kube-api-access-xfzfl" pod "1579a23a-e983-4da2-b9be-427fe00d9de3" (UID: "1579a23a-e983-4da2-b9be-427fe00d9de3"). InnerVolumeSpecName "kube-api-access-xfzfl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:30:42.264895 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:42.264866 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91be6fd5-032c-4abe-aedf-e753fe1d7386-kube-api-access-6v8l4\" (UniqueName: \"kubernetes.io/projected/91be6fd5-032c-4abe-aedf-e753fe1d7386-kube-api-access-6v8l4\") pod \"91be6fd5-032c-4abe-aedf-e753fe1d7386\" (UID: \"91be6fd5-032c-4abe-aedf-e753fe1d7386\") " Nov 03 01:30:42.265089 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:42.264923 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91be6fd5-032c-4abe-aedf-e753fe1d7386-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91be6fd5-032c-4abe-aedf-e753fe1d7386-maas-api-service-ca\") pod \"91be6fd5-032c-4abe-aedf-e753fe1d7386\" (UID: \"91be6fd5-032c-4abe-aedf-e753fe1d7386\") " Nov 03 01:30:42.265425 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:42.265395 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91be6fd5-032c-4abe-aedf-e753fe1d7386-maas-api-service-ca" pod "91be6fd5-032c-4abe-aedf-e753fe1d7386" (UID: "91be6fd5-032c-4abe-aedf-e753fe1d7386"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:30:42.265708 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:42.265694 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91be6fd5-032c-4abe-aedf-e753fe1d7386-maas-api-service-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:30:42.265764 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:42.265711 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xfzfl\" (UniqueName: \"kubernetes.io/projected/1579a23a-e983-4da2-b9be-427fe00d9de3-kube-api-access-xfzfl\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:30:42.267269 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:42.267239 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91be6fd5-032c-4abe-aedf-e753fe1d7386-kube-api-access-6v8l4" pod "91be6fd5-032c-4abe-aedf-e753fe1d7386" (UID: "91be6fd5-032c-4abe-aedf-e753fe1d7386"). InnerVolumeSpecName "kube-api-access-6v8l4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:30:42.367933 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:42.367899 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6v8l4\" (UniqueName: \"kubernetes.io/projected/91be6fd5-032c-4abe-aedf-e753fe1d7386-kube-api-access-6v8l4\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:30:42.481352 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:42.473623 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b4971a-29894490-f47mr" event={"ID":"58e1f190-d19c-43f7-afe2-e3079e8d9814","Type":"ContainerStarted","Data":"11cf6c12b18f0447b28523d7254b0dbd2c9069c620a41eb614bef3d0775941f5"} Nov 03 01:30:42.481352 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:42.476858 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6c0ec0-29894490-5p6qm" event={"ID":"645536c5-e930-48f6-bbf2-f4591ee8c368","Type":"ContainerDied","Data":"225f8191f984a56b6389838cc8d7b0a502d93e0912088f1146edb524f6f74cb7"} Nov 03 01:30:42.481352 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:42.476924 4421 scope.go:122] "RemoveContainer" containerID="06396ff9d32b8d9239968e0d95a686e7b0f19fc374e053bb510f0e7c907ac69f" Nov 03 01:30:42.481352 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:42.477036 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6c0ec0-29894490-5p6qm" Nov 03 01:30:42.491043 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:42.491012 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-af8244-29894490-5dpb4"] Nov 03 01:30:42.501882 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:42.500308 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894490-tqrss" event={"ID":"1579a23a-e983-4da2-b9be-427fe00d9de3","Type":"ContainerDied","Data":"5df1dfe9c6f53724b2b897fb900f750529ed36edd1a24749a941488dc6382dc4"} Nov 03 01:30:42.501882 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:42.500419 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894490-tqrss" Nov 03 01:30:42.503905 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:42.503880 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-83755a-29894490-txc6m" event={"ID":"91be6fd5-032c-4abe-aedf-e753fe1d7386","Type":"ContainerDied","Data":"e8a7be46d79042cb048e237851e0ee1d0a1243d7352d29e83af44a615f9f5714"} Nov 03 01:30:42.504029 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:42.503979 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-83755a-29894490-txc6m" Nov 03 01:30:42.514098 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:42.514077 4421 scope.go:122] "RemoveContainer" containerID="4f7310994ee9acb40e799a703243b6963d66b926d57df513d736c9e7cf3fe476" Nov 03 01:30:42.528143 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:42.528121 4421 scope.go:122] "RemoveContainer" containerID="9a90275e55ee5c7e5bb0de4d9ea4d2f37db0bb9994c79f7aec682c461a3645ff" Nov 03 01:30:42.543479 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:42.543453 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6c0ec0-29894490-5p6qm"] Nov 03 01:30:42.550602 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:42.550576 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6c0ec0-29894490-5p6qm"] Nov 03 01:30:42.562218 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:42.562191 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894490-tqrss"] Nov 03 01:30:42.567316 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:42.567258 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894490-tqrss"] Nov 03 01:30:42.582647 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:42.582619 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-83755a-29894490-txc6m"] Nov 03 01:30:42.610351 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:42.610308 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-83755a-29894490-txc6m"] Nov 03 01:30:42.673028 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:42.672998 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-af8244-29894490-5dpb4" Nov 03 01:30:42.776144 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:42.776113 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b230406-0b7d-4d69-ac8e-b5e57164a5e4-kube-api-access-hz872\" (UniqueName: \"kubernetes.io/projected/1b230406-0b7d-4d69-ac8e-b5e57164a5e4-kube-api-access-hz872\") pod \"1b230406-0b7d-4d69-ac8e-b5e57164a5e4\" (UID: \"1b230406-0b7d-4d69-ac8e-b5e57164a5e4\") " Nov 03 01:30:42.776361 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:42.776250 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b230406-0b7d-4d69-ac8e-b5e57164a5e4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b230406-0b7d-4d69-ac8e-b5e57164a5e4-maas-api-service-ca\") pod \"1b230406-0b7d-4d69-ac8e-b5e57164a5e4\" (UID: \"1b230406-0b7d-4d69-ac8e-b5e57164a5e4\") " Nov 03 01:30:42.776613 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:42.776589 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b230406-0b7d-4d69-ac8e-b5e57164a5e4-maas-api-service-ca" pod "1b230406-0b7d-4d69-ac8e-b5e57164a5e4" (UID: "1b230406-0b7d-4d69-ac8e-b5e57164a5e4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:30:42.776826 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:42.776808 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b230406-0b7d-4d69-ac8e-b5e57164a5e4-maas-api-service-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:30:42.778481 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:42.778459 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b230406-0b7d-4d69-ac8e-b5e57164a5e4-kube-api-access-hz872" pod "1b230406-0b7d-4d69-ac8e-b5e57164a5e4" (UID: "1b230406-0b7d-4d69-ac8e-b5e57164a5e4"). InnerVolumeSpecName "kube-api-access-hz872". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:30:42.878414 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:42.878375 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hz872\" (UniqueName: \"kubernetes.io/projected/1b230406-0b7d-4d69-ac8e-b5e57164a5e4-kube-api-access-hz872\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:30:43.510392 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:43.510358 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b4971a-29894490-f47mr"] Nov 03 01:30:43.526020 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:43.525991 4421 generic.go:356] "Generic (PLEG): container finished" podID="58e1f190-d19c-43f7-afe2-e3079e8d9814" containerID="11cf6c12b18f0447b28523d7254b0dbd2c9069c620a41eb614bef3d0775941f5" exitCode=60 Nov 03 01:30:43.526185 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:43.526081 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b4971a-29894490-f47mr" event={"ID":"58e1f190-d19c-43f7-afe2-e3079e8d9814","Type":"ContainerDied","Data":"11cf6c12b18f0447b28523d7254b0dbd2c9069c620a41eb614bef3d0775941f5"} Nov 03 01:30:43.526185 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:43.526124 4421 scope.go:122] "RemoveContainer" containerID="051e25efc59c5ce0dfd6c49b46314c37975afef02b23709765aca9ac7822cf48" Nov 03 01:30:43.529894 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:43.529867 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-af8244-29894490-5dpb4" Nov 03 01:30:43.530015 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:43.529907 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-af8244-29894490-5dpb4" event={"ID":"1b230406-0b7d-4d69-ac8e-b5e57164a5e4","Type":"ContainerDied","Data":"068ee3e7952fc7e4c4c29466a5ebe1e39976600347bb1ab24e8a53be1f658548"} Nov 03 01:30:43.534817 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:43.534792 4421 generic.go:356] "Generic (PLEG): container finished" podID="6bb86426-5f37-4afc-9d67-83399abe9862" containerID="69f2f6da1191fe75fe7cbcee8d8a7df17983135334c71d41a0b267ff0d131583" exitCode=60 Nov 03 01:30:43.534927 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:43.534868 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6fec8a-29894490-r4cj4" event={"ID":"6bb86426-5f37-4afc-9d67-83399abe9862","Type":"ContainerDied","Data":"69f2f6da1191fe75fe7cbcee8d8a7df17983135334c71d41a0b267ff0d131583"} Nov 03 01:30:43.535435 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:43.535413 4421 scope.go:122] "RemoveContainer" containerID="69f2f6da1191fe75fe7cbcee8d8a7df17983135334c71d41a0b267ff0d131583" Nov 03 01:30:43.535863 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:30:43.535838 4421 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-6fec8a-29894490-r4cj4_odh-ai-gateway-infra(6bb86426-5f37-4afc-9d67-83399abe9862)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6fec8a-29894490-r4cj4" podUID="6bb86426-5f37-4afc-9d67-83399abe9862" Nov 03 01:30:43.543737 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:43.543709 4421 scope.go:122] "RemoveContainer" containerID="6878591712a8ab8fa23267a6cc7f107937e5f29416d647e7a91b9bbb16486479" Nov 03 01:30:43.559722 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:43.559699 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-af8244-29894490-5dpb4"] Nov 03 01:30:43.562553 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:43.562516 4421 scope.go:122] "RemoveContainer" containerID="6e23d727b4fc44eb70480422e0405a9335258953cc202d9ec9a9d0b16c5498a3" Nov 03 01:30:43.568427 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:43.568399 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-af8244-29894490-5dpb4"] Nov 03 01:30:43.729486 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:43.729461 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b4971a-29894490-f47mr" Nov 03 01:30:43.758386 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:43.758351 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-4tt4f"] Nov 03 01:30:43.759368 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:43.759346 4421 cpu_manager.go:414] "containerMap: removing container" podUID="10c1b824-d0d5-4b84-8961-c41c36242725" containerName="cleanup" Nov 03 01:30:43.759368 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:43.759368 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10c1b824-d0d5-4b84-8961-c41c36242725" containerName="cleanup" Nov 03 01:30:43.759505 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:43.759386 4421 cpu_manager.go:414] "containerMap: removing container" podUID="91be6fd5-032c-4abe-aedf-e753fe1d7386" containerName="cleanup" Nov 03 01:30:43.759505 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:43.759392 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91be6fd5-032c-4abe-aedf-e753fe1d7386" containerName="cleanup" Nov 03 01:30:43.759505 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:43.759405 4421 cpu_manager.go:414] "containerMap: removing container" podUID="1b230406-0b7d-4d69-ac8e-b5e57164a5e4" containerName="cleanup" Nov 03 01:30:43.759505 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:43.759410 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b230406-0b7d-4d69-ac8e-b5e57164a5e4" containerName="cleanup" Nov 03 01:30:43.759505 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:43.759418 4421 cpu_manager.go:414] "containerMap: removing container" podUID="58e1f190-d19c-43f7-afe2-e3079e8d9814" containerName="cleanup" Nov 03 01:30:43.759505 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:43.759423 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58e1f190-d19c-43f7-afe2-e3079e8d9814" containerName="cleanup" Nov 03 01:30:43.759505 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:43.759462 4421 cpu_manager.go:414] "containerMap: removing container" podUID="1579a23a-e983-4da2-b9be-427fe00d9de3" containerName="cleanup" Nov 03 01:30:43.759505 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:43.759468 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1579a23a-e983-4da2-b9be-427fe00d9de3" containerName="cleanup" Nov 03 01:30:43.759505 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:43.759478 4421 cpu_manager.go:414] "containerMap: removing container" podUID="917a9367-6e36-4f93-a9a6-b65bfd868704" containerName="cleanup" Nov 03 01:30:43.759505 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:43.759484 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="917a9367-6e36-4f93-a9a6-b65bfd868704" containerName="cleanup" Nov 03 01:30:43.759505 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:43.759491 4421 cpu_manager.go:414] "containerMap: removing container" podUID="645536c5-e930-48f6-bbf2-f4591ee8c368" containerName="cleanup" Nov 03 01:30:43.759505 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:43.759496 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="645536c5-e930-48f6-bbf2-f4591ee8c368" containerName="cleanup" Nov 03 01:30:43.759892 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:43.759516 4421 cpu_manager.go:414] "containerMap: removing container" podUID="10c1b824-d0d5-4b84-8961-c41c36242725" containerName="cleanup" Nov 03 01:30:43.759892 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:43.759520 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10c1b824-d0d5-4b84-8961-c41c36242725" containerName="cleanup" Nov 03 01:30:43.759892 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:43.759531 4421 cpu_manager.go:414] "containerMap: removing container" podUID="1b230406-0b7d-4d69-ac8e-b5e57164a5e4" containerName="cleanup" Nov 03 01:30:43.759892 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:43.759535 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b230406-0b7d-4d69-ac8e-b5e57164a5e4" containerName="cleanup" Nov 03 01:30:43.759892 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:43.759543 4421 cpu_manager.go:414] "containerMap: removing container" podUID="917a9367-6e36-4f93-a9a6-b65bfd868704" containerName="cleanup" Nov 03 01:30:43.759892 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:43.759547 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="917a9367-6e36-4f93-a9a6-b65bfd868704" containerName="cleanup" Nov 03 01:30:43.759892 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:43.759558 4421 cpu_manager.go:414] "containerMap: removing container" podUID="645536c5-e930-48f6-bbf2-f4591ee8c368" containerName="cleanup" Nov 03 01:30:43.759892 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:43.759562 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="645536c5-e930-48f6-bbf2-f4591ee8c368" containerName="cleanup" Nov 03 01:30:43.759892 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:43.759569 4421 cpu_manager.go:414] "containerMap: removing container" podUID="e9bfb106-b718-4909-a826-79040347a857" containerName="cleanup" Nov 03 01:30:43.759892 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:43.759573 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9bfb106-b718-4909-a826-79040347a857" containerName="cleanup" Nov 03 01:30:43.759892 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:43.759579 4421 cpu_manager.go:414] "containerMap: removing container" podUID="91be6fd5-032c-4abe-aedf-e753fe1d7386" containerName="cleanup" Nov 03 01:30:43.759892 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:43.759583 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91be6fd5-032c-4abe-aedf-e753fe1d7386" containerName="cleanup" Nov 03 01:30:43.759892 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:43.759590 4421 cpu_manager.go:414] "containerMap: removing container" podUID="917a9367-6e36-4f93-a9a6-b65bfd868704" containerName="cleanup" Nov 03 01:30:43.759892 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:43.759594 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="917a9367-6e36-4f93-a9a6-b65bfd868704" containerName="cleanup" Nov 03 01:30:43.759892 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:43.759600 4421 cpu_manager.go:414] "containerMap: removing container" podUID="91be6fd5-032c-4abe-aedf-e753fe1d7386" containerName="cleanup" Nov 03 01:30:43.759892 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:43.759604 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91be6fd5-032c-4abe-aedf-e753fe1d7386" containerName="cleanup" Nov 03 01:30:43.759892 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:43.759620 4421 cpu_manager.go:414] "containerMap: removing container" podUID="1579a23a-e983-4da2-b9be-427fe00d9de3" containerName="cleanup" Nov 03 01:30:43.759892 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:43.759625 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1579a23a-e983-4da2-b9be-427fe00d9de3" containerName="cleanup" Nov 03 01:30:43.759892 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:43.759632 4421 cpu_manager.go:414] "containerMap: removing container" podUID="e9bfb106-b718-4909-a826-79040347a857" containerName="cleanup" Nov 03 01:30:43.759892 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:43.759637 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9bfb106-b718-4909-a826-79040347a857" containerName="cleanup" Nov 03 01:30:43.759892 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:43.759644 4421 cpu_manager.go:414] "containerMap: removing container" podUID="e9bfb106-b718-4909-a826-79040347a857" containerName="cleanup" Nov 03 01:30:43.759892 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:43.759648 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9bfb106-b718-4909-a826-79040347a857" containerName="cleanup" Nov 03 01:30:43.759892 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:43.759656 4421 cpu_manager.go:414] "containerMap: removing container" podUID="645536c5-e930-48f6-bbf2-f4591ee8c368" containerName="cleanup" Nov 03 01:30:43.759892 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:43.759660 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="645536c5-e930-48f6-bbf2-f4591ee8c368" containerName="cleanup" Nov 03 01:30:43.759892 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:43.759671 4421 cpu_manager.go:414] "containerMap: removing container" podUID="58e1f190-d19c-43f7-afe2-e3079e8d9814" containerName="cleanup" Nov 03 01:30:43.759892 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:43.759675 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58e1f190-d19c-43f7-afe2-e3079e8d9814" containerName="cleanup" Nov 03 01:30:43.759892 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:43.759816 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="10c1b824-d0d5-4b84-8961-c41c36242725" containerName="cleanup" Nov 03 01:30:43.759892 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:43.759829 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="10c1b824-d0d5-4b84-8961-c41c36242725" containerName="cleanup" Nov 03 01:30:43.759892 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:43.759839 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b230406-0b7d-4d69-ac8e-b5e57164a5e4" containerName="cleanup" Nov 03 01:30:43.759892 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:43.759847 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="917a9367-6e36-4f93-a9a6-b65bfd868704" containerName="cleanup" Nov 03 01:30:43.759892 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:43.759855 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="645536c5-e930-48f6-bbf2-f4591ee8c368" containerName="cleanup" Nov 03 01:30:43.759892 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:43.759860 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9bfb106-b718-4909-a826-79040347a857" containerName="cleanup" Nov 03 01:30:43.759892 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:43.759865 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="91be6fd5-032c-4abe-aedf-e753fe1d7386" containerName="cleanup" Nov 03 01:30:43.759892 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:43.759869 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="917a9367-6e36-4f93-a9a6-b65bfd868704" containerName="cleanup" Nov 03 01:30:43.759892 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:43.759875 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="91be6fd5-032c-4abe-aedf-e753fe1d7386" containerName="cleanup" Nov 03 01:30:43.759892 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:43.759880 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="1579a23a-e983-4da2-b9be-427fe00d9de3" containerName="cleanup" Nov 03 01:30:43.759892 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:43.759886 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="58e1f190-d19c-43f7-afe2-e3079e8d9814" containerName="cleanup" Nov 03 01:30:43.759892 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:43.759893 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9bfb106-b718-4909-a826-79040347a857" containerName="cleanup" Nov 03 01:30:43.759892 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:43.759900 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9bfb106-b718-4909-a826-79040347a857" containerName="cleanup" Nov 03 01:30:43.759892 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:43.759907 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="645536c5-e930-48f6-bbf2-f4591ee8c368" containerName="cleanup" Nov 03 01:30:43.759892 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:43.759914 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="58e1f190-d19c-43f7-afe2-e3079e8d9814" containerName="cleanup" Nov 03 01:30:43.763584 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:43.759918 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="10c1b824-d0d5-4b84-8961-c41c36242725" containerName="cleanup" Nov 03 01:30:43.763584 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:43.759926 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="58e1f190-d19c-43f7-afe2-e3079e8d9814" containerName="cleanup" Nov 03 01:30:43.763584 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:43.759932 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="1579a23a-e983-4da2-b9be-427fe00d9de3" containerName="cleanup" Nov 03 01:30:43.763584 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:43.759938 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b230406-0b7d-4d69-ac8e-b5e57164a5e4" containerName="cleanup" Nov 03 01:30:43.763860 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:43.763843 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-4tt4f" Nov 03 01:30:43.776443 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:43.776415 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-4tt4f"] Nov 03 01:30:43.901065 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:43.901001 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58e1f190-d19c-43f7-afe2-e3079e8d9814-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/58e1f190-d19c-43f7-afe2-e3079e8d9814-maas-api-service-ca\") pod \"58e1f190-d19c-43f7-afe2-e3079e8d9814\" (UID: \"58e1f190-d19c-43f7-afe2-e3079e8d9814\") " Nov 03 01:30:43.901339 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:43.901301 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58e1f190-d19c-43f7-afe2-e3079e8d9814-kube-api-access-rv7d2\" (UniqueName: \"kubernetes.io/projected/58e1f190-d19c-43f7-afe2-e3079e8d9814-kube-api-access-rv7d2\") pod \"58e1f190-d19c-43f7-afe2-e3079e8d9814\" (UID: \"58e1f190-d19c-43f7-afe2-e3079e8d9814\") " Nov 03 01:30:43.901791 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:43.901766 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58e1f190-d19c-43f7-afe2-e3079e8d9814-maas-api-service-ca" pod "58e1f190-d19c-43f7-afe2-e3079e8d9814" (UID: "58e1f190-d19c-43f7-afe2-e3079e8d9814"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:30:43.901846 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:43.901782 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b645b6f2-58be-45c4-9a6f-6785a1047bea-istio-data\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-4tt4f\" (UID: \"b645b6f2-58be-45c4-9a6f-6785a1047bea\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-4tt4f" Nov 03 01:30:43.901946 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:43.901929 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvfzl\" (UniqueName: \"kubernetes.io/projected/b645b6f2-58be-45c4-9a6f-6785a1047bea-kube-api-access-rvfzl\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-4tt4f\" (UID: \"b645b6f2-58be-45c4-9a6f-6785a1047bea\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-4tt4f" Nov 03 01:30:43.902084 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:43.902064 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b645b6f2-58be-45c4-9a6f-6785a1047bea-istiod-ca-cert\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-4tt4f\" (UID: \"b645b6f2-58be-45c4-9a6f-6785a1047bea\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-4tt4f" Nov 03 01:30:43.902211 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:43.902102 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b645b6f2-58be-45c4-9a6f-6785a1047bea-workload-certs\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-4tt4f\" (UID: \"b645b6f2-58be-45c4-9a6f-6785a1047bea\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-4tt4f" Nov 03 01:30:43.902211 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:43.902139 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b645b6f2-58be-45c4-9a6f-6785a1047bea-istio-podinfo\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-4tt4f\" (UID: \"b645b6f2-58be-45c4-9a6f-6785a1047bea\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-4tt4f" Nov 03 01:30:43.902361 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:43.902340 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b645b6f2-58be-45c4-9a6f-6785a1047bea-istio-envoy\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-4tt4f\" (UID: \"b645b6f2-58be-45c4-9a6f-6785a1047bea\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-4tt4f" Nov 03 01:30:43.902424 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:43.902406 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b645b6f2-58be-45c4-9a6f-6785a1047bea-istio-token\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-4tt4f\" (UID: \"b645b6f2-58be-45c4-9a6f-6785a1047bea\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-4tt4f" Nov 03 01:30:43.902479 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:43.902463 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b645b6f2-58be-45c4-9a6f-6785a1047bea-credential-socket\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-4tt4f\" (UID: \"b645b6f2-58be-45c4-9a6f-6785a1047bea\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-4tt4f" Nov 03 01:30:43.902564 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:43.902545 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b645b6f2-58be-45c4-9a6f-6785a1047bea-workload-socket\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-4tt4f\" (UID: \"b645b6f2-58be-45c4-9a6f-6785a1047bea\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-4tt4f" Nov 03 01:30:43.902732 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:43.902721 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/58e1f190-d19c-43f7-afe2-e3079e8d9814-maas-api-service-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:30:43.903906 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:43.903882 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58e1f190-d19c-43f7-afe2-e3079e8d9814-kube-api-access-rv7d2" pod "58e1f190-d19c-43f7-afe2-e3079e8d9814" (UID: "58e1f190-d19c-43f7-afe2-e3079e8d9814"). InnerVolumeSpecName "kube-api-access-rv7d2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:30:43.983298 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:43.983247 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1579a23a-e983-4da2-b9be-427fe00d9de3" path="/var/lib/kubelet/pods/1579a23a-e983-4da2-b9be-427fe00d9de3/volumes" Nov 03 01:30:43.983903 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:43.983887 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b230406-0b7d-4d69-ac8e-b5e57164a5e4" path="/var/lib/kubelet/pods/1b230406-0b7d-4d69-ac8e-b5e57164a5e4/volumes" Nov 03 01:30:43.984393 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:43.984375 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="645536c5-e930-48f6-bbf2-f4591ee8c368" path="/var/lib/kubelet/pods/645536c5-e930-48f6-bbf2-f4591ee8c368/volumes" Nov 03 01:30:43.985141 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:43.985125 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91be6fd5-032c-4abe-aedf-e753fe1d7386" path="/var/lib/kubelet/pods/91be6fd5-032c-4abe-aedf-e753fe1d7386/volumes" Nov 03 01:30:44.046351 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:44.026100 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b645b6f2-58be-45c4-9a6f-6785a1047bea-istio-token\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-4tt4f\" (UID: \"b645b6f2-58be-45c4-9a6f-6785a1047bea\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-4tt4f" Nov 03 01:30:44.046692 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:44.046662 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b645b6f2-58be-45c4-9a6f-6785a1047bea-credential-socket\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-4tt4f\" (UID: \"b645b6f2-58be-45c4-9a6f-6785a1047bea\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-4tt4f" Nov 03 01:30:44.046903 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:44.046879 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b645b6f2-58be-45c4-9a6f-6785a1047bea-workload-socket\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-4tt4f\" (UID: \"b645b6f2-58be-45c4-9a6f-6785a1047bea\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-4tt4f" Nov 03 01:30:44.047233 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:44.047211 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b645b6f2-58be-45c4-9a6f-6785a1047bea-istio-data\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-4tt4f\" (UID: \"b645b6f2-58be-45c4-9a6f-6785a1047bea\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-4tt4f" Nov 03 01:30:44.047724 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:44.047695 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvfzl\" (UniqueName: \"kubernetes.io/projected/b645b6f2-58be-45c4-9a6f-6785a1047bea-kube-api-access-rvfzl\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-4tt4f\" (UID: \"b645b6f2-58be-45c4-9a6f-6785a1047bea\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-4tt4f" Nov 03 01:30:44.048004 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:44.047984 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b645b6f2-58be-45c4-9a6f-6785a1047bea-istiod-ca-cert\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-4tt4f\" (UID: \"b645b6f2-58be-45c4-9a6f-6785a1047bea\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-4tt4f" Nov 03 01:30:44.048178 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:44.048160 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b645b6f2-58be-45c4-9a6f-6785a1047bea-workload-certs\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-4tt4f\" (UID: \"b645b6f2-58be-45c4-9a6f-6785a1047bea\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-4tt4f" Nov 03 01:30:44.061307 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:44.048470 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b645b6f2-58be-45c4-9a6f-6785a1047bea-istio-podinfo\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-4tt4f\" (UID: \"b645b6f2-58be-45c4-9a6f-6785a1047bea\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-4tt4f" Nov 03 01:30:44.072944 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:44.069456 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b645b6f2-58be-45c4-9a6f-6785a1047bea-istio-envoy\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-4tt4f\" (UID: \"b645b6f2-58be-45c4-9a6f-6785a1047bea\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-4tt4f" Nov 03 01:30:44.072944 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:44.069682 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rv7d2\" (UniqueName: \"kubernetes.io/projected/58e1f190-d19c-43f7-afe2-e3079e8d9814-kube-api-access-rv7d2\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:30:44.072944 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:44.058881 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b645b6f2-58be-45c4-9a6f-6785a1047bea-workload-socket\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-4tt4f\" (UID: \"b645b6f2-58be-45c4-9a6f-6785a1047bea\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-4tt4f" Nov 03 01:30:44.072944 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:44.065705 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b645b6f2-58be-45c4-9a6f-6785a1047bea-istio-data\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-4tt4f\" (UID: \"b645b6f2-58be-45c4-9a6f-6785a1047bea\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-4tt4f" Nov 03 01:30:44.072944 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:44.066393 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b645b6f2-58be-45c4-9a6f-6785a1047bea-istiod-ca-cert\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-4tt4f\" (UID: \"b645b6f2-58be-45c4-9a6f-6785a1047bea\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-4tt4f" Nov 03 01:30:44.072944 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:44.066900 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b645b6f2-58be-45c4-9a6f-6785a1047bea-workload-certs\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-4tt4f\" (UID: \"b645b6f2-58be-45c4-9a6f-6785a1047bea\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-4tt4f" Nov 03 01:30:44.072944 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:44.044524 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b645b6f2-58be-45c4-9a6f-6785a1047bea-istio-token\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-4tt4f\" (UID: \"b645b6f2-58be-45c4-9a6f-6785a1047bea\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-4tt4f" Nov 03 01:30:44.072944 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:44.058464 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b645b6f2-58be-45c4-9a6f-6785a1047bea-credential-socket\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-4tt4f\" (UID: \"b645b6f2-58be-45c4-9a6f-6785a1047bea\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-4tt4f" Nov 03 01:30:44.072944 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:44.072012 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b645b6f2-58be-45c4-9a6f-6785a1047bea-istio-podinfo\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-4tt4f\" (UID: \"b645b6f2-58be-45c4-9a6f-6785a1047bea\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-4tt4f" Nov 03 01:30:44.085293 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:44.075080 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvfzl\" (UniqueName: \"kubernetes.io/projected/b645b6f2-58be-45c4-9a6f-6785a1047bea-kube-api-access-rvfzl\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-4tt4f\" (UID: \"b645b6f2-58be-45c4-9a6f-6785a1047bea\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-4tt4f" Nov 03 01:30:44.117773 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:44.108430 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b645b6f2-58be-45c4-9a6f-6785a1047bea-istio-envoy\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-4tt4f\" (UID: \"b645b6f2-58be-45c4-9a6f-6785a1047bea\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-4tt4f" Nov 03 01:30:44.387380 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:44.387242 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-4tt4f" Nov 03 01:30:44.549895 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:44.549866 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b4971a-29894490-f47mr" Nov 03 01:30:44.550334 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:44.549898 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b4971a-29894490-f47mr" event={"ID":"58e1f190-d19c-43f7-afe2-e3079e8d9814","Type":"ContainerDied","Data":"fc8bcdbec76a5f24638e8f6f4f223a9299f9fdede8cb9affd05e22a6b0eca739"} Nov 03 01:30:44.550334 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:44.549954 4421 scope.go:122] "RemoveContainer" containerID="11cf6c12b18f0447b28523d7254b0dbd2c9069c620a41eb614bef3d0775941f5" Nov 03 01:30:44.570608 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:44.570564 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b4971a-29894490-f47mr"] Nov 03 01:30:44.576610 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:44.576582 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b4971a-29894490-f47mr"] Nov 03 01:30:44.586487 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:44.586450 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6fec8a-29894490-r4cj4"] Nov 03 01:30:44.691314 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:44.687675 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-4tt4f"] Nov 03 01:30:44.691314 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:30:44.689516 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 01:30:44.693811 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:30:44.693779 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb645b6f2_58be_45c4_9a6f_6785a1047bea.slice/crio-d28b1487e660fd3e4be7ce8b45602b4c6679fb226c53b02cce829101dcbec6bf WatchSource:0}: Error finding container d28b1487e660fd3e4be7ce8b45602b4c6679fb226c53b02cce829101dcbec6bf: Status 404 returned error can't find the container with id d28b1487e660fd3e4be7ce8b45602b4c6679fb226c53b02cce829101dcbec6bf Nov 03 01:30:44.698169 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:44.698130 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:30:44.698282 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:44.698209 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:30:44.698282 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:44.698259 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:30:44.761641 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:44.761613 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6fec8a-29894490-r4cj4" Nov 03 01:30:44.901848 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:44.892053 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6bb86426-5f37-4afc-9d67-83399abe9862-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bb86426-5f37-4afc-9d67-83399abe9862-maas-api-service-ca\") pod \"6bb86426-5f37-4afc-9d67-83399abe9862\" (UID: \"6bb86426-5f37-4afc-9d67-83399abe9862\") " Nov 03 01:30:44.901848 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:44.892223 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6bb86426-5f37-4afc-9d67-83399abe9862-kube-api-access-l577x\" (UniqueName: \"kubernetes.io/projected/6bb86426-5f37-4afc-9d67-83399abe9862-kube-api-access-l577x\") pod \"6bb86426-5f37-4afc-9d67-83399abe9862\" (UID: \"6bb86426-5f37-4afc-9d67-83399abe9862\") " Nov 03 01:30:44.906918 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:44.906873 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bb86426-5f37-4afc-9d67-83399abe9862-maas-api-service-ca" pod "6bb86426-5f37-4afc-9d67-83399abe9862" (UID: "6bb86426-5f37-4afc-9d67-83399abe9862"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:30:44.907656 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:44.907623 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bb86426-5f37-4afc-9d67-83399abe9862-kube-api-access-l577x" pod "6bb86426-5f37-4afc-9d67-83399abe9862" (UID: "6bb86426-5f37-4afc-9d67-83399abe9862"). InnerVolumeSpecName "kube-api-access-l577x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:30:44.994678 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:44.994642 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bb86426-5f37-4afc-9d67-83399abe9862-maas-api-service-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:30:44.994678 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:44.994678 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l577x\" (UniqueName: \"kubernetes.io/projected/6bb86426-5f37-4afc-9d67-83399abe9862-kube-api-access-l577x\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:30:45.618845 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:45.618802 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6fec8a-29894490-r4cj4" event={"ID":"6bb86426-5f37-4afc-9d67-83399abe9862","Type":"ContainerDied","Data":"8f648f87e721607e4305a103b9586634a08fcb619a40e32248d857bac4bc788a"} Nov 03 01:30:45.619512 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:45.619490 4421 scope.go:122] "RemoveContainer" containerID="69f2f6da1191fe75fe7cbcee8d8a7df17983135334c71d41a0b267ff0d131583" Nov 03 01:30:45.619849 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:45.619830 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6fec8a-29894490-r4cj4" Nov 03 01:30:45.623400 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:45.623371 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-4tt4f" event={"ID":"b645b6f2-58be-45c4-9a6f-6785a1047bea","Type":"ContainerStarted","Data":"38f6c23c7753df1e2d6b4de8eabe27e5046425a24da317c37e9e24936a4adf82"} Nov 03 01:30:45.623701 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:45.623681 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-4tt4f" event={"ID":"b645b6f2-58be-45c4-9a6f-6785a1047bea","Type":"ContainerStarted","Data":"d28b1487e660fd3e4be7ce8b45602b4c6679fb226c53b02cce829101dcbec6bf"} Nov 03 01:30:45.645860 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:45.645803 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-4tt4f" podStartSLOduration=266.645784251 podStartE2EDuration="4m26.645784251s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:26:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:30:45.642238626 +0000 UTC m=+2987.813241719" watchObservedRunningTime="2026-11-03 01:30:45.645784251 +0000 UTC m=+2987.816787348" Nov 03 01:30:45.662011 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:45.661973 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6fec8a-29894490-r4cj4"] Nov 03 01:30:45.667616 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:45.667582 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6fec8a-29894490-r4cj4"] Nov 03 01:30:45.982425 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:45.982346 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58e1f190-d19c-43f7-afe2-e3079e8d9814" path="/var/lib/kubelet/pods/58e1f190-d19c-43f7-afe2-e3079e8d9814/volumes" Nov 03 01:30:45.982943 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:45.982924 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6bb86426-5f37-4afc-9d67-83399abe9862" path="/var/lib/kubelet/pods/6bb86426-5f37-4afc-9d67-83399abe9862/volumes" Nov 03 01:30:46.388296 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:46.388253 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-4tt4f" Nov 03 01:30:46.389583 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:46.389554 4421 patch_prober.go:29] interesting pod/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-4tt4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.85:15021/healthz/ready\": dial tcp 10.134.1.85:15021: connect: connection refused" start-of-body= Nov 03 01:30:46.389734 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:46.389606 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-4tt4f" podUID="b645b6f2-58be-45c4-9a6f-6785a1047bea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.85:15021/healthz/ready\": dial tcp 10.134.1.85:15021: connect: connection refused" Nov 03 01:30:47.388060 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:47.388022 4421 patch_prober.go:29] interesting pod/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-4tt4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.85:15021/healthz/ready\": dial tcp 10.134.1.85:15021: connect: connection refused" start-of-body= Nov 03 01:30:47.388601 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:47.388100 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-4tt4f" podUID="b645b6f2-58be-45c4-9a6f-6785a1047bea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.85:15021/healthz/ready\": dial tcp 10.134.1.85:15021: connect: connection refused" Nov 03 01:30:48.387554 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:48.387518 4421 patch_prober.go:29] interesting pod/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-4tt4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.85:15021/healthz/ready\": dial tcp 10.134.1.85:15021: connect: connection refused" start-of-body= Nov 03 01:30:48.387714 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:48.387573 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-4tt4f" podUID="b645b6f2-58be-45c4-9a6f-6785a1047bea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.85:15021/healthz/ready\": dial tcp 10.134.1.85:15021: connect: connection refused" Nov 03 01:30:49.388477 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:49.388442 4421 patch_prober.go:29] interesting pod/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-4tt4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.85:15021/healthz/ready\": dial tcp 10.134.1.85:15021: connect: connection refused" start-of-body= Nov 03 01:30:49.388967 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:49.388505 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-4tt4f" podUID="b645b6f2-58be-45c4-9a6f-6785a1047bea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.85:15021/healthz/ready\": dial tcp 10.134.1.85:15021: connect: connection refused" Nov 03 01:30:50.387591 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:50.387559 4421 patch_prober.go:29] interesting pod/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-4tt4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.85:15021/healthz/ready\": dial tcp 10.134.1.85:15021: connect: connection refused" start-of-body= Nov 03 01:30:50.387756 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:50.387615 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-4tt4f" podUID="b645b6f2-58be-45c4-9a6f-6785a1047bea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.85:15021/healthz/ready\": dial tcp 10.134.1.85:15021: connect: connection refused" Nov 03 01:30:51.387915 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:51.387882 4421 patch_prober.go:29] interesting pod/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-4tt4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.85:15021/healthz/ready\": dial tcp 10.134.1.85:15021: connect: connection refused" start-of-body= Nov 03 01:30:51.388392 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:51.387946 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-4tt4f" podUID="b645b6f2-58be-45c4-9a6f-6785a1047bea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.85:15021/healthz/ready\": dial tcp 10.134.1.85:15021: connect: connection refused" Nov 03 01:30:52.394291 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:52.394240 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-4tt4f" Nov 03 01:30:52.394908 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:52.394884 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-4tt4f" Nov 03 01:30:52.395555 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:30:52.395534 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-4tt4f" Nov 03 01:31:05.013131 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:05.013096 4421 scope.go:122] "RemoveContainer" containerID="0b344147dabf5a9c489eca52ff337bf3730d7d1ccc55b93ca73f00185e244c4d" Nov 03 01:31:05.026687 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:05.026652 4421 scope.go:122] "RemoveContainer" containerID="a815d1e2aaf9f68e0b75cd8024b9cb7911738a8edf96b9728cb50bb0b988e611" Nov 03 01:31:06.136369 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:06.136322 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dbbdccc9-lp88t"] Nov 03 01:31:06.156794 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:06.156753 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dbbdccc9-lp88t"] Nov 03 01:31:07.982480 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:07.982446 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f7644c6-d7bd-4e28-ab78-ed80a99428dc" path="/var/lib/kubelet/pods/3f7644c6-d7bd-4e28-ab78-ed80a99428dc/volumes" Nov 03 01:31:12.226797 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:12.226700 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-nccrq"] Nov 03 01:31:12.227352 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:12.226950 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-nccrq" podUID="a0d8360b-b4c3-483e-8b00-412893aee9a6" containerName="istio-proxy" containerID="cri-o://3bf31bc22bf58fa760a72288611e2065dad3745f00cb10885401226f9b1f2a44" gracePeriod=30 Nov 03 01:31:14.337894 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:14.337855 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-mxvfl"] Nov 03 01:31:14.338372 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:14.338106 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-mxvfl" podUID="5a1e781a-b4bc-4e84-85ef-18b48bf77f99" containerName="istio-proxy" containerID="cri-o://0f6d27237cee407e3c0c756d4a58ba7518d85cbc28093f1f3b69d9ebf144cf95" gracePeriod=30 Nov 03 01:31:14.677572 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:31:14.677474 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 01:31:17.031207 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:17.031169 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5c9e521-qx24h"] Nov 03 01:31:17.036197 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:17.036172 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5c9e521-qx24h"] Nov 03 01:31:17.530152 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:17.530113 4421 generic.go:356] "Generic (PLEG): container finished" podID="a0d8360b-b4c3-483e-8b00-412893aee9a6" containerID="3bf31bc22bf58fa760a72288611e2065dad3745f00cb10885401226f9b1f2a44" exitCode=0 Nov 03 01:31:17.530355 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:17.530188 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-nccrq" event={"ID":"a0d8360b-b4c3-483e-8b00-412893aee9a6","Type":"ContainerDied","Data":"3bf31bc22bf58fa760a72288611e2065dad3745f00cb10885401226f9b1f2a44"} Nov 03 01:31:17.530355 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:17.530251 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-nccrq" event={"ID":"a0d8360b-b4c3-483e-8b00-412893aee9a6","Type":"ContainerDied","Data":"58d0683ab935e771010a40747d5bac5841b2305e02e74f90be840adeba1d03f5"} Nov 03 01:31:17.530355 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:17.530319 4421 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="58d0683ab935e771010a40747d5bac5841b2305e02e74f90be840adeba1d03f5" Nov 03 01:31:17.546700 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:17.546673 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-nccrq" Nov 03 01:31:17.580776 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:17.580744 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-gvmnj"] Nov 03 01:31:17.581676 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:17.581661 4421 cpu_manager.go:414] "containerMap: removing container" podUID="6bb86426-5f37-4afc-9d67-83399abe9862" containerName="cleanup" Nov 03 01:31:17.581729 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:17.581678 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bb86426-5f37-4afc-9d67-83399abe9862" containerName="cleanup" Nov 03 01:31:17.581729 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:17.581692 4421 cpu_manager.go:414] "containerMap: removing container" podUID="6bb86426-5f37-4afc-9d67-83399abe9862" containerName="cleanup" Nov 03 01:31:17.581729 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:17.581698 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bb86426-5f37-4afc-9d67-83399abe9862" containerName="cleanup" Nov 03 01:31:17.581729 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:17.581720 4421 cpu_manager.go:414] "containerMap: removing container" podUID="58e1f190-d19c-43f7-afe2-e3079e8d9814" containerName="cleanup" Nov 03 01:31:17.581874 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:17.581729 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58e1f190-d19c-43f7-afe2-e3079e8d9814" containerName="cleanup" Nov 03 01:31:17.581874 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:17.581744 4421 cpu_manager.go:414] "containerMap: removing container" podUID="1579a23a-e983-4da2-b9be-427fe00d9de3" containerName="cleanup" Nov 03 01:31:17.581874 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:17.581748 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1579a23a-e983-4da2-b9be-427fe00d9de3" containerName="cleanup" Nov 03 01:31:17.581874 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:17.581756 4421 cpu_manager.go:414] "containerMap: removing container" podUID="a0d8360b-b4c3-483e-8b00-412893aee9a6" containerName="istio-proxy" Nov 03 01:31:17.581874 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:17.581761 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0d8360b-b4c3-483e-8b00-412893aee9a6" containerName="istio-proxy" Nov 03 01:31:17.581874 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:17.581786 4421 cpu_manager.go:414] "containerMap: removing container" podUID="10c1b824-d0d5-4b84-8961-c41c36242725" containerName="cleanup" Nov 03 01:31:17.581874 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:17.581791 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10c1b824-d0d5-4b84-8961-c41c36242725" containerName="cleanup" Nov 03 01:31:17.581874 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:17.581810 4421 cpu_manager.go:414] "containerMap: removing container" podUID="1b230406-0b7d-4d69-ac8e-b5e57164a5e4" containerName="cleanup" Nov 03 01:31:17.581874 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:17.581825 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b230406-0b7d-4d69-ac8e-b5e57164a5e4" containerName="cleanup" Nov 03 01:31:17.582164 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:17.581991 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bb86426-5f37-4afc-9d67-83399abe9862" containerName="cleanup" Nov 03 01:31:17.582164 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:17.582000 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bb86426-5f37-4afc-9d67-83399abe9862" containerName="cleanup" Nov 03 01:31:17.582164 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:17.582009 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="1579a23a-e983-4da2-b9be-427fe00d9de3" containerName="cleanup" Nov 03 01:31:17.582164 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:17.582018 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0d8360b-b4c3-483e-8b00-412893aee9a6" containerName="istio-proxy" Nov 03 01:31:17.582164 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:17.582028 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bb86426-5f37-4afc-9d67-83399abe9862" containerName="cleanup" Nov 03 01:31:17.582164 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:17.582034 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b230406-0b7d-4d69-ac8e-b5e57164a5e4" containerName="cleanup" Nov 03 01:31:17.582164 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:17.582043 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="645536c5-e930-48f6-bbf2-f4591ee8c368" containerName="cleanup" Nov 03 01:31:17.582164 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:17.582052 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="917a9367-6e36-4f93-a9a6-b65bfd868704" containerName="cleanup" Nov 03 01:31:17.582164 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:17.582057 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="91be6fd5-032c-4abe-aedf-e753fe1d7386" containerName="cleanup" Nov 03 01:31:17.586879 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:17.586854 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-gvmnj" Nov 03 01:31:17.589441 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:17.588474 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0d8360b-b4c3-483e-8b00-412893aee9a6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0d8360b-b4c3-483e-8b00-412893aee9a6-istiod-ca-cert\") pod \"a0d8360b-b4c3-483e-8b00-412893aee9a6\" (UID: \"a0d8360b-b4c3-483e-8b00-412893aee9a6\") " Nov 03 01:31:17.589779 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:17.589755 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0d8360b-b4c3-483e-8b00-412893aee9a6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0d8360b-b4c3-483e-8b00-412893aee9a6-istio-data\") pod \"a0d8360b-b4c3-483e-8b00-412893aee9a6\" (UID: \"a0d8360b-b4c3-483e-8b00-412893aee9a6\") " Nov 03 01:31:17.589959 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:17.589944 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0d8360b-b4c3-483e-8b00-412893aee9a6-istio-token\" (UniqueName: \"kubernetes.io/projected/a0d8360b-b4c3-483e-8b00-412893aee9a6-istio-token\") pod \"a0d8360b-b4c3-483e-8b00-412893aee9a6\" (UID: \"a0d8360b-b4c3-483e-8b00-412893aee9a6\") " Nov 03 01:31:17.590473 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:17.590440 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0d8360b-b4c3-483e-8b00-412893aee9a6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0d8360b-b4c3-483e-8b00-412893aee9a6-workload-certs\") pod \"a0d8360b-b4c3-483e-8b00-412893aee9a6\" (UID: \"a0d8360b-b4c3-483e-8b00-412893aee9a6\") " Nov 03 01:31:17.590999 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:17.590881 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0d8360b-b4c3-483e-8b00-412893aee9a6-workload-certs" pod "a0d8360b-b4c3-483e-8b00-412893aee9a6" (UID: "a0d8360b-b4c3-483e-8b00-412893aee9a6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:31:17.591657 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:17.591260 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a0d8360b-b4c3-483e-8b00-412893aee9a6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0d8360b-b4c3-483e-8b00-412893aee9a6-istio-podinfo\") pod \"a0d8360b-b4c3-483e-8b00-412893aee9a6\" (UID: \"a0d8360b-b4c3-483e-8b00-412893aee9a6\") " Nov 03 01:31:17.592002 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:17.591985 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0d8360b-b4c3-483e-8b00-412893aee9a6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0d8360b-b4c3-483e-8b00-412893aee9a6-istio-envoy\") pod \"a0d8360b-b4c3-483e-8b00-412893aee9a6\" (UID: \"a0d8360b-b4c3-483e-8b00-412893aee9a6\") " Nov 03 01:31:17.592495 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:17.592473 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0d8360b-b4c3-483e-8b00-412893aee9a6-kube-api-access-k447z\" (UniqueName: \"kubernetes.io/projected/a0d8360b-b4c3-483e-8b00-412893aee9a6-kube-api-access-k447z\") pod \"a0d8360b-b4c3-483e-8b00-412893aee9a6\" (UID: \"a0d8360b-b4c3-483e-8b00-412893aee9a6\") " Nov 03 01:31:17.592755 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:17.592729 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0d8360b-b4c3-483e-8b00-412893aee9a6-istio-data" pod "a0d8360b-b4c3-483e-8b00-412893aee9a6" (UID: "a0d8360b-b4c3-483e-8b00-412893aee9a6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:31:17.592953 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:17.592934 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0d8360b-b4c3-483e-8b00-412893aee9a6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0d8360b-b4c3-483e-8b00-412893aee9a6-workload-socket\") pod \"a0d8360b-b4c3-483e-8b00-412893aee9a6\" (UID: \"a0d8360b-b4c3-483e-8b00-412893aee9a6\") " Nov 03 01:31:17.593089 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:17.593073 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0d8360b-b4c3-483e-8b00-412893aee9a6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0d8360b-b4c3-483e-8b00-412893aee9a6-credential-socket\") pod \"a0d8360b-b4c3-483e-8b00-412893aee9a6\" (UID: \"a0d8360b-b4c3-483e-8b00-412893aee9a6\") " Nov 03 01:31:17.594478 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:17.594458 4421 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0d8360b-b4c3-483e-8b00-412893aee9a6-istio-data\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:31:17.596589 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:17.596569 4421 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0d8360b-b4c3-483e-8b00-412893aee9a6-workload-certs\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:31:17.597013 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:17.594747 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0d8360b-b4c3-483e-8b00-412893aee9a6-istiod-ca-cert" pod "a0d8360b-b4c3-483e-8b00-412893aee9a6" (UID: "a0d8360b-b4c3-483e-8b00-412893aee9a6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:31:17.597142 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:17.596725 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0d8360b-b4c3-483e-8b00-412893aee9a6-credential-socket" pod "a0d8360b-b4c3-483e-8b00-412893aee9a6" (UID: "a0d8360b-b4c3-483e-8b00-412893aee9a6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:31:17.598415 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:17.598389 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0d8360b-b4c3-483e-8b00-412893aee9a6-workload-socket" pod "a0d8360b-b4c3-483e-8b00-412893aee9a6" (UID: "a0d8360b-b4c3-483e-8b00-412893aee9a6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:31:17.615949 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:17.614909 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a0d8360b-b4c3-483e-8b00-412893aee9a6-istio-podinfo" pod "a0d8360b-b4c3-483e-8b00-412893aee9a6" (UID: "a0d8360b-b4c3-483e-8b00-412893aee9a6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:31:17.622295 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:17.622205 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0d8360b-b4c3-483e-8b00-412893aee9a6-istio-envoy" pod "a0d8360b-b4c3-483e-8b00-412893aee9a6" (UID: "a0d8360b-b4c3-483e-8b00-412893aee9a6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:31:17.626323 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:17.624751 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0d8360b-b4c3-483e-8b00-412893aee9a6-istio-token" pod "a0d8360b-b4c3-483e-8b00-412893aee9a6" (UID: "a0d8360b-b4c3-483e-8b00-412893aee9a6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:31:17.630791 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:17.630746 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0d8360b-b4c3-483e-8b00-412893aee9a6-kube-api-access-k447z" pod "a0d8360b-b4c3-483e-8b00-412893aee9a6" (UID: "a0d8360b-b4c3-483e-8b00-412893aee9a6"). InnerVolumeSpecName "kube-api-access-k447z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:31:17.633513 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:17.633479 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-gvmnj"] Nov 03 01:31:17.698553 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:17.698514 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/812a530a-f597-4049-a40a-2dcfb8ba6c74-istio-token\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-gvmnj\" (UID: \"812a530a-f597-4049-a40a-2dcfb8ba6c74\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-gvmnj" Nov 03 01:31:17.698775 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:17.698635 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhcqn\" (UniqueName: \"kubernetes.io/projected/812a530a-f597-4049-a40a-2dcfb8ba6c74-kube-api-access-vhcqn\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-gvmnj\" (UID: \"812a530a-f597-4049-a40a-2dcfb8ba6c74\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-gvmnj" Nov 03 01:31:17.698775 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:17.698684 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/812a530a-f597-4049-a40a-2dcfb8ba6c74-istio-podinfo\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-gvmnj\" (UID: \"812a530a-f597-4049-a40a-2dcfb8ba6c74\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-gvmnj" Nov 03 01:31:17.698948 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:17.698918 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/812a530a-f597-4049-a40a-2dcfb8ba6c74-credential-socket\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-gvmnj\" (UID: \"812a530a-f597-4049-a40a-2dcfb8ba6c74\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-gvmnj" Nov 03 01:31:17.699123 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:17.699095 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/812a530a-f597-4049-a40a-2dcfb8ba6c74-workload-socket\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-gvmnj\" (UID: \"812a530a-f597-4049-a40a-2dcfb8ba6c74\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-gvmnj" Nov 03 01:31:17.699215 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:17.699202 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/812a530a-f597-4049-a40a-2dcfb8ba6c74-istiod-ca-cert\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-gvmnj\" (UID: \"812a530a-f597-4049-a40a-2dcfb8ba6c74\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-gvmnj" Nov 03 01:31:17.699323 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:17.699248 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/812a530a-f597-4049-a40a-2dcfb8ba6c74-workload-certs\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-gvmnj\" (UID: \"812a530a-f597-4049-a40a-2dcfb8ba6c74\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-gvmnj" Nov 03 01:31:17.699323 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:17.699308 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/812a530a-f597-4049-a40a-2dcfb8ba6c74-istio-data\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-gvmnj\" (UID: \"812a530a-f597-4049-a40a-2dcfb8ba6c74\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-gvmnj" Nov 03 01:31:17.699455 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:17.699433 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/812a530a-f597-4049-a40a-2dcfb8ba6c74-istio-envoy\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-gvmnj\" (UID: \"812a530a-f597-4049-a40a-2dcfb8ba6c74\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-gvmnj" Nov 03 01:31:17.699642 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:17.699628 4421 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0d8360b-b4c3-483e-8b00-412893aee9a6-credential-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:31:17.699684 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:17.699648 4421 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0d8360b-b4c3-483e-8b00-412893aee9a6-istiod-ca-cert\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:31:17.699684 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:17.699659 4421 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0d8360b-b4c3-483e-8b00-412893aee9a6-istio-token\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:31:17.699684 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:17.699673 4421 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0d8360b-b4c3-483e-8b00-412893aee9a6-istio-podinfo\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:31:17.699779 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:17.699688 4421 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0d8360b-b4c3-483e-8b00-412893aee9a6-istio-envoy\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:31:17.699779 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:17.699700 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k447z\" (UniqueName: \"kubernetes.io/projected/a0d8360b-b4c3-483e-8b00-412893aee9a6-kube-api-access-k447z\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:31:17.699779 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:17.699711 4421 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0d8360b-b4c3-483e-8b00-412893aee9a6-workload-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:31:17.839884 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:17.827943 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/812a530a-f597-4049-a40a-2dcfb8ba6c74-istio-podinfo\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-gvmnj\" (UID: \"812a530a-f597-4049-a40a-2dcfb8ba6c74\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-gvmnj" Nov 03 01:31:17.839884 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:17.828140 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/812a530a-f597-4049-a40a-2dcfb8ba6c74-credential-socket\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-gvmnj\" (UID: \"812a530a-f597-4049-a40a-2dcfb8ba6c74\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-gvmnj" Nov 03 01:31:17.839884 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:17.828258 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/812a530a-f597-4049-a40a-2dcfb8ba6c74-workload-socket\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-gvmnj\" (UID: \"812a530a-f597-4049-a40a-2dcfb8ba6c74\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-gvmnj" Nov 03 01:31:17.839884 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:17.828343 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/812a530a-f597-4049-a40a-2dcfb8ba6c74-istiod-ca-cert\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-gvmnj\" (UID: \"812a530a-f597-4049-a40a-2dcfb8ba6c74\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-gvmnj" Nov 03 01:31:17.839884 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:17.828395 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/812a530a-f597-4049-a40a-2dcfb8ba6c74-workload-certs\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-gvmnj\" (UID: \"812a530a-f597-4049-a40a-2dcfb8ba6c74\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-gvmnj" Nov 03 01:31:17.839884 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:17.828427 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/812a530a-f597-4049-a40a-2dcfb8ba6c74-istio-data\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-gvmnj\" (UID: \"812a530a-f597-4049-a40a-2dcfb8ba6c74\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-gvmnj" Nov 03 01:31:17.839884 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:17.828496 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/812a530a-f597-4049-a40a-2dcfb8ba6c74-istio-envoy\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-gvmnj\" (UID: \"812a530a-f597-4049-a40a-2dcfb8ba6c74\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-gvmnj" Nov 03 01:31:17.839884 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:17.828546 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/812a530a-f597-4049-a40a-2dcfb8ba6c74-istio-token\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-gvmnj\" (UID: \"812a530a-f597-4049-a40a-2dcfb8ba6c74\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-gvmnj" Nov 03 01:31:17.839884 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:17.828615 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhcqn\" (UniqueName: \"kubernetes.io/projected/812a530a-f597-4049-a40a-2dcfb8ba6c74-kube-api-access-vhcqn\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-gvmnj\" (UID: \"812a530a-f597-4049-a40a-2dcfb8ba6c74\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-gvmnj" Nov 03 01:31:17.839884 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:17.832724 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/812a530a-f597-4049-a40a-2dcfb8ba6c74-workload-certs\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-gvmnj\" (UID: \"812a530a-f597-4049-a40a-2dcfb8ba6c74\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-gvmnj" Nov 03 01:31:17.839884 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:17.833015 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/812a530a-f597-4049-a40a-2dcfb8ba6c74-credential-socket\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-gvmnj\" (UID: \"812a530a-f597-4049-a40a-2dcfb8ba6c74\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-gvmnj" Nov 03 01:31:17.839884 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:17.833443 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/812a530a-f597-4049-a40a-2dcfb8ba6c74-workload-socket\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-gvmnj\" (UID: \"812a530a-f597-4049-a40a-2dcfb8ba6c74\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-gvmnj" Nov 03 01:31:17.839884 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:17.834164 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/812a530a-f597-4049-a40a-2dcfb8ba6c74-istiod-ca-cert\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-gvmnj\" (UID: \"812a530a-f597-4049-a40a-2dcfb8ba6c74\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-gvmnj" Nov 03 01:31:17.839884 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:17.835894 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/812a530a-f597-4049-a40a-2dcfb8ba6c74-istio-data\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-gvmnj\" (UID: \"812a530a-f597-4049-a40a-2dcfb8ba6c74\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-gvmnj" Nov 03 01:31:17.857260 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:17.852086 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/812a530a-f597-4049-a40a-2dcfb8ba6c74-istio-podinfo\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-gvmnj\" (UID: \"812a530a-f597-4049-a40a-2dcfb8ba6c74\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-gvmnj" Nov 03 01:31:17.864967 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:17.864913 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/812a530a-f597-4049-a40a-2dcfb8ba6c74-istio-envoy\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-gvmnj\" (UID: \"812a530a-f597-4049-a40a-2dcfb8ba6c74\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-gvmnj" Nov 03 01:31:17.885986 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:17.884041 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhcqn\" (UniqueName: \"kubernetes.io/projected/812a530a-f597-4049-a40a-2dcfb8ba6c74-kube-api-access-vhcqn\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-gvmnj\" (UID: \"812a530a-f597-4049-a40a-2dcfb8ba6c74\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-gvmnj" Nov 03 01:31:17.885986 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:17.885401 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/812a530a-f597-4049-a40a-2dcfb8ba6c74-istio-token\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-gvmnj\" (UID: \"812a530a-f597-4049-a40a-2dcfb8ba6c74\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-gvmnj" Nov 03 01:31:17.885986 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:17.885495 4421 patch_prober.go:29] interesting pod/e2e-extmod-a-d25759-openshift-default-d47fd9785-mxvfl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 01:31:17.885986 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:17.885543 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-mxvfl" podUID="5a1e781a-b4bc-4e84-85ef-18b48bf77f99" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 01:31:17.966418 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:17.966381 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-gvmnj" Nov 03 01:31:17.983834 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:17.983811 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2907a10d-55cc-4321-b4ec-169e4d8d0de2" path="/var/lib/kubelet/pods/2907a10d-55cc-4321-b4ec-169e4d8d0de2/volumes" Nov 03 01:31:18.257995 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:31:18.257963 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod812a530a_f597_4049_a40a_2dcfb8ba6c74.slice/crio-2a64502f7c1b61e1fe90ed30de8fa1b1950fd23d12164b1701b793e6b48feb3e WatchSource:0}: Error finding container 2a64502f7c1b61e1fe90ed30de8fa1b1950fd23d12164b1701b793e6b48feb3e: Status 404 returned error can't find the container with id 2a64502f7c1b61e1fe90ed30de8fa1b1950fd23d12164b1701b793e6b48feb3e Nov 03 01:31:18.258864 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:18.258839 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-gvmnj"] Nov 03 01:31:18.261232 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:18.261196 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:31:18.261375 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:18.261297 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:31:18.261375 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:18.261349 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:31:18.541364 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:18.541252 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-gvmnj" event={"ID":"812a530a-f597-4049-a40a-2dcfb8ba6c74","Type":"ContainerStarted","Data":"b177a223eb4b7553bf143b9f7b2045e22a53266c156107de37060f64bb62bc28"} Nov 03 01:31:18.541364 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:18.541328 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-gvmnj" event={"ID":"812a530a-f597-4049-a40a-2dcfb8ba6c74","Type":"ContainerStarted","Data":"2a64502f7c1b61e1fe90ed30de8fa1b1950fd23d12164b1701b793e6b48feb3e"} Nov 03 01:31:18.541608 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:18.541547 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-nccrq" Nov 03 01:31:18.566655 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:18.566565 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-gvmnj" podStartSLOduration=299.566552191 podStartE2EDuration="4m59.566552191s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:26:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:31:18.561378317 +0000 UTC m=+3020.732381413" watchObservedRunningTime="2026-11-03 01:31:18.566552191 +0000 UTC m=+3020.737555286" Nov 03 01:31:18.593090 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:18.593047 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-nccrq"] Nov 03 01:31:18.597918 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:18.597882 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-nccrq"] Nov 03 01:31:18.981874 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:18.971995 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-gvmnj" Nov 03 01:31:19.253563 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:19.253532 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-pm4c4"] Nov 03 01:31:19.253760 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:19.253736 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-pm4c4" podUID="af8d7fc5-bed0-4ebf-be1c-44805efec25a" containerName="istio-proxy" containerID="cri-o://fa60a24944e6d148d83da74de731ae827243e458dcb289705fab872206721c09" gracePeriod=30 Nov 03 01:31:19.553336 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:19.553226 4421 generic.go:356] "Generic (PLEG): container finished" podID="5a1e781a-b4bc-4e84-85ef-18b48bf77f99" containerID="0f6d27237cee407e3c0c756d4a58ba7518d85cbc28093f1f3b69d9ebf144cf95" exitCode=0 Nov 03 01:31:19.553831 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:19.553804 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-mxvfl" event={"ID":"5a1e781a-b4bc-4e84-85ef-18b48bf77f99","Type":"ContainerDied","Data":"0f6d27237cee407e3c0c756d4a58ba7518d85cbc28093f1f3b69d9ebf144cf95"} Nov 03 01:31:19.751724 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:19.751695 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-mxvfl" Nov 03 01:31:19.874708 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:19.874605 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a1e781a-b4bc-4e84-85ef-18b48bf77f99-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a1e781a-b4bc-4e84-85ef-18b48bf77f99-istio-data\") pod \"5a1e781a-b4bc-4e84-85ef-18b48bf77f99\" (UID: \"5a1e781a-b4bc-4e84-85ef-18b48bf77f99\") " Nov 03 01:31:19.874708 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:19.874677 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a1e781a-b4bc-4e84-85ef-18b48bf77f99-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a1e781a-b4bc-4e84-85ef-18b48bf77f99-istio-envoy\") pod \"5a1e781a-b4bc-4e84-85ef-18b48bf77f99\" (UID: \"5a1e781a-b4bc-4e84-85ef-18b48bf77f99\") " Nov 03 01:31:19.874957 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:19.874723 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a1e781a-b4bc-4e84-85ef-18b48bf77f99-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a1e781a-b4bc-4e84-85ef-18b48bf77f99-workload-certs\") pod \"5a1e781a-b4bc-4e84-85ef-18b48bf77f99\" (UID: \"5a1e781a-b4bc-4e84-85ef-18b48bf77f99\") " Nov 03 01:31:19.874957 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:19.874879 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a1e781a-b4bc-4e84-85ef-18b48bf77f99-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a1e781a-b4bc-4e84-85ef-18b48bf77f99-istiod-ca-cert\") pod \"5a1e781a-b4bc-4e84-85ef-18b48bf77f99\" (UID: \"5a1e781a-b4bc-4e84-85ef-18b48bf77f99\") " Nov 03 01:31:19.875076 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:19.874992 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5a1e781a-b4bc-4e84-85ef-18b48bf77f99-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a1e781a-b4bc-4e84-85ef-18b48bf77f99-istio-podinfo\") pod \"5a1e781a-b4bc-4e84-85ef-18b48bf77f99\" (UID: \"5a1e781a-b4bc-4e84-85ef-18b48bf77f99\") " Nov 03 01:31:19.875076 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:19.875029 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a1e781a-b4bc-4e84-85ef-18b48bf77f99-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a1e781a-b4bc-4e84-85ef-18b48bf77f99-credential-socket\") pod \"5a1e781a-b4bc-4e84-85ef-18b48bf77f99\" (UID: \"5a1e781a-b4bc-4e84-85ef-18b48bf77f99\") " Nov 03 01:31:19.875185 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:19.875165 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a1e781a-b4bc-4e84-85ef-18b48bf77f99-kube-api-access-kcwwl\" (UniqueName: \"kubernetes.io/projected/5a1e781a-b4bc-4e84-85ef-18b48bf77f99-kube-api-access-kcwwl\") pod \"5a1e781a-b4bc-4e84-85ef-18b48bf77f99\" (UID: \"5a1e781a-b4bc-4e84-85ef-18b48bf77f99\") " Nov 03 01:31:19.875298 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:19.875194 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a1e781a-b4bc-4e84-85ef-18b48bf77f99-workload-certs" pod "5a1e781a-b4bc-4e84-85ef-18b48bf77f99" (UID: "5a1e781a-b4bc-4e84-85ef-18b48bf77f99"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:31:19.875298 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:19.875211 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a1e781a-b4bc-4e84-85ef-18b48bf77f99-istio-token\" (UniqueName: \"kubernetes.io/projected/5a1e781a-b4bc-4e84-85ef-18b48bf77f99-istio-token\") pod \"5a1e781a-b4bc-4e84-85ef-18b48bf77f99\" (UID: \"5a1e781a-b4bc-4e84-85ef-18b48bf77f99\") " Nov 03 01:31:19.875521 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:19.875436 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a1e781a-b4bc-4e84-85ef-18b48bf77f99-istio-data" pod "5a1e781a-b4bc-4e84-85ef-18b48bf77f99" (UID: "5a1e781a-b4bc-4e84-85ef-18b48bf77f99"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:31:19.875521 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:19.875463 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a1e781a-b4bc-4e84-85ef-18b48bf77f99-istiod-ca-cert" pod "5a1e781a-b4bc-4e84-85ef-18b48bf77f99" (UID: "5a1e781a-b4bc-4e84-85ef-18b48bf77f99"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:31:19.875665 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:19.875650 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a1e781a-b4bc-4e84-85ef-18b48bf77f99-credential-socket" pod "5a1e781a-b4bc-4e84-85ef-18b48bf77f99" (UID: "5a1e781a-b4bc-4e84-85ef-18b48bf77f99"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:31:19.875964 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:19.875937 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a1e781a-b4bc-4e84-85ef-18b48bf77f99-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a1e781a-b4bc-4e84-85ef-18b48bf77f99-workload-socket\") pod \"5a1e781a-b4bc-4e84-85ef-18b48bf77f99\" (UID: \"5a1e781a-b4bc-4e84-85ef-18b48bf77f99\") " Nov 03 01:31:19.876351 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:19.876326 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a1e781a-b4bc-4e84-85ef-18b48bf77f99-workload-socket" pod "5a1e781a-b4bc-4e84-85ef-18b48bf77f99" (UID: "5a1e781a-b4bc-4e84-85ef-18b48bf77f99"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:31:19.877308 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:19.877285 4421 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a1e781a-b4bc-4e84-85ef-18b48bf77f99-istiod-ca-cert\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:31:19.877472 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:19.877456 4421 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a1e781a-b4bc-4e84-85ef-18b48bf77f99-credential-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:31:19.877567 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:19.877555 4421 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a1e781a-b4bc-4e84-85ef-18b48bf77f99-workload-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:31:19.877659 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:19.877647 4421 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a1e781a-b4bc-4e84-85ef-18b48bf77f99-istio-data\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:31:19.877756 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:19.877745 4421 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a1e781a-b4bc-4e84-85ef-18b48bf77f99-workload-certs\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:31:19.878435 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:19.878397 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a1e781a-b4bc-4e84-85ef-18b48bf77f99-istio-envoy" pod "5a1e781a-b4bc-4e84-85ef-18b48bf77f99" (UID: "5a1e781a-b4bc-4e84-85ef-18b48bf77f99"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:31:19.879820 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:19.879738 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a1e781a-b4bc-4e84-85ef-18b48bf77f99-istio-token" pod "5a1e781a-b4bc-4e84-85ef-18b48bf77f99" (UID: "5a1e781a-b4bc-4e84-85ef-18b48bf77f99"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:31:19.879936 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:19.879833 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a1e781a-b4bc-4e84-85ef-18b48bf77f99-kube-api-access-kcwwl" pod "5a1e781a-b4bc-4e84-85ef-18b48bf77f99" (UID: "5a1e781a-b4bc-4e84-85ef-18b48bf77f99"). InnerVolumeSpecName "kube-api-access-kcwwl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:31:19.890798 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:19.890741 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5a1e781a-b4bc-4e84-85ef-18b48bf77f99-istio-podinfo" pod "5a1e781a-b4bc-4e84-85ef-18b48bf77f99" (UID: "5a1e781a-b4bc-4e84-85ef-18b48bf77f99"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:31:19.968750 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:19.968716 4421 patch_prober.go:29] interesting pod/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-gvmnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.86:15021/healthz/ready\": dial tcp 10.134.1.86:15021: connect: connection refused" start-of-body= Nov 03 01:31:19.968932 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:19.968771 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-gvmnj" podUID="812a530a-f597-4049-a40a-2dcfb8ba6c74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.86:15021/healthz/ready\": dial tcp 10.134.1.86:15021: connect: connection refused" Nov 03 01:31:19.979411 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:19.979383 4421 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a1e781a-b4bc-4e84-85ef-18b48bf77f99-istio-envoy\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:31:19.979411 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:19.979410 4421 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a1e781a-b4bc-4e84-85ef-18b48bf77f99-istio-podinfo\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:31:19.979671 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:19.979422 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kcwwl\" (UniqueName: \"kubernetes.io/projected/5a1e781a-b4bc-4e84-85ef-18b48bf77f99-kube-api-access-kcwwl\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:31:19.979671 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:19.979432 4421 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a1e781a-b4bc-4e84-85ef-18b48bf77f99-istio-token\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:31:19.982880 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:19.982853 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0d8360b-b4c3-483e-8b00-412893aee9a6" path="/var/lib/kubelet/pods/a0d8360b-b4c3-483e-8b00-412893aee9a6/volumes" Nov 03 01:31:20.564254 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:20.564223 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-mxvfl" Nov 03 01:31:20.564254 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:20.564244 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-mxvfl" event={"ID":"5a1e781a-b4bc-4e84-85ef-18b48bf77f99","Type":"ContainerDied","Data":"93248c0a06bc1359bef6ef5cac03f2d17106b351d924265d56b6919069ca06cf"} Nov 03 01:31:20.564775 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:20.564344 4421 scope.go:122] "RemoveContainer" containerID="0f6d27237cee407e3c0c756d4a58ba7518d85cbc28093f1f3b69d9ebf144cf95" Nov 03 01:31:20.590333 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:20.590296 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-mxvfl"] Nov 03 01:31:20.595315 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:20.595289 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-mxvfl"] Nov 03 01:31:20.973880 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:20.973721 4421 patch_prober.go:29] interesting pod/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-gvmnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.86:15021/healthz/ready\": dial tcp 10.134.1.86:15021: connect: connection refused" start-of-body= Nov 03 01:31:20.976010 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:20.975963 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-gvmnj" podUID="812a530a-f597-4049-a40a-2dcfb8ba6c74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.86:15021/healthz/ready\": dial tcp 10.134.1.86:15021: connect: connection refused" Nov 03 01:31:21.967892 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:21.967857 4421 patch_prober.go:29] interesting pod/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-gvmnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.86:15021/healthz/ready\": dial tcp 10.134.1.86:15021: connect: connection refused" start-of-body= Nov 03 01:31:21.968419 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:21.967915 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-gvmnj" podUID="812a530a-f597-4049-a40a-2dcfb8ba6c74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.86:15021/healthz/ready\": dial tcp 10.134.1.86:15021: connect: connection refused" Nov 03 01:31:21.983734 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:21.983699 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a1e781a-b4bc-4e84-85ef-18b48bf77f99" path="/var/lib/kubelet/pods/5a1e781a-b4bc-4e84-85ef-18b48bf77f99/volumes" Nov 03 01:31:22.972664 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:22.967699 4421 patch_prober.go:29] interesting pod/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-gvmnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.86:15021/healthz/ready\": dial tcp 10.134.1.86:15021: connect: connection refused" start-of-body= Nov 03 01:31:22.972664 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:22.967795 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-gvmnj" podUID="812a530a-f597-4049-a40a-2dcfb8ba6c74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.86:15021/healthz/ready\": dial tcp 10.134.1.86:15021: connect: connection refused" Nov 03 01:31:23.966761 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:23.966724 4421 patch_prober.go:29] interesting pod/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-gvmnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.86:15021/healthz/ready\": dial tcp 10.134.1.86:15021: connect: connection refused" start-of-body= Nov 03 01:31:23.966966 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:23.966789 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-gvmnj" podUID="812a530a-f597-4049-a40a-2dcfb8ba6c74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.86:15021/healthz/ready\": dial tcp 10.134.1.86:15021: connect: connection refused" Nov 03 01:31:24.872368 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:24.872246 4421 generic.go:356] "Generic (PLEG): container finished" podID="af8d7fc5-bed0-4ebf-be1c-44805efec25a" containerID="fa60a24944e6d148d83da74de731ae827243e458dcb289705fab872206721c09" exitCode=0 Nov 03 01:31:24.883724 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:24.873266 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-pm4c4" event={"ID":"af8d7fc5-bed0-4ebf-be1c-44805efec25a","Type":"ContainerDied","Data":"fa60a24944e6d148d83da74de731ae827243e458dcb289705fab872206721c09"} Nov 03 01:31:24.967200 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:24.967171 4421 patch_prober.go:29] interesting pod/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-gvmnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.86:15021/healthz/ready\": dial tcp 10.134.1.86:15021: connect: connection refused" start-of-body= Nov 03 01:31:24.967374 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:24.967237 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-gvmnj" podUID="812a530a-f597-4049-a40a-2dcfb8ba6c74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.86:15021/healthz/ready\": dial tcp 10.134.1.86:15021: connect: connection refused" Nov 03 01:31:25.003644 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:25.003612 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-pm4c4" Nov 03 01:31:25.154811 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:25.151021 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af8d7fc5-bed0-4ebf-be1c-44805efec25a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af8d7fc5-bed0-4ebf-be1c-44805efec25a-credential-socket\") pod \"af8d7fc5-bed0-4ebf-be1c-44805efec25a\" (UID: \"af8d7fc5-bed0-4ebf-be1c-44805efec25a\") " Nov 03 01:31:25.155179 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:25.155147 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af8d7fc5-bed0-4ebf-be1c-44805efec25a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af8d7fc5-bed0-4ebf-be1c-44805efec25a-workload-socket\") pod \"af8d7fc5-bed0-4ebf-be1c-44805efec25a\" (UID: \"af8d7fc5-bed0-4ebf-be1c-44805efec25a\") " Nov 03 01:31:25.155555 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:25.155535 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/af8d7fc5-bed0-4ebf-be1c-44805efec25a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af8d7fc5-bed0-4ebf-be1c-44805efec25a-istio-podinfo\") pod \"af8d7fc5-bed0-4ebf-be1c-44805efec25a\" (UID: \"af8d7fc5-bed0-4ebf-be1c-44805efec25a\") " Nov 03 01:31:25.155806 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:25.155784 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af8d7fc5-bed0-4ebf-be1c-44805efec25a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af8d7fc5-bed0-4ebf-be1c-44805efec25a-istio-envoy\") pod \"af8d7fc5-bed0-4ebf-be1c-44805efec25a\" (UID: \"af8d7fc5-bed0-4ebf-be1c-44805efec25a\") " Nov 03 01:31:25.156043 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:25.156014 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af8d7fc5-bed0-4ebf-be1c-44805efec25a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af8d7fc5-bed0-4ebf-be1c-44805efec25a-istio-data\") pod \"af8d7fc5-bed0-4ebf-be1c-44805efec25a\" (UID: \"af8d7fc5-bed0-4ebf-be1c-44805efec25a\") " Nov 03 01:31:25.156184 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:25.156098 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af8d7fc5-bed0-4ebf-be1c-44805efec25a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af8d7fc5-bed0-4ebf-be1c-44805efec25a-workload-certs\") pod \"af8d7fc5-bed0-4ebf-be1c-44805efec25a\" (UID: \"af8d7fc5-bed0-4ebf-be1c-44805efec25a\") " Nov 03 01:31:25.156252 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:25.156208 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af8d7fc5-bed0-4ebf-be1c-44805efec25a-kube-api-access-wfbdl\" (UniqueName: \"kubernetes.io/projected/af8d7fc5-bed0-4ebf-be1c-44805efec25a-kube-api-access-wfbdl\") pod \"af8d7fc5-bed0-4ebf-be1c-44805efec25a\" (UID: \"af8d7fc5-bed0-4ebf-be1c-44805efec25a\") " Nov 03 01:31:25.156252 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:25.156249 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af8d7fc5-bed0-4ebf-be1c-44805efec25a-istio-token\" (UniqueName: \"kubernetes.io/projected/af8d7fc5-bed0-4ebf-be1c-44805efec25a-istio-token\") pod \"af8d7fc5-bed0-4ebf-be1c-44805efec25a\" (UID: \"af8d7fc5-bed0-4ebf-be1c-44805efec25a\") " Nov 03 01:31:25.156446 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:25.156327 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af8d7fc5-bed0-4ebf-be1c-44805efec25a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af8d7fc5-bed0-4ebf-be1c-44805efec25a-istiod-ca-cert\") pod \"af8d7fc5-bed0-4ebf-be1c-44805efec25a\" (UID: \"af8d7fc5-bed0-4ebf-be1c-44805efec25a\") " Nov 03 01:31:25.158572 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:25.158541 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af8d7fc5-bed0-4ebf-be1c-44805efec25a-credential-socket" pod "af8d7fc5-bed0-4ebf-be1c-44805efec25a" (UID: "af8d7fc5-bed0-4ebf-be1c-44805efec25a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:31:25.160283 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:25.160249 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af8d7fc5-bed0-4ebf-be1c-44805efec25a-istio-data" pod "af8d7fc5-bed0-4ebf-be1c-44805efec25a" (UID: "af8d7fc5-bed0-4ebf-be1c-44805efec25a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:31:25.160394 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:25.160331 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af8d7fc5-bed0-4ebf-be1c-44805efec25a-workload-socket" pod "af8d7fc5-bed0-4ebf-be1c-44805efec25a" (UID: "af8d7fc5-bed0-4ebf-be1c-44805efec25a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:31:25.162243 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:25.162216 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af8d7fc5-bed0-4ebf-be1c-44805efec25a-workload-certs" pod "af8d7fc5-bed0-4ebf-be1c-44805efec25a" (UID: "af8d7fc5-bed0-4ebf-be1c-44805efec25a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:31:25.162748 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:25.162722 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/af8d7fc5-bed0-4ebf-be1c-44805efec25a-istio-podinfo" pod "af8d7fc5-bed0-4ebf-be1c-44805efec25a" (UID: "af8d7fc5-bed0-4ebf-be1c-44805efec25a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:31:25.162841 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:25.162744 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af8d7fc5-bed0-4ebf-be1c-44805efec25a-kube-api-access-wfbdl" pod "af8d7fc5-bed0-4ebf-be1c-44805efec25a" (UID: "af8d7fc5-bed0-4ebf-be1c-44805efec25a"). InnerVolumeSpecName "kube-api-access-wfbdl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:31:25.162841 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:25.162767 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af8d7fc5-bed0-4ebf-be1c-44805efec25a-istio-token" pod "af8d7fc5-bed0-4ebf-be1c-44805efec25a" (UID: "af8d7fc5-bed0-4ebf-be1c-44805efec25a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:31:25.163479 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:25.163454 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af8d7fc5-bed0-4ebf-be1c-44805efec25a-istiod-ca-cert" pod "af8d7fc5-bed0-4ebf-be1c-44805efec25a" (UID: "af8d7fc5-bed0-4ebf-be1c-44805efec25a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:31:25.164024 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:25.164005 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af8d7fc5-bed0-4ebf-be1c-44805efec25a-istio-envoy" pod "af8d7fc5-bed0-4ebf-be1c-44805efec25a" (UID: "af8d7fc5-bed0-4ebf-be1c-44805efec25a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:31:25.260199 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:25.260161 4421 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af8d7fc5-bed0-4ebf-be1c-44805efec25a-credential-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:31:25.260199 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:25.260198 4421 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af8d7fc5-bed0-4ebf-be1c-44805efec25a-workload-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:31:25.260433 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:25.260212 4421 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af8d7fc5-bed0-4ebf-be1c-44805efec25a-istio-podinfo\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:31:25.260433 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:25.260222 4421 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af8d7fc5-bed0-4ebf-be1c-44805efec25a-istio-envoy\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:31:25.260433 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:25.260230 4421 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af8d7fc5-bed0-4ebf-be1c-44805efec25a-istio-data\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:31:25.260433 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:25.260239 4421 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af8d7fc5-bed0-4ebf-be1c-44805efec25a-workload-certs\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:31:25.260433 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:25.260248 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wfbdl\" (UniqueName: \"kubernetes.io/projected/af8d7fc5-bed0-4ebf-be1c-44805efec25a-kube-api-access-wfbdl\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:31:25.260433 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:25.260258 4421 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af8d7fc5-bed0-4ebf-be1c-44805efec25a-istio-token\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:31:25.260433 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:25.260266 4421 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af8d7fc5-bed0-4ebf-be1c-44805efec25a-istiod-ca-cert\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:31:25.882022 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:25.881991 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-pm4c4" Nov 03 01:31:25.882022 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:25.882000 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-pm4c4" event={"ID":"af8d7fc5-bed0-4ebf-be1c-44805efec25a","Type":"ContainerDied","Data":"1bf8296d1c117d5a3bb1ea9b6ee7984822ad1c9b060cfa145e4d868167c549e7"} Nov 03 01:31:25.882628 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:25.882092 4421 scope.go:122] "RemoveContainer" containerID="fa60a24944e6d148d83da74de731ae827243e458dcb289705fab872206721c09" Nov 03 01:31:25.910185 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:25.910160 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-pm4c4"] Nov 03 01:31:25.915835 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:25.915812 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-pm4c4"] Nov 03 01:31:25.984699 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:25.984663 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af8d7fc5-bed0-4ebf-be1c-44805efec25a" path="/var/lib/kubelet/pods/af8d7fc5-bed0-4ebf-be1c-44805efec25a/volumes" Nov 03 01:31:25.985370 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:25.985345 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-gvmnj" Nov 03 01:31:25.985832 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:25.985810 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-gvmnj" Nov 03 01:31:25.986741 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:25.986720 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-gvmnj" Nov 03 01:31:29.023243 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:29.023196 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-048e6381-wwgl5"] Nov 03 01:31:29.028191 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:29.028160 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-048e6381-wwgl5"] Nov 03 01:31:30.065384 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:30.065348 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb88a728-a121-47b2-84a0-a5ef8ba91de6" path="/var/lib/kubelet/pods/fb88a728-a121-47b2-84a0-a5ef8ba91de6/volumes" Nov 03 01:31:38.022567 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:38.022522 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c67e94fe-m5ldg"] Nov 03 01:31:38.027047 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:38.027022 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c67e94fe-m5ldg"] Nov 03 01:31:39.984638 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:31:39.984601 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e1fd99f-0a01-4ae3-8d80-1788ae33e332" path="/var/lib/kubelet/pods/6e1fd99f-0a01-4ae3-8d80-1788ae33e332/volumes" Nov 03 01:31:44.684932 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:31:44.684885 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 01:32:05.152184 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:32:05.152146 4421 scope.go:122] "RemoveContainer" containerID="b31ea700cceb57b7be39745ae51066822042d8df5dcc05e34b55723af0e5a5a3" Nov 03 01:32:05.165558 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:32:05.165531 4421 scope.go:122] "RemoveContainer" containerID="443d16093546a0bf0378f12426effa19e7504067d5adeeaa4ead18d0e0e76459" Nov 03 01:32:05.176774 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:32:05.176750 4421 scope.go:122] "RemoveContainer" containerID="67c897df89aad8918d042dec6841eff2be6de9a96585fe2d0e69e07b16429332" Nov 03 01:32:05.191665 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:32:05.191642 4421 scope.go:122] "RemoveContainer" containerID="3bf31bc22bf58fa760a72288611e2065dad3745f00cb10885401226f9b1f2a44" Nov 03 01:32:14.675258 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:32:14.675222 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 01:32:44.672220 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:32:44.672137 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 01:32:55.029654 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:32:55.029617 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31ccefb4-n9r9z"] Nov 03 01:32:55.034123 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:32:55.034081 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31ccefb4-n9r9z"] Nov 03 01:32:55.219790 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:32:55.219756 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2af124a-x8h45"] Nov 03 01:32:55.220738 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:32:55.220714 4421 cpu_manager.go:414] "containerMap: removing container" podUID="5a1e781a-b4bc-4e84-85ef-18b48bf77f99" containerName="istio-proxy" Nov 03 01:32:55.220738 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:32:55.220737 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a1e781a-b4bc-4e84-85ef-18b48bf77f99" containerName="istio-proxy" Nov 03 01:32:55.220940 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:32:55.220763 4421 cpu_manager.go:414] "containerMap: removing container" podUID="af8d7fc5-bed0-4ebf-be1c-44805efec25a" containerName="istio-proxy" Nov 03 01:32:55.220940 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:32:55.220769 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af8d7fc5-bed0-4ebf-be1c-44805efec25a" containerName="istio-proxy" Nov 03 01:32:55.220940 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:32:55.220789 4421 cpu_manager.go:414] "containerMap: removing container" podUID="6bb86426-5f37-4afc-9d67-83399abe9862" containerName="cleanup" Nov 03 01:32:55.220940 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:32:55.220797 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bb86426-5f37-4afc-9d67-83399abe9862" containerName="cleanup" Nov 03 01:32:55.221185 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:32:55.220973 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a1e781a-b4bc-4e84-85ef-18b48bf77f99" containerName="istio-proxy" Nov 03 01:32:55.221185 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:32:55.220987 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="af8d7fc5-bed0-4ebf-be1c-44805efec25a" containerName="istio-proxy" Nov 03 01:32:55.225612 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:32:55.225589 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2af124a-x8h45" Nov 03 01:32:55.229819 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:32:55.229792 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:32:55.230209 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:32:55.230169 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6q4xp\"" Nov 03 01:32:55.234212 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:32:55.234186 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2af124a-x8h45"] Nov 03 01:32:55.296806 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:32:55.296703 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e663e56-bfe1-48b5-bc2d-b1b56b20595a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c2af124a-x8h45\" (UID: \"3e663e56-bfe1-48b5-bc2d-b1b56b20595a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2af124a-x8h45" Nov 03 01:32:55.296998 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:32:55.296843 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjvnx\" (UniqueName: \"kubernetes.io/projected/3e663e56-bfe1-48b5-bc2d-b1b56b20595a-kube-api-access-rjvnx\") pod \"maas-api-revoke-sub-c2af124a-x8h45\" (UID: \"3e663e56-bfe1-48b5-bc2d-b1b56b20595a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2af124a-x8h45" Nov 03 01:32:55.297111 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:32:55.297095 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e663e56-bfe1-48b5-bc2d-b1b56b20595a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c2af124a-x8h45\" (UID: \"3e663e56-bfe1-48b5-bc2d-b1b56b20595a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2af124a-x8h45" Nov 03 01:32:55.399005 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:32:55.398965 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e663e56-bfe1-48b5-bc2d-b1b56b20595a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c2af124a-x8h45\" (UID: \"3e663e56-bfe1-48b5-bc2d-b1b56b20595a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2af124a-x8h45" Nov 03 01:32:55.399246 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:32:55.399147 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e663e56-bfe1-48b5-bc2d-b1b56b20595a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c2af124a-x8h45\" (UID: \"3e663e56-bfe1-48b5-bc2d-b1b56b20595a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2af124a-x8h45" Nov 03 01:32:55.399246 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:32:55.399201 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjvnx\" (UniqueName: \"kubernetes.io/projected/3e663e56-bfe1-48b5-bc2d-b1b56b20595a-kube-api-access-rjvnx\") pod \"maas-api-revoke-sub-c2af124a-x8h45\" (UID: \"3e663e56-bfe1-48b5-bc2d-b1b56b20595a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2af124a-x8h45" Nov 03 01:32:55.400021 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:32:55.399997 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e663e56-bfe1-48b5-bc2d-b1b56b20595a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c2af124a-x8h45\" (UID: \"3e663e56-bfe1-48b5-bc2d-b1b56b20595a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2af124a-x8h45" Nov 03 01:32:55.407753 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:32:55.407730 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjvnx\" (UniqueName: \"kubernetes.io/projected/3e663e56-bfe1-48b5-bc2d-b1b56b20595a-kube-api-access-rjvnx\") pod \"maas-api-revoke-sub-c2af124a-x8h45\" (UID: \"3e663e56-bfe1-48b5-bc2d-b1b56b20595a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2af124a-x8h45" Nov 03 01:32:55.407873 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:32:55.407730 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e663e56-bfe1-48b5-bc2d-b1b56b20595a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c2af124a-x8h45\" (UID: \"3e663e56-bfe1-48b5-bc2d-b1b56b20595a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2af124a-x8h45" Nov 03 01:32:55.560300 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:32:55.560191 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2af124a-x8h45" Nov 03 01:32:55.928295 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:32:55.928247 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2af124a-x8h45"] Nov 03 01:32:55.929070 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:32:55.929031 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e663e56_bfe1_48b5_bc2d_b1b56b20595a.slice/crio-2ceed6b9f0be4f45561f298e55fa69847a6e53da36ca174d019213f9a6965743 WatchSource:0}: Error finding container 2ceed6b9f0be4f45561f298e55fa69847a6e53da36ca174d019213f9a6965743: Status 404 returned error can't find the container with id 2ceed6b9f0be4f45561f298e55fa69847a6e53da36ca174d019213f9a6965743 Nov 03 01:32:55.986006 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:32:55.985968 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6bf7762e-6620-4f61-bd57-a988ed382691" path="/var/lib/kubelet/pods/6bf7762e-6620-4f61-bd57-a988ed382691/volumes" Nov 03 01:32:56.267072 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:32:56.267040 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2af124a-x8h45" event={"ID":"3e663e56-bfe1-48b5-bc2d-b1b56b20595a","Type":"ContainerStarted","Data":"2ceed6b9f0be4f45561f298e55fa69847a6e53da36ca174d019213f9a6965743"} Nov 03 01:32:57.277443 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:32:57.277347 4421 generic.go:356] "Generic (PLEG): container finished" podID="3e663e56-bfe1-48b5-bc2d-b1b56b20595a" containerID="8c6f903b6fcefb1ad9f4b4df0760344808c50204fb9ec5f1af0c28e28fc483e5" exitCode=0 Nov 03 01:32:57.277443 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:32:57.277425 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2af124a-x8h45" event={"ID":"3e663e56-bfe1-48b5-bc2d-b1b56b20595a","Type":"ContainerDied","Data":"8c6f903b6fcefb1ad9f4b4df0760344808c50204fb9ec5f1af0c28e28fc483e5"} Nov 03 01:32:58.476201 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:32:58.476171 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2af124a-x8h45" Nov 03 01:32:58.605794 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:32:58.605701 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e663e56-bfe1-48b5-bc2d-b1b56b20595a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e663e56-bfe1-48b5-bc2d-b1b56b20595a-maas-api-openshift-ca\") pod \"3e663e56-bfe1-48b5-bc2d-b1b56b20595a\" (UID: \"3e663e56-bfe1-48b5-bc2d-b1b56b20595a\") " Nov 03 01:32:58.605956 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:32:58.605847 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3e663e56-bfe1-48b5-bc2d-b1b56b20595a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e663e56-bfe1-48b5-bc2d-b1b56b20595a-maas-api-xks-ca\") pod \"3e663e56-bfe1-48b5-bc2d-b1b56b20595a\" (UID: \"3e663e56-bfe1-48b5-bc2d-b1b56b20595a\") " Nov 03 01:32:58.605956 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:32:58.605943 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e663e56-bfe1-48b5-bc2d-b1b56b20595a-kube-api-access-rjvnx\" (UniqueName: \"kubernetes.io/projected/3e663e56-bfe1-48b5-bc2d-b1b56b20595a-kube-api-access-rjvnx\") pod \"3e663e56-bfe1-48b5-bc2d-b1b56b20595a\" (UID: \"3e663e56-bfe1-48b5-bc2d-b1b56b20595a\") " Nov 03 01:32:58.606104 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:32:58.606076 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e663e56-bfe1-48b5-bc2d-b1b56b20595a-maas-api-openshift-ca" pod "3e663e56-bfe1-48b5-bc2d-b1b56b20595a" (UID: "3e663e56-bfe1-48b5-bc2d-b1b56b20595a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:32:58.606826 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:32:58.606800 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e663e56-bfe1-48b5-bc2d-b1b56b20595a-maas-api-openshift-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:32:58.609096 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:32:58.609068 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e663e56-bfe1-48b5-bc2d-b1b56b20595a-kube-api-access-rjvnx" pod "3e663e56-bfe1-48b5-bc2d-b1b56b20595a" (UID: "3e663e56-bfe1-48b5-bc2d-b1b56b20595a"). InnerVolumeSpecName "kube-api-access-rjvnx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:32:58.609246 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:32:58.609077 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e663e56-bfe1-48b5-bc2d-b1b56b20595a-maas-api-xks-ca" pod "3e663e56-bfe1-48b5-bc2d-b1b56b20595a" (UID: "3e663e56-bfe1-48b5-bc2d-b1b56b20595a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:32:58.708231 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:32:58.708199 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e663e56-bfe1-48b5-bc2d-b1b56b20595a-maas-api-xks-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:32:58.708231 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:32:58.708229 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rjvnx\" (UniqueName: \"kubernetes.io/projected/3e663e56-bfe1-48b5-bc2d-b1b56b20595a-kube-api-access-rjvnx\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:32:59.297116 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:32:59.297090 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2af124a-x8h45" Nov 03 01:32:59.297337 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:32:59.297119 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2af124a-x8h45" event={"ID":"3e663e56-bfe1-48b5-bc2d-b1b56b20595a","Type":"ContainerDied","Data":"2ceed6b9f0be4f45561f298e55fa69847a6e53da36ca174d019213f9a6965743"} Nov 03 01:32:59.297337 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:32:59.297166 4421 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2ceed6b9f0be4f45561f298e55fa69847a6e53da36ca174d019213f9a6965743" Nov 03 01:33:05.305011 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:05.304971 4421 scope.go:122] "RemoveContainer" containerID="d2be94e4b4d1becf36ddae56b1a30f6c113cbaeb864532e9203eb964bd48da05" Nov 03 01:33:05.467994 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:05.467966 4421 scope.go:122] "RemoveContainer" containerID="d92b00f9bb0ec3513d94e150a12836f44e0fd2aadf5635980f4bf6f7b7117ec8" Nov 03 01:33:05.482524 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:05.482478 4421 scope.go:122] "RemoveContainer" containerID="dbd348316291fbaa1f21a8d30694af3833bd10a53501410d0953973375476206" Nov 03 01:33:06.582332 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:06.582295 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b3fb80d-cvnnn"] Nov 03 01:33:06.583213 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:06.583195 4421 cpu_manager.go:414] "containerMap: removing container" podUID="3e663e56-bfe1-48b5-bc2d-b1b56b20595a" containerName="revoke-keys" Nov 03 01:33:06.583263 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:06.583214 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e663e56-bfe1-48b5-bc2d-b1b56b20595a" containerName="revoke-keys" Nov 03 01:33:06.583393 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:06.583382 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e663e56-bfe1-48b5-bc2d-b1b56b20595a" containerName="revoke-keys" Nov 03 01:33:06.588973 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:06.588948 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b3fb80d-cvnnn" Nov 03 01:33:06.592292 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:06.592255 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:33:06.593709 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:06.593680 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6q4xp\"" Nov 03 01:33:06.594299 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:06.594264 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b3fb80d-cvnnn"] Nov 03 01:33:06.648666 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:06.648612 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1da4b72f-2094-43a3-b99a-1da5849d1082-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b3fb80d-cvnnn\" (UID: \"1da4b72f-2094-43a3-b99a-1da5849d1082\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b3fb80d-cvnnn" Nov 03 01:33:06.649053 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:06.649031 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g494j\" (UniqueName: \"kubernetes.io/projected/1da4b72f-2094-43a3-b99a-1da5849d1082-kube-api-access-g494j\") pod \"maas-api-revoke-sub-0b3fb80d-cvnnn\" (UID: \"1da4b72f-2094-43a3-b99a-1da5849d1082\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b3fb80d-cvnnn" Nov 03 01:33:06.649165 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:06.649143 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1da4b72f-2094-43a3-b99a-1da5849d1082-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b3fb80d-cvnnn\" (UID: \"1da4b72f-2094-43a3-b99a-1da5849d1082\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b3fb80d-cvnnn" Nov 03 01:33:06.770300 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:06.770242 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g494j\" (UniqueName: \"kubernetes.io/projected/1da4b72f-2094-43a3-b99a-1da5849d1082-kube-api-access-g494j\") pod \"maas-api-revoke-sub-0b3fb80d-cvnnn\" (UID: \"1da4b72f-2094-43a3-b99a-1da5849d1082\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b3fb80d-cvnnn" Nov 03 01:33:06.770627 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:06.770603 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1da4b72f-2094-43a3-b99a-1da5849d1082-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b3fb80d-cvnnn\" (UID: \"1da4b72f-2094-43a3-b99a-1da5849d1082\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b3fb80d-cvnnn" Nov 03 01:33:06.771039 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:06.771009 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1da4b72f-2094-43a3-b99a-1da5849d1082-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b3fb80d-cvnnn\" (UID: \"1da4b72f-2094-43a3-b99a-1da5849d1082\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b3fb80d-cvnnn" Nov 03 01:33:06.772662 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:06.771832 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1da4b72f-2094-43a3-b99a-1da5849d1082-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b3fb80d-cvnnn\" (UID: \"1da4b72f-2094-43a3-b99a-1da5849d1082\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b3fb80d-cvnnn" Nov 03 01:33:06.788357 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:06.784461 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1da4b72f-2094-43a3-b99a-1da5849d1082-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b3fb80d-cvnnn\" (UID: \"1da4b72f-2094-43a3-b99a-1da5849d1082\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b3fb80d-cvnnn" Nov 03 01:33:06.788357 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:06.787792 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g494j\" (UniqueName: \"kubernetes.io/projected/1da4b72f-2094-43a3-b99a-1da5849d1082-kube-api-access-g494j\") pod \"maas-api-revoke-sub-0b3fb80d-cvnnn\" (UID: \"1da4b72f-2094-43a3-b99a-1da5849d1082\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b3fb80d-cvnnn" Nov 03 01:33:06.915324 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:06.915205 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b3fb80d-cvnnn" Nov 03 01:33:07.029563 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:07.029196 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f462289d-9ngww"] Nov 03 01:33:07.033773 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:07.033743 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f462289d-9ngww"] Nov 03 01:33:07.200429 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:07.200397 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b3fb80d-cvnnn"] Nov 03 01:33:07.200633 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:33:07.200614 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1da4b72f_2094_43a3_b99a_1da5849d1082.slice/crio-53bc3bc96f0021e3261b75b24305d0a4f843384177cabd198ad9dd7f6fee3ed7 WatchSource:0}: Error finding container 53bc3bc96f0021e3261b75b24305d0a4f843384177cabd198ad9dd7f6fee3ed7: Status 404 returned error can't find the container with id 53bc3bc96f0021e3261b75b24305d0a4f843384177cabd198ad9dd7f6fee3ed7 Nov 03 01:33:07.479694 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:07.479654 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b3fb80d-cvnnn" event={"ID":"1da4b72f-2094-43a3-b99a-1da5849d1082","Type":"ContainerStarted","Data":"53bc3bc96f0021e3261b75b24305d0a4f843384177cabd198ad9dd7f6fee3ed7"} Nov 03 01:33:07.984096 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:07.984060 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d06ea1eb-b168-4b5e-be57-cb4e5a45d911" path="/var/lib/kubelet/pods/d06ea1eb-b168-4b5e-be57-cb4e5a45d911/volumes" Nov 03 01:33:08.490863 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:08.490807 4421 generic.go:356] "Generic (PLEG): container finished" podID="1da4b72f-2094-43a3-b99a-1da5849d1082" containerID="50d7c3c76b192320124562f51823595cdbdee54e7d3e2e68426897b2695cb248" exitCode=0 Nov 03 01:33:08.491064 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:08.490941 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b3fb80d-cvnnn" event={"ID":"1da4b72f-2094-43a3-b99a-1da5849d1082","Type":"ContainerDied","Data":"50d7c3c76b192320124562f51823595cdbdee54e7d3e2e68426897b2695cb248"} Nov 03 01:33:09.680026 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:09.680002 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b3fb80d-cvnnn" Nov 03 01:33:09.783047 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:09.759224 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1da4b72f-2094-43a3-b99a-1da5849d1082-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1da4b72f-2094-43a3-b99a-1da5849d1082-maas-api-openshift-ca\") pod \"1da4b72f-2094-43a3-b99a-1da5849d1082\" (UID: \"1da4b72f-2094-43a3-b99a-1da5849d1082\") " Nov 03 01:33:09.783512 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:09.783475 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1da4b72f-2094-43a3-b99a-1da5849d1082-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1da4b72f-2094-43a3-b99a-1da5849d1082-maas-api-xks-ca\") pod \"1da4b72f-2094-43a3-b99a-1da5849d1082\" (UID: \"1da4b72f-2094-43a3-b99a-1da5849d1082\") " Nov 03 01:33:09.784068 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:09.784042 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1da4b72f-2094-43a3-b99a-1da5849d1082-kube-api-access-g494j\" (UniqueName: \"kubernetes.io/projected/1da4b72f-2094-43a3-b99a-1da5849d1082-kube-api-access-g494j\") pod \"1da4b72f-2094-43a3-b99a-1da5849d1082\" (UID: \"1da4b72f-2094-43a3-b99a-1da5849d1082\") " Nov 03 01:33:09.786469 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:09.776855 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1da4b72f-2094-43a3-b99a-1da5849d1082-maas-api-openshift-ca" pod "1da4b72f-2094-43a3-b99a-1da5849d1082" (UID: "1da4b72f-2094-43a3-b99a-1da5849d1082"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:33:09.787997 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:09.787975 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1da4b72f-2094-43a3-b99a-1da5849d1082-maas-api-openshift-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:33:09.795116 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:09.795065 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1da4b72f-2094-43a3-b99a-1da5849d1082-kube-api-access-g494j" pod "1da4b72f-2094-43a3-b99a-1da5849d1082" (UID: "1da4b72f-2094-43a3-b99a-1da5849d1082"). InnerVolumeSpecName "kube-api-access-g494j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:33:09.831817 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:09.831761 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1da4b72f-2094-43a3-b99a-1da5849d1082-maas-api-xks-ca" pod "1da4b72f-2094-43a3-b99a-1da5849d1082" (UID: "1da4b72f-2094-43a3-b99a-1da5849d1082"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:33:09.889960 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:09.889924 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g494j\" (UniqueName: \"kubernetes.io/projected/1da4b72f-2094-43a3-b99a-1da5849d1082-kube-api-access-g494j\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:33:09.889960 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:09.889955 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1da4b72f-2094-43a3-b99a-1da5849d1082-maas-api-xks-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:33:10.554460 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:10.554415 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b3fb80d-cvnnn" event={"ID":"1da4b72f-2094-43a3-b99a-1da5849d1082","Type":"ContainerDied","Data":"53bc3bc96f0021e3261b75b24305d0a4f843384177cabd198ad9dd7f6fee3ed7"} Nov 03 01:33:10.554844 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:10.554739 4421 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="53bc3bc96f0021e3261b75b24305d0a4f843384177cabd198ad9dd7f6fee3ed7" Nov 03 01:33:10.555003 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:10.554667 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b3fb80d-cvnnn" Nov 03 01:33:13.079342 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:13.079304 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-9v2m2"] Nov 03 01:33:13.080222 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:13.080198 4421 cpu_manager.go:414] "containerMap: removing container" podUID="1da4b72f-2094-43a3-b99a-1da5849d1082" containerName="revoke-keys" Nov 03 01:33:13.080222 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:13.080220 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1da4b72f-2094-43a3-b99a-1da5849d1082" containerName="revoke-keys" Nov 03 01:33:13.080488 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:13.080440 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="1da4b72f-2094-43a3-b99a-1da5849d1082" containerName="revoke-keys" Nov 03 01:33:13.085234 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:13.085206 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-9v2m2" Nov 03 01:33:13.095664 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:13.095628 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-9v2m2"] Nov 03 01:33:13.097620 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:13.097591 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-737bf2d8-openshift-default-dockercfg-7f9q9\"" Nov 03 01:33:13.179868 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:13.179830 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52a0f195-418d-42be-8222-b030c8f6ba55-istio-envoy\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-9v2m2\" (UID: \"52a0f195-418d-42be-8222-b030c8f6ba55\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-9v2m2" Nov 03 01:33:13.180054 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:13.179968 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52a0f195-418d-42be-8222-b030c8f6ba55-credential-socket\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-9v2m2\" (UID: \"52a0f195-418d-42be-8222-b030c8f6ba55\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-9v2m2" Nov 03 01:33:13.180054 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:13.180015 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52a0f195-418d-42be-8222-b030c8f6ba55-istio-data\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-9v2m2\" (UID: \"52a0f195-418d-42be-8222-b030c8f6ba55\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-9v2m2" Nov 03 01:33:13.180149 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:13.180091 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52a0f195-418d-42be-8222-b030c8f6ba55-istiod-ca-cert\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-9v2m2\" (UID: \"52a0f195-418d-42be-8222-b030c8f6ba55\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-9v2m2" Nov 03 01:33:13.180206 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:13.180191 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52a0f195-418d-42be-8222-b030c8f6ba55-istio-podinfo\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-9v2m2\" (UID: \"52a0f195-418d-42be-8222-b030c8f6ba55\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-9v2m2" Nov 03 01:33:13.180381 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:13.180368 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lp648\" (UniqueName: \"kubernetes.io/projected/52a0f195-418d-42be-8222-b030c8f6ba55-kube-api-access-lp648\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-9v2m2\" (UID: \"52a0f195-418d-42be-8222-b030c8f6ba55\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-9v2m2" Nov 03 01:33:13.180430 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:13.180400 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52a0f195-418d-42be-8222-b030c8f6ba55-istio-token\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-9v2m2\" (UID: \"52a0f195-418d-42be-8222-b030c8f6ba55\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-9v2m2" Nov 03 01:33:13.180467 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:13.180432 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52a0f195-418d-42be-8222-b030c8f6ba55-workload-socket\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-9v2m2\" (UID: \"52a0f195-418d-42be-8222-b030c8f6ba55\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-9v2m2" Nov 03 01:33:13.180545 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:13.180520 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52a0f195-418d-42be-8222-b030c8f6ba55-workload-certs\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-9v2m2\" (UID: \"52a0f195-418d-42be-8222-b030c8f6ba55\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-9v2m2" Nov 03 01:33:13.299392 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:13.297406 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52a0f195-418d-42be-8222-b030c8f6ba55-workload-socket\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-9v2m2\" (UID: \"52a0f195-418d-42be-8222-b030c8f6ba55\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-9v2m2" Nov 03 01:33:13.299392 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:13.297492 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52a0f195-418d-42be-8222-b030c8f6ba55-workload-certs\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-9v2m2\" (UID: \"52a0f195-418d-42be-8222-b030c8f6ba55\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-9v2m2" Nov 03 01:33:13.299392 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:13.297531 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52a0f195-418d-42be-8222-b030c8f6ba55-istio-envoy\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-9v2m2\" (UID: \"52a0f195-418d-42be-8222-b030c8f6ba55\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-9v2m2" Nov 03 01:33:13.299392 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:13.297675 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52a0f195-418d-42be-8222-b030c8f6ba55-credential-socket\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-9v2m2\" (UID: \"52a0f195-418d-42be-8222-b030c8f6ba55\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-9v2m2" Nov 03 01:33:13.299392 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:13.297754 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52a0f195-418d-42be-8222-b030c8f6ba55-istio-data\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-9v2m2\" (UID: \"52a0f195-418d-42be-8222-b030c8f6ba55\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-9v2m2" Nov 03 01:33:13.299392 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:13.297804 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52a0f195-418d-42be-8222-b030c8f6ba55-istiod-ca-cert\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-9v2m2\" (UID: \"52a0f195-418d-42be-8222-b030c8f6ba55\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-9v2m2" Nov 03 01:33:13.299392 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:13.297904 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52a0f195-418d-42be-8222-b030c8f6ba55-istio-podinfo\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-9v2m2\" (UID: \"52a0f195-418d-42be-8222-b030c8f6ba55\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-9v2m2" Nov 03 01:33:13.299392 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:13.298116 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lp648\" (UniqueName: \"kubernetes.io/projected/52a0f195-418d-42be-8222-b030c8f6ba55-kube-api-access-lp648\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-9v2m2\" (UID: \"52a0f195-418d-42be-8222-b030c8f6ba55\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-9v2m2" Nov 03 01:33:13.299392 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:13.298162 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52a0f195-418d-42be-8222-b030c8f6ba55-istio-token\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-9v2m2\" (UID: \"52a0f195-418d-42be-8222-b030c8f6ba55\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-9v2m2" Nov 03 01:33:13.310306 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:13.310066 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52a0f195-418d-42be-8222-b030c8f6ba55-credential-socket\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-9v2m2\" (UID: \"52a0f195-418d-42be-8222-b030c8f6ba55\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-9v2m2" Nov 03 01:33:13.310497 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:13.310367 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52a0f195-418d-42be-8222-b030c8f6ba55-workload-socket\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-9v2m2\" (UID: \"52a0f195-418d-42be-8222-b030c8f6ba55\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-9v2m2" Nov 03 01:33:13.314303 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:13.310621 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52a0f195-418d-42be-8222-b030c8f6ba55-workload-certs\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-9v2m2\" (UID: \"52a0f195-418d-42be-8222-b030c8f6ba55\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-9v2m2" Nov 03 01:33:13.314303 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:13.312093 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52a0f195-418d-42be-8222-b030c8f6ba55-istiod-ca-cert\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-9v2m2\" (UID: \"52a0f195-418d-42be-8222-b030c8f6ba55\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-9v2m2" Nov 03 01:33:13.314303 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:13.312361 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52a0f195-418d-42be-8222-b030c8f6ba55-istio-data\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-9v2m2\" (UID: \"52a0f195-418d-42be-8222-b030c8f6ba55\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-9v2m2" Nov 03 01:33:13.344928 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:13.344826 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52a0f195-418d-42be-8222-b030c8f6ba55-istio-podinfo\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-9v2m2\" (UID: \"52a0f195-418d-42be-8222-b030c8f6ba55\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-9v2m2" Nov 03 01:33:13.355664 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:13.355628 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52a0f195-418d-42be-8222-b030c8f6ba55-istio-envoy\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-9v2m2\" (UID: \"52a0f195-418d-42be-8222-b030c8f6ba55\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-9v2m2" Nov 03 01:33:13.367297 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:13.366090 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52a0f195-418d-42be-8222-b030c8f6ba55-istio-token\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-9v2m2\" (UID: \"52a0f195-418d-42be-8222-b030c8f6ba55\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-9v2m2" Nov 03 01:33:13.407294 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:13.407120 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lp648\" (UniqueName: \"kubernetes.io/projected/52a0f195-418d-42be-8222-b030c8f6ba55-kube-api-access-lp648\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-9v2m2\" (UID: \"52a0f195-418d-42be-8222-b030c8f6ba55\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-9v2m2" Nov 03 01:33:13.426306 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:13.414598 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-9v2m2" Nov 03 01:33:14.054680 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:33:14.054635 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52a0f195_418d_42be_8222_b030c8f6ba55.slice/crio-1b0d1384754a12d1e6502fb7e13d4ead5ec340130f5aeb3c815d5f2873c49f28 WatchSource:0}: Error finding container 1b0d1384754a12d1e6502fb7e13d4ead5ec340130f5aeb3c815d5f2873c49f28: Status 404 returned error can't find the container with id 1b0d1384754a12d1e6502fb7e13d4ead5ec340130f5aeb3c815d5f2873c49f28 Nov 03 01:33:14.055020 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:14.054992 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-9v2m2"] Nov 03 01:33:14.058770 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:14.058723 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:33:14.058889 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:14.058822 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:33:14.058889 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:14.058867 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:33:14.685703 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:33:14.685644 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 01:33:14.702057 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:14.702004 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-9v2m2" event={"ID":"52a0f195-418d-42be-8222-b030c8f6ba55","Type":"ContainerStarted","Data":"41749612022ba7f45899b639631594c7f0c9a087212cf7b716fb6eddb81975d3"} Nov 03 01:33:14.702234 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:14.702083 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-9v2m2" event={"ID":"52a0f195-418d-42be-8222-b030c8f6ba55","Type":"ContainerStarted","Data":"1b0d1384754a12d1e6502fb7e13d4ead5ec340130f5aeb3c815d5f2873c49f28"} Nov 03 01:33:14.729656 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:14.729597 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-9v2m2" podStartSLOduration=1.729577643 podStartE2EDuration="1.729577643s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:33:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:33:14.722497846 +0000 UTC m=+3136.893500944" watchObservedRunningTime="2026-11-03 01:33:14.729577643 +0000 UTC m=+3136.900580739" Nov 03 01:33:15.415345 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:15.415302 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-9v2m2" Nov 03 01:33:15.424414 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:15.424381 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-9v2m2" Nov 03 01:33:15.710591 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:15.710495 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-9v2m2" Nov 03 01:33:15.711403 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:15.711382 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-9v2m2" Nov 03 01:33:18.187324 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:18.187291 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-737bf2d8-66d45589cc-n7m7l"] Nov 03 01:33:18.192096 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:18.192072 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-737bf2d8-66d45589cc-n7m7l" Nov 03 01:33:18.198595 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:18.198539 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-737bf2d8-dockercfg-dz8hq\"" Nov 03 01:33:18.203191 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:18.203163 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-737bf2d8\"" Nov 03 01:33:18.206749 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:18.206333 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-737bf2d8-66d45589cc-n7m7l"] Nov 03 01:33:18.221331 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:18.221254 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-737bf2d8-6ff985b587-vb7qc"] Nov 03 01:33:18.239765 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:18.239732 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-737bf2d8-6ff985b587-vb7qc" Nov 03 01:33:18.240080 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:18.240055 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-737bf2d8-6ff985b587-vb7qc"] Nov 03 01:33:18.291572 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:18.291536 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txx8g\" (UniqueName: \"kubernetes.io/projected/e6eaa842-3947-496e-8b82-01a51ef80298-kube-api-access-txx8g\") pod \"payload-processing-e2e-mt-737bf2d8-66d45589cc-n7m7l\" (UID: \"e6eaa842-3947-496e-8b82-01a51ef80298\") " pod="openshift-ingress/payload-processing-e2e-mt-737bf2d8-66d45589cc-n7m7l" Nov 03 01:33:18.291927 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:18.291903 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6eaa842-3947-496e-8b82-01a51ef80298-plugins-config-volume\") pod \"payload-processing-e2e-mt-737bf2d8-66d45589cc-n7m7l\" (UID: \"e6eaa842-3947-496e-8b82-01a51ef80298\") " pod="openshift-ingress/payload-processing-e2e-mt-737bf2d8-66d45589cc-n7m7l" Nov 03 01:33:18.397298 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:18.396667 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad4cba32-8044-4971-8e5d-bd5aa09cfef9-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-737bf2d8-6ff985b587-vb7qc\" (UID: \"ad4cba32-8044-4971-8e5d-bd5aa09cfef9\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-737bf2d8-6ff985b587-vb7qc" Nov 03 01:33:18.397298 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:18.396875 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6eaa842-3947-496e-8b82-01a51ef80298-plugins-config-volume\") pod \"payload-processing-e2e-mt-737bf2d8-66d45589cc-n7m7l\" (UID: \"e6eaa842-3947-496e-8b82-01a51ef80298\") " pod="openshift-ingress/payload-processing-e2e-mt-737bf2d8-66d45589cc-n7m7l" Nov 03 01:33:18.397298 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:18.396925 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7wx6\" (UniqueName: \"kubernetes.io/projected/ad4cba32-8044-4971-8e5d-bd5aa09cfef9-kube-api-access-z7wx6\") pod \"payload-pre-processing-e2e-mt-737bf2d8-6ff985b587-vb7qc\" (UID: \"ad4cba32-8044-4971-8e5d-bd5aa09cfef9\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-737bf2d8-6ff985b587-vb7qc" Nov 03 01:33:18.397298 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:18.397017 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txx8g\" (UniqueName: \"kubernetes.io/projected/e6eaa842-3947-496e-8b82-01a51ef80298-kube-api-access-txx8g\") pod \"payload-processing-e2e-mt-737bf2d8-66d45589cc-n7m7l\" (UID: \"e6eaa842-3947-496e-8b82-01a51ef80298\") " pod="openshift-ingress/payload-processing-e2e-mt-737bf2d8-66d45589cc-n7m7l" Nov 03 01:33:18.401299 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:18.398920 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6eaa842-3947-496e-8b82-01a51ef80298-plugins-config-volume\") pod \"payload-processing-e2e-mt-737bf2d8-66d45589cc-n7m7l\" (UID: \"e6eaa842-3947-496e-8b82-01a51ef80298\") " pod="openshift-ingress/payload-processing-e2e-mt-737bf2d8-66d45589cc-n7m7l" Nov 03 01:33:18.465903 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:18.462018 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-txx8g\" (UniqueName: \"kubernetes.io/projected/e6eaa842-3947-496e-8b82-01a51ef80298-kube-api-access-txx8g\") pod \"payload-processing-e2e-mt-737bf2d8-66d45589cc-n7m7l\" (UID: \"e6eaa842-3947-496e-8b82-01a51ef80298\") " pod="openshift-ingress/payload-processing-e2e-mt-737bf2d8-66d45589cc-n7m7l" Nov 03 01:33:18.499302 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:18.498543 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad4cba32-8044-4971-8e5d-bd5aa09cfef9-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-737bf2d8-6ff985b587-vb7qc\" (UID: \"ad4cba32-8044-4971-8e5d-bd5aa09cfef9\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-737bf2d8-6ff985b587-vb7qc" Nov 03 01:33:18.499302 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:18.498753 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7wx6\" (UniqueName: \"kubernetes.io/projected/ad4cba32-8044-4971-8e5d-bd5aa09cfef9-kube-api-access-z7wx6\") pod \"payload-pre-processing-e2e-mt-737bf2d8-6ff985b587-vb7qc\" (UID: \"ad4cba32-8044-4971-8e5d-bd5aa09cfef9\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-737bf2d8-6ff985b587-vb7qc" Nov 03 01:33:18.503302 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:18.500160 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad4cba32-8044-4971-8e5d-bd5aa09cfef9-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-737bf2d8-6ff985b587-vb7qc\" (UID: \"ad4cba32-8044-4971-8e5d-bd5aa09cfef9\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-737bf2d8-6ff985b587-vb7qc" Nov 03 01:33:18.545495 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:18.541488 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7wx6\" (UniqueName: \"kubernetes.io/projected/ad4cba32-8044-4971-8e5d-bd5aa09cfef9-kube-api-access-z7wx6\") pod \"payload-pre-processing-e2e-mt-737bf2d8-6ff985b587-vb7qc\" (UID: \"ad4cba32-8044-4971-8e5d-bd5aa09cfef9\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-737bf2d8-6ff985b587-vb7qc" Nov 03 01:33:18.545495 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:18.542707 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-737bf2d8-66d45589cc-n7m7l" Nov 03 01:33:18.568295 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:18.559698 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-737bf2d8-6ff985b587-vb7qc" Nov 03 01:33:19.126832 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:19.126805 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-737bf2d8-66d45589cc-n7m7l"] Nov 03 01:33:19.128135 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:33:19.128106 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6eaa842_3947_496e_8b82_01a51ef80298.slice/crio-13e6c44acb61c4c3368cc80dc97d591cd9bbc9ae29e2b69a94ac3fe9d670f7e0 WatchSource:0}: Error finding container 13e6c44acb61c4c3368cc80dc97d591cd9bbc9ae29e2b69a94ac3fe9d670f7e0: Status 404 returned error can't find the container with id 13e6c44acb61c4c3368cc80dc97d591cd9bbc9ae29e2b69a94ac3fe9d670f7e0 Nov 03 01:33:19.129146 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:33:19.129118 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad4cba32_8044_4971_8e5d_bd5aa09cfef9.slice/crio-b89b1c70224f69aaf35868afb78e3f1cb8fb78d30c0cc425c1f6acecadec7ad2 WatchSource:0}: Error finding container b89b1c70224f69aaf35868afb78e3f1cb8fb78d30c0cc425c1f6acecadec7ad2: Status 404 returned error can't find the container with id b89b1c70224f69aaf35868afb78e3f1cb8fb78d30c0cc425c1f6acecadec7ad2 Nov 03 01:33:19.132465 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:19.132429 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-737bf2d8-6ff985b587-vb7qc"] Nov 03 01:33:19.798195 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:19.798137 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-737bf2d8-6ff985b587-vb7qc" event={"ID":"ad4cba32-8044-4971-8e5d-bd5aa09cfef9","Type":"ContainerStarted","Data":"cf0dfa4f7564237d56b92c3f5300f6463d63244794e2f19129826a2812b90332"} Nov 03 01:33:19.798892 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:19.798263 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-737bf2d8-6ff985b587-vb7qc" event={"ID":"ad4cba32-8044-4971-8e5d-bd5aa09cfef9","Type":"ContainerStarted","Data":"b89b1c70224f69aaf35868afb78e3f1cb8fb78d30c0cc425c1f6acecadec7ad2"} Nov 03 01:33:19.805312 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:19.799782 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-737bf2d8-6ff985b587-vb7qc" Nov 03 01:33:19.805312 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:19.802509 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-737bf2d8-66d45589cc-n7m7l" event={"ID":"e6eaa842-3947-496e-8b82-01a51ef80298","Type":"ContainerStarted","Data":"aa32a6b6804d03bf0d9e829eff973868a0841f0c54101f262a2dc5b6b1e1ec2e"} Nov 03 01:33:19.805312 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:19.802583 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-737bf2d8-66d45589cc-n7m7l" event={"ID":"e6eaa842-3947-496e-8b82-01a51ef80298","Type":"ContainerStarted","Data":"13e6c44acb61c4c3368cc80dc97d591cd9bbc9ae29e2b69a94ac3fe9d670f7e0"} Nov 03 01:33:19.805312 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:19.803705 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-737bf2d8-66d45589cc-n7m7l" Nov 03 01:33:19.828124 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:19.828065 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-737bf2d8-6ff985b587-vb7qc" podStartSLOduration=1.828045696 podStartE2EDuration="1.828045696s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:33:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:33:19.824590622 +0000 UTC m=+3141.995593720" watchObservedRunningTime="2026-11-03 01:33:19.828045696 +0000 UTC m=+3141.999048792" Nov 03 01:33:19.845860 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:19.845809 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-737bf2d8-66d45589cc-n7m7l" podStartSLOduration=1.845794068 podStartE2EDuration="1.845794068s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:33:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:33:19.84446528 +0000 UTC m=+3142.015468376" watchObservedRunningTime="2026-11-03 01:33:19.845794068 +0000 UTC m=+3142.016797186" Nov 03 01:33:23.688133 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:23.688101 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ee618c-7bc658cb5-2kgml"] Nov 03 01:33:28.694308 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:28.689413 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-ee618c-7bc658cb5-2kgml" podUID="157f33b5-87a2-4d03-8288-a8e0cc90de31" containerName="payload-processing" containerID="cri-o://dcd8edadc6ada4b6f7df7691368602f193a6de408eeaf39e65ff806f7aff509e" gracePeriod=25 Nov 03 01:33:28.925339 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:28.924380 4421 generic.go:356] "Generic (PLEG): container finished" podID="157f33b5-87a2-4d03-8288-a8e0cc90de31" containerID="dcd8edadc6ada4b6f7df7691368602f193a6de408eeaf39e65ff806f7aff509e" exitCode=0 Nov 03 01:33:28.925339 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:28.924450 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-ee618c-7bc658cb5-2kgml" event={"ID":"157f33b5-87a2-4d03-8288-a8e0cc90de31","Type":"ContainerDied","Data":"dcd8edadc6ada4b6f7df7691368602f193a6de408eeaf39e65ff806f7aff509e"} Nov 03 01:33:29.033112 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:29.033081 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-ee618c-7bc658cb5-2kgml" Nov 03 01:33:29.124329 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:29.124293 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/157f33b5-87a2-4d03-8288-a8e0cc90de31-kube-api-access-psx52\" (UniqueName: \"kubernetes.io/projected/157f33b5-87a2-4d03-8288-a8e0cc90de31-kube-api-access-psx52\") pod \"157f33b5-87a2-4d03-8288-a8e0cc90de31\" (UID: \"157f33b5-87a2-4d03-8288-a8e0cc90de31\") " Nov 03 01:33:29.125174 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:29.124533 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/157f33b5-87a2-4d03-8288-a8e0cc90de31-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/157f33b5-87a2-4d03-8288-a8e0cc90de31-plugins-config-volume\") pod \"157f33b5-87a2-4d03-8288-a8e0cc90de31\" (UID: \"157f33b5-87a2-4d03-8288-a8e0cc90de31\") " Nov 03 01:33:29.125174 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:29.124969 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/157f33b5-87a2-4d03-8288-a8e0cc90de31-plugins-config-volume" pod "157f33b5-87a2-4d03-8288-a8e0cc90de31" (UID: "157f33b5-87a2-4d03-8288-a8e0cc90de31"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:33:29.125465 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:29.125442 4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/157f33b5-87a2-4d03-8288-a8e0cc90de31-plugins-config-volume\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:33:29.127660 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:29.127629 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/157f33b5-87a2-4d03-8288-a8e0cc90de31-kube-api-access-psx52" pod "157f33b5-87a2-4d03-8288-a8e0cc90de31" (UID: "157f33b5-87a2-4d03-8288-a8e0cc90de31"). InnerVolumeSpecName "kube-api-access-psx52". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:33:29.305999 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:29.278212 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-psx52\" (UniqueName: \"kubernetes.io/projected/157f33b5-87a2-4d03-8288-a8e0cc90de31-kube-api-access-psx52\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:33:30.006840 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:30.006807 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-ee618c-7bc658cb5-2kgml" Nov 03 01:33:30.021548 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:30.021501 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-ee618c-7bc658cb5-2kgml" event={"ID":"157f33b5-87a2-4d03-8288-a8e0cc90de31","Type":"ContainerDied","Data":"07db8329cc3207078b1e161481ccd1135832982914900d3942504c92a16c4793"} Nov 03 01:33:30.021754 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:30.021560 4421 scope.go:122] "RemoveContainer" containerID="dcd8edadc6ada4b6f7df7691368602f193a6de408eeaf39e65ff806f7aff509e" Nov 03 01:33:30.035496 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:30.035420 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ee618c-7bc658cb5-2kgml"] Nov 03 01:33:30.040963 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:30.040936 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ee618c-7bc658cb5-2kgml"] Nov 03 01:33:31.863299 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:31.857451 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-737bf2d8-66d45589cc-n7m7l" Nov 03 01:33:31.863299 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:31.860187 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-737bf2d8-6ff985b587-vb7qc" Nov 03 01:33:32.692315 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:32.692256 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="157f33b5-87a2-4d03-8288-a8e0cc90de31" path="/var/lib/kubelet/pods/157f33b5-87a2-4d03-8288-a8e0cc90de31/volumes" Nov 03 01:33:37.068381 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:37.067650 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-vhjlx"] Nov 03 01:33:37.069314 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:37.069259 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-vhjlx" podUID="1e2345c1-2a99-4b03-b078-eebe1f198589" containerName="istio-proxy" containerID="cri-o://69702dff6eed0fd1f37433181868ea6b682951c24343e3e04dc3116e146c03c3" gracePeriod=30 Nov 03 01:33:37.125422 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:37.124715 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zgdl8"] Nov 03 01:33:37.125422 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:37.124961 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zgdl8" podUID="920eea17-797a-4c19-9f09-36f121782d2e" containerName="istio-proxy" containerID="cri-o://1cf56a802f20d20a896f34cee26bb9327a975f2779269509d11017a42b7c66c6" gracePeriod=30 Nov 03 01:33:37.270480 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:37.260100 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-gvmnj"] Nov 03 01:33:37.270480 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:37.260376 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-gvmnj" podUID="812a530a-f597-4049-a40a-2dcfb8ba6c74" containerName="istio-proxy" containerID="cri-o://b177a223eb4b7553bf143b9f7b2045e22a53266c156107de37060f64bb62bc28" gracePeriod=30 Nov 03 01:33:37.389186 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:37.388193 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-x8tb9"] Nov 03 01:33:37.389186 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:37.388478 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-x8tb9" podUID="79c7fff7-62df-44a2-aa3e-f24696d71b03" containerName="istio-proxy" containerID="cri-o://d74595c5bdaade18ab20fc49ed721cc7212db6df5232f69a0d362df3e9687ab6" gracePeriod=30 Nov 03 01:33:37.436112 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:37.436036 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-4tt4f"] Nov 03 01:33:37.438294 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:37.436712 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-4tt4f" podUID="b645b6f2-58be-45c4-9a6f-6785a1047bea" containerName="istio-proxy" containerID="cri-o://38f6c23c7753df1e2d6b4de8eabe27e5046425a24da317c37e9e24936a4adf82" gracePeriod=30 Nov 03 01:33:38.033150 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:38.033113 4421 patch_prober.go:29] interesting pod/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-x8tb9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 01:33:38.034075 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:38.033176 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-x8tb9" podUID="79c7fff7-62df-44a2-aa3e-f24696d71b03" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 01:33:40.989571 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:40.989533 4421 patch_prober.go:29] interesting pod/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-gvmnj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 01:33:40.990484 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:40.989600 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-gvmnj" podUID="812a530a-f597-4049-a40a-2dcfb8ba6c74" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 01:33:42.550055 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:42.550015 4421 generic.go:356] "Generic (PLEG): container finished" podID="920eea17-797a-4c19-9f09-36f121782d2e" containerID="1cf56a802f20d20a896f34cee26bb9327a975f2779269509d11017a42b7c66c6" exitCode=0 Nov 03 01:33:42.551204 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:42.550134 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zgdl8" event={"ID":"920eea17-797a-4c19-9f09-36f121782d2e","Type":"ContainerDied","Data":"1cf56a802f20d20a896f34cee26bb9327a975f2779269509d11017a42b7c66c6"} Nov 03 01:33:42.552703 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:42.552678 4421 generic.go:356] "Generic (PLEG): container finished" podID="1e2345c1-2a99-4b03-b078-eebe1f198589" containerID="69702dff6eed0fd1f37433181868ea6b682951c24343e3e04dc3116e146c03c3" exitCode=0 Nov 03 01:33:42.552810 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:42.552767 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-vhjlx" event={"ID":"1e2345c1-2a99-4b03-b078-eebe1f198589","Type":"ContainerDied","Data":"69702dff6eed0fd1f37433181868ea6b682951c24343e3e04dc3116e146c03c3"} Nov 03 01:33:42.552893 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:42.552827 4421 scope.go:122] "RemoveContainer" containerID="751bf328470f6f5ab404fb70bf67aa7c9c20ff91cd278e623b26099e33f00960" Nov 03 01:33:42.557360 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:42.556173 4421 generic.go:356] "Generic (PLEG): container finished" podID="812a530a-f597-4049-a40a-2dcfb8ba6c74" containerID="b177a223eb4b7553bf143b9f7b2045e22a53266c156107de37060f64bb62bc28" exitCode=0 Nov 03 01:33:42.557360 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:42.556215 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-gvmnj" event={"ID":"812a530a-f597-4049-a40a-2dcfb8ba6c74","Type":"ContainerDied","Data":"b177a223eb4b7553bf143b9f7b2045e22a53266c156107de37060f64bb62bc28"} Nov 03 01:33:43.119338 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.119244 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zgdl8" Nov 03 01:33:43.148802 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.148768 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-vhjlx" Nov 03 01:33:43.166803 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.166751 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-x8tb9" Nov 03 01:33:43.178798 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.178772 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-4tt4f" Nov 03 01:33:43.205728 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.205695 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-gvmnj" Nov 03 01:33:43.310873 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.310835 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/79c7fff7-62df-44a2-aa3e-f24696d71b03-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79c7fff7-62df-44a2-aa3e-f24696d71b03-istio-podinfo\") pod \"79c7fff7-62df-44a2-aa3e-f24696d71b03\" (UID: \"79c7fff7-62df-44a2-aa3e-f24696d71b03\") " Nov 03 01:33:43.311076 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.310926 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e2345c1-2a99-4b03-b078-eebe1f198589-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e2345c1-2a99-4b03-b078-eebe1f198589-workload-certs\") pod \"1e2345c1-2a99-4b03-b078-eebe1f198589\" (UID: \"1e2345c1-2a99-4b03-b078-eebe1f198589\") " Nov 03 01:33:43.311076 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.310965 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e2345c1-2a99-4b03-b078-eebe1f198589-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e2345c1-2a99-4b03-b078-eebe1f198589-workload-socket\") pod \"1e2345c1-2a99-4b03-b078-eebe1f198589\" (UID: \"1e2345c1-2a99-4b03-b078-eebe1f198589\") " Nov 03 01:33:43.311076 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.310995 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/920eea17-797a-4c19-9f09-36f121782d2e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/920eea17-797a-4c19-9f09-36f121782d2e-istio-envoy\") pod \"920eea17-797a-4c19-9f09-36f121782d2e\" (UID: \"920eea17-797a-4c19-9f09-36f121782d2e\") " Nov 03 01:33:43.311076 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.311031 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b645b6f2-58be-45c4-9a6f-6785a1047bea-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b645b6f2-58be-45c4-9a6f-6785a1047bea-workload-socket\") pod \"b645b6f2-58be-45c4-9a6f-6785a1047bea\" (UID: \"b645b6f2-58be-45c4-9a6f-6785a1047bea\") " Nov 03 01:33:43.347552 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.311079 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b645b6f2-58be-45c4-9a6f-6785a1047bea-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b645b6f2-58be-45c4-9a6f-6785a1047bea-istio-podinfo\") pod \"b645b6f2-58be-45c4-9a6f-6785a1047bea\" (UID: \"b645b6f2-58be-45c4-9a6f-6785a1047bea\") " Nov 03 01:33:43.347552 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.311110 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/812a530a-f597-4049-a40a-2dcfb8ba6c74-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/812a530a-f597-4049-a40a-2dcfb8ba6c74-istio-data\") pod \"812a530a-f597-4049-a40a-2dcfb8ba6c74\" (UID: \"812a530a-f597-4049-a40a-2dcfb8ba6c74\") " Nov 03 01:33:43.347552 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.311144 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79c7fff7-62df-44a2-aa3e-f24696d71b03-istio-token\" (UniqueName: \"kubernetes.io/projected/79c7fff7-62df-44a2-aa3e-f24696d71b03-istio-token\") pod \"79c7fff7-62df-44a2-aa3e-f24696d71b03\" (UID: \"79c7fff7-62df-44a2-aa3e-f24696d71b03\") " Nov 03 01:33:43.347552 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.311184 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79c7fff7-62df-44a2-aa3e-f24696d71b03-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79c7fff7-62df-44a2-aa3e-f24696d71b03-istio-envoy\") pod \"79c7fff7-62df-44a2-aa3e-f24696d71b03\" (UID: \"79c7fff7-62df-44a2-aa3e-f24696d71b03\") " Nov 03 01:33:43.347552 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.311221 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b645b6f2-58be-45c4-9a6f-6785a1047bea-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b645b6f2-58be-45c4-9a6f-6785a1047bea-istio-data\") pod \"b645b6f2-58be-45c4-9a6f-6785a1047bea\" (UID: \"b645b6f2-58be-45c4-9a6f-6785a1047bea\") " Nov 03 01:33:43.347552 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.311249 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79c7fff7-62df-44a2-aa3e-f24696d71b03-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79c7fff7-62df-44a2-aa3e-f24696d71b03-workload-certs\") pod \"79c7fff7-62df-44a2-aa3e-f24696d71b03\" (UID: \"79c7fff7-62df-44a2-aa3e-f24696d71b03\") " Nov 03 01:33:43.347552 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.311317 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/812a530a-f597-4049-a40a-2dcfb8ba6c74-istio-token\" (UniqueName: \"kubernetes.io/projected/812a530a-f597-4049-a40a-2dcfb8ba6c74-istio-token\") pod \"812a530a-f597-4049-a40a-2dcfb8ba6c74\" (UID: \"812a530a-f597-4049-a40a-2dcfb8ba6c74\") " Nov 03 01:33:43.347552 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.311366 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/920eea17-797a-4c19-9f09-36f121782d2e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/920eea17-797a-4c19-9f09-36f121782d2e-workload-certs\") pod \"920eea17-797a-4c19-9f09-36f121782d2e\" (UID: \"920eea17-797a-4c19-9f09-36f121782d2e\") " Nov 03 01:33:43.347552 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.311411 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79c7fff7-62df-44a2-aa3e-f24696d71b03-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79c7fff7-62df-44a2-aa3e-f24696d71b03-credential-socket\") pod \"79c7fff7-62df-44a2-aa3e-f24696d71b03\" (UID: \"79c7fff7-62df-44a2-aa3e-f24696d71b03\") " Nov 03 01:33:43.347552 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.311481 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/812a530a-f597-4049-a40a-2dcfb8ba6c74-kube-api-access-vhcqn\" (UniqueName: \"kubernetes.io/projected/812a530a-f597-4049-a40a-2dcfb8ba6c74-kube-api-access-vhcqn\") pod \"812a530a-f597-4049-a40a-2dcfb8ba6c74\" (UID: \"812a530a-f597-4049-a40a-2dcfb8ba6c74\") " Nov 03 01:33:43.347552 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.311518 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b645b6f2-58be-45c4-9a6f-6785a1047bea-kube-api-access-rvfzl\" (UniqueName: \"kubernetes.io/projected/b645b6f2-58be-45c4-9a6f-6785a1047bea-kube-api-access-rvfzl\") pod \"b645b6f2-58be-45c4-9a6f-6785a1047bea\" (UID: \"b645b6f2-58be-45c4-9a6f-6785a1047bea\") " Nov 03 01:33:43.347552 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.311548 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e2345c1-2a99-4b03-b078-eebe1f198589-istio-token\" (UniqueName: \"kubernetes.io/projected/1e2345c1-2a99-4b03-b078-eebe1f198589-istio-token\") pod \"1e2345c1-2a99-4b03-b078-eebe1f198589\" (UID: \"1e2345c1-2a99-4b03-b078-eebe1f198589\") " Nov 03 01:33:43.347552 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.311584 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/920eea17-797a-4c19-9f09-36f121782d2e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/920eea17-797a-4c19-9f09-36f121782d2e-istio-podinfo\") pod \"920eea17-797a-4c19-9f09-36f121782d2e\" (UID: \"920eea17-797a-4c19-9f09-36f121782d2e\") " Nov 03 01:33:43.347552 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.311613 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/920eea17-797a-4c19-9f09-36f121782d2e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/920eea17-797a-4c19-9f09-36f121782d2e-workload-socket\") pod \"920eea17-797a-4c19-9f09-36f121782d2e\" (UID: \"920eea17-797a-4c19-9f09-36f121782d2e\") " Nov 03 01:33:43.347552 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.311646 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79c7fff7-62df-44a2-aa3e-f24696d71b03-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79c7fff7-62df-44a2-aa3e-f24696d71b03-istiod-ca-cert\") pod \"79c7fff7-62df-44a2-aa3e-f24696d71b03\" (UID: \"79c7fff7-62df-44a2-aa3e-f24696d71b03\") " Nov 03 01:33:43.347552 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.311674 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/920eea17-797a-4c19-9f09-36f121782d2e-kube-api-access-qg594\" (UniqueName: \"kubernetes.io/projected/920eea17-797a-4c19-9f09-36f121782d2e-kube-api-access-qg594\") pod \"920eea17-797a-4c19-9f09-36f121782d2e\" (UID: \"920eea17-797a-4c19-9f09-36f121782d2e\") " Nov 03 01:33:43.347552 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.311771 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/920eea17-797a-4c19-9f09-36f121782d2e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/920eea17-797a-4c19-9f09-36f121782d2e-credential-socket\") pod \"920eea17-797a-4c19-9f09-36f121782d2e\" (UID: \"920eea17-797a-4c19-9f09-36f121782d2e\") " Nov 03 01:33:43.347552 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.311844 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79c7fff7-62df-44a2-aa3e-f24696d71b03-kube-api-access-kdbdb\" (UniqueName: \"kubernetes.io/projected/79c7fff7-62df-44a2-aa3e-f24696d71b03-kube-api-access-kdbdb\") pod \"79c7fff7-62df-44a2-aa3e-f24696d71b03\" (UID: \"79c7fff7-62df-44a2-aa3e-f24696d71b03\") " Nov 03 01:33:43.347552 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.311893 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b645b6f2-58be-45c4-9a6f-6785a1047bea-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b645b6f2-58be-45c4-9a6f-6785a1047bea-workload-certs\") pod \"b645b6f2-58be-45c4-9a6f-6785a1047bea\" (UID: \"b645b6f2-58be-45c4-9a6f-6785a1047bea\") " Nov 03 01:33:43.347552 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.311922 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/812a530a-f597-4049-a40a-2dcfb8ba6c74-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/812a530a-f597-4049-a40a-2dcfb8ba6c74-istio-envoy\") pod \"812a530a-f597-4049-a40a-2dcfb8ba6c74\" (UID: \"812a530a-f597-4049-a40a-2dcfb8ba6c74\") " Nov 03 01:33:43.347552 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.311958 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79c7fff7-62df-44a2-aa3e-f24696d71b03-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79c7fff7-62df-44a2-aa3e-f24696d71b03-workload-socket\") pod \"79c7fff7-62df-44a2-aa3e-f24696d71b03\" (UID: \"79c7fff7-62df-44a2-aa3e-f24696d71b03\") " Nov 03 01:33:43.347552 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.312004 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e2345c1-2a99-4b03-b078-eebe1f198589-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e2345c1-2a99-4b03-b078-eebe1f198589-istio-envoy\") pod \"1e2345c1-2a99-4b03-b078-eebe1f198589\" (UID: \"1e2345c1-2a99-4b03-b078-eebe1f198589\") " Nov 03 01:33:43.347552 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.312069 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/920eea17-797a-4c19-9f09-36f121782d2e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/920eea17-797a-4c19-9f09-36f121782d2e-istiod-ca-cert\") pod \"920eea17-797a-4c19-9f09-36f121782d2e\" (UID: \"920eea17-797a-4c19-9f09-36f121782d2e\") " Nov 03 01:33:43.347552 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.312138 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1e2345c1-2a99-4b03-b078-eebe1f198589-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e2345c1-2a99-4b03-b078-eebe1f198589-istio-podinfo\") pod \"1e2345c1-2a99-4b03-b078-eebe1f198589\" (UID: \"1e2345c1-2a99-4b03-b078-eebe1f198589\") " Nov 03 01:33:43.347552 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.312167 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79c7fff7-62df-44a2-aa3e-f24696d71b03-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79c7fff7-62df-44a2-aa3e-f24696d71b03-istio-data\") pod \"79c7fff7-62df-44a2-aa3e-f24696d71b03\" (UID: \"79c7fff7-62df-44a2-aa3e-f24696d71b03\") " Nov 03 01:33:43.347552 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.312193 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b645b6f2-58be-45c4-9a6f-6785a1047bea-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b645b6f2-58be-45c4-9a6f-6785a1047bea-credential-socket\") pod \"b645b6f2-58be-45c4-9a6f-6785a1047bea\" (UID: \"b645b6f2-58be-45c4-9a6f-6785a1047bea\") " Nov 03 01:33:43.347552 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.312289 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/812a530a-f597-4049-a40a-2dcfb8ba6c74-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/812a530a-f597-4049-a40a-2dcfb8ba6c74-istiod-ca-cert\") pod \"812a530a-f597-4049-a40a-2dcfb8ba6c74\" (UID: \"812a530a-f597-4049-a40a-2dcfb8ba6c74\") " Nov 03 01:33:43.347552 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.312344 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/920eea17-797a-4c19-9f09-36f121782d2e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/920eea17-797a-4c19-9f09-36f121782d2e-istio-data\") pod \"920eea17-797a-4c19-9f09-36f121782d2e\" (UID: \"920eea17-797a-4c19-9f09-36f121782d2e\") " Nov 03 01:33:43.347552 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.312409 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b645b6f2-58be-45c4-9a6f-6785a1047bea-istio-token\" (UniqueName: \"kubernetes.io/projected/b645b6f2-58be-45c4-9a6f-6785a1047bea-istio-token\") pod \"b645b6f2-58be-45c4-9a6f-6785a1047bea\" (UID: \"b645b6f2-58be-45c4-9a6f-6785a1047bea\") " Nov 03 01:33:43.347552 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.312445 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e2345c1-2a99-4b03-b078-eebe1f198589-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e2345c1-2a99-4b03-b078-eebe1f198589-istio-data\") pod \"1e2345c1-2a99-4b03-b078-eebe1f198589\" (UID: \"1e2345c1-2a99-4b03-b078-eebe1f198589\") " Nov 03 01:33:43.347552 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.312477 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/920eea17-797a-4c19-9f09-36f121782d2e-istio-token\" (UniqueName: \"kubernetes.io/projected/920eea17-797a-4c19-9f09-36f121782d2e-istio-token\") pod \"920eea17-797a-4c19-9f09-36f121782d2e\" (UID: \"920eea17-797a-4c19-9f09-36f121782d2e\") " Nov 03 01:33:43.347552 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.312599 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/812a530a-f597-4049-a40a-2dcfb8ba6c74-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/812a530a-f597-4049-a40a-2dcfb8ba6c74-istio-podinfo\") pod \"812a530a-f597-4049-a40a-2dcfb8ba6c74\" (UID: \"812a530a-f597-4049-a40a-2dcfb8ba6c74\") " Nov 03 01:33:43.347552 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.312629 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/812a530a-f597-4049-a40a-2dcfb8ba6c74-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/812a530a-f597-4049-a40a-2dcfb8ba6c74-workload-socket\") pod \"812a530a-f597-4049-a40a-2dcfb8ba6c74\" (UID: \"812a530a-f597-4049-a40a-2dcfb8ba6c74\") " Nov 03 01:33:43.347552 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.312709 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e2345c1-2a99-4b03-b078-eebe1f198589-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e2345c1-2a99-4b03-b078-eebe1f198589-credential-socket\") pod \"1e2345c1-2a99-4b03-b078-eebe1f198589\" (UID: \"1e2345c1-2a99-4b03-b078-eebe1f198589\") " Nov 03 01:33:43.347552 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.312754 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e2345c1-2a99-4b03-b078-eebe1f198589-kube-api-access-45hxf\" (UniqueName: \"kubernetes.io/projected/1e2345c1-2a99-4b03-b078-eebe1f198589-kube-api-access-45hxf\") pod \"1e2345c1-2a99-4b03-b078-eebe1f198589\" (UID: \"1e2345c1-2a99-4b03-b078-eebe1f198589\") " Nov 03 01:33:43.347552 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.312869 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/812a530a-f597-4049-a40a-2dcfb8ba6c74-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/812a530a-f597-4049-a40a-2dcfb8ba6c74-credential-socket\") pod \"812a530a-f597-4049-a40a-2dcfb8ba6c74\" (UID: \"812a530a-f597-4049-a40a-2dcfb8ba6c74\") " Nov 03 01:33:43.347552 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.312900 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b645b6f2-58be-45c4-9a6f-6785a1047bea-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b645b6f2-58be-45c4-9a6f-6785a1047bea-istio-envoy\") pod \"b645b6f2-58be-45c4-9a6f-6785a1047bea\" (UID: \"b645b6f2-58be-45c4-9a6f-6785a1047bea\") " Nov 03 01:33:43.347552 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.312952 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b645b6f2-58be-45c4-9a6f-6785a1047bea-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b645b6f2-58be-45c4-9a6f-6785a1047bea-istiod-ca-cert\") pod \"b645b6f2-58be-45c4-9a6f-6785a1047bea\" (UID: \"b645b6f2-58be-45c4-9a6f-6785a1047bea\") " Nov 03 01:33:43.347552 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.312989 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e2345c1-2a99-4b03-b078-eebe1f198589-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e2345c1-2a99-4b03-b078-eebe1f198589-istiod-ca-cert\") pod \"1e2345c1-2a99-4b03-b078-eebe1f198589\" (UID: \"1e2345c1-2a99-4b03-b078-eebe1f198589\") " Nov 03 01:33:43.347552 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.313019 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/812a530a-f597-4049-a40a-2dcfb8ba6c74-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/812a530a-f597-4049-a40a-2dcfb8ba6c74-workload-certs\") pod \"812a530a-f597-4049-a40a-2dcfb8ba6c74\" (UID: \"812a530a-f597-4049-a40a-2dcfb8ba6c74\") " Nov 03 01:33:43.347552 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.321019 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e2345c1-2a99-4b03-b078-eebe1f198589-workload-certs" pod "1e2345c1-2a99-4b03-b078-eebe1f198589" (UID: "1e2345c1-2a99-4b03-b078-eebe1f198589"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:33:43.347552 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.321040 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/812a530a-f597-4049-a40a-2dcfb8ba6c74-workload-certs" pod "812a530a-f597-4049-a40a-2dcfb8ba6c74" (UID: "812a530a-f597-4049-a40a-2dcfb8ba6c74"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:33:43.347552 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.321287 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/812a530a-f597-4049-a40a-2dcfb8ba6c74-istio-data" pod "812a530a-f597-4049-a40a-2dcfb8ba6c74" (UID: "812a530a-f597-4049-a40a-2dcfb8ba6c74"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:33:43.347552 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.321331 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b645b6f2-58be-45c4-9a6f-6785a1047bea-workload-socket" pod "b645b6f2-58be-45c4-9a6f-6785a1047bea" (UID: "b645b6f2-58be-45c4-9a6f-6785a1047bea"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:33:43.347552 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.321601 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79c7fff7-62df-44a2-aa3e-f24696d71b03-workload-socket" pod "79c7fff7-62df-44a2-aa3e-f24696d71b03" (UID: "79c7fff7-62df-44a2-aa3e-f24696d71b03"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:33:43.347552 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.321893 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e2345c1-2a99-4b03-b078-eebe1f198589-istio-data" pod "1e2345c1-2a99-4b03-b078-eebe1f198589" (UID: "1e2345c1-2a99-4b03-b078-eebe1f198589"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:33:43.347552 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.321023 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e2345c1-2a99-4b03-b078-eebe1f198589-workload-socket" pod "1e2345c1-2a99-4b03-b078-eebe1f198589" (UID: "1e2345c1-2a99-4b03-b078-eebe1f198589"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:33:43.347552 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.322193 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b645b6f2-58be-45c4-9a6f-6785a1047bea-credential-socket" pod "b645b6f2-58be-45c4-9a6f-6785a1047bea" (UID: "b645b6f2-58be-45c4-9a6f-6785a1047bea"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:33:43.347552 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.334386 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/920eea17-797a-4c19-9f09-36f121782d2e-istiod-ca-cert" pod "920eea17-797a-4c19-9f09-36f121782d2e" (UID: "920eea17-797a-4c19-9f09-36f121782d2e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:33:43.637325 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.357727 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79c7fff7-62df-44a2-aa3e-f24696d71b03-istio-data" pod "79c7fff7-62df-44a2-aa3e-f24696d71b03" (UID: "79c7fff7-62df-44a2-aa3e-f24696d71b03"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:33:43.637325 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.412753 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/812a530a-f597-4049-a40a-2dcfb8ba6c74-credential-socket" pod "812a530a-f597-4049-a40a-2dcfb8ba6c74" (UID: "812a530a-f597-4049-a40a-2dcfb8ba6c74"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:33:43.637325 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.414701 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/812a530a-f597-4049-a40a-2dcfb8ba6c74-istiod-ca-cert" pod "812a530a-f597-4049-a40a-2dcfb8ba6c74" (UID: "812a530a-f597-4049-a40a-2dcfb8ba6c74"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:33:43.637325 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.414914 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/920eea17-797a-4c19-9f09-36f121782d2e-istio-data" pod "920eea17-797a-4c19-9f09-36f121782d2e" (UID: "920eea17-797a-4c19-9f09-36f121782d2e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:33:43.637325 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.415226 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b645b6f2-58be-45c4-9a6f-6785a1047bea-istio-data" pod "b645b6f2-58be-45c4-9a6f-6785a1047bea" (UID: "b645b6f2-58be-45c4-9a6f-6785a1047bea"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:33:43.637325 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.415601 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79c7fff7-62df-44a2-aa3e-f24696d71b03-workload-certs" pod "79c7fff7-62df-44a2-aa3e-f24696d71b03" (UID: "79c7fff7-62df-44a2-aa3e-f24696d71b03"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:33:43.637325 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.429611 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/920eea17-797a-4c19-9f09-36f121782d2e-workload-certs" pod "920eea17-797a-4c19-9f09-36f121782d2e" (UID: "920eea17-797a-4c19-9f09-36f121782d2e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:33:43.637325 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.429817 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79c7fff7-62df-44a2-aa3e-f24696d71b03-credential-socket" pod "79c7fff7-62df-44a2-aa3e-f24696d71b03" (UID: "79c7fff7-62df-44a2-aa3e-f24696d71b03"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:33:43.637325 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.430740 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/812a530a-f597-4049-a40a-2dcfb8ba6c74-workload-socket" pod "812a530a-f597-4049-a40a-2dcfb8ba6c74" (UID: "812a530a-f597-4049-a40a-2dcfb8ba6c74"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:33:43.637325 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.431483 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b645b6f2-58be-45c4-9a6f-6785a1047bea-istiod-ca-cert" pod "b645b6f2-58be-45c4-9a6f-6785a1047bea" (UID: "b645b6f2-58be-45c4-9a6f-6785a1047bea"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:33:43.637325 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.431665 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79c7fff7-62df-44a2-aa3e-f24696d71b03-istiod-ca-cert" pod "79c7fff7-62df-44a2-aa3e-f24696d71b03" (UID: "79c7fff7-62df-44a2-aa3e-f24696d71b03"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:33:43.637325 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.431827 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e2345c1-2a99-4b03-b078-eebe1f198589-istiod-ca-cert" pod "1e2345c1-2a99-4b03-b078-eebe1f198589" (UID: "1e2345c1-2a99-4b03-b078-eebe1f198589"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:33:43.637325 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.433015 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b645b6f2-58be-45c4-9a6f-6785a1047bea-workload-certs" pod "b645b6f2-58be-45c4-9a6f-6785a1047bea" (UID: "b645b6f2-58be-45c4-9a6f-6785a1047bea"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:33:43.637325 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.433118 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e2345c1-2a99-4b03-b078-eebe1f198589-credential-socket" pod "1e2345c1-2a99-4b03-b078-eebe1f198589" (UID: "1e2345c1-2a99-4b03-b078-eebe1f198589"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:33:43.637325 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.433605 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/920eea17-797a-4c19-9f09-36f121782d2e-credential-socket" pod "920eea17-797a-4c19-9f09-36f121782d2e" (UID: "920eea17-797a-4c19-9f09-36f121782d2e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:33:43.637325 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.433912 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/920eea17-797a-4c19-9f09-36f121782d2e-workload-socket" pod "920eea17-797a-4c19-9f09-36f121782d2e" (UID: "920eea17-797a-4c19-9f09-36f121782d2e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:33:43.637325 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.434311 4421 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/812a530a-f597-4049-a40a-2dcfb8ba6c74-istiod-ca-cert\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:33:43.637325 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.434332 4421 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/920eea17-797a-4c19-9f09-36f121782d2e-istio-data\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:33:43.637325 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.434342 4421 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e2345c1-2a99-4b03-b078-eebe1f198589-istio-data\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:33:43.637325 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.434351 4421 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/812a530a-f597-4049-a40a-2dcfb8ba6c74-workload-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:33:43.637325 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.434360 4421 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e2345c1-2a99-4b03-b078-eebe1f198589-credential-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:33:43.637325 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.434368 4421 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/812a530a-f597-4049-a40a-2dcfb8ba6c74-credential-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:33:43.637325 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.434378 4421 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b645b6f2-58be-45c4-9a6f-6785a1047bea-istiod-ca-cert\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:33:43.637325 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.434387 4421 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e2345c1-2a99-4b03-b078-eebe1f198589-istiod-ca-cert\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:33:43.637325 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.434400 4421 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/812a530a-f597-4049-a40a-2dcfb8ba6c74-workload-certs\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:33:43.637325 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.434412 4421 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e2345c1-2a99-4b03-b078-eebe1f198589-workload-certs\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:33:43.637325 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.434422 4421 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e2345c1-2a99-4b03-b078-eebe1f198589-workload-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:33:43.637325 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.434432 4421 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b645b6f2-58be-45c4-9a6f-6785a1047bea-workload-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:33:43.637325 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.434440 4421 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/812a530a-f597-4049-a40a-2dcfb8ba6c74-istio-data\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:33:43.637325 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.434449 4421 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b645b6f2-58be-45c4-9a6f-6785a1047bea-istio-data\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:33:43.637325 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.434458 4421 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79c7fff7-62df-44a2-aa3e-f24696d71b03-workload-certs\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:33:43.637325 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.434467 4421 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/920eea17-797a-4c19-9f09-36f121782d2e-workload-certs\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:33:43.637325 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.434481 4421 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79c7fff7-62df-44a2-aa3e-f24696d71b03-credential-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:33:43.637325 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.434492 4421 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/920eea17-797a-4c19-9f09-36f121782d2e-workload-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:33:43.637325 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.434501 4421 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79c7fff7-62df-44a2-aa3e-f24696d71b03-istiod-ca-cert\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:33:43.637325 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.434517 4421 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/920eea17-797a-4c19-9f09-36f121782d2e-credential-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:33:43.637325 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.434526 4421 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b645b6f2-58be-45c4-9a6f-6785a1047bea-workload-certs\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:33:43.637325 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.434536 4421 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79c7fff7-62df-44a2-aa3e-f24696d71b03-workload-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:33:43.637325 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.434549 4421 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/920eea17-797a-4c19-9f09-36f121782d2e-istiod-ca-cert\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:33:43.637325 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.434563 4421 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79c7fff7-62df-44a2-aa3e-f24696d71b03-istio-data\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:33:43.637325 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.434579 4421 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b645b6f2-58be-45c4-9a6f-6785a1047bea-credential-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:33:43.637325 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.568195 4421 generic.go:356] "Generic (PLEG): container finished" podID="b645b6f2-58be-45c4-9a6f-6785a1047bea" containerID="38f6c23c7753df1e2d6b4de8eabe27e5046425a24da317c37e9e24936a4adf82" exitCode=0 Nov 03 01:33:43.637325 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.568321 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-4tt4f" Nov 03 01:33:43.637325 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.568313 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-4tt4f" event={"ID":"b645b6f2-58be-45c4-9a6f-6785a1047bea","Type":"ContainerDied","Data":"38f6c23c7753df1e2d6b4de8eabe27e5046425a24da317c37e9e24936a4adf82"} Nov 03 01:33:43.637325 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.568451 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-4tt4f" event={"ID":"b645b6f2-58be-45c4-9a6f-6785a1047bea","Type":"ContainerDied","Data":"d28b1487e660fd3e4be7ce8b45602b4c6679fb226c53b02cce829101dcbec6bf"} Nov 03 01:33:43.637325 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.568484 4421 scope.go:122] "RemoveContainer" containerID="38f6c23c7753df1e2d6b4de8eabe27e5046425a24da317c37e9e24936a4adf82" Nov 03 01:33:43.637325 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.570486 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zgdl8" Nov 03 01:33:43.637325 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.570505 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zgdl8" event={"ID":"920eea17-797a-4c19-9f09-36f121782d2e","Type":"ContainerDied","Data":"72338106d88185586eddfd3755eb712102e2d30a941fe9caba3f1caa0e6a1435"} Nov 03 01:33:43.637325 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.572235 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-vhjlx" event={"ID":"1e2345c1-2a99-4b03-b078-eebe1f198589","Type":"ContainerDied","Data":"b4255f57b4864f07879f04d0762b47722fc529ae128fdc08adb488343ab6ef86"} Nov 03 01:33:43.637325 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.572321 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-vhjlx" Nov 03 01:33:43.637325 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.574612 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-gvmnj" Nov 03 01:33:43.637325 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.574632 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-gvmnj" event={"ID":"812a530a-f597-4049-a40a-2dcfb8ba6c74","Type":"ContainerDied","Data":"2a64502f7c1b61e1fe90ed30de8fa1b1950fd23d12164b1701b793e6b48feb3e"} Nov 03 01:33:43.637325 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.592097 4421 generic.go:356] "Generic (PLEG): container finished" podID="79c7fff7-62df-44a2-aa3e-f24696d71b03" containerID="d74595c5bdaade18ab20fc49ed721cc7212db6df5232f69a0d362df3e9687ab6" exitCode=0 Nov 03 01:33:43.637325 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.592156 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-x8tb9" Nov 03 01:33:43.637325 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.592204 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-x8tb9" event={"ID":"79c7fff7-62df-44a2-aa3e-f24696d71b03","Type":"ContainerDied","Data":"d74595c5bdaade18ab20fc49ed721cc7212db6df5232f69a0d362df3e9687ab6"} Nov 03 01:33:43.637325 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.592249 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-x8tb9" event={"ID":"79c7fff7-62df-44a2-aa3e-f24696d71b03","Type":"ContainerDied","Data":"209ca632afd1755e2e87cd58903e10b1dc1a7ef524d009cf8a7740a13727b94d"} Nov 03 01:33:43.709603 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.702026 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/920eea17-797a-4c19-9f09-36f121782d2e-kube-api-access-qg594" pod "920eea17-797a-4c19-9f09-36f121782d2e" (UID: "920eea17-797a-4c19-9f09-36f121782d2e"). InnerVolumeSpecName "kube-api-access-qg594". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:33:43.709603 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.704102 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1e2345c1-2a99-4b03-b078-eebe1f198589-istio-podinfo" pod "1e2345c1-2a99-4b03-b078-eebe1f198589" (UID: "1e2345c1-2a99-4b03-b078-eebe1f198589"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:33:43.709603 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.704226 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/79c7fff7-62df-44a2-aa3e-f24696d71b03-istio-podinfo" pod "79c7fff7-62df-44a2-aa3e-f24696d71b03" (UID: "79c7fff7-62df-44a2-aa3e-f24696d71b03"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:33:43.709603 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.704260 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79c7fff7-62df-44a2-aa3e-f24696d71b03-istio-token" pod "79c7fff7-62df-44a2-aa3e-f24696d71b03" (UID: "79c7fff7-62df-44a2-aa3e-f24696d71b03"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:33:43.709603 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.704564 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79c7fff7-62df-44a2-aa3e-f24696d71b03-kube-api-access-kdbdb" pod "79c7fff7-62df-44a2-aa3e-f24696d71b03" (UID: "79c7fff7-62df-44a2-aa3e-f24696d71b03"). InnerVolumeSpecName "kube-api-access-kdbdb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:33:43.709603 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.704674 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/812a530a-f597-4049-a40a-2dcfb8ba6c74-istio-podinfo" pod "812a530a-f597-4049-a40a-2dcfb8ba6c74" (UID: "812a530a-f597-4049-a40a-2dcfb8ba6c74"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:33:43.709603 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.704687 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/812a530a-f597-4049-a40a-2dcfb8ba6c74-istio-token" pod "812a530a-f597-4049-a40a-2dcfb8ba6c74" (UID: "812a530a-f597-4049-a40a-2dcfb8ba6c74"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:33:43.709603 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.705426 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/812a530a-f597-4049-a40a-2dcfb8ba6c74-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/812a530a-f597-4049-a40a-2dcfb8ba6c74-istio-podinfo\") pod \"812a530a-f597-4049-a40a-2dcfb8ba6c74\" (UID: \"812a530a-f597-4049-a40a-2dcfb8ba6c74\") " Nov 03 01:33:43.709603 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.705679 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79c7fff7-62df-44a2-aa3e-f24696d71b03-istio-token\" (UniqueName: \"kubernetes.io/projected/79c7fff7-62df-44a2-aa3e-f24696d71b03-istio-token\") pod \"79c7fff7-62df-44a2-aa3e-f24696d71b03\" (UID: \"79c7fff7-62df-44a2-aa3e-f24696d71b03\") " Nov 03 01:33:43.709603 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:33:43.705864 4421 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/812a530a-f597-4049-a40a-2dcfb8ba6c74/volumes/kubernetes.io~downward-api/istio-podinfo Nov 03 01:33:43.709603 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.705884 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/812a530a-f597-4049-a40a-2dcfb8ba6c74-istio-podinfo" pod "812a530a-f597-4049-a40a-2dcfb8ba6c74" (UID: "812a530a-f597-4049-a40a-2dcfb8ba6c74"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:33:43.709603 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:33:43.706013 4421 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/79c7fff7-62df-44a2-aa3e-f24696d71b03/volumes/kubernetes.io~projected/istio-token Nov 03 01:33:43.709603 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.706042 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79c7fff7-62df-44a2-aa3e-f24696d71b03-istio-token" pod "79c7fff7-62df-44a2-aa3e-f24696d71b03" (UID: "79c7fff7-62df-44a2-aa3e-f24696d71b03"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:33:43.709603 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.706501 4421 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79c7fff7-62df-44a2-aa3e-f24696d71b03-istio-podinfo\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:33:43.709603 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.706533 4421 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79c7fff7-62df-44a2-aa3e-f24696d71b03-istio-token\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:33:43.709603 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.706548 4421 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/812a530a-f597-4049-a40a-2dcfb8ba6c74-istio-token\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:33:43.709603 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.706566 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qg594\" (UniqueName: \"kubernetes.io/projected/920eea17-797a-4c19-9f09-36f121782d2e-kube-api-access-qg594\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:33:43.709603 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.706581 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdbdb\" (UniqueName: \"kubernetes.io/projected/79c7fff7-62df-44a2-aa3e-f24696d71b03-kube-api-access-kdbdb\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:33:43.709603 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.706597 4421 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e2345c1-2a99-4b03-b078-eebe1f198589-istio-podinfo\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:33:43.709603 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.706611 4421 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/812a530a-f597-4049-a40a-2dcfb8ba6c74-istio-podinfo\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:33:43.709603 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.708397 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/920eea17-797a-4c19-9f09-36f121782d2e-istio-podinfo" pod "920eea17-797a-4c19-9f09-36f121782d2e" (UID: "920eea17-797a-4c19-9f09-36f121782d2e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:33:43.709603 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.708407 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/920eea17-797a-4c19-9f09-36f121782d2e-istio-envoy" pod "920eea17-797a-4c19-9f09-36f121782d2e" (UID: "920eea17-797a-4c19-9f09-36f121782d2e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:33:43.709603 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.708497 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/812a530a-f597-4049-a40a-2dcfb8ba6c74-kube-api-access-vhcqn" pod "812a530a-f597-4049-a40a-2dcfb8ba6c74" (UID: "812a530a-f597-4049-a40a-2dcfb8ba6c74"). InnerVolumeSpecName "kube-api-access-vhcqn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:33:43.709603 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.708535 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b645b6f2-58be-45c4-9a6f-6785a1047bea-istio-token" pod "b645b6f2-58be-45c4-9a6f-6785a1047bea" (UID: "b645b6f2-58be-45c4-9a6f-6785a1047bea"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:33:43.720455 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.710903 4421 scope.go:122] "RemoveContainer" containerID="38f6c23c7753df1e2d6b4de8eabe27e5046425a24da317c37e9e24936a4adf82" Nov 03 01:33:43.720455 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.712700 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/812a530a-f597-4049-a40a-2dcfb8ba6c74-istio-envoy" pod "812a530a-f597-4049-a40a-2dcfb8ba6c74" (UID: "812a530a-f597-4049-a40a-2dcfb8ba6c74"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:33:43.720455 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:33:43.713545 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38f6c23c7753df1e2d6b4de8eabe27e5046425a24da317c37e9e24936a4adf82\": container with ID starting with 38f6c23c7753df1e2d6b4de8eabe27e5046425a24da317c37e9e24936a4adf82 not found: ID does not exist" containerID="38f6c23c7753df1e2d6b4de8eabe27e5046425a24da317c37e9e24936a4adf82" Nov 03 01:33:43.720455 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.713589 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38f6c23c7753df1e2d6b4de8eabe27e5046425a24da317c37e9e24936a4adf82"} err="failed to get container status \"38f6c23c7753df1e2d6b4de8eabe27e5046425a24da317c37e9e24936a4adf82\": rpc error: code = NotFound desc = could not find container \"38f6c23c7753df1e2d6b4de8eabe27e5046425a24da317c37e9e24936a4adf82\": container with ID starting with 38f6c23c7753df1e2d6b4de8eabe27e5046425a24da317c37e9e24936a4adf82 not found: ID does not exist" Nov 03 01:33:43.720455 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.713614 4421 scope.go:122] "RemoveContainer" containerID="1cf56a802f20d20a896f34cee26bb9327a975f2779269509d11017a42b7c66c6" Nov 03 01:33:43.720455 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.713673 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b645b6f2-58be-45c4-9a6f-6785a1047bea-istio-envoy" pod "b645b6f2-58be-45c4-9a6f-6785a1047bea" (UID: "b645b6f2-58be-45c4-9a6f-6785a1047bea"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:33:43.720455 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.715646 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b645b6f2-58be-45c4-9a6f-6785a1047bea-istio-podinfo" pod "b645b6f2-58be-45c4-9a6f-6785a1047bea" (UID: "b645b6f2-58be-45c4-9a6f-6785a1047bea"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:33:43.720455 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.716049 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e2345c1-2a99-4b03-b078-eebe1f198589-istio-token" pod "1e2345c1-2a99-4b03-b078-eebe1f198589" (UID: "1e2345c1-2a99-4b03-b078-eebe1f198589"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:33:43.720455 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.717650 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79c7fff7-62df-44a2-aa3e-f24696d71b03-istio-envoy" pod "79c7fff7-62df-44a2-aa3e-f24696d71b03" (UID: "79c7fff7-62df-44a2-aa3e-f24696d71b03"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:33:43.720455 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.719801 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/920eea17-797a-4c19-9f09-36f121782d2e-istio-token" pod "920eea17-797a-4c19-9f09-36f121782d2e" (UID: "920eea17-797a-4c19-9f09-36f121782d2e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:33:43.730211 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.720882 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e2345c1-2a99-4b03-b078-eebe1f198589-istio-envoy" pod "1e2345c1-2a99-4b03-b078-eebe1f198589" (UID: "1e2345c1-2a99-4b03-b078-eebe1f198589"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:33:43.730211 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.720893 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b645b6f2-58be-45c4-9a6f-6785a1047bea-kube-api-access-rvfzl" pod "b645b6f2-58be-45c4-9a6f-6785a1047bea" (UID: "b645b6f2-58be-45c4-9a6f-6785a1047bea"). InnerVolumeSpecName "kube-api-access-rvfzl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:33:43.730211 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.720957 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e2345c1-2a99-4b03-b078-eebe1f198589-kube-api-access-45hxf" pod "1e2345c1-2a99-4b03-b078-eebe1f198589" (UID: "1e2345c1-2a99-4b03-b078-eebe1f198589"). InnerVolumeSpecName "kube-api-access-45hxf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:33:43.772548 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.772527 4421 scope.go:122] "RemoveContainer" containerID="69702dff6eed0fd1f37433181868ea6b682951c24343e3e04dc3116e146c03c3" Nov 03 01:33:43.785068 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.785046 4421 scope.go:122] "RemoveContainer" containerID="b177a223eb4b7553bf143b9f7b2045e22a53266c156107de37060f64bb62bc28" Nov 03 01:33:43.797363 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.796799 4421 scope.go:122] "RemoveContainer" containerID="d74595c5bdaade18ab20fc49ed721cc7212db6df5232f69a0d362df3e9687ab6" Nov 03 01:33:43.809935 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.809104 4421 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b645b6f2-58be-45c4-9a6f-6785a1047bea-istio-envoy\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:33:43.809935 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.809137 4421 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/920eea17-797a-4c19-9f09-36f121782d2e-istio-envoy\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:33:43.809935 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.809185 4421 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b645b6f2-58be-45c4-9a6f-6785a1047bea-istio-podinfo\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:33:43.809935 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.809201 4421 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79c7fff7-62df-44a2-aa3e-f24696d71b03-istio-envoy\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:33:43.809935 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.809215 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhcqn\" (UniqueName: \"kubernetes.io/projected/812a530a-f597-4049-a40a-2dcfb8ba6c74-kube-api-access-vhcqn\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:33:43.809935 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.809230 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvfzl\" (UniqueName: \"kubernetes.io/projected/b645b6f2-58be-45c4-9a6f-6785a1047bea-kube-api-access-rvfzl\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:33:43.809935 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.809260 4421 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e2345c1-2a99-4b03-b078-eebe1f198589-istio-token\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:33:43.809935 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.809299 4421 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/920eea17-797a-4c19-9f09-36f121782d2e-istio-podinfo\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:33:43.809935 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.809315 4421 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/812a530a-f597-4049-a40a-2dcfb8ba6c74-istio-envoy\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:33:43.809935 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.809338 4421 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e2345c1-2a99-4b03-b078-eebe1f198589-istio-envoy\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:33:43.809935 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.809352 4421 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b645b6f2-58be-45c4-9a6f-6785a1047bea-istio-token\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:33:43.809935 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.809385 4421 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/920eea17-797a-4c19-9f09-36f121782d2e-istio-token\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:33:43.809935 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.809399 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45hxf\" (UniqueName: \"kubernetes.io/projected/1e2345c1-2a99-4b03-b078-eebe1f198589-kube-api-access-45hxf\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:33:43.821026 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.813155 4421 scope.go:122] "RemoveContainer" containerID="d74595c5bdaade18ab20fc49ed721cc7212db6df5232f69a0d362df3e9687ab6" Nov 03 01:33:43.821026 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:33:43.813548 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d74595c5bdaade18ab20fc49ed721cc7212db6df5232f69a0d362df3e9687ab6\": container with ID starting with d74595c5bdaade18ab20fc49ed721cc7212db6df5232f69a0d362df3e9687ab6 not found: ID does not exist" containerID="d74595c5bdaade18ab20fc49ed721cc7212db6df5232f69a0d362df3e9687ab6" Nov 03 01:33:43.821026 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.813584 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d74595c5bdaade18ab20fc49ed721cc7212db6df5232f69a0d362df3e9687ab6"} err="failed to get container status \"d74595c5bdaade18ab20fc49ed721cc7212db6df5232f69a0d362df3e9687ab6\": rpc error: code = NotFound desc = could not find container \"d74595c5bdaade18ab20fc49ed721cc7212db6df5232f69a0d362df3e9687ab6\": container with ID starting with d74595c5bdaade18ab20fc49ed721cc7212db6df5232f69a0d362df3e9687ab6 not found: ID does not exist" Nov 03 01:33:44.048171 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.900873 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zgdl8"] Nov 03 01:33:44.048171 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.905901 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zgdl8"] Nov 03 01:33:44.048171 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.917079 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-4tt4f"] Nov 03 01:33:44.048171 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.921814 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-4tt4f"] Nov 03 01:33:44.048171 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.931400 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-gvmnj"] Nov 03 01:33:44.048171 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.937255 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-gvmnj"] Nov 03 01:33:44.048171 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.948489 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-vhjlx"] Nov 03 01:33:44.048171 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.954153 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-vhjlx"] Nov 03 01:33:44.048171 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:43.966793 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-x8tb9"] Nov 03 01:33:44.048171 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:44.003552 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e2345c1-2a99-4b03-b078-eebe1f198589" path="/var/lib/kubelet/pods/1e2345c1-2a99-4b03-b078-eebe1f198589/volumes" Nov 03 01:33:44.153749 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:44.064037 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="812a530a-f597-4049-a40a-2dcfb8ba6c74" path="/var/lib/kubelet/pods/812a530a-f597-4049-a40a-2dcfb8ba6c74/volumes" Nov 03 01:33:44.153749 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:44.064633 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="920eea17-797a-4c19-9f09-36f121782d2e" path="/var/lib/kubelet/pods/920eea17-797a-4c19-9f09-36f121782d2e/volumes" Nov 03 01:33:44.153749 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:44.065098 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b645b6f2-58be-45c4-9a6f-6785a1047bea" path="/var/lib/kubelet/pods/b645b6f2-58be-45c4-9a6f-6785a1047bea/volumes" Nov 03 01:33:44.153749 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:44.065746 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-x8tb9"] Nov 03 01:33:44.300380 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:44.298537 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-5bfc51-6c7fc6b99f-kdhcn"] Nov 03 01:33:44.915055 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:33:44.915010 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 01:33:45.982056 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:45.982019 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79c7fff7-62df-44a2-aa3e-f24696d71b03" path="/var/lib/kubelet/pods/79c7fff7-62df-44a2-aa3e-f24696d71b03/volumes" Nov 03 01:33:49.298858 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:49.298809 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5bfc51-6c7fc6b99f-kdhcn" podUID="1758fd0f-1c79-41fe-be74-474ba88962ca" containerName="payload-pre-processing" containerID="cri-o://24254f8efebde82101fba3587d2b4d21e199d080797c90107ca55a1575aa0b90" gracePeriod=25 Nov 03 01:33:51.475684 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:51.475641 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5bfc51-6c7fc6b99f-kdhcn" podUID="1758fd0f-1c79-41fe-be74-474ba88962ca" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:33:51.625582 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:51.625541 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-5bfc51-openshift-default-54d8846889-sc6sj"] Nov 03 01:33:51.625850 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:51.625821 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-5bfc51-openshift-default-54d8846889-sc6sj" podUID="781e1acc-4bc9-489f-8df6-6b5f8c10ba1b" containerName="istio-proxy" containerID="cri-o://c4a61a0ded25ac4ec91681148656a34e9d166be8de77bc84446b246b23a8b181" gracePeriod=30 Nov 03 01:33:51.780518 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:51.780477 4421 patch_prober.go:29] interesting pod/e2e-inf-b-5bfc51-openshift-default-54d8846889-sc6sj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 01:33:51.788460 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:51.780741 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-5bfc51-openshift-default-54d8846889-sc6sj" podUID="781e1acc-4bc9-489f-8df6-6b5f8c10ba1b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 01:33:56.964885 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:56.964856 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5bfc51-openshift-default-54d8846889-sc6sj" Nov 03 01:33:57.028779 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:57.028739 4421 generic.go:356] "Generic (PLEG): container finished" podID="781e1acc-4bc9-489f-8df6-6b5f8c10ba1b" containerID="c4a61a0ded25ac4ec91681148656a34e9d166be8de77bc84446b246b23a8b181" exitCode=0 Nov 03 01:33:57.028963 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:57.028803 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5bfc51-openshift-default-54d8846889-sc6sj" Nov 03 01:33:57.028963 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:57.028825 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5bfc51-openshift-default-54d8846889-sc6sj" event={"ID":"781e1acc-4bc9-489f-8df6-6b5f8c10ba1b","Type":"ContainerDied","Data":"c4a61a0ded25ac4ec91681148656a34e9d166be8de77bc84446b246b23a8b181"} Nov 03 01:33:57.028963 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:57.028884 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5bfc51-openshift-default-54d8846889-sc6sj" event={"ID":"781e1acc-4bc9-489f-8df6-6b5f8c10ba1b","Type":"ContainerDied","Data":"7857e624a87614454db50b098f5dc7732cedbcaf34f73e08697a5f19a26e3b4a"} Nov 03 01:33:57.028963 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:57.028914 4421 scope.go:122] "RemoveContainer" containerID="c4a61a0ded25ac4ec91681148656a34e9d166be8de77bc84446b246b23a8b181" Nov 03 01:33:57.041618 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:57.041592 4421 scope.go:122] "RemoveContainer" containerID="39421a5a8e7703794bb42341d5ced6151daf742935511581e9300f6e52d11b57" Nov 03 01:33:57.053455 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:57.053381 4421 scope.go:122] "RemoveContainer" containerID="c4a61a0ded25ac4ec91681148656a34e9d166be8de77bc84446b246b23a8b181" Nov 03 01:33:57.053733 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:33:57.053708 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4a61a0ded25ac4ec91681148656a34e9d166be8de77bc84446b246b23a8b181\": container with ID starting with c4a61a0ded25ac4ec91681148656a34e9d166be8de77bc84446b246b23a8b181 not found: ID does not exist" containerID="c4a61a0ded25ac4ec91681148656a34e9d166be8de77bc84446b246b23a8b181" Nov 03 01:33:57.053836 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:57.053744 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4a61a0ded25ac4ec91681148656a34e9d166be8de77bc84446b246b23a8b181"} err="failed to get container status \"c4a61a0ded25ac4ec91681148656a34e9d166be8de77bc84446b246b23a8b181\": rpc error: code = NotFound desc = could not find container \"c4a61a0ded25ac4ec91681148656a34e9d166be8de77bc84446b246b23a8b181\": container with ID starting with c4a61a0ded25ac4ec91681148656a34e9d166be8de77bc84446b246b23a8b181 not found: ID does not exist" Nov 03 01:33:57.053836 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:57.053771 4421 scope.go:122] "RemoveContainer" containerID="39421a5a8e7703794bb42341d5ced6151daf742935511581e9300f6e52d11b57" Nov 03 01:33:57.054066 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:33:57.054037 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39421a5a8e7703794bb42341d5ced6151daf742935511581e9300f6e52d11b57\": container with ID starting with 39421a5a8e7703794bb42341d5ced6151daf742935511581e9300f6e52d11b57 not found: ID does not exist" containerID="39421a5a8e7703794bb42341d5ced6151daf742935511581e9300f6e52d11b57" Nov 03 01:33:57.054213 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:57.054070 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39421a5a8e7703794bb42341d5ced6151daf742935511581e9300f6e52d11b57"} err="failed to get container status \"39421a5a8e7703794bb42341d5ced6151daf742935511581e9300f6e52d11b57\": rpc error: code = NotFound desc = could not find container \"39421a5a8e7703794bb42341d5ced6151daf742935511581e9300f6e52d11b57\": container with ID starting with 39421a5a8e7703794bb42341d5ced6151daf742935511581e9300f6e52d11b57 not found: ID does not exist" Nov 03 01:33:57.095677 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:57.095641 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/781e1acc-4bc9-489f-8df6-6b5f8c10ba1b-istio-token\" (UniqueName: \"kubernetes.io/projected/781e1acc-4bc9-489f-8df6-6b5f8c10ba1b-istio-token\") pod \"781e1acc-4bc9-489f-8df6-6b5f8c10ba1b\" (UID: \"781e1acc-4bc9-489f-8df6-6b5f8c10ba1b\") " Nov 03 01:33:57.095889 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:57.095831 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/781e1acc-4bc9-489f-8df6-6b5f8c10ba1b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/781e1acc-4bc9-489f-8df6-6b5f8c10ba1b-istio-envoy\") pod \"781e1acc-4bc9-489f-8df6-6b5f8c10ba1b\" (UID: \"781e1acc-4bc9-489f-8df6-6b5f8c10ba1b\") " Nov 03 01:33:57.095956 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:57.095898 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/781e1acc-4bc9-489f-8df6-6b5f8c10ba1b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/781e1acc-4bc9-489f-8df6-6b5f8c10ba1b-istio-data\") pod \"781e1acc-4bc9-489f-8df6-6b5f8c10ba1b\" (UID: \"781e1acc-4bc9-489f-8df6-6b5f8c10ba1b\") " Nov 03 01:33:57.096065 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:57.096040 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/781e1acc-4bc9-489f-8df6-6b5f8c10ba1b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/781e1acc-4bc9-489f-8df6-6b5f8c10ba1b-istio-podinfo\") pod \"781e1acc-4bc9-489f-8df6-6b5f8c10ba1b\" (UID: \"781e1acc-4bc9-489f-8df6-6b5f8c10ba1b\") " Nov 03 01:33:57.096169 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:57.096147 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/781e1acc-4bc9-489f-8df6-6b5f8c10ba1b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/781e1acc-4bc9-489f-8df6-6b5f8c10ba1b-istiod-ca-cert\") pod \"781e1acc-4bc9-489f-8df6-6b5f8c10ba1b\" (UID: \"781e1acc-4bc9-489f-8df6-6b5f8c10ba1b\") " Nov 03 01:33:57.096344 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:57.096217 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/781e1acc-4bc9-489f-8df6-6b5f8c10ba1b-istio-data" pod "781e1acc-4bc9-489f-8df6-6b5f8c10ba1b" (UID: "781e1acc-4bc9-489f-8df6-6b5f8c10ba1b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:33:57.096344 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:57.096263 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/781e1acc-4bc9-489f-8df6-6b5f8c10ba1b-kube-api-access-ssrq5\" (UniqueName: \"kubernetes.io/projected/781e1acc-4bc9-489f-8df6-6b5f8c10ba1b-kube-api-access-ssrq5\") pod \"781e1acc-4bc9-489f-8df6-6b5f8c10ba1b\" (UID: \"781e1acc-4bc9-489f-8df6-6b5f8c10ba1b\") " Nov 03 01:33:57.096483 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:57.096354 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/781e1acc-4bc9-489f-8df6-6b5f8c10ba1b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/781e1acc-4bc9-489f-8df6-6b5f8c10ba1b-workload-socket\") pod \"781e1acc-4bc9-489f-8df6-6b5f8c10ba1b\" (UID: \"781e1acc-4bc9-489f-8df6-6b5f8c10ba1b\") " Nov 03 01:33:57.096483 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:57.096390 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/781e1acc-4bc9-489f-8df6-6b5f8c10ba1b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/781e1acc-4bc9-489f-8df6-6b5f8c10ba1b-credential-socket\") pod \"781e1acc-4bc9-489f-8df6-6b5f8c10ba1b\" (UID: \"781e1acc-4bc9-489f-8df6-6b5f8c10ba1b\") " Nov 03 01:33:57.096483 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:57.096457 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/781e1acc-4bc9-489f-8df6-6b5f8c10ba1b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/781e1acc-4bc9-489f-8df6-6b5f8c10ba1b-workload-certs\") pod \"781e1acc-4bc9-489f-8df6-6b5f8c10ba1b\" (UID: \"781e1acc-4bc9-489f-8df6-6b5f8c10ba1b\") " Nov 03 01:33:57.096693 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:57.096667 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/781e1acc-4bc9-489f-8df6-6b5f8c10ba1b-istiod-ca-cert" pod "781e1acc-4bc9-489f-8df6-6b5f8c10ba1b" (UID: "781e1acc-4bc9-489f-8df6-6b5f8c10ba1b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:33:57.096959 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:57.096935 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/781e1acc-4bc9-489f-8df6-6b5f8c10ba1b-workload-socket" pod "781e1acc-4bc9-489f-8df6-6b5f8c10ba1b" (UID: "781e1acc-4bc9-489f-8df6-6b5f8c10ba1b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:33:57.097328 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:57.097184 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/781e1acc-4bc9-489f-8df6-6b5f8c10ba1b-credential-socket" pod "781e1acc-4bc9-489f-8df6-6b5f8c10ba1b" (UID: "781e1acc-4bc9-489f-8df6-6b5f8c10ba1b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:33:57.097707 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:57.097480 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/781e1acc-4bc9-489f-8df6-6b5f8c10ba1b-workload-certs" pod "781e1acc-4bc9-489f-8df6-6b5f8c10ba1b" (UID: "781e1acc-4bc9-489f-8df6-6b5f8c10ba1b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:33:57.097839 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:57.097726 4421 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/781e1acc-4bc9-489f-8df6-6b5f8c10ba1b-istio-data\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:33:57.097839 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:57.097749 4421 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/781e1acc-4bc9-489f-8df6-6b5f8c10ba1b-istiod-ca-cert\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:33:57.097839 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:57.097765 4421 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/781e1acc-4bc9-489f-8df6-6b5f8c10ba1b-workload-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:33:57.097839 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:57.097780 4421 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/781e1acc-4bc9-489f-8df6-6b5f8c10ba1b-credential-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:33:57.097839 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:57.097797 4421 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/781e1acc-4bc9-489f-8df6-6b5f8c10ba1b-workload-certs\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:33:57.100443 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:57.100422 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/781e1acc-4bc9-489f-8df6-6b5f8c10ba1b-istio-envoy" pod "781e1acc-4bc9-489f-8df6-6b5f8c10ba1b" (UID: "781e1acc-4bc9-489f-8df6-6b5f8c10ba1b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:33:57.145655 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:57.144153 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/781e1acc-4bc9-489f-8df6-6b5f8c10ba1b-istio-podinfo" pod "781e1acc-4bc9-489f-8df6-6b5f8c10ba1b" (UID: "781e1acc-4bc9-489f-8df6-6b5f8c10ba1b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:33:57.154623 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:57.154528 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/781e1acc-4bc9-489f-8df6-6b5f8c10ba1b-istio-token" pod "781e1acc-4bc9-489f-8df6-6b5f8c10ba1b" (UID: "781e1acc-4bc9-489f-8df6-6b5f8c10ba1b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:33:57.165556 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:57.165484 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/781e1acc-4bc9-489f-8df6-6b5f8c10ba1b-kube-api-access-ssrq5" pod "781e1acc-4bc9-489f-8df6-6b5f8c10ba1b" (UID: "781e1acc-4bc9-489f-8df6-6b5f8c10ba1b"). InnerVolumeSpecName "kube-api-access-ssrq5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:33:57.199297 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:57.199189 4421 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/781e1acc-4bc9-489f-8df6-6b5f8c10ba1b-istio-podinfo\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:33:57.199297 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:57.199230 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ssrq5\" (UniqueName: \"kubernetes.io/projected/781e1acc-4bc9-489f-8df6-6b5f8c10ba1b-kube-api-access-ssrq5\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:33:57.199297 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:57.199249 4421 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/781e1acc-4bc9-489f-8df6-6b5f8c10ba1b-istio-token\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:33:57.199297 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:57.199262 4421 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/781e1acc-4bc9-489f-8df6-6b5f8c10ba1b-istio-envoy\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:33:57.358457 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:57.358413 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-5bfc51-openshift-default-54d8846889-sc6sj"] Nov 03 01:33:57.363516 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:57.363484 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-5bfc51-openshift-default-54d8846889-sc6sj"] Nov 03 01:33:57.983644 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:33:57.983605 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="781e1acc-4bc9-489f-8df6-6b5f8c10ba1b" path="/var/lib/kubelet/pods/781e1acc-4bc9-489f-8df6-6b5f8c10ba1b/volumes" Nov 03 01:34:01.475323 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:01.475267 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5bfc51-6c7fc6b99f-kdhcn" podUID="1758fd0f-1c79-41fe-be74-474ba88962ca" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:34:05.535537 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:05.535497 4421 scope.go:122] "RemoveContainer" containerID="3956e74d8c81b1a31ac85be391532806d5ba2ccc1c5bd7eacec13186b950eacb" Nov 03 01:34:05.548713 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:05.548692 4421 scope.go:122] "RemoveContainer" containerID="82496076a6fe00149ba48e215e7b265b317fc263b9a661dcb61afcb37ed3b110" Nov 03 01:34:09.628479 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:09.628449 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5bfc51-6c7fc6b99f-kdhcn" Nov 03 01:34:09.724268 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:09.724233 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1758fd0f-1c79-41fe-be74-474ba88962ca-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1758fd0f-1c79-41fe-be74-474ba88962ca-plugins-config-volume\") pod \"1758fd0f-1c79-41fe-be74-474ba88962ca\" (UID: \"1758fd0f-1c79-41fe-be74-474ba88962ca\") " Nov 03 01:34:09.724521 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:09.724410 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1758fd0f-1c79-41fe-be74-474ba88962ca-kube-api-access-cnxmw\" (UniqueName: \"kubernetes.io/projected/1758fd0f-1c79-41fe-be74-474ba88962ca-kube-api-access-cnxmw\") pod \"1758fd0f-1c79-41fe-be74-474ba88962ca\" (UID: \"1758fd0f-1c79-41fe-be74-474ba88962ca\") " Nov 03 01:34:09.725536 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:09.725463 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1758fd0f-1c79-41fe-be74-474ba88962ca-plugins-config-volume" pod "1758fd0f-1c79-41fe-be74-474ba88962ca" (UID: "1758fd0f-1c79-41fe-be74-474ba88962ca"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:34:09.726404 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:09.726189 4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1758fd0f-1c79-41fe-be74-474ba88962ca-plugins-config-volume\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:34:09.727215 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:09.727171 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1758fd0f-1c79-41fe-be74-474ba88962ca-kube-api-access-cnxmw" pod "1758fd0f-1c79-41fe-be74-474ba88962ca" (UID: "1758fd0f-1c79-41fe-be74-474ba88962ca"). InnerVolumeSpecName "kube-api-access-cnxmw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:34:09.831396 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:09.827837 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cnxmw\" (UniqueName: \"kubernetes.io/projected/1758fd0f-1c79-41fe-be74-474ba88962ca-kube-api-access-cnxmw\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:34:10.258624 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:10.258582 4421 generic.go:356] "Generic (PLEG): container finished" podID="1758fd0f-1c79-41fe-be74-474ba88962ca" containerID="24254f8efebde82101fba3587d2b4d21e199d080797c90107ca55a1575aa0b90" exitCode=0 Nov 03 01:34:10.259522 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:10.258636 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5bfc51-6c7fc6b99f-kdhcn" Nov 03 01:34:10.259522 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:10.258643 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5bfc51-6c7fc6b99f-kdhcn" event={"ID":"1758fd0f-1c79-41fe-be74-474ba88962ca","Type":"ContainerDied","Data":"24254f8efebde82101fba3587d2b4d21e199d080797c90107ca55a1575aa0b90"} Nov 03 01:34:10.259522 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:10.258705 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5bfc51-6c7fc6b99f-kdhcn" event={"ID":"1758fd0f-1c79-41fe-be74-474ba88962ca","Type":"ContainerDied","Data":"9935c56a019ed82f5a582426dfe6dd1cdce1e64e975c42d0f674b6a6f2a682f0"} Nov 03 01:34:10.259522 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:10.258722 4421 scope.go:122] "RemoveContainer" containerID="24254f8efebde82101fba3587d2b4d21e199d080797c90107ca55a1575aa0b90" Nov 03 01:34:10.272086 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:10.272061 4421 scope.go:122] "RemoveContainer" containerID="24254f8efebde82101fba3587d2b4d21e199d080797c90107ca55a1575aa0b90" Nov 03 01:34:10.272441 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:34:10.272418 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24254f8efebde82101fba3587d2b4d21e199d080797c90107ca55a1575aa0b90\": container with ID starting with 24254f8efebde82101fba3587d2b4d21e199d080797c90107ca55a1575aa0b90 not found: ID does not exist" containerID="24254f8efebde82101fba3587d2b4d21e199d080797c90107ca55a1575aa0b90" Nov 03 01:34:10.273310 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:10.272451 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24254f8efebde82101fba3587d2b4d21e199d080797c90107ca55a1575aa0b90"} err="failed to get container status \"24254f8efebde82101fba3587d2b4d21e199d080797c90107ca55a1575aa0b90\": rpc error: code = NotFound desc = could not find container \"24254f8efebde82101fba3587d2b4d21e199d080797c90107ca55a1575aa0b90\": container with ID starting with 24254f8efebde82101fba3587d2b4d21e199d080797c90107ca55a1575aa0b90 not found: ID does not exist" Nov 03 01:34:10.279432 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:10.279407 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-5bfc51-6c7fc6b99f-kdhcn"] Nov 03 01:34:10.283754 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:10.283728 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-5bfc51-6c7fc6b99f-kdhcn"] Nov 03 01:34:11.984291 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:11.984243 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1758fd0f-1c79-41fe-be74-474ba88962ca" path="/var/lib/kubelet/pods/1758fd0f-1c79-41fe-be74-474ba88962ca/volumes" Nov 03 01:34:12.372385 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:12.372353 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed3dfac7-qrblz"] Nov 03 01:34:12.373764 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:12.373738 4421 cpu_manager.go:414] "containerMap: removing container" podUID="b645b6f2-58be-45c4-9a6f-6785a1047bea" containerName="istio-proxy" Nov 03 01:34:12.373932 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:12.373918 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b645b6f2-58be-45c4-9a6f-6785a1047bea" containerName="istio-proxy" Nov 03 01:34:12.374036 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:12.374025 4421 cpu_manager.go:414] "containerMap: removing container" podUID="1e2345c1-2a99-4b03-b078-eebe1f198589" containerName="istio-proxy" Nov 03 01:34:12.377846 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:12.377822 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e2345c1-2a99-4b03-b078-eebe1f198589" containerName="istio-proxy" Nov 03 01:34:12.378093 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:12.378076 4421 cpu_manager.go:414] "containerMap: removing container" podUID="1e2345c1-2a99-4b03-b078-eebe1f198589" containerName="istio-proxy" Nov 03 01:34:12.378200 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:12.378188 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e2345c1-2a99-4b03-b078-eebe1f198589" containerName="istio-proxy" Nov 03 01:34:12.378366 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:12.378353 4421 cpu_manager.go:414] "containerMap: removing container" podUID="79c7fff7-62df-44a2-aa3e-f24696d71b03" containerName="istio-proxy" Nov 03 01:34:12.378462 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:12.378450 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79c7fff7-62df-44a2-aa3e-f24696d71b03" containerName="istio-proxy" Nov 03 01:34:12.378585 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:12.378572 4421 cpu_manager.go:414] "containerMap: removing container" podUID="157f33b5-87a2-4d03-8288-a8e0cc90de31" containerName="payload-processing" Nov 03 01:34:12.378690 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:12.378679 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="157f33b5-87a2-4d03-8288-a8e0cc90de31" containerName="payload-processing" Nov 03 01:34:12.378795 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:12.378784 4421 cpu_manager.go:414] "containerMap: removing container" podUID="781e1acc-4bc9-489f-8df6-6b5f8c10ba1b" containerName="istio-proxy" Nov 03 01:34:12.378876 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:12.378865 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="781e1acc-4bc9-489f-8df6-6b5f8c10ba1b" containerName="istio-proxy" Nov 03 01:34:12.378968 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:12.378957 4421 cpu_manager.go:414] "containerMap: removing container" podUID="920eea17-797a-4c19-9f09-36f121782d2e" containerName="istio-proxy" Nov 03 01:34:12.379118 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:12.379106 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="920eea17-797a-4c19-9f09-36f121782d2e" containerName="istio-proxy" Nov 03 01:34:12.379227 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:12.379216 4421 cpu_manager.go:414] "containerMap: removing container" podUID="812a530a-f597-4049-a40a-2dcfb8ba6c74" containerName="istio-proxy" Nov 03 01:34:12.380788 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:12.379300 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="812a530a-f597-4049-a40a-2dcfb8ba6c74" containerName="istio-proxy" Nov 03 01:34:12.380788 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:12.379320 4421 cpu_manager.go:414] "containerMap: removing container" podUID="1758fd0f-1c79-41fe-be74-474ba88962ca" containerName="payload-pre-processing" Nov 03 01:34:12.380788 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:12.379328 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1758fd0f-1c79-41fe-be74-474ba88962ca" containerName="payload-pre-processing" Nov 03 01:34:12.380788 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:12.379345 4421 cpu_manager.go:414] "containerMap: removing container" podUID="781e1acc-4bc9-489f-8df6-6b5f8c10ba1b" containerName="istio-proxy" Nov 03 01:34:12.380788 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:12.379352 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="781e1acc-4bc9-489f-8df6-6b5f8c10ba1b" containerName="istio-proxy" Nov 03 01:34:12.380788 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:12.379631 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="79c7fff7-62df-44a2-aa3e-f24696d71b03" containerName="istio-proxy" Nov 03 01:34:12.380788 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:12.379652 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="157f33b5-87a2-4d03-8288-a8e0cc90de31" containerName="payload-processing" Nov 03 01:34:12.380788 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:12.379662 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="781e1acc-4bc9-489f-8df6-6b5f8c10ba1b" containerName="istio-proxy" Nov 03 01:34:12.380788 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:12.379672 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="920eea17-797a-4c19-9f09-36f121782d2e" containerName="istio-proxy" Nov 03 01:34:12.380788 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:12.379685 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="812a530a-f597-4049-a40a-2dcfb8ba6c74" containerName="istio-proxy" Nov 03 01:34:12.380788 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:12.379695 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="1758fd0f-1c79-41fe-be74-474ba88962ca" containerName="payload-pre-processing" Nov 03 01:34:12.380788 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:12.379704 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="781e1acc-4bc9-489f-8df6-6b5f8c10ba1b" containerName="istio-proxy" Nov 03 01:34:12.380788 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:12.379718 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="b645b6f2-58be-45c4-9a6f-6785a1047bea" containerName="istio-proxy" Nov 03 01:34:12.380788 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:12.379727 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e2345c1-2a99-4b03-b078-eebe1f198589" containerName="istio-proxy" Nov 03 01:34:12.396213 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:12.396188 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed3dfac7-qrblz" Nov 03 01:34:12.401469 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:12.401443 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:34:12.408291 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:12.402644 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed3dfac7-qrblz"] Nov 03 01:34:12.408676 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:12.408655 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6q4xp\"" Nov 03 01:34:12.426547 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:12.426511 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fec6be89-9f7a-4db5-aa30-e862be8c53d4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ed3dfac7-qrblz\" (UID: \"fec6be89-9f7a-4db5-aa30-e862be8c53d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed3dfac7-qrblz" Nov 03 01:34:12.456453 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:12.456421 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fec6be89-9f7a-4db5-aa30-e862be8c53d4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ed3dfac7-qrblz\" (UID: \"fec6be89-9f7a-4db5-aa30-e862be8c53d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed3dfac7-qrblz" Nov 03 01:34:12.456845 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:12.456824 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htn4v\" (UniqueName: \"kubernetes.io/projected/fec6be89-9f7a-4db5-aa30-e862be8c53d4-kube-api-access-htn4v\") pod \"maas-api-revoke-sub-ed3dfac7-qrblz\" (UID: \"fec6be89-9f7a-4db5-aa30-e862be8c53d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed3dfac7-qrblz" Nov 03 01:34:12.655458 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:12.622185 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fec6be89-9f7a-4db5-aa30-e862be8c53d4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ed3dfac7-qrblz\" (UID: \"fec6be89-9f7a-4db5-aa30-e862be8c53d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed3dfac7-qrblz" Nov 03 01:34:12.655458 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:12.622312 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fec6be89-9f7a-4db5-aa30-e862be8c53d4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ed3dfac7-qrblz\" (UID: \"fec6be89-9f7a-4db5-aa30-e862be8c53d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed3dfac7-qrblz" Nov 03 01:34:12.655458 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:12.622390 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htn4v\" (UniqueName: \"kubernetes.io/projected/fec6be89-9f7a-4db5-aa30-e862be8c53d4-kube-api-access-htn4v\") pod \"maas-api-revoke-sub-ed3dfac7-qrblz\" (UID: \"fec6be89-9f7a-4db5-aa30-e862be8c53d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed3dfac7-qrblz" Nov 03 01:34:12.655458 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:12.623438 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fec6be89-9f7a-4db5-aa30-e862be8c53d4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ed3dfac7-qrblz\" (UID: \"fec6be89-9f7a-4db5-aa30-e862be8c53d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed3dfac7-qrblz" Nov 03 01:34:12.737299 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:12.716843 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fec6be89-9f7a-4db5-aa30-e862be8c53d4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ed3dfac7-qrblz\" (UID: \"fec6be89-9f7a-4db5-aa30-e862be8c53d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed3dfac7-qrblz" Nov 03 01:34:12.762252 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:12.741968 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-htn4v\" (UniqueName: \"kubernetes.io/projected/fec6be89-9f7a-4db5-aa30-e862be8c53d4-kube-api-access-htn4v\") pod \"maas-api-revoke-sub-ed3dfac7-qrblz\" (UID: \"fec6be89-9f7a-4db5-aa30-e862be8c53d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed3dfac7-qrblz" Nov 03 01:34:12.782297 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:12.780237 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed3dfac7-qrblz" Nov 03 01:34:13.180962 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:13.180914 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed3dfac7-qrblz"] Nov 03 01:34:13.340449 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:13.340411 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed3dfac7-qrblz" event={"ID":"fec6be89-9f7a-4db5-aa30-e862be8c53d4","Type":"ContainerStarted","Data":"ec235067ccaee94418cd24e4ebcc454502e09ab63259454b13d5994b942b576a"} Nov 03 01:34:14.394879 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:14.394474 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed3dfac7-qrblz" event={"ID":"fec6be89-9f7a-4db5-aa30-e862be8c53d4","Type":"ContainerStarted","Data":"0d6bf2d0d19470b6bda9530f344af79ea8233019d657ec96eb95074152b18bf4"} Nov 03 01:34:14.441399 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:14.441319 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed3dfac7-qrblz" podStartSLOduration=2.441299103 podStartE2EDuration="2.441299103s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:34:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:34:14.426823901 +0000 UTC m=+3196.597826999" watchObservedRunningTime="2026-11-03 01:34:14.441299103 +0000 UTC m=+3196.612302208" Nov 03 01:34:14.676909 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:34:14.676827 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 01:34:15.403660 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:15.403619 4421 generic.go:356] "Generic (PLEG): container finished" podID="fec6be89-9f7a-4db5-aa30-e862be8c53d4" containerID="0d6bf2d0d19470b6bda9530f344af79ea8233019d657ec96eb95074152b18bf4" exitCode=0 Nov 03 01:34:15.404860 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:15.403700 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed3dfac7-qrblz" event={"ID":"fec6be89-9f7a-4db5-aa30-e862be8c53d4","Type":"ContainerDied","Data":"0d6bf2d0d19470b6bda9530f344af79ea8233019d657ec96eb95074152b18bf4"} Nov 03 01:34:16.659728 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:16.659699 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed3dfac7-qrblz" Nov 03 01:34:16.699692 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:16.699591 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fec6be89-9f7a-4db5-aa30-e862be8c53d4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fec6be89-9f7a-4db5-aa30-e862be8c53d4-maas-api-xks-ca\") pod \"fec6be89-9f7a-4db5-aa30-e862be8c53d4\" (UID: \"fec6be89-9f7a-4db5-aa30-e862be8c53d4\") " Nov 03 01:34:16.699938 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:16.699748 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fec6be89-9f7a-4db5-aa30-e862be8c53d4-kube-api-access-htn4v\" (UniqueName: \"kubernetes.io/projected/fec6be89-9f7a-4db5-aa30-e862be8c53d4-kube-api-access-htn4v\") pod \"fec6be89-9f7a-4db5-aa30-e862be8c53d4\" (UID: \"fec6be89-9f7a-4db5-aa30-e862be8c53d4\") " Nov 03 01:34:16.699938 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:16.699794 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fec6be89-9f7a-4db5-aa30-e862be8c53d4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fec6be89-9f7a-4db5-aa30-e862be8c53d4-maas-api-openshift-ca\") pod \"fec6be89-9f7a-4db5-aa30-e862be8c53d4\" (UID: \"fec6be89-9f7a-4db5-aa30-e862be8c53d4\") " Nov 03 01:34:16.700480 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:16.700451 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fec6be89-9f7a-4db5-aa30-e862be8c53d4-maas-api-openshift-ca" pod "fec6be89-9f7a-4db5-aa30-e862be8c53d4" (UID: "fec6be89-9f7a-4db5-aa30-e862be8c53d4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:34:16.700618 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:16.700598 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fec6be89-9f7a-4db5-aa30-e862be8c53d4-maas-api-openshift-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:34:16.702096 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:16.702065 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fec6be89-9f7a-4db5-aa30-e862be8c53d4-kube-api-access-htn4v" pod "fec6be89-9f7a-4db5-aa30-e862be8c53d4" (UID: "fec6be89-9f7a-4db5-aa30-e862be8c53d4"). InnerVolumeSpecName "kube-api-access-htn4v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:34:16.702903 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:16.702438 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fec6be89-9f7a-4db5-aa30-e862be8c53d4-maas-api-xks-ca" pod "fec6be89-9f7a-4db5-aa30-e862be8c53d4" (UID: "fec6be89-9f7a-4db5-aa30-e862be8c53d4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:34:16.803171 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:16.803135 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-htn4v\" (UniqueName: \"kubernetes.io/projected/fec6be89-9f7a-4db5-aa30-e862be8c53d4-kube-api-access-htn4v\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:34:16.803171 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:16.803168 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fec6be89-9f7a-4db5-aa30-e862be8c53d4-maas-api-xks-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:34:17.445499 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:17.445469 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed3dfac7-qrblz" Nov 03 01:34:17.447173 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:17.445495 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed3dfac7-qrblz" event={"ID":"fec6be89-9f7a-4db5-aa30-e862be8c53d4","Type":"ContainerDied","Data":"ec235067ccaee94418cd24e4ebcc454502e09ab63259454b13d5994b942b576a"} Nov 03 01:34:17.447173 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:17.445545 4421 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ec235067ccaee94418cd24e4ebcc454502e09ab63259454b13d5994b942b576a" Nov 03 01:34:19.541938 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:19.541901 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-7hhxt"] Nov 03 01:34:19.542527 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:19.542140 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-7hhxt" podUID="2995b3fc-d4cd-49ba-a988-470f220f30f3" containerName="istio-proxy" containerID="cri-o://8c4a39e5f8ae3453ec86398c3489eae2a8a04cf5e22eb2bcc3c2c3acfa27b030" gracePeriod=30 Nov 03 01:34:21.485483 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:21.485449 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-x67zh"] Nov 03 01:34:21.485993 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:21.485653 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-x67zh" podUID="894c4be7-9b51-47c4-87d7-05eee1e8bcbc" containerName="istio-proxy" containerID="cri-o://ff2d9570d49bca7bc41207d1b8e57265a35a82a4f07644979b5ccf8083a18db8" gracePeriod=30 Nov 03 01:34:24.989214 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:24.989166 4421 patch_prober.go:29] interesting pod/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-x67zh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 01:34:24.989753 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:24.989238 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-x67zh" podUID="894c4be7-9b51-47c4-87d7-05eee1e8bcbc" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 01:34:25.305892 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:25.271058 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-7hhxt" Nov 03 01:34:25.345603 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:25.345560 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2995b3fc-d4cd-49ba-a988-470f220f30f3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2995b3fc-d4cd-49ba-a988-470f220f30f3-credential-socket\") pod \"2995b3fc-d4cd-49ba-a988-470f220f30f3\" (UID: \"2995b3fc-d4cd-49ba-a988-470f220f30f3\") " Nov 03 01:34:25.345774 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:25.345655 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2995b3fc-d4cd-49ba-a988-470f220f30f3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2995b3fc-d4cd-49ba-a988-470f220f30f3-istiod-ca-cert\") pod \"2995b3fc-d4cd-49ba-a988-470f220f30f3\" (UID: \"2995b3fc-d4cd-49ba-a988-470f220f30f3\") " Nov 03 01:34:25.345774 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:25.345698 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2995b3fc-d4cd-49ba-a988-470f220f30f3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2995b3fc-d4cd-49ba-a988-470f220f30f3-workload-certs\") pod \"2995b3fc-d4cd-49ba-a988-470f220f30f3\" (UID: \"2995b3fc-d4cd-49ba-a988-470f220f30f3\") " Nov 03 01:34:25.345947 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:25.345777 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2995b3fc-d4cd-49ba-a988-470f220f30f3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2995b3fc-d4cd-49ba-a988-470f220f30f3-workload-socket\") pod \"2995b3fc-d4cd-49ba-a988-470f220f30f3\" (UID: \"2995b3fc-d4cd-49ba-a988-470f220f30f3\") " Nov 03 01:34:25.345947 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:25.345806 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2995b3fc-d4cd-49ba-a988-470f220f30f3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2995b3fc-d4cd-49ba-a988-470f220f30f3-istio-data\") pod \"2995b3fc-d4cd-49ba-a988-470f220f30f3\" (UID: \"2995b3fc-d4cd-49ba-a988-470f220f30f3\") " Nov 03 01:34:25.345947 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:25.345836 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2995b3fc-d4cd-49ba-a988-470f220f30f3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2995b3fc-d4cd-49ba-a988-470f220f30f3-istio-envoy\") pod \"2995b3fc-d4cd-49ba-a988-470f220f30f3\" (UID: \"2995b3fc-d4cd-49ba-a988-470f220f30f3\") " Nov 03 01:34:25.345947 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:25.345861 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2995b3fc-d4cd-49ba-a988-470f220f30f3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2995b3fc-d4cd-49ba-a988-470f220f30f3-istio-podinfo\") pod \"2995b3fc-d4cd-49ba-a988-470f220f30f3\" (UID: \"2995b3fc-d4cd-49ba-a988-470f220f30f3\") " Nov 03 01:34:25.346153 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:25.346126 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2995b3fc-d4cd-49ba-a988-470f220f30f3-istio-token\" (UniqueName: \"kubernetes.io/projected/2995b3fc-d4cd-49ba-a988-470f220f30f3-istio-token\") pod \"2995b3fc-d4cd-49ba-a988-470f220f30f3\" (UID: \"2995b3fc-d4cd-49ba-a988-470f220f30f3\") " Nov 03 01:34:25.346329 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:25.346267 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2995b3fc-d4cd-49ba-a988-470f220f30f3-kube-api-access-gq4hc\" (UniqueName: \"kubernetes.io/projected/2995b3fc-d4cd-49ba-a988-470f220f30f3-kube-api-access-gq4hc\") pod \"2995b3fc-d4cd-49ba-a988-470f220f30f3\" (UID: \"2995b3fc-d4cd-49ba-a988-470f220f30f3\") " Nov 03 01:34:25.373693 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:25.373612 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2995b3fc-d4cd-49ba-a988-470f220f30f3-credential-socket" pod "2995b3fc-d4cd-49ba-a988-470f220f30f3" (UID: "2995b3fc-d4cd-49ba-a988-470f220f30f3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:34:25.373693 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:25.373634 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2995b3fc-d4cd-49ba-a988-470f220f30f3-workload-socket" pod "2995b3fc-d4cd-49ba-a988-470f220f30f3" (UID: "2995b3fc-d4cd-49ba-a988-470f220f30f3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:34:25.374122 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:25.374077 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2995b3fc-d4cd-49ba-a988-470f220f30f3-istio-data" pod "2995b3fc-d4cd-49ba-a988-470f220f30f3" (UID: "2995b3fc-d4cd-49ba-a988-470f220f30f3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:34:25.407846 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:25.374479 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2995b3fc-d4cd-49ba-a988-470f220f30f3-workload-certs" pod "2995b3fc-d4cd-49ba-a988-470f220f30f3" (UID: "2995b3fc-d4cd-49ba-a988-470f220f30f3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:34:25.407846 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:25.374621 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2995b3fc-d4cd-49ba-a988-470f220f30f3-istiod-ca-cert" pod "2995b3fc-d4cd-49ba-a988-470f220f30f3" (UID: "2995b3fc-d4cd-49ba-a988-470f220f30f3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:34:25.407846 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:25.376155 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2995b3fc-d4cd-49ba-a988-470f220f30f3-istio-token" pod "2995b3fc-d4cd-49ba-a988-470f220f30f3" (UID: "2995b3fc-d4cd-49ba-a988-470f220f30f3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:34:25.407846 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:25.376258 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2995b3fc-d4cd-49ba-a988-470f220f30f3-istio-podinfo" pod "2995b3fc-d4cd-49ba-a988-470f220f30f3" (UID: "2995b3fc-d4cd-49ba-a988-470f220f30f3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:34:25.407846 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:25.376331 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2995b3fc-d4cd-49ba-a988-470f220f30f3-kube-api-access-gq4hc" pod "2995b3fc-d4cd-49ba-a988-470f220f30f3" (UID: "2995b3fc-d4cd-49ba-a988-470f220f30f3"). InnerVolumeSpecName "kube-api-access-gq4hc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:34:25.407846 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:25.376496 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2995b3fc-d4cd-49ba-a988-470f220f30f3-istio-envoy" pod "2995b3fc-d4cd-49ba-a988-470f220f30f3" (UID: "2995b3fc-d4cd-49ba-a988-470f220f30f3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:34:25.448498 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:25.448455 4421 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2995b3fc-d4cd-49ba-a988-470f220f30f3-istio-token\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:34:25.448498 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:25.448502 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gq4hc\" (UniqueName: \"kubernetes.io/projected/2995b3fc-d4cd-49ba-a988-470f220f30f3-kube-api-access-gq4hc\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:34:25.515013 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:25.448515 4421 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2995b3fc-d4cd-49ba-a988-470f220f30f3-credential-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:34:25.515013 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:25.448525 4421 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2995b3fc-d4cd-49ba-a988-470f220f30f3-istiod-ca-cert\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:34:25.515013 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:25.448534 4421 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2995b3fc-d4cd-49ba-a988-470f220f30f3-workload-certs\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:34:25.515013 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:25.448543 4421 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2995b3fc-d4cd-49ba-a988-470f220f30f3-workload-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:34:25.515013 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:25.448551 4421 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2995b3fc-d4cd-49ba-a988-470f220f30f3-istio-data\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:34:25.515013 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:25.448559 4421 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2995b3fc-d4cd-49ba-a988-470f220f30f3-istio-envoy\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:34:25.515013 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:25.448569 4421 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2995b3fc-d4cd-49ba-a988-470f220f30f3-istio-podinfo\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:34:25.565018 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:25.564554 4421 generic.go:356] "Generic (PLEG): container finished" podID="2995b3fc-d4cd-49ba-a988-470f220f30f3" containerID="8c4a39e5f8ae3453ec86398c3489eae2a8a04cf5e22eb2bcc3c2c3acfa27b030" exitCode=0 Nov 03 01:34:25.565600 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:25.565563 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-7hhxt" Nov 03 01:34:25.568802 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:25.568539 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-7hhxt" event={"ID":"2995b3fc-d4cd-49ba-a988-470f220f30f3","Type":"ContainerDied","Data":"8c4a39e5f8ae3453ec86398c3489eae2a8a04cf5e22eb2bcc3c2c3acfa27b030"} Nov 03 01:34:25.568802 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:25.568604 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-7hhxt" event={"ID":"2995b3fc-d4cd-49ba-a988-470f220f30f3","Type":"ContainerDied","Data":"cadd4bf7a0abdff58e4f19c35343660f4e10bbdc4acc4d28948379faf91bc44e"} Nov 03 01:34:25.568802 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:25.568637 4421 scope.go:122] "RemoveContainer" containerID="8c4a39e5f8ae3453ec86398c3489eae2a8a04cf5e22eb2bcc3c2c3acfa27b030" Nov 03 01:34:25.606240 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:25.606194 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-7hhxt"] Nov 03 01:34:25.610818 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:25.610786 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-7hhxt"] Nov 03 01:34:25.793645 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:25.793623 4421 scope.go:122] "RemoveContainer" containerID="8c4a39e5f8ae3453ec86398c3489eae2a8a04cf5e22eb2bcc3c2c3acfa27b030" Nov 03 01:34:25.794019 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:34:25.793989 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c4a39e5f8ae3453ec86398c3489eae2a8a04cf5e22eb2bcc3c2c3acfa27b030\": container with ID starting with 8c4a39e5f8ae3453ec86398c3489eae2a8a04cf5e22eb2bcc3c2c3acfa27b030 not found: ID does not exist" containerID="8c4a39e5f8ae3453ec86398c3489eae2a8a04cf5e22eb2bcc3c2c3acfa27b030" Nov 03 01:34:25.794137 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:25.794025 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c4a39e5f8ae3453ec86398c3489eae2a8a04cf5e22eb2bcc3c2c3acfa27b030"} err="failed to get container status \"8c4a39e5f8ae3453ec86398c3489eae2a8a04cf5e22eb2bcc3c2c3acfa27b030\": rpc error: code = NotFound desc = could not find container \"8c4a39e5f8ae3453ec86398c3489eae2a8a04cf5e22eb2bcc3c2c3acfa27b030\": container with ID starting with 8c4a39e5f8ae3453ec86398c3489eae2a8a04cf5e22eb2bcc3c2c3acfa27b030 not found: ID does not exist" Nov 03 01:34:25.984845 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:25.984759 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2995b3fc-d4cd-49ba-a988-470f220f30f3" path="/var/lib/kubelet/pods/2995b3fc-d4cd-49ba-a988-470f220f30f3/volumes" Nov 03 01:34:27.591077 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:27.591040 4421 generic.go:356] "Generic (PLEG): container finished" podID="894c4be7-9b51-47c4-87d7-05eee1e8bcbc" containerID="ff2d9570d49bca7bc41207d1b8e57265a35a82a4f07644979b5ccf8083a18db8" exitCode=0 Nov 03 01:34:27.591566 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:27.591110 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-x67zh" event={"ID":"894c4be7-9b51-47c4-87d7-05eee1e8bcbc","Type":"ContainerDied","Data":"ff2d9570d49bca7bc41207d1b8e57265a35a82a4f07644979b5ccf8083a18db8"} Nov 03 01:34:27.591566 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:27.591173 4421 scope.go:122] "RemoveContainer" containerID="222e52cd77cb070a4e69d96d60d2a4c50bebd5e667232db677487b8cf0b86476" Nov 03 01:34:27.775291 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:27.775256 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-x67zh" Nov 03 01:34:27.916789 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:27.916742 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/894c4be7-9b51-47c4-87d7-05eee1e8bcbc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/894c4be7-9b51-47c4-87d7-05eee1e8bcbc-workload-certs\") pod \"894c4be7-9b51-47c4-87d7-05eee1e8bcbc\" (UID: \"894c4be7-9b51-47c4-87d7-05eee1e8bcbc\") " Nov 03 01:34:27.917314 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:27.916881 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/894c4be7-9b51-47c4-87d7-05eee1e8bcbc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/894c4be7-9b51-47c4-87d7-05eee1e8bcbc-credential-socket\") pod \"894c4be7-9b51-47c4-87d7-05eee1e8bcbc\" (UID: \"894c4be7-9b51-47c4-87d7-05eee1e8bcbc\") " Nov 03 01:34:27.917314 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:27.916952 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/894c4be7-9b51-47c4-87d7-05eee1e8bcbc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/894c4be7-9b51-47c4-87d7-05eee1e8bcbc-istiod-ca-cert\") pod \"894c4be7-9b51-47c4-87d7-05eee1e8bcbc\" (UID: \"894c4be7-9b51-47c4-87d7-05eee1e8bcbc\") " Nov 03 01:34:27.917314 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:27.916996 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/894c4be7-9b51-47c4-87d7-05eee1e8bcbc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/894c4be7-9b51-47c4-87d7-05eee1e8bcbc-workload-socket\") pod \"894c4be7-9b51-47c4-87d7-05eee1e8bcbc\" (UID: \"894c4be7-9b51-47c4-87d7-05eee1e8bcbc\") " Nov 03 01:34:27.917314 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:27.917031 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/894c4be7-9b51-47c4-87d7-05eee1e8bcbc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/894c4be7-9b51-47c4-87d7-05eee1e8bcbc-istio-podinfo\") pod \"894c4be7-9b51-47c4-87d7-05eee1e8bcbc\" (UID: \"894c4be7-9b51-47c4-87d7-05eee1e8bcbc\") " Nov 03 01:34:27.917314 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:27.917113 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/894c4be7-9b51-47c4-87d7-05eee1e8bcbc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/894c4be7-9b51-47c4-87d7-05eee1e8bcbc-istio-envoy\") pod \"894c4be7-9b51-47c4-87d7-05eee1e8bcbc\" (UID: \"894c4be7-9b51-47c4-87d7-05eee1e8bcbc\") " Nov 03 01:34:27.917314 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:27.917180 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/894c4be7-9b51-47c4-87d7-05eee1e8bcbc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/894c4be7-9b51-47c4-87d7-05eee1e8bcbc-istio-data\") pod \"894c4be7-9b51-47c4-87d7-05eee1e8bcbc\" (UID: \"894c4be7-9b51-47c4-87d7-05eee1e8bcbc\") " Nov 03 01:34:27.917314 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:27.917221 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/894c4be7-9b51-47c4-87d7-05eee1e8bcbc-kube-api-access-7wbpc\" (UniqueName: \"kubernetes.io/projected/894c4be7-9b51-47c4-87d7-05eee1e8bcbc-kube-api-access-7wbpc\") pod \"894c4be7-9b51-47c4-87d7-05eee1e8bcbc\" (UID: \"894c4be7-9b51-47c4-87d7-05eee1e8bcbc\") " Nov 03 01:34:27.917832 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:27.917490 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/894c4be7-9b51-47c4-87d7-05eee1e8bcbc-istio-token\" (UniqueName: \"kubernetes.io/projected/894c4be7-9b51-47c4-87d7-05eee1e8bcbc-istio-token\") pod \"894c4be7-9b51-47c4-87d7-05eee1e8bcbc\" (UID: \"894c4be7-9b51-47c4-87d7-05eee1e8bcbc\") " Nov 03 01:34:27.918232 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:27.918194 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/894c4be7-9b51-47c4-87d7-05eee1e8bcbc-workload-socket" pod "894c4be7-9b51-47c4-87d7-05eee1e8bcbc" (UID: "894c4be7-9b51-47c4-87d7-05eee1e8bcbc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:34:27.921165 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:27.918568 4421 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/894c4be7-9b51-47c4-87d7-05eee1e8bcbc-workload-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:34:27.921165 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:27.918838 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/894c4be7-9b51-47c4-87d7-05eee1e8bcbc-workload-certs" pod "894c4be7-9b51-47c4-87d7-05eee1e8bcbc" (UID: "894c4be7-9b51-47c4-87d7-05eee1e8bcbc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:34:27.921165 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:27.919039 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/894c4be7-9b51-47c4-87d7-05eee1e8bcbc-credential-socket" pod "894c4be7-9b51-47c4-87d7-05eee1e8bcbc" (UID: "894c4be7-9b51-47c4-87d7-05eee1e8bcbc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:34:27.921165 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:27.919354 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/894c4be7-9b51-47c4-87d7-05eee1e8bcbc-istiod-ca-cert" pod "894c4be7-9b51-47c4-87d7-05eee1e8bcbc" (UID: "894c4be7-9b51-47c4-87d7-05eee1e8bcbc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:34:27.927359 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:27.921572 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/894c4be7-9b51-47c4-87d7-05eee1e8bcbc-istio-token" pod "894c4be7-9b51-47c4-87d7-05eee1e8bcbc" (UID: "894c4be7-9b51-47c4-87d7-05eee1e8bcbc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:34:27.927359 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:27.921697 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/894c4be7-9b51-47c4-87d7-05eee1e8bcbc-istio-data" pod "894c4be7-9b51-47c4-87d7-05eee1e8bcbc" (UID: "894c4be7-9b51-47c4-87d7-05eee1e8bcbc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:34:27.927359 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:27.926661 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/894c4be7-9b51-47c4-87d7-05eee1e8bcbc-istio-podinfo" pod "894c4be7-9b51-47c4-87d7-05eee1e8bcbc" (UID: "894c4be7-9b51-47c4-87d7-05eee1e8bcbc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:34:27.927639 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:27.927500 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/894c4be7-9b51-47c4-87d7-05eee1e8bcbc-istio-envoy" pod "894c4be7-9b51-47c4-87d7-05eee1e8bcbc" (UID: "894c4be7-9b51-47c4-87d7-05eee1e8bcbc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:34:27.927745 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:27.927723 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/894c4be7-9b51-47c4-87d7-05eee1e8bcbc-kube-api-access-7wbpc" pod "894c4be7-9b51-47c4-87d7-05eee1e8bcbc" (UID: "894c4be7-9b51-47c4-87d7-05eee1e8bcbc"). InnerVolumeSpecName "kube-api-access-7wbpc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:34:28.020114 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:28.020075 4421 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/894c4be7-9b51-47c4-87d7-05eee1e8bcbc-istio-token\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:34:28.020114 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:28.020113 4421 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/894c4be7-9b51-47c4-87d7-05eee1e8bcbc-workload-certs\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:34:28.020361 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:28.020124 4421 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/894c4be7-9b51-47c4-87d7-05eee1e8bcbc-credential-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:34:28.020361 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:28.020133 4421 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/894c4be7-9b51-47c4-87d7-05eee1e8bcbc-istiod-ca-cert\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:34:28.020361 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:28.020143 4421 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/894c4be7-9b51-47c4-87d7-05eee1e8bcbc-istio-podinfo\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:34:28.020361 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:28.020156 4421 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/894c4be7-9b51-47c4-87d7-05eee1e8bcbc-istio-envoy\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:34:28.020361 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:28.020167 4421 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/894c4be7-9b51-47c4-87d7-05eee1e8bcbc-istio-data\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:34:28.020361 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:28.020176 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7wbpc\" (UniqueName: \"kubernetes.io/projected/894c4be7-9b51-47c4-87d7-05eee1e8bcbc-kube-api-access-7wbpc\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:34:28.226440 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:28.226330 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-b5cbp"] Nov 03 01:34:28.228031 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:28.227998 4421 cpu_manager.go:414] "containerMap: removing container" podUID="2995b3fc-d4cd-49ba-a988-470f220f30f3" containerName="istio-proxy" Nov 03 01:34:28.229411 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:28.229389 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2995b3fc-d4cd-49ba-a988-470f220f30f3" containerName="istio-proxy" Nov 03 01:34:28.229593 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:28.229579 4421 cpu_manager.go:414] "containerMap: removing container" podUID="fec6be89-9f7a-4db5-aa30-e862be8c53d4" containerName="revoke-keys" Nov 03 01:34:28.229705 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:28.229691 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fec6be89-9f7a-4db5-aa30-e862be8c53d4" containerName="revoke-keys" Nov 03 01:34:28.229867 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:28.229854 4421 cpu_manager.go:414] "containerMap: removing container" podUID="894c4be7-9b51-47c4-87d7-05eee1e8bcbc" containerName="istio-proxy" Nov 03 01:34:28.229965 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:28.229955 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="894c4be7-9b51-47c4-87d7-05eee1e8bcbc" containerName="istio-proxy" Nov 03 01:34:28.230086 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:28.230077 4421 cpu_manager.go:414] "containerMap: removing container" podUID="894c4be7-9b51-47c4-87d7-05eee1e8bcbc" containerName="istio-proxy" Nov 03 01:34:28.237226 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:28.230155 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="894c4be7-9b51-47c4-87d7-05eee1e8bcbc" containerName="istio-proxy" Nov 03 01:34:28.237226 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:28.230378 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="894c4be7-9b51-47c4-87d7-05eee1e8bcbc" containerName="istio-proxy" Nov 03 01:34:28.237226 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:28.230389 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e2345c1-2a99-4b03-b078-eebe1f198589" containerName="istio-proxy" Nov 03 01:34:28.237226 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:28.230409 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="2995b3fc-d4cd-49ba-a988-470f220f30f3" containerName="istio-proxy" Nov 03 01:34:28.237226 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:28.230422 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="fec6be89-9f7a-4db5-aa30-e862be8c53d4" containerName="revoke-keys" Nov 03 01:34:28.255460 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:28.246742 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-b5cbp" Nov 03 01:34:28.276979 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:28.276894 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-b5cbp"] Nov 03 01:34:28.328897 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:28.328853 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b553f012-3d8c-4a9a-b514-410db05c2f0a-credential-socket\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-b5cbp\" (UID: \"b553f012-3d8c-4a9a-b514-410db05c2f0a\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-b5cbp" Nov 03 01:34:28.329075 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:28.328976 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b553f012-3d8c-4a9a-b514-410db05c2f0a-istio-data\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-b5cbp\" (UID: \"b553f012-3d8c-4a9a-b514-410db05c2f0a\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-b5cbp" Nov 03 01:34:28.329075 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:28.329015 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b553f012-3d8c-4a9a-b514-410db05c2f0a-istiod-ca-cert\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-b5cbp\" (UID: \"b553f012-3d8c-4a9a-b514-410db05c2f0a\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-b5cbp" Nov 03 01:34:28.329075 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:28.329054 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b553f012-3d8c-4a9a-b514-410db05c2f0a-workload-certs\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-b5cbp\" (UID: \"b553f012-3d8c-4a9a-b514-410db05c2f0a\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-b5cbp" Nov 03 01:34:28.329192 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:28.329113 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b553f012-3d8c-4a9a-b514-410db05c2f0a-istio-podinfo\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-b5cbp\" (UID: \"b553f012-3d8c-4a9a-b514-410db05c2f0a\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-b5cbp" Nov 03 01:34:28.329312 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:28.329261 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b553f012-3d8c-4a9a-b514-410db05c2f0a-istio-token\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-b5cbp\" (UID: \"b553f012-3d8c-4a9a-b514-410db05c2f0a\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-b5cbp" Nov 03 01:34:28.329391 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:28.329376 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b553f012-3d8c-4a9a-b514-410db05c2f0a-workload-socket\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-b5cbp\" (UID: \"b553f012-3d8c-4a9a-b514-410db05c2f0a\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-b5cbp" Nov 03 01:34:28.329494 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:28.329475 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jh68r\" (UniqueName: \"kubernetes.io/projected/b553f012-3d8c-4a9a-b514-410db05c2f0a-kube-api-access-jh68r\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-b5cbp\" (UID: \"b553f012-3d8c-4a9a-b514-410db05c2f0a\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-b5cbp" Nov 03 01:34:28.329663 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:28.329648 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b553f012-3d8c-4a9a-b514-410db05c2f0a-istio-envoy\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-b5cbp\" (UID: \"b553f012-3d8c-4a9a-b514-410db05c2f0a\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-b5cbp" Nov 03 01:34:28.431662 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:28.431618 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b553f012-3d8c-4a9a-b514-410db05c2f0a-credential-socket\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-b5cbp\" (UID: \"b553f012-3d8c-4a9a-b514-410db05c2f0a\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-b5cbp" Nov 03 01:34:28.431954 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:28.431697 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b553f012-3d8c-4a9a-b514-410db05c2f0a-istio-data\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-b5cbp\" (UID: \"b553f012-3d8c-4a9a-b514-410db05c2f0a\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-b5cbp" Nov 03 01:34:28.431954 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:28.431719 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b553f012-3d8c-4a9a-b514-410db05c2f0a-istiod-ca-cert\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-b5cbp\" (UID: \"b553f012-3d8c-4a9a-b514-410db05c2f0a\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-b5cbp" Nov 03 01:34:28.431954 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:28.431847 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b553f012-3d8c-4a9a-b514-410db05c2f0a-workload-certs\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-b5cbp\" (UID: \"b553f012-3d8c-4a9a-b514-410db05c2f0a\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-b5cbp" Nov 03 01:34:28.431954 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:28.431900 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b553f012-3d8c-4a9a-b514-410db05c2f0a-istio-podinfo\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-b5cbp\" (UID: \"b553f012-3d8c-4a9a-b514-410db05c2f0a\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-b5cbp" Nov 03 01:34:28.432343 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:28.431992 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b553f012-3d8c-4a9a-b514-410db05c2f0a-istio-token\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-b5cbp\" (UID: \"b553f012-3d8c-4a9a-b514-410db05c2f0a\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-b5cbp" Nov 03 01:34:28.432343 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:28.432076 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b553f012-3d8c-4a9a-b514-410db05c2f0a-credential-socket\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-b5cbp\" (UID: \"b553f012-3d8c-4a9a-b514-410db05c2f0a\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-b5cbp" Nov 03 01:34:28.432343 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:28.432110 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b553f012-3d8c-4a9a-b514-410db05c2f0a-workload-socket\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-b5cbp\" (UID: \"b553f012-3d8c-4a9a-b514-410db05c2f0a\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-b5cbp" Nov 03 01:34:28.432343 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:28.432227 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jh68r\" (UniqueName: \"kubernetes.io/projected/b553f012-3d8c-4a9a-b514-410db05c2f0a-kube-api-access-jh68r\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-b5cbp\" (UID: \"b553f012-3d8c-4a9a-b514-410db05c2f0a\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-b5cbp" Nov 03 01:34:28.432343 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:28.432243 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b553f012-3d8c-4a9a-b514-410db05c2f0a-workload-certs\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-b5cbp\" (UID: \"b553f012-3d8c-4a9a-b514-410db05c2f0a\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-b5cbp" Nov 03 01:34:28.432663 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:28.432423 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b553f012-3d8c-4a9a-b514-410db05c2f0a-istio-envoy\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-b5cbp\" (UID: \"b553f012-3d8c-4a9a-b514-410db05c2f0a\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-b5cbp" Nov 03 01:34:28.432663 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:28.432428 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b553f012-3d8c-4a9a-b514-410db05c2f0a-workload-socket\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-b5cbp\" (UID: \"b553f012-3d8c-4a9a-b514-410db05c2f0a\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-b5cbp" Nov 03 01:34:28.432663 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:28.432531 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b553f012-3d8c-4a9a-b514-410db05c2f0a-istiod-ca-cert\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-b5cbp\" (UID: \"b553f012-3d8c-4a9a-b514-410db05c2f0a\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-b5cbp" Nov 03 01:34:28.432790 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:28.432712 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b553f012-3d8c-4a9a-b514-410db05c2f0a-istio-data\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-b5cbp\" (UID: \"b553f012-3d8c-4a9a-b514-410db05c2f0a\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-b5cbp" Nov 03 01:34:28.434854 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:28.434827 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b553f012-3d8c-4a9a-b514-410db05c2f0a-istio-envoy\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-b5cbp\" (UID: \"b553f012-3d8c-4a9a-b514-410db05c2f0a\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-b5cbp" Nov 03 01:34:28.435226 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:28.435207 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b553f012-3d8c-4a9a-b514-410db05c2f0a-istio-podinfo\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-b5cbp\" (UID: \"b553f012-3d8c-4a9a-b514-410db05c2f0a\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-b5cbp" Nov 03 01:34:28.441375 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:28.441343 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b553f012-3d8c-4a9a-b514-410db05c2f0a-istio-token\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-b5cbp\" (UID: \"b553f012-3d8c-4a9a-b514-410db05c2f0a\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-b5cbp" Nov 03 01:34:28.441480 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:28.441443 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jh68r\" (UniqueName: \"kubernetes.io/projected/b553f012-3d8c-4a9a-b514-410db05c2f0a-kube-api-access-jh68r\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-b5cbp\" (UID: \"b553f012-3d8c-4a9a-b514-410db05c2f0a\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-b5cbp" Nov 03 01:34:28.588941 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:28.588910 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-b5cbp" Nov 03 01:34:28.601394 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:28.601360 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-x67zh" event={"ID":"894c4be7-9b51-47c4-87d7-05eee1e8bcbc","Type":"ContainerDied","Data":"fdd2613d60209122cf80912ea65b7a571f6b0c50b2f5889a0664177f7f022bb3"} Nov 03 01:34:28.601394 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:28.601390 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-x67zh" Nov 03 01:34:28.601897 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:28.601427 4421 scope.go:122] "RemoveContainer" containerID="ff2d9570d49bca7bc41207d1b8e57265a35a82a4f07644979b5ccf8083a18db8" Nov 03 01:34:28.646733 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:28.646703 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-x67zh"] Nov 03 01:34:28.651336 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:28.651313 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-x67zh"] Nov 03 01:34:28.976294 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:28.975950 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-b5cbp"] Nov 03 01:34:29.013400 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:29.011219 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:34:29.013400 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:29.011341 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:34:29.013400 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:29.011383 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:34:29.568110 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:29.568067 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-wxr6x"] Nov 03 01:34:29.568502 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:29.568337 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-wxr6x" podUID="d56b90c8-05c3-4c59-a204-79eb70d4a172" containerName="istio-proxy" containerID="cri-o://456097de640039b54e16601e495e6a3f66020f72fa020b0ddbba108503f0990f" gracePeriod=30 Nov 03 01:34:29.621332 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:29.618288 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-b5cbp" event={"ID":"b553f012-3d8c-4a9a-b514-410db05c2f0a","Type":"ContainerStarted","Data":"9e37991ee0afd467692f5ef93bf3b08d01e9305ee77387dcf1874c6cedb5f047"} Nov 03 01:34:29.621332 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:29.618356 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-b5cbp" event={"ID":"b553f012-3d8c-4a9a-b514-410db05c2f0a","Type":"ContainerStarted","Data":"b475864076a988b4cbd349c0b60c8209b70292fa73936ff19a8a8b88f6d16574"} Nov 03 01:34:29.640870 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:29.640792 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-b5cbp" podStartSLOduration=1.64077062 podStartE2EDuration="1.64077062s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:34:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:34:29.638469874 +0000 UTC m=+3211.809472962" watchObservedRunningTime="2026-11-03 01:34:29.64077062 +0000 UTC m=+3211.811773718" Nov 03 01:34:29.983169 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:29.983079 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="894c4be7-9b51-47c4-87d7-05eee1e8bcbc" path="/var/lib/kubelet/pods/894c4be7-9b51-47c4-87d7-05eee1e8bcbc/volumes" Nov 03 01:34:30.451300 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:30.448069 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9gm5g"] Nov 03 01:34:30.451300 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:30.448435 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9gm5g" podUID="8b76e085-63e9-4361-bab9-d6a88b76c35a" containerName="istio-proxy" containerID="cri-o://ed0aac922366879df461e8e0d690987aed3bd8654c2d20d1ae9b1f4c18a4bf55" gracePeriod=30 Nov 03 01:34:30.589820 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:30.589769 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-b5cbp" Nov 03 01:34:30.591721 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:30.591692 4421 patch_prober.go:29] interesting pod/e2e-mt-737bf2d8-openshift-default-56dc96b957-b5cbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.93:15021/healthz/ready\": dial tcp 10.134.1.93:15021: connect: connection refused" start-of-body= Nov 03 01:34:30.591886 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:30.591753 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-b5cbp" podUID="b553f012-3d8c-4a9a-b514-410db05c2f0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.93:15021/healthz/ready\": dial tcp 10.134.1.93:15021: connect: connection refused" Nov 03 01:34:31.589523 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:31.589488 4421 patch_prober.go:29] interesting pod/e2e-mt-737bf2d8-openshift-default-56dc96b957-b5cbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.93:15021/healthz/ready\": dial tcp 10.134.1.93:15021: connect: connection refused" start-of-body= Nov 03 01:34:31.590061 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:31.589544 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-b5cbp" podUID="b553f012-3d8c-4a9a-b514-410db05c2f0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.93:15021/healthz/ready\": dial tcp 10.134.1.93:15021: connect: connection refused" Nov 03 01:34:31.768514 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:31.768468 4421 patch_prober.go:29] interesting pod/e2e-mt-737bf2d8-openshift-default-56dc96b957-9v2m2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 01:34:31.768702 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:31.768539 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-9v2m2" podUID="52a0f195-418d-42be-8222-b030c8f6ba55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 01:34:32.600300 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:32.595266 4421 patch_prober.go:29] interesting pod/e2e-mt-737bf2d8-openshift-default-56dc96b957-b5cbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.93:15021/healthz/ready\": dial tcp 10.134.1.93:15021: connect: connection refused" start-of-body= Nov 03 01:34:32.600300 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:32.595343 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-b5cbp" podUID="b553f012-3d8c-4a9a-b514-410db05c2f0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.93:15021/healthz/ready\": dial tcp 10.134.1.93:15021: connect: connection refused" Nov 03 01:34:33.590321 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:33.590250 4421 patch_prober.go:29] interesting pod/e2e-mt-737bf2d8-openshift-default-56dc96b957-b5cbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.93:15021/healthz/ready\": dial tcp 10.134.1.93:15021: connect: connection refused" start-of-body= Nov 03 01:34:33.590525 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:33.590341 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-b5cbp" podUID="b553f012-3d8c-4a9a-b514-410db05c2f0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.93:15021/healthz/ready\": dial tcp 10.134.1.93:15021: connect: connection refused" Nov 03 01:34:33.684193 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:33.684160 4421 generic.go:356] "Generic (PLEG): container finished" podID="52a0f195-418d-42be-8222-b030c8f6ba55" containerID="41749612022ba7f45899b639631594c7f0c9a087212cf7b716fb6eddb81975d3" exitCode=0 Nov 03 01:34:33.684651 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:33.684227 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-9v2m2" event={"ID":"52a0f195-418d-42be-8222-b030c8f6ba55","Type":"ContainerDied","Data":"41749612022ba7f45899b639631594c7f0c9a087212cf7b716fb6eddb81975d3"} Nov 03 01:34:33.684749 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:33.684734 4421 scope.go:122] "RemoveContainer" containerID="41749612022ba7f45899b639631594c7f0c9a087212cf7b716fb6eddb81975d3" Nov 03 01:34:33.686860 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:33.686825 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:34:33.686976 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:33.686914 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:34:33.686976 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:33.686967 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:34:34.505156 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:34.505119 4421 patch_prober.go:29] interesting pod/e2e-extmod-a-d25759-openshift-default-d47fd9785-wxr6x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 01:34:34.505350 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:34.505172 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-wxr6x" podUID="d56b90c8-05c3-4c59-a204-79eb70d4a172" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 01:34:34.591347 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:34.590243 4421 patch_prober.go:29] interesting pod/e2e-mt-737bf2d8-openshift-default-56dc96b957-b5cbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.93:15021/healthz/ready\": dial tcp 10.134.1.93:15021: connect: connection refused" start-of-body= Nov 03 01:34:34.591347 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:34.590313 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-b5cbp" podUID="b553f012-3d8c-4a9a-b514-410db05c2f0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.93:15021/healthz/ready\": dial tcp 10.134.1.93:15021: connect: connection refused" Nov 03 01:34:34.694056 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:34.694024 4421 generic.go:356] "Generic (PLEG): container finished" podID="d56b90c8-05c3-4c59-a204-79eb70d4a172" containerID="456097de640039b54e16601e495e6a3f66020f72fa020b0ddbba108503f0990f" exitCode=0 Nov 03 01:34:34.694516 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:34.694098 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-wxr6x" event={"ID":"d56b90c8-05c3-4c59-a204-79eb70d4a172","Type":"ContainerDied","Data":"456097de640039b54e16601e495e6a3f66020f72fa020b0ddbba108503f0990f"} Nov 03 01:34:34.696747 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:34.696715 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-9v2m2" event={"ID":"52a0f195-418d-42be-8222-b030c8f6ba55","Type":"ContainerStarted","Data":"ac321584e82282af33485c24ee8379fdb77edee07abd643508c5b845fd39082b"} Nov 03 01:34:35.164686 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:35.164654 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-wxr6x" Nov 03 01:34:35.270985 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:35.270949 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-ddd653-openshift-default-779ff6465-6hnrl"] Nov 03 01:34:35.272202 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:35.272162 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-ddd653-openshift-default-779ff6465-6hnrl" podUID="8b337af9-fa55-447d-9555-7852367061f5" containerName="istio-proxy" containerID="cri-o://9c8617669bb3974e29df388e4cd520abd543b06add22c069b572607542e2224e" gracePeriod=30 Nov 03 01:34:35.352032 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:35.351882 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d56b90c8-05c3-4c59-a204-79eb70d4a172-kube-api-access-tjnw8\" (UniqueName: \"kubernetes.io/projected/d56b90c8-05c3-4c59-a204-79eb70d4a172-kube-api-access-tjnw8\") pod \"d56b90c8-05c3-4c59-a204-79eb70d4a172\" (UID: \"d56b90c8-05c3-4c59-a204-79eb70d4a172\") " Nov 03 01:34:35.374016 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:35.352333 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d56b90c8-05c3-4c59-a204-79eb70d4a172-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d56b90c8-05c3-4c59-a204-79eb70d4a172-credential-socket\") pod \"d56b90c8-05c3-4c59-a204-79eb70d4a172\" (UID: \"d56b90c8-05c3-4c59-a204-79eb70d4a172\") " Nov 03 01:34:35.374016 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:35.373793 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d56b90c8-05c3-4c59-a204-79eb70d4a172-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d56b90c8-05c3-4c59-a204-79eb70d4a172-istio-envoy\") pod \"d56b90c8-05c3-4c59-a204-79eb70d4a172\" (UID: \"d56b90c8-05c3-4c59-a204-79eb70d4a172\") " Nov 03 01:34:35.374016 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:35.373835 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d56b90c8-05c3-4c59-a204-79eb70d4a172-istio-token\" (UniqueName: \"kubernetes.io/projected/d56b90c8-05c3-4c59-a204-79eb70d4a172-istio-token\") pod \"d56b90c8-05c3-4c59-a204-79eb70d4a172\" (UID: \"d56b90c8-05c3-4c59-a204-79eb70d4a172\") " Nov 03 01:34:35.374016 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:35.373930 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d56b90c8-05c3-4c59-a204-79eb70d4a172-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d56b90c8-05c3-4c59-a204-79eb70d4a172-istio-data\") pod \"d56b90c8-05c3-4c59-a204-79eb70d4a172\" (UID: \"d56b90c8-05c3-4c59-a204-79eb70d4a172\") " Nov 03 01:34:35.374463 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:35.374129 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d56b90c8-05c3-4c59-a204-79eb70d4a172-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d56b90c8-05c3-4c59-a204-79eb70d4a172-workload-socket\") pod \"d56b90c8-05c3-4c59-a204-79eb70d4a172\" (UID: \"d56b90c8-05c3-4c59-a204-79eb70d4a172\") " Nov 03 01:34:35.374463 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:35.374183 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d56b90c8-05c3-4c59-a204-79eb70d4a172-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d56b90c8-05c3-4c59-a204-79eb70d4a172-workload-certs\") pod \"d56b90c8-05c3-4c59-a204-79eb70d4a172\" (UID: \"d56b90c8-05c3-4c59-a204-79eb70d4a172\") " Nov 03 01:34:35.374463 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:35.374219 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d56b90c8-05c3-4c59-a204-79eb70d4a172-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d56b90c8-05c3-4c59-a204-79eb70d4a172-istio-podinfo\") pod \"d56b90c8-05c3-4c59-a204-79eb70d4a172\" (UID: \"d56b90c8-05c3-4c59-a204-79eb70d4a172\") " Nov 03 01:34:35.374463 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:35.374442 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d56b90c8-05c3-4c59-a204-79eb70d4a172-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d56b90c8-05c3-4c59-a204-79eb70d4a172-istiod-ca-cert\") pod \"d56b90c8-05c3-4c59-a204-79eb70d4a172\" (UID: \"d56b90c8-05c3-4c59-a204-79eb70d4a172\") " Nov 03 01:34:35.377929 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:35.377883 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d56b90c8-05c3-4c59-a204-79eb70d4a172-istio-data" pod "d56b90c8-05c3-4c59-a204-79eb70d4a172" (UID: "d56b90c8-05c3-4c59-a204-79eb70d4a172"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:34:35.378195 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:35.378165 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d56b90c8-05c3-4c59-a204-79eb70d4a172-credential-socket" pod "d56b90c8-05c3-4c59-a204-79eb70d4a172" (UID: "d56b90c8-05c3-4c59-a204-79eb70d4a172"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:34:35.378481 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:35.378438 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d56b90c8-05c3-4c59-a204-79eb70d4a172-workload-certs" pod "d56b90c8-05c3-4c59-a204-79eb70d4a172" (UID: "d56b90c8-05c3-4c59-a204-79eb70d4a172"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:34:35.381137 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:35.381112 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d56b90c8-05c3-4c59-a204-79eb70d4a172-kube-api-access-tjnw8" pod "d56b90c8-05c3-4c59-a204-79eb70d4a172" (UID: "d56b90c8-05c3-4c59-a204-79eb70d4a172"). InnerVolumeSpecName "kube-api-access-tjnw8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:34:35.381284 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:35.381239 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d56b90c8-05c3-4c59-a204-79eb70d4a172-istio-token" pod "d56b90c8-05c3-4c59-a204-79eb70d4a172" (UID: "d56b90c8-05c3-4c59-a204-79eb70d4a172"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:34:35.381405 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:35.381388 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d56b90c8-05c3-4c59-a204-79eb70d4a172-workload-socket" pod "d56b90c8-05c3-4c59-a204-79eb70d4a172" (UID: "d56b90c8-05c3-4c59-a204-79eb70d4a172"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:34:35.381482 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:35.381400 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d56b90c8-05c3-4c59-a204-79eb70d4a172-istiod-ca-cert" pod "d56b90c8-05c3-4c59-a204-79eb70d4a172" (UID: "d56b90c8-05c3-4c59-a204-79eb70d4a172"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:34:35.381548 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:35.381530 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d56b90c8-05c3-4c59-a204-79eb70d4a172-istio-envoy" pod "d56b90c8-05c3-4c59-a204-79eb70d4a172" (UID: "d56b90c8-05c3-4c59-a204-79eb70d4a172"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:34:35.383637 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:35.383611 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d56b90c8-05c3-4c59-a204-79eb70d4a172-istio-podinfo" pod "d56b90c8-05c3-4c59-a204-79eb70d4a172" (UID: "d56b90c8-05c3-4c59-a204-79eb70d4a172"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:34:35.454391 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:35.434732 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-9v2m2" Nov 03 01:34:35.454391 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:35.438388 4421 patch_prober.go:29] interesting pod/e2e-mt-737bf2d8-openshift-default-56dc96b957-9v2m2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.89:15021/healthz/ready\": dial tcp 10.134.1.89:15021: connect: connection refused" start-of-body= Nov 03 01:34:35.454391 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:35.438446 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-9v2m2" podUID="52a0f195-418d-42be-8222-b030c8f6ba55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.89:15021/healthz/ready\": dial tcp 10.134.1.89:15021: connect: connection refused" Nov 03 01:34:35.476713 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:35.476682 4421 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d56b90c8-05c3-4c59-a204-79eb70d4a172-istiod-ca-cert\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:34:35.476713 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:35.476714 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjnw8\" (UniqueName: \"kubernetes.io/projected/d56b90c8-05c3-4c59-a204-79eb70d4a172-kube-api-access-tjnw8\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:34:35.476876 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:35.476725 4421 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d56b90c8-05c3-4c59-a204-79eb70d4a172-credential-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:34:35.476876 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:35.476734 4421 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d56b90c8-05c3-4c59-a204-79eb70d4a172-istio-envoy\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:34:35.476876 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:35.476743 4421 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d56b90c8-05c3-4c59-a204-79eb70d4a172-istio-token\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:34:35.476876 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:35.476751 4421 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d56b90c8-05c3-4c59-a204-79eb70d4a172-istio-data\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:34:35.476876 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:35.476759 4421 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d56b90c8-05c3-4c59-a204-79eb70d4a172-workload-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:34:35.476876 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:35.476767 4421 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d56b90c8-05c3-4c59-a204-79eb70d4a172-workload-certs\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:34:35.476876 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:35.476775 4421 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d56b90c8-05c3-4c59-a204-79eb70d4a172-istio-podinfo\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:34:35.590124 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:35.589672 4421 patch_prober.go:29] interesting pod/e2e-mt-737bf2d8-openshift-default-56dc96b957-b5cbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.93:15021/healthz/ready\": dial tcp 10.134.1.93:15021: connect: connection refused" start-of-body= Nov 03 01:34:35.590124 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:35.589734 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-b5cbp" podUID="b553f012-3d8c-4a9a-b514-410db05c2f0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.93:15021/healthz/ready\": dial tcp 10.134.1.93:15021: connect: connection refused" Nov 03 01:34:35.788300 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:35.777379 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-wxr6x" event={"ID":"d56b90c8-05c3-4c59-a204-79eb70d4a172","Type":"ContainerDied","Data":"6747158065bb2d75cfea8f16ff5d4a2a88df4c6717714b41e088255be4d98f2e"} Nov 03 01:34:35.788300 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:35.777458 4421 scope.go:122] "RemoveContainer" containerID="456097de640039b54e16601e495e6a3f66020f72fa020b0ddbba108503f0990f" Nov 03 01:34:35.788300 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:35.777661 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-wxr6x" Nov 03 01:34:35.803383 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:35.803347 4421 generic.go:356] "Generic (PLEG): container finished" podID="8b76e085-63e9-4361-bab9-d6a88b76c35a" containerID="ed0aac922366879df461e8e0d690987aed3bd8654c2d20d1ae9b1f4c18a4bf55" exitCode=0 Nov 03 01:34:35.804480 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:35.804448 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9gm5g" event={"ID":"8b76e085-63e9-4361-bab9-d6a88b76c35a","Type":"ContainerDied","Data":"ed0aac922366879df461e8e0d690987aed3bd8654c2d20d1ae9b1f4c18a4bf55"} Nov 03 01:34:35.862205 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:35.862164 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-wxr6x"] Nov 03 01:34:35.869883 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:35.869738 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-wxr6x"] Nov 03 01:34:36.051219 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:36.051183 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d56b90c8-05c3-4c59-a204-79eb70d4a172" path="/var/lib/kubelet/pods/d56b90c8-05c3-4c59-a204-79eb70d4a172/volumes" Nov 03 01:34:36.088760 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:36.088731 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-020af9-openshift-default-d77df45d5-vjg9x"] Nov 03 01:34:36.088948 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:36.088927 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-020af9-openshift-default-d77df45d5-vjg9x" podUID="8fb3426e-5493-4655-a9e9-e011e9073012" containerName="istio-proxy" containerID="cri-o://cb7345056853e5742c1811f4fc210b604d8848d627f1ea560ff1b0fc2d781373" gracePeriod=30 Nov 03 01:34:36.099781 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:36.099756 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9gm5g" Nov 03 01:34:36.256090 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:36.253580 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b76e085-63e9-4361-bab9-d6a88b76c35a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b76e085-63e9-4361-bab9-d6a88b76c35a-workload-certs\") pod \"8b76e085-63e9-4361-bab9-d6a88b76c35a\" (UID: \"8b76e085-63e9-4361-bab9-d6a88b76c35a\") " Nov 03 01:34:36.256090 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:36.253863 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b76e085-63e9-4361-bab9-d6a88b76c35a-istio-token\" (UniqueName: \"kubernetes.io/projected/8b76e085-63e9-4361-bab9-d6a88b76c35a-istio-token\") pod \"8b76e085-63e9-4361-bab9-d6a88b76c35a\" (UID: \"8b76e085-63e9-4361-bab9-d6a88b76c35a\") " Nov 03 01:34:36.256090 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:36.253937 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b76e085-63e9-4361-bab9-d6a88b76c35a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b76e085-63e9-4361-bab9-d6a88b76c35a-istio-envoy\") pod \"8b76e085-63e9-4361-bab9-d6a88b76c35a\" (UID: \"8b76e085-63e9-4361-bab9-d6a88b76c35a\") " Nov 03 01:34:36.256090 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:36.253994 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b76e085-63e9-4361-bab9-d6a88b76c35a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b76e085-63e9-4361-bab9-d6a88b76c35a-credential-socket\") pod \"8b76e085-63e9-4361-bab9-d6a88b76c35a\" (UID: \"8b76e085-63e9-4361-bab9-d6a88b76c35a\") " Nov 03 01:34:36.256090 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:36.254053 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b76e085-63e9-4361-bab9-d6a88b76c35a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b76e085-63e9-4361-bab9-d6a88b76c35a-istio-data\") pod \"8b76e085-63e9-4361-bab9-d6a88b76c35a\" (UID: \"8b76e085-63e9-4361-bab9-d6a88b76c35a\") " Nov 03 01:34:36.256090 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:36.254121 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8b76e085-63e9-4361-bab9-d6a88b76c35a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b76e085-63e9-4361-bab9-d6a88b76c35a-istio-podinfo\") pod \"8b76e085-63e9-4361-bab9-d6a88b76c35a\" (UID: \"8b76e085-63e9-4361-bab9-d6a88b76c35a\") " Nov 03 01:34:36.256090 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:36.254199 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b76e085-63e9-4361-bab9-d6a88b76c35a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b76e085-63e9-4361-bab9-d6a88b76c35a-workload-socket\") pod \"8b76e085-63e9-4361-bab9-d6a88b76c35a\" (UID: \"8b76e085-63e9-4361-bab9-d6a88b76c35a\") " Nov 03 01:34:36.256090 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:36.254266 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b76e085-63e9-4361-bab9-d6a88b76c35a-kube-api-access-xsw98\" (UniqueName: \"kubernetes.io/projected/8b76e085-63e9-4361-bab9-d6a88b76c35a-kube-api-access-xsw98\") pod \"8b76e085-63e9-4361-bab9-d6a88b76c35a\" (UID: \"8b76e085-63e9-4361-bab9-d6a88b76c35a\") " Nov 03 01:34:36.256090 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:36.255669 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b76e085-63e9-4361-bab9-d6a88b76c35a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b76e085-63e9-4361-bab9-d6a88b76c35a-istiod-ca-cert\") pod \"8b76e085-63e9-4361-bab9-d6a88b76c35a\" (UID: \"8b76e085-63e9-4361-bab9-d6a88b76c35a\") " Nov 03 01:34:36.256090 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:36.255045 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b76e085-63e9-4361-bab9-d6a88b76c35a-credential-socket" pod "8b76e085-63e9-4361-bab9-d6a88b76c35a" (UID: "8b76e085-63e9-4361-bab9-d6a88b76c35a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:34:36.256090 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:36.255228 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b76e085-63e9-4361-bab9-d6a88b76c35a-istio-data" pod "8b76e085-63e9-4361-bab9-d6a88b76c35a" (UID: "8b76e085-63e9-4361-bab9-d6a88b76c35a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:34:36.256090 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:36.255337 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b76e085-63e9-4361-bab9-d6a88b76c35a-workload-certs" pod "8b76e085-63e9-4361-bab9-d6a88b76c35a" (UID: "8b76e085-63e9-4361-bab9-d6a88b76c35a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:34:36.256090 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:36.255629 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b76e085-63e9-4361-bab9-d6a88b76c35a-workload-socket" pod "8b76e085-63e9-4361-bab9-d6a88b76c35a" (UID: "8b76e085-63e9-4361-bab9-d6a88b76c35a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:34:36.257059 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:36.256150 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b76e085-63e9-4361-bab9-d6a88b76c35a-istiod-ca-cert" pod "8b76e085-63e9-4361-bab9-d6a88b76c35a" (UID: "8b76e085-63e9-4361-bab9-d6a88b76c35a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:34:36.257059 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:36.256550 4421 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b76e085-63e9-4361-bab9-d6a88b76c35a-workload-certs\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:34:36.257059 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:36.256579 4421 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b76e085-63e9-4361-bab9-d6a88b76c35a-credential-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:34:36.257059 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:36.256596 4421 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b76e085-63e9-4361-bab9-d6a88b76c35a-istio-data\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:34:36.257059 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:36.256611 4421 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b76e085-63e9-4361-bab9-d6a88b76c35a-workload-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:34:36.257059 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:36.256625 4421 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b76e085-63e9-4361-bab9-d6a88b76c35a-istiod-ca-cert\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:34:36.263404 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:36.263370 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8b76e085-63e9-4361-bab9-d6a88b76c35a-istio-podinfo" pod "8b76e085-63e9-4361-bab9-d6a88b76c35a" (UID: "8b76e085-63e9-4361-bab9-d6a88b76c35a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:34:36.263556 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:36.263403 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b76e085-63e9-4361-bab9-d6a88b76c35a-istio-envoy" pod "8b76e085-63e9-4361-bab9-d6a88b76c35a" (UID: "8b76e085-63e9-4361-bab9-d6a88b76c35a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:34:36.263556 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:36.263448 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b76e085-63e9-4361-bab9-d6a88b76c35a-kube-api-access-xsw98" pod "8b76e085-63e9-4361-bab9-d6a88b76c35a" (UID: "8b76e085-63e9-4361-bab9-d6a88b76c35a"). InnerVolumeSpecName "kube-api-access-xsw98". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:34:36.263556 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:36.263481 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b76e085-63e9-4361-bab9-d6a88b76c35a-istio-token" pod "8b76e085-63e9-4361-bab9-d6a88b76c35a" (UID: "8b76e085-63e9-4361-bab9-d6a88b76c35a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:34:36.358785 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:36.358741 4421 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b76e085-63e9-4361-bab9-d6a88b76c35a-istio-podinfo\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:34:36.358785 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:36.358777 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xsw98\" (UniqueName: \"kubernetes.io/projected/8b76e085-63e9-4361-bab9-d6a88b76c35a-kube-api-access-xsw98\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:34:36.358785 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:36.358787 4421 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b76e085-63e9-4361-bab9-d6a88b76c35a-istio-token\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:34:36.358785 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:36.358797 4421 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b76e085-63e9-4361-bab9-d6a88b76c35a-istio-envoy\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:34:36.429300 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:36.420780 4421 patch_prober.go:29] interesting pod/e2e-mt-737bf2d8-openshift-default-56dc96b957-9v2m2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.89:15021/healthz/ready\": dial tcp 10.134.1.89:15021: connect: connection refused" start-of-body= Nov 03 01:34:36.429300 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:36.420849 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-9v2m2" podUID="52a0f195-418d-42be-8222-b030c8f6ba55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.89:15021/healthz/ready\": dial tcp 10.134.1.89:15021: connect: connection refused" Nov 03 01:34:36.596211 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:36.596129 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-b5cbp" Nov 03 01:34:36.597186 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:36.596778 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-b5cbp" Nov 03 01:34:36.598058 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:36.597317 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-b5cbp" Nov 03 01:34:36.739302 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:36.738141 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ad7b8cb-xd298"] Nov 03 01:34:36.739521 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:36.739500 4421 cpu_manager.go:414] "containerMap: removing container" podUID="8b76e085-63e9-4361-bab9-d6a88b76c35a" containerName="istio-proxy" Nov 03 01:34:36.739592 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:36.739523 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b76e085-63e9-4361-bab9-d6a88b76c35a" containerName="istio-proxy" Nov 03 01:34:36.739648 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:36.739608 4421 cpu_manager.go:414] "containerMap: removing container" podUID="d56b90c8-05c3-4c59-a204-79eb70d4a172" containerName="istio-proxy" Nov 03 01:34:36.739648 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:36.739617 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d56b90c8-05c3-4c59-a204-79eb70d4a172" containerName="istio-proxy" Nov 03 01:34:36.743959 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:36.739830 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="894c4be7-9b51-47c4-87d7-05eee1e8bcbc" containerName="istio-proxy" Nov 03 01:34:36.743959 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:36.739851 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="d56b90c8-05c3-4c59-a204-79eb70d4a172" containerName="istio-proxy" Nov 03 01:34:36.743959 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:36.739871 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b76e085-63e9-4361-bab9-d6a88b76c35a" containerName="istio-proxy" Nov 03 01:34:36.750894 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:36.750867 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ad7b8cb-xd298" Nov 03 01:34:36.766689 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:36.766656 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6q4xp\"" Nov 03 01:34:36.769344 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:36.768836 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:34:36.769344 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:36.769263 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ad7b8cb-xd298"] Nov 03 01:34:36.842186 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:36.842150 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9gm5g" Nov 03 01:34:36.842186 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:36.842167 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9gm5g" event={"ID":"8b76e085-63e9-4361-bab9-d6a88b76c35a","Type":"ContainerDied","Data":"2a242c4645f2cffbccf114b7add9870d49b48edf70fe76789b47bf675f0d2c4b"} Nov 03 01:34:36.842821 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:36.842237 4421 scope.go:122] "RemoveContainer" containerID="ed0aac922366879df461e8e0d690987aed3bd8654c2d20d1ae9b1f4c18a4bf55" Nov 03 01:34:36.868383 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:36.868349 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9gm5g"] Nov 03 01:34:36.874216 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:36.874190 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-9gm5g"] Nov 03 01:34:36.890721 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:36.890692 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d2f4a61-1127-4b56-b639-13742aa394b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9ad7b8cb-xd298\" (UID: \"9d2f4a61-1127-4b56-b639-13742aa394b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ad7b8cb-xd298" Nov 03 01:34:36.890931 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:36.890916 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjnq5\" (UniqueName: \"kubernetes.io/projected/9d2f4a61-1127-4b56-b639-13742aa394b0-kube-api-access-xjnq5\") pod \"maas-api-revoke-sub-9ad7b8cb-xd298\" (UID: \"9d2f4a61-1127-4b56-b639-13742aa394b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ad7b8cb-xd298" Nov 03 01:34:36.890992 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:36.890943 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d2f4a61-1127-4b56-b639-13742aa394b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9ad7b8cb-xd298\" (UID: \"9d2f4a61-1127-4b56-b639-13742aa394b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ad7b8cb-xd298" Nov 03 01:34:36.992384 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:36.992346 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjnq5\" (UniqueName: \"kubernetes.io/projected/9d2f4a61-1127-4b56-b639-13742aa394b0-kube-api-access-xjnq5\") pod \"maas-api-revoke-sub-9ad7b8cb-xd298\" (UID: \"9d2f4a61-1127-4b56-b639-13742aa394b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ad7b8cb-xd298" Nov 03 01:34:36.992580 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:36.992488 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d2f4a61-1127-4b56-b639-13742aa394b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9ad7b8cb-xd298\" (UID: \"9d2f4a61-1127-4b56-b639-13742aa394b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ad7b8cb-xd298" Nov 03 01:34:36.992752 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:36.992731 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d2f4a61-1127-4b56-b639-13742aa394b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9ad7b8cb-xd298\" (UID: \"9d2f4a61-1127-4b56-b639-13742aa394b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ad7b8cb-xd298" Nov 03 01:34:36.993162 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:36.993139 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d2f4a61-1127-4b56-b639-13742aa394b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9ad7b8cb-xd298\" (UID: \"9d2f4a61-1127-4b56-b639-13742aa394b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ad7b8cb-xd298" Nov 03 01:34:36.995090 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:36.995061 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d2f4a61-1127-4b56-b639-13742aa394b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9ad7b8cb-xd298\" (UID: \"9d2f4a61-1127-4b56-b639-13742aa394b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ad7b8cb-xd298" Nov 03 01:34:36.999985 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:36.999962 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjnq5\" (UniqueName: \"kubernetes.io/projected/9d2f4a61-1127-4b56-b639-13742aa394b0-kube-api-access-xjnq5\") pod \"maas-api-revoke-sub-9ad7b8cb-xd298\" (UID: \"9d2f4a61-1127-4b56-b639-13742aa394b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ad7b8cb-xd298" Nov 03 01:34:37.086974 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:37.086938 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ad7b8cb-xd298" Nov 03 01:34:37.430985 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:37.430887 4421 patch_prober.go:29] interesting pod/e2e-mt-737bf2d8-openshift-default-56dc96b957-9v2m2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.89:15021/healthz/ready\": dial tcp 10.134.1.89:15021: connect: connection refused" start-of-body= Nov 03 01:34:37.430985 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:37.430964 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-9v2m2" podUID="52a0f195-418d-42be-8222-b030c8f6ba55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.89:15021/healthz/ready\": dial tcp 10.134.1.89:15021: connect: connection refused" Nov 03 01:34:37.535316 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:37.503010 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ad7b8cb-xd298"] Nov 03 01:34:37.962496 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:37.962379 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ad7b8cb-xd298" event={"ID":"9d2f4a61-1127-4b56-b639-13742aa394b0","Type":"ContainerStarted","Data":"ef15fb96c1f32be9d1092ed8af74aad8c2f6e56844370969cfad6e25838c2fbd"} Nov 03 01:34:37.990094 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:37.990056 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b76e085-63e9-4361-bab9-d6a88b76c35a" path="/var/lib/kubelet/pods/8b76e085-63e9-4361-bab9-d6a88b76c35a/volumes" Nov 03 01:34:38.039710 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:38.039666 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2767ec4a-rg2kd"] Nov 03 01:34:38.044638 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:38.044611 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2767ec4a-rg2kd"] Nov 03 01:34:38.423300 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:38.415924 4421 patch_prober.go:29] interesting pod/e2e-mt-737bf2d8-openshift-default-56dc96b957-9v2m2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.89:15021/healthz/ready\": dial tcp 10.134.1.89:15021: connect: connection refused" start-of-body= Nov 03 01:34:38.423300 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:38.415995 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-9v2m2" podUID="52a0f195-418d-42be-8222-b030c8f6ba55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.89:15021/healthz/ready\": dial tcp 10.134.1.89:15021: connect: connection refused" Nov 03 01:34:38.537300 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:38.537018 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-027c19-openshift-default-58d5bc7645-6jphv"] Nov 03 01:34:38.537300 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:38.537210 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-027c19-openshift-default-58d5bc7645-6jphv" podUID="c4fa33da-32e1-40cb-b89d-8b25ab268fb0" containerName="istio-proxy" containerID="cri-o://c390525ebabf64e8a9a245c797763de55fe02a5985c830758a5a35f4d35f7340" gracePeriod=30 Nov 03 01:34:38.971701 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:38.971673 4421 generic.go:356] "Generic (PLEG): container finished" podID="9d2f4a61-1127-4b56-b639-13742aa394b0" containerID="b590451a28a979db1b8c7570c6d79145ac726ca815310f08d2cf9ef25a083b13" exitCode=0 Nov 03 01:34:38.972097 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:38.971808 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ad7b8cb-xd298" event={"ID":"9d2f4a61-1127-4b56-b639-13742aa394b0","Type":"ContainerDied","Data":"b590451a28a979db1b8c7570c6d79145ac726ca815310f08d2cf9ef25a083b13"} Nov 03 01:34:39.424300 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:39.423790 4421 patch_prober.go:29] interesting pod/e2e-mt-737bf2d8-openshift-default-56dc96b957-9v2m2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.89:15021/healthz/ready\": dial tcp 10.134.1.89:15021: connect: connection refused" start-of-body= Nov 03 01:34:39.424300 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:39.423854 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-9v2m2" podUID="52a0f195-418d-42be-8222-b030c8f6ba55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.89:15021/healthz/ready\": dial tcp 10.134.1.89:15021: connect: connection refused" Nov 03 01:34:39.985222 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:39.985178 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de29d9b6-da5a-4b65-99b9-996326dac022" path="/var/lib/kubelet/pods/de29d9b6-da5a-4b65-99b9-996326dac022/volumes" Nov 03 01:34:40.291804 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:40.291776 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ad7b8cb-xd298" Nov 03 01:34:40.394485 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:40.394449 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d2f4a61-1127-4b56-b639-13742aa394b0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d2f4a61-1127-4b56-b639-13742aa394b0-maas-api-openshift-ca\") pod \"9d2f4a61-1127-4b56-b639-13742aa394b0\" (UID: \"9d2f4a61-1127-4b56-b639-13742aa394b0\") " Nov 03 01:34:40.394857 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:40.394828 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d2f4a61-1127-4b56-b639-13742aa394b0-maas-api-openshift-ca" pod "9d2f4a61-1127-4b56-b639-13742aa394b0" (UID: "9d2f4a61-1127-4b56-b639-13742aa394b0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:34:40.394913 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:40.394862 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d2f4a61-1127-4b56-b639-13742aa394b0-kube-api-access-xjnq5\" (UniqueName: \"kubernetes.io/projected/9d2f4a61-1127-4b56-b639-13742aa394b0-kube-api-access-xjnq5\") pod \"9d2f4a61-1127-4b56-b639-13742aa394b0\" (UID: \"9d2f4a61-1127-4b56-b639-13742aa394b0\") " Nov 03 01:34:40.394973 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:40.394951 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d2f4a61-1127-4b56-b639-13742aa394b0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d2f4a61-1127-4b56-b639-13742aa394b0-maas-api-xks-ca\") pod \"9d2f4a61-1127-4b56-b639-13742aa394b0\" (UID: \"9d2f4a61-1127-4b56-b639-13742aa394b0\") " Nov 03 01:34:40.395667 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:40.395637 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d2f4a61-1127-4b56-b639-13742aa394b0-maas-api-openshift-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:34:40.397009 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:40.396986 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d2f4a61-1127-4b56-b639-13742aa394b0-maas-api-xks-ca" pod "9d2f4a61-1127-4b56-b639-13742aa394b0" (UID: "9d2f4a61-1127-4b56-b639-13742aa394b0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:34:40.397255 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:40.397239 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d2f4a61-1127-4b56-b639-13742aa394b0-kube-api-access-xjnq5" pod "9d2f4a61-1127-4b56-b639-13742aa394b0" (UID: "9d2f4a61-1127-4b56-b639-13742aa394b0"). InnerVolumeSpecName "kube-api-access-xjnq5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:34:40.508367 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:40.499253 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjnq5\" (UniqueName: \"kubernetes.io/projected/9d2f4a61-1127-4b56-b639-13742aa394b0-kube-api-access-xjnq5\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:34:40.508367 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:40.499311 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d2f4a61-1127-4b56-b639-13742aa394b0-maas-api-xks-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:34:40.580744 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:40.579713 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-9v2m2" Nov 03 01:34:40.580744 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:40.580219 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-9v2m2" Nov 03 01:34:40.585904 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:40.585852 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-9v2m2" Nov 03 01:34:40.666038 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:40.666014 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-ddd653-openshift-default-779ff6465-6hnrl" Nov 03 01:34:40.702619 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:40.702587 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8b337af9-fa55-447d-9555-7852367061f5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b337af9-fa55-447d-9555-7852367061f5-istio-podinfo\") pod \"8b337af9-fa55-447d-9555-7852367061f5\" (UID: \"8b337af9-fa55-447d-9555-7852367061f5\") " Nov 03 01:34:40.702816 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:40.702675 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b337af9-fa55-447d-9555-7852367061f5-kube-api-access-4zfnm\" (UniqueName: \"kubernetes.io/projected/8b337af9-fa55-447d-9555-7852367061f5-kube-api-access-4zfnm\") pod \"8b337af9-fa55-447d-9555-7852367061f5\" (UID: \"8b337af9-fa55-447d-9555-7852367061f5\") " Nov 03 01:34:40.702816 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:40.702782 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b337af9-fa55-447d-9555-7852367061f5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b337af9-fa55-447d-9555-7852367061f5-workload-certs\") pod \"8b337af9-fa55-447d-9555-7852367061f5\" (UID: \"8b337af9-fa55-447d-9555-7852367061f5\") " Nov 03 01:34:40.702973 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:40.702842 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b337af9-fa55-447d-9555-7852367061f5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b337af9-fa55-447d-9555-7852367061f5-istio-data\") pod \"8b337af9-fa55-447d-9555-7852367061f5\" (UID: \"8b337af9-fa55-447d-9555-7852367061f5\") " Nov 03 01:34:40.702973 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:40.702963 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b337af9-fa55-447d-9555-7852367061f5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b337af9-fa55-447d-9555-7852367061f5-credential-socket\") pod \"8b337af9-fa55-447d-9555-7852367061f5\" (UID: \"8b337af9-fa55-447d-9555-7852367061f5\") " Nov 03 01:34:40.703102 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:40.703000 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b337af9-fa55-447d-9555-7852367061f5-istio-token\" (UniqueName: \"kubernetes.io/projected/8b337af9-fa55-447d-9555-7852367061f5-istio-token\") pod \"8b337af9-fa55-447d-9555-7852367061f5\" (UID: \"8b337af9-fa55-447d-9555-7852367061f5\") " Nov 03 01:34:40.703102 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:40.703034 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b337af9-fa55-447d-9555-7852367061f5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b337af9-fa55-447d-9555-7852367061f5-istio-envoy\") pod \"8b337af9-fa55-447d-9555-7852367061f5\" (UID: \"8b337af9-fa55-447d-9555-7852367061f5\") " Nov 03 01:34:40.703102 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:40.703073 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b337af9-fa55-447d-9555-7852367061f5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b337af9-fa55-447d-9555-7852367061f5-workload-socket\") pod \"8b337af9-fa55-447d-9555-7852367061f5\" (UID: \"8b337af9-fa55-447d-9555-7852367061f5\") " Nov 03 01:34:40.703975 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:40.703107 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b337af9-fa55-447d-9555-7852367061f5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b337af9-fa55-447d-9555-7852367061f5-istiod-ca-cert\") pod \"8b337af9-fa55-447d-9555-7852367061f5\" (UID: \"8b337af9-fa55-447d-9555-7852367061f5\") " Nov 03 01:34:40.703975 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:40.703496 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b337af9-fa55-447d-9555-7852367061f5-credential-socket" pod "8b337af9-fa55-447d-9555-7852367061f5" (UID: "8b337af9-fa55-447d-9555-7852367061f5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:34:40.703975 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:40.703718 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b337af9-fa55-447d-9555-7852367061f5-istio-data" pod "8b337af9-fa55-447d-9555-7852367061f5" (UID: "8b337af9-fa55-447d-9555-7852367061f5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:34:40.703975 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:40.703815 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b337af9-fa55-447d-9555-7852367061f5-istiod-ca-cert" pod "8b337af9-fa55-447d-9555-7852367061f5" (UID: "8b337af9-fa55-447d-9555-7852367061f5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:34:40.703975 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:40.703933 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b337af9-fa55-447d-9555-7852367061f5-workload-socket" pod "8b337af9-fa55-447d-9555-7852367061f5" (UID: "8b337af9-fa55-447d-9555-7852367061f5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:34:40.703975 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:40.703962 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b337af9-fa55-447d-9555-7852367061f5-workload-certs" pod "8b337af9-fa55-447d-9555-7852367061f5" (UID: "8b337af9-fa55-447d-9555-7852367061f5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:34:40.705058 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:40.704158 4421 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b337af9-fa55-447d-9555-7852367061f5-credential-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:34:40.705058 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:40.704183 4421 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b337af9-fa55-447d-9555-7852367061f5-workload-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:34:40.705058 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:40.704201 4421 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b337af9-fa55-447d-9555-7852367061f5-istiod-ca-cert\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:34:40.705058 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:40.704216 4421 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b337af9-fa55-447d-9555-7852367061f5-workload-certs\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:34:40.705058 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:40.704229 4421 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b337af9-fa55-447d-9555-7852367061f5-istio-data\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:34:40.705554 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:40.705532 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8b337af9-fa55-447d-9555-7852367061f5-istio-podinfo" pod "8b337af9-fa55-447d-9555-7852367061f5" (UID: "8b337af9-fa55-447d-9555-7852367061f5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:34:40.706342 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:40.706317 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b337af9-fa55-447d-9555-7852367061f5-kube-api-access-4zfnm" pod "8b337af9-fa55-447d-9555-7852367061f5" (UID: "8b337af9-fa55-447d-9555-7852367061f5"). InnerVolumeSpecName "kube-api-access-4zfnm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:34:40.706444 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:40.706361 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b337af9-fa55-447d-9555-7852367061f5-istio-token" pod "8b337af9-fa55-447d-9555-7852367061f5" (UID: "8b337af9-fa55-447d-9555-7852367061f5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:34:40.707396 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:40.707365 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b337af9-fa55-447d-9555-7852367061f5-istio-envoy" pod "8b337af9-fa55-447d-9555-7852367061f5" (UID: "8b337af9-fa55-447d-9555-7852367061f5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:34:40.808452 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:40.808345 4421 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b337af9-fa55-447d-9555-7852367061f5-istio-podinfo\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:34:40.808452 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:40.808390 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4zfnm\" (UniqueName: \"kubernetes.io/projected/8b337af9-fa55-447d-9555-7852367061f5-kube-api-access-4zfnm\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:34:40.808452 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:40.808414 4421 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b337af9-fa55-447d-9555-7852367061f5-istio-token\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:34:40.808452 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:40.808430 4421 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b337af9-fa55-447d-9555-7852367061f5-istio-envoy\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:34:40.992993 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:40.992956 4421 generic.go:356] "Generic (PLEG): container finished" podID="8b337af9-fa55-447d-9555-7852367061f5" containerID="9c8617669bb3974e29df388e4cd520abd543b06add22c069b572607542e2224e" exitCode=0 Nov 03 01:34:40.993470 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:40.993028 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-ddd653-openshift-default-779ff6465-6hnrl" Nov 03 01:34:40.993470 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:40.993030 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-ddd653-openshift-default-779ff6465-6hnrl" event={"ID":"8b337af9-fa55-447d-9555-7852367061f5","Type":"ContainerDied","Data":"9c8617669bb3974e29df388e4cd520abd543b06add22c069b572607542e2224e"} Nov 03 01:34:40.993470 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:40.993090 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-ddd653-openshift-default-779ff6465-6hnrl" event={"ID":"8b337af9-fa55-447d-9555-7852367061f5","Type":"ContainerDied","Data":"ccf98034c97b8e6b040bd68fccdb1d7d675256b070a34c40342e286b0b2472c5"} Nov 03 01:34:40.993470 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:40.993111 4421 scope.go:122] "RemoveContainer" containerID="9c8617669bb3974e29df388e4cd520abd543b06add22c069b572607542e2224e" Nov 03 01:34:40.995930 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:40.995898 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ad7b8cb-xd298" event={"ID":"9d2f4a61-1127-4b56-b639-13742aa394b0","Type":"ContainerDied","Data":"ef15fb96c1f32be9d1092ed8af74aad8c2f6e56844370969cfad6e25838c2fbd"} Nov 03 01:34:40.996044 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:40.995963 4421 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ef15fb96c1f32be9d1092ed8af74aad8c2f6e56844370969cfad6e25838c2fbd" Nov 03 01:34:40.996044 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:40.995990 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ad7b8cb-xd298" Nov 03 01:34:41.010513 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:41.010477 4421 scope.go:122] "RemoveContainer" containerID="9c8617669bb3974e29df388e4cd520abd543b06add22c069b572607542e2224e" Nov 03 01:34:41.010919 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:34:41.010884 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c8617669bb3974e29df388e4cd520abd543b06add22c069b572607542e2224e\": container with ID starting with 9c8617669bb3974e29df388e4cd520abd543b06add22c069b572607542e2224e not found: ID does not exist" containerID="9c8617669bb3974e29df388e4cd520abd543b06add22c069b572607542e2224e" Nov 03 01:34:41.011074 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:41.010931 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c8617669bb3974e29df388e4cd520abd543b06add22c069b572607542e2224e"} err="failed to get container status \"9c8617669bb3974e29df388e4cd520abd543b06add22c069b572607542e2224e\": rpc error: code = NotFound desc = could not find container \"9c8617669bb3974e29df388e4cd520abd543b06add22c069b572607542e2224e\": container with ID starting with 9c8617669bb3974e29df388e4cd520abd543b06add22c069b572607542e2224e not found: ID does not exist" Nov 03 01:34:41.026140 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:41.026100 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-ddd653-openshift-default-779ff6465-6hnrl"] Nov 03 01:34:41.030690 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:41.030654 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-ddd653-openshift-default-779ff6465-6hnrl"] Nov 03 01:34:41.436175 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:41.436149 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-020af9-openshift-default-d77df45d5-vjg9x" Nov 03 01:34:41.531071 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:41.531025 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fb3426e-5493-4655-a9e9-e011e9073012-istio-token\" (UniqueName: \"kubernetes.io/projected/8fb3426e-5493-4655-a9e9-e011e9073012-istio-token\") pod \"8fb3426e-5493-4655-a9e9-e011e9073012\" (UID: \"8fb3426e-5493-4655-a9e9-e011e9073012\") " Nov 03 01:34:41.531313 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:41.531143 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fb3426e-5493-4655-a9e9-e011e9073012-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fb3426e-5493-4655-a9e9-e011e9073012-workload-socket\") pod \"8fb3426e-5493-4655-a9e9-e011e9073012\" (UID: \"8fb3426e-5493-4655-a9e9-e011e9073012\") " Nov 03 01:34:41.531420 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:41.531329 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fb3426e-5493-4655-a9e9-e011e9073012-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fb3426e-5493-4655-a9e9-e011e9073012-istio-envoy\") pod \"8fb3426e-5493-4655-a9e9-e011e9073012\" (UID: \"8fb3426e-5493-4655-a9e9-e011e9073012\") " Nov 03 01:34:41.531420 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:41.531370 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8fb3426e-5493-4655-a9e9-e011e9073012-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fb3426e-5493-4655-a9e9-e011e9073012-istio-podinfo\") pod \"8fb3426e-5493-4655-a9e9-e011e9073012\" (UID: \"8fb3426e-5493-4655-a9e9-e011e9073012\") " Nov 03 01:34:41.531547 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:41.531418 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8fb3426e-5493-4655-a9e9-e011e9073012-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fb3426e-5493-4655-a9e9-e011e9073012-istiod-ca-cert\") pod \"8fb3426e-5493-4655-a9e9-e011e9073012\" (UID: \"8fb3426e-5493-4655-a9e9-e011e9073012\") " Nov 03 01:34:41.531547 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:41.531449 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fb3426e-5493-4655-a9e9-e011e9073012-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fb3426e-5493-4655-a9e9-e011e9073012-credential-socket\") pod \"8fb3426e-5493-4655-a9e9-e011e9073012\" (UID: \"8fb3426e-5493-4655-a9e9-e011e9073012\") " Nov 03 01:34:41.531547 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:41.531492 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fb3426e-5493-4655-a9e9-e011e9073012-kube-api-access-jpvxn\" (UniqueName: \"kubernetes.io/projected/8fb3426e-5493-4655-a9e9-e011e9073012-kube-api-access-jpvxn\") pod \"8fb3426e-5493-4655-a9e9-e011e9073012\" (UID: \"8fb3426e-5493-4655-a9e9-e011e9073012\") " Nov 03 01:34:41.533162 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:41.531559 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fb3426e-5493-4655-a9e9-e011e9073012-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fb3426e-5493-4655-a9e9-e011e9073012-istio-data\") pod \"8fb3426e-5493-4655-a9e9-e011e9073012\" (UID: \"8fb3426e-5493-4655-a9e9-e011e9073012\") " Nov 03 01:34:41.533162 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:41.531627 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fb3426e-5493-4655-a9e9-e011e9073012-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fb3426e-5493-4655-a9e9-e011e9073012-workload-certs\") pod \"8fb3426e-5493-4655-a9e9-e011e9073012\" (UID: \"8fb3426e-5493-4655-a9e9-e011e9073012\") " Nov 03 01:34:41.533162 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:41.531980 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fb3426e-5493-4655-a9e9-e011e9073012-workload-socket" pod "8fb3426e-5493-4655-a9e9-e011e9073012" (UID: "8fb3426e-5493-4655-a9e9-e011e9073012"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:34:41.533162 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:41.532152 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fb3426e-5493-4655-a9e9-e011e9073012-istiod-ca-cert" pod "8fb3426e-5493-4655-a9e9-e011e9073012" (UID: "8fb3426e-5493-4655-a9e9-e011e9073012"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:34:41.533162 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:41.532238 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fb3426e-5493-4655-a9e9-e011e9073012-workload-certs" pod "8fb3426e-5493-4655-a9e9-e011e9073012" (UID: "8fb3426e-5493-4655-a9e9-e011e9073012"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:34:41.533162 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:41.532386 4421 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fb3426e-5493-4655-a9e9-e011e9073012-istiod-ca-cert\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:34:41.533162 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:41.532396 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fb3426e-5493-4655-a9e9-e011e9073012-credential-socket" pod "8fb3426e-5493-4655-a9e9-e011e9073012" (UID: "8fb3426e-5493-4655-a9e9-e011e9073012"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:34:41.533162 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:41.532408 4421 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fb3426e-5493-4655-a9e9-e011e9073012-workload-certs\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:34:41.533162 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:41.532454 4421 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fb3426e-5493-4655-a9e9-e011e9073012-workload-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:34:41.533162 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:41.532420 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fb3426e-5493-4655-a9e9-e011e9073012-istio-data" pod "8fb3426e-5493-4655-a9e9-e011e9073012" (UID: "8fb3426e-5493-4655-a9e9-e011e9073012"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:34:41.534556 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:41.533713 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fb3426e-5493-4655-a9e9-e011e9073012-istio-token" pod "8fb3426e-5493-4655-a9e9-e011e9073012" (UID: "8fb3426e-5493-4655-a9e9-e011e9073012"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:34:41.534556 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:41.534389 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fb3426e-5493-4655-a9e9-e011e9073012-istio-envoy" pod "8fb3426e-5493-4655-a9e9-e011e9073012" (UID: "8fb3426e-5493-4655-a9e9-e011e9073012"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:34:41.534556 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:41.534442 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fb3426e-5493-4655-a9e9-e011e9073012-kube-api-access-jpvxn" pod "8fb3426e-5493-4655-a9e9-e011e9073012" (UID: "8fb3426e-5493-4655-a9e9-e011e9073012"). InnerVolumeSpecName "kube-api-access-jpvxn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:34:41.534556 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:41.534513 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8fb3426e-5493-4655-a9e9-e011e9073012-istio-podinfo" pod "8fb3426e-5493-4655-a9e9-e011e9073012" (UID: "8fb3426e-5493-4655-a9e9-e011e9073012"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:34:41.633519 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:41.633477 4421 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fb3426e-5493-4655-a9e9-e011e9073012-istio-envoy\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:34:41.633519 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:41.633511 4421 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fb3426e-5493-4655-a9e9-e011e9073012-istio-podinfo\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:34:41.633519 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:41.633521 4421 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fb3426e-5493-4655-a9e9-e011e9073012-credential-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:34:41.633803 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:41.633533 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jpvxn\" (UniqueName: \"kubernetes.io/projected/8fb3426e-5493-4655-a9e9-e011e9073012-kube-api-access-jpvxn\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:34:41.633803 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:41.633542 4421 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fb3426e-5493-4655-a9e9-e011e9073012-istio-data\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:34:41.633803 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:41.633551 4421 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fb3426e-5493-4655-a9e9-e011e9073012-istio-token\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:34:41.982034 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:41.982002 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b337af9-fa55-447d-9555-7852367061f5" path="/var/lib/kubelet/pods/8b337af9-fa55-447d-9555-7852367061f5/volumes" Nov 03 01:34:42.009523 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:42.009486 4421 generic.go:356] "Generic (PLEG): container finished" podID="8fb3426e-5493-4655-a9e9-e011e9073012" containerID="cb7345056853e5742c1811f4fc210b604d8848d627f1ea560ff1b0fc2d781373" exitCode=0 Nov 03 01:34:42.010014 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:42.009561 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-020af9-openshift-default-d77df45d5-vjg9x" Nov 03 01:34:42.010014 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:42.009570 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-020af9-openshift-default-d77df45d5-vjg9x" event={"ID":"8fb3426e-5493-4655-a9e9-e011e9073012","Type":"ContainerDied","Data":"cb7345056853e5742c1811f4fc210b604d8848d627f1ea560ff1b0fc2d781373"} Nov 03 01:34:42.010014 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:42.009621 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-020af9-openshift-default-d77df45d5-vjg9x" event={"ID":"8fb3426e-5493-4655-a9e9-e011e9073012","Type":"ContainerDied","Data":"288abde84800e5c9324a1a74d4d366ffde9a740cc3eee99e2466c7a129168281"} Nov 03 01:34:42.010014 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:42.009643 4421 scope.go:122] "RemoveContainer" containerID="cb7345056853e5742c1811f4fc210b604d8848d627f1ea560ff1b0fc2d781373" Nov 03 01:34:42.021829 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:42.021813 4421 scope.go:122] "RemoveContainer" containerID="cb7345056853e5742c1811f4fc210b604d8848d627f1ea560ff1b0fc2d781373" Nov 03 01:34:42.022118 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:34:42.022095 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb7345056853e5742c1811f4fc210b604d8848d627f1ea560ff1b0fc2d781373\": container with ID starting with cb7345056853e5742c1811f4fc210b604d8848d627f1ea560ff1b0fc2d781373 not found: ID does not exist" containerID="cb7345056853e5742c1811f4fc210b604d8848d627f1ea560ff1b0fc2d781373" Nov 03 01:34:42.022229 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:42.022126 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb7345056853e5742c1811f4fc210b604d8848d627f1ea560ff1b0fc2d781373"} err="failed to get container status \"cb7345056853e5742c1811f4fc210b604d8848d627f1ea560ff1b0fc2d781373\": rpc error: code = NotFound desc = could not find container \"cb7345056853e5742c1811f4fc210b604d8848d627f1ea560ff1b0fc2d781373\": container with ID starting with cb7345056853e5742c1811f4fc210b604d8848d627f1ea560ff1b0fc2d781373 not found: ID does not exist" Nov 03 01:34:42.031587 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:42.031554 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-020af9-openshift-default-d77df45d5-vjg9x"] Nov 03 01:34:42.035532 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:42.035513 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-020af9-openshift-default-d77df45d5-vjg9x"] Nov 03 01:34:43.851632 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:43.851603 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-027c19-openshift-default-58d5bc7645-6jphv" Nov 03 01:34:43.892235 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:43.892199 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4fa33da-32e1-40cb-b89d-8b25ab268fb0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4fa33da-32e1-40cb-b89d-8b25ab268fb0-istio-envoy\") pod \"c4fa33da-32e1-40cb-b89d-8b25ab268fb0\" (UID: \"c4fa33da-32e1-40cb-b89d-8b25ab268fb0\") " Nov 03 01:34:43.892446 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:43.892299 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4fa33da-32e1-40cb-b89d-8b25ab268fb0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4fa33da-32e1-40cb-b89d-8b25ab268fb0-workload-certs\") pod \"c4fa33da-32e1-40cb-b89d-8b25ab268fb0\" (UID: \"c4fa33da-32e1-40cb-b89d-8b25ab268fb0\") " Nov 03 01:34:43.892446 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:43.892355 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c4fa33da-32e1-40cb-b89d-8b25ab268fb0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4fa33da-32e1-40cb-b89d-8b25ab268fb0-istio-podinfo\") pod \"c4fa33da-32e1-40cb-b89d-8b25ab268fb0\" (UID: \"c4fa33da-32e1-40cb-b89d-8b25ab268fb0\") " Nov 03 01:34:43.892546 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:43.892486 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4fa33da-32e1-40cb-b89d-8b25ab268fb0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4fa33da-32e1-40cb-b89d-8b25ab268fb0-istio-data\") pod \"c4fa33da-32e1-40cb-b89d-8b25ab268fb0\" (UID: \"c4fa33da-32e1-40cb-b89d-8b25ab268fb0\") " Nov 03 01:34:43.892546 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:43.892517 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4fa33da-32e1-40cb-b89d-8b25ab268fb0-istio-token\" (UniqueName: \"kubernetes.io/projected/c4fa33da-32e1-40cb-b89d-8b25ab268fb0-istio-token\") pod \"c4fa33da-32e1-40cb-b89d-8b25ab268fb0\" (UID: \"c4fa33da-32e1-40cb-b89d-8b25ab268fb0\") " Nov 03 01:34:43.892649 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:43.892550 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4fa33da-32e1-40cb-b89d-8b25ab268fb0-kube-api-access-pwpn5\" (UniqueName: \"kubernetes.io/projected/c4fa33da-32e1-40cb-b89d-8b25ab268fb0-kube-api-access-pwpn5\") pod \"c4fa33da-32e1-40cb-b89d-8b25ab268fb0\" (UID: \"c4fa33da-32e1-40cb-b89d-8b25ab268fb0\") " Nov 03 01:34:43.892649 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:43.892578 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4fa33da-32e1-40cb-b89d-8b25ab268fb0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4fa33da-32e1-40cb-b89d-8b25ab268fb0-workload-socket\") pod \"c4fa33da-32e1-40cb-b89d-8b25ab268fb0\" (UID: \"c4fa33da-32e1-40cb-b89d-8b25ab268fb0\") " Nov 03 01:34:43.892759 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:43.892665 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4fa33da-32e1-40cb-b89d-8b25ab268fb0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4fa33da-32e1-40cb-b89d-8b25ab268fb0-credential-socket\") pod \"c4fa33da-32e1-40cb-b89d-8b25ab268fb0\" (UID: \"c4fa33da-32e1-40cb-b89d-8b25ab268fb0\") " Nov 03 01:34:43.892811 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:43.892759 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4fa33da-32e1-40cb-b89d-8b25ab268fb0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4fa33da-32e1-40cb-b89d-8b25ab268fb0-istiod-ca-cert\") pod \"c4fa33da-32e1-40cb-b89d-8b25ab268fb0\" (UID: \"c4fa33da-32e1-40cb-b89d-8b25ab268fb0\") " Nov 03 01:34:43.893288 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:43.893224 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4fa33da-32e1-40cb-b89d-8b25ab268fb0-istio-data" pod "c4fa33da-32e1-40cb-b89d-8b25ab268fb0" (UID: "c4fa33da-32e1-40cb-b89d-8b25ab268fb0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:34:43.893650 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:43.893610 4421 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4fa33da-32e1-40cb-b89d-8b25ab268fb0-istio-data\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:34:43.894374 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:43.894343 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4fa33da-32e1-40cb-b89d-8b25ab268fb0-workload-socket" pod "c4fa33da-32e1-40cb-b89d-8b25ab268fb0" (UID: "c4fa33da-32e1-40cb-b89d-8b25ab268fb0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:34:43.896455 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:43.895975 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4fa33da-32e1-40cb-b89d-8b25ab268fb0-istio-envoy" pod "c4fa33da-32e1-40cb-b89d-8b25ab268fb0" (UID: "c4fa33da-32e1-40cb-b89d-8b25ab268fb0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:34:43.896455 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:43.896202 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4fa33da-32e1-40cb-b89d-8b25ab268fb0-istio-token" pod "c4fa33da-32e1-40cb-b89d-8b25ab268fb0" (UID: "c4fa33da-32e1-40cb-b89d-8b25ab268fb0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:34:43.896455 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:43.896230 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4fa33da-32e1-40cb-b89d-8b25ab268fb0-workload-certs" pod "c4fa33da-32e1-40cb-b89d-8b25ab268fb0" (UID: "c4fa33da-32e1-40cb-b89d-8b25ab268fb0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:34:43.896734 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:43.896469 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4fa33da-32e1-40cb-b89d-8b25ab268fb0-credential-socket" pod "c4fa33da-32e1-40cb-b89d-8b25ab268fb0" (UID: "c4fa33da-32e1-40cb-b89d-8b25ab268fb0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:34:43.896734 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:43.896650 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4fa33da-32e1-40cb-b89d-8b25ab268fb0-istiod-ca-cert" pod "c4fa33da-32e1-40cb-b89d-8b25ab268fb0" (UID: "c4fa33da-32e1-40cb-b89d-8b25ab268fb0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:34:43.898221 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:43.898195 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c4fa33da-32e1-40cb-b89d-8b25ab268fb0-istio-podinfo" pod "c4fa33da-32e1-40cb-b89d-8b25ab268fb0" (UID: "c4fa33da-32e1-40cb-b89d-8b25ab268fb0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:34:43.898358 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:43.898200 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4fa33da-32e1-40cb-b89d-8b25ab268fb0-kube-api-access-pwpn5" pod "c4fa33da-32e1-40cb-b89d-8b25ab268fb0" (UID: "c4fa33da-32e1-40cb-b89d-8b25ab268fb0"). InnerVolumeSpecName "kube-api-access-pwpn5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:34:43.982173 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:43.982135 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8fb3426e-5493-4655-a9e9-e011e9073012" path="/var/lib/kubelet/pods/8fb3426e-5493-4655-a9e9-e011e9073012/volumes" Nov 03 01:34:44.005307 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:44.000436 4421 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4fa33da-32e1-40cb-b89d-8b25ab268fb0-workload-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:34:44.005307 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:44.000487 4421 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4fa33da-32e1-40cb-b89d-8b25ab268fb0-credential-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:34:44.005307 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:44.000508 4421 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4fa33da-32e1-40cb-b89d-8b25ab268fb0-istiod-ca-cert\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:34:44.005307 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:44.000525 4421 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4fa33da-32e1-40cb-b89d-8b25ab268fb0-istio-envoy\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:34:44.005307 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:44.000658 4421 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4fa33da-32e1-40cb-b89d-8b25ab268fb0-workload-certs\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:34:44.005307 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:44.000714 4421 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4fa33da-32e1-40cb-b89d-8b25ab268fb0-istio-podinfo\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:34:44.005307 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:44.000732 4421 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4fa33da-32e1-40cb-b89d-8b25ab268fb0-istio-token\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:34:44.005307 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:44.000750 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwpn5\" (UniqueName: \"kubernetes.io/projected/c4fa33da-32e1-40cb-b89d-8b25ab268fb0-kube-api-access-pwpn5\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:34:44.058798 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:44.058768 4421 generic.go:356] "Generic (PLEG): container finished" podID="c4fa33da-32e1-40cb-b89d-8b25ab268fb0" containerID="c390525ebabf64e8a9a245c797763de55fe02a5985c830758a5a35f4d35f7340" exitCode=0 Nov 03 01:34:44.058966 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:44.058812 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-027c19-openshift-default-58d5bc7645-6jphv" Nov 03 01:34:44.058966 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:44.058856 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-027c19-openshift-default-58d5bc7645-6jphv" event={"ID":"c4fa33da-32e1-40cb-b89d-8b25ab268fb0","Type":"ContainerDied","Data":"c390525ebabf64e8a9a245c797763de55fe02a5985c830758a5a35f4d35f7340"} Nov 03 01:34:44.058966 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:44.058926 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-027c19-openshift-default-58d5bc7645-6jphv" event={"ID":"c4fa33da-32e1-40cb-b89d-8b25ab268fb0","Type":"ContainerDied","Data":"6ff510f4d83e1826980b83e4351383b3b05bf7c7503fd5f9773eaae8f75e331c"} Nov 03 01:34:44.058966 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:44.058957 4421 scope.go:122] "RemoveContainer" containerID="c390525ebabf64e8a9a245c797763de55fe02a5985c830758a5a35f4d35f7340" Nov 03 01:34:44.071956 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:44.071937 4421 scope.go:122] "RemoveContainer" containerID="c390525ebabf64e8a9a245c797763de55fe02a5985c830758a5a35f4d35f7340" Nov 03 01:34:44.072319 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:34:44.072263 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c390525ebabf64e8a9a245c797763de55fe02a5985c830758a5a35f4d35f7340\": container with ID starting with c390525ebabf64e8a9a245c797763de55fe02a5985c830758a5a35f4d35f7340 not found: ID does not exist" containerID="c390525ebabf64e8a9a245c797763de55fe02a5985c830758a5a35f4d35f7340" Nov 03 01:34:44.072319 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:44.072306 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c390525ebabf64e8a9a245c797763de55fe02a5985c830758a5a35f4d35f7340"} err="failed to get container status \"c390525ebabf64e8a9a245c797763de55fe02a5985c830758a5a35f4d35f7340\": rpc error: code = NotFound desc = could not find container \"c390525ebabf64e8a9a245c797763de55fe02a5985c830758a5a35f4d35f7340\": container with ID starting with c390525ebabf64e8a9a245c797763de55fe02a5985c830758a5a35f4d35f7340 not found: ID does not exist" Nov 03 01:34:44.082684 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:44.082487 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-027c19-openshift-default-58d5bc7645-6jphv"] Nov 03 01:34:44.086423 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:44.086400 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-027c19-openshift-default-58d5bc7645-6jphv"] Nov 03 01:34:44.671363 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:34:44.671320 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 01:34:45.991585 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:45.991535 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4fa33da-32e1-40cb-b89d-8b25ab268fb0" path="/var/lib/kubelet/pods/c4fa33da-32e1-40cb-b89d-8b25ab268fb0/volumes" Nov 03 01:34:47.808503 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:47.808459 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c05bb0ed-69cvl"] Nov 03 01:34:47.809632 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:47.809598 4421 cpu_manager.go:414] "containerMap: removing container" podUID="8fb3426e-5493-4655-a9e9-e011e9073012" containerName="istio-proxy" Nov 03 01:34:47.809632 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:47.809624 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fb3426e-5493-4655-a9e9-e011e9073012" containerName="istio-proxy" Nov 03 01:34:47.809814 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:47.809649 4421 cpu_manager.go:414] "containerMap: removing container" podUID="c4fa33da-32e1-40cb-b89d-8b25ab268fb0" containerName="istio-proxy" Nov 03 01:34:47.809814 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:47.809658 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4fa33da-32e1-40cb-b89d-8b25ab268fb0" containerName="istio-proxy" Nov 03 01:34:47.809814 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:47.809704 4421 cpu_manager.go:414] "containerMap: removing container" podUID="8b337af9-fa55-447d-9555-7852367061f5" containerName="istio-proxy" Nov 03 01:34:47.809814 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:47.809712 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b337af9-fa55-447d-9555-7852367061f5" containerName="istio-proxy" Nov 03 01:34:47.809814 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:47.809723 4421 cpu_manager.go:414] "containerMap: removing container" podUID="9d2f4a61-1127-4b56-b639-13742aa394b0" containerName="revoke-keys" Nov 03 01:34:47.809814 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:47.809729 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d2f4a61-1127-4b56-b639-13742aa394b0" containerName="revoke-keys" Nov 03 01:34:47.810038 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:47.809898 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4fa33da-32e1-40cb-b89d-8b25ab268fb0" containerName="istio-proxy" Nov 03 01:34:47.810038 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:47.809925 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b337af9-fa55-447d-9555-7852367061f5" containerName="istio-proxy" Nov 03 01:34:47.810038 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:47.809937 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d2f4a61-1127-4b56-b639-13742aa394b0" containerName="revoke-keys" Nov 03 01:34:47.810038 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:47.809961 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fb3426e-5493-4655-a9e9-e011e9073012" containerName="istio-proxy" Nov 03 01:34:47.817356 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:47.816014 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c05bb0ed-69cvl" Nov 03 01:34:47.824933 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:47.824794 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:34:47.824933 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:47.824902 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6q4xp\"" Nov 03 01:34:47.825380 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:47.825027 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c05bb0ed-69cvl"] Nov 03 01:34:47.978151 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:47.977934 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lq72n\" (UniqueName: \"kubernetes.io/projected/434cc10e-873d-4d79-a3e2-93f475b37ccf-kube-api-access-lq72n\") pod \"maas-api-revoke-sub-c05bb0ed-69cvl\" (UID: \"434cc10e-873d-4d79-a3e2-93f475b37ccf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c05bb0ed-69cvl" Nov 03 01:34:47.978151 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:47.977993 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/434cc10e-873d-4d79-a3e2-93f475b37ccf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c05bb0ed-69cvl\" (UID: \"434cc10e-873d-4d79-a3e2-93f475b37ccf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c05bb0ed-69cvl" Nov 03 01:34:47.978151 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:47.978029 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/434cc10e-873d-4d79-a3e2-93f475b37ccf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c05bb0ed-69cvl\" (UID: \"434cc10e-873d-4d79-a3e2-93f475b37ccf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c05bb0ed-69cvl" Nov 03 01:34:48.079761 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:48.079674 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lq72n\" (UniqueName: \"kubernetes.io/projected/434cc10e-873d-4d79-a3e2-93f475b37ccf-kube-api-access-lq72n\") pod \"maas-api-revoke-sub-c05bb0ed-69cvl\" (UID: \"434cc10e-873d-4d79-a3e2-93f475b37ccf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c05bb0ed-69cvl" Nov 03 01:34:48.079761 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:48.079723 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/434cc10e-873d-4d79-a3e2-93f475b37ccf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c05bb0ed-69cvl\" (UID: \"434cc10e-873d-4d79-a3e2-93f475b37ccf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c05bb0ed-69cvl" Nov 03 01:34:48.079761 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:48.079751 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/434cc10e-873d-4d79-a3e2-93f475b37ccf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c05bb0ed-69cvl\" (UID: \"434cc10e-873d-4d79-a3e2-93f475b37ccf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c05bb0ed-69cvl" Nov 03 01:34:48.080499 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:48.080475 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/434cc10e-873d-4d79-a3e2-93f475b37ccf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c05bb0ed-69cvl\" (UID: \"434cc10e-873d-4d79-a3e2-93f475b37ccf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c05bb0ed-69cvl" Nov 03 01:34:48.082293 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:48.082261 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/434cc10e-873d-4d79-a3e2-93f475b37ccf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c05bb0ed-69cvl\" (UID: \"434cc10e-873d-4d79-a3e2-93f475b37ccf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c05bb0ed-69cvl" Nov 03 01:34:48.088369 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:48.088346 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lq72n\" (UniqueName: \"kubernetes.io/projected/434cc10e-873d-4d79-a3e2-93f475b37ccf-kube-api-access-lq72n\") pod \"maas-api-revoke-sub-c05bb0ed-69cvl\" (UID: \"434cc10e-873d-4d79-a3e2-93f475b37ccf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c05bb0ed-69cvl" Nov 03 01:34:48.156140 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:48.156104 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c05bb0ed-69cvl" Nov 03 01:34:48.440849 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:48.440823 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c05bb0ed-69cvl"] Nov 03 01:34:48.441728 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:34:48.441695 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod434cc10e_873d_4d79_a3e2_93f475b37ccf.slice/crio-a66a4065846adc9b8a925dccc796231f81ad02507a87bca9f68fc0e36a014555 WatchSource:0}: Error finding container a66a4065846adc9b8a925dccc796231f81ad02507a87bca9f68fc0e36a014555: Status 404 returned error can't find the container with id a66a4065846adc9b8a925dccc796231f81ad02507a87bca9f68fc0e36a014555 Nov 03 01:34:49.114560 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:49.114519 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c05bb0ed-69cvl" event={"ID":"434cc10e-873d-4d79-a3e2-93f475b37ccf","Type":"ContainerStarted","Data":"426d3623a95d23577d85b059b270ae8c86282c77aa9586917f76a24c49ab9861"} Nov 03 01:34:49.115018 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:49.114580 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c05bb0ed-69cvl" event={"ID":"434cc10e-873d-4d79-a3e2-93f475b37ccf","Type":"ContainerStarted","Data":"a66a4065846adc9b8a925dccc796231f81ad02507a87bca9f68fc0e36a014555"} Nov 03 01:34:49.141895 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:49.141828 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c05bb0ed-69cvl" podStartSLOduration=2.141808113 podStartE2EDuration="2.141808113s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:34:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:34:49.13590016 +0000 UTC m=+3231.306903254" watchObservedRunningTime="2026-11-03 01:34:49.141808113 +0000 UTC m=+3231.312811210" Nov 03 01:34:50.128918 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:50.128877 4421 generic.go:356] "Generic (PLEG): container finished" podID="434cc10e-873d-4d79-a3e2-93f475b37ccf" containerID="426d3623a95d23577d85b059b270ae8c86282c77aa9586917f76a24c49ab9861" exitCode=0 Nov 03 01:34:50.129494 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:50.129037 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c05bb0ed-69cvl" event={"ID":"434cc10e-873d-4d79-a3e2-93f475b37ccf","Type":"ContainerDied","Data":"426d3623a95d23577d85b059b270ae8c86282c77aa9586917f76a24c49ab9861"} Nov 03 01:34:52.019917 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:52.019879 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c05bb0ed-69cvl" Nov 03 01:34:52.073298 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:52.070348 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/434cc10e-873d-4d79-a3e2-93f475b37ccf-kube-api-access-lq72n\" (UniqueName: \"kubernetes.io/projected/434cc10e-873d-4d79-a3e2-93f475b37ccf-kube-api-access-lq72n\") pod \"434cc10e-873d-4d79-a3e2-93f475b37ccf\" (UID: \"434cc10e-873d-4d79-a3e2-93f475b37ccf\") " Nov 03 01:34:52.073298 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:52.070410 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/434cc10e-873d-4d79-a3e2-93f475b37ccf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/434cc10e-873d-4d79-a3e2-93f475b37ccf-maas-api-openshift-ca\") pod \"434cc10e-873d-4d79-a3e2-93f475b37ccf\" (UID: \"434cc10e-873d-4d79-a3e2-93f475b37ccf\") " Nov 03 01:34:52.073298 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:52.070576 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/434cc10e-873d-4d79-a3e2-93f475b37ccf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/434cc10e-873d-4d79-a3e2-93f475b37ccf-maas-api-xks-ca\") pod \"434cc10e-873d-4d79-a3e2-93f475b37ccf\" (UID: \"434cc10e-873d-4d79-a3e2-93f475b37ccf\") " Nov 03 01:34:52.077302 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:52.075265 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/434cc10e-873d-4d79-a3e2-93f475b37ccf-maas-api-openshift-ca" pod "434cc10e-873d-4d79-a3e2-93f475b37ccf" (UID: "434cc10e-873d-4d79-a3e2-93f475b37ccf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:34:52.150347 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:52.144527 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/434cc10e-873d-4d79-a3e2-93f475b37ccf-kube-api-access-lq72n" pod "434cc10e-873d-4d79-a3e2-93f475b37ccf" (UID: "434cc10e-873d-4d79-a3e2-93f475b37ccf"). InnerVolumeSpecName "kube-api-access-lq72n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:34:52.193297 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:52.180079 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lq72n\" (UniqueName: \"kubernetes.io/projected/434cc10e-873d-4d79-a3e2-93f475b37ccf-kube-api-access-lq72n\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:34:52.193297 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:52.180133 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/434cc10e-873d-4d79-a3e2-93f475b37ccf-maas-api-openshift-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:34:52.200302 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:52.194658 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/434cc10e-873d-4d79-a3e2-93f475b37ccf-maas-api-xks-ca" pod "434cc10e-873d-4d79-a3e2-93f475b37ccf" (UID: "434cc10e-873d-4d79-a3e2-93f475b37ccf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:34:52.224298 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:52.217879 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c05bb0ed-69cvl" event={"ID":"434cc10e-873d-4d79-a3e2-93f475b37ccf","Type":"ContainerDied","Data":"a66a4065846adc9b8a925dccc796231f81ad02507a87bca9f68fc0e36a014555"} Nov 03 01:34:52.224298 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:52.217951 4421 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a66a4065846adc9b8a925dccc796231f81ad02507a87bca9f68fc0e36a014555" Nov 03 01:34:52.224298 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:52.218029 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c05bb0ed-69cvl" Nov 03 01:34:52.292050 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:34:52.292014 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/434cc10e-873d-4d79-a3e2-93f475b37ccf-maas-api-xks-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:35:02.773953 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:02.773915 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-sjhcb"] Nov 03 01:35:02.775159 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:02.774123 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-sjhcb" podUID="d1d81d36-60e4-4f12-81e6-7c5be483af12" containerName="istio-proxy" containerID="cri-o://10b36843f4f9c99389d43f4eeb582894ba750cd8d4209716a59be05217067d09" gracePeriod=30 Nov 03 01:35:04.828647 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:04.828593 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-b6afd6-7db7d566cf-rkxmr"] Nov 03 01:35:06.539991 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:06.539951 4421 patch_prober.go:29] interesting pod/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-sjhcb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 01:35:06.540467 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:06.540022 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-sjhcb" podUID="d1d81d36-60e4-4f12-81e6-7c5be483af12" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 01:35:08.076984 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:08.076957 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-sjhcb" Nov 03 01:35:08.109570 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:08.109544 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1d81d36-60e4-4f12-81e6-7c5be483af12-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1d81d36-60e4-4f12-81e6-7c5be483af12-workload-certs\") pod \"d1d81d36-60e4-4f12-81e6-7c5be483af12\" (UID: \"d1d81d36-60e4-4f12-81e6-7c5be483af12\") " Nov 03 01:35:08.109743 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:08.109678 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1d81d36-60e4-4f12-81e6-7c5be483af12-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1d81d36-60e4-4f12-81e6-7c5be483af12-workload-socket\") pod \"d1d81d36-60e4-4f12-81e6-7c5be483af12\" (UID: \"d1d81d36-60e4-4f12-81e6-7c5be483af12\") " Nov 03 01:35:08.109820 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:08.109748 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1d81d36-60e4-4f12-81e6-7c5be483af12-kube-api-access-c9vv6\" (UniqueName: \"kubernetes.io/projected/d1d81d36-60e4-4f12-81e6-7c5be483af12-kube-api-access-c9vv6\") pod \"d1d81d36-60e4-4f12-81e6-7c5be483af12\" (UID: \"d1d81d36-60e4-4f12-81e6-7c5be483af12\") " Nov 03 01:35:08.109820 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:08.109778 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1d81d36-60e4-4f12-81e6-7c5be483af12-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1d81d36-60e4-4f12-81e6-7c5be483af12-istio-envoy\") pod \"d1d81d36-60e4-4f12-81e6-7c5be483af12\" (UID: \"d1d81d36-60e4-4f12-81e6-7c5be483af12\") " Nov 03 01:35:08.109937 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:08.109889 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d1d81d36-60e4-4f12-81e6-7c5be483af12-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1d81d36-60e4-4f12-81e6-7c5be483af12-istio-podinfo\") pod \"d1d81d36-60e4-4f12-81e6-7c5be483af12\" (UID: \"d1d81d36-60e4-4f12-81e6-7c5be483af12\") " Nov 03 01:35:08.109996 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:08.109955 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1d81d36-60e4-4f12-81e6-7c5be483af12-istio-token\" (UniqueName: \"kubernetes.io/projected/d1d81d36-60e4-4f12-81e6-7c5be483af12-istio-token\") pod \"d1d81d36-60e4-4f12-81e6-7c5be483af12\" (UID: \"d1d81d36-60e4-4f12-81e6-7c5be483af12\") " Nov 03 01:35:08.110054 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:08.109996 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1d81d36-60e4-4f12-81e6-7c5be483af12-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1d81d36-60e4-4f12-81e6-7c5be483af12-istio-data\") pod \"d1d81d36-60e4-4f12-81e6-7c5be483af12\" (UID: \"d1d81d36-60e4-4f12-81e6-7c5be483af12\") " Nov 03 01:35:08.110054 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:08.110032 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1d81d36-60e4-4f12-81e6-7c5be483af12-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1d81d36-60e4-4f12-81e6-7c5be483af12-credential-socket\") pod \"d1d81d36-60e4-4f12-81e6-7c5be483af12\" (UID: \"d1d81d36-60e4-4f12-81e6-7c5be483af12\") " Nov 03 01:35:08.111849 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:08.110081 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1d81d36-60e4-4f12-81e6-7c5be483af12-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1d81d36-60e4-4f12-81e6-7c5be483af12-istiod-ca-cert\") pod \"d1d81d36-60e4-4f12-81e6-7c5be483af12\" (UID: \"d1d81d36-60e4-4f12-81e6-7c5be483af12\") " Nov 03 01:35:08.111849 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:08.110503 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1d81d36-60e4-4f12-81e6-7c5be483af12-workload-socket" pod "d1d81d36-60e4-4f12-81e6-7c5be483af12" (UID: "d1d81d36-60e4-4f12-81e6-7c5be483af12"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:35:08.111849 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:08.110895 4421 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1d81d36-60e4-4f12-81e6-7c5be483af12-workload-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:35:08.111849 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:08.111137 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1d81d36-60e4-4f12-81e6-7c5be483af12-istiod-ca-cert" pod "d1d81d36-60e4-4f12-81e6-7c5be483af12" (UID: "d1d81d36-60e4-4f12-81e6-7c5be483af12"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:35:08.111849 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:08.111408 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1d81d36-60e4-4f12-81e6-7c5be483af12-credential-socket" pod "d1d81d36-60e4-4f12-81e6-7c5be483af12" (UID: "d1d81d36-60e4-4f12-81e6-7c5be483af12"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:35:08.111849 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:08.111463 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1d81d36-60e4-4f12-81e6-7c5be483af12-istio-data" pod "d1d81d36-60e4-4f12-81e6-7c5be483af12" (UID: "d1d81d36-60e4-4f12-81e6-7c5be483af12"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:35:08.111849 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:08.111609 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1d81d36-60e4-4f12-81e6-7c5be483af12-workload-certs" pod "d1d81d36-60e4-4f12-81e6-7c5be483af12" (UID: "d1d81d36-60e4-4f12-81e6-7c5be483af12"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:35:08.113903 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:08.112894 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1d81d36-60e4-4f12-81e6-7c5be483af12-istio-envoy" pod "d1d81d36-60e4-4f12-81e6-7c5be483af12" (UID: "d1d81d36-60e4-4f12-81e6-7c5be483af12"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:35:08.113903 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:08.113232 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1d81d36-60e4-4f12-81e6-7c5be483af12-istio-token" pod "d1d81d36-60e4-4f12-81e6-7c5be483af12" (UID: "d1d81d36-60e4-4f12-81e6-7c5be483af12"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:35:08.113903 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:08.113349 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1d81d36-60e4-4f12-81e6-7c5be483af12-kube-api-access-c9vv6" pod "d1d81d36-60e4-4f12-81e6-7c5be483af12" (UID: "d1d81d36-60e4-4f12-81e6-7c5be483af12"). InnerVolumeSpecName "kube-api-access-c9vv6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:35:08.113903 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:08.113890 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d1d81d36-60e4-4f12-81e6-7c5be483af12-istio-podinfo" pod "d1d81d36-60e4-4f12-81e6-7c5be483af12" (UID: "d1d81d36-60e4-4f12-81e6-7c5be483af12"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:35:08.215129 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:08.215092 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9vv6\" (UniqueName: \"kubernetes.io/projected/d1d81d36-60e4-4f12-81e6-7c5be483af12-kube-api-access-c9vv6\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:35:08.215129 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:08.215130 4421 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1d81d36-60e4-4f12-81e6-7c5be483af12-istio-envoy\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:35:08.215551 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:08.215144 4421 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1d81d36-60e4-4f12-81e6-7c5be483af12-istio-podinfo\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:35:08.215551 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:08.215158 4421 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1d81d36-60e4-4f12-81e6-7c5be483af12-istio-token\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:35:08.215551 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:08.215170 4421 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1d81d36-60e4-4f12-81e6-7c5be483af12-istio-data\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:35:08.215551 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:08.215181 4421 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1d81d36-60e4-4f12-81e6-7c5be483af12-credential-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:35:08.215551 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:08.215193 4421 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1d81d36-60e4-4f12-81e6-7c5be483af12-istiod-ca-cert\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:35:08.215551 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:08.215206 4421 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1d81d36-60e4-4f12-81e6-7c5be483af12-workload-certs\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:35:08.410745 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:08.410649 4421 generic.go:356] "Generic (PLEG): container finished" podID="d1d81d36-60e4-4f12-81e6-7c5be483af12" containerID="10b36843f4f9c99389d43f4eeb582894ba750cd8d4209716a59be05217067d09" exitCode=0 Nov 03 01:35:08.410745 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:08.410712 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-sjhcb" Nov 03 01:35:08.410745 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:08.410732 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-sjhcb" event={"ID":"d1d81d36-60e4-4f12-81e6-7c5be483af12","Type":"ContainerDied","Data":"10b36843f4f9c99389d43f4eeb582894ba750cd8d4209716a59be05217067d09"} Nov 03 01:35:08.411044 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:08.410786 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-sjhcb" event={"ID":"d1d81d36-60e4-4f12-81e6-7c5be483af12","Type":"ContainerDied","Data":"80e56c89512261a0f15e115b325ab065c89f61494d6f8072bc29ec82e0e7a4e8"} Nov 03 01:35:08.411044 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:08.410808 4421 scope.go:122] "RemoveContainer" containerID="10b36843f4f9c99389d43f4eeb582894ba750cd8d4209716a59be05217067d09" Nov 03 01:35:08.437441 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:08.437414 4421 scope.go:122] "RemoveContainer" containerID="88843295f0b313641db6de1e5c6e5f67887fa1739aa9a8995bcf4d96a892cfeb" Nov 03 01:35:08.443627 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:08.443602 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-sjhcb"] Nov 03 01:35:08.447555 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:08.447529 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-sjhcb"] Nov 03 01:35:08.457018 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:08.456995 4421 scope.go:122] "RemoveContainer" containerID="10b36843f4f9c99389d43f4eeb582894ba750cd8d4209716a59be05217067d09" Nov 03 01:35:08.457385 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:35:08.457364 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10b36843f4f9c99389d43f4eeb582894ba750cd8d4209716a59be05217067d09\": container with ID starting with 10b36843f4f9c99389d43f4eeb582894ba750cd8d4209716a59be05217067d09 not found: ID does not exist" containerID="10b36843f4f9c99389d43f4eeb582894ba750cd8d4209716a59be05217067d09" Nov 03 01:35:08.457457 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:08.457398 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10b36843f4f9c99389d43f4eeb582894ba750cd8d4209716a59be05217067d09"} err="failed to get container status \"10b36843f4f9c99389d43f4eeb582894ba750cd8d4209716a59be05217067d09\": rpc error: code = NotFound desc = could not find container \"10b36843f4f9c99389d43f4eeb582894ba750cd8d4209716a59be05217067d09\": container with ID starting with 10b36843f4f9c99389d43f4eeb582894ba750cd8d4209716a59be05217067d09 not found: ID does not exist" Nov 03 01:35:08.457457 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:08.457418 4421 scope.go:122] "RemoveContainer" containerID="88843295f0b313641db6de1e5c6e5f67887fa1739aa9a8995bcf4d96a892cfeb" Nov 03 01:35:08.457763 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:35:08.457746 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88843295f0b313641db6de1e5c6e5f67887fa1739aa9a8995bcf4d96a892cfeb\": container with ID starting with 88843295f0b313641db6de1e5c6e5f67887fa1739aa9a8995bcf4d96a892cfeb not found: ID does not exist" containerID="88843295f0b313641db6de1e5c6e5f67887fa1739aa9a8995bcf4d96a892cfeb" Nov 03 01:35:08.457814 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:08.457767 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88843295f0b313641db6de1e5c6e5f67887fa1739aa9a8995bcf4d96a892cfeb"} err="failed to get container status \"88843295f0b313641db6de1e5c6e5f67887fa1739aa9a8995bcf4d96a892cfeb\": rpc error: code = NotFound desc = could not find container \"88843295f0b313641db6de1e5c6e5f67887fa1739aa9a8995bcf4d96a892cfeb\": container with ID starting with 88843295f0b313641db6de1e5c6e5f67887fa1739aa9a8995bcf4d96a892cfeb not found: ID does not exist" Nov 03 01:35:09.829762 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:09.829707 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b6afd6-7db7d566cf-rkxmr" podUID="18de1ce4-36e1-4a3c-90cc-ecbac5dcfc80" containerName="payload-pre-processing" containerID="cri-o://0b370d615a26cca3f7c32796de54eb12c2a3fb3fee464e67cc8c0c5acb0379a8" gracePeriod=25 Nov 03 01:35:09.981728 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:09.981693 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1d81d36-60e4-4f12-81e6-7c5be483af12" path="/var/lib/kubelet/pods/d1d81d36-60e4-4f12-81e6-7c5be483af12/volumes" Nov 03 01:35:13.332301 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:13.326192 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-vhvf6"] Nov 03 01:35:13.332301 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:13.327427 4421 cpu_manager.go:414] "containerMap: removing container" podUID="434cc10e-873d-4d79-a3e2-93f475b37ccf" containerName="revoke-keys" Nov 03 01:35:13.332301 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:13.327453 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="434cc10e-873d-4d79-a3e2-93f475b37ccf" containerName="revoke-keys" Nov 03 01:35:13.332301 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:13.327521 4421 cpu_manager.go:414] "containerMap: removing container" podUID="d1d81d36-60e4-4f12-81e6-7c5be483af12" containerName="istio-proxy" Nov 03 01:35:13.332301 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:13.327529 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1d81d36-60e4-4f12-81e6-7c5be483af12" containerName="istio-proxy" Nov 03 01:35:13.332301 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:13.327569 4421 cpu_manager.go:414] "containerMap: removing container" podUID="d1d81d36-60e4-4f12-81e6-7c5be483af12" containerName="istio-proxy" Nov 03 01:35:13.332301 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:13.327578 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1d81d36-60e4-4f12-81e6-7c5be483af12" containerName="istio-proxy" Nov 03 01:35:13.332301 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:13.327770 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1d81d36-60e4-4f12-81e6-7c5be483af12" containerName="istio-proxy" Nov 03 01:35:13.332301 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:13.327797 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="434cc10e-873d-4d79-a3e2-93f475b37ccf" containerName="revoke-keys" Nov 03 01:35:13.343295 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:13.338463 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-vhvf6" Nov 03 01:35:13.351299 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:13.343922 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-vhvf6"] Nov 03 01:35:13.423328 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:13.423078 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4246b3f1-fc55-4059-9983-06dbeca10768-istio-token\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-vhvf6\" (UID: \"4246b3f1-fc55-4059-9983-06dbeca10768\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-vhvf6" Nov 03 01:35:13.423328 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:13.423214 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4246b3f1-fc55-4059-9983-06dbeca10768-workload-certs\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-vhvf6\" (UID: \"4246b3f1-fc55-4059-9983-06dbeca10768\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-vhvf6" Nov 03 01:35:13.423328 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:13.423251 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4246b3f1-fc55-4059-9983-06dbeca10768-credential-socket\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-vhvf6\" (UID: \"4246b3f1-fc55-4059-9983-06dbeca10768\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-vhvf6" Nov 03 01:35:13.423328 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:13.423310 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4246b3f1-fc55-4059-9983-06dbeca10768-workload-socket\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-vhvf6\" (UID: \"4246b3f1-fc55-4059-9983-06dbeca10768\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-vhvf6" Nov 03 01:35:13.423731 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:13.423431 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4246b3f1-fc55-4059-9983-06dbeca10768-istio-envoy\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-vhvf6\" (UID: \"4246b3f1-fc55-4059-9983-06dbeca10768\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-vhvf6" Nov 03 01:35:13.423731 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:13.423472 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4246b3f1-fc55-4059-9983-06dbeca10768-istio-data\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-vhvf6\" (UID: \"4246b3f1-fc55-4059-9983-06dbeca10768\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-vhvf6" Nov 03 01:35:13.423731 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:13.423520 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4246b3f1-fc55-4059-9983-06dbeca10768-istio-podinfo\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-vhvf6\" (UID: \"4246b3f1-fc55-4059-9983-06dbeca10768\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-vhvf6" Nov 03 01:35:13.423731 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:13.423660 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4246b3f1-fc55-4059-9983-06dbeca10768-istiod-ca-cert\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-vhvf6\" (UID: \"4246b3f1-fc55-4059-9983-06dbeca10768\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-vhvf6" Nov 03 01:35:13.423731 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:13.423691 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfw97\" (UniqueName: \"kubernetes.io/projected/4246b3f1-fc55-4059-9983-06dbeca10768-kube-api-access-gfw97\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-vhvf6\" (UID: \"4246b3f1-fc55-4059-9983-06dbeca10768\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-vhvf6" Nov 03 01:35:13.525238 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:13.525199 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4246b3f1-fc55-4059-9983-06dbeca10768-istio-token\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-vhvf6\" (UID: \"4246b3f1-fc55-4059-9983-06dbeca10768\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-vhvf6" Nov 03 01:35:13.525451 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:13.525357 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4246b3f1-fc55-4059-9983-06dbeca10768-workload-certs\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-vhvf6\" (UID: \"4246b3f1-fc55-4059-9983-06dbeca10768\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-vhvf6" Nov 03 01:35:13.525451 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:13.525388 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4246b3f1-fc55-4059-9983-06dbeca10768-credential-socket\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-vhvf6\" (UID: \"4246b3f1-fc55-4059-9983-06dbeca10768\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-vhvf6" Nov 03 01:35:13.525451 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:13.525430 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4246b3f1-fc55-4059-9983-06dbeca10768-workload-socket\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-vhvf6\" (UID: \"4246b3f1-fc55-4059-9983-06dbeca10768\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-vhvf6" Nov 03 01:35:13.525737 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:13.525536 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4246b3f1-fc55-4059-9983-06dbeca10768-istio-envoy\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-vhvf6\" (UID: \"4246b3f1-fc55-4059-9983-06dbeca10768\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-vhvf6" Nov 03 01:35:13.525737 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:13.525569 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4246b3f1-fc55-4059-9983-06dbeca10768-istio-data\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-vhvf6\" (UID: \"4246b3f1-fc55-4059-9983-06dbeca10768\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-vhvf6" Nov 03 01:35:13.525737 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:13.525613 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4246b3f1-fc55-4059-9983-06dbeca10768-istio-podinfo\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-vhvf6\" (UID: \"4246b3f1-fc55-4059-9983-06dbeca10768\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-vhvf6" Nov 03 01:35:13.525915 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:13.525741 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4246b3f1-fc55-4059-9983-06dbeca10768-istiod-ca-cert\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-vhvf6\" (UID: \"4246b3f1-fc55-4059-9983-06dbeca10768\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-vhvf6" Nov 03 01:35:13.525915 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:13.525773 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfw97\" (UniqueName: \"kubernetes.io/projected/4246b3f1-fc55-4059-9983-06dbeca10768-kube-api-access-gfw97\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-vhvf6\" (UID: \"4246b3f1-fc55-4059-9983-06dbeca10768\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-vhvf6" Nov 03 01:35:13.525915 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:13.525853 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4246b3f1-fc55-4059-9983-06dbeca10768-workload-certs\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-vhvf6\" (UID: \"4246b3f1-fc55-4059-9983-06dbeca10768\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-vhvf6" Nov 03 01:35:13.525915 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:13.525880 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4246b3f1-fc55-4059-9983-06dbeca10768-workload-socket\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-vhvf6\" (UID: \"4246b3f1-fc55-4059-9983-06dbeca10768\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-vhvf6" Nov 03 01:35:13.525915 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:13.525900 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4246b3f1-fc55-4059-9983-06dbeca10768-credential-socket\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-vhvf6\" (UID: \"4246b3f1-fc55-4059-9983-06dbeca10768\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-vhvf6" Nov 03 01:35:13.526164 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:13.526143 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4246b3f1-fc55-4059-9983-06dbeca10768-istio-data\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-vhvf6\" (UID: \"4246b3f1-fc55-4059-9983-06dbeca10768\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-vhvf6" Nov 03 01:35:13.526594 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:13.526566 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4246b3f1-fc55-4059-9983-06dbeca10768-istiod-ca-cert\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-vhvf6\" (UID: \"4246b3f1-fc55-4059-9983-06dbeca10768\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-vhvf6" Nov 03 01:35:13.528328 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:13.528303 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4246b3f1-fc55-4059-9983-06dbeca10768-istio-envoy\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-vhvf6\" (UID: \"4246b3f1-fc55-4059-9983-06dbeca10768\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-vhvf6" Nov 03 01:35:13.528463 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:13.528445 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4246b3f1-fc55-4059-9983-06dbeca10768-istio-podinfo\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-vhvf6\" (UID: \"4246b3f1-fc55-4059-9983-06dbeca10768\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-vhvf6" Nov 03 01:35:13.537684 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:13.537656 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4246b3f1-fc55-4059-9983-06dbeca10768-istio-token\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-vhvf6\" (UID: \"4246b3f1-fc55-4059-9983-06dbeca10768\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-vhvf6" Nov 03 01:35:13.538234 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:13.538211 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfw97\" (UniqueName: \"kubernetes.io/projected/4246b3f1-fc55-4059-9983-06dbeca10768-kube-api-access-gfw97\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-vhvf6\" (UID: \"4246b3f1-fc55-4059-9983-06dbeca10768\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-vhvf6" Nov 03 01:35:13.716354 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:13.715672 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-vhvf6" Nov 03 01:35:14.019178 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:14.016999 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-vhvf6"] Nov 03 01:35:14.024492 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:14.024435 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:35:14.024642 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:14.024546 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:35:14.024642 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:14.024591 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:35:14.475170 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:14.475075 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-vhvf6" event={"ID":"4246b3f1-fc55-4059-9983-06dbeca10768","Type":"ContainerStarted","Data":"1aa0aa1962028996bc494bc47f39e170cc90bc0396bbaef258ad31ab6fc6fb3f"} Nov 03 01:35:14.475595 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:14.475180 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-vhvf6" event={"ID":"4246b3f1-fc55-4059-9983-06dbeca10768","Type":"ContainerStarted","Data":"fb4c46c3bcd1de600170b4238946aa6ac465d4c63daed44e21c63f8b17778822"} Nov 03 01:35:14.502925 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:14.502856 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-vhvf6" podStartSLOduration=1.5028419830000002 podStartE2EDuration="1.502841983s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:35:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:35:14.496500547 +0000 UTC m=+3256.667503644" watchObservedRunningTime="2026-11-03 01:35:14.502841983 +0000 UTC m=+3256.673845080" Nov 03 01:35:14.663299 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:14.660085 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6dac96c1-8tdw5"] Nov 03 01:35:14.663299 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:14.661868 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1d81d36-60e4-4f12-81e6-7c5be483af12" containerName="istio-proxy" Nov 03 01:35:14.682353 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:14.676577 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dac96c1-8tdw5" Nov 03 01:35:14.691999 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:14.691586 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:35:14.691999 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:14.691742 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6q4xp\"" Nov 03 01:35:14.716496 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:14.716466 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-vhvf6" Nov 03 01:35:14.723561 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:14.722452 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6dac96c1-8tdw5"] Nov 03 01:35:14.795818 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:14.794603 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8mvz\" (UniqueName: \"kubernetes.io/projected/03015bfc-aaac-4f5a-b909-790a18fc61f3-kube-api-access-h8mvz\") pod \"maas-api-revoke-sub-6dac96c1-8tdw5\" (UID: \"03015bfc-aaac-4f5a-b909-790a18fc61f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dac96c1-8tdw5" Nov 03 01:35:14.795818 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:14.794715 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03015bfc-aaac-4f5a-b909-790a18fc61f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6dac96c1-8tdw5\" (UID: \"03015bfc-aaac-4f5a-b909-790a18fc61f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dac96c1-8tdw5" Nov 03 01:35:14.795818 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:14.794757 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03015bfc-aaac-4f5a-b909-790a18fc61f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6dac96c1-8tdw5\" (UID: \"03015bfc-aaac-4f5a-b909-790a18fc61f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dac96c1-8tdw5" Nov 03 01:35:14.846940 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:35:14.846906 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 01:35:14.899775 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:14.899725 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8mvz\" (UniqueName: \"kubernetes.io/projected/03015bfc-aaac-4f5a-b909-790a18fc61f3-kube-api-access-h8mvz\") pod \"maas-api-revoke-sub-6dac96c1-8tdw5\" (UID: \"03015bfc-aaac-4f5a-b909-790a18fc61f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dac96c1-8tdw5" Nov 03 01:35:14.900659 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:14.899856 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03015bfc-aaac-4f5a-b909-790a18fc61f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6dac96c1-8tdw5\" (UID: \"03015bfc-aaac-4f5a-b909-790a18fc61f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dac96c1-8tdw5" Nov 03 01:35:14.900659 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:14.899945 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03015bfc-aaac-4f5a-b909-790a18fc61f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6dac96c1-8tdw5\" (UID: \"03015bfc-aaac-4f5a-b909-790a18fc61f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dac96c1-8tdw5" Nov 03 01:35:14.901010 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:14.900977 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03015bfc-aaac-4f5a-b909-790a18fc61f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6dac96c1-8tdw5\" (UID: \"03015bfc-aaac-4f5a-b909-790a18fc61f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dac96c1-8tdw5" Nov 03 01:35:14.908742 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:14.908706 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03015bfc-aaac-4f5a-b909-790a18fc61f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6dac96c1-8tdw5\" (UID: \"03015bfc-aaac-4f5a-b909-790a18fc61f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dac96c1-8tdw5" Nov 03 01:35:14.912785 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:14.912630 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8mvz\" (UniqueName: \"kubernetes.io/projected/03015bfc-aaac-4f5a-b909-790a18fc61f3-kube-api-access-h8mvz\") pod \"maas-api-revoke-sub-6dac96c1-8tdw5\" (UID: \"03015bfc-aaac-4f5a-b909-790a18fc61f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dac96c1-8tdw5" Nov 03 01:35:15.039369 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:15.033872 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dac96c1-8tdw5" Nov 03 01:35:15.357358 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:15.356944 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6dac96c1-8tdw5"] Nov 03 01:35:15.381299 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:35:15.373625 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03015bfc_aaac_4f5a_b909_790a18fc61f3.slice/crio-9acc074bcc26cf9727c6196f0cae54019e52752250645ebf9ab63879e7f17f90 WatchSource:0}: Error finding container 9acc074bcc26cf9727c6196f0cae54019e52752250645ebf9ab63879e7f17f90: Status 404 returned error can't find the container with id 9acc074bcc26cf9727c6196f0cae54019e52752250645ebf9ab63879e7f17f90 Nov 03 01:35:15.488358 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:15.488329 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dac96c1-8tdw5" event={"ID":"03015bfc-aaac-4f5a-b909-790a18fc61f3","Type":"ContainerStarted","Data":"9acc074bcc26cf9727c6196f0cae54019e52752250645ebf9ab63879e7f17f90"} Nov 03 01:35:15.733428 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:15.732731 4421 patch_prober.go:29] interesting pod/e2e-mt-737bf2d8-openshift-default-56dc96b957-vhvf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.96:15021/healthz/ready\": dial tcp 10.134.1.96:15021: connect: connection refused" start-of-body= Nov 03 01:35:15.733428 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:15.732788 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-vhvf6" podUID="4246b3f1-fc55-4059-9983-06dbeca10768" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.96:15021/healthz/ready\": dial tcp 10.134.1.96:15021: connect: connection refused" Nov 03 01:35:16.481603 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:16.481439 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b6afd6-7db7d566cf-rkxmr" podUID="18de1ce4-36e1-4a3c-90cc-ecbac5dcfc80" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:35:16.499165 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:16.499126 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dac96c1-8tdw5" event={"ID":"03015bfc-aaac-4f5a-b909-790a18fc61f3","Type":"ContainerStarted","Data":"b53c38ff45bbf00536acbb28cdf51ca62bb7a00e6280f941ea73f4f37bb8e649"} Nov 03 01:35:16.534224 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:16.534164 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dac96c1-8tdw5" podStartSLOduration=2.534141941 podStartE2EDuration="2.534141941s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:35:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:35:16.522569082 +0000 UTC m=+3258.693572181" watchObservedRunningTime="2026-11-03 01:35:16.534141941 +0000 UTC m=+3258.705145036" Nov 03 01:35:16.732345 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:16.724004 4421 patch_prober.go:29] interesting pod/e2e-mt-737bf2d8-openshift-default-56dc96b957-vhvf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.96:15021/healthz/ready\": dial tcp 10.134.1.96:15021: connect: connection refused" start-of-body= Nov 03 01:35:16.732345 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:16.724063 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-vhvf6" podUID="4246b3f1-fc55-4059-9983-06dbeca10768" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.96:15021/healthz/ready\": dial tcp 10.134.1.96:15021: connect: connection refused" Nov 03 01:35:17.509955 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:17.509917 4421 generic.go:356] "Generic (PLEG): container finished" podID="03015bfc-aaac-4f5a-b909-790a18fc61f3" containerID="b53c38ff45bbf00536acbb28cdf51ca62bb7a00e6280f941ea73f4f37bb8e649" exitCode=0 Nov 03 01:35:17.510690 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:17.510648 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dac96c1-8tdw5" event={"ID":"03015bfc-aaac-4f5a-b909-790a18fc61f3","Type":"ContainerDied","Data":"b53c38ff45bbf00536acbb28cdf51ca62bb7a00e6280f941ea73f4f37bb8e649"} Nov 03 01:35:17.724299 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:17.717802 4421 patch_prober.go:29] interesting pod/e2e-mt-737bf2d8-openshift-default-56dc96b957-vhvf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.96:15021/healthz/ready\": dial tcp 10.134.1.96:15021: connect: connection refused" start-of-body= Nov 03 01:35:17.724299 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:17.717872 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-vhvf6" podUID="4246b3f1-fc55-4059-9983-06dbeca10768" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.96:15021/healthz/ready\": dial tcp 10.134.1.96:15021: connect: connection refused" Nov 03 01:35:18.742175 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:18.742137 4421 patch_prober.go:29] interesting pod/e2e-mt-737bf2d8-openshift-default-56dc96b957-vhvf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.96:15021/healthz/ready\": dial tcp 10.134.1.96:15021: connect: connection refused" start-of-body= Nov 03 01:35:18.742740 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:18.742715 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-vhvf6" podUID="4246b3f1-fc55-4059-9983-06dbeca10768" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.96:15021/healthz/ready\": dial tcp 10.134.1.96:15021: connect: connection refused" Nov 03 01:35:18.787525 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:18.787498 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dac96c1-8tdw5" Nov 03 01:35:18.904421 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:18.904336 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03015bfc-aaac-4f5a-b909-790a18fc61f3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03015bfc-aaac-4f5a-b909-790a18fc61f3-maas-api-openshift-ca\") pod \"03015bfc-aaac-4f5a-b909-790a18fc61f3\" (UID: \"03015bfc-aaac-4f5a-b909-790a18fc61f3\") " Nov 03 01:35:18.904586 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:18.904563 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03015bfc-aaac-4f5a-b909-790a18fc61f3-kube-api-access-h8mvz\" (UniqueName: \"kubernetes.io/projected/03015bfc-aaac-4f5a-b909-790a18fc61f3-kube-api-access-h8mvz\") pod \"03015bfc-aaac-4f5a-b909-790a18fc61f3\" (UID: \"03015bfc-aaac-4f5a-b909-790a18fc61f3\") " Nov 03 01:35:18.904643 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:18.904605 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/03015bfc-aaac-4f5a-b909-790a18fc61f3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03015bfc-aaac-4f5a-b909-790a18fc61f3-maas-api-xks-ca\") pod \"03015bfc-aaac-4f5a-b909-790a18fc61f3\" (UID: \"03015bfc-aaac-4f5a-b909-790a18fc61f3\") " Nov 03 01:35:18.904714 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:18.904688 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03015bfc-aaac-4f5a-b909-790a18fc61f3-maas-api-openshift-ca" pod "03015bfc-aaac-4f5a-b909-790a18fc61f3" (UID: "03015bfc-aaac-4f5a-b909-790a18fc61f3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:35:18.919296 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:18.905447 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03015bfc-aaac-4f5a-b909-790a18fc61f3-maas-api-openshift-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:35:18.931544 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:18.931505 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03015bfc-aaac-4f5a-b909-790a18fc61f3-kube-api-access-h8mvz" pod "03015bfc-aaac-4f5a-b909-790a18fc61f3" (UID: "03015bfc-aaac-4f5a-b909-790a18fc61f3"). InnerVolumeSpecName "kube-api-access-h8mvz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:35:18.932416 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:18.932383 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03015bfc-aaac-4f5a-b909-790a18fc61f3-maas-api-xks-ca" pod "03015bfc-aaac-4f5a-b909-790a18fc61f3" (UID: "03015bfc-aaac-4f5a-b909-790a18fc61f3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:35:19.031311 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:19.023910 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8mvz\" (UniqueName: \"kubernetes.io/projected/03015bfc-aaac-4f5a-b909-790a18fc61f3-kube-api-access-h8mvz\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:35:19.031311 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:19.030379 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03015bfc-aaac-4f5a-b909-790a18fc61f3-maas-api-xks-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:35:19.555451 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:19.555419 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dac96c1-8tdw5" Nov 03 01:35:19.555451 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:19.555448 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dac96c1-8tdw5" event={"ID":"03015bfc-aaac-4f5a-b909-790a18fc61f3","Type":"ContainerDied","Data":"9acc074bcc26cf9727c6196f0cae54019e52752250645ebf9ab63879e7f17f90"} Nov 03 01:35:19.555765 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:19.555517 4421 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9acc074bcc26cf9727c6196f0cae54019e52752250645ebf9ab63879e7f17f90" Nov 03 01:35:19.721304 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:19.717386 4421 patch_prober.go:29] interesting pod/e2e-mt-737bf2d8-openshift-default-56dc96b957-vhvf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.96:15021/healthz/ready\": dial tcp 10.134.1.96:15021: connect: connection refused" start-of-body= Nov 03 01:35:19.721304 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:19.717454 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-vhvf6" podUID="4246b3f1-fc55-4059-9983-06dbeca10768" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.96:15021/healthz/ready\": dial tcp 10.134.1.96:15021: connect: connection refused" Nov 03 01:35:20.103393 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:20.103357 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-b6afd6-openshift-default-78d46d7dcc-dg79z"] Nov 03 01:35:20.105222 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:20.105172 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-b6afd6-openshift-default-78d46d7dcc-dg79z" podUID="50b59bdf-0949-473f-8849-bb5648c6cba1" containerName="istio-proxy" containerID="cri-o://157d6621f49d6e3d9575bc04f56fd034ae3f052d0e4e15f5e2ca92cb14e760dc" gracePeriod=30 Nov 03 01:35:20.716613 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:20.716573 4421 patch_prober.go:29] interesting pod/e2e-mt-737bf2d8-openshift-default-56dc96b957-vhvf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.96:15021/healthz/ready\": dial tcp 10.134.1.96:15021: connect: connection refused" start-of-body= Nov 03 01:35:20.716801 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:20.716631 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-vhvf6" podUID="4246b3f1-fc55-4059-9983-06dbeca10768" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.96:15021/healthz/ready\": dial tcp 10.134.1.96:15021: connect: connection refused" Nov 03 01:35:21.723843 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:21.723809 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-vhvf6" Nov 03 01:35:21.724420 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:21.724394 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-vhvf6" Nov 03 01:35:21.725288 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:21.725251 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-vhvf6" Nov 03 01:35:22.897663 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:22.897627 4421 patch_prober.go:29] interesting pod/e2e-inf-a-b6afd6-openshift-default-78d46d7dcc-dg79z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 01:35:22.898082 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:22.897685 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-b6afd6-openshift-default-78d46d7dcc-dg79z" podUID="50b59bdf-0949-473f-8849-bb5648c6cba1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 01:35:25.617056 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:25.616866 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b6afd6-openshift-default-78d46d7dcc-dg79z" Nov 03 01:35:25.625028 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:25.624373 4421 generic.go:356] "Generic (PLEG): container finished" podID="50b59bdf-0949-473f-8849-bb5648c6cba1" containerID="157d6621f49d6e3d9575bc04f56fd034ae3f052d0e4e15f5e2ca92cb14e760dc" exitCode=0 Nov 03 01:35:25.625028 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:25.624518 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b6afd6-openshift-default-78d46d7dcc-dg79z" event={"ID":"50b59bdf-0949-473f-8849-bb5648c6cba1","Type":"ContainerDied","Data":"157d6621f49d6e3d9575bc04f56fd034ae3f052d0e4e15f5e2ca92cb14e760dc"} Nov 03 01:35:25.625028 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:25.624579 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b6afd6-openshift-default-78d46d7dcc-dg79z" event={"ID":"50b59bdf-0949-473f-8849-bb5648c6cba1","Type":"ContainerDied","Data":"fc5052938d5a2a6ef4a1e69afadee14bd0057a16400deb40a8d42320f8784725"} Nov 03 01:35:25.625028 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:25.624613 4421 scope.go:122] "RemoveContainer" containerID="157d6621f49d6e3d9575bc04f56fd034ae3f052d0e4e15f5e2ca92cb14e760dc" Nov 03 01:35:25.625028 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:25.624794 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b6afd6-openshift-default-78d46d7dcc-dg79z" Nov 03 01:35:25.689047 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:25.689013 4421 scope.go:122] "RemoveContainer" containerID="6a0daece2fecdfdfc172298e4260507ca77c25680ec24beb166c78a92f218ce7" Nov 03 01:35:25.707187 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:25.707164 4421 scope.go:122] "RemoveContainer" containerID="157d6621f49d6e3d9575bc04f56fd034ae3f052d0e4e15f5e2ca92cb14e760dc" Nov 03 01:35:25.708047 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:35:25.708018 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"157d6621f49d6e3d9575bc04f56fd034ae3f052d0e4e15f5e2ca92cb14e760dc\": container with ID starting with 157d6621f49d6e3d9575bc04f56fd034ae3f052d0e4e15f5e2ca92cb14e760dc not found: ID does not exist" containerID="157d6621f49d6e3d9575bc04f56fd034ae3f052d0e4e15f5e2ca92cb14e760dc" Nov 03 01:35:25.708138 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:25.708058 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"157d6621f49d6e3d9575bc04f56fd034ae3f052d0e4e15f5e2ca92cb14e760dc"} err="failed to get container status \"157d6621f49d6e3d9575bc04f56fd034ae3f052d0e4e15f5e2ca92cb14e760dc\": rpc error: code = NotFound desc = could not find container \"157d6621f49d6e3d9575bc04f56fd034ae3f052d0e4e15f5e2ca92cb14e760dc\": container with ID starting with 157d6621f49d6e3d9575bc04f56fd034ae3f052d0e4e15f5e2ca92cb14e760dc not found: ID does not exist" Nov 03 01:35:25.708138 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:25.708080 4421 scope.go:122] "RemoveContainer" containerID="6a0daece2fecdfdfc172298e4260507ca77c25680ec24beb166c78a92f218ce7" Nov 03 01:35:25.708430 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:35:25.708411 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a0daece2fecdfdfc172298e4260507ca77c25680ec24beb166c78a92f218ce7\": container with ID starting with 6a0daece2fecdfdfc172298e4260507ca77c25680ec24beb166c78a92f218ce7 not found: ID does not exist" containerID="6a0daece2fecdfdfc172298e4260507ca77c25680ec24beb166c78a92f218ce7" Nov 03 01:35:25.708488 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:25.708435 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a0daece2fecdfdfc172298e4260507ca77c25680ec24beb166c78a92f218ce7"} err="failed to get container status \"6a0daece2fecdfdfc172298e4260507ca77c25680ec24beb166c78a92f218ce7\": rpc error: code = NotFound desc = could not find container \"6a0daece2fecdfdfc172298e4260507ca77c25680ec24beb166c78a92f218ce7\": container with ID starting with 6a0daece2fecdfdfc172298e4260507ca77c25680ec24beb166c78a92f218ce7 not found: ID does not exist" Nov 03 01:35:25.712053 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:25.712024 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50b59bdf-0949-473f-8849-bb5648c6cba1-kube-api-access-gqzq2\" (UniqueName: \"kubernetes.io/projected/50b59bdf-0949-473f-8849-bb5648c6cba1-kube-api-access-gqzq2\") pod \"50b59bdf-0949-473f-8849-bb5648c6cba1\" (UID: \"50b59bdf-0949-473f-8849-bb5648c6cba1\") " Nov 03 01:35:25.712184 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:25.712127 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50b59bdf-0949-473f-8849-bb5648c6cba1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50b59bdf-0949-473f-8849-bb5648c6cba1-istio-envoy\") pod \"50b59bdf-0949-473f-8849-bb5648c6cba1\" (UID: \"50b59bdf-0949-473f-8849-bb5648c6cba1\") " Nov 03 01:35:25.712257 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:25.712216 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50b59bdf-0949-473f-8849-bb5648c6cba1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50b59bdf-0949-473f-8849-bb5648c6cba1-workload-socket\") pod \"50b59bdf-0949-473f-8849-bb5648c6cba1\" (UID: \"50b59bdf-0949-473f-8849-bb5648c6cba1\") " Nov 03 01:35:25.712374 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:25.712267 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50b59bdf-0949-473f-8849-bb5648c6cba1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50b59bdf-0949-473f-8849-bb5648c6cba1-credential-socket\") pod \"50b59bdf-0949-473f-8849-bb5648c6cba1\" (UID: \"50b59bdf-0949-473f-8849-bb5648c6cba1\") " Nov 03 01:35:25.712436 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:25.712394 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/50b59bdf-0949-473f-8849-bb5648c6cba1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50b59bdf-0949-473f-8849-bb5648c6cba1-istio-podinfo\") pod \"50b59bdf-0949-473f-8849-bb5648c6cba1\" (UID: \"50b59bdf-0949-473f-8849-bb5648c6cba1\") " Nov 03 01:35:25.712494 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:25.712454 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50b59bdf-0949-473f-8849-bb5648c6cba1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50b59bdf-0949-473f-8849-bb5648c6cba1-workload-certs\") pod \"50b59bdf-0949-473f-8849-bb5648c6cba1\" (UID: \"50b59bdf-0949-473f-8849-bb5648c6cba1\") " Nov 03 01:35:25.712565 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:25.712512 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50b59bdf-0949-473f-8849-bb5648c6cba1-istio-token\" (UniqueName: \"kubernetes.io/projected/50b59bdf-0949-473f-8849-bb5648c6cba1-istio-token\") pod \"50b59bdf-0949-473f-8849-bb5648c6cba1\" (UID: \"50b59bdf-0949-473f-8849-bb5648c6cba1\") " Nov 03 01:35:25.712565 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:25.712541 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50b59bdf-0949-473f-8849-bb5648c6cba1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50b59bdf-0949-473f-8849-bb5648c6cba1-istiod-ca-cert\") pod \"50b59bdf-0949-473f-8849-bb5648c6cba1\" (UID: \"50b59bdf-0949-473f-8849-bb5648c6cba1\") " Nov 03 01:35:25.712688 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:25.712592 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50b59bdf-0949-473f-8849-bb5648c6cba1-credential-socket" pod "50b59bdf-0949-473f-8849-bb5648c6cba1" (UID: "50b59bdf-0949-473f-8849-bb5648c6cba1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:35:25.712688 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:25.712602 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50b59bdf-0949-473f-8849-bb5648c6cba1-workload-socket" pod "50b59bdf-0949-473f-8849-bb5648c6cba1" (UID: "50b59bdf-0949-473f-8849-bb5648c6cba1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:35:25.712688 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:25.712645 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50b59bdf-0949-473f-8849-bb5648c6cba1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50b59bdf-0949-473f-8849-bb5648c6cba1-istio-data\") pod \"50b59bdf-0949-473f-8849-bb5648c6cba1\" (UID: \"50b59bdf-0949-473f-8849-bb5648c6cba1\") " Nov 03 01:35:25.712845 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:25.712827 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50b59bdf-0949-473f-8849-bb5648c6cba1-workload-certs" pod "50b59bdf-0949-473f-8849-bb5648c6cba1" (UID: "50b59bdf-0949-473f-8849-bb5648c6cba1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:35:25.713130 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:25.713102 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50b59bdf-0949-473f-8849-bb5648c6cba1-istio-data" pod "50b59bdf-0949-473f-8849-bb5648c6cba1" (UID: "50b59bdf-0949-473f-8849-bb5648c6cba1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:35:25.713224 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:25.713110 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50b59bdf-0949-473f-8849-bb5648c6cba1-istiod-ca-cert" pod "50b59bdf-0949-473f-8849-bb5648c6cba1" (UID: "50b59bdf-0949-473f-8849-bb5648c6cba1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:35:25.713984 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:25.713396 4421 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50b59bdf-0949-473f-8849-bb5648c6cba1-workload-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:35:25.713984 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:25.713421 4421 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50b59bdf-0949-473f-8849-bb5648c6cba1-credential-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:35:25.713984 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:25.713437 4421 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50b59bdf-0949-473f-8849-bb5648c6cba1-workload-certs\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:35:25.713984 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:25.713452 4421 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50b59bdf-0949-473f-8849-bb5648c6cba1-istiod-ca-cert\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:35:25.713984 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:25.713469 4421 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50b59bdf-0949-473f-8849-bb5648c6cba1-istio-data\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:35:25.715197 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:25.715171 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/50b59bdf-0949-473f-8849-bb5648c6cba1-istio-podinfo" pod "50b59bdf-0949-473f-8849-bb5648c6cba1" (UID: "50b59bdf-0949-473f-8849-bb5648c6cba1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:35:25.715352 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:25.715216 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50b59bdf-0949-473f-8849-bb5648c6cba1-istio-token" pod "50b59bdf-0949-473f-8849-bb5648c6cba1" (UID: "50b59bdf-0949-473f-8849-bb5648c6cba1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:35:25.715352 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:25.715315 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50b59bdf-0949-473f-8849-bb5648c6cba1-istio-envoy" pod "50b59bdf-0949-473f-8849-bb5648c6cba1" (UID: "50b59bdf-0949-473f-8849-bb5648c6cba1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:35:25.715504 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:25.715449 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50b59bdf-0949-473f-8849-bb5648c6cba1-kube-api-access-gqzq2" pod "50b59bdf-0949-473f-8849-bb5648c6cba1" (UID: "50b59bdf-0949-473f-8849-bb5648c6cba1"). InnerVolumeSpecName "kube-api-access-gqzq2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:35:25.760763 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:25.755625 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a78aa3c2-hk7dm"] Nov 03 01:35:25.760763 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:25.756944 4421 cpu_manager.go:414] "containerMap: removing container" podUID="50b59bdf-0949-473f-8849-bb5648c6cba1" containerName="istio-proxy" Nov 03 01:35:25.760763 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:25.756968 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50b59bdf-0949-473f-8849-bb5648c6cba1" containerName="istio-proxy" Nov 03 01:35:25.760763 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:25.757063 4421 cpu_manager.go:414] "containerMap: removing container" podUID="03015bfc-aaac-4f5a-b909-790a18fc61f3" containerName="revoke-keys" Nov 03 01:35:25.760763 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:25.757073 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03015bfc-aaac-4f5a-b909-790a18fc61f3" containerName="revoke-keys" Nov 03 01:35:25.760763 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:25.757313 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="03015bfc-aaac-4f5a-b909-790a18fc61f3" containerName="revoke-keys" Nov 03 01:35:25.760763 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:25.757337 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="50b59bdf-0949-473f-8849-bb5648c6cba1" containerName="istio-proxy" Nov 03 01:35:25.760763 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:25.757350 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="50b59bdf-0949-473f-8849-bb5648c6cba1" containerName="istio-proxy" Nov 03 01:35:25.763641 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:25.763614 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a78aa3c2-hk7dm"] Nov 03 01:35:25.763779 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:25.763736 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a78aa3c2-hk7dm" Nov 03 01:35:25.767591 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:25.767562 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:35:25.767774 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:25.767643 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6q4xp\"" Nov 03 01:35:25.816377 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:25.815422 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad30bab7-985e-4174-b75b-26d590a6acfe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a78aa3c2-hk7dm\" (UID: \"ad30bab7-985e-4174-b75b-26d590a6acfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a78aa3c2-hk7dm" Nov 03 01:35:25.816377 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:25.815470 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad30bab7-985e-4174-b75b-26d590a6acfe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a78aa3c2-hk7dm\" (UID: \"ad30bab7-985e-4174-b75b-26d590a6acfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a78aa3c2-hk7dm" Nov 03 01:35:25.816377 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:25.815826 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wh6fh\" (UniqueName: \"kubernetes.io/projected/ad30bab7-985e-4174-b75b-26d590a6acfe-kube-api-access-wh6fh\") pod \"maas-api-revoke-sub-a78aa3c2-hk7dm\" (UID: \"ad30bab7-985e-4174-b75b-26d590a6acfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a78aa3c2-hk7dm" Nov 03 01:35:25.816377 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:25.815920 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqzq2\" (UniqueName: \"kubernetes.io/projected/50b59bdf-0949-473f-8849-bb5648c6cba1-kube-api-access-gqzq2\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:35:25.816377 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:25.815937 4421 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50b59bdf-0949-473f-8849-bb5648c6cba1-istio-envoy\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:35:25.816377 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:25.815952 4421 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50b59bdf-0949-473f-8849-bb5648c6cba1-istio-podinfo\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:35:25.816377 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:25.815966 4421 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50b59bdf-0949-473f-8849-bb5648c6cba1-istio-token\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:35:25.917978 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:25.917936 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wh6fh\" (UniqueName: \"kubernetes.io/projected/ad30bab7-985e-4174-b75b-26d590a6acfe-kube-api-access-wh6fh\") pod \"maas-api-revoke-sub-a78aa3c2-hk7dm\" (UID: \"ad30bab7-985e-4174-b75b-26d590a6acfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a78aa3c2-hk7dm" Nov 03 01:35:25.918198 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:25.918085 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad30bab7-985e-4174-b75b-26d590a6acfe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a78aa3c2-hk7dm\" (UID: \"ad30bab7-985e-4174-b75b-26d590a6acfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a78aa3c2-hk7dm" Nov 03 01:35:25.918198 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:25.918116 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad30bab7-985e-4174-b75b-26d590a6acfe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a78aa3c2-hk7dm\" (UID: \"ad30bab7-985e-4174-b75b-26d590a6acfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a78aa3c2-hk7dm" Nov 03 01:35:25.919201 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:25.919172 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad30bab7-985e-4174-b75b-26d590a6acfe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a78aa3c2-hk7dm\" (UID: \"ad30bab7-985e-4174-b75b-26d590a6acfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a78aa3c2-hk7dm" Nov 03 01:35:25.921665 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:25.921643 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad30bab7-985e-4174-b75b-26d590a6acfe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a78aa3c2-hk7dm\" (UID: \"ad30bab7-985e-4174-b75b-26d590a6acfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a78aa3c2-hk7dm" Nov 03 01:35:25.927288 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:25.927256 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wh6fh\" (UniqueName: \"kubernetes.io/projected/ad30bab7-985e-4174-b75b-26d590a6acfe-kube-api-access-wh6fh\") pod \"maas-api-revoke-sub-a78aa3c2-hk7dm\" (UID: \"ad30bab7-985e-4174-b75b-26d590a6acfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a78aa3c2-hk7dm" Nov 03 01:35:25.952788 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:25.952754 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-b6afd6-openshift-default-78d46d7dcc-dg79z"] Nov 03 01:35:25.959804 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:25.959774 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-b6afd6-openshift-default-78d46d7dcc-dg79z"] Nov 03 01:35:25.984552 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:25.984520 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50b59bdf-0949-473f-8849-bb5648c6cba1" path="/var/lib/kubelet/pods/50b59bdf-0949-473f-8849-bb5648c6cba1/volumes" Nov 03 01:35:26.084878 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:26.084790 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a78aa3c2-hk7dm" Nov 03 01:35:26.504302 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:26.489892 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a78aa3c2-hk7dm"] Nov 03 01:35:26.511340 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:26.508176 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b6afd6-7db7d566cf-rkxmr" podUID="18de1ce4-36e1-4a3c-90cc-ecbac5dcfc80" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:35:26.635599 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:26.635563 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a78aa3c2-hk7dm" event={"ID":"ad30bab7-985e-4174-b75b-26d590a6acfe","Type":"ContainerStarted","Data":"6cba4b828413d60e01979a9af1b4af1ebda6838dc74e5477861ed4cb85b280cc"} Nov 03 01:35:27.645155 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:27.645117 4421 generic.go:356] "Generic (PLEG): container finished" podID="ad30bab7-985e-4174-b75b-26d590a6acfe" containerID="d0a44dc039c752dabfdfb9cab8ebcd92a27ea27138a85f86a13d61695b8dff4b" exitCode=0 Nov 03 01:35:27.645608 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:27.645199 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a78aa3c2-hk7dm" event={"ID":"ad30bab7-985e-4174-b75b-26d590a6acfe","Type":"ContainerDied","Data":"d0a44dc039c752dabfdfb9cab8ebcd92a27ea27138a85f86a13d61695b8dff4b"} Nov 03 01:35:28.824393 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:28.824368 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a78aa3c2-hk7dm" Nov 03 01:35:28.876623 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:28.876578 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad30bab7-985e-4174-b75b-26d590a6acfe-kube-api-access-wh6fh\" (UniqueName: \"kubernetes.io/projected/ad30bab7-985e-4174-b75b-26d590a6acfe-kube-api-access-wh6fh\") pod \"ad30bab7-985e-4174-b75b-26d590a6acfe\" (UID: \"ad30bab7-985e-4174-b75b-26d590a6acfe\") " Nov 03 01:35:28.876833 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:28.876654 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad30bab7-985e-4174-b75b-26d590a6acfe-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad30bab7-985e-4174-b75b-26d590a6acfe-maas-api-xks-ca\") pod \"ad30bab7-985e-4174-b75b-26d590a6acfe\" (UID: \"ad30bab7-985e-4174-b75b-26d590a6acfe\") " Nov 03 01:35:28.876833 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:28.876766 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad30bab7-985e-4174-b75b-26d590a6acfe-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad30bab7-985e-4174-b75b-26d590a6acfe-maas-api-openshift-ca\") pod \"ad30bab7-985e-4174-b75b-26d590a6acfe\" (UID: \"ad30bab7-985e-4174-b75b-26d590a6acfe\") " Nov 03 01:35:28.877208 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:28.877176 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad30bab7-985e-4174-b75b-26d590a6acfe-maas-api-openshift-ca" pod "ad30bab7-985e-4174-b75b-26d590a6acfe" (UID: "ad30bab7-985e-4174-b75b-26d590a6acfe"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:35:28.877430 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:28.877404 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad30bab7-985e-4174-b75b-26d590a6acfe-maas-api-openshift-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:35:28.879512 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:28.879486 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad30bab7-985e-4174-b75b-26d590a6acfe-kube-api-access-wh6fh" pod "ad30bab7-985e-4174-b75b-26d590a6acfe" (UID: "ad30bab7-985e-4174-b75b-26d590a6acfe"). InnerVolumeSpecName "kube-api-access-wh6fh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:35:28.879612 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:28.879477 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad30bab7-985e-4174-b75b-26d590a6acfe-maas-api-xks-ca" pod "ad30bab7-985e-4174-b75b-26d590a6acfe" (UID: "ad30bab7-985e-4174-b75b-26d590a6acfe"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:35:28.979319 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:28.979216 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wh6fh\" (UniqueName: \"kubernetes.io/projected/ad30bab7-985e-4174-b75b-26d590a6acfe-kube-api-access-wh6fh\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:35:28.979319 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:28.979253 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad30bab7-985e-4174-b75b-26d590a6acfe-maas-api-xks-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:35:29.661150 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:29.661113 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a78aa3c2-hk7dm" Nov 03 01:35:29.661150 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:29.661146 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a78aa3c2-hk7dm" event={"ID":"ad30bab7-985e-4174-b75b-26d590a6acfe","Type":"ContainerDied","Data":"6cba4b828413d60e01979a9af1b4af1ebda6838dc74e5477861ed4cb85b280cc"} Nov 03 01:35:29.661426 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:29.661203 4421 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6cba4b828413d60e01979a9af1b4af1ebda6838dc74e5477861ed4cb85b280cc" Nov 03 01:35:30.291372 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:30.291342 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b6afd6-7db7d566cf-rkxmr" Nov 03 01:35:30.411160 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:30.411126 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18de1ce4-36e1-4a3c-90cc-ecbac5dcfc80-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18de1ce4-36e1-4a3c-90cc-ecbac5dcfc80-plugins-config-volume\") pod \"18de1ce4-36e1-4a3c-90cc-ecbac5dcfc80\" (UID: \"18de1ce4-36e1-4a3c-90cc-ecbac5dcfc80\") " Nov 03 01:35:30.411362 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:30.411311 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18de1ce4-36e1-4a3c-90cc-ecbac5dcfc80-kube-api-access-wlmlw\" (UniqueName: \"kubernetes.io/projected/18de1ce4-36e1-4a3c-90cc-ecbac5dcfc80-kube-api-access-wlmlw\") pod \"18de1ce4-36e1-4a3c-90cc-ecbac5dcfc80\" (UID: \"18de1ce4-36e1-4a3c-90cc-ecbac5dcfc80\") " Nov 03 01:35:30.411539 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:30.411517 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18de1ce4-36e1-4a3c-90cc-ecbac5dcfc80-plugins-config-volume" pod "18de1ce4-36e1-4a3c-90cc-ecbac5dcfc80" (UID: "18de1ce4-36e1-4a3c-90cc-ecbac5dcfc80"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:35:30.411970 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:30.411951 4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18de1ce4-36e1-4a3c-90cc-ecbac5dcfc80-plugins-config-volume\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:35:30.413638 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:30.413613 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18de1ce4-36e1-4a3c-90cc-ecbac5dcfc80-kube-api-access-wlmlw" pod "18de1ce4-36e1-4a3c-90cc-ecbac5dcfc80" (UID: "18de1ce4-36e1-4a3c-90cc-ecbac5dcfc80"). InnerVolumeSpecName "kube-api-access-wlmlw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:35:30.513790 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:30.513741 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wlmlw\" (UniqueName: \"kubernetes.io/projected/18de1ce4-36e1-4a3c-90cc-ecbac5dcfc80-kube-api-access-wlmlw\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:35:30.672583 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:30.672544 4421 generic.go:356] "Generic (PLEG): container finished" podID="18de1ce4-36e1-4a3c-90cc-ecbac5dcfc80" containerID="0b370d615a26cca3f7c32796de54eb12c2a3fb3fee464e67cc8c0c5acb0379a8" exitCode=0 Nov 03 01:35:30.672816 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:30.672592 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b6afd6-7db7d566cf-rkxmr" Nov 03 01:35:30.672816 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:30.672615 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b6afd6-7db7d566cf-rkxmr" event={"ID":"18de1ce4-36e1-4a3c-90cc-ecbac5dcfc80","Type":"ContainerDied","Data":"0b370d615a26cca3f7c32796de54eb12c2a3fb3fee464e67cc8c0c5acb0379a8"} Nov 03 01:35:30.672816 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:30.672673 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b6afd6-7db7d566cf-rkxmr" event={"ID":"18de1ce4-36e1-4a3c-90cc-ecbac5dcfc80","Type":"ContainerDied","Data":"8d6251233781c5d4701a829135afab2c6ca67341b7bd3e30d11ea3aa253a8f44"} Nov 03 01:35:30.672816 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:30.672702 4421 scope.go:122] "RemoveContainer" containerID="0b370d615a26cca3f7c32796de54eb12c2a3fb3fee464e67cc8c0c5acb0379a8" Nov 03 01:35:30.686043 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:30.686021 4421 scope.go:122] "RemoveContainer" containerID="0b370d615a26cca3f7c32796de54eb12c2a3fb3fee464e67cc8c0c5acb0379a8" Nov 03 01:35:30.686366 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:35:30.686340 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b370d615a26cca3f7c32796de54eb12c2a3fb3fee464e67cc8c0c5acb0379a8\": container with ID starting with 0b370d615a26cca3f7c32796de54eb12c2a3fb3fee464e67cc8c0c5acb0379a8 not found: ID does not exist" containerID="0b370d615a26cca3f7c32796de54eb12c2a3fb3fee464e67cc8c0c5acb0379a8" Nov 03 01:35:30.686442 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:30.686376 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b370d615a26cca3f7c32796de54eb12c2a3fb3fee464e67cc8c0c5acb0379a8"} err="failed to get container status \"0b370d615a26cca3f7c32796de54eb12c2a3fb3fee464e67cc8c0c5acb0379a8\": rpc error: code = NotFound desc = could not find container \"0b370d615a26cca3f7c32796de54eb12c2a3fb3fee464e67cc8c0c5acb0379a8\": container with ID starting with 0b370d615a26cca3f7c32796de54eb12c2a3fb3fee464e67cc8c0c5acb0379a8 not found: ID does not exist" Nov 03 01:35:30.699761 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:30.699722 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-b6afd6-7db7d566cf-rkxmr"] Nov 03 01:35:30.703986 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:30.703950 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-b6afd6-7db7d566cf-rkxmr"] Nov 03 01:35:31.982644 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:31.982596 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18de1ce4-36e1-4a3c-90cc-ecbac5dcfc80" path="/var/lib/kubelet/pods/18de1ce4-36e1-4a3c-90cc-ecbac5dcfc80/volumes" Nov 03 01:35:44.799623 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:35:44.799540 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 01:35:45.502195 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:45.502151 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-737bf2d8-66d45589cc-n7m7l"] Nov 03 01:35:45.513240 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:45.513212 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-737bf2d8-6ff985b587-vb7qc"] Nov 03 01:35:50.508328 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:50.503671 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-737bf2d8-66d45589cc-n7m7l" podUID="e6eaa842-3947-496e-8b82-01a51ef80298" containerName="payload-processing" containerID="cri-o://aa32a6b6804d03bf0d9e829eff973868a0841f0c54101f262a2dc5b6b1e1ec2e" gracePeriod=25 Nov 03 01:35:50.518594 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:50.517251 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-737bf2d8-6ff985b587-vb7qc" podUID="ad4cba32-8044-4971-8e5d-bd5aa09cfef9" containerName="payload-pre-processing" containerID="cri-o://cf0dfa4f7564237d56b92c3f5300f6463d63244794e2f19129826a2812b90332" gracePeriod=25 Nov 03 01:35:50.868380 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:50.868349 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-737bf2d8-66d45589cc-n7m7l" Nov 03 01:35:50.903574 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:50.903545 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-737bf2d8-6ff985b587-vb7qc" Nov 03 01:35:50.909903 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:50.909876 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6eaa842-3947-496e-8b82-01a51ef80298-kube-api-access-txx8g\" (UniqueName: \"kubernetes.io/projected/e6eaa842-3947-496e-8b82-01a51ef80298-kube-api-access-txx8g\") pod \"e6eaa842-3947-496e-8b82-01a51ef80298\" (UID: \"e6eaa842-3947-496e-8b82-01a51ef80298\") " Nov 03 01:35:50.910366 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:50.910340 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6eaa842-3947-496e-8b82-01a51ef80298-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6eaa842-3947-496e-8b82-01a51ef80298-plugins-config-volume\") pod \"e6eaa842-3947-496e-8b82-01a51ef80298\" (UID: \"e6eaa842-3947-496e-8b82-01a51ef80298\") " Nov 03 01:35:50.911586 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:50.911559 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6eaa842-3947-496e-8b82-01a51ef80298-plugins-config-volume" pod "e6eaa842-3947-496e-8b82-01a51ef80298" (UID: "e6eaa842-3947-496e-8b82-01a51ef80298"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:35:50.916786 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:50.916751 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6eaa842-3947-496e-8b82-01a51ef80298-kube-api-access-txx8g" pod "e6eaa842-3947-496e-8b82-01a51ef80298" (UID: "e6eaa842-3947-496e-8b82-01a51ef80298"). InnerVolumeSpecName "kube-api-access-txx8g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:35:50.931207 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:50.931174 4421 generic.go:356] "Generic (PLEG): container finished" podID="ad4cba32-8044-4971-8e5d-bd5aa09cfef9" containerID="cf0dfa4f7564237d56b92c3f5300f6463d63244794e2f19129826a2812b90332" exitCode=0 Nov 03 01:35:50.931483 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:50.931191 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-737bf2d8-6ff985b587-vb7qc" Nov 03 01:35:50.936793 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:50.936753 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-737bf2d8-6ff985b587-vb7qc" event={"ID":"ad4cba32-8044-4971-8e5d-bd5aa09cfef9","Type":"ContainerDied","Data":"cf0dfa4f7564237d56b92c3f5300f6463d63244794e2f19129826a2812b90332"} Nov 03 01:35:50.937153 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:50.936836 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-737bf2d8-6ff985b587-vb7qc" event={"ID":"ad4cba32-8044-4971-8e5d-bd5aa09cfef9","Type":"ContainerDied","Data":"b89b1c70224f69aaf35868afb78e3f1cb8fb78d30c0cc425c1f6acecadec7ad2"} Nov 03 01:35:50.937153 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:50.936872 4421 scope.go:122] "RemoveContainer" containerID="cf0dfa4f7564237d56b92c3f5300f6463d63244794e2f19129826a2812b90332" Nov 03 01:35:50.940380 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:50.940348 4421 generic.go:356] "Generic (PLEG): container finished" podID="e6eaa842-3947-496e-8b82-01a51ef80298" containerID="aa32a6b6804d03bf0d9e829eff973868a0841f0c54101f262a2dc5b6b1e1ec2e" exitCode=0 Nov 03 01:35:50.940521 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:50.940394 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-737bf2d8-66d45589cc-n7m7l" Nov 03 01:35:50.941017 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:50.940473 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-737bf2d8-66d45589cc-n7m7l" event={"ID":"e6eaa842-3947-496e-8b82-01a51ef80298","Type":"ContainerDied","Data":"aa32a6b6804d03bf0d9e829eff973868a0841f0c54101f262a2dc5b6b1e1ec2e"} Nov 03 01:35:50.941017 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:50.940953 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-737bf2d8-66d45589cc-n7m7l" event={"ID":"e6eaa842-3947-496e-8b82-01a51ef80298","Type":"ContainerDied","Data":"13e6c44acb61c4c3368cc80dc97d591cd9bbc9ae29e2b69a94ac3fe9d670f7e0"} Nov 03 01:35:50.955196 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:50.955177 4421 scope.go:122] "RemoveContainer" containerID="cf0dfa4f7564237d56b92c3f5300f6463d63244794e2f19129826a2812b90332" Nov 03 01:35:50.955579 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:35:50.955554 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf0dfa4f7564237d56b92c3f5300f6463d63244794e2f19129826a2812b90332\": container with ID starting with cf0dfa4f7564237d56b92c3f5300f6463d63244794e2f19129826a2812b90332 not found: ID does not exist" containerID="cf0dfa4f7564237d56b92c3f5300f6463d63244794e2f19129826a2812b90332" Nov 03 01:35:50.955694 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:50.955594 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf0dfa4f7564237d56b92c3f5300f6463d63244794e2f19129826a2812b90332"} err="failed to get container status \"cf0dfa4f7564237d56b92c3f5300f6463d63244794e2f19129826a2812b90332\": rpc error: code = NotFound desc = could not find container \"cf0dfa4f7564237d56b92c3f5300f6463d63244794e2f19129826a2812b90332\": container with ID starting with cf0dfa4f7564237d56b92c3f5300f6463d63244794e2f19129826a2812b90332 not found: ID does not exist" Nov 03 01:35:50.955694 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:50.955620 4421 scope.go:122] "RemoveContainer" containerID="aa32a6b6804d03bf0d9e829eff973868a0841f0c54101f262a2dc5b6b1e1ec2e" Nov 03 01:35:50.972405 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:50.972371 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-737bf2d8-66d45589cc-n7m7l"] Nov 03 01:35:50.979726 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:50.979691 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-737bf2d8-66d45589cc-n7m7l"] Nov 03 01:35:50.988641 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:50.988450 4421 scope.go:122] "RemoveContainer" containerID="aa32a6b6804d03bf0d9e829eff973868a0841f0c54101f262a2dc5b6b1e1ec2e" Nov 03 01:35:50.990494 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:35:50.990452 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa32a6b6804d03bf0d9e829eff973868a0841f0c54101f262a2dc5b6b1e1ec2e\": container with ID starting with aa32a6b6804d03bf0d9e829eff973868a0841f0c54101f262a2dc5b6b1e1ec2e not found: ID does not exist" containerID="aa32a6b6804d03bf0d9e829eff973868a0841f0c54101f262a2dc5b6b1e1ec2e" Nov 03 01:35:50.990641 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:50.990502 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa32a6b6804d03bf0d9e829eff973868a0841f0c54101f262a2dc5b6b1e1ec2e"} err="failed to get container status \"aa32a6b6804d03bf0d9e829eff973868a0841f0c54101f262a2dc5b6b1e1ec2e\": rpc error: code = NotFound desc = could not find container \"aa32a6b6804d03bf0d9e829eff973868a0841f0c54101f262a2dc5b6b1e1ec2e\": container with ID starting with aa32a6b6804d03bf0d9e829eff973868a0841f0c54101f262a2dc5b6b1e1ec2e not found: ID does not exist" Nov 03 01:35:51.012127 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:51.012096 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad4cba32-8044-4971-8e5d-bd5aa09cfef9-kube-api-access-z7wx6\" (UniqueName: \"kubernetes.io/projected/ad4cba32-8044-4971-8e5d-bd5aa09cfef9-kube-api-access-z7wx6\") pod \"ad4cba32-8044-4971-8e5d-bd5aa09cfef9\" (UID: \"ad4cba32-8044-4971-8e5d-bd5aa09cfef9\") " Nov 03 01:35:51.012313 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:51.012223 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad4cba32-8044-4971-8e5d-bd5aa09cfef9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad4cba32-8044-4971-8e5d-bd5aa09cfef9-plugins-config-volume\") pod \"ad4cba32-8044-4971-8e5d-bd5aa09cfef9\" (UID: \"ad4cba32-8044-4971-8e5d-bd5aa09cfef9\") " Nov 03 01:35:51.012673 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:51.012642 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad4cba32-8044-4971-8e5d-bd5aa09cfef9-plugins-config-volume" pod "ad4cba32-8044-4971-8e5d-bd5aa09cfef9" (UID: "ad4cba32-8044-4971-8e5d-bd5aa09cfef9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:35:51.012941 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:51.012925 4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad4cba32-8044-4971-8e5d-bd5aa09cfef9-plugins-config-volume\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:35:51.013015 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:51.012943 4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6eaa842-3947-496e-8b82-01a51ef80298-plugins-config-volume\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:35:51.013015 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:51.012953 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-txx8g\" (UniqueName: \"kubernetes.io/projected/e6eaa842-3947-496e-8b82-01a51ef80298-kube-api-access-txx8g\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:35:51.016835 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:51.016808 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad4cba32-8044-4971-8e5d-bd5aa09cfef9-kube-api-access-z7wx6" pod "ad4cba32-8044-4971-8e5d-bd5aa09cfef9" (UID: "ad4cba32-8044-4971-8e5d-bd5aa09cfef9"). InnerVolumeSpecName "kube-api-access-z7wx6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:35:51.114612 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:51.114508 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7wx6\" (UniqueName: \"kubernetes.io/projected/ad4cba32-8044-4971-8e5d-bd5aa09cfef9-kube-api-access-z7wx6\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:35:51.298987 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:51.298952 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-737bf2d8-6ff985b587-vb7qc"] Nov 03 01:35:51.303098 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:51.303067 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-737bf2d8-6ff985b587-vb7qc"] Nov 03 01:35:51.980736 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:51.980648 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad4cba32-8044-4971-8e5d-bd5aa09cfef9" path="/var/lib/kubelet/pods/ad4cba32-8044-4971-8e5d-bd5aa09cfef9/volumes" Nov 03 01:35:51.981213 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:51.981197 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6eaa842-3947-496e-8b82-01a51ef80298" path="/var/lib/kubelet/pods/e6eaa842-3947-496e-8b82-01a51ef80298/volumes" Nov 03 01:35:53.411411 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:53.411375 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1230bd0-qx57v"] Nov 03 01:35:53.412204 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:53.412185 4421 cpu_manager.go:414] "containerMap: removing container" podUID="ad30bab7-985e-4174-b75b-26d590a6acfe" containerName="revoke-keys" Nov 03 01:35:53.412257 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:53.412206 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad30bab7-985e-4174-b75b-26d590a6acfe" containerName="revoke-keys" Nov 03 01:35:53.412257 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:53.412225 4421 cpu_manager.go:414] "containerMap: removing container" podUID="ad4cba32-8044-4971-8e5d-bd5aa09cfef9" containerName="payload-pre-processing" Nov 03 01:35:53.412257 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:53.412231 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad4cba32-8044-4971-8e5d-bd5aa09cfef9" containerName="payload-pre-processing" Nov 03 01:35:53.412401 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:53.412265 4421 cpu_manager.go:414] "containerMap: removing container" podUID="50b59bdf-0949-473f-8849-bb5648c6cba1" containerName="istio-proxy" Nov 03 01:35:53.412401 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:53.412284 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50b59bdf-0949-473f-8849-bb5648c6cba1" containerName="istio-proxy" Nov 03 01:35:53.412401 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:53.412291 4421 cpu_manager.go:414] "containerMap: removing container" podUID="e6eaa842-3947-496e-8b82-01a51ef80298" containerName="payload-processing" Nov 03 01:35:53.412401 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:53.412297 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6eaa842-3947-496e-8b82-01a51ef80298" containerName="payload-processing" Nov 03 01:35:53.412401 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:53.412305 4421 cpu_manager.go:414] "containerMap: removing container" podUID="18de1ce4-36e1-4a3c-90cc-ecbac5dcfc80" containerName="payload-pre-processing" Nov 03 01:35:53.412401 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:53.412310 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18de1ce4-36e1-4a3c-90cc-ecbac5dcfc80" containerName="payload-pre-processing" Nov 03 01:35:53.412699 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:53.412453 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6eaa842-3947-496e-8b82-01a51ef80298" containerName="payload-processing" Nov 03 01:35:53.412699 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:53.412479 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="18de1ce4-36e1-4a3c-90cc-ecbac5dcfc80" containerName="payload-pre-processing" Nov 03 01:35:53.412699 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:53.412499 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad30bab7-985e-4174-b75b-26d590a6acfe" containerName="revoke-keys" Nov 03 01:35:53.412699 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:53.412514 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad4cba32-8044-4971-8e5d-bd5aa09cfef9" containerName="payload-pre-processing" Nov 03 01:35:53.418237 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:53.418207 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1230bd0-qx57v" Nov 03 01:35:53.421737 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:53.421710 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:35:53.422874 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:53.422845 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6q4xp\"" Nov 03 01:35:53.423571 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:53.423544 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1230bd0-qx57v"] Nov 03 01:35:53.481318 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:53.472774 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/845cc619-fae7-4de0-a6a9-33d2e4f6ce87-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f1230bd0-qx57v\" (UID: \"845cc619-fae7-4de0-a6a9-33d2e4f6ce87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1230bd0-qx57v" Nov 03 01:35:53.481318 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:53.472861 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nfjv\" (UniqueName: \"kubernetes.io/projected/845cc619-fae7-4de0-a6a9-33d2e4f6ce87-kube-api-access-9nfjv\") pod \"maas-api-revoke-sub-f1230bd0-qx57v\" (UID: \"845cc619-fae7-4de0-a6a9-33d2e4f6ce87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1230bd0-qx57v" Nov 03 01:35:53.481318 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:53.473125 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/845cc619-fae7-4de0-a6a9-33d2e4f6ce87-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f1230bd0-qx57v\" (UID: \"845cc619-fae7-4de0-a6a9-33d2e4f6ce87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1230bd0-qx57v" Nov 03 01:35:53.574653 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:53.574620 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/845cc619-fae7-4de0-a6a9-33d2e4f6ce87-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f1230bd0-qx57v\" (UID: \"845cc619-fae7-4de0-a6a9-33d2e4f6ce87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1230bd0-qx57v" Nov 03 01:35:53.574842 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:53.574673 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nfjv\" (UniqueName: \"kubernetes.io/projected/845cc619-fae7-4de0-a6a9-33d2e4f6ce87-kube-api-access-9nfjv\") pod \"maas-api-revoke-sub-f1230bd0-qx57v\" (UID: \"845cc619-fae7-4de0-a6a9-33d2e4f6ce87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1230bd0-qx57v" Nov 03 01:35:53.574842 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:53.574740 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/845cc619-fae7-4de0-a6a9-33d2e4f6ce87-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f1230bd0-qx57v\" (UID: \"845cc619-fae7-4de0-a6a9-33d2e4f6ce87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1230bd0-qx57v" Nov 03 01:35:53.575306 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:53.575251 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/845cc619-fae7-4de0-a6a9-33d2e4f6ce87-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f1230bd0-qx57v\" (UID: \"845cc619-fae7-4de0-a6a9-33d2e4f6ce87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1230bd0-qx57v" Nov 03 01:35:53.586533 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:53.586503 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/845cc619-fae7-4de0-a6a9-33d2e4f6ce87-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f1230bd0-qx57v\" (UID: \"845cc619-fae7-4de0-a6a9-33d2e4f6ce87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1230bd0-qx57v" Nov 03 01:35:53.587082 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:53.587058 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nfjv\" (UniqueName: \"kubernetes.io/projected/845cc619-fae7-4de0-a6a9-33d2e4f6ce87-kube-api-access-9nfjv\") pod \"maas-api-revoke-sub-f1230bd0-qx57v\" (UID: \"845cc619-fae7-4de0-a6a9-33d2e4f6ce87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1230bd0-qx57v" Nov 03 01:35:53.737951 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:53.737876 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1230bd0-qx57v" Nov 03 01:35:54.025593 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:35:54.025564 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod845cc619_fae7_4de0_a6a9_33d2e4f6ce87.slice/crio-7608b81161f151bf3abdad17de252e60ba487116230f7f9462295eaf3533b3fe WatchSource:0}: Error finding container 7608b81161f151bf3abdad17de252e60ba487116230f7f9462295eaf3533b3fe: Status 404 returned error can't find the container with id 7608b81161f151bf3abdad17de252e60ba487116230f7f9462295eaf3533b3fe Nov 03 01:35:54.025720 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:54.025685 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1230bd0-qx57v"] Nov 03 01:35:54.976015 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:54.975965 4421 generic.go:356] "Generic (PLEG): container finished" podID="845cc619-fae7-4de0-a6a9-33d2e4f6ce87" containerID="65214d143721edd3fb04d3c1fb1fc69ba46c5937217121e32162451088bf159a" exitCode=0 Nov 03 01:35:54.976517 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:54.976045 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1230bd0-qx57v" event={"ID":"845cc619-fae7-4de0-a6a9-33d2e4f6ce87","Type":"ContainerDied","Data":"65214d143721edd3fb04d3c1fb1fc69ba46c5937217121e32162451088bf159a"} Nov 03 01:35:54.976517 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:54.976101 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1230bd0-qx57v" event={"ID":"845cc619-fae7-4de0-a6a9-33d2e4f6ce87","Type":"ContainerStarted","Data":"7608b81161f151bf3abdad17de252e60ba487116230f7f9462295eaf3533b3fe"} Nov 03 01:35:56.213298 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:56.209214 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1230bd0-qx57v" Nov 03 01:35:56.333298 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:56.326899 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/845cc619-fae7-4de0-a6a9-33d2e4f6ce87-kube-api-access-9nfjv\" (UniqueName: \"kubernetes.io/projected/845cc619-fae7-4de0-a6a9-33d2e4f6ce87-kube-api-access-9nfjv\") pod \"845cc619-fae7-4de0-a6a9-33d2e4f6ce87\" (UID: \"845cc619-fae7-4de0-a6a9-33d2e4f6ce87\") " Nov 03 01:35:56.333298 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:56.327056 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/845cc619-fae7-4de0-a6a9-33d2e4f6ce87-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/845cc619-fae7-4de0-a6a9-33d2e4f6ce87-maas-api-openshift-ca\") pod \"845cc619-fae7-4de0-a6a9-33d2e4f6ce87\" (UID: \"845cc619-fae7-4de0-a6a9-33d2e4f6ce87\") " Nov 03 01:35:56.333298 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:56.327217 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/845cc619-fae7-4de0-a6a9-33d2e4f6ce87-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/845cc619-fae7-4de0-a6a9-33d2e4f6ce87-maas-api-xks-ca\") pod \"845cc619-fae7-4de0-a6a9-33d2e4f6ce87\" (UID: \"845cc619-fae7-4de0-a6a9-33d2e4f6ce87\") " Nov 03 01:35:56.333298 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:56.333159 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/845cc619-fae7-4de0-a6a9-33d2e4f6ce87-maas-api-openshift-ca" pod "845cc619-fae7-4de0-a6a9-33d2e4f6ce87" (UID: "845cc619-fae7-4de0-a6a9-33d2e4f6ce87"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:35:56.392299 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:56.385553 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/845cc619-fae7-4de0-a6a9-33d2e4f6ce87-kube-api-access-9nfjv" pod "845cc619-fae7-4de0-a6a9-33d2e4f6ce87" (UID: "845cc619-fae7-4de0-a6a9-33d2e4f6ce87"). InnerVolumeSpecName "kube-api-access-9nfjv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:35:56.442298 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:56.424435 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/845cc619-fae7-4de0-a6a9-33d2e4f6ce87-maas-api-xks-ca" pod "845cc619-fae7-4de0-a6a9-33d2e4f6ce87" (UID: "845cc619-fae7-4de0-a6a9-33d2e4f6ce87"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:35:56.442298 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:56.435882 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9nfjv\" (UniqueName: \"kubernetes.io/projected/845cc619-fae7-4de0-a6a9-33d2e4f6ce87-kube-api-access-9nfjv\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:35:56.442298 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:56.435923 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/845cc619-fae7-4de0-a6a9-33d2e4f6ce87-maas-api-openshift-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:35:56.442298 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:56.435941 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/845cc619-fae7-4de0-a6a9-33d2e4f6ce87-maas-api-xks-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:35:56.999460 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:56.999429 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1230bd0-qx57v" Nov 03 01:35:56.999756 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:56.999458 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1230bd0-qx57v" event={"ID":"845cc619-fae7-4de0-a6a9-33d2e4f6ce87","Type":"ContainerDied","Data":"7608b81161f151bf3abdad17de252e60ba487116230f7f9462295eaf3533b3fe"} Nov 03 01:35:56.999918 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:35:56.999797 4421 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7608b81161f151bf3abdad17de252e60ba487116230f7f9462295eaf3533b3fe" Nov 03 01:36:01.952758 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:01.952564 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-9v2m2"] Nov 03 01:36:01.953447 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:01.952786 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-9v2m2" podUID="52a0f195-418d-42be-8222-b030c8f6ba55" containerName="istio-proxy" containerID="cri-o://ac321584e82282af33485c24ee8379fdb77edee07abd643508c5b845fd39082b" gracePeriod=30 Nov 03 01:36:01.959304 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:01.959241 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-b5cbp"] Nov 03 01:36:01.959509 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:01.959485 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-b5cbp" podUID="b553f012-3d8c-4a9a-b514-410db05c2f0a" containerName="istio-proxy" containerID="cri-o://9e37991ee0afd467692f5ef93bf3b08d01e9305ee77387dcf1874c6cedb5f047" gracePeriod=30 Nov 03 01:36:01.966388 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:01.966359 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-vhvf6"] Nov 03 01:36:01.966678 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:01.966626 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-vhvf6" podUID="4246b3f1-fc55-4059-9983-06dbeca10768" containerName="istio-proxy" containerID="cri-o://1aa0aa1962028996bc494bc47f39e170cc90bc0396bbaef258ad31ab6fc6fb3f" gracePeriod=30 Nov 03 01:36:05.716187 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:05.716148 4421 scope.go:122] "RemoveContainer" containerID="876417d98447d80450b3399706caca23ada482a1d659849487ad6e6ec51d88a0" Nov 03 01:36:06.033559 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:06.033519 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-5dab6a-6c4b59c89-txkqj"] Nov 03 01:36:06.599315 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:06.599235 4421 patch_prober.go:29] interesting pod/e2e-mt-737bf2d8-openshift-default-56dc96b957-b5cbp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 01:36:06.599525 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:06.599338 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-b5cbp" podUID="b553f012-3d8c-4a9a-b514-410db05c2f0a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 01:36:06.726949 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:06.726910 4421 patch_prober.go:29] interesting pod/e2e-mt-737bf2d8-openshift-default-56dc96b957-vhvf6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 01:36:06.727389 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:06.726981 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-vhvf6" podUID="4246b3f1-fc55-4059-9983-06dbeca10768" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 01:36:07.319171 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:07.319124 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-b5cbp" Nov 03 01:36:07.376001 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:07.375969 4421 generic.go:356] "Generic (PLEG): container finished" podID="4246b3f1-fc55-4059-9983-06dbeca10768" containerID="1aa0aa1962028996bc494bc47f39e170cc90bc0396bbaef258ad31ab6fc6fb3f" exitCode=0 Nov 03 01:36:07.376172 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:07.376053 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-vhvf6" event={"ID":"4246b3f1-fc55-4059-9983-06dbeca10768","Type":"ContainerDied","Data":"1aa0aa1962028996bc494bc47f39e170cc90bc0396bbaef258ad31ab6fc6fb3f"} Nov 03 01:36:07.376172 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:07.376096 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-vhvf6" event={"ID":"4246b3f1-fc55-4059-9983-06dbeca10768","Type":"ContainerDied","Data":"fb4c46c3bcd1de600170b4238946aa6ac465d4c63daed44e21c63f8b17778822"} Nov 03 01:36:07.376172 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:07.376119 4421 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fb4c46c3bcd1de600170b4238946aa6ac465d4c63daed44e21c63f8b17778822" Nov 03 01:36:07.376503 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:07.376483 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-vhvf6" Nov 03 01:36:07.378052 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:07.378024 4421 generic.go:356] "Generic (PLEG): container finished" podID="b553f012-3d8c-4a9a-b514-410db05c2f0a" containerID="9e37991ee0afd467692f5ef93bf3b08d01e9305ee77387dcf1874c6cedb5f047" exitCode=0 Nov 03 01:36:07.378266 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:07.378242 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-b5cbp" event={"ID":"b553f012-3d8c-4a9a-b514-410db05c2f0a","Type":"ContainerDied","Data":"9e37991ee0afd467692f5ef93bf3b08d01e9305ee77387dcf1874c6cedb5f047"} Nov 03 01:36:07.378378 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:07.378319 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-b5cbp" event={"ID":"b553f012-3d8c-4a9a-b514-410db05c2f0a","Type":"ContainerDied","Data":"b475864076a988b4cbd349c0b60c8209b70292fa73936ff19a8a8b88f6d16574"} Nov 03 01:36:07.378378 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:07.378355 4421 scope.go:122] "RemoveContainer" containerID="9e37991ee0afd467692f5ef93bf3b08d01e9305ee77387dcf1874c6cedb5f047" Nov 03 01:36:07.378911 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:07.378888 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-b5cbp" Nov 03 01:36:07.384437 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:07.384406 4421 generic.go:356] "Generic (PLEG): container finished" podID="52a0f195-418d-42be-8222-b030c8f6ba55" containerID="ac321584e82282af33485c24ee8379fdb77edee07abd643508c5b845fd39082b" exitCode=0 Nov 03 01:36:07.384599 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:07.384579 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-9v2m2" event={"ID":"52a0f195-418d-42be-8222-b030c8f6ba55","Type":"ContainerDied","Data":"ac321584e82282af33485c24ee8379fdb77edee07abd643508c5b845fd39082b"} Nov 03 01:36:07.384692 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:07.384635 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-9v2m2" event={"ID":"52a0f195-418d-42be-8222-b030c8f6ba55","Type":"ContainerDied","Data":"1b0d1384754a12d1e6502fb7e13d4ead5ec340130f5aeb3c815d5f2873c49f28"} Nov 03 01:36:07.384692 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:07.384662 4421 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1b0d1384754a12d1e6502fb7e13d4ead5ec340130f5aeb3c815d5f2873c49f28" Nov 03 01:36:07.385440 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:07.385413 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-9v2m2" Nov 03 01:36:07.398091 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:07.398064 4421 scope.go:122] "RemoveContainer" containerID="9e37991ee0afd467692f5ef93bf3b08d01e9305ee77387dcf1874c6cedb5f047" Nov 03 01:36:07.399348 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:36:07.399313 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e37991ee0afd467692f5ef93bf3b08d01e9305ee77387dcf1874c6cedb5f047\": container with ID starting with 9e37991ee0afd467692f5ef93bf3b08d01e9305ee77387dcf1874c6cedb5f047 not found: ID does not exist" containerID="9e37991ee0afd467692f5ef93bf3b08d01e9305ee77387dcf1874c6cedb5f047" Nov 03 01:36:07.399498 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:07.399356 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e37991ee0afd467692f5ef93bf3b08d01e9305ee77387dcf1874c6cedb5f047"} err="failed to get container status \"9e37991ee0afd467692f5ef93bf3b08d01e9305ee77387dcf1874c6cedb5f047\": rpc error: code = NotFound desc = could not find container \"9e37991ee0afd467692f5ef93bf3b08d01e9305ee77387dcf1874c6cedb5f047\": container with ID starting with 9e37991ee0afd467692f5ef93bf3b08d01e9305ee77387dcf1874c6cedb5f047 not found: ID does not exist" Nov 03 01:36:07.399498 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:07.399383 4421 scope.go:122] "RemoveContainer" containerID="41749612022ba7f45899b639631594c7f0c9a087212cf7b716fb6eddb81975d3" Nov 03 01:36:07.432592 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:07.432557 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4246b3f1-fc55-4059-9983-06dbeca10768-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4246b3f1-fc55-4059-9983-06dbeca10768-istio-envoy\") pod \"4246b3f1-fc55-4059-9983-06dbeca10768\" (UID: \"4246b3f1-fc55-4059-9983-06dbeca10768\") " Nov 03 01:36:07.432813 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:07.432614 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4246b3f1-fc55-4059-9983-06dbeca10768-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4246b3f1-fc55-4059-9983-06dbeca10768-istio-podinfo\") pod \"4246b3f1-fc55-4059-9983-06dbeca10768\" (UID: \"4246b3f1-fc55-4059-9983-06dbeca10768\") " Nov 03 01:36:07.432813 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:07.432657 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b553f012-3d8c-4a9a-b514-410db05c2f0a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b553f012-3d8c-4a9a-b514-410db05c2f0a-workload-socket\") pod \"b553f012-3d8c-4a9a-b514-410db05c2f0a\" (UID: \"b553f012-3d8c-4a9a-b514-410db05c2f0a\") " Nov 03 01:36:07.432813 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:07.432686 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b553f012-3d8c-4a9a-b514-410db05c2f0a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b553f012-3d8c-4a9a-b514-410db05c2f0a-istio-podinfo\") pod \"b553f012-3d8c-4a9a-b514-410db05c2f0a\" (UID: \"b553f012-3d8c-4a9a-b514-410db05c2f0a\") " Nov 03 01:36:07.432813 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:07.432723 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4246b3f1-fc55-4059-9983-06dbeca10768-istio-token\" (UniqueName: \"kubernetes.io/projected/4246b3f1-fc55-4059-9983-06dbeca10768-istio-token\") pod \"4246b3f1-fc55-4059-9983-06dbeca10768\" (UID: \"4246b3f1-fc55-4059-9983-06dbeca10768\") " Nov 03 01:36:07.432813 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:07.432758 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b553f012-3d8c-4a9a-b514-410db05c2f0a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b553f012-3d8c-4a9a-b514-410db05c2f0a-istio-data\") pod \"b553f012-3d8c-4a9a-b514-410db05c2f0a\" (UID: \"b553f012-3d8c-4a9a-b514-410db05c2f0a\") " Nov 03 01:36:07.434190 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:07.432842 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4246b3f1-fc55-4059-9983-06dbeca10768-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4246b3f1-fc55-4059-9983-06dbeca10768-credential-socket\") pod \"4246b3f1-fc55-4059-9983-06dbeca10768\" (UID: \"4246b3f1-fc55-4059-9983-06dbeca10768\") " Nov 03 01:36:07.434190 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:07.432860 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4246b3f1-fc55-4059-9983-06dbeca10768-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4246b3f1-fc55-4059-9983-06dbeca10768-istio-data\") pod \"4246b3f1-fc55-4059-9983-06dbeca10768\" (UID: \"4246b3f1-fc55-4059-9983-06dbeca10768\") " Nov 03 01:36:07.434190 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:07.432883 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4246b3f1-fc55-4059-9983-06dbeca10768-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4246b3f1-fc55-4059-9983-06dbeca10768-workload-socket\") pod \"4246b3f1-fc55-4059-9983-06dbeca10768\" (UID: \"4246b3f1-fc55-4059-9983-06dbeca10768\") " Nov 03 01:36:07.434190 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:07.432914 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4246b3f1-fc55-4059-9983-06dbeca10768-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4246b3f1-fc55-4059-9983-06dbeca10768-istiod-ca-cert\") pod \"4246b3f1-fc55-4059-9983-06dbeca10768\" (UID: \"4246b3f1-fc55-4059-9983-06dbeca10768\") " Nov 03 01:36:07.434190 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:07.432937 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b553f012-3d8c-4a9a-b514-410db05c2f0a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b553f012-3d8c-4a9a-b514-410db05c2f0a-istiod-ca-cert\") pod \"b553f012-3d8c-4a9a-b514-410db05c2f0a\" (UID: \"b553f012-3d8c-4a9a-b514-410db05c2f0a\") " Nov 03 01:36:07.434190 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:07.432993 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b553f012-3d8c-4a9a-b514-410db05c2f0a-kube-api-access-jh68r\" (UniqueName: \"kubernetes.io/projected/b553f012-3d8c-4a9a-b514-410db05c2f0a-kube-api-access-jh68r\") pod \"b553f012-3d8c-4a9a-b514-410db05c2f0a\" (UID: \"b553f012-3d8c-4a9a-b514-410db05c2f0a\") " Nov 03 01:36:07.434190 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:07.433038 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b553f012-3d8c-4a9a-b514-410db05c2f0a-istio-token\" (UniqueName: \"kubernetes.io/projected/b553f012-3d8c-4a9a-b514-410db05c2f0a-istio-token\") pod \"b553f012-3d8c-4a9a-b514-410db05c2f0a\" (UID: \"b553f012-3d8c-4a9a-b514-410db05c2f0a\") " Nov 03 01:36:07.434190 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:07.433073 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4246b3f1-fc55-4059-9983-06dbeca10768-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4246b3f1-fc55-4059-9983-06dbeca10768-workload-certs\") pod \"4246b3f1-fc55-4059-9983-06dbeca10768\" (UID: \"4246b3f1-fc55-4059-9983-06dbeca10768\") " Nov 03 01:36:07.434190 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:07.433105 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b553f012-3d8c-4a9a-b514-410db05c2f0a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b553f012-3d8c-4a9a-b514-410db05c2f0a-istio-envoy\") pod \"b553f012-3d8c-4a9a-b514-410db05c2f0a\" (UID: \"b553f012-3d8c-4a9a-b514-410db05c2f0a\") " Nov 03 01:36:07.434190 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:07.433127 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b553f012-3d8c-4a9a-b514-410db05c2f0a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b553f012-3d8c-4a9a-b514-410db05c2f0a-workload-certs\") pod \"b553f012-3d8c-4a9a-b514-410db05c2f0a\" (UID: \"b553f012-3d8c-4a9a-b514-410db05c2f0a\") " Nov 03 01:36:07.434190 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:07.433162 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b553f012-3d8c-4a9a-b514-410db05c2f0a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b553f012-3d8c-4a9a-b514-410db05c2f0a-credential-socket\") pod \"b553f012-3d8c-4a9a-b514-410db05c2f0a\" (UID: \"b553f012-3d8c-4a9a-b514-410db05c2f0a\") " Nov 03 01:36:07.434190 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:07.433196 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4246b3f1-fc55-4059-9983-06dbeca10768-kube-api-access-gfw97\" (UniqueName: \"kubernetes.io/projected/4246b3f1-fc55-4059-9983-06dbeca10768-kube-api-access-gfw97\") pod \"4246b3f1-fc55-4059-9983-06dbeca10768\" (UID: \"4246b3f1-fc55-4059-9983-06dbeca10768\") " Nov 03 01:36:07.434883 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:07.434359 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4246b3f1-fc55-4059-9983-06dbeca10768-workload-socket" pod "4246b3f1-fc55-4059-9983-06dbeca10768" (UID: "4246b3f1-fc55-4059-9983-06dbeca10768"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:36:07.435575 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:07.435544 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b553f012-3d8c-4a9a-b514-410db05c2f0a-istiod-ca-cert" pod "b553f012-3d8c-4a9a-b514-410db05c2f0a" (UID: "b553f012-3d8c-4a9a-b514-410db05c2f0a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:36:07.435931 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:07.435905 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4246b3f1-fc55-4059-9983-06dbeca10768-istiod-ca-cert" pod "4246b3f1-fc55-4059-9983-06dbeca10768" (UID: "4246b3f1-fc55-4059-9983-06dbeca10768"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:36:07.437431 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:07.436570 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4246b3f1-fc55-4059-9983-06dbeca10768-workload-certs" pod "4246b3f1-fc55-4059-9983-06dbeca10768" (UID: "4246b3f1-fc55-4059-9983-06dbeca10768"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:36:07.437431 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:07.436639 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b553f012-3d8c-4a9a-b514-410db05c2f0a-workload-certs" pod "b553f012-3d8c-4a9a-b514-410db05c2f0a" (UID: "b553f012-3d8c-4a9a-b514-410db05c2f0a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:36:07.437431 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:07.436774 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b553f012-3d8c-4a9a-b514-410db05c2f0a-credential-socket" pod "b553f012-3d8c-4a9a-b514-410db05c2f0a" (UID: "b553f012-3d8c-4a9a-b514-410db05c2f0a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:36:07.437431 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:07.436829 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4246b3f1-fc55-4059-9983-06dbeca10768-credential-socket" pod "4246b3f1-fc55-4059-9983-06dbeca10768" (UID: "4246b3f1-fc55-4059-9983-06dbeca10768"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:36:07.437431 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:07.437000 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b553f012-3d8c-4a9a-b514-410db05c2f0a-istio-data" pod "b553f012-3d8c-4a9a-b514-410db05c2f0a" (UID: "b553f012-3d8c-4a9a-b514-410db05c2f0a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:36:07.437431 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:07.437050 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b553f012-3d8c-4a9a-b514-410db05c2f0a-workload-socket" pod "b553f012-3d8c-4a9a-b514-410db05c2f0a" (UID: "b553f012-3d8c-4a9a-b514-410db05c2f0a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:36:07.437431 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:07.437190 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4246b3f1-fc55-4059-9983-06dbeca10768-istio-data" pod "4246b3f1-fc55-4059-9983-06dbeca10768" (UID: "4246b3f1-fc55-4059-9983-06dbeca10768"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:36:07.437431 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:07.437229 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4246b3f1-fc55-4059-9983-06dbeca10768-istio-envoy" pod "4246b3f1-fc55-4059-9983-06dbeca10768" (UID: "4246b3f1-fc55-4059-9983-06dbeca10768"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:36:07.437431 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:07.437410 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b553f012-3d8c-4a9a-b514-410db05c2f0a-istio-token" pod "b553f012-3d8c-4a9a-b514-410db05c2f0a" (UID: "b553f012-3d8c-4a9a-b514-410db05c2f0a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:36:07.438018 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:07.437808 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4246b3f1-fc55-4059-9983-06dbeca10768-kube-api-access-gfw97" pod "4246b3f1-fc55-4059-9983-06dbeca10768" (UID: "4246b3f1-fc55-4059-9983-06dbeca10768"). InnerVolumeSpecName "kube-api-access-gfw97". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:36:07.438620 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:07.438593 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b553f012-3d8c-4a9a-b514-410db05c2f0a-kube-api-access-jh68r" pod "b553f012-3d8c-4a9a-b514-410db05c2f0a" (UID: "b553f012-3d8c-4a9a-b514-410db05c2f0a"). InnerVolumeSpecName "kube-api-access-jh68r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:36:07.438729 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:07.438686 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4246b3f1-fc55-4059-9983-06dbeca10768-istio-token" pod "4246b3f1-fc55-4059-9983-06dbeca10768" (UID: "4246b3f1-fc55-4059-9983-06dbeca10768"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:36:07.441392 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:07.441351 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4246b3f1-fc55-4059-9983-06dbeca10768-istio-podinfo" pod "4246b3f1-fc55-4059-9983-06dbeca10768" (UID: "4246b3f1-fc55-4059-9983-06dbeca10768"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:36:07.442524 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:07.442491 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b553f012-3d8c-4a9a-b514-410db05c2f0a-istio-podinfo" pod "b553f012-3d8c-4a9a-b514-410db05c2f0a" (UID: "b553f012-3d8c-4a9a-b514-410db05c2f0a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:36:07.443544 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:07.443516 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b553f012-3d8c-4a9a-b514-410db05c2f0a-istio-envoy" pod "b553f012-3d8c-4a9a-b514-410db05c2f0a" (UID: "b553f012-3d8c-4a9a-b514-410db05c2f0a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:36:07.547302 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:07.541897 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52a0f195-418d-42be-8222-b030c8f6ba55-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52a0f195-418d-42be-8222-b030c8f6ba55-credential-socket\") pod \"52a0f195-418d-42be-8222-b030c8f6ba55\" (UID: \"52a0f195-418d-42be-8222-b030c8f6ba55\") " Nov 03 01:36:07.547302 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:07.541959 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52a0f195-418d-42be-8222-b030c8f6ba55-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52a0f195-418d-42be-8222-b030c8f6ba55-workload-certs\") pod \"52a0f195-418d-42be-8222-b030c8f6ba55\" (UID: \"52a0f195-418d-42be-8222-b030c8f6ba55\") " Nov 03 01:36:07.547302 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:07.542043 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/52a0f195-418d-42be-8222-b030c8f6ba55-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52a0f195-418d-42be-8222-b030c8f6ba55-istio-podinfo\") pod \"52a0f195-418d-42be-8222-b030c8f6ba55\" (UID: \"52a0f195-418d-42be-8222-b030c8f6ba55\") " Nov 03 01:36:07.547302 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:07.542331 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52a0f195-418d-42be-8222-b030c8f6ba55-kube-api-access-lp648\" (UniqueName: \"kubernetes.io/projected/52a0f195-418d-42be-8222-b030c8f6ba55-kube-api-access-lp648\") pod \"52a0f195-418d-42be-8222-b030c8f6ba55\" (UID: \"52a0f195-418d-42be-8222-b030c8f6ba55\") " Nov 03 01:36:07.547302 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:07.542459 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52a0f195-418d-42be-8222-b030c8f6ba55-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52a0f195-418d-42be-8222-b030c8f6ba55-workload-socket\") pod \"52a0f195-418d-42be-8222-b030c8f6ba55\" (UID: \"52a0f195-418d-42be-8222-b030c8f6ba55\") " Nov 03 01:36:07.547302 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:07.542533 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52a0f195-418d-42be-8222-b030c8f6ba55-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52a0f195-418d-42be-8222-b030c8f6ba55-istio-envoy\") pod \"52a0f195-418d-42be-8222-b030c8f6ba55\" (UID: \"52a0f195-418d-42be-8222-b030c8f6ba55\") " Nov 03 01:36:07.547302 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:07.542668 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52a0f195-418d-42be-8222-b030c8f6ba55-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52a0f195-418d-42be-8222-b030c8f6ba55-istiod-ca-cert\") pod \"52a0f195-418d-42be-8222-b030c8f6ba55\" (UID: \"52a0f195-418d-42be-8222-b030c8f6ba55\") " Nov 03 01:36:07.547302 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:07.542706 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52a0f195-418d-42be-8222-b030c8f6ba55-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52a0f195-418d-42be-8222-b030c8f6ba55-istio-data\") pod \"52a0f195-418d-42be-8222-b030c8f6ba55\" (UID: \"52a0f195-418d-42be-8222-b030c8f6ba55\") " Nov 03 01:36:07.547302 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:07.542744 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52a0f195-418d-42be-8222-b030c8f6ba55-istio-token\" (UniqueName: \"kubernetes.io/projected/52a0f195-418d-42be-8222-b030c8f6ba55-istio-token\") pod \"52a0f195-418d-42be-8222-b030c8f6ba55\" (UID: \"52a0f195-418d-42be-8222-b030c8f6ba55\") " Nov 03 01:36:07.547302 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:07.543492 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gfw97\" (UniqueName: \"kubernetes.io/projected/4246b3f1-fc55-4059-9983-06dbeca10768-kube-api-access-gfw97\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:36:07.547302 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:07.543522 4421 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4246b3f1-fc55-4059-9983-06dbeca10768-istio-envoy\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:36:07.547302 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:07.543538 4421 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4246b3f1-fc55-4059-9983-06dbeca10768-istio-podinfo\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:36:07.547302 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:07.543554 4421 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b553f012-3d8c-4a9a-b514-410db05c2f0a-workload-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:36:07.547302 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:07.543567 4421 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b553f012-3d8c-4a9a-b514-410db05c2f0a-istio-podinfo\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:36:07.547302 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:07.543596 4421 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4246b3f1-fc55-4059-9983-06dbeca10768-istio-token\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:36:07.547302 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:07.543611 4421 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b553f012-3d8c-4a9a-b514-410db05c2f0a-istio-data\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:36:07.547302 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:07.543626 4421 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4246b3f1-fc55-4059-9983-06dbeca10768-credential-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:36:07.547302 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:07.543641 4421 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4246b3f1-fc55-4059-9983-06dbeca10768-istio-data\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:36:07.547302 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:07.543656 4421 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4246b3f1-fc55-4059-9983-06dbeca10768-workload-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:36:07.549829 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:07.543670 4421 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4246b3f1-fc55-4059-9983-06dbeca10768-istiod-ca-cert\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:36:07.549829 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:07.549198 4421 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b553f012-3d8c-4a9a-b514-410db05c2f0a-istiod-ca-cert\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:36:07.549829 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:07.549231 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jh68r\" (UniqueName: \"kubernetes.io/projected/b553f012-3d8c-4a9a-b514-410db05c2f0a-kube-api-access-jh68r\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:36:07.549829 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:07.549248 4421 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b553f012-3d8c-4a9a-b514-410db05c2f0a-istio-token\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:36:07.549829 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:07.549263 4421 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4246b3f1-fc55-4059-9983-06dbeca10768-workload-certs\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:36:07.549829 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:07.549298 4421 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b553f012-3d8c-4a9a-b514-410db05c2f0a-istio-envoy\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:36:07.549829 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:07.549322 4421 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b553f012-3d8c-4a9a-b514-410db05c2f0a-workload-certs\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:36:07.549829 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:07.549349 4421 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b553f012-3d8c-4a9a-b514-410db05c2f0a-credential-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:36:07.557469 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:07.550218 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52a0f195-418d-42be-8222-b030c8f6ba55-credential-socket" pod "52a0f195-418d-42be-8222-b030c8f6ba55" (UID: "52a0f195-418d-42be-8222-b030c8f6ba55"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:36:07.557469 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:07.554534 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52a0f195-418d-42be-8222-b030c8f6ba55-istiod-ca-cert" pod "52a0f195-418d-42be-8222-b030c8f6ba55" (UID: "52a0f195-418d-42be-8222-b030c8f6ba55"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:36:07.557469 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:07.554840 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52a0f195-418d-42be-8222-b030c8f6ba55-istio-data" pod "52a0f195-418d-42be-8222-b030c8f6ba55" (UID: "52a0f195-418d-42be-8222-b030c8f6ba55"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:36:07.557469 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:07.556558 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52a0f195-418d-42be-8222-b030c8f6ba55-workload-certs" pod "52a0f195-418d-42be-8222-b030c8f6ba55" (UID: "52a0f195-418d-42be-8222-b030c8f6ba55"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:36:07.557469 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:07.556810 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52a0f195-418d-42be-8222-b030c8f6ba55-workload-socket" pod "52a0f195-418d-42be-8222-b030c8f6ba55" (UID: "52a0f195-418d-42be-8222-b030c8f6ba55"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:36:07.576075 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:07.576001 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/52a0f195-418d-42be-8222-b030c8f6ba55-istio-podinfo" pod "52a0f195-418d-42be-8222-b030c8f6ba55" (UID: "52a0f195-418d-42be-8222-b030c8f6ba55"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:36:07.576075 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:07.576032 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52a0f195-418d-42be-8222-b030c8f6ba55-istio-token" pod "52a0f195-418d-42be-8222-b030c8f6ba55" (UID: "52a0f195-418d-42be-8222-b030c8f6ba55"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:36:07.576665 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:07.576627 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52a0f195-418d-42be-8222-b030c8f6ba55-istio-envoy" pod "52a0f195-418d-42be-8222-b030c8f6ba55" (UID: "52a0f195-418d-42be-8222-b030c8f6ba55"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:36:07.577083 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:07.577052 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52a0f195-418d-42be-8222-b030c8f6ba55-kube-api-access-lp648" pod "52a0f195-418d-42be-8222-b030c8f6ba55" (UID: "52a0f195-418d-42be-8222-b030c8f6ba55"). InnerVolumeSpecName "kube-api-access-lp648". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:36:07.650622 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:07.650582 4421 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52a0f195-418d-42be-8222-b030c8f6ba55-credential-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:36:07.650622 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:07.650620 4421 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52a0f195-418d-42be-8222-b030c8f6ba55-workload-certs\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:36:07.650622 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:07.650629 4421 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52a0f195-418d-42be-8222-b030c8f6ba55-istio-podinfo\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:36:07.652554 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:07.650638 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lp648\" (UniqueName: \"kubernetes.io/projected/52a0f195-418d-42be-8222-b030c8f6ba55-kube-api-access-lp648\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:36:07.652554 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:07.650650 4421 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52a0f195-418d-42be-8222-b030c8f6ba55-workload-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:36:07.652554 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:07.650658 4421 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52a0f195-418d-42be-8222-b030c8f6ba55-istio-envoy\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:36:07.652554 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:07.650668 4421 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52a0f195-418d-42be-8222-b030c8f6ba55-istiod-ca-cert\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:36:07.652554 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:07.650676 4421 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52a0f195-418d-42be-8222-b030c8f6ba55-istio-data\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:36:07.652554 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:07.650685 4421 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52a0f195-418d-42be-8222-b030c8f6ba55-istio-token\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:36:07.710971 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:07.710939 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-b5cbp"] Nov 03 01:36:07.714991 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:07.714963 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-b5cbp"] Nov 03 01:36:07.981821 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:07.981732 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b553f012-3d8c-4a9a-b514-410db05c2f0a" path="/var/lib/kubelet/pods/b553f012-3d8c-4a9a-b514-410db05c2f0a/volumes" Nov 03 01:36:08.406235 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:08.406204 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-9v2m2" Nov 03 01:36:08.408020 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:08.407989 4421 generic.go:356] "Generic (PLEG): container finished" podID="05c74cad-b107-403b-bffc-af74dea66076" containerID="52f533bb4c8a15e2597427f81efc7c95ff1857307d03a59375f513687b8a55a0" exitCode=0 Nov 03 01:36:08.408176 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:08.408059 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-5ws4g" event={"ID":"05c74cad-b107-403b-bffc-af74dea66076","Type":"ContainerDied","Data":"52f533bb4c8a15e2597427f81efc7c95ff1857307d03a59375f513687b8a55a0"} Nov 03 01:36:08.408176 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:08.408124 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-vhvf6" Nov 03 01:36:08.408741 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:08.408723 4421 scope.go:122] "RemoveContainer" containerID="52f533bb4c8a15e2597427f81efc7c95ff1857307d03a59375f513687b8a55a0" Nov 03 01:36:08.410811 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:08.410780 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:36:08.410943 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:08.410848 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:36:08.410943 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:08.410879 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:36:08.461223 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:08.461189 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-vhvf6"] Nov 03 01:36:08.470606 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:08.470570 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-vhvf6"] Nov 03 01:36:08.488257 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:08.488228 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-9v2m2"] Nov 03 01:36:08.494496 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:08.494466 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-9v2m2"] Nov 03 01:36:09.435794 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:09.435751 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-5ws4g" event={"ID":"05c74cad-b107-403b-bffc-af74dea66076","Type":"ContainerStarted","Data":"35d21581f8d905ac294fb78104bcd18118fbf08cebe90ca75f0d9833b8c127e5"} Nov 03 01:36:09.981707 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:09.981657 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4246b3f1-fc55-4059-9983-06dbeca10768" path="/var/lib/kubelet/pods/4246b3f1-fc55-4059-9983-06dbeca10768/volumes" Nov 03 01:36:09.982344 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:09.982319 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52a0f195-418d-42be-8222-b030c8f6ba55" path="/var/lib/kubelet/pods/52a0f195-418d-42be-8222-b030c8f6ba55/volumes" Nov 03 01:36:10.226313 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:10.225791 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-5ws4g" Nov 03 01:36:10.226313 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:10.225940 4421 patch_prober.go:29] interesting pod/e2e-shared-a-w2-531447-openshift-default-7995d46767-5ws4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" start-of-body= Nov 03 01:36:10.226313 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:10.225985 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-5ws4g" podUID="05c74cad-b107-403b-bffc-af74dea66076" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" Nov 03 01:36:11.034888 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:11.034837 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-5dab6a-6c4b59c89-txkqj" podUID="896b4a5e-a44c-48a3-a5ee-aaef0ddad638" containerName="payload-processing" containerID="cri-o://c8bc28dc919ef1123a5e72cb7b7a8bbf2f798e6cdde506de1c25a1da48f6b937" gracePeriod=25 Nov 03 01:36:11.227298 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:11.217859 4421 patch_prober.go:29] interesting pod/e2e-shared-a-w2-531447-openshift-default-7995d46767-5ws4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" start-of-body= Nov 03 01:36:11.227298 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:11.217952 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-5ws4g" podUID="05c74cad-b107-403b-bffc-af74dea66076" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" Nov 03 01:36:12.227792 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:12.224751 4421 patch_prober.go:29] interesting pod/e2e-shared-a-w2-531447-openshift-default-7995d46767-5ws4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" start-of-body= Nov 03 01:36:12.227792 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:12.224822 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-5ws4g" podUID="05c74cad-b107-403b-bffc-af74dea66076" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" Nov 03 01:36:12.267211 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:12.267165 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-5dab6a-6c4b59c89-txkqj" podUID="896b4a5e-a44c-48a3-a5ee-aaef0ddad638" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:36:13.230926 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:13.230882 4421 patch_prober.go:29] interesting pod/e2e-shared-a-w2-531447-openshift-default-7995d46767-5ws4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" start-of-body= Nov 03 01:36:13.231419 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:13.230957 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-5ws4g" podUID="05c74cad-b107-403b-bffc-af74dea66076" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" Nov 03 01:36:14.216959 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:14.216912 4421 patch_prober.go:29] interesting pod/e2e-shared-a-w2-531447-openshift-default-7995d46767-5ws4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" start-of-body= Nov 03 01:36:14.217296 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:14.217243 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-5ws4g" podUID="05c74cad-b107-403b-bffc-af74dea66076" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" Nov 03 01:36:14.677545 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:36:14.677508 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 01:36:15.221786 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:15.221748 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-5ws4g" Nov 03 01:36:15.222262 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:15.222243 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-5ws4g" Nov 03 01:36:15.223113 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:15.223093 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-5ws4g" Nov 03 01:36:22.283429 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:22.283381 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-5dab6a-6c4b59c89-txkqj" podUID="896b4a5e-a44c-48a3-a5ee-aaef0ddad638" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:36:25.350351 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:25.350312 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-ddd653-5958b7b98-78g9l"] Nov 03 01:36:25.352129 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:25.352090 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ddd653-5958b7b98-78g9l" podUID="816d576b-324a-492c-b81f-cf3d4a6dfe92" containerName="maas-api" containerID="cri-o://3b2d5350ed7342b7dab9a48c5cbd8e79cae127963aae8a75e2d2008ea195e362" gracePeriod=30 Nov 03 01:36:25.706573 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:25.706362 4421 generic.go:356] "Generic (PLEG): container finished" podID="816d576b-324a-492c-b81f-cf3d4a6dfe92" containerID="3b2d5350ed7342b7dab9a48c5cbd8e79cae127963aae8a75e2d2008ea195e362" exitCode=0 Nov 03 01:36:25.706573 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:25.706435 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ddd653-5958b7b98-78g9l" event={"ID":"816d576b-324a-492c-b81f-cf3d4a6dfe92","Type":"ContainerDied","Data":"3b2d5350ed7342b7dab9a48c5cbd8e79cae127963aae8a75e2d2008ea195e362"} Nov 03 01:36:25.780019 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:25.779993 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ddd653-5958b7b98-78g9l" Nov 03 01:36:25.823685 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:25.823648 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/816d576b-324a-492c-b81f-cf3d4a6dfe92-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/816d576b-324a-492c-b81f-cf3d4a6dfe92-maas-api-tls\") pod \"816d576b-324a-492c-b81f-cf3d4a6dfe92\" (UID: \"816d576b-324a-492c-b81f-cf3d4a6dfe92\") " Nov 03 01:36:25.823883 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:25.823694 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/816d576b-324a-492c-b81f-cf3d4a6dfe92-metrics-certs\" (UniqueName: \"kubernetes.io/secret/816d576b-324a-492c-b81f-cf3d4a6dfe92-metrics-certs\") pod \"816d576b-324a-492c-b81f-cf3d4a6dfe92\" (UID: \"816d576b-324a-492c-b81f-cf3d4a6dfe92\") " Nov 03 01:36:25.823883 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:25.823855 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/816d576b-324a-492c-b81f-cf3d4a6dfe92-kube-api-access-vw8kk\" (UniqueName: \"kubernetes.io/projected/816d576b-324a-492c-b81f-cf3d4a6dfe92-kube-api-access-vw8kk\") pod \"816d576b-324a-492c-b81f-cf3d4a6dfe92\" (UID: \"816d576b-324a-492c-b81f-cf3d4a6dfe92\") " Nov 03 01:36:25.824019 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:25.823902 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/816d576b-324a-492c-b81f-cf3d4a6dfe92-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/816d576b-324a-492c-b81f-cf3d4a6dfe92-openshift-service-ca\") pod \"816d576b-324a-492c-b81f-cf3d4a6dfe92\" (UID: \"816d576b-324a-492c-b81f-cf3d4a6dfe92\") " Nov 03 01:36:25.824672 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:25.824642 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/816d576b-324a-492c-b81f-cf3d4a6dfe92-openshift-service-ca" pod "816d576b-324a-492c-b81f-cf3d4a6dfe92" (UID: "816d576b-324a-492c-b81f-cf3d4a6dfe92"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:36:25.827249 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:25.827224 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/816d576b-324a-492c-b81f-cf3d4a6dfe92-metrics-certs" pod "816d576b-324a-492c-b81f-cf3d4a6dfe92" (UID: "816d576b-324a-492c-b81f-cf3d4a6dfe92"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:36:25.827748 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:25.827471 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/816d576b-324a-492c-b81f-cf3d4a6dfe92-kube-api-access-vw8kk" pod "816d576b-324a-492c-b81f-cf3d4a6dfe92" (UID: "816d576b-324a-492c-b81f-cf3d4a6dfe92"). InnerVolumeSpecName "kube-api-access-vw8kk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:36:25.827970 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:25.827882 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/816d576b-324a-492c-b81f-cf3d4a6dfe92-maas-api-tls" pod "816d576b-324a-492c-b81f-cf3d4a6dfe92" (UID: "816d576b-324a-492c-b81f-cf3d4a6dfe92"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:36:25.960131 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:25.950591 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vw8kk\" (UniqueName: \"kubernetes.io/projected/816d576b-324a-492c-b81f-cf3d4a6dfe92-kube-api-access-vw8kk\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:36:25.960131 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:25.950630 4421 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/816d576b-324a-492c-b81f-cf3d4a6dfe92-openshift-service-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:36:25.960131 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:25.950650 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/816d576b-324a-492c-b81f-cf3d4a6dfe92-maas-api-tls\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:36:25.960131 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:25.950667 4421 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/816d576b-324a-492c-b81f-cf3d4a6dfe92-metrics-certs\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:36:26.716547 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:26.716515 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ddd653-5958b7b98-78g9l" Nov 03 01:36:26.716547 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:26.716539 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ddd653-5958b7b98-78g9l" event={"ID":"816d576b-324a-492c-b81f-cf3d4a6dfe92","Type":"ContainerDied","Data":"4f57e64b1e44ecc8c7ae00b90e7670c8a2bd49fdaa7477b2563db7ee5e63ea11"} Nov 03 01:36:26.717047 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:26.716598 4421 scope.go:122] "RemoveContainer" containerID="3b2d5350ed7342b7dab9a48c5cbd8e79cae127963aae8a75e2d2008ea195e362" Nov 03 01:36:26.738355 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:26.738324 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-ddd653-5958b7b98-78g9l"] Nov 03 01:36:26.742118 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:26.742096 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-ddd653-5958b7b98-78g9l"] Nov 03 01:36:27.996212 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:27.996166 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="816d576b-324a-492c-b81f-cf3d4a6dfe92" path="/var/lib/kubelet/pods/816d576b-324a-492c-b81f-cf3d4a6dfe92/volumes" Nov 03 01:36:28.406550 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:28.406511 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-5dab6a-openshift-default-6ff495656c-bxn8s"] Nov 03 01:36:28.406835 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:28.406804 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-5dab6a-openshift-default-6ff495656c-bxn8s" podUID="e0c12073-7ff8-413c-9940-978f4b600778" containerName="istio-proxy" containerID="cri-o://249bb0fe1738567e86a1979d8fcfbba2130e5107a49c6e70218a77527379e944" gracePeriod=30 Nov 03 01:36:28.950823 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:28.950773 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jvqst"] Nov 03 01:36:28.951062 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:28.951040 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jvqst" podUID="8669aad1-230a-4b5f-be97-2e7ecd2a9de3" containerName="discovery" containerID="cri-o://933aca6d122054cbdea416716a95e170fef97edca0e9b76c7491b9be36487dc9" gracePeriod=30 Nov 03 01:36:29.322011 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:29.321977 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jvqst" Nov 03 01:36:29.420782 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:29.420748 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8669aad1-230a-4b5f-be97-2e7ecd2a9de3-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/8669aad1-230a-4b5f-be97-2e7ecd2a9de3-istio-csr-ca-configmap\") pod \"8669aad1-230a-4b5f-be97-2e7ecd2a9de3\" (UID: \"8669aad1-230a-4b5f-be97-2e7ecd2a9de3\") " Nov 03 01:36:29.421004 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:29.420845 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8669aad1-230a-4b5f-be97-2e7ecd2a9de3-cacerts\" (UniqueName: \"kubernetes.io/secret/8669aad1-230a-4b5f-be97-2e7ecd2a9de3-cacerts\") pod \"8669aad1-230a-4b5f-be97-2e7ecd2a9de3\" (UID: \"8669aad1-230a-4b5f-be97-2e7ecd2a9de3\") " Nov 03 01:36:29.421004 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:29.420910 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8669aad1-230a-4b5f-be97-2e7ecd2a9de3-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/8669aad1-230a-4b5f-be97-2e7ecd2a9de3-local-certs\") pod \"8669aad1-230a-4b5f-be97-2e7ecd2a9de3\" (UID: \"8669aad1-230a-4b5f-be97-2e7ecd2a9de3\") " Nov 03 01:36:29.421004 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:29.420957 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8669aad1-230a-4b5f-be97-2e7ecd2a9de3-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8669aad1-230a-4b5f-be97-2e7ecd2a9de3-istio-kubeconfig\") pod \"8669aad1-230a-4b5f-be97-2e7ecd2a9de3\" (UID: \"8669aad1-230a-4b5f-be97-2e7ecd2a9de3\") " Nov 03 01:36:29.421004 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:29.420991 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8669aad1-230a-4b5f-be97-2e7ecd2a9de3-istio-token\" (UniqueName: \"kubernetes.io/projected/8669aad1-230a-4b5f-be97-2e7ecd2a9de3-istio-token\") pod \"8669aad1-230a-4b5f-be97-2e7ecd2a9de3\" (UID: \"8669aad1-230a-4b5f-be97-2e7ecd2a9de3\") " Nov 03 01:36:29.421199 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:29.421080 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8669aad1-230a-4b5f-be97-2e7ecd2a9de3-kube-api-access-nzcsz\" (UniqueName: \"kubernetes.io/projected/8669aad1-230a-4b5f-be97-2e7ecd2a9de3-kube-api-access-nzcsz\") pod \"8669aad1-230a-4b5f-be97-2e7ecd2a9de3\" (UID: \"8669aad1-230a-4b5f-be97-2e7ecd2a9de3\") " Nov 03 01:36:29.421199 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:29.421113 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8669aad1-230a-4b5f-be97-2e7ecd2a9de3-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/8669aad1-230a-4b5f-be97-2e7ecd2a9de3-istio-csr-dns-cert\") pod \"8669aad1-230a-4b5f-be97-2e7ecd2a9de3\" (UID: \"8669aad1-230a-4b5f-be97-2e7ecd2a9de3\") " Nov 03 01:36:29.422502 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:29.422463 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8669aad1-230a-4b5f-be97-2e7ecd2a9de3-istio-csr-ca-configmap" pod "8669aad1-230a-4b5f-be97-2e7ecd2a9de3" (UID: "8669aad1-230a-4b5f-be97-2e7ecd2a9de3"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:36:29.424460 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:29.424424 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8669aad1-230a-4b5f-be97-2e7ecd2a9de3-istio-csr-dns-cert" pod "8669aad1-230a-4b5f-be97-2e7ecd2a9de3" (UID: "8669aad1-230a-4b5f-be97-2e7ecd2a9de3"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:36:29.425355 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:29.425314 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8669aad1-230a-4b5f-be97-2e7ecd2a9de3-istio-token" pod "8669aad1-230a-4b5f-be97-2e7ecd2a9de3" (UID: "8669aad1-230a-4b5f-be97-2e7ecd2a9de3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:36:29.425461 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:29.425364 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8669aad1-230a-4b5f-be97-2e7ecd2a9de3-local-certs" pod "8669aad1-230a-4b5f-be97-2e7ecd2a9de3" (UID: "8669aad1-230a-4b5f-be97-2e7ecd2a9de3"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:36:29.425461 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:29.425426 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8669aad1-230a-4b5f-be97-2e7ecd2a9de3-kube-api-access-nzcsz" pod "8669aad1-230a-4b5f-be97-2e7ecd2a9de3" (UID: "8669aad1-230a-4b5f-be97-2e7ecd2a9de3"). InnerVolumeSpecName "kube-api-access-nzcsz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:36:29.425631 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:29.425607 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8669aad1-230a-4b5f-be97-2e7ecd2a9de3-istio-kubeconfig" pod "8669aad1-230a-4b5f-be97-2e7ecd2a9de3" (UID: "8669aad1-230a-4b5f-be97-2e7ecd2a9de3"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:36:29.428145 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:29.428111 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8669aad1-230a-4b5f-be97-2e7ecd2a9de3-cacerts" pod "8669aad1-230a-4b5f-be97-2e7ecd2a9de3" (UID: "8669aad1-230a-4b5f-be97-2e7ecd2a9de3"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:36:29.522990 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:29.522950 4421 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/8669aad1-230a-4b5f-be97-2e7ecd2a9de3-istio-csr-ca-configmap\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:36:29.522990 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:29.522983 4421 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/8669aad1-230a-4b5f-be97-2e7ecd2a9de3-cacerts\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:36:29.522990 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:29.522993 4421 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/8669aad1-230a-4b5f-be97-2e7ecd2a9de3-local-certs\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:36:29.523335 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:29.523002 4421 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8669aad1-230a-4b5f-be97-2e7ecd2a9de3-istio-kubeconfig\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:36:29.523335 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:29.523011 4421 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8669aad1-230a-4b5f-be97-2e7ecd2a9de3-istio-token\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:36:29.523335 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:29.523019 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nzcsz\" (UniqueName: \"kubernetes.io/projected/8669aad1-230a-4b5f-be97-2e7ecd2a9de3-kube-api-access-nzcsz\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:36:29.523335 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:29.523028 4421 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/8669aad1-230a-4b5f-be97-2e7ecd2a9de3-istio-csr-dns-cert\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:36:29.609691 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:29.599320 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w4-6c0ec0/sub-test-model-6c0ec0-kserve-666b858575-xpltp"] Nov 03 01:36:29.611575 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:29.609969 4421 cpu_manager.go:414] "containerMap: removing container" podUID="8669aad1-230a-4b5f-be97-2e7ecd2a9de3" containerName="discovery" Nov 03 01:36:29.611575 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:29.610007 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8669aad1-230a-4b5f-be97-2e7ecd2a9de3" containerName="discovery" Nov 03 01:36:29.611575 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:29.610042 4421 cpu_manager.go:414] "containerMap: removing container" podUID="52a0f195-418d-42be-8222-b030c8f6ba55" containerName="istio-proxy" Nov 03 01:36:29.611575 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:29.610051 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52a0f195-418d-42be-8222-b030c8f6ba55" containerName="istio-proxy" Nov 03 01:36:29.611575 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:29.610073 4421 cpu_manager.go:414] "containerMap: removing container" podUID="b553f012-3d8c-4a9a-b514-410db05c2f0a" containerName="istio-proxy" Nov 03 01:36:29.611575 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:29.610080 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b553f012-3d8c-4a9a-b514-410db05c2f0a" containerName="istio-proxy" Nov 03 01:36:29.611575 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:29.610117 4421 cpu_manager.go:414] "containerMap: removing container" podUID="816d576b-324a-492c-b81f-cf3d4a6dfe92" containerName="maas-api" Nov 03 01:36:29.611575 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:29.610124 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="816d576b-324a-492c-b81f-cf3d4a6dfe92" containerName="maas-api" Nov 03 01:36:29.611575 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:29.610155 4421 cpu_manager.go:414] "containerMap: removing container" podUID="4246b3f1-fc55-4059-9983-06dbeca10768" containerName="istio-proxy" Nov 03 01:36:29.611575 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:29.610163 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4246b3f1-fc55-4059-9983-06dbeca10768" containerName="istio-proxy" Nov 03 01:36:29.611575 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:29.610201 4421 cpu_manager.go:414] "containerMap: removing container" podUID="845cc619-fae7-4de0-a6a9-33d2e4f6ce87" containerName="revoke-keys" Nov 03 01:36:29.611575 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:29.610208 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="845cc619-fae7-4de0-a6a9-33d2e4f6ce87" containerName="revoke-keys" Nov 03 01:36:29.611575 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:29.610461 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="52a0f195-418d-42be-8222-b030c8f6ba55" containerName="istio-proxy" Nov 03 01:36:29.611575 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:29.610485 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="8669aad1-230a-4b5f-be97-2e7ecd2a9de3" containerName="discovery" Nov 03 01:36:29.611575 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:29.610497 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="52a0f195-418d-42be-8222-b030c8f6ba55" containerName="istio-proxy" Nov 03 01:36:29.611575 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:29.610509 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="b553f012-3d8c-4a9a-b514-410db05c2f0a" containerName="istio-proxy" Nov 03 01:36:29.611575 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:29.610523 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="816d576b-324a-492c-b81f-cf3d4a6dfe92" containerName="maas-api" Nov 03 01:36:29.611575 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:29.610538 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="4246b3f1-fc55-4059-9983-06dbeca10768" containerName="istio-proxy" Nov 03 01:36:29.611575 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:29.610557 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="845cc619-fae7-4de0-a6a9-33d2e4f6ce87" containerName="revoke-keys" Nov 03 01:36:29.615241 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:29.615217 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-6c0ec0/sub-test-model-6c0ec0-kserve-666b858575-xpltp"] Nov 03 01:36:29.615363 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:29.615349 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-6c0ec0/sub-test-model-6c0ec0-kserve-666b858575-xpltp" Nov 03 01:36:29.622087 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:29.622057 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-6c0ec0\"/\"default-dockercfg-kbnv9\"" Nov 03 01:36:29.623037 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:29.622993 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-6c0ec0\"/\"kube-root-ca.crt\"" Nov 03 01:36:29.623037 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:29.623020 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-6c0ec0\"/\"sub-test-model-6c0ec0-kserve-self-signed-certs\"" Nov 03 01:36:29.623316 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:29.623064 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-6c0ec0\"/\"openshift-service-ca.crt\"" Nov 03 01:36:29.726378 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:29.725716 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc3c2641-8192-4c01-a52c-29e3cd6de1f8-dshm\") pod \"sub-test-model-6c0ec0-kserve-666b858575-xpltp\" (UID: \"bc3c2641-8192-4c01-a52c-29e3cd6de1f8\") " pod="ai-tenant-e2e-shared-a-w4-6c0ec0/sub-test-model-6c0ec0-kserve-666b858575-xpltp" Nov 03 01:36:29.726378 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:29.725777 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxbmt\" (UniqueName: \"kubernetes.io/projected/bc3c2641-8192-4c01-a52c-29e3cd6de1f8-kube-api-access-kxbmt\") pod \"sub-test-model-6c0ec0-kserve-666b858575-xpltp\" (UID: \"bc3c2641-8192-4c01-a52c-29e3cd6de1f8\") " pod="ai-tenant-e2e-shared-a-w4-6c0ec0/sub-test-model-6c0ec0-kserve-666b858575-xpltp" Nov 03 01:36:29.726378 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:29.725940 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc3c2641-8192-4c01-a52c-29e3cd6de1f8-model-cache\") pod \"sub-test-model-6c0ec0-kserve-666b858575-xpltp\" (UID: \"bc3c2641-8192-4c01-a52c-29e3cd6de1f8\") " pod="ai-tenant-e2e-shared-a-w4-6c0ec0/sub-test-model-6c0ec0-kserve-666b858575-xpltp" Nov 03 01:36:29.726378 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:29.725978 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc3c2641-8192-4c01-a52c-29e3cd6de1f8-home\") pod \"sub-test-model-6c0ec0-kserve-666b858575-xpltp\" (UID: \"bc3c2641-8192-4c01-a52c-29e3cd6de1f8\") " pod="ai-tenant-e2e-shared-a-w4-6c0ec0/sub-test-model-6c0ec0-kserve-666b858575-xpltp" Nov 03 01:36:29.726378 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:29.726076 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc3c2641-8192-4c01-a52c-29e3cd6de1f8-tmp-dir\") pod \"sub-test-model-6c0ec0-kserve-666b858575-xpltp\" (UID: \"bc3c2641-8192-4c01-a52c-29e3cd6de1f8\") " pod="ai-tenant-e2e-shared-a-w4-6c0ec0/sub-test-model-6c0ec0-kserve-666b858575-xpltp" Nov 03 01:36:29.726378 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:29.726157 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc3c2641-8192-4c01-a52c-29e3cd6de1f8-tls-certs\") pod \"sub-test-model-6c0ec0-kserve-666b858575-xpltp\" (UID: \"bc3c2641-8192-4c01-a52c-29e3cd6de1f8\") " pod="ai-tenant-e2e-shared-a-w4-6c0ec0/sub-test-model-6c0ec0-kserve-666b858575-xpltp" Nov 03 01:36:29.816811 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:29.816769 4421 generic.go:356] "Generic (PLEG): container finished" podID="8669aad1-230a-4b5f-be97-2e7ecd2a9de3" containerID="933aca6d122054cbdea416716a95e170fef97edca0e9b76c7491b9be36487dc9" exitCode=0 Nov 03 01:36:29.817026 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:29.816867 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jvqst" Nov 03 01:36:29.817191 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:29.817162 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jvqst" event={"ID":"8669aad1-230a-4b5f-be97-2e7ecd2a9de3","Type":"ContainerDied","Data":"933aca6d122054cbdea416716a95e170fef97edca0e9b76c7491b9be36487dc9"} Nov 03 01:36:29.817259 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:29.817219 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jvqst" event={"ID":"8669aad1-230a-4b5f-be97-2e7ecd2a9de3","Type":"ContainerDied","Data":"b49b100725f54df2c201f34a83803ee5f8796bcc1ecb6c9eed02761ba956f7eb"} Nov 03 01:36:29.817259 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:29.817245 4421 scope.go:122] "RemoveContainer" containerID="933aca6d122054cbdea416716a95e170fef97edca0e9b76c7491b9be36487dc9" Nov 03 01:36:29.827688 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:29.827653 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc3c2641-8192-4c01-a52c-29e3cd6de1f8-tls-certs\") pod \"sub-test-model-6c0ec0-kserve-666b858575-xpltp\" (UID: \"bc3c2641-8192-4c01-a52c-29e3cd6de1f8\") " pod="ai-tenant-e2e-shared-a-w4-6c0ec0/sub-test-model-6c0ec0-kserve-666b858575-xpltp" Nov 03 01:36:29.828753 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:29.828728 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc3c2641-8192-4c01-a52c-29e3cd6de1f8-dshm\") pod \"sub-test-model-6c0ec0-kserve-666b858575-xpltp\" (UID: \"bc3c2641-8192-4c01-a52c-29e3cd6de1f8\") " pod="ai-tenant-e2e-shared-a-w4-6c0ec0/sub-test-model-6c0ec0-kserve-666b858575-xpltp" Nov 03 01:36:29.830382 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:29.830348 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxbmt\" (UniqueName: \"kubernetes.io/projected/bc3c2641-8192-4c01-a52c-29e3cd6de1f8-kube-api-access-kxbmt\") pod \"sub-test-model-6c0ec0-kserve-666b858575-xpltp\" (UID: \"bc3c2641-8192-4c01-a52c-29e3cd6de1f8\") " pod="ai-tenant-e2e-shared-a-w4-6c0ec0/sub-test-model-6c0ec0-kserve-666b858575-xpltp" Nov 03 01:36:29.831430 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:29.831372 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc3c2641-8192-4c01-a52c-29e3cd6de1f8-model-cache\") pod \"sub-test-model-6c0ec0-kserve-666b858575-xpltp\" (UID: \"bc3c2641-8192-4c01-a52c-29e3cd6de1f8\") " pod="ai-tenant-e2e-shared-a-w4-6c0ec0/sub-test-model-6c0ec0-kserve-666b858575-xpltp" Nov 03 01:36:29.831990 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:29.831973 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc3c2641-8192-4c01-a52c-29e3cd6de1f8-model-cache\") pod \"sub-test-model-6c0ec0-kserve-666b858575-xpltp\" (UID: \"bc3c2641-8192-4c01-a52c-29e3cd6de1f8\") " pod="ai-tenant-e2e-shared-a-w4-6c0ec0/sub-test-model-6c0ec0-kserve-666b858575-xpltp" Nov 03 01:36:29.832150 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:29.832134 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc3c2641-8192-4c01-a52c-29e3cd6de1f8-home\") pod \"sub-test-model-6c0ec0-kserve-666b858575-xpltp\" (UID: \"bc3c2641-8192-4c01-a52c-29e3cd6de1f8\") " pod="ai-tenant-e2e-shared-a-w4-6c0ec0/sub-test-model-6c0ec0-kserve-666b858575-xpltp" Nov 03 01:36:29.832396 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:29.832384 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc3c2641-8192-4c01-a52c-29e3cd6de1f8-tmp-dir\") pod \"sub-test-model-6c0ec0-kserve-666b858575-xpltp\" (UID: \"bc3c2641-8192-4c01-a52c-29e3cd6de1f8\") " pod="ai-tenant-e2e-shared-a-w4-6c0ec0/sub-test-model-6c0ec0-kserve-666b858575-xpltp" Nov 03 01:36:29.833004 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:29.832631 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc3c2641-8192-4c01-a52c-29e3cd6de1f8-home\") pod \"sub-test-model-6c0ec0-kserve-666b858575-xpltp\" (UID: \"bc3c2641-8192-4c01-a52c-29e3cd6de1f8\") " pod="ai-tenant-e2e-shared-a-w4-6c0ec0/sub-test-model-6c0ec0-kserve-666b858575-xpltp" Nov 03 01:36:29.833004 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:29.832682 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc3c2641-8192-4c01-a52c-29e3cd6de1f8-tmp-dir\") pod \"sub-test-model-6c0ec0-kserve-666b858575-xpltp\" (UID: \"bc3c2641-8192-4c01-a52c-29e3cd6de1f8\") " pod="ai-tenant-e2e-shared-a-w4-6c0ec0/sub-test-model-6c0ec0-kserve-666b858575-xpltp" Nov 03 01:36:29.835212 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:29.835155 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc3c2641-8192-4c01-a52c-29e3cd6de1f8-tls-certs\") pod \"sub-test-model-6c0ec0-kserve-666b858575-xpltp\" (UID: \"bc3c2641-8192-4c01-a52c-29e3cd6de1f8\") " pod="ai-tenant-e2e-shared-a-w4-6c0ec0/sub-test-model-6c0ec0-kserve-666b858575-xpltp" Nov 03 01:36:29.835738 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:29.835653 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc3c2641-8192-4c01-a52c-29e3cd6de1f8-dshm\") pod \"sub-test-model-6c0ec0-kserve-666b858575-xpltp\" (UID: \"bc3c2641-8192-4c01-a52c-29e3cd6de1f8\") " pod="ai-tenant-e2e-shared-a-w4-6c0ec0/sub-test-model-6c0ec0-kserve-666b858575-xpltp" Nov 03 01:36:29.838839 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:29.838808 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxbmt\" (UniqueName: \"kubernetes.io/projected/bc3c2641-8192-4c01-a52c-29e3cd6de1f8-kube-api-access-kxbmt\") pod \"sub-test-model-6c0ec0-kserve-666b858575-xpltp\" (UID: \"bc3c2641-8192-4c01-a52c-29e3cd6de1f8\") " pod="ai-tenant-e2e-shared-a-w4-6c0ec0/sub-test-model-6c0ec0-kserve-666b858575-xpltp" Nov 03 01:36:29.847050 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:29.846927 4421 scope.go:122] "RemoveContainer" containerID="933aca6d122054cbdea416716a95e170fef97edca0e9b76c7491b9be36487dc9" Nov 03 01:36:29.847847 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:36:29.847805 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"933aca6d122054cbdea416716a95e170fef97edca0e9b76c7491b9be36487dc9\": container with ID starting with 933aca6d122054cbdea416716a95e170fef97edca0e9b76c7491b9be36487dc9 not found: ID does not exist" containerID="933aca6d122054cbdea416716a95e170fef97edca0e9b76c7491b9be36487dc9" Nov 03 01:36:29.847970 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:29.847844 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"933aca6d122054cbdea416716a95e170fef97edca0e9b76c7491b9be36487dc9"} err="failed to get container status \"933aca6d122054cbdea416716a95e170fef97edca0e9b76c7491b9be36487dc9\": rpc error: code = NotFound desc = could not find container \"933aca6d122054cbdea416716a95e170fef97edca0e9b76c7491b9be36487dc9\": container with ID starting with 933aca6d122054cbdea416716a95e170fef97edca0e9b76c7491b9be36487dc9 not found: ID does not exist" Nov 03 01:36:29.848920 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:29.848898 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jvqst"] Nov 03 01:36:29.852637 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:29.852615 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jvqst"] Nov 03 01:36:29.931956 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:29.931865 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-6c0ec0/sub-test-model-6c0ec0-kserve-666b858575-xpltp" Nov 03 01:36:29.988952 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:29.988915 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8669aad1-230a-4b5f-be97-2e7ecd2a9de3" path="/var/lib/kubelet/pods/8669aad1-230a-4b5f-be97-2e7ecd2a9de3/volumes" Nov 03 01:36:30.314528 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:30.314449 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-6c0ec0/sub-test-model-6c0ec0-kserve-666b858575-xpltp"] Nov 03 01:36:30.315594 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:36:30.315572 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbc3c2641_8192_4c01_a52c_29e3cd6de1f8.slice/crio-6cedbc0fd939b1bbc3bfe96b059bb92c1f070041cd30061d9568ca129ad9ffc8 WatchSource:0}: Error finding container 6cedbc0fd939b1bbc3bfe96b059bb92c1f070041cd30061d9568ca129ad9ffc8: Status 404 returned error can't find the container with id 6cedbc0fd939b1bbc3bfe96b059bb92c1f070041cd30061d9568ca129ad9ffc8 Nov 03 01:36:30.829921 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:30.829882 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-6c0ec0/sub-test-model-6c0ec0-kserve-666b858575-xpltp" event={"ID":"bc3c2641-8192-4c01-a52c-29e3cd6de1f8","Type":"ContainerStarted","Data":"9a41b6c191fbbb6b19d8629ecb0cf5fe41dffc624c50b5563b93aa83c2641175"} Nov 03 01:36:30.829921 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:30.829932 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-6c0ec0/sub-test-model-6c0ec0-kserve-666b858575-xpltp" event={"ID":"bc3c2641-8192-4c01-a52c-29e3cd6de1f8","Type":"ContainerStarted","Data":"6cedbc0fd939b1bbc3bfe96b059bb92c1f070041cd30061d9568ca129ad9ffc8"} Nov 03 01:36:30.851915 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:30.851858 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w4-6c0ec0/sub-test-model-6c0ec0-kserve-666b858575-xpltp" podStartSLOduration=1.851844962 podStartE2EDuration="1.851844962s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:36:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:36:30.848542801 +0000 UTC m=+3333.019545897" watchObservedRunningTime="2026-11-03 01:36:30.851844962 +0000 UTC m=+3333.022848058" Nov 03 01:36:30.932496 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:30.932445 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-6c0ec0/sub-test-model-6c0ec0-kserve-666b858575-xpltp" Nov 03 01:36:31.325617 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:31.325583 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-5dab6a-6c4b59c89-txkqj" Nov 03 01:36:31.355318 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:31.355258 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/896b4a5e-a44c-48a3-a5ee-aaef0ddad638-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/896b4a5e-a44c-48a3-a5ee-aaef0ddad638-plugins-config-volume\") pod \"896b4a5e-a44c-48a3-a5ee-aaef0ddad638\" (UID: \"896b4a5e-a44c-48a3-a5ee-aaef0ddad638\") " Nov 03 01:36:31.355483 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:31.355389 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/896b4a5e-a44c-48a3-a5ee-aaef0ddad638-kube-api-access-nrmsz\" (UniqueName: \"kubernetes.io/projected/896b4a5e-a44c-48a3-a5ee-aaef0ddad638-kube-api-access-nrmsz\") pod \"896b4a5e-a44c-48a3-a5ee-aaef0ddad638\" (UID: \"896b4a5e-a44c-48a3-a5ee-aaef0ddad638\") " Nov 03 01:36:31.355697 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:31.355672 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/896b4a5e-a44c-48a3-a5ee-aaef0ddad638-plugins-config-volume" pod "896b4a5e-a44c-48a3-a5ee-aaef0ddad638" (UID: "896b4a5e-a44c-48a3-a5ee-aaef0ddad638"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:36:31.356143 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:31.356121 4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/896b4a5e-a44c-48a3-a5ee-aaef0ddad638-plugins-config-volume\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:36:31.358814 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:31.358785 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/896b4a5e-a44c-48a3-a5ee-aaef0ddad638-kube-api-access-nrmsz" pod "896b4a5e-a44c-48a3-a5ee-aaef0ddad638" (UID: "896b4a5e-a44c-48a3-a5ee-aaef0ddad638"). InnerVolumeSpecName "kube-api-access-nrmsz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:36:31.457488 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:31.457389 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrmsz\" (UniqueName: \"kubernetes.io/projected/896b4a5e-a44c-48a3-a5ee-aaef0ddad638-kube-api-access-nrmsz\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:36:31.840814 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:31.840779 4421 generic.go:356] "Generic (PLEG): container finished" podID="896b4a5e-a44c-48a3-a5ee-aaef0ddad638" containerID="c8bc28dc919ef1123a5e72cb7b7a8bbf2f798e6cdde506de1c25a1da48f6b937" exitCode=0 Nov 03 01:36:31.841411 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:31.840848 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-5dab6a-6c4b59c89-txkqj" Nov 03 01:36:31.841411 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:31.840872 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-5dab6a-6c4b59c89-txkqj" event={"ID":"896b4a5e-a44c-48a3-a5ee-aaef0ddad638","Type":"ContainerDied","Data":"c8bc28dc919ef1123a5e72cb7b7a8bbf2f798e6cdde506de1c25a1da48f6b937"} Nov 03 01:36:31.841411 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:31.840954 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-5dab6a-6c4b59c89-txkqj" event={"ID":"896b4a5e-a44c-48a3-a5ee-aaef0ddad638","Type":"ContainerDied","Data":"5b61f3b19264594cb01990e007fe5e426059dc436f4f69f6dff5e4e4d195d5bf"} Nov 03 01:36:31.841411 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:31.840986 4421 scope.go:122] "RemoveContainer" containerID="c8bc28dc919ef1123a5e72cb7b7a8bbf2f798e6cdde506de1c25a1da48f6b937" Nov 03 01:36:31.854341 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:31.854315 4421 scope.go:122] "RemoveContainer" containerID="c8bc28dc919ef1123a5e72cb7b7a8bbf2f798e6cdde506de1c25a1da48f6b937" Nov 03 01:36:31.854743 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:36:31.854719 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8bc28dc919ef1123a5e72cb7b7a8bbf2f798e6cdde506de1c25a1da48f6b937\": container with ID starting with c8bc28dc919ef1123a5e72cb7b7a8bbf2f798e6cdde506de1c25a1da48f6b937 not found: ID does not exist" containerID="c8bc28dc919ef1123a5e72cb7b7a8bbf2f798e6cdde506de1c25a1da48f6b937" Nov 03 01:36:31.854852 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:31.854753 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8bc28dc919ef1123a5e72cb7b7a8bbf2f798e6cdde506de1c25a1da48f6b937"} err="failed to get container status \"c8bc28dc919ef1123a5e72cb7b7a8bbf2f798e6cdde506de1c25a1da48f6b937\": rpc error: code = NotFound desc = could not find container \"c8bc28dc919ef1123a5e72cb7b7a8bbf2f798e6cdde506de1c25a1da48f6b937\": container with ID starting with c8bc28dc919ef1123a5e72cb7b7a8bbf2f798e6cdde506de1c25a1da48f6b937 not found: ID does not exist" Nov 03 01:36:31.859341 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:31.859316 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-c9sjv"] Nov 03 01:36:31.860086 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:31.860063 4421 cpu_manager.go:414] "containerMap: removing container" podUID="896b4a5e-a44c-48a3-a5ee-aaef0ddad638" containerName="payload-processing" Nov 03 01:36:31.860086 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:31.860084 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="896b4a5e-a44c-48a3-a5ee-aaef0ddad638" containerName="payload-processing" Nov 03 01:36:31.860262 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:31.860125 4421 cpu_manager.go:414] "containerMap: removing container" podUID="52a0f195-418d-42be-8222-b030c8f6ba55" containerName="istio-proxy" Nov 03 01:36:31.860262 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:31.860130 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52a0f195-418d-42be-8222-b030c8f6ba55" containerName="istio-proxy" Nov 03 01:36:31.860262 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:31.860241 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="896b4a5e-a44c-48a3-a5ee-aaef0ddad638" containerName="payload-processing" Nov 03 01:36:31.865229 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:31.865176 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-c9sjv" Nov 03 01:36:31.874347 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:31.874168 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-5dab6a-6c4b59c89-txkqj"] Nov 03 01:36:31.881351 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:31.881247 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-c9sjv"] Nov 03 01:36:31.886638 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:31.886605 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-5dab6a-6c4b59c89-txkqj"] Nov 03 01:36:31.964522 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:31.964479 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73bae713-0108-4120-8486-a35dbf68b0c9-workload-socket\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-c9sjv\" (UID: \"73bae713-0108-4120-8486-a35dbf68b0c9\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-c9sjv" Nov 03 01:36:31.964705 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:31.964556 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73bae713-0108-4120-8486-a35dbf68b0c9-credential-socket\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-c9sjv\" (UID: \"73bae713-0108-4120-8486-a35dbf68b0c9\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-c9sjv" Nov 03 01:36:31.964705 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:31.964631 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73bae713-0108-4120-8486-a35dbf68b0c9-workload-certs\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-c9sjv\" (UID: \"73bae713-0108-4120-8486-a35dbf68b0c9\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-c9sjv" Nov 03 01:36:31.964786 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:31.964720 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73bae713-0108-4120-8486-a35dbf68b0c9-istiod-ca-cert\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-c9sjv\" (UID: \"73bae713-0108-4120-8486-a35dbf68b0c9\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-c9sjv" Nov 03 01:36:31.964938 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:31.964901 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73bae713-0108-4120-8486-a35dbf68b0c9-istio-podinfo\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-c9sjv\" (UID: \"73bae713-0108-4120-8486-a35dbf68b0c9\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-c9sjv" Nov 03 01:36:31.965092 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:31.964968 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73bae713-0108-4120-8486-a35dbf68b0c9-istio-token\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-c9sjv\" (UID: \"73bae713-0108-4120-8486-a35dbf68b0c9\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-c9sjv" Nov 03 01:36:31.965092 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:31.964998 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzbp8\" (UniqueName: \"kubernetes.io/projected/73bae713-0108-4120-8486-a35dbf68b0c9-kube-api-access-pzbp8\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-c9sjv\" (UID: \"73bae713-0108-4120-8486-a35dbf68b0c9\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-c9sjv" Nov 03 01:36:31.965092 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:31.965042 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73bae713-0108-4120-8486-a35dbf68b0c9-istio-envoy\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-c9sjv\" (UID: \"73bae713-0108-4120-8486-a35dbf68b0c9\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-c9sjv" Nov 03 01:36:31.965242 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:31.965151 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73bae713-0108-4120-8486-a35dbf68b0c9-istio-data\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-c9sjv\" (UID: \"73bae713-0108-4120-8486-a35dbf68b0c9\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-c9sjv" Nov 03 01:36:31.981599 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:31.981571 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="896b4a5e-a44c-48a3-a5ee-aaef0ddad638" path="/var/lib/kubelet/pods/896b4a5e-a44c-48a3-a5ee-aaef0ddad638/volumes" Nov 03 01:36:32.066572 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:32.066533 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73bae713-0108-4120-8486-a35dbf68b0c9-istio-podinfo\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-c9sjv\" (UID: \"73bae713-0108-4120-8486-a35dbf68b0c9\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-c9sjv" Nov 03 01:36:32.066572 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:32.066579 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73bae713-0108-4120-8486-a35dbf68b0c9-istio-token\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-c9sjv\" (UID: \"73bae713-0108-4120-8486-a35dbf68b0c9\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-c9sjv" Nov 03 01:36:32.066914 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:32.066699 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzbp8\" (UniqueName: \"kubernetes.io/projected/73bae713-0108-4120-8486-a35dbf68b0c9-kube-api-access-pzbp8\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-c9sjv\" (UID: \"73bae713-0108-4120-8486-a35dbf68b0c9\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-c9sjv" Nov 03 01:36:32.066914 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:32.066749 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73bae713-0108-4120-8486-a35dbf68b0c9-istio-envoy\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-c9sjv\" (UID: \"73bae713-0108-4120-8486-a35dbf68b0c9\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-c9sjv" Nov 03 01:36:32.066914 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:32.066791 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73bae713-0108-4120-8486-a35dbf68b0c9-istio-data\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-c9sjv\" (UID: \"73bae713-0108-4120-8486-a35dbf68b0c9\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-c9sjv" Nov 03 01:36:32.068707 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:32.066987 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73bae713-0108-4120-8486-a35dbf68b0c9-workload-socket\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-c9sjv\" (UID: \"73bae713-0108-4120-8486-a35dbf68b0c9\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-c9sjv" Nov 03 01:36:32.068707 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:32.067088 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73bae713-0108-4120-8486-a35dbf68b0c9-credential-socket\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-c9sjv\" (UID: \"73bae713-0108-4120-8486-a35dbf68b0c9\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-c9sjv" Nov 03 01:36:32.068707 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:32.067130 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73bae713-0108-4120-8486-a35dbf68b0c9-workload-certs\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-c9sjv\" (UID: \"73bae713-0108-4120-8486-a35dbf68b0c9\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-c9sjv" Nov 03 01:36:32.068707 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:32.067189 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73bae713-0108-4120-8486-a35dbf68b0c9-istiod-ca-cert\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-c9sjv\" (UID: \"73bae713-0108-4120-8486-a35dbf68b0c9\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-c9sjv" Nov 03 01:36:32.068707 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:32.067227 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73bae713-0108-4120-8486-a35dbf68b0c9-istio-data\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-c9sjv\" (UID: \"73bae713-0108-4120-8486-a35dbf68b0c9\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-c9sjv" Nov 03 01:36:32.068707 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:32.067356 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73bae713-0108-4120-8486-a35dbf68b0c9-workload-socket\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-c9sjv\" (UID: \"73bae713-0108-4120-8486-a35dbf68b0c9\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-c9sjv" Nov 03 01:36:32.068707 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:32.067443 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73bae713-0108-4120-8486-a35dbf68b0c9-credential-socket\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-c9sjv\" (UID: \"73bae713-0108-4120-8486-a35dbf68b0c9\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-c9sjv" Nov 03 01:36:32.068707 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:32.067686 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73bae713-0108-4120-8486-a35dbf68b0c9-workload-certs\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-c9sjv\" (UID: \"73bae713-0108-4120-8486-a35dbf68b0c9\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-c9sjv" Nov 03 01:36:32.068707 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:32.068095 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73bae713-0108-4120-8486-a35dbf68b0c9-istiod-ca-cert\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-c9sjv\" (UID: \"73bae713-0108-4120-8486-a35dbf68b0c9\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-c9sjv" Nov 03 01:36:32.069429 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:32.069126 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73bae713-0108-4120-8486-a35dbf68b0c9-istio-envoy\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-c9sjv\" (UID: \"73bae713-0108-4120-8486-a35dbf68b0c9\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-c9sjv" Nov 03 01:36:32.069429 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:32.069250 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73bae713-0108-4120-8486-a35dbf68b0c9-istio-podinfo\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-c9sjv\" (UID: \"73bae713-0108-4120-8486-a35dbf68b0c9\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-c9sjv" Nov 03 01:36:32.077393 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:32.077373 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73bae713-0108-4120-8486-a35dbf68b0c9-istio-token\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-c9sjv\" (UID: \"73bae713-0108-4120-8486-a35dbf68b0c9\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-c9sjv" Nov 03 01:36:32.079714 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:32.079690 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzbp8\" (UniqueName: \"kubernetes.io/projected/73bae713-0108-4120-8486-a35dbf68b0c9-kube-api-access-pzbp8\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-c9sjv\" (UID: \"73bae713-0108-4120-8486-a35dbf68b0c9\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-c9sjv" Nov 03 01:36:32.190317 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:32.190199 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-c9sjv" Nov 03 01:36:32.524070 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:32.524032 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-c9sjv"] Nov 03 01:36:32.524237 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:36:32.524209 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73bae713_0108_4120_8486_a35dbf68b0c9.slice/crio-62027605b2b3ba4b1731586e5f45a6f58042c8bb194468bb2ecaac84df7b95fb WatchSource:0}: Error finding container 62027605b2b3ba4b1731586e5f45a6f58042c8bb194468bb2ecaac84df7b95fb: Status 404 returned error can't find the container with id 62027605b2b3ba4b1731586e5f45a6f58042c8bb194468bb2ecaac84df7b95fb Nov 03 01:36:32.527585 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:32.527541 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:36:32.527718 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:32.527634 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:36:32.527718 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:32.527681 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:36:32.852682 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:32.852575 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-c9sjv" event={"ID":"73bae713-0108-4120-8486-a35dbf68b0c9","Type":"ContainerStarted","Data":"a5f6e226d649f869467b1020f02969e268556d71065246f2002bf534df9c178a"} Nov 03 01:36:32.852682 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:32.852644 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-c9sjv" event={"ID":"73bae713-0108-4120-8486-a35dbf68b0c9","Type":"ContainerStarted","Data":"62027605b2b3ba4b1731586e5f45a6f58042c8bb194468bb2ecaac84df7b95fb"} Nov 03 01:36:32.871839 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:32.871784 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-c9sjv" podStartSLOduration=1.8717683649999999 podStartE2EDuration="1.871768365s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:36:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:36:32.870336148 +0000 UTC m=+3335.041339244" watchObservedRunningTime="2026-11-03 01:36:32.871768365 +0000 UTC m=+3335.042771461" Nov 03 01:36:33.190396 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:33.190305 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-c9sjv" Nov 03 01:36:33.192151 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:33.192123 4421 patch_prober.go:29] interesting pod/e2e-shared-a-w2-531447-openshift-default-7995d46767-c9sjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.101:15021/healthz/ready\": dial tcp 10.134.1.101:15021: connect: connection refused" start-of-body= Nov 03 01:36:33.192318 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:33.192170 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-c9sjv" podUID="73bae713-0108-4120-8486-a35dbf68b0c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.101:15021/healthz/ready\": dial tcp 10.134.1.101:15021: connect: connection refused" Nov 03 01:36:33.236298 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:33.236070 4421 patch_prober.go:29] interesting pod/e2e-inf-b-5dab6a-openshift-default-6ff495656c-bxn8s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 01:36:33.236298 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:33.236137 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-5dab6a-openshift-default-6ff495656c-bxn8s" podUID="e0c12073-7ff8-413c-9940-978f4b600778" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 01:36:33.776769 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:33.776740 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5dab6a-openshift-default-6ff495656c-bxn8s" Nov 03 01:36:33.865237 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:33.865193 4421 generic.go:356] "Generic (PLEG): container finished" podID="e0c12073-7ff8-413c-9940-978f4b600778" containerID="249bb0fe1738567e86a1979d8fcfbba2130e5107a49c6e70218a77527379e944" exitCode=0 Nov 03 01:36:33.865708 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:33.865236 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5dab6a-openshift-default-6ff495656c-bxn8s" event={"ID":"e0c12073-7ff8-413c-9940-978f4b600778","Type":"ContainerDied","Data":"249bb0fe1738567e86a1979d8fcfbba2130e5107a49c6e70218a77527379e944"} Nov 03 01:36:33.865708 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:33.865254 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5dab6a-openshift-default-6ff495656c-bxn8s" Nov 03 01:36:33.865708 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:33.865329 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5dab6a-openshift-default-6ff495656c-bxn8s" event={"ID":"e0c12073-7ff8-413c-9940-978f4b600778","Type":"ContainerDied","Data":"a10011ecb5c5c1f2e6aa6ec2891d49ecbbd492c9b499e0e072369f4d6990fc7b"} Nov 03 01:36:33.865708 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:33.865369 4421 scope.go:122] "RemoveContainer" containerID="249bb0fe1738567e86a1979d8fcfbba2130e5107a49c6e70218a77527379e944" Nov 03 01:36:33.879689 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:33.879662 4421 scope.go:122] "RemoveContainer" containerID="4fd927cf38a5c09578c9fa47907e1dbebcfc4f5b21825dd9733c61fb91f2a3f9" Nov 03 01:36:33.890608 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:33.890589 4421 scope.go:122] "RemoveContainer" containerID="249bb0fe1738567e86a1979d8fcfbba2130e5107a49c6e70218a77527379e944" Nov 03 01:36:33.890917 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:36:33.890895 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"249bb0fe1738567e86a1979d8fcfbba2130e5107a49c6e70218a77527379e944\": container with ID starting with 249bb0fe1738567e86a1979d8fcfbba2130e5107a49c6e70218a77527379e944 not found: ID does not exist" containerID="249bb0fe1738567e86a1979d8fcfbba2130e5107a49c6e70218a77527379e944" Nov 03 01:36:33.890964 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:33.890928 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"249bb0fe1738567e86a1979d8fcfbba2130e5107a49c6e70218a77527379e944"} err="failed to get container status \"249bb0fe1738567e86a1979d8fcfbba2130e5107a49c6e70218a77527379e944\": rpc error: code = NotFound desc = could not find container \"249bb0fe1738567e86a1979d8fcfbba2130e5107a49c6e70218a77527379e944\": container with ID starting with 249bb0fe1738567e86a1979d8fcfbba2130e5107a49c6e70218a77527379e944 not found: ID does not exist" Nov 03 01:36:33.890964 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:33.890946 4421 scope.go:122] "RemoveContainer" containerID="4fd927cf38a5c09578c9fa47907e1dbebcfc4f5b21825dd9733c61fb91f2a3f9" Nov 03 01:36:33.891821 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:36:33.891184 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4fd927cf38a5c09578c9fa47907e1dbebcfc4f5b21825dd9733c61fb91f2a3f9\": container with ID starting with 4fd927cf38a5c09578c9fa47907e1dbebcfc4f5b21825dd9733c61fb91f2a3f9 not found: ID does not exist" containerID="4fd927cf38a5c09578c9fa47907e1dbebcfc4f5b21825dd9733c61fb91f2a3f9" Nov 03 01:36:33.891821 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:33.891217 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fd927cf38a5c09578c9fa47907e1dbebcfc4f5b21825dd9733c61fb91f2a3f9"} err="failed to get container status \"4fd927cf38a5c09578c9fa47907e1dbebcfc4f5b21825dd9733c61fb91f2a3f9\": rpc error: code = NotFound desc = could not find container \"4fd927cf38a5c09578c9fa47907e1dbebcfc4f5b21825dd9733c61fb91f2a3f9\": container with ID starting with 4fd927cf38a5c09578c9fa47907e1dbebcfc4f5b21825dd9733c61fb91f2a3f9 not found: ID does not exist" Nov 03 01:36:33.895745 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:33.895724 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0c12073-7ff8-413c-9940-978f4b600778-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0c12073-7ff8-413c-9940-978f4b600778-istio-envoy\") pod \"e0c12073-7ff8-413c-9940-978f4b600778\" (UID: \"e0c12073-7ff8-413c-9940-978f4b600778\") " Nov 03 01:36:33.895873 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:33.895858 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0c12073-7ff8-413c-9940-978f4b600778-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0c12073-7ff8-413c-9940-978f4b600778-workload-socket\") pod \"e0c12073-7ff8-413c-9940-978f4b600778\" (UID: \"e0c12073-7ff8-413c-9940-978f4b600778\") " Nov 03 01:36:33.895942 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:33.895911 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0c12073-7ff8-413c-9940-978f4b600778-kube-api-access-96mbm\" (UniqueName: \"kubernetes.io/projected/e0c12073-7ff8-413c-9940-978f4b600778-kube-api-access-96mbm\") pod \"e0c12073-7ff8-413c-9940-978f4b600778\" (UID: \"e0c12073-7ff8-413c-9940-978f4b600778\") " Nov 03 01:36:33.896004 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:33.895993 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0c12073-7ff8-413c-9940-978f4b600778-istio-token\" (UniqueName: \"kubernetes.io/projected/e0c12073-7ff8-413c-9940-978f4b600778-istio-token\") pod \"e0c12073-7ff8-413c-9940-978f4b600778\" (UID: \"e0c12073-7ff8-413c-9940-978f4b600778\") " Nov 03 01:36:33.896063 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:33.896026 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0c12073-7ff8-413c-9940-978f4b600778-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0c12073-7ff8-413c-9940-978f4b600778-workload-certs\") pod \"e0c12073-7ff8-413c-9940-978f4b600778\" (UID: \"e0c12073-7ff8-413c-9940-978f4b600778\") " Nov 03 01:36:33.896142 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:33.896126 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0c12073-7ff8-413c-9940-978f4b600778-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0c12073-7ff8-413c-9940-978f4b600778-istio-data\") pod \"e0c12073-7ff8-413c-9940-978f4b600778\" (UID: \"e0c12073-7ff8-413c-9940-978f4b600778\") " Nov 03 01:36:33.896737 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:33.896231 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0c12073-7ff8-413c-9940-978f4b600778-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0c12073-7ff8-413c-9940-978f4b600778-istiod-ca-cert\") pod \"e0c12073-7ff8-413c-9940-978f4b600778\" (UID: \"e0c12073-7ff8-413c-9940-978f4b600778\") " Nov 03 01:36:33.896737 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:33.896254 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e0c12073-7ff8-413c-9940-978f4b600778-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0c12073-7ff8-413c-9940-978f4b600778-istio-podinfo\") pod \"e0c12073-7ff8-413c-9940-978f4b600778\" (UID: \"e0c12073-7ff8-413c-9940-978f4b600778\") " Nov 03 01:36:33.896737 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:33.896295 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0c12073-7ff8-413c-9940-978f4b600778-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0c12073-7ff8-413c-9940-978f4b600778-credential-socket\") pod \"e0c12073-7ff8-413c-9940-978f4b600778\" (UID: \"e0c12073-7ff8-413c-9940-978f4b600778\") " Nov 03 01:36:33.896737 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:33.896575 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0c12073-7ff8-413c-9940-978f4b600778-istiod-ca-cert" pod "e0c12073-7ff8-413c-9940-978f4b600778" (UID: "e0c12073-7ff8-413c-9940-978f4b600778"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:36:33.896997 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:33.896766 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0c12073-7ff8-413c-9940-978f4b600778-istio-data" pod "e0c12073-7ff8-413c-9940-978f4b600778" (UID: "e0c12073-7ff8-413c-9940-978f4b600778"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:36:33.896997 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:33.896781 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0c12073-7ff8-413c-9940-978f4b600778-credential-socket" pod "e0c12073-7ff8-413c-9940-978f4b600778" (UID: "e0c12073-7ff8-413c-9940-978f4b600778"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:36:33.896997 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:33.896822 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0c12073-7ff8-413c-9940-978f4b600778-workload-certs" pod "e0c12073-7ff8-413c-9940-978f4b600778" (UID: "e0c12073-7ff8-413c-9940-978f4b600778"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:36:33.897175 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:33.897006 4421 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0c12073-7ff8-413c-9940-978f4b600778-istiod-ca-cert\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:36:33.897175 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:33.897022 4421 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0c12073-7ff8-413c-9940-978f4b600778-credential-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:36:33.897175 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:33.897031 4421 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0c12073-7ff8-413c-9940-978f4b600778-workload-certs\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:36:33.897175 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:33.897041 4421 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0c12073-7ff8-413c-9940-978f4b600778-istio-data\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:36:33.897484 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:33.897253 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0c12073-7ff8-413c-9940-978f4b600778-workload-socket" pod "e0c12073-7ff8-413c-9940-978f4b600778" (UID: "e0c12073-7ff8-413c-9940-978f4b600778"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:36:33.898818 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:33.898789 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0c12073-7ff8-413c-9940-978f4b600778-kube-api-access-96mbm" pod "e0c12073-7ff8-413c-9940-978f4b600778" (UID: "e0c12073-7ff8-413c-9940-978f4b600778"). InnerVolumeSpecName "kube-api-access-96mbm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:36:33.898942 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:33.898917 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0c12073-7ff8-413c-9940-978f4b600778-istio-envoy" pod "e0c12073-7ff8-413c-9940-978f4b600778" (UID: "e0c12073-7ff8-413c-9940-978f4b600778"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:36:33.899013 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:33.898919 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e0c12073-7ff8-413c-9940-978f4b600778-istio-podinfo" pod "e0c12073-7ff8-413c-9940-978f4b600778" (UID: "e0c12073-7ff8-413c-9940-978f4b600778"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:36:33.899457 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:33.899438 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0c12073-7ff8-413c-9940-978f4b600778-istio-token" pod "e0c12073-7ff8-413c-9940-978f4b600778" (UID: "e0c12073-7ff8-413c-9940-978f4b600778"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:36:33.998769 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:33.998739 4421 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0c12073-7ff8-413c-9940-978f4b600778-istio-podinfo\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:36:33.998769 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:33.998771 4421 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0c12073-7ff8-413c-9940-978f4b600778-istio-envoy\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:36:33.999000 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:33.998788 4421 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0c12073-7ff8-413c-9940-978f4b600778-workload-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:36:33.999000 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:33.998802 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96mbm\" (UniqueName: \"kubernetes.io/projected/e0c12073-7ff8-413c-9940-978f4b600778-kube-api-access-96mbm\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:36:33.999000 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:33.998817 4421 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0c12073-7ff8-413c-9940-978f4b600778-istio-token\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:36:34.190488 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:34.190453 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-5dab6a-openshift-default-6ff495656c-bxn8s"] Nov 03 01:36:34.190811 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:34.190785 4421 patch_prober.go:29] interesting pod/e2e-shared-a-w2-531447-openshift-default-7995d46767-c9sjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.101:15021/healthz/ready\": dial tcp 10.134.1.101:15021: connect: connection refused" start-of-body= Nov 03 01:36:34.190910 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:34.190851 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-c9sjv" podUID="73bae713-0108-4120-8486-a35dbf68b0c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.101:15021/healthz/ready\": dial tcp 10.134.1.101:15021: connect: connection refused" Nov 03 01:36:34.196116 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:34.196089 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-5dab6a-openshift-default-6ff495656c-bxn8s"] Nov 03 01:36:35.214304 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:35.213341 4421 patch_prober.go:29] interesting pod/e2e-shared-a-w2-531447-openshift-default-7995d46767-c9sjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.101:15021/healthz/ready\": dial tcp 10.134.1.101:15021: connect: connection refused" start-of-body= Nov 03 01:36:35.214304 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:35.213424 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-c9sjv" podUID="73bae713-0108-4120-8486-a35dbf68b0c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.101:15021/healthz/ready\": dial tcp 10.134.1.101:15021: connect: connection refused" Nov 03 01:36:36.008294 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:36.007514 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/simulator-w3-83755a-kserve-77d6664559-psqtg" podUID="b0a4c234-af58-4da0-a1a4-e1f167953f6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Nov 03 01:36:36.049332 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:36.032524 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0c12073-7ff8-413c-9940-978f4b600778" path="/var/lib/kubelet/pods/e0c12073-7ff8-413c-9940-978f4b600778/volumes" Nov 03 01:36:36.194710 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:36.194672 4421 patch_prober.go:29] interesting pod/e2e-shared-a-w2-531447-openshift-default-7995d46767-c9sjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.101:15021/healthz/ready\": dial tcp 10.134.1.101:15021: connect: connection refused" start-of-body= Nov 03 01:36:36.194999 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:36.194969 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-c9sjv" podUID="73bae713-0108-4120-8486-a35dbf68b0c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.101:15021/healthz/ready\": dial tcp 10.134.1.101:15021: connect: connection refused" Nov 03 01:36:36.391896 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:36.391783 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/simulator-w2-f7680d-kserve-85bdc86cd6-8gxrg" podUID="2bdcf521-cebd-4d6b-8554-1b60db8b0f3c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 03 01:36:36.667368 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:36.666364 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/premium-w3-83755a-kserve-7c9d66988b-lt4xd" podUID="c0c6b02e-2ecb-428e-a13d-be6d7cd1ba39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:36:36.848066 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:36.846963 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-650be0/premium-w4-650be0-kserve-7bf8555dfb-z97n5" podUID="40601dc5-808f-456f-826f-6e0c35966ada" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 01:36:37.191126 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:37.191086 4421 patch_prober.go:29] interesting pod/e2e-shared-a-w2-531447-openshift-default-7995d46767-c9sjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.101:15021/healthz/ready\": dial tcp 10.134.1.101:15021: connect: connection refused" start-of-body= Nov 03 01:36:37.191488 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:37.191453 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-c9sjv" podUID="73bae713-0108-4120-8486-a35dbf68b0c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.101:15021/healthz/ready\": dial tcp 10.134.1.101:15021: connect: connection refused" Nov 03 01:36:38.191741 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:38.191700 4421 patch_prober.go:29] interesting pod/e2e-shared-a-w2-531447-openshift-default-7995d46767-c9sjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.101:15021/healthz/ready\": dial tcp 10.134.1.101:15021: connect: connection refused" start-of-body= Nov 03 01:36:38.192414 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:38.192343 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-c9sjv" podUID="73bae713-0108-4120-8486-a35dbf68b0c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.101:15021/healthz/ready\": dial tcp 10.134.1.101:15021: connect: connection refused" Nov 03 01:36:39.191434 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:39.191388 4421 patch_prober.go:29] interesting pod/e2e-shared-a-w2-531447-openshift-default-7995d46767-c9sjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.101:15021/healthz/ready\": dial tcp 10.134.1.101:15021: connect: connection refused" start-of-body= Nov 03 01:36:39.191656 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:39.191462 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-c9sjv" podUID="73bae713-0108-4120-8486-a35dbf68b0c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.101:15021/healthz/ready\": dial tcp 10.134.1.101:15021: connect: connection refused" Nov 03 01:36:39.932553 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:39.932521 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w4-6c0ec0/sub-test-model-6c0ec0-kserve-666b858575-xpltp" Nov 03 01:36:39.945915 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:39.945869 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w4-6c0ec0/sub-test-model-6c0ec0-kserve-666b858575-xpltp" Nov 03 01:36:40.127907 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:40.127873 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-6c0ec0/sub-test-model-6c0ec0-kserve-666b858575-xpltp" Nov 03 01:36:40.200033 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:40.198172 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-c9sjv" Nov 03 01:36:40.200033 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:40.199633 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-c9sjv" Nov 03 01:36:40.201541 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:40.201518 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-c9sjv" Nov 03 01:36:44.671924 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:36:44.671880 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 01:36:46.875680 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:46.875646 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-jcsn2"] Nov 03 01:36:46.876706 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:46.876679 4421 cpu_manager.go:414] "containerMap: removing container" podUID="e0c12073-7ff8-413c-9940-978f4b600778" containerName="istio-proxy" Nov 03 01:36:46.876706 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:46.876701 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0c12073-7ff8-413c-9940-978f4b600778" containerName="istio-proxy" Nov 03 01:36:46.876925 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:46.876731 4421 cpu_manager.go:414] "containerMap: removing container" podUID="e0c12073-7ff8-413c-9940-978f4b600778" containerName="istio-proxy" Nov 03 01:36:46.876925 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:46.876737 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0c12073-7ff8-413c-9940-978f4b600778" containerName="istio-proxy" Nov 03 01:36:46.876925 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:46.876920 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0c12073-7ff8-413c-9940-978f4b600778" containerName="istio-proxy" Nov 03 01:36:46.882590 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:46.882557 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-jcsn2" Nov 03 01:36:46.894300 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:46.894253 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-jcsn2"] Nov 03 01:36:46.913107 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:46.912566 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2x8c\" (UniqueName: \"kubernetes.io/projected/2233c8fb-b2bc-4987-82de-c06a99a7d396-kube-api-access-g2x8c\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-jcsn2\" (UID: \"2233c8fb-b2bc-4987-82de-c06a99a7d396\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-jcsn2" Nov 03 01:36:46.913107 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:46.912644 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2233c8fb-b2bc-4987-82de-c06a99a7d396-workload-certs\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-jcsn2\" (UID: \"2233c8fb-b2bc-4987-82de-c06a99a7d396\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-jcsn2" Nov 03 01:36:46.913107 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:46.912761 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2233c8fb-b2bc-4987-82de-c06a99a7d396-istio-data\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-jcsn2\" (UID: \"2233c8fb-b2bc-4987-82de-c06a99a7d396\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-jcsn2" Nov 03 01:36:46.913107 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:46.912806 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2233c8fb-b2bc-4987-82de-c06a99a7d396-istio-token\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-jcsn2\" (UID: \"2233c8fb-b2bc-4987-82de-c06a99a7d396\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-jcsn2" Nov 03 01:36:46.913107 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:46.912837 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2233c8fb-b2bc-4987-82de-c06a99a7d396-workload-socket\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-jcsn2\" (UID: \"2233c8fb-b2bc-4987-82de-c06a99a7d396\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-jcsn2" Nov 03 01:36:46.913107 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:46.912942 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2233c8fb-b2bc-4987-82de-c06a99a7d396-credential-socket\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-jcsn2\" (UID: \"2233c8fb-b2bc-4987-82de-c06a99a7d396\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-jcsn2" Nov 03 01:36:46.913107 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:46.913008 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2233c8fb-b2bc-4987-82de-c06a99a7d396-istiod-ca-cert\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-jcsn2\" (UID: \"2233c8fb-b2bc-4987-82de-c06a99a7d396\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-jcsn2" Nov 03 01:36:46.913107 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:46.913060 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2233c8fb-b2bc-4987-82de-c06a99a7d396-istio-podinfo\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-jcsn2\" (UID: \"2233c8fb-b2bc-4987-82de-c06a99a7d396\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-jcsn2" Nov 03 01:36:46.913107 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:46.913100 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2233c8fb-b2bc-4987-82de-c06a99a7d396-istio-envoy\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-jcsn2\" (UID: \"2233c8fb-b2bc-4987-82de-c06a99a7d396\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-jcsn2" Nov 03 01:36:47.014860 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:47.014820 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2233c8fb-b2bc-4987-82de-c06a99a7d396-istio-envoy\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-jcsn2\" (UID: \"2233c8fb-b2bc-4987-82de-c06a99a7d396\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-jcsn2" Nov 03 01:36:47.015136 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:47.014924 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2x8c\" (UniqueName: \"kubernetes.io/projected/2233c8fb-b2bc-4987-82de-c06a99a7d396-kube-api-access-g2x8c\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-jcsn2\" (UID: \"2233c8fb-b2bc-4987-82de-c06a99a7d396\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-jcsn2" Nov 03 01:36:47.015136 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:47.014969 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2233c8fb-b2bc-4987-82de-c06a99a7d396-workload-certs\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-jcsn2\" (UID: \"2233c8fb-b2bc-4987-82de-c06a99a7d396\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-jcsn2" Nov 03 01:36:47.015136 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:47.015069 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2233c8fb-b2bc-4987-82de-c06a99a7d396-istio-data\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-jcsn2\" (UID: \"2233c8fb-b2bc-4987-82de-c06a99a7d396\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-jcsn2" Nov 03 01:36:47.015136 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:47.015110 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2233c8fb-b2bc-4987-82de-c06a99a7d396-istio-token\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-jcsn2\" (UID: \"2233c8fb-b2bc-4987-82de-c06a99a7d396\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-jcsn2" Nov 03 01:36:47.015414 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:47.015139 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2233c8fb-b2bc-4987-82de-c06a99a7d396-workload-socket\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-jcsn2\" (UID: \"2233c8fb-b2bc-4987-82de-c06a99a7d396\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-jcsn2" Nov 03 01:36:47.015414 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:47.015232 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2233c8fb-b2bc-4987-82de-c06a99a7d396-credential-socket\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-jcsn2\" (UID: \"2233c8fb-b2bc-4987-82de-c06a99a7d396\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-jcsn2" Nov 03 01:36:47.015414 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:47.015314 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2233c8fb-b2bc-4987-82de-c06a99a7d396-istiod-ca-cert\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-jcsn2\" (UID: \"2233c8fb-b2bc-4987-82de-c06a99a7d396\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-jcsn2" Nov 03 01:36:47.015414 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:47.015370 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2233c8fb-b2bc-4987-82de-c06a99a7d396-istio-podinfo\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-jcsn2\" (UID: \"2233c8fb-b2bc-4987-82de-c06a99a7d396\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-jcsn2" Nov 03 01:36:47.015609 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:47.015439 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2233c8fb-b2bc-4987-82de-c06a99a7d396-workload-certs\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-jcsn2\" (UID: \"2233c8fb-b2bc-4987-82de-c06a99a7d396\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-jcsn2" Nov 03 01:36:47.015609 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:47.015495 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2233c8fb-b2bc-4987-82de-c06a99a7d396-istio-data\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-jcsn2\" (UID: \"2233c8fb-b2bc-4987-82de-c06a99a7d396\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-jcsn2" Nov 03 01:36:47.015609 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:47.015530 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2233c8fb-b2bc-4987-82de-c06a99a7d396-workload-socket\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-jcsn2\" (UID: \"2233c8fb-b2bc-4987-82de-c06a99a7d396\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-jcsn2" Nov 03 01:36:47.015740 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:47.015649 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2233c8fb-b2bc-4987-82de-c06a99a7d396-credential-socket\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-jcsn2\" (UID: \"2233c8fb-b2bc-4987-82de-c06a99a7d396\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-jcsn2" Nov 03 01:36:47.015947 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:47.015919 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2233c8fb-b2bc-4987-82de-c06a99a7d396-istiod-ca-cert\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-jcsn2\" (UID: \"2233c8fb-b2bc-4987-82de-c06a99a7d396\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-jcsn2" Nov 03 01:36:47.017424 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:47.017397 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2233c8fb-b2bc-4987-82de-c06a99a7d396-istio-envoy\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-jcsn2\" (UID: \"2233c8fb-b2bc-4987-82de-c06a99a7d396\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-jcsn2" Nov 03 01:36:47.017827 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:47.017810 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2233c8fb-b2bc-4987-82de-c06a99a7d396-istio-podinfo\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-jcsn2\" (UID: \"2233c8fb-b2bc-4987-82de-c06a99a7d396\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-jcsn2" Nov 03 01:36:47.022865 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:47.022841 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2x8c\" (UniqueName: \"kubernetes.io/projected/2233c8fb-b2bc-4987-82de-c06a99a7d396-kube-api-access-g2x8c\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-jcsn2\" (UID: \"2233c8fb-b2bc-4987-82de-c06a99a7d396\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-jcsn2" Nov 03 01:36:47.022959 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:47.022848 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2233c8fb-b2bc-4987-82de-c06a99a7d396-istio-token\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-jcsn2\" (UID: \"2233c8fb-b2bc-4987-82de-c06a99a7d396\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-jcsn2" Nov 03 01:36:47.215872 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:47.215767 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-jcsn2" Nov 03 01:36:47.418287 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:47.418243 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w4-4a697f/sub-test-model-4a697f-kserve-588b784c4c-j5qfn"] Nov 03 01:36:47.419608 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:47.419583 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0c12073-7ff8-413c-9940-978f4b600778" containerName="istio-proxy" Nov 03 01:36:47.432039 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:47.430498 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-4a697f/sub-test-model-4a697f-kserve-588b784c4c-j5qfn" Nov 03 01:36:47.432039 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:47.430755 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-4a697f/sub-test-model-4a697f-kserve-588b784c4c-j5qfn"] Nov 03 01:36:47.435783 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:47.435754 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-4a697f\"/\"kube-root-ca.crt\"" Nov 03 01:36:47.435783 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:47.435772 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-4a697f\"/\"openshift-service-ca.crt\"" Nov 03 01:36:47.437016 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:47.436884 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-4a697f\"/\"default-dockercfg-q8vgn\"" Nov 03 01:36:47.437016 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:47.436998 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-4a697f\"/\"sub-test-model-4a697f-kserve-self-signed-certs\"" Nov 03 01:36:47.512167 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:47.512135 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-jcsn2"] Nov 03 01:36:47.513114 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:36:47.513083 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2233c8fb_b2bc_4987_82de_c06a99a7d396.slice/crio-b5ff7e40da5f9d7ff6e98b918152392a64c534a8d5d1648bbd3cda08310e360a WatchSource:0}: Error finding container b5ff7e40da5f9d7ff6e98b918152392a64c534a8d5d1648bbd3cda08310e360a: Status 404 returned error can't find the container with id b5ff7e40da5f9d7ff6e98b918152392a64c534a8d5d1648bbd3cda08310e360a Nov 03 01:36:47.516261 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:47.516225 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:36:47.516358 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:47.516340 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:36:47.516925 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:47.516372 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:36:47.534700 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:47.534672 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0494e957-033d-42d6-8b56-ebca9f26b75b-dshm\") pod \"sub-test-model-4a697f-kserve-588b784c4c-j5qfn\" (UID: \"0494e957-033d-42d6-8b56-ebca9f26b75b\") " pod="ai-tenant-e2e-shared-b-w4-4a697f/sub-test-model-4a697f-kserve-588b784c4c-j5qfn" Nov 03 01:36:47.534820 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:47.534722 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0494e957-033d-42d6-8b56-ebca9f26b75b-home\") pod \"sub-test-model-4a697f-kserve-588b784c4c-j5qfn\" (UID: \"0494e957-033d-42d6-8b56-ebca9f26b75b\") " pod="ai-tenant-e2e-shared-b-w4-4a697f/sub-test-model-4a697f-kserve-588b784c4c-j5qfn" Nov 03 01:36:47.534874 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:47.534844 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0494e957-033d-42d6-8b56-ebca9f26b75b-model-cache\") pod \"sub-test-model-4a697f-kserve-588b784c4c-j5qfn\" (UID: \"0494e957-033d-42d6-8b56-ebca9f26b75b\") " pod="ai-tenant-e2e-shared-b-w4-4a697f/sub-test-model-4a697f-kserve-588b784c4c-j5qfn" Nov 03 01:36:47.534924 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:47.534892 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0494e957-033d-42d6-8b56-ebca9f26b75b-tmp-dir\") pod \"sub-test-model-4a697f-kserve-588b784c4c-j5qfn\" (UID: \"0494e957-033d-42d6-8b56-ebca9f26b75b\") " pod="ai-tenant-e2e-shared-b-w4-4a697f/sub-test-model-4a697f-kserve-588b784c4c-j5qfn" Nov 03 01:36:47.535003 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:47.534982 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0494e957-033d-42d6-8b56-ebca9f26b75b-tls-certs\") pod \"sub-test-model-4a697f-kserve-588b784c4c-j5qfn\" (UID: \"0494e957-033d-42d6-8b56-ebca9f26b75b\") " pod="ai-tenant-e2e-shared-b-w4-4a697f/sub-test-model-4a697f-kserve-588b784c4c-j5qfn" Nov 03 01:36:47.535184 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:47.535168 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2jxc\" (UniqueName: \"kubernetes.io/projected/0494e957-033d-42d6-8b56-ebca9f26b75b-kube-api-access-r2jxc\") pod \"sub-test-model-4a697f-kserve-588b784c4c-j5qfn\" (UID: \"0494e957-033d-42d6-8b56-ebca9f26b75b\") " pod="ai-tenant-e2e-shared-b-w4-4a697f/sub-test-model-4a697f-kserve-588b784c4c-j5qfn" Nov 03 01:36:47.649295 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:47.648287 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2jxc\" (UniqueName: \"kubernetes.io/projected/0494e957-033d-42d6-8b56-ebca9f26b75b-kube-api-access-r2jxc\") pod \"sub-test-model-4a697f-kserve-588b784c4c-j5qfn\" (UID: \"0494e957-033d-42d6-8b56-ebca9f26b75b\") " pod="ai-tenant-e2e-shared-b-w4-4a697f/sub-test-model-4a697f-kserve-588b784c4c-j5qfn" Nov 03 01:36:47.649295 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:47.648444 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0494e957-033d-42d6-8b56-ebca9f26b75b-dshm\") pod \"sub-test-model-4a697f-kserve-588b784c4c-j5qfn\" (UID: \"0494e957-033d-42d6-8b56-ebca9f26b75b\") " pod="ai-tenant-e2e-shared-b-w4-4a697f/sub-test-model-4a697f-kserve-588b784c4c-j5qfn" Nov 03 01:36:47.649295 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:47.648480 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0494e957-033d-42d6-8b56-ebca9f26b75b-home\") pod \"sub-test-model-4a697f-kserve-588b784c4c-j5qfn\" (UID: \"0494e957-033d-42d6-8b56-ebca9f26b75b\") " pod="ai-tenant-e2e-shared-b-w4-4a697f/sub-test-model-4a697f-kserve-588b784c4c-j5qfn" Nov 03 01:36:47.649295 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:47.648573 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0494e957-033d-42d6-8b56-ebca9f26b75b-model-cache\") pod \"sub-test-model-4a697f-kserve-588b784c4c-j5qfn\" (UID: \"0494e957-033d-42d6-8b56-ebca9f26b75b\") " pod="ai-tenant-e2e-shared-b-w4-4a697f/sub-test-model-4a697f-kserve-588b784c4c-j5qfn" Nov 03 01:36:47.649295 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:47.648624 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0494e957-033d-42d6-8b56-ebca9f26b75b-tmp-dir\") pod \"sub-test-model-4a697f-kserve-588b784c4c-j5qfn\" (UID: \"0494e957-033d-42d6-8b56-ebca9f26b75b\") " pod="ai-tenant-e2e-shared-b-w4-4a697f/sub-test-model-4a697f-kserve-588b784c4c-j5qfn" Nov 03 01:36:47.649295 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:47.648657 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0494e957-033d-42d6-8b56-ebca9f26b75b-tls-certs\") pod \"sub-test-model-4a697f-kserve-588b784c4c-j5qfn\" (UID: \"0494e957-033d-42d6-8b56-ebca9f26b75b\") " pod="ai-tenant-e2e-shared-b-w4-4a697f/sub-test-model-4a697f-kserve-588b784c4c-j5qfn" Nov 03 01:36:47.654296 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:47.650614 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0494e957-033d-42d6-8b56-ebca9f26b75b-model-cache\") pod \"sub-test-model-4a697f-kserve-588b784c4c-j5qfn\" (UID: \"0494e957-033d-42d6-8b56-ebca9f26b75b\") " pod="ai-tenant-e2e-shared-b-w4-4a697f/sub-test-model-4a697f-kserve-588b784c4c-j5qfn" Nov 03 01:36:47.654296 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:47.650680 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0494e957-033d-42d6-8b56-ebca9f26b75b-tmp-dir\") pod \"sub-test-model-4a697f-kserve-588b784c4c-j5qfn\" (UID: \"0494e957-033d-42d6-8b56-ebca9f26b75b\") " pod="ai-tenant-e2e-shared-b-w4-4a697f/sub-test-model-4a697f-kserve-588b784c4c-j5qfn" Nov 03 01:36:47.654296 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:47.650858 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0494e957-033d-42d6-8b56-ebca9f26b75b-home\") pod \"sub-test-model-4a697f-kserve-588b784c4c-j5qfn\" (UID: \"0494e957-033d-42d6-8b56-ebca9f26b75b\") " pod="ai-tenant-e2e-shared-b-w4-4a697f/sub-test-model-4a697f-kserve-588b784c4c-j5qfn" Nov 03 01:36:47.685362 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:47.679819 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0494e957-033d-42d6-8b56-ebca9f26b75b-dshm\") pod \"sub-test-model-4a697f-kserve-588b784c4c-j5qfn\" (UID: \"0494e957-033d-42d6-8b56-ebca9f26b75b\") " pod="ai-tenant-e2e-shared-b-w4-4a697f/sub-test-model-4a697f-kserve-588b784c4c-j5qfn" Nov 03 01:36:47.685362 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:47.684245 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0494e957-033d-42d6-8b56-ebca9f26b75b-tls-certs\") pod \"sub-test-model-4a697f-kserve-588b784c4c-j5qfn\" (UID: \"0494e957-033d-42d6-8b56-ebca9f26b75b\") " pod="ai-tenant-e2e-shared-b-w4-4a697f/sub-test-model-4a697f-kserve-588b784c4c-j5qfn" Nov 03 01:36:47.685362 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:47.684373 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2jxc\" (UniqueName: \"kubernetes.io/projected/0494e957-033d-42d6-8b56-ebca9f26b75b-kube-api-access-r2jxc\") pod \"sub-test-model-4a697f-kserve-588b784c4c-j5qfn\" (UID: \"0494e957-033d-42d6-8b56-ebca9f26b75b\") " pod="ai-tenant-e2e-shared-b-w4-4a697f/sub-test-model-4a697f-kserve-588b784c4c-j5qfn" Nov 03 01:36:47.749198 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:47.749163 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-4a697f/sub-test-model-4a697f-kserve-588b784c4c-j5qfn" Nov 03 01:36:48.037327 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:48.037218 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-4a697f/sub-test-model-4a697f-kserve-588b784c4c-j5qfn"] Nov 03 01:36:48.184341 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:48.184305 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-jcsn2" event={"ID":"2233c8fb-b2bc-4987-82de-c06a99a7d396","Type":"ContainerStarted","Data":"0fdac1b8e75bb68607bd72e909890c6790f7ac9fe81871b6e7c0931a585e1c34"} Nov 03 01:36:48.184724 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:48.184380 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-jcsn2" event={"ID":"2233c8fb-b2bc-4987-82de-c06a99a7d396","Type":"ContainerStarted","Data":"b5ff7e40da5f9d7ff6e98b918152392a64c534a8d5d1648bbd3cda08310e360a"} Nov 03 01:36:48.186488 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:48.186425 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-4a697f/sub-test-model-4a697f-kserve-588b784c4c-j5qfn" event={"ID":"0494e957-033d-42d6-8b56-ebca9f26b75b","Type":"ContainerStarted","Data":"a724d2c9a38ae879b14a60f1249832a5ce6f76d26eb69f7482827d1053d18901"} Nov 03 01:36:48.186488 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:48.186478 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-4a697f/sub-test-model-4a697f-kserve-588b784c4c-j5qfn" event={"ID":"0494e957-033d-42d6-8b56-ebca9f26b75b","Type":"ContainerStarted","Data":"e038a6c2cab1da5bcd977a7b1637d72e3922bbd0dc37a6eb6834a7d88eaba461"} Nov 03 01:36:48.208037 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:48.207965 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-jcsn2" podStartSLOduration=2.207944956 podStartE2EDuration="2.207944956s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:36:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:36:48.202770077 +0000 UTC m=+3350.373773195" watchObservedRunningTime="2026-11-03 01:36:48.207944956 +0000 UTC m=+3350.378948052" Nov 03 01:36:48.221912 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:48.221872 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-jcsn2" Nov 03 01:36:48.223747 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:48.223611 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w4-4a697f/sub-test-model-4a697f-kserve-588b784c4c-j5qfn" podStartSLOduration=1.223592626 podStartE2EDuration="1.223592626s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:36:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:36:48.221940693 +0000 UTC m=+3350.392943802" watchObservedRunningTime="2026-11-03 01:36:48.223592626 +0000 UTC m=+3350.394595724" Nov 03 01:36:48.225670 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:48.225635 4421 patch_prober.go:29] interesting pod/e2e-shared-a-w2-531447-openshift-default-7995d46767-jcsn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.102:15021/healthz/ready\": dial tcp 10.134.1.102:15021: connect: connection refused" start-of-body= Nov 03 01:36:48.225816 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:48.225690 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-jcsn2" podUID="2233c8fb-b2bc-4987-82de-c06a99a7d396" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.102:15021/healthz/ready\": dial tcp 10.134.1.102:15021: connect: connection refused" Nov 03 01:36:48.750444 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:48.750384 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-4a697f/sub-test-model-4a697f-kserve-588b784c4c-j5qfn" Nov 03 01:36:49.224931 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:49.224826 4421 patch_prober.go:29] interesting pod/e2e-shared-a-w2-531447-openshift-default-7995d46767-jcsn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.102:15021/healthz/ready\": dial tcp 10.134.1.102:15021: connect: connection refused" start-of-body= Nov 03 01:36:49.225629 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:49.225589 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-jcsn2" podUID="2233c8fb-b2bc-4987-82de-c06a99a7d396" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.102:15021/healthz/ready\": dial tcp 10.134.1.102:15021: connect: connection refused" Nov 03 01:36:50.234665 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:50.234624 4421 patch_prober.go:29] interesting pod/e2e-shared-a-w2-531447-openshift-default-7995d46767-jcsn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.102:15021/healthz/ready\": dial tcp 10.134.1.102:15021: connect: connection refused" start-of-body= Nov 03 01:36:50.235248 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:50.235215 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-jcsn2" podUID="2233c8fb-b2bc-4987-82de-c06a99a7d396" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.102:15021/healthz/ready\": dial tcp 10.134.1.102:15021: connect: connection refused" Nov 03 01:36:51.221948 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:51.221907 4421 patch_prober.go:29] interesting pod/e2e-shared-a-w2-531447-openshift-default-7995d46767-jcsn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.102:15021/healthz/ready\": dial tcp 10.134.1.102:15021: connect: connection refused" start-of-body= Nov 03 01:36:51.230446 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:51.222155 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-jcsn2" podUID="2233c8fb-b2bc-4987-82de-c06a99a7d396" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.102:15021/healthz/ready\": dial tcp 10.134.1.102:15021: connect: connection refused" Nov 03 01:36:52.230301 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:52.223010 4421 patch_prober.go:29] interesting pod/e2e-shared-a-w2-531447-openshift-default-7995d46767-jcsn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.102:15021/healthz/ready\": dial tcp 10.134.1.102:15021: connect: connection refused" start-of-body= Nov 03 01:36:52.230301 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:52.223080 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-jcsn2" podUID="2233c8fb-b2bc-4987-82de-c06a99a7d396" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.102:15021/healthz/ready\": dial tcp 10.134.1.102:15021: connect: connection refused" Nov 03 01:36:53.224671 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:53.224631 4421 patch_prober.go:29] interesting pod/e2e-shared-a-w2-531447-openshift-default-7995d46767-jcsn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.102:15021/healthz/ready\": dial tcp 10.134.1.102:15021: connect: connection refused" start-of-body= Nov 03 01:36:53.225027 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:53.224993 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-jcsn2" podUID="2233c8fb-b2bc-4987-82de-c06a99a7d396" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.102:15021/healthz/ready\": dial tcp 10.134.1.102:15021: connect: connection refused" Nov 03 01:36:54.252947 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:54.252914 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-jcsn2" Nov 03 01:36:54.254099 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:54.254071 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-jcsn2" Nov 03 01:36:54.255746 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:54.255725 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-jcsn2" Nov 03 01:36:54.755528 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:54.754560 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-be59df-7499f75b58-xgcq7"] Nov 03 01:36:54.755528 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:54.754890 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-be59df-7499f75b58-xgcq7" podUID="5c6b86d5-52d4-449c-90c4-3d5bad417c00" containerName="maas-api" containerID="cri-o://b91771ea32f82017c0e920b30d0a5ba8dbd437c3211196c47149504a52d5597e" gracePeriod=30 Nov 03 01:36:55.215665 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:55.215632 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-be59df-7499f75b58-xgcq7" Nov 03 01:36:55.275338 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:55.275256 4421 generic.go:356] "Generic (PLEG): container finished" podID="5c6b86d5-52d4-449c-90c4-3d5bad417c00" containerID="b91771ea32f82017c0e920b30d0a5ba8dbd437c3211196c47149504a52d5597e" exitCode=0 Nov 03 01:36:55.275338 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:55.275327 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-be59df-7499f75b58-xgcq7" event={"ID":"5c6b86d5-52d4-449c-90c4-3d5bad417c00","Type":"ContainerDied","Data":"b91771ea32f82017c0e920b30d0a5ba8dbd437c3211196c47149504a52d5597e"} Nov 03 01:36:55.275919 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:55.275382 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-be59df-7499f75b58-xgcq7" event={"ID":"5c6b86d5-52d4-449c-90c4-3d5bad417c00","Type":"ContainerDied","Data":"8eb1e483271feb22c2a9e2c3854c5fed00c5f77b6b34863c9b755a8ba65e6db1"} Nov 03 01:36:55.275919 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:55.275321 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-be59df-7499f75b58-xgcq7" Nov 03 01:36:55.275919 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:55.275405 4421 scope.go:122] "RemoveContainer" containerID="b91771ea32f82017c0e920b30d0a5ba8dbd437c3211196c47149504a52d5597e" Nov 03 01:36:55.290167 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:55.290143 4421 scope.go:122] "RemoveContainer" containerID="b91771ea32f82017c0e920b30d0a5ba8dbd437c3211196c47149504a52d5597e" Nov 03 01:36:55.290607 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:36:55.290581 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b91771ea32f82017c0e920b30d0a5ba8dbd437c3211196c47149504a52d5597e\": container with ID starting with b91771ea32f82017c0e920b30d0a5ba8dbd437c3211196c47149504a52d5597e not found: ID does not exist" containerID="b91771ea32f82017c0e920b30d0a5ba8dbd437c3211196c47149504a52d5597e" Nov 03 01:36:55.290742 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:55.290619 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b91771ea32f82017c0e920b30d0a5ba8dbd437c3211196c47149504a52d5597e"} err="failed to get container status \"b91771ea32f82017c0e920b30d0a5ba8dbd437c3211196c47149504a52d5597e\": rpc error: code = NotFound desc = could not find container \"b91771ea32f82017c0e920b30d0a5ba8dbd437c3211196c47149504a52d5597e\": container with ID starting with b91771ea32f82017c0e920b30d0a5ba8dbd437c3211196c47149504a52d5597e not found: ID does not exist" Nov 03 01:36:55.342485 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:55.342445 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c6b86d5-52d4-449c-90c4-3d5bad417c00-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c6b86d5-52d4-449c-90c4-3d5bad417c00-openshift-service-ca\") pod \"5c6b86d5-52d4-449c-90c4-3d5bad417c00\" (UID: \"5c6b86d5-52d4-449c-90c4-3d5bad417c00\") " Nov 03 01:36:55.342759 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:55.342673 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5c6b86d5-52d4-449c-90c4-3d5bad417c00-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c6b86d5-52d4-449c-90c4-3d5bad417c00-maas-api-tls\") pod \"5c6b86d5-52d4-449c-90c4-3d5bad417c00\" (UID: \"5c6b86d5-52d4-449c-90c4-3d5bad417c00\") " Nov 03 01:36:55.342863 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:55.342841 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c6b86d5-52d4-449c-90c4-3d5bad417c00-kube-api-access-p8fxd\" (UniqueName: \"kubernetes.io/projected/5c6b86d5-52d4-449c-90c4-3d5bad417c00-kube-api-access-p8fxd\") pod \"5c6b86d5-52d4-449c-90c4-3d5bad417c00\" (UID: \"5c6b86d5-52d4-449c-90c4-3d5bad417c00\") " Nov 03 01:36:55.342939 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:55.342906 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5c6b86d5-52d4-449c-90c4-3d5bad417c00-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c6b86d5-52d4-449c-90c4-3d5bad417c00-metrics-certs\") pod \"5c6b86d5-52d4-449c-90c4-3d5bad417c00\" (UID: \"5c6b86d5-52d4-449c-90c4-3d5bad417c00\") " Nov 03 01:36:55.343046 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:55.342941 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c6b86d5-52d4-449c-90c4-3d5bad417c00-openshift-service-ca" pod "5c6b86d5-52d4-449c-90c4-3d5bad417c00" (UID: "5c6b86d5-52d4-449c-90c4-3d5bad417c00"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:36:55.343584 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:55.343554 4421 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c6b86d5-52d4-449c-90c4-3d5bad417c00-openshift-service-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:36:55.345675 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:55.345640 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c6b86d5-52d4-449c-90c4-3d5bad417c00-kube-api-access-p8fxd" pod "5c6b86d5-52d4-449c-90c4-3d5bad417c00" (UID: "5c6b86d5-52d4-449c-90c4-3d5bad417c00"). InnerVolumeSpecName "kube-api-access-p8fxd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:36:55.345823 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:55.345751 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c6b86d5-52d4-449c-90c4-3d5bad417c00-maas-api-tls" pod "5c6b86d5-52d4-449c-90c4-3d5bad417c00" (UID: "5c6b86d5-52d4-449c-90c4-3d5bad417c00"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:36:55.346007 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:55.345967 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c6b86d5-52d4-449c-90c4-3d5bad417c00-metrics-certs" pod "5c6b86d5-52d4-449c-90c4-3d5bad417c00" (UID: "5c6b86d5-52d4-449c-90c4-3d5bad417c00"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:36:55.447597 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:55.446963 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c6b86d5-52d4-449c-90c4-3d5bad417c00-maas-api-tls\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:36:55.447597 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:55.447006 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8fxd\" (UniqueName: \"kubernetes.io/projected/5c6b86d5-52d4-449c-90c4-3d5bad417c00-kube-api-access-p8fxd\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:36:55.447597 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:55.447021 4421 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c6b86d5-52d4-449c-90c4-3d5bad417c00-metrics-certs\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:36:55.602574 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:55.602530 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-be59df-7499f75b58-xgcq7"] Nov 03 01:36:55.606402 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:55.606374 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-be59df-7499f75b58-xgcq7"] Nov 03 01:36:55.981873 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:55.981782 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c6b86d5-52d4-449c-90c4-3d5bad417c00" path="/var/lib/kubelet/pods/5c6b86d5-52d4-449c-90c4-3d5bad417c00/volumes" Nov 03 01:36:57.749358 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:57.749307 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w4-4a697f/sub-test-model-4a697f-kserve-588b784c4c-j5qfn" Nov 03 01:36:57.762409 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:57.762383 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w4-4a697f/sub-test-model-4a697f-kserve-588b784c4c-j5qfn" Nov 03 01:36:58.311225 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:36:58.311193 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-4a697f/sub-test-model-4a697f-kserve-588b784c4c-j5qfn" Nov 03 01:37:01.431019 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:01.430974 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-717217/rate-test-model-717217-kserve-74ddd7bbcc-4kxbf"] Nov 03 01:37:01.432204 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:01.432107 4421 cpu_manager.go:414] "containerMap: removing container" podUID="5c6b86d5-52d4-449c-90c4-3d5bad417c00" containerName="maas-api" Nov 03 01:37:01.432204 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:01.432131 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c6b86d5-52d4-449c-90c4-3d5bad417c00" containerName="maas-api" Nov 03 01:37:01.432462 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:01.432357 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c6b86d5-52d4-449c-90c4-3d5bad417c00" containerName="maas-api" Nov 03 01:37:01.437364 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:01.437342 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-717217/rate-test-model-717217-kserve-74ddd7bbcc-4kxbf" Nov 03 01:37:01.447529 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:01.447495 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-717217\"/\"kube-root-ca.crt\"" Nov 03 01:37:01.448343 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:01.448318 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-717217\"/\"openshift-service-ca.crt\"" Nov 03 01:37:01.455890 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:01.455853 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-717217/rate-test-model-717217-kserve-74ddd7bbcc-4kxbf"] Nov 03 01:37:01.472683 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:01.472635 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aee71e38-f4ab-40e6-a3ad-1fe79bf0c63f-model-cache\") pod \"rate-test-model-717217-kserve-74ddd7bbcc-4kxbf\" (UID: \"aee71e38-f4ab-40e6-a3ad-1fe79bf0c63f\") " pod="ai-tenant-e2e-shared-a-w3-717217/rate-test-model-717217-kserve-74ddd7bbcc-4kxbf" Nov 03 01:37:01.472871 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:01.472757 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aee71e38-f4ab-40e6-a3ad-1fe79bf0c63f-tls-certs\") pod \"rate-test-model-717217-kserve-74ddd7bbcc-4kxbf\" (UID: \"aee71e38-f4ab-40e6-a3ad-1fe79bf0c63f\") " pod="ai-tenant-e2e-shared-a-w3-717217/rate-test-model-717217-kserve-74ddd7bbcc-4kxbf" Nov 03 01:37:01.472871 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:01.472844 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aee71e38-f4ab-40e6-a3ad-1fe79bf0c63f-home\") pod \"rate-test-model-717217-kserve-74ddd7bbcc-4kxbf\" (UID: \"aee71e38-f4ab-40e6-a3ad-1fe79bf0c63f\") " pod="ai-tenant-e2e-shared-a-w3-717217/rate-test-model-717217-kserve-74ddd7bbcc-4kxbf" Nov 03 01:37:01.473015 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:01.472902 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aee71e38-f4ab-40e6-a3ad-1fe79bf0c63f-tmp-dir\") pod \"rate-test-model-717217-kserve-74ddd7bbcc-4kxbf\" (UID: \"aee71e38-f4ab-40e6-a3ad-1fe79bf0c63f\") " pod="ai-tenant-e2e-shared-a-w3-717217/rate-test-model-717217-kserve-74ddd7bbcc-4kxbf" Nov 03 01:37:01.473015 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:01.472963 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9bn9\" (UniqueName: \"kubernetes.io/projected/aee71e38-f4ab-40e6-a3ad-1fe79bf0c63f-kube-api-access-j9bn9\") pod \"rate-test-model-717217-kserve-74ddd7bbcc-4kxbf\" (UID: \"aee71e38-f4ab-40e6-a3ad-1fe79bf0c63f\") " pod="ai-tenant-e2e-shared-a-w3-717217/rate-test-model-717217-kserve-74ddd7bbcc-4kxbf" Nov 03 01:37:01.473156 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:01.473045 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aee71e38-f4ab-40e6-a3ad-1fe79bf0c63f-dshm\") pod \"rate-test-model-717217-kserve-74ddd7bbcc-4kxbf\" (UID: \"aee71e38-f4ab-40e6-a3ad-1fe79bf0c63f\") " pod="ai-tenant-e2e-shared-a-w3-717217/rate-test-model-717217-kserve-74ddd7bbcc-4kxbf" Nov 03 01:37:01.480346 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:01.477564 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-717217\"/\"rate-test-model-717217-kserve-self-signed-certs\"" Nov 03 01:37:01.480346 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:01.477646 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-717217\"/\"default-dockercfg-4z75p\"" Nov 03 01:37:01.574366 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:01.574331 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aee71e38-f4ab-40e6-a3ad-1fe79bf0c63f-home\") pod \"rate-test-model-717217-kserve-74ddd7bbcc-4kxbf\" (UID: \"aee71e38-f4ab-40e6-a3ad-1fe79bf0c63f\") " pod="ai-tenant-e2e-shared-a-w3-717217/rate-test-model-717217-kserve-74ddd7bbcc-4kxbf" Nov 03 01:37:01.574561 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:01.574380 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aee71e38-f4ab-40e6-a3ad-1fe79bf0c63f-tmp-dir\") pod \"rate-test-model-717217-kserve-74ddd7bbcc-4kxbf\" (UID: \"aee71e38-f4ab-40e6-a3ad-1fe79bf0c63f\") " pod="ai-tenant-e2e-shared-a-w3-717217/rate-test-model-717217-kserve-74ddd7bbcc-4kxbf" Nov 03 01:37:01.574561 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:01.574428 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9bn9\" (UniqueName: \"kubernetes.io/projected/aee71e38-f4ab-40e6-a3ad-1fe79bf0c63f-kube-api-access-j9bn9\") pod \"rate-test-model-717217-kserve-74ddd7bbcc-4kxbf\" (UID: \"aee71e38-f4ab-40e6-a3ad-1fe79bf0c63f\") " pod="ai-tenant-e2e-shared-a-w3-717217/rate-test-model-717217-kserve-74ddd7bbcc-4kxbf" Nov 03 01:37:01.574561 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:01.574490 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aee71e38-f4ab-40e6-a3ad-1fe79bf0c63f-dshm\") pod \"rate-test-model-717217-kserve-74ddd7bbcc-4kxbf\" (UID: \"aee71e38-f4ab-40e6-a3ad-1fe79bf0c63f\") " pod="ai-tenant-e2e-shared-a-w3-717217/rate-test-model-717217-kserve-74ddd7bbcc-4kxbf" Nov 03 01:37:01.574746 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:01.574609 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aee71e38-f4ab-40e6-a3ad-1fe79bf0c63f-model-cache\") pod \"rate-test-model-717217-kserve-74ddd7bbcc-4kxbf\" (UID: \"aee71e38-f4ab-40e6-a3ad-1fe79bf0c63f\") " pod="ai-tenant-e2e-shared-a-w3-717217/rate-test-model-717217-kserve-74ddd7bbcc-4kxbf" Nov 03 01:37:01.574746 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:01.574684 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aee71e38-f4ab-40e6-a3ad-1fe79bf0c63f-tls-certs\") pod \"rate-test-model-717217-kserve-74ddd7bbcc-4kxbf\" (UID: \"aee71e38-f4ab-40e6-a3ad-1fe79bf0c63f\") " pod="ai-tenant-e2e-shared-a-w3-717217/rate-test-model-717217-kserve-74ddd7bbcc-4kxbf" Nov 03 01:37:01.574897 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:01.574797 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aee71e38-f4ab-40e6-a3ad-1fe79bf0c63f-home\") pod \"rate-test-model-717217-kserve-74ddd7bbcc-4kxbf\" (UID: \"aee71e38-f4ab-40e6-a3ad-1fe79bf0c63f\") " pod="ai-tenant-e2e-shared-a-w3-717217/rate-test-model-717217-kserve-74ddd7bbcc-4kxbf" Nov 03 01:37:01.574996 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:01.574960 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aee71e38-f4ab-40e6-a3ad-1fe79bf0c63f-tmp-dir\") pod \"rate-test-model-717217-kserve-74ddd7bbcc-4kxbf\" (UID: \"aee71e38-f4ab-40e6-a3ad-1fe79bf0c63f\") " pod="ai-tenant-e2e-shared-a-w3-717217/rate-test-model-717217-kserve-74ddd7bbcc-4kxbf" Nov 03 01:37:01.575134 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:01.575114 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aee71e38-f4ab-40e6-a3ad-1fe79bf0c63f-model-cache\") pod \"rate-test-model-717217-kserve-74ddd7bbcc-4kxbf\" (UID: \"aee71e38-f4ab-40e6-a3ad-1fe79bf0c63f\") " pod="ai-tenant-e2e-shared-a-w3-717217/rate-test-model-717217-kserve-74ddd7bbcc-4kxbf" Nov 03 01:37:01.576901 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:01.576873 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aee71e38-f4ab-40e6-a3ad-1fe79bf0c63f-dshm\") pod \"rate-test-model-717217-kserve-74ddd7bbcc-4kxbf\" (UID: \"aee71e38-f4ab-40e6-a3ad-1fe79bf0c63f\") " pod="ai-tenant-e2e-shared-a-w3-717217/rate-test-model-717217-kserve-74ddd7bbcc-4kxbf" Nov 03 01:37:01.577201 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:01.577181 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aee71e38-f4ab-40e6-a3ad-1fe79bf0c63f-tls-certs\") pod \"rate-test-model-717217-kserve-74ddd7bbcc-4kxbf\" (UID: \"aee71e38-f4ab-40e6-a3ad-1fe79bf0c63f\") " pod="ai-tenant-e2e-shared-a-w3-717217/rate-test-model-717217-kserve-74ddd7bbcc-4kxbf" Nov 03 01:37:01.582501 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:01.582480 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9bn9\" (UniqueName: \"kubernetes.io/projected/aee71e38-f4ab-40e6-a3ad-1fe79bf0c63f-kube-api-access-j9bn9\") pod \"rate-test-model-717217-kserve-74ddd7bbcc-4kxbf\" (UID: \"aee71e38-f4ab-40e6-a3ad-1fe79bf0c63f\") " pod="ai-tenant-e2e-shared-a-w3-717217/rate-test-model-717217-kserve-74ddd7bbcc-4kxbf" Nov 03 01:37:01.766812 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:01.766772 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-717217/rate-test-model-717217-kserve-74ddd7bbcc-4kxbf" Nov 03 01:37:01.924933 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:01.924889 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-tlggp"] Nov 03 01:37:01.936699 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:01.936657 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-tlggp" Nov 03 01:37:01.937257 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:01.936659 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-tlggp"] Nov 03 01:37:01.983216 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:01.983173 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e32924b-f5e1-4ecc-8a7d-08647aebf8fd-istio-envoy\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-tlggp\" (UID: \"8e32924b-f5e1-4ecc-8a7d-08647aebf8fd\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-tlggp" Nov 03 01:37:01.983442 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:01.983231 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tghxw\" (UniqueName: \"kubernetes.io/projected/8e32924b-f5e1-4ecc-8a7d-08647aebf8fd-kube-api-access-tghxw\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-tlggp\" (UID: \"8e32924b-f5e1-4ecc-8a7d-08647aebf8fd\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-tlggp" Nov 03 01:37:01.983442 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:01.983421 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e32924b-f5e1-4ecc-8a7d-08647aebf8fd-workload-socket\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-tlggp\" (UID: \"8e32924b-f5e1-4ecc-8a7d-08647aebf8fd\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-tlggp" Nov 03 01:37:01.983589 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:01.983446 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e32924b-f5e1-4ecc-8a7d-08647aebf8fd-istio-token\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-tlggp\" (UID: \"8e32924b-f5e1-4ecc-8a7d-08647aebf8fd\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-tlggp" Nov 03 01:37:01.983589 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:01.983481 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e32924b-f5e1-4ecc-8a7d-08647aebf8fd-istio-data\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-tlggp\" (UID: \"8e32924b-f5e1-4ecc-8a7d-08647aebf8fd\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-tlggp" Nov 03 01:37:01.983712 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:01.983636 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e32924b-f5e1-4ecc-8a7d-08647aebf8fd-workload-certs\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-tlggp\" (UID: \"8e32924b-f5e1-4ecc-8a7d-08647aebf8fd\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-tlggp" Nov 03 01:37:01.983712 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:01.983698 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e32924b-f5e1-4ecc-8a7d-08647aebf8fd-istiod-ca-cert\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-tlggp\" (UID: \"8e32924b-f5e1-4ecc-8a7d-08647aebf8fd\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-tlggp" Nov 03 01:37:01.984052 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:01.984030 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e32924b-f5e1-4ecc-8a7d-08647aebf8fd-istio-podinfo\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-tlggp\" (UID: \"8e32924b-f5e1-4ecc-8a7d-08647aebf8fd\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-tlggp" Nov 03 01:37:01.984162 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:01.984099 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e32924b-f5e1-4ecc-8a7d-08647aebf8fd-credential-socket\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-tlggp\" (UID: \"8e32924b-f5e1-4ecc-8a7d-08647aebf8fd\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-tlggp" Nov 03 01:37:02.066889 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:37:02.066856 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaee71e38_f4ab_40e6_a3ad_1fe79bf0c63f.slice/crio-685ff6abf86b1a6f6d83b45bded6df02fa25a01aca8cc733a27cb1053499e1ef WatchSource:0}: Error finding container 685ff6abf86b1a6f6d83b45bded6df02fa25a01aca8cc733a27cb1053499e1ef: Status 404 returned error can't find the container with id 685ff6abf86b1a6f6d83b45bded6df02fa25a01aca8cc733a27cb1053499e1ef Nov 03 01:37:02.068900 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:02.068845 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-717217/rate-test-model-717217-kserve-74ddd7bbcc-4kxbf"] Nov 03 01:37:02.086429 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:02.086395 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e32924b-f5e1-4ecc-8a7d-08647aebf8fd-istio-envoy\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-tlggp\" (UID: \"8e32924b-f5e1-4ecc-8a7d-08647aebf8fd\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-tlggp" Nov 03 01:37:02.086575 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:02.086462 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tghxw\" (UniqueName: \"kubernetes.io/projected/8e32924b-f5e1-4ecc-8a7d-08647aebf8fd-kube-api-access-tghxw\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-tlggp\" (UID: \"8e32924b-f5e1-4ecc-8a7d-08647aebf8fd\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-tlggp" Nov 03 01:37:02.086575 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:02.086537 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e32924b-f5e1-4ecc-8a7d-08647aebf8fd-workload-socket\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-tlggp\" (UID: \"8e32924b-f5e1-4ecc-8a7d-08647aebf8fd\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-tlggp" Nov 03 01:37:02.086575 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:02.086573 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e32924b-f5e1-4ecc-8a7d-08647aebf8fd-istio-token\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-tlggp\" (UID: \"8e32924b-f5e1-4ecc-8a7d-08647aebf8fd\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-tlggp" Nov 03 01:37:02.086772 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:02.086607 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e32924b-f5e1-4ecc-8a7d-08647aebf8fd-istio-data\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-tlggp\" (UID: \"8e32924b-f5e1-4ecc-8a7d-08647aebf8fd\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-tlggp" Nov 03 01:37:02.086772 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:02.086667 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e32924b-f5e1-4ecc-8a7d-08647aebf8fd-workload-certs\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-tlggp\" (UID: \"8e32924b-f5e1-4ecc-8a7d-08647aebf8fd\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-tlggp" Nov 03 01:37:02.086772 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:02.086707 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e32924b-f5e1-4ecc-8a7d-08647aebf8fd-istiod-ca-cert\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-tlggp\" (UID: \"8e32924b-f5e1-4ecc-8a7d-08647aebf8fd\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-tlggp" Nov 03 01:37:02.086935 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:02.086776 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e32924b-f5e1-4ecc-8a7d-08647aebf8fd-istio-podinfo\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-tlggp\" (UID: \"8e32924b-f5e1-4ecc-8a7d-08647aebf8fd\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-tlggp" Nov 03 01:37:02.086935 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:02.086825 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e32924b-f5e1-4ecc-8a7d-08647aebf8fd-credential-socket\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-tlggp\" (UID: \"8e32924b-f5e1-4ecc-8a7d-08647aebf8fd\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-tlggp" Nov 03 01:37:02.087249 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:02.087224 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e32924b-f5e1-4ecc-8a7d-08647aebf8fd-credential-socket\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-tlggp\" (UID: \"8e32924b-f5e1-4ecc-8a7d-08647aebf8fd\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-tlggp" Nov 03 01:37:02.087386 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:02.087315 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e32924b-f5e1-4ecc-8a7d-08647aebf8fd-workload-certs\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-tlggp\" (UID: \"8e32924b-f5e1-4ecc-8a7d-08647aebf8fd\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-tlggp" Nov 03 01:37:02.087493 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:02.087454 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e32924b-f5e1-4ecc-8a7d-08647aebf8fd-workload-socket\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-tlggp\" (UID: \"8e32924b-f5e1-4ecc-8a7d-08647aebf8fd\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-tlggp" Nov 03 01:37:02.087773 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:02.087755 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e32924b-f5e1-4ecc-8a7d-08647aebf8fd-istio-data\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-tlggp\" (UID: \"8e32924b-f5e1-4ecc-8a7d-08647aebf8fd\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-tlggp" Nov 03 01:37:02.087850 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:02.087772 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e32924b-f5e1-4ecc-8a7d-08647aebf8fd-istiod-ca-cert\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-tlggp\" (UID: \"8e32924b-f5e1-4ecc-8a7d-08647aebf8fd\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-tlggp" Nov 03 01:37:02.090415 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:02.090390 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e32924b-f5e1-4ecc-8a7d-08647aebf8fd-istio-envoy\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-tlggp\" (UID: \"8e32924b-f5e1-4ecc-8a7d-08647aebf8fd\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-tlggp" Nov 03 01:37:02.090631 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:02.090611 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e32924b-f5e1-4ecc-8a7d-08647aebf8fd-istio-podinfo\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-tlggp\" (UID: \"8e32924b-f5e1-4ecc-8a7d-08647aebf8fd\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-tlggp" Nov 03 01:37:02.096051 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:02.096027 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e32924b-f5e1-4ecc-8a7d-08647aebf8fd-istio-token\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-tlggp\" (UID: \"8e32924b-f5e1-4ecc-8a7d-08647aebf8fd\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-tlggp" Nov 03 01:37:02.096204 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:02.096181 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tghxw\" (UniqueName: \"kubernetes.io/projected/8e32924b-f5e1-4ecc-8a7d-08647aebf8fd-kube-api-access-tghxw\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-tlggp\" (UID: \"8e32924b-f5e1-4ecc-8a7d-08647aebf8fd\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-tlggp" Nov 03 01:37:02.259644 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:02.259607 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-tlggp" Nov 03 01:37:02.465475 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:02.465441 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-717217/rate-test-model-717217-kserve-74ddd7bbcc-4kxbf" event={"ID":"aee71e38-f4ab-40e6-a3ad-1fe79bf0c63f","Type":"ContainerStarted","Data":"f395b119afd548c8c73c11058798f5d2d4381f429f3d07ea06d6ae29a02e5275"} Nov 03 01:37:02.465927 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:02.465908 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-717217/rate-test-model-717217-kserve-74ddd7bbcc-4kxbf" event={"ID":"aee71e38-f4ab-40e6-a3ad-1fe79bf0c63f","Type":"ContainerStarted","Data":"685ff6abf86b1a6f6d83b45bded6df02fa25a01aca8cc733a27cb1053499e1ef"} Nov 03 01:37:02.486236 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:02.486188 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-717217/rate-test-model-717217-kserve-74ddd7bbcc-4kxbf" podStartSLOduration=1.486174788 podStartE2EDuration="1.486174788s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:37:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:37:02.481644709 +0000 UTC m=+3364.652647806" watchObservedRunningTime="2026-11-03 01:37:02.486174788 +0000 UTC m=+3364.657177916" Nov 03 01:37:02.548186 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:02.548160 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-tlggp"] Nov 03 01:37:02.550893 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:37:02.550863 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e32924b_f5e1_4ecc_8a7d_08647aebf8fd.slice/crio-d64967260ae9b5601b7e241b83941d160d41a777f4c5f13b4bded6bca78f8f04 WatchSource:0}: Error finding container d64967260ae9b5601b7e241b83941d160d41a777f4c5f13b4bded6bca78f8f04: Status 404 returned error can't find the container with id d64967260ae9b5601b7e241b83941d160d41a777f4c5f13b4bded6bca78f8f04 Nov 03 01:37:02.553871 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:02.553834 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:37:02.553988 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:02.553909 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:37:02.553988 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:02.553940 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:37:02.768089 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:02.768029 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-717217/rate-test-model-717217-kserve-74ddd7bbcc-4kxbf" Nov 03 01:37:03.474861 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:03.474819 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-tlggp" event={"ID":"8e32924b-f5e1-4ecc-8a7d-08647aebf8fd","Type":"ContainerStarted","Data":"1c547b60d1d0b1ad263ddb29c394402893bbbeb36320a99b4367ca055962796b"} Nov 03 01:37:03.475306 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:03.474882 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-tlggp" event={"ID":"8e32924b-f5e1-4ecc-8a7d-08647aebf8fd","Type":"ContainerStarted","Data":"d64967260ae9b5601b7e241b83941d160d41a777f4c5f13b4bded6bca78f8f04"} Nov 03 01:37:03.497438 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:03.497383 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-tlggp" podStartSLOduration=2.4973638 podStartE2EDuration="2.4973638s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:37:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:37:03.494489175 +0000 UTC m=+3365.665492265" watchObservedRunningTime="2026-11-03 01:37:03.4973638 +0000 UTC m=+3365.668366898" Nov 03 01:37:04.259973 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:04.259897 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-tlggp" Nov 03 01:37:04.261335 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:04.261305 4421 patch_prober.go:29] interesting pod/e2e-shared-a-w2-531447-openshift-default-7995d46767-tlggp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.105:15021/healthz/ready\": dial tcp 10.134.1.105:15021: connect: connection refused" start-of-body= Nov 03 01:37:04.261480 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:04.261366 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-tlggp" podUID="8e32924b-f5e1-4ecc-8a7d-08647aebf8fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.105:15021/healthz/ready\": dial tcp 10.134.1.105:15021: connect: connection refused" Nov 03 01:37:05.260788 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:05.260750 4421 patch_prober.go:29] interesting pod/e2e-shared-a-w2-531447-openshift-default-7995d46767-tlggp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.105:15021/healthz/ready\": dial tcp 10.134.1.105:15021: connect: connection refused" start-of-body= Nov 03 01:37:05.261175 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:05.260808 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-tlggp" podUID="8e32924b-f5e1-4ecc-8a7d-08647aebf8fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.105:15021/healthz/ready\": dial tcp 10.134.1.105:15021: connect: connection refused" Nov 03 01:37:06.260514 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:06.260482 4421 patch_prober.go:29] interesting pod/e2e-shared-a-w2-531447-openshift-default-7995d46767-tlggp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.105:15021/healthz/ready\": dial tcp 10.134.1.105:15021: connect: connection refused" start-of-body= Nov 03 01:37:06.260687 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:06.260536 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-tlggp" podUID="8e32924b-f5e1-4ecc-8a7d-08647aebf8fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.105:15021/healthz/ready\": dial tcp 10.134.1.105:15021: connect: connection refused" Nov 03 01:37:07.260663 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:07.260620 4421 patch_prober.go:29] interesting pod/e2e-shared-a-w2-531447-openshift-default-7995d46767-tlggp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.105:15021/healthz/ready\": dial tcp 10.134.1.105:15021: connect: connection refused" start-of-body= Nov 03 01:37:07.261162 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:07.260681 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-tlggp" podUID="8e32924b-f5e1-4ecc-8a7d-08647aebf8fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.105:15021/healthz/ready\": dial tcp 10.134.1.105:15021: connect: connection refused" Nov 03 01:37:08.042951 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:08.042912 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-50e623-6d6668784c-kp496"] Nov 03 01:37:08.261300 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:08.260649 4421 patch_prober.go:29] interesting pod/e2e-shared-a-w2-531447-openshift-default-7995d46767-tlggp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.105:15021/healthz/ready\": dial tcp 10.134.1.105:15021: connect: connection refused" start-of-body= Nov 03 01:37:08.261300 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:08.260713 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-tlggp" podUID="8e32924b-f5e1-4ecc-8a7d-08647aebf8fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.105:15021/healthz/ready\": dial tcp 10.134.1.105:15021: connect: connection refused" Nov 03 01:37:09.370687 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:09.370648 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-tlggp" Nov 03 01:37:09.372220 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:09.372191 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-tlggp" Nov 03 01:37:09.373232 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:09.373200 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-tlggp" Nov 03 01:37:11.767127 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:11.767028 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-717217/rate-test-model-717217-kserve-74ddd7bbcc-4kxbf" Nov 03 01:37:11.780178 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:11.780147 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-717217/rate-test-model-717217-kserve-74ddd7bbcc-4kxbf" Nov 03 01:37:12.748232 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:12.748198 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-717217/rate-test-model-717217-kserve-74ddd7bbcc-4kxbf" Nov 03 01:37:13.043711 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:13.043591 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-50e623-6d6668784c-kp496" podUID="d12f07a3-b300-4368-b119-f860d1d62b5a" containerName="payload-processing" containerID="cri-o://f50315af9d472cb97c2fa507fa868fca5b42f5f694097606ab0f8ab7082e780b" gracePeriod=25 Nov 03 01:37:13.351398 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:13.351370 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-50e623-6d6668784c-kp496" Nov 03 01:37:13.417561 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:13.417528 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d12f07a3-b300-4368-b119-f860d1d62b5a-kube-api-access-d2kz7\" (UniqueName: \"kubernetes.io/projected/d12f07a3-b300-4368-b119-f860d1d62b5a-kube-api-access-d2kz7\") pod \"d12f07a3-b300-4368-b119-f860d1d62b5a\" (UID: \"d12f07a3-b300-4368-b119-f860d1d62b5a\") " Nov 03 01:37:13.417757 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:13.417658 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d12f07a3-b300-4368-b119-f860d1d62b5a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d12f07a3-b300-4368-b119-f860d1d62b5a-plugins-config-volume\") pod \"d12f07a3-b300-4368-b119-f860d1d62b5a\" (UID: \"d12f07a3-b300-4368-b119-f860d1d62b5a\") " Nov 03 01:37:13.418057 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:13.418028 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d12f07a3-b300-4368-b119-f860d1d62b5a-plugins-config-volume" pod "d12f07a3-b300-4368-b119-f860d1d62b5a" (UID: "d12f07a3-b300-4368-b119-f860d1d62b5a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:37:13.418332 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:13.418315 4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d12f07a3-b300-4368-b119-f860d1d62b5a-plugins-config-volume\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:37:13.420088 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:13.420059 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d12f07a3-b300-4368-b119-f860d1d62b5a-kube-api-access-d2kz7" pod "d12f07a3-b300-4368-b119-f860d1d62b5a" (UID: "d12f07a3-b300-4368-b119-f860d1d62b5a"). InnerVolumeSpecName "kube-api-access-d2kz7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:37:13.519504 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:13.519467 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2kz7\" (UniqueName: \"kubernetes.io/projected/d12f07a3-b300-4368-b119-f860d1d62b5a-kube-api-access-d2kz7\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:37:13.747118 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:13.747081 4421 generic.go:356] "Generic (PLEG): container finished" podID="d12f07a3-b300-4368-b119-f860d1d62b5a" containerID="f50315af9d472cb97c2fa507fa868fca5b42f5f694097606ab0f8ab7082e780b" exitCode=0 Nov 03 01:37:13.747310 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:13.747148 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-50e623-6d6668784c-kp496" Nov 03 01:37:13.747310 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:13.747153 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-50e623-6d6668784c-kp496" event={"ID":"d12f07a3-b300-4368-b119-f860d1d62b5a","Type":"ContainerDied","Data":"f50315af9d472cb97c2fa507fa868fca5b42f5f694097606ab0f8ab7082e780b"} Nov 03 01:37:13.747310 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:13.747204 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-50e623-6d6668784c-kp496" event={"ID":"d12f07a3-b300-4368-b119-f860d1d62b5a","Type":"ContainerDied","Data":"e2a278ca1e83884d4368a6d3d080d2a839c8ebb06638e2df5129a107ddd9f793"} Nov 03 01:37:13.747310 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:13.747226 4421 scope.go:122] "RemoveContainer" containerID="f50315af9d472cb97c2fa507fa868fca5b42f5f694097606ab0f8ab7082e780b" Nov 03 01:37:13.761035 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:13.761017 4421 scope.go:122] "RemoveContainer" containerID="f50315af9d472cb97c2fa507fa868fca5b42f5f694097606ab0f8ab7082e780b" Nov 03 01:37:13.761353 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:37:13.761332 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f50315af9d472cb97c2fa507fa868fca5b42f5f694097606ab0f8ab7082e780b\": container with ID starting with f50315af9d472cb97c2fa507fa868fca5b42f5f694097606ab0f8ab7082e780b not found: ID does not exist" containerID="f50315af9d472cb97c2fa507fa868fca5b42f5f694097606ab0f8ab7082e780b" Nov 03 01:37:13.761400 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:13.761361 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f50315af9d472cb97c2fa507fa868fca5b42f5f694097606ab0f8ab7082e780b"} err="failed to get container status \"f50315af9d472cb97c2fa507fa868fca5b42f5f694097606ab0f8ab7082e780b\": rpc error: code = NotFound desc = could not find container \"f50315af9d472cb97c2fa507fa868fca5b42f5f694097606ab0f8ab7082e780b\": container with ID starting with f50315af9d472cb97c2fa507fa868fca5b42f5f694097606ab0f8ab7082e780b not found: ID does not exist" Nov 03 01:37:13.815932 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:13.815624 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-50e623-6d6668784c-kp496"] Nov 03 01:37:13.824743 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:13.824707 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-50e623-6d6668784c-kp496"] Nov 03 01:37:13.981392 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:13.981350 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d12f07a3-b300-4368-b119-f860d1d62b5a" path="/var/lib/kubelet/pods/d12f07a3-b300-4368-b119-f860d1d62b5a/volumes" Nov 03 01:37:14.675151 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:37:14.675116 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 01:37:15.535870 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:15.535834 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-96hvv"] Nov 03 01:37:15.536111 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:15.536084 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-96hvv" podUID="f890f900-55cb-4e5f-b25d-8f5e899bc5c7" containerName="istio-proxy" containerID="cri-o://abcc3adc4d6cdff2cfb43a436232425ed085e2f44b8634aac12f68463233d03c" gracePeriod=30 Nov 03 01:37:17.886564 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:17.886517 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-556c14/rate-test-model-556c14-kserve-5ccd9784cc-s8fq2"] Nov 03 01:37:17.887672 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:17.887647 4421 cpu_manager.go:414] "containerMap: removing container" podUID="d12f07a3-b300-4368-b119-f860d1d62b5a" containerName="payload-processing" Nov 03 01:37:17.887752 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:17.887675 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d12f07a3-b300-4368-b119-f860d1d62b5a" containerName="payload-processing" Nov 03 01:37:17.887913 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:17.887898 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="d12f07a3-b300-4368-b119-f860d1d62b5a" containerName="payload-processing" Nov 03 01:37:17.896610 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:17.896582 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-556c14/rate-test-model-556c14-kserve-5ccd9784cc-s8fq2" Nov 03 01:37:17.899228 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:17.899199 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-556c14/rate-test-model-556c14-kserve-5ccd9784cc-s8fq2"] Nov 03 01:37:17.902638 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:17.902030 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-556c14\"/\"openshift-service-ca.crt\"" Nov 03 01:37:17.902638 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:17.902089 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-556c14\"/\"rate-test-model-556c14-kserve-self-signed-certs\"" Nov 03 01:37:17.902638 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:17.902134 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-556c14\"/\"default-dockercfg-mv62m\"" Nov 03 01:37:17.902638 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:17.902217 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-556c14\"/\"kube-root-ca.crt\"" Nov 03 01:37:17.995704 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:17.995671 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m48rr\" (UniqueName: \"kubernetes.io/projected/141cb342-025b-46eb-83b8-cbd2a60df0c0-kube-api-access-m48rr\") pod \"rate-test-model-556c14-kserve-5ccd9784cc-s8fq2\" (UID: \"141cb342-025b-46eb-83b8-cbd2a60df0c0\") " pod="ai-tenant-e2e-shared-b-w3-556c14/rate-test-model-556c14-kserve-5ccd9784cc-s8fq2" Nov 03 01:37:17.995908 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:17.995738 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/141cb342-025b-46eb-83b8-cbd2a60df0c0-home\") pod \"rate-test-model-556c14-kserve-5ccd9784cc-s8fq2\" (UID: \"141cb342-025b-46eb-83b8-cbd2a60df0c0\") " pod="ai-tenant-e2e-shared-b-w3-556c14/rate-test-model-556c14-kserve-5ccd9784cc-s8fq2" Nov 03 01:37:17.995908 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:17.995897 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/141cb342-025b-46eb-83b8-cbd2a60df0c0-model-cache\") pod \"rate-test-model-556c14-kserve-5ccd9784cc-s8fq2\" (UID: \"141cb342-025b-46eb-83b8-cbd2a60df0c0\") " pod="ai-tenant-e2e-shared-b-w3-556c14/rate-test-model-556c14-kserve-5ccd9784cc-s8fq2" Nov 03 01:37:17.996004 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:17.995933 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/141cb342-025b-46eb-83b8-cbd2a60df0c0-dshm\") pod \"rate-test-model-556c14-kserve-5ccd9784cc-s8fq2\" (UID: \"141cb342-025b-46eb-83b8-cbd2a60df0c0\") " pod="ai-tenant-e2e-shared-b-w3-556c14/rate-test-model-556c14-kserve-5ccd9784cc-s8fq2" Nov 03 01:37:17.996046 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:17.996001 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/141cb342-025b-46eb-83b8-cbd2a60df0c0-tmp-dir\") pod \"rate-test-model-556c14-kserve-5ccd9784cc-s8fq2\" (UID: \"141cb342-025b-46eb-83b8-cbd2a60df0c0\") " pod="ai-tenant-e2e-shared-b-w3-556c14/rate-test-model-556c14-kserve-5ccd9784cc-s8fq2" Nov 03 01:37:17.996046 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:17.996036 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/141cb342-025b-46eb-83b8-cbd2a60df0c0-tls-certs\") pod \"rate-test-model-556c14-kserve-5ccd9784cc-s8fq2\" (UID: \"141cb342-025b-46eb-83b8-cbd2a60df0c0\") " pod="ai-tenant-e2e-shared-b-w3-556c14/rate-test-model-556c14-kserve-5ccd9784cc-s8fq2" Nov 03 01:37:18.123300 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:18.122495 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/141cb342-025b-46eb-83b8-cbd2a60df0c0-model-cache\") pod \"rate-test-model-556c14-kserve-5ccd9784cc-s8fq2\" (UID: \"141cb342-025b-46eb-83b8-cbd2a60df0c0\") " pod="ai-tenant-e2e-shared-b-w3-556c14/rate-test-model-556c14-kserve-5ccd9784cc-s8fq2" Nov 03 01:37:18.123300 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:18.122574 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/141cb342-025b-46eb-83b8-cbd2a60df0c0-dshm\") pod \"rate-test-model-556c14-kserve-5ccd9784cc-s8fq2\" (UID: \"141cb342-025b-46eb-83b8-cbd2a60df0c0\") " pod="ai-tenant-e2e-shared-b-w3-556c14/rate-test-model-556c14-kserve-5ccd9784cc-s8fq2" Nov 03 01:37:18.123300 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:18.122624 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/141cb342-025b-46eb-83b8-cbd2a60df0c0-tmp-dir\") pod \"rate-test-model-556c14-kserve-5ccd9784cc-s8fq2\" (UID: \"141cb342-025b-46eb-83b8-cbd2a60df0c0\") " pod="ai-tenant-e2e-shared-b-w3-556c14/rate-test-model-556c14-kserve-5ccd9784cc-s8fq2" Nov 03 01:37:18.123300 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:18.122678 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/141cb342-025b-46eb-83b8-cbd2a60df0c0-tls-certs\") pod \"rate-test-model-556c14-kserve-5ccd9784cc-s8fq2\" (UID: \"141cb342-025b-46eb-83b8-cbd2a60df0c0\") " pod="ai-tenant-e2e-shared-b-w3-556c14/rate-test-model-556c14-kserve-5ccd9784cc-s8fq2" Nov 03 01:37:18.123300 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:18.122863 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m48rr\" (UniqueName: \"kubernetes.io/projected/141cb342-025b-46eb-83b8-cbd2a60df0c0-kube-api-access-m48rr\") pod \"rate-test-model-556c14-kserve-5ccd9784cc-s8fq2\" (UID: \"141cb342-025b-46eb-83b8-cbd2a60df0c0\") " pod="ai-tenant-e2e-shared-b-w3-556c14/rate-test-model-556c14-kserve-5ccd9784cc-s8fq2" Nov 03 01:37:18.123300 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:18.122976 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/141cb342-025b-46eb-83b8-cbd2a60df0c0-home\") pod \"rate-test-model-556c14-kserve-5ccd9784cc-s8fq2\" (UID: \"141cb342-025b-46eb-83b8-cbd2a60df0c0\") " pod="ai-tenant-e2e-shared-b-w3-556c14/rate-test-model-556c14-kserve-5ccd9784cc-s8fq2" Nov 03 01:37:18.139360 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:18.124149 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/141cb342-025b-46eb-83b8-cbd2a60df0c0-model-cache\") pod \"rate-test-model-556c14-kserve-5ccd9784cc-s8fq2\" (UID: \"141cb342-025b-46eb-83b8-cbd2a60df0c0\") " pod="ai-tenant-e2e-shared-b-w3-556c14/rate-test-model-556c14-kserve-5ccd9784cc-s8fq2" Nov 03 01:37:18.139360 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:18.127589 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/141cb342-025b-46eb-83b8-cbd2a60df0c0-home\") pod \"rate-test-model-556c14-kserve-5ccd9784cc-s8fq2\" (UID: \"141cb342-025b-46eb-83b8-cbd2a60df0c0\") " pod="ai-tenant-e2e-shared-b-w3-556c14/rate-test-model-556c14-kserve-5ccd9784cc-s8fq2" Nov 03 01:37:18.139360 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:18.127835 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/141cb342-025b-46eb-83b8-cbd2a60df0c0-tmp-dir\") pod \"rate-test-model-556c14-kserve-5ccd9784cc-s8fq2\" (UID: \"141cb342-025b-46eb-83b8-cbd2a60df0c0\") " pod="ai-tenant-e2e-shared-b-w3-556c14/rate-test-model-556c14-kserve-5ccd9784cc-s8fq2" Nov 03 01:37:18.178955 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:18.178926 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m48rr\" (UniqueName: \"kubernetes.io/projected/141cb342-025b-46eb-83b8-cbd2a60df0c0-kube-api-access-m48rr\") pod \"rate-test-model-556c14-kserve-5ccd9784cc-s8fq2\" (UID: \"141cb342-025b-46eb-83b8-cbd2a60df0c0\") " pod="ai-tenant-e2e-shared-b-w3-556c14/rate-test-model-556c14-kserve-5ccd9784cc-s8fq2" Nov 03 01:37:18.181043 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:18.180977 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/141cb342-025b-46eb-83b8-cbd2a60df0c0-tls-certs\") pod \"rate-test-model-556c14-kserve-5ccd9784cc-s8fq2\" (UID: \"141cb342-025b-46eb-83b8-cbd2a60df0c0\") " pod="ai-tenant-e2e-shared-b-w3-556c14/rate-test-model-556c14-kserve-5ccd9784cc-s8fq2" Nov 03 01:37:18.196300 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:18.189663 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/141cb342-025b-46eb-83b8-cbd2a60df0c0-dshm\") pod \"rate-test-model-556c14-kserve-5ccd9784cc-s8fq2\" (UID: \"141cb342-025b-46eb-83b8-cbd2a60df0c0\") " pod="ai-tenant-e2e-shared-b-w3-556c14/rate-test-model-556c14-kserve-5ccd9784cc-s8fq2" Nov 03 01:37:18.217254 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:18.217216 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-556c14/rate-test-model-556c14-kserve-5ccd9784cc-s8fq2" Nov 03 01:37:18.507145 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:18.507117 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-556c14/rate-test-model-556c14-kserve-5ccd9784cc-s8fq2"] Nov 03 01:37:18.508168 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:37:18.508140 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod141cb342_025b_46eb_83b8_cbd2a60df0c0.slice/crio-6304b5fc34cb38cea1c3828049e655f48a53cfdfcddfd0dd6ff75597f1fe16cc WatchSource:0}: Error finding container 6304b5fc34cb38cea1c3828049e655f48a53cfdfcddfd0dd6ff75597f1fe16cc: Status 404 returned error can't find the container with id 6304b5fc34cb38cea1c3828049e655f48a53cfdfcddfd0dd6ff75597f1fe16cc Nov 03 01:37:18.821659 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:18.821568 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-556c14/rate-test-model-556c14-kserve-5ccd9784cc-s8fq2" event={"ID":"141cb342-025b-46eb-83b8-cbd2a60df0c0","Type":"ContainerStarted","Data":"01092830cf39cda527bba975b19fae5025e0ad65e1f647a3dc67810494b8c5b7"} Nov 03 01:37:18.821659 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:18.821610 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-556c14/rate-test-model-556c14-kserve-5ccd9784cc-s8fq2" event={"ID":"141cb342-025b-46eb-83b8-cbd2a60df0c0","Type":"ContainerStarted","Data":"6304b5fc34cb38cea1c3828049e655f48a53cfdfcddfd0dd6ff75597f1fe16cc"} Nov 03 01:37:18.838709 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:18.838651 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-556c14/rate-test-model-556c14-kserve-5ccd9784cc-s8fq2" podStartSLOduration=1.838636422 podStartE2EDuration="1.838636422s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:37:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:37:18.837728401 +0000 UTC m=+3381.008731497" watchObservedRunningTime="2026-11-03 01:37:18.838636422 +0000 UTC m=+3381.009639518" Nov 03 01:37:19.217909 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:19.217822 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-556c14/rate-test-model-556c14-kserve-5ccd9784cc-s8fq2" Nov 03 01:37:20.878730 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:20.878689 4421 generic.go:356] "Generic (PLEG): container finished" podID="f890f900-55cb-4e5f-b25d-8f5e899bc5c7" containerID="abcc3adc4d6cdff2cfb43a436232425ed085e2f44b8634aac12f68463233d03c" exitCode=0 Nov 03 01:37:20.879767 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:20.879739 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-96hvv" event={"ID":"f890f900-55cb-4e5f-b25d-8f5e899bc5c7","Type":"ContainerDied","Data":"abcc3adc4d6cdff2cfb43a436232425ed085e2f44b8634aac12f68463233d03c"} Nov 03 01:37:20.879937 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:20.879840 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-96hvv" event={"ID":"f890f900-55cb-4e5f-b25d-8f5e899bc5c7","Type":"ContainerDied","Data":"d51c99bf88db8a11698844c5fe654e29dc7a88c3879c3d6e80eb0f81ce6184f9"} Nov 03 01:37:20.879937 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:20.879866 4421 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d51c99bf88db8a11698844c5fe654e29dc7a88c3879c3d6e80eb0f81ce6184f9" Nov 03 01:37:20.888009 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:20.887983 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-96hvv" Nov 03 01:37:20.974527 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:20.974486 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f890f900-55cb-4e5f-b25d-8f5e899bc5c7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f890f900-55cb-4e5f-b25d-8f5e899bc5c7-workload-socket\") pod \"f890f900-55cb-4e5f-b25d-8f5e899bc5c7\" (UID: \"f890f900-55cb-4e5f-b25d-8f5e899bc5c7\") " Nov 03 01:37:20.974745 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:20.974562 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f890f900-55cb-4e5f-b25d-8f5e899bc5c7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f890f900-55cb-4e5f-b25d-8f5e899bc5c7-istio-podinfo\") pod \"f890f900-55cb-4e5f-b25d-8f5e899bc5c7\" (UID: \"f890f900-55cb-4e5f-b25d-8f5e899bc5c7\") " Nov 03 01:37:20.974745 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:20.974719 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f890f900-55cb-4e5f-b25d-8f5e899bc5c7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f890f900-55cb-4e5f-b25d-8f5e899bc5c7-istio-envoy\") pod \"f890f900-55cb-4e5f-b25d-8f5e899bc5c7\" (UID: \"f890f900-55cb-4e5f-b25d-8f5e899bc5c7\") " Nov 03 01:37:20.974885 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:20.974749 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f890f900-55cb-4e5f-b25d-8f5e899bc5c7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f890f900-55cb-4e5f-b25d-8f5e899bc5c7-istio-data\") pod \"f890f900-55cb-4e5f-b25d-8f5e899bc5c7\" (UID: \"f890f900-55cb-4e5f-b25d-8f5e899bc5c7\") " Nov 03 01:37:20.974885 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:20.974824 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f890f900-55cb-4e5f-b25d-8f5e899bc5c7-istio-token\" (UniqueName: \"kubernetes.io/projected/f890f900-55cb-4e5f-b25d-8f5e899bc5c7-istio-token\") pod \"f890f900-55cb-4e5f-b25d-8f5e899bc5c7\" (UID: \"f890f900-55cb-4e5f-b25d-8f5e899bc5c7\") " Nov 03 01:37:20.974885 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:20.974856 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f890f900-55cb-4e5f-b25d-8f5e899bc5c7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f890f900-55cb-4e5f-b25d-8f5e899bc5c7-istiod-ca-cert\") pod \"f890f900-55cb-4e5f-b25d-8f5e899bc5c7\" (UID: \"f890f900-55cb-4e5f-b25d-8f5e899bc5c7\") " Nov 03 01:37:20.974885 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:20.974851 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f890f900-55cb-4e5f-b25d-8f5e899bc5c7-workload-socket" pod "f890f900-55cb-4e5f-b25d-8f5e899bc5c7" (UID: "f890f900-55cb-4e5f-b25d-8f5e899bc5c7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:37:20.975110 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:20.974898 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f890f900-55cb-4e5f-b25d-8f5e899bc5c7-kube-api-access-pdqz6\" (UniqueName: \"kubernetes.io/projected/f890f900-55cb-4e5f-b25d-8f5e899bc5c7-kube-api-access-pdqz6\") pod \"f890f900-55cb-4e5f-b25d-8f5e899bc5c7\" (UID: \"f890f900-55cb-4e5f-b25d-8f5e899bc5c7\") " Nov 03 01:37:20.975110 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:20.974967 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f890f900-55cb-4e5f-b25d-8f5e899bc5c7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f890f900-55cb-4e5f-b25d-8f5e899bc5c7-workload-certs\") pod \"f890f900-55cb-4e5f-b25d-8f5e899bc5c7\" (UID: \"f890f900-55cb-4e5f-b25d-8f5e899bc5c7\") " Nov 03 01:37:20.975110 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:20.975075 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f890f900-55cb-4e5f-b25d-8f5e899bc5c7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f890f900-55cb-4e5f-b25d-8f5e899bc5c7-credential-socket\") pod \"f890f900-55cb-4e5f-b25d-8f5e899bc5c7\" (UID: \"f890f900-55cb-4e5f-b25d-8f5e899bc5c7\") " Nov 03 01:37:20.975401 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:20.975373 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f890f900-55cb-4e5f-b25d-8f5e899bc5c7-istiod-ca-cert" pod "f890f900-55cb-4e5f-b25d-8f5e899bc5c7" (UID: "f890f900-55cb-4e5f-b25d-8f5e899bc5c7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:37:20.976205 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:20.975622 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f890f900-55cb-4e5f-b25d-8f5e899bc5c7-credential-socket" pod "f890f900-55cb-4e5f-b25d-8f5e899bc5c7" (UID: "f890f900-55cb-4e5f-b25d-8f5e899bc5c7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:37:20.976205 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:20.975735 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f890f900-55cb-4e5f-b25d-8f5e899bc5c7-workload-certs" pod "f890f900-55cb-4e5f-b25d-8f5e899bc5c7" (UID: "f890f900-55cb-4e5f-b25d-8f5e899bc5c7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:37:20.976205 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:20.976117 4421 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f890f900-55cb-4e5f-b25d-8f5e899bc5c7-credential-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:37:20.976205 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:20.976140 4421 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f890f900-55cb-4e5f-b25d-8f5e899bc5c7-workload-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:37:20.976205 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:20.976154 4421 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f890f900-55cb-4e5f-b25d-8f5e899bc5c7-istiod-ca-cert\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:37:20.976205 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:20.976168 4421 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f890f900-55cb-4e5f-b25d-8f5e899bc5c7-workload-certs\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:37:20.977135 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:20.975569 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f890f900-55cb-4e5f-b25d-8f5e899bc5c7-istio-data" pod "f890f900-55cb-4e5f-b25d-8f5e899bc5c7" (UID: "f890f900-55cb-4e5f-b25d-8f5e899bc5c7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:37:20.977846 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:20.977819 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f890f900-55cb-4e5f-b25d-8f5e899bc5c7-istio-envoy" pod "f890f900-55cb-4e5f-b25d-8f5e899bc5c7" (UID: "f890f900-55cb-4e5f-b25d-8f5e899bc5c7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:37:20.979011 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:20.978981 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f890f900-55cb-4e5f-b25d-8f5e899bc5c7-kube-api-access-pdqz6" pod "f890f900-55cb-4e5f-b25d-8f5e899bc5c7" (UID: "f890f900-55cb-4e5f-b25d-8f5e899bc5c7"). InnerVolumeSpecName "kube-api-access-pdqz6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:37:20.981743 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:20.981699 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f890f900-55cb-4e5f-b25d-8f5e899bc5c7-istio-token" pod "f890f900-55cb-4e5f-b25d-8f5e899bc5c7" (UID: "f890f900-55cb-4e5f-b25d-8f5e899bc5c7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:37:20.981999 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:20.981982 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f890f900-55cb-4e5f-b25d-8f5e899bc5c7-istio-podinfo" pod "f890f900-55cb-4e5f-b25d-8f5e899bc5c7" (UID: "f890f900-55cb-4e5f-b25d-8f5e899bc5c7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:37:21.078137 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:21.078045 4421 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f890f900-55cb-4e5f-b25d-8f5e899bc5c7-istio-data\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:37:21.078137 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:21.078089 4421 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f890f900-55cb-4e5f-b25d-8f5e899bc5c7-istio-token\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:37:21.078137 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:21.078106 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdqz6\" (UniqueName: \"kubernetes.io/projected/f890f900-55cb-4e5f-b25d-8f5e899bc5c7-kube-api-access-pdqz6\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:37:21.078137 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:21.078122 4421 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f890f900-55cb-4e5f-b25d-8f5e899bc5c7-istio-podinfo\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:37:21.078137 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:21.078137 4421 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f890f900-55cb-4e5f-b25d-8f5e899bc5c7-istio-envoy\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:37:21.514586 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:21.514492 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-020af9-66449c479f-bmnx4"] Nov 03 01:37:21.528327 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:21.528293 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-020af9-7fcf787fd7-9fgx6"] Nov 03 01:37:21.903365 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:21.902893 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-96hvv" Nov 03 01:37:21.933936 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:21.933897 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-96hvv"] Nov 03 01:37:21.938693 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:21.938658 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-96hvv"] Nov 03 01:37:21.981264 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:21.981233 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f890f900-55cb-4e5f-b25d-8f5e899bc5c7" path="/var/lib/kubelet/pods/f890f900-55cb-4e5f-b25d-8f5e899bc5c7/volumes" Nov 03 01:37:26.516116 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:26.516052 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-020af9-66449c479f-bmnx4" podUID="c671f325-44a0-497f-afb5-24f3871cc496" containerName="payload-processing" containerID="cri-o://e445366f56173baa16d4e190841a3e0dff3eaffcbe18c6d11775c966f22c74fe" gracePeriod=25 Nov 03 01:37:26.528845 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:26.528799 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-020af9-7fcf787fd7-9fgx6" podUID="7f5f8dc1-5619-4ffb-b968-06869fa59c39" containerName="payload-pre-processing" containerID="cri-o://6630ccebf1f1ad6d0a9d1180bf83ad096d3e8150dcaea1b51fe21615822b338c" gracePeriod=25 Nov 03 01:37:28.217868 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:28.217826 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-556c14/rate-test-model-556c14-kserve-5ccd9784cc-s8fq2" Nov 03 01:37:28.230918 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:28.230888 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-556c14/rate-test-model-556c14-kserve-5ccd9784cc-s8fq2" Nov 03 01:37:29.006478 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:29.006447 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-556c14/rate-test-model-556c14-kserve-5ccd9784cc-s8fq2" Nov 03 01:37:32.836784 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:32.836743 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-020af9-7fcf787fd7-9fgx6" podUID="7f5f8dc1-5619-4ffb-b968-06869fa59c39" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:37:32.837310 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:32.836774 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-020af9-66449c479f-bmnx4" podUID="c671f325-44a0-497f-afb5-24f3871cc496" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:37:40.308062 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:40.308027 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-031e492b-hpfg6"] Nov 03 01:37:40.308735 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:40.308720 4421 cpu_manager.go:414] "containerMap: removing container" podUID="f890f900-55cb-4e5f-b25d-8f5e899bc5c7" containerName="istio-proxy" Nov 03 01:37:40.308793 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:40.308735 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f890f900-55cb-4e5f-b25d-8f5e899bc5c7" containerName="istio-proxy" Nov 03 01:37:40.308886 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:40.308877 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="f890f900-55cb-4e5f-b25d-8f5e899bc5c7" containerName="istio-proxy" Nov 03 01:37:40.312591 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:40.312568 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-031e492b-hpfg6" Nov 03 01:37:40.316060 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:40.316033 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:37:40.317525 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:40.317503 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6q4xp\"" Nov 03 01:37:40.320429 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:40.320399 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-031e492b-hpfg6"] Nov 03 01:37:40.397723 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:40.397688 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/617b1b92-68e6-47cd-ac46-2542031c7efc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-031e492b-hpfg6\" (UID: \"617b1b92-68e6-47cd-ac46-2542031c7efc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-031e492b-hpfg6" Nov 03 01:37:40.397903 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:40.397821 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbg2w\" (UniqueName: \"kubernetes.io/projected/617b1b92-68e6-47cd-ac46-2542031c7efc-kube-api-access-rbg2w\") pod \"maas-api-revoke-sub-031e492b-hpfg6\" (UID: \"617b1b92-68e6-47cd-ac46-2542031c7efc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-031e492b-hpfg6" Nov 03 01:37:40.397903 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:40.397879 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/617b1b92-68e6-47cd-ac46-2542031c7efc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-031e492b-hpfg6\" (UID: \"617b1b92-68e6-47cd-ac46-2542031c7efc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-031e492b-hpfg6" Nov 03 01:37:40.499705 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:40.499666 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/617b1b92-68e6-47cd-ac46-2542031c7efc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-031e492b-hpfg6\" (UID: \"617b1b92-68e6-47cd-ac46-2542031c7efc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-031e492b-hpfg6" Nov 03 01:37:40.499904 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:40.499758 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbg2w\" (UniqueName: \"kubernetes.io/projected/617b1b92-68e6-47cd-ac46-2542031c7efc-kube-api-access-rbg2w\") pod \"maas-api-revoke-sub-031e492b-hpfg6\" (UID: \"617b1b92-68e6-47cd-ac46-2542031c7efc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-031e492b-hpfg6" Nov 03 01:37:40.499904 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:40.499821 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/617b1b92-68e6-47cd-ac46-2542031c7efc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-031e492b-hpfg6\" (UID: \"617b1b92-68e6-47cd-ac46-2542031c7efc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-031e492b-hpfg6" Nov 03 01:37:40.500667 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:40.500631 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/617b1b92-68e6-47cd-ac46-2542031c7efc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-031e492b-hpfg6\" (UID: \"617b1b92-68e6-47cd-ac46-2542031c7efc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-031e492b-hpfg6" Nov 03 01:37:40.502356 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:40.502330 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/617b1b92-68e6-47cd-ac46-2542031c7efc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-031e492b-hpfg6\" (UID: \"617b1b92-68e6-47cd-ac46-2542031c7efc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-031e492b-hpfg6" Nov 03 01:37:40.507863 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:40.507837 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbg2w\" (UniqueName: \"kubernetes.io/projected/617b1b92-68e6-47cd-ac46-2542031c7efc-kube-api-access-rbg2w\") pod \"maas-api-revoke-sub-031e492b-hpfg6\" (UID: \"617b1b92-68e6-47cd-ac46-2542031c7efc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-031e492b-hpfg6" Nov 03 01:37:40.641784 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:40.641679 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-031e492b-hpfg6" Nov 03 01:37:40.947777 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:40.947739 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-031e492b-hpfg6"] Nov 03 01:37:41.192392 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:41.192306 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-031e492b-hpfg6" event={"ID":"617b1b92-68e6-47cd-ac46-2542031c7efc","Type":"ContainerStarted","Data":"0d600004a921bb248c2431b17e1824a6a73518bdb76d4ece72dcc980ecd6a8e8"} Nov 03 01:37:42.201756 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:42.201666 4421 generic.go:356] "Generic (PLEG): container finished" podID="617b1b92-68e6-47cd-ac46-2542031c7efc" containerID="d665d84272233cca08640f0873e58eba761cb71da4ed66269713a193e85bc5d2" exitCode=0 Nov 03 01:37:42.201756 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:42.201749 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-031e492b-hpfg6" event={"ID":"617b1b92-68e6-47cd-ac46-2542031c7efc","Type":"ContainerDied","Data":"d665d84272233cca08640f0873e58eba761cb71da4ed66269713a193e85bc5d2"} Nov 03 01:37:42.836390 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:42.836349 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-020af9-66449c479f-bmnx4" podUID="c671f325-44a0-497f-afb5-24f3871cc496" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:37:42.836576 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:42.836359 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-020af9-7fcf787fd7-9fgx6" podUID="7f5f8dc1-5619-4ffb-b968-06869fa59c39" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:37:43.378366 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:43.378338 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-031e492b-hpfg6" Nov 03 01:37:43.475924 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:43.475841 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/617b1b92-68e6-47cd-ac46-2542031c7efc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/617b1b92-68e6-47cd-ac46-2542031c7efc-maas-api-openshift-ca\") pod \"617b1b92-68e6-47cd-ac46-2542031c7efc\" (UID: \"617b1b92-68e6-47cd-ac46-2542031c7efc\") " Nov 03 01:37:43.476729 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:43.475981 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/617b1b92-68e6-47cd-ac46-2542031c7efc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/617b1b92-68e6-47cd-ac46-2542031c7efc-maas-api-xks-ca\") pod \"617b1b92-68e6-47cd-ac46-2542031c7efc\" (UID: \"617b1b92-68e6-47cd-ac46-2542031c7efc\") " Nov 03 01:37:43.476729 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:43.476007 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/617b1b92-68e6-47cd-ac46-2542031c7efc-kube-api-access-rbg2w\" (UniqueName: \"kubernetes.io/projected/617b1b92-68e6-47cd-ac46-2542031c7efc-kube-api-access-rbg2w\") pod \"617b1b92-68e6-47cd-ac46-2542031c7efc\" (UID: \"617b1b92-68e6-47cd-ac46-2542031c7efc\") " Nov 03 01:37:43.476729 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:43.476194 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/617b1b92-68e6-47cd-ac46-2542031c7efc-maas-api-openshift-ca" pod "617b1b92-68e6-47cd-ac46-2542031c7efc" (UID: "617b1b92-68e6-47cd-ac46-2542031c7efc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:37:43.476729 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:43.476628 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/617b1b92-68e6-47cd-ac46-2542031c7efc-maas-api-openshift-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:37:43.478358 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:43.478333 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/617b1b92-68e6-47cd-ac46-2542031c7efc-maas-api-xks-ca" pod "617b1b92-68e6-47cd-ac46-2542031c7efc" (UID: "617b1b92-68e6-47cd-ac46-2542031c7efc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:37:43.478473 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:43.478419 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/617b1b92-68e6-47cd-ac46-2542031c7efc-kube-api-access-rbg2w" pod "617b1b92-68e6-47cd-ac46-2542031c7efc" (UID: "617b1b92-68e6-47cd-ac46-2542031c7efc"). InnerVolumeSpecName "kube-api-access-rbg2w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:37:43.578230 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:43.578189 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/617b1b92-68e6-47cd-ac46-2542031c7efc-maas-api-xks-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:37:43.578230 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:43.578226 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbg2w\" (UniqueName: \"kubernetes.io/projected/617b1b92-68e6-47cd-ac46-2542031c7efc-kube-api-access-rbg2w\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:37:44.218253 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:44.218221 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-031e492b-hpfg6" Nov 03 01:37:44.218444 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:44.218264 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-031e492b-hpfg6" event={"ID":"617b1b92-68e6-47cd-ac46-2542031c7efc","Type":"ContainerDied","Data":"0d600004a921bb248c2431b17e1824a6a73518bdb76d4ece72dcc980ecd6a8e8"} Nov 03 01:37:44.218444 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:44.218335 4421 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0d600004a921bb248c2431b17e1824a6a73518bdb76d4ece72dcc980ecd6a8e8" Nov 03 01:37:44.671014 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:37:44.670967 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 01:37:46.821696 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:46.821657 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-020af9-66449c479f-bmnx4" Nov 03 01:37:46.909194 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:46.909163 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-020af9-7fcf787fd7-9fgx6" Nov 03 01:37:46.929011 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:46.928934 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c671f325-44a0-497f-afb5-24f3871cc496-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c671f325-44a0-497f-afb5-24f3871cc496-plugins-config-volume\") pod \"c671f325-44a0-497f-afb5-24f3871cc496\" (UID: \"c671f325-44a0-497f-afb5-24f3871cc496\") " Nov 03 01:37:46.929525 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:46.929486 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c671f325-44a0-497f-afb5-24f3871cc496-kube-api-access-lk9wg\" (UniqueName: \"kubernetes.io/projected/c671f325-44a0-497f-afb5-24f3871cc496-kube-api-access-lk9wg\") pod \"c671f325-44a0-497f-afb5-24f3871cc496\" (UID: \"c671f325-44a0-497f-afb5-24f3871cc496\") " Nov 03 01:37:46.931858 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:46.931824 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c671f325-44a0-497f-afb5-24f3871cc496-plugins-config-volume" pod "c671f325-44a0-497f-afb5-24f3871cc496" (UID: "c671f325-44a0-497f-afb5-24f3871cc496"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:37:46.932963 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:46.932936 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c671f325-44a0-497f-afb5-24f3871cc496-kube-api-access-lk9wg" pod "c671f325-44a0-497f-afb5-24f3871cc496" (UID: "c671f325-44a0-497f-afb5-24f3871cc496"). InnerVolumeSpecName "kube-api-access-lk9wg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:37:47.031630 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:47.031535 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f5f8dc1-5619-4ffb-b968-06869fa59c39-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f5f8dc1-5619-4ffb-b968-06869fa59c39-plugins-config-volume\") pod \"7f5f8dc1-5619-4ffb-b968-06869fa59c39\" (UID: \"7f5f8dc1-5619-4ffb-b968-06869fa59c39\") " Nov 03 01:37:47.031630 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:47.031605 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f5f8dc1-5619-4ffb-b968-06869fa59c39-kube-api-access-tcfbn\" (UniqueName: \"kubernetes.io/projected/7f5f8dc1-5619-4ffb-b968-06869fa59c39-kube-api-access-tcfbn\") pod \"7f5f8dc1-5619-4ffb-b968-06869fa59c39\" (UID: \"7f5f8dc1-5619-4ffb-b968-06869fa59c39\") " Nov 03 01:37:47.031898 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:47.031872 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f5f8dc1-5619-4ffb-b968-06869fa59c39-plugins-config-volume" pod "7f5f8dc1-5619-4ffb-b968-06869fa59c39" (UID: "7f5f8dc1-5619-4ffb-b968-06869fa59c39"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:37:47.032328 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:47.032292 4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c671f325-44a0-497f-afb5-24f3871cc496-plugins-config-volume\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:37:47.032328 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:47.032317 4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f5f8dc1-5619-4ffb-b968-06869fa59c39-plugins-config-volume\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:37:47.032474 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:47.032331 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lk9wg\" (UniqueName: \"kubernetes.io/projected/c671f325-44a0-497f-afb5-24f3871cc496-kube-api-access-lk9wg\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:37:47.033798 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:47.033775 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f5f8dc1-5619-4ffb-b968-06869fa59c39-kube-api-access-tcfbn" pod "7f5f8dc1-5619-4ffb-b968-06869fa59c39" (UID: "7f5f8dc1-5619-4ffb-b968-06869fa59c39"). InnerVolumeSpecName "kube-api-access-tcfbn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:37:47.133398 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:47.133365 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tcfbn\" (UniqueName: \"kubernetes.io/projected/7f5f8dc1-5619-4ffb-b968-06869fa59c39-kube-api-access-tcfbn\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:37:47.242715 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:47.242677 4421 generic.go:356] "Generic (PLEG): container finished" podID="c671f325-44a0-497f-afb5-24f3871cc496" containerID="e445366f56173baa16d4e190841a3e0dff3eaffcbe18c6d11775c966f22c74fe" exitCode=0 Nov 03 01:37:47.242918 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:47.242735 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-020af9-66449c479f-bmnx4" Nov 03 01:37:47.242918 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:47.242767 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-020af9-66449c479f-bmnx4" event={"ID":"c671f325-44a0-497f-afb5-24f3871cc496","Type":"ContainerDied","Data":"e445366f56173baa16d4e190841a3e0dff3eaffcbe18c6d11775c966f22c74fe"} Nov 03 01:37:47.242918 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:47.242824 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-020af9-66449c479f-bmnx4" event={"ID":"c671f325-44a0-497f-afb5-24f3871cc496","Type":"ContainerDied","Data":"111cd64b5758df0e0a7bf53bab3376b7e06e099f1a09cf0cfa867f59939f5370"} Nov 03 01:37:47.242918 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:47.242856 4421 scope.go:122] "RemoveContainer" containerID="e445366f56173baa16d4e190841a3e0dff3eaffcbe18c6d11775c966f22c74fe" Nov 03 01:37:47.244509 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:47.244483 4421 generic.go:356] "Generic (PLEG): container finished" podID="7f5f8dc1-5619-4ffb-b968-06869fa59c39" containerID="6630ccebf1f1ad6d0a9d1180bf83ad096d3e8150dcaea1b51fe21615822b338c" exitCode=0 Nov 03 01:37:47.244694 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:47.244530 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-020af9-7fcf787fd7-9fgx6" Nov 03 01:37:47.244694 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:47.244570 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-020af9-7fcf787fd7-9fgx6" event={"ID":"7f5f8dc1-5619-4ffb-b968-06869fa59c39","Type":"ContainerDied","Data":"6630ccebf1f1ad6d0a9d1180bf83ad096d3e8150dcaea1b51fe21615822b338c"} Nov 03 01:37:47.244694 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:47.244625 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-020af9-7fcf787fd7-9fgx6" event={"ID":"7f5f8dc1-5619-4ffb-b968-06869fa59c39","Type":"ContainerDied","Data":"7f8f9a835cd8a74893a1af9a7a937a0dd44210308b0e84343b8b2b57d7037cd2"} Nov 03 01:37:47.256973 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:47.256949 4421 scope.go:122] "RemoveContainer" containerID="e445366f56173baa16d4e190841a3e0dff3eaffcbe18c6d11775c966f22c74fe" Nov 03 01:37:47.257379 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:37:47.257352 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e445366f56173baa16d4e190841a3e0dff3eaffcbe18c6d11775c966f22c74fe\": container with ID starting with e445366f56173baa16d4e190841a3e0dff3eaffcbe18c6d11775c966f22c74fe not found: ID does not exist" containerID="e445366f56173baa16d4e190841a3e0dff3eaffcbe18c6d11775c966f22c74fe" Nov 03 01:37:47.257472 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:47.257392 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e445366f56173baa16d4e190841a3e0dff3eaffcbe18c6d11775c966f22c74fe"} err="failed to get container status \"e445366f56173baa16d4e190841a3e0dff3eaffcbe18c6d11775c966f22c74fe\": rpc error: code = NotFound desc = could not find container \"e445366f56173baa16d4e190841a3e0dff3eaffcbe18c6d11775c966f22c74fe\": container with ID starting with e445366f56173baa16d4e190841a3e0dff3eaffcbe18c6d11775c966f22c74fe not found: ID does not exist" Nov 03 01:37:47.257472 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:47.257418 4421 scope.go:122] "RemoveContainer" containerID="6630ccebf1f1ad6d0a9d1180bf83ad096d3e8150dcaea1b51fe21615822b338c" Nov 03 01:37:47.270253 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:47.270208 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-020af9-66449c479f-bmnx4"] Nov 03 01:37:47.271903 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:47.271881 4421 scope.go:122] "RemoveContainer" containerID="6630ccebf1f1ad6d0a9d1180bf83ad096d3e8150dcaea1b51fe21615822b338c" Nov 03 01:37:47.272357 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:37:47.272326 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6630ccebf1f1ad6d0a9d1180bf83ad096d3e8150dcaea1b51fe21615822b338c\": container with ID starting with 6630ccebf1f1ad6d0a9d1180bf83ad096d3e8150dcaea1b51fe21615822b338c not found: ID does not exist" containerID="6630ccebf1f1ad6d0a9d1180bf83ad096d3e8150dcaea1b51fe21615822b338c" Nov 03 01:37:47.272475 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:47.272362 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6630ccebf1f1ad6d0a9d1180bf83ad096d3e8150dcaea1b51fe21615822b338c"} err="failed to get container status \"6630ccebf1f1ad6d0a9d1180bf83ad096d3e8150dcaea1b51fe21615822b338c\": rpc error: code = NotFound desc = could not find container \"6630ccebf1f1ad6d0a9d1180bf83ad096d3e8150dcaea1b51fe21615822b338c\": container with ID starting with 6630ccebf1f1ad6d0a9d1180bf83ad096d3e8150dcaea1b51fe21615822b338c not found: ID does not exist" Nov 03 01:37:47.275002 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:47.274981 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-020af9-66449c479f-bmnx4"] Nov 03 01:37:47.285652 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:47.285594 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-020af9-7fcf787fd7-9fgx6"] Nov 03 01:37:47.288867 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:47.288847 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-020af9-7fcf787fd7-9fgx6"] Nov 03 01:37:47.983024 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:47.982945 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f5f8dc1-5619-4ffb-b968-06869fa59c39" path="/var/lib/kubelet/pods/7f5f8dc1-5619-4ffb-b968-06869fa59c39/volumes" Nov 03 01:37:47.983461 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:47.983446 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c671f325-44a0-497f-afb5-24f3871cc496" path="/var/lib/kubelet/pods/c671f325-44a0-497f-afb5-24f3871cc496/volumes" Nov 03 01:37:52.229170 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:52.229132 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w1-4af1f7/rate-test-model-4af1f7-kserve-6b4d9677cc-6pz6m"] Nov 03 01:37:52.230371 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:52.230207 4421 cpu_manager.go:414] "containerMap: removing container" podUID="617b1b92-68e6-47cd-ac46-2542031c7efc" containerName="revoke-keys" Nov 03 01:37:52.230371 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:52.230226 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="617b1b92-68e6-47cd-ac46-2542031c7efc" containerName="revoke-keys" Nov 03 01:37:52.230371 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:52.230239 4421 cpu_manager.go:414] "containerMap: removing container" podUID="c671f325-44a0-497f-afb5-24f3871cc496" containerName="payload-processing" Nov 03 01:37:52.230371 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:52.230245 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c671f325-44a0-497f-afb5-24f3871cc496" containerName="payload-processing" Nov 03 01:37:52.230371 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:52.230303 4421 cpu_manager.go:414] "containerMap: removing container" podUID="7f5f8dc1-5619-4ffb-b968-06869fa59c39" containerName="payload-pre-processing" Nov 03 01:37:52.230371 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:52.230312 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f5f8dc1-5619-4ffb-b968-06869fa59c39" containerName="payload-pre-processing" Nov 03 01:37:52.230718 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:52.230438 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f5f8dc1-5619-4ffb-b968-06869fa59c39" containerName="payload-pre-processing" Nov 03 01:37:52.230718 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:52.230451 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="617b1b92-68e6-47cd-ac46-2542031c7efc" containerName="revoke-keys" Nov 03 01:37:52.230718 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:52.230459 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="c671f325-44a0-497f-afb5-24f3871cc496" containerName="payload-processing" Nov 03 01:37:52.235709 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:52.235683 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-4af1f7/rate-test-model-4af1f7-kserve-6b4d9677cc-6pz6m" Nov 03 01:37:52.238334 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:52.238288 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a57251c-8bd7-449e-ac32-563fe0f99711-home\") pod \"rate-test-model-4af1f7-kserve-6b4d9677cc-6pz6m\" (UID: \"6a57251c-8bd7-449e-ac32-563fe0f99711\") " pod="ai-tenant-e2e-shared-b-w1-4af1f7/rate-test-model-4af1f7-kserve-6b4d9677cc-6pz6m" Nov 03 01:37:52.238473 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:52.238420 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lstp\" (UniqueName: \"kubernetes.io/projected/6a57251c-8bd7-449e-ac32-563fe0f99711-kube-api-access-5lstp\") pod \"rate-test-model-4af1f7-kserve-6b4d9677cc-6pz6m\" (UID: \"6a57251c-8bd7-449e-ac32-563fe0f99711\") " pod="ai-tenant-e2e-shared-b-w1-4af1f7/rate-test-model-4af1f7-kserve-6b4d9677cc-6pz6m" Nov 03 01:37:52.238553 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:52.238477 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a57251c-8bd7-449e-ac32-563fe0f99711-model-cache\") pod \"rate-test-model-4af1f7-kserve-6b4d9677cc-6pz6m\" (UID: \"6a57251c-8bd7-449e-ac32-563fe0f99711\") " pod="ai-tenant-e2e-shared-b-w1-4af1f7/rate-test-model-4af1f7-kserve-6b4d9677cc-6pz6m" Nov 03 01:37:52.238553 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:52.238529 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a57251c-8bd7-449e-ac32-563fe0f99711-tmp-dir\") pod \"rate-test-model-4af1f7-kserve-6b4d9677cc-6pz6m\" (UID: \"6a57251c-8bd7-449e-ac32-563fe0f99711\") " pod="ai-tenant-e2e-shared-b-w1-4af1f7/rate-test-model-4af1f7-kserve-6b4d9677cc-6pz6m" Nov 03 01:37:52.238669 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:52.238565 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a57251c-8bd7-449e-ac32-563fe0f99711-tls-certs\") pod \"rate-test-model-4af1f7-kserve-6b4d9677cc-6pz6m\" (UID: \"6a57251c-8bd7-449e-ac32-563fe0f99711\") " pod="ai-tenant-e2e-shared-b-w1-4af1f7/rate-test-model-4af1f7-kserve-6b4d9677cc-6pz6m" Nov 03 01:37:52.238669 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:52.238661 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a57251c-8bd7-449e-ac32-563fe0f99711-dshm\") pod \"rate-test-model-4af1f7-kserve-6b4d9677cc-6pz6m\" (UID: \"6a57251c-8bd7-449e-ac32-563fe0f99711\") " pod="ai-tenant-e2e-shared-b-w1-4af1f7/rate-test-model-4af1f7-kserve-6b4d9677cc-6pz6m" Nov 03 01:37:52.271302 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:52.254694 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-4af1f7\"/\"openshift-service-ca.crt\"" Nov 03 01:37:52.271302 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:52.255391 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-4af1f7\"/\"kube-root-ca.crt\"" Nov 03 01:37:52.271302 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:52.255499 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-4af1f7\"/\"default-dockercfg-6c6n6\"" Nov 03 01:37:52.271302 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:52.255601 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-4af1f7\"/\"rate-test-model-4af1f7-kserve-self-signed-certs\"" Nov 03 01:37:52.271302 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:52.257096 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-4af1f7/rate-test-model-4af1f7-kserve-6b4d9677cc-6pz6m"] Nov 03 01:37:52.339941 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:52.339907 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a57251c-8bd7-449e-ac32-563fe0f99711-home\") pod \"rate-test-model-4af1f7-kserve-6b4d9677cc-6pz6m\" (UID: \"6a57251c-8bd7-449e-ac32-563fe0f99711\") " pod="ai-tenant-e2e-shared-b-w1-4af1f7/rate-test-model-4af1f7-kserve-6b4d9677cc-6pz6m" Nov 03 01:37:52.340130 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:52.339998 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lstp\" (UniqueName: \"kubernetes.io/projected/6a57251c-8bd7-449e-ac32-563fe0f99711-kube-api-access-5lstp\") pod \"rate-test-model-4af1f7-kserve-6b4d9677cc-6pz6m\" (UID: \"6a57251c-8bd7-449e-ac32-563fe0f99711\") " pod="ai-tenant-e2e-shared-b-w1-4af1f7/rate-test-model-4af1f7-kserve-6b4d9677cc-6pz6m" Nov 03 01:37:52.340130 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:52.340036 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a57251c-8bd7-449e-ac32-563fe0f99711-model-cache\") pod \"rate-test-model-4af1f7-kserve-6b4d9677cc-6pz6m\" (UID: \"6a57251c-8bd7-449e-ac32-563fe0f99711\") " pod="ai-tenant-e2e-shared-b-w1-4af1f7/rate-test-model-4af1f7-kserve-6b4d9677cc-6pz6m" Nov 03 01:37:52.340259 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:52.340178 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a57251c-8bd7-449e-ac32-563fe0f99711-tmp-dir\") pod \"rate-test-model-4af1f7-kserve-6b4d9677cc-6pz6m\" (UID: \"6a57251c-8bd7-449e-ac32-563fe0f99711\") " pod="ai-tenant-e2e-shared-b-w1-4af1f7/rate-test-model-4af1f7-kserve-6b4d9677cc-6pz6m" Nov 03 01:37:52.340259 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:52.340229 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a57251c-8bd7-449e-ac32-563fe0f99711-tls-certs\") pod \"rate-test-model-4af1f7-kserve-6b4d9677cc-6pz6m\" (UID: \"6a57251c-8bd7-449e-ac32-563fe0f99711\") " pod="ai-tenant-e2e-shared-b-w1-4af1f7/rate-test-model-4af1f7-kserve-6b4d9677cc-6pz6m" Nov 03 01:37:52.340387 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:52.340299 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a57251c-8bd7-449e-ac32-563fe0f99711-home\") pod \"rate-test-model-4af1f7-kserve-6b4d9677cc-6pz6m\" (UID: \"6a57251c-8bd7-449e-ac32-563fe0f99711\") " pod="ai-tenant-e2e-shared-b-w1-4af1f7/rate-test-model-4af1f7-kserve-6b4d9677cc-6pz6m" Nov 03 01:37:52.340387 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:52.340365 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a57251c-8bd7-449e-ac32-563fe0f99711-model-cache\") pod \"rate-test-model-4af1f7-kserve-6b4d9677cc-6pz6m\" (UID: \"6a57251c-8bd7-449e-ac32-563fe0f99711\") " pod="ai-tenant-e2e-shared-b-w1-4af1f7/rate-test-model-4af1f7-kserve-6b4d9677cc-6pz6m" Nov 03 01:37:52.340387 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:52.340380 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a57251c-8bd7-449e-ac32-563fe0f99711-dshm\") pod \"rate-test-model-4af1f7-kserve-6b4d9677cc-6pz6m\" (UID: \"6a57251c-8bd7-449e-ac32-563fe0f99711\") " pod="ai-tenant-e2e-shared-b-w1-4af1f7/rate-test-model-4af1f7-kserve-6b4d9677cc-6pz6m" Nov 03 01:37:52.340533 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:52.340498 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a57251c-8bd7-449e-ac32-563fe0f99711-tmp-dir\") pod \"rate-test-model-4af1f7-kserve-6b4d9677cc-6pz6m\" (UID: \"6a57251c-8bd7-449e-ac32-563fe0f99711\") " pod="ai-tenant-e2e-shared-b-w1-4af1f7/rate-test-model-4af1f7-kserve-6b4d9677cc-6pz6m" Nov 03 01:37:52.342644 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:52.342625 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a57251c-8bd7-449e-ac32-563fe0f99711-dshm\") pod \"rate-test-model-4af1f7-kserve-6b4d9677cc-6pz6m\" (UID: \"6a57251c-8bd7-449e-ac32-563fe0f99711\") " pod="ai-tenant-e2e-shared-b-w1-4af1f7/rate-test-model-4af1f7-kserve-6b4d9677cc-6pz6m" Nov 03 01:37:52.342907 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:52.342889 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a57251c-8bd7-449e-ac32-563fe0f99711-tls-certs\") pod \"rate-test-model-4af1f7-kserve-6b4d9677cc-6pz6m\" (UID: \"6a57251c-8bd7-449e-ac32-563fe0f99711\") " pod="ai-tenant-e2e-shared-b-w1-4af1f7/rate-test-model-4af1f7-kserve-6b4d9677cc-6pz6m" Nov 03 01:37:52.349446 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:52.349419 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lstp\" (UniqueName: \"kubernetes.io/projected/6a57251c-8bd7-449e-ac32-563fe0f99711-kube-api-access-5lstp\") pod \"rate-test-model-4af1f7-kserve-6b4d9677cc-6pz6m\" (UID: \"6a57251c-8bd7-449e-ac32-563fe0f99711\") " pod="ai-tenant-e2e-shared-b-w1-4af1f7/rate-test-model-4af1f7-kserve-6b4d9677cc-6pz6m" Nov 03 01:37:52.590160 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:52.590119 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-4af1f7/rate-test-model-4af1f7-kserve-6b4d9677cc-6pz6m" Nov 03 01:37:52.882840 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:52.882809 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-4af1f7/rate-test-model-4af1f7-kserve-6b4d9677cc-6pz6m"] Nov 03 01:37:52.883839 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:37:52.883810 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6a57251c_8bd7_449e_ac32_563fe0f99711.slice/crio-dadabe136be9daf89ca24bcd994fea9f410befca57443bbf066fe568fca43540 WatchSource:0}: Error finding container dadabe136be9daf89ca24bcd994fea9f410befca57443bbf066fe568fca43540: Status 404 returned error can't find the container with id dadabe136be9daf89ca24bcd994fea9f410befca57443bbf066fe568fca43540 Nov 03 01:37:53.317013 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:53.316970 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-4af1f7/rate-test-model-4af1f7-kserve-6b4d9677cc-6pz6m" event={"ID":"6a57251c-8bd7-449e-ac32-563fe0f99711","Type":"ContainerStarted","Data":"d21569a00857a12296071ac8e1289a495e00f991f7d3ca8cf15ece2a0739934a"} Nov 03 01:37:53.317013 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:53.317017 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-4af1f7/rate-test-model-4af1f7-kserve-6b4d9677cc-6pz6m" event={"ID":"6a57251c-8bd7-449e-ac32-563fe0f99711","Type":"ContainerStarted","Data":"dadabe136be9daf89ca24bcd994fea9f410befca57443bbf066fe568fca43540"} Nov 03 01:37:53.335783 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:53.335726 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w1-4af1f7/rate-test-model-4af1f7-kserve-6b4d9677cc-6pz6m" podStartSLOduration=1.335709399 podStartE2EDuration="1.335709399s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:37:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:37:53.332422985 +0000 UTC m=+3415.503426083" watchObservedRunningTime="2026-11-03 01:37:53.335709399 +0000 UTC m=+3415.506712495" Nov 03 01:37:53.590937 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:53.590851 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w1-4af1f7/rate-test-model-4af1f7-kserve-6b4d9677cc-6pz6m" Nov 03 01:37:57.872637 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:57.872603 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9fa9240-slh8k"] Nov 03 01:37:57.877119 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:57.877095 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9fa9240-slh8k" Nov 03 01:37:57.883490 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:57.883462 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9fa9240-slh8k"] Nov 03 01:37:57.905600 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:57.905570 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6q4xp\"" Nov 03 01:37:57.905600 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:57.905572 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:37:57.934217 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:57.934185 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10544138-8e08-412d-9551-6e31301544ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c9fa9240-slh8k\" (UID: \"10544138-8e08-412d-9551-6e31301544ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9fa9240-slh8k" Nov 03 01:37:57.934445 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:57.934384 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2d5sc\" (UniqueName: \"kubernetes.io/projected/10544138-8e08-412d-9551-6e31301544ee-kube-api-access-2d5sc\") pod \"maas-api-revoke-sub-c9fa9240-slh8k\" (UID: \"10544138-8e08-412d-9551-6e31301544ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9fa9240-slh8k" Nov 03 01:37:57.934582 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:57.934563 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10544138-8e08-412d-9551-6e31301544ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c9fa9240-slh8k\" (UID: \"10544138-8e08-412d-9551-6e31301544ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9fa9240-slh8k" Nov 03 01:37:58.036484 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:58.036444 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10544138-8e08-412d-9551-6e31301544ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c9fa9240-slh8k\" (UID: \"10544138-8e08-412d-9551-6e31301544ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9fa9240-slh8k" Nov 03 01:37:58.036772 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:58.036639 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2d5sc\" (UniqueName: \"kubernetes.io/projected/10544138-8e08-412d-9551-6e31301544ee-kube-api-access-2d5sc\") pod \"maas-api-revoke-sub-c9fa9240-slh8k\" (UID: \"10544138-8e08-412d-9551-6e31301544ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9fa9240-slh8k" Nov 03 01:37:58.037093 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:58.037070 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10544138-8e08-412d-9551-6e31301544ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c9fa9240-slh8k\" (UID: \"10544138-8e08-412d-9551-6e31301544ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9fa9240-slh8k" Nov 03 01:37:58.037190 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:58.037156 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10544138-8e08-412d-9551-6e31301544ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c9fa9240-slh8k\" (UID: \"10544138-8e08-412d-9551-6e31301544ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9fa9240-slh8k" Nov 03 01:37:58.039969 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:58.039935 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10544138-8e08-412d-9551-6e31301544ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c9fa9240-slh8k\" (UID: \"10544138-8e08-412d-9551-6e31301544ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9fa9240-slh8k" Nov 03 01:37:58.044853 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:58.044827 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2d5sc\" (UniqueName: \"kubernetes.io/projected/10544138-8e08-412d-9551-6e31301544ee-kube-api-access-2d5sc\") pod \"maas-api-revoke-sub-c9fa9240-slh8k\" (UID: \"10544138-8e08-412d-9551-6e31301544ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9fa9240-slh8k" Nov 03 01:37:58.100755 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:58.100724 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-2f9a3a-8697684787-rcsf2"] Nov 03 01:37:58.122752 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:58.122648 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-2f9a3a-78c44fb86c-ckvcs"] Nov 03 01:37:58.195885 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:58.195852 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9fa9240-slh8k" Nov 03 01:37:58.529626 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:58.529580 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9fa9240-slh8k"] Nov 03 01:37:58.532630 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:37:58.532539 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10544138_8e08_412d_9551_6e31301544ee.slice/crio-acf097d5e00f427c88d000b098d49d1cd98b025af4bdc9cd1ddac73183b861a5 WatchSource:0}: Error finding container acf097d5e00f427c88d000b098d49d1cd98b025af4bdc9cd1ddac73183b861a5: Status 404 returned error can't find the container with id acf097d5e00f427c88d000b098d49d1cd98b025af4bdc9cd1ddac73183b861a5 Nov 03 01:37:59.028401 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:59.028364 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2af124a-x8h45"] Nov 03 01:37:59.033413 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:59.033384 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2af124a-x8h45"] Nov 03 01:37:59.430570 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:59.430526 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9fa9240-slh8k" event={"ID":"10544138-8e08-412d-9551-6e31301544ee","Type":"ContainerStarted","Data":"88c028d3c4647c20692797a7273d01a7ea3fc7ad08b3678980326c23faca4651"} Nov 03 01:37:59.430786 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:59.430596 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9fa9240-slh8k" event={"ID":"10544138-8e08-412d-9551-6e31301544ee","Type":"ContainerStarted","Data":"acf097d5e00f427c88d000b098d49d1cd98b025af4bdc9cd1ddac73183b861a5"} Nov 03 01:37:59.446954 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:59.446893 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9fa9240-slh8k" podStartSLOduration=2.446874079 podStartE2EDuration="2.446874079s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:37:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:37:59.446169272 +0000 UTC m=+3421.617172368" watchObservedRunningTime="2026-11-03 01:37:59.446874079 +0000 UTC m=+3421.617877176" Nov 03 01:37:59.981731 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:37:59.981684 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e663e56-bfe1-48b5-bc2d-b1b56b20595a" path="/var/lib/kubelet/pods/3e663e56-bfe1-48b5-bc2d-b1b56b20595a/volumes" Nov 03 01:38:00.439230 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:00.439196 4421 generic.go:356] "Generic (PLEG): container finished" podID="10544138-8e08-412d-9551-6e31301544ee" containerID="88c028d3c4647c20692797a7273d01a7ea3fc7ad08b3678980326c23faca4651" exitCode=0 Nov 03 01:38:00.439659 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:00.439294 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9fa9240-slh8k" event={"ID":"10544138-8e08-412d-9551-6e31301544ee","Type":"ContainerDied","Data":"88c028d3c4647c20692797a7273d01a7ea3fc7ad08b3678980326c23faca4651"} Nov 03 01:38:01.608836 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:01.608810 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9fa9240-slh8k" Nov 03 01:38:01.724583 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:01.724480 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10544138-8e08-412d-9551-6e31301544ee-kube-api-access-2d5sc\" (UniqueName: \"kubernetes.io/projected/10544138-8e08-412d-9551-6e31301544ee-kube-api-access-2d5sc\") pod \"10544138-8e08-412d-9551-6e31301544ee\" (UID: \"10544138-8e08-412d-9551-6e31301544ee\") " Nov 03 01:38:01.724583 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:01.724571 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/10544138-8e08-412d-9551-6e31301544ee-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10544138-8e08-412d-9551-6e31301544ee-maas-api-xks-ca\") pod \"10544138-8e08-412d-9551-6e31301544ee\" (UID: \"10544138-8e08-412d-9551-6e31301544ee\") " Nov 03 01:38:01.724794 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:01.724731 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10544138-8e08-412d-9551-6e31301544ee-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10544138-8e08-412d-9551-6e31301544ee-maas-api-openshift-ca\") pod \"10544138-8e08-412d-9551-6e31301544ee\" (UID: \"10544138-8e08-412d-9551-6e31301544ee\") " Nov 03 01:38:01.725094 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:01.725058 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10544138-8e08-412d-9551-6e31301544ee-maas-api-openshift-ca" pod "10544138-8e08-412d-9551-6e31301544ee" (UID: "10544138-8e08-412d-9551-6e31301544ee"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:38:01.726724 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:01.726703 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10544138-8e08-412d-9551-6e31301544ee-maas-api-xks-ca" pod "10544138-8e08-412d-9551-6e31301544ee" (UID: "10544138-8e08-412d-9551-6e31301544ee"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:38:01.726880 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:01.726860 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10544138-8e08-412d-9551-6e31301544ee-kube-api-access-2d5sc" pod "10544138-8e08-412d-9551-6e31301544ee" (UID: "10544138-8e08-412d-9551-6e31301544ee"). InnerVolumeSpecName "kube-api-access-2d5sc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:38:01.826061 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:01.826030 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10544138-8e08-412d-9551-6e31301544ee-maas-api-openshift-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:38:01.826061 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:01.826058 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2d5sc\" (UniqueName: \"kubernetes.io/projected/10544138-8e08-412d-9551-6e31301544ee-kube-api-access-2d5sc\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:38:01.826061 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:01.826069 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10544138-8e08-412d-9551-6e31301544ee-maas-api-xks-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:38:02.456288 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:02.456242 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9fa9240-slh8k" Nov 03 01:38:02.456490 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:02.456313 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9fa9240-slh8k" event={"ID":"10544138-8e08-412d-9551-6e31301544ee","Type":"ContainerDied","Data":"acf097d5e00f427c88d000b098d49d1cd98b025af4bdc9cd1ddac73183b861a5"} Nov 03 01:38:02.456490 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:02.456366 4421 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="acf097d5e00f427c88d000b098d49d1cd98b025af4bdc9cd1ddac73183b861a5" Nov 03 01:38:02.590573 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:02.590536 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w1-4af1f7/rate-test-model-4af1f7-kserve-6b4d9677cc-6pz6m" Nov 03 01:38:02.603489 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:02.603458 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w1-4af1f7/rate-test-model-4af1f7-kserve-6b4d9677cc-6pz6m" Nov 03 01:38:03.101385 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:03.101341 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-2f9a3a-8697684787-rcsf2" podUID="d2cd8dba-bbe9-4bdf-9fca-94a4cb839ca1" containerName="payload-processing" containerID="cri-o://30dfeaf9971d86ed46f702780413f66434be9d5d6f3dc72379b8636c23c4ce0f" gracePeriod=25 Nov 03 01:38:03.123715 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:03.123674 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2f9a3a-78c44fb86c-ckvcs" podUID="d3b636b3-5a2e-467d-a364-f8c57824521f" containerName="payload-pre-processing" containerID="cri-o://12283813be518b911286b92c20e397bc64157fd0595a93b0fe1a238b297563da" gracePeriod=25 Nov 03 01:38:03.475673 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:03.475591 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w1-4af1f7/rate-test-model-4af1f7-kserve-6b4d9677cc-6pz6m" Nov 03 01:38:05.968562 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:05.968530 4421 scope.go:122] "RemoveContainer" containerID="abcc3adc4d6cdff2cfb43a436232425ed085e2f44b8634aac12f68463233d03c" Nov 03 01:38:10.056199 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:10.056157 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-2f9a3a-8697684787-rcsf2" podUID="d2cd8dba-bbe9-4bdf-9fca-94a4cb839ca1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:38:10.061084 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:10.061041 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2f9a3a-78c44fb86c-ckvcs" podUID="d3b636b3-5a2e-467d-a364-f8c57824521f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:38:10.064863 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:10.064830 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b3fb80d-cvnnn"] Nov 03 01:38:10.069182 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:10.069151 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b3fb80d-cvnnn"] Nov 03 01:38:11.983092 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:11.983050 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1da4b72f-2094-43a3-b99a-1da5849d1082" path="/var/lib/kubelet/pods/1da4b72f-2094-43a3-b99a-1da5849d1082/volumes" Nov 03 01:38:14.671172 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:38:14.671129 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 01:38:20.055254 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:20.055212 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-2f9a3a-8697684787-rcsf2" podUID="d2cd8dba-bbe9-4bdf-9fca-94a4cb839ca1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:38:20.055691 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:20.055264 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2f9a3a-78c44fb86c-ckvcs" podUID="d3b636b3-5a2e-467d-a364-f8c57824521f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:38:23.527742 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:23.527714 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-2f9a3a-8697684787-rcsf2" Nov 03 01:38:23.531809 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:23.531734 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2f9a3a-78c44fb86c-ckvcs" Nov 03 01:38:23.667794 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:23.667704 4421 generic.go:356] "Generic (PLEG): container finished" podID="d3b636b3-5a2e-467d-a364-f8c57824521f" containerID="12283813be518b911286b92c20e397bc64157fd0595a93b0fe1a238b297563da" exitCode=0 Nov 03 01:38:23.667794 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:23.667742 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2f9a3a-78c44fb86c-ckvcs" event={"ID":"d3b636b3-5a2e-467d-a364-f8c57824521f","Type":"ContainerDied","Data":"12283813be518b911286b92c20e397bc64157fd0595a93b0fe1a238b297563da"} Nov 03 01:38:23.667794 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:23.667773 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2f9a3a-78c44fb86c-ckvcs" Nov 03 01:38:23.668090 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:23.667814 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2f9a3a-78c44fb86c-ckvcs" event={"ID":"d3b636b3-5a2e-467d-a364-f8c57824521f","Type":"ContainerDied","Data":"4cde18fe17352751e54fbe1c8d57e355717ed1e33825f016e943308f467b4fd2"} Nov 03 01:38:23.668090 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:23.667846 4421 scope.go:122] "RemoveContainer" containerID="12283813be518b911286b92c20e397bc64157fd0595a93b0fe1a238b297563da" Nov 03 01:38:23.669350 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:23.669291 4421 generic.go:356] "Generic (PLEG): container finished" podID="d2cd8dba-bbe9-4bdf-9fca-94a4cb839ca1" containerID="30dfeaf9971d86ed46f702780413f66434be9d5d6f3dc72379b8636c23c4ce0f" exitCode=0 Nov 03 01:38:23.669485 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:23.669387 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-2f9a3a-8697684787-rcsf2" Nov 03 01:38:23.669485 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:23.669413 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-2f9a3a-8697684787-rcsf2" event={"ID":"d2cd8dba-bbe9-4bdf-9fca-94a4cb839ca1","Type":"ContainerDied","Data":"30dfeaf9971d86ed46f702780413f66434be9d5d6f3dc72379b8636c23c4ce0f"} Nov 03 01:38:23.669485 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:23.669467 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-2f9a3a-8697684787-rcsf2" event={"ID":"d2cd8dba-bbe9-4bdf-9fca-94a4cb839ca1","Type":"ContainerDied","Data":"772e19accab7237d80f147fe70a9857ee131c89af669bb0c68939b8b831e9eb9"} Nov 03 01:38:23.676600 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:23.676571 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3b636b3-5a2e-467d-a364-f8c57824521f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3b636b3-5a2e-467d-a364-f8c57824521f-plugins-config-volume\") pod \"d3b636b3-5a2e-467d-a364-f8c57824521f\" (UID: \"d3b636b3-5a2e-467d-a364-f8c57824521f\") " Nov 03 01:38:23.676750 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:23.676687 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2cd8dba-bbe9-4bdf-9fca-94a4cb839ca1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2cd8dba-bbe9-4bdf-9fca-94a4cb839ca1-plugins-config-volume\") pod \"d2cd8dba-bbe9-4bdf-9fca-94a4cb839ca1\" (UID: \"d2cd8dba-bbe9-4bdf-9fca-94a4cb839ca1\") " Nov 03 01:38:23.676839 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:23.676815 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2cd8dba-bbe9-4bdf-9fca-94a4cb839ca1-kube-api-access-rjst9\" (UniqueName: \"kubernetes.io/projected/d2cd8dba-bbe9-4bdf-9fca-94a4cb839ca1-kube-api-access-rjst9\") pod \"d2cd8dba-bbe9-4bdf-9fca-94a4cb839ca1\" (UID: \"d2cd8dba-bbe9-4bdf-9fca-94a4cb839ca1\") " Nov 03 01:38:23.676958 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:23.676867 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3b636b3-5a2e-467d-a364-f8c57824521f-kube-api-access-bx6zw\" (UniqueName: \"kubernetes.io/projected/d3b636b3-5a2e-467d-a364-f8c57824521f-kube-api-access-bx6zw\") pod \"d3b636b3-5a2e-467d-a364-f8c57824521f\" (UID: \"d3b636b3-5a2e-467d-a364-f8c57824521f\") " Nov 03 01:38:23.677037 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:23.677012 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3b636b3-5a2e-467d-a364-f8c57824521f-plugins-config-volume" pod "d3b636b3-5a2e-467d-a364-f8c57824521f" (UID: "d3b636b3-5a2e-467d-a364-f8c57824521f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:38:23.677499 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:23.677474 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2cd8dba-bbe9-4bdf-9fca-94a4cb839ca1-plugins-config-volume" pod "d2cd8dba-bbe9-4bdf-9fca-94a4cb839ca1" (UID: "d2cd8dba-bbe9-4bdf-9fca-94a4cb839ca1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:38:23.677614 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:23.677488 4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3b636b3-5a2e-467d-a364-f8c57824521f-plugins-config-volume\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:38:23.679547 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:23.679518 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2cd8dba-bbe9-4bdf-9fca-94a4cb839ca1-kube-api-access-rjst9" pod "d2cd8dba-bbe9-4bdf-9fca-94a4cb839ca1" (UID: "d2cd8dba-bbe9-4bdf-9fca-94a4cb839ca1"). InnerVolumeSpecName "kube-api-access-rjst9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:38:23.679695 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:23.679574 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3b636b3-5a2e-467d-a364-f8c57824521f-kube-api-access-bx6zw" pod "d3b636b3-5a2e-467d-a364-f8c57824521f" (UID: "d3b636b3-5a2e-467d-a364-f8c57824521f"). InnerVolumeSpecName "kube-api-access-bx6zw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:38:23.684446 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:23.684423 4421 scope.go:122] "RemoveContainer" containerID="12283813be518b911286b92c20e397bc64157fd0595a93b0fe1a238b297563da" Nov 03 01:38:23.684838 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:38:23.684816 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12283813be518b911286b92c20e397bc64157fd0595a93b0fe1a238b297563da\": container with ID starting with 12283813be518b911286b92c20e397bc64157fd0595a93b0fe1a238b297563da not found: ID does not exist" containerID="12283813be518b911286b92c20e397bc64157fd0595a93b0fe1a238b297563da" Nov 03 01:38:23.684916 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:23.684850 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12283813be518b911286b92c20e397bc64157fd0595a93b0fe1a238b297563da"} err="failed to get container status \"12283813be518b911286b92c20e397bc64157fd0595a93b0fe1a238b297563da\": rpc error: code = NotFound desc = could not find container \"12283813be518b911286b92c20e397bc64157fd0595a93b0fe1a238b297563da\": container with ID starting with 12283813be518b911286b92c20e397bc64157fd0595a93b0fe1a238b297563da not found: ID does not exist" Nov 03 01:38:23.684916 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:23.684870 4421 scope.go:122] "RemoveContainer" containerID="30dfeaf9971d86ed46f702780413f66434be9d5d6f3dc72379b8636c23c4ce0f" Nov 03 01:38:23.698018 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:23.697999 4421 scope.go:122] "RemoveContainer" containerID="30dfeaf9971d86ed46f702780413f66434be9d5d6f3dc72379b8636c23c4ce0f" Nov 03 01:38:23.698342 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:38:23.698311 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30dfeaf9971d86ed46f702780413f66434be9d5d6f3dc72379b8636c23c4ce0f\": container with ID starting with 30dfeaf9971d86ed46f702780413f66434be9d5d6f3dc72379b8636c23c4ce0f not found: ID does not exist" containerID="30dfeaf9971d86ed46f702780413f66434be9d5d6f3dc72379b8636c23c4ce0f" Nov 03 01:38:23.698441 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:23.698340 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30dfeaf9971d86ed46f702780413f66434be9d5d6f3dc72379b8636c23c4ce0f"} err="failed to get container status \"30dfeaf9971d86ed46f702780413f66434be9d5d6f3dc72379b8636c23c4ce0f\": rpc error: code = NotFound desc = could not find container \"30dfeaf9971d86ed46f702780413f66434be9d5d6f3dc72379b8636c23c4ce0f\": container with ID starting with 30dfeaf9971d86ed46f702780413f66434be9d5d6f3dc72379b8636c23c4ce0f not found: ID does not exist" Nov 03 01:38:23.778782 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:23.778734 4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2cd8dba-bbe9-4bdf-9fca-94a4cb839ca1-plugins-config-volume\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:38:23.778782 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:23.778770 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rjst9\" (UniqueName: \"kubernetes.io/projected/d2cd8dba-bbe9-4bdf-9fca-94a4cb839ca1-kube-api-access-rjst9\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:38:23.778782 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:23.778784 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bx6zw\" (UniqueName: \"kubernetes.io/projected/d3b636b3-5a2e-467d-a364-f8c57824521f-kube-api-access-bx6zw\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:38:24.005970 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:24.005881 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-2f9a3a-78c44fb86c-ckvcs"] Nov 03 01:38:24.009584 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:24.009558 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-2f9a3a-78c44fb86c-ckvcs"] Nov 03 01:38:24.019985 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:24.019956 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-2f9a3a-8697684787-rcsf2"] Nov 03 01:38:24.023900 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:24.023879 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-2f9a3a-8697684787-rcsf2"] Nov 03 01:38:25.981397 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:25.981363 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2cd8dba-bbe9-4bdf-9fca-94a4cb839ca1" path="/var/lib/kubelet/pods/d2cd8dba-bbe9-4bdf-9fca-94a4cb839ca1/volumes" Nov 03 01:38:25.982427 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:25.982408 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3b636b3-5a2e-467d-a364-f8c57824521f" path="/var/lib/kubelet/pods/d3b636b3-5a2e-467d-a364-f8c57824521f/volumes" Nov 03 01:38:28.953389 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:28.953351 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xmmdd"] Nov 03 01:38:28.953952 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:28.953601 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xmmdd" podUID="c931c147-75a3-4a38-99ff-26d0aa98a38a" containerName="istio-proxy" containerID="cri-o://27c7b0cb36fd8ff37135e7f6e5af8b67d75d1f1e3376f811659e6845627e8df6" gracePeriod=30 Nov 03 01:38:32.853413 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:32.853379 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-ztm5t"] Nov 03 01:38:32.855102 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:32.855053 4421 cpu_manager.go:414] "containerMap: removing container" podUID="d2cd8dba-bbe9-4bdf-9fca-94a4cb839ca1" containerName="payload-processing" Nov 03 01:38:32.856522 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:32.856492 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2cd8dba-bbe9-4bdf-9fca-94a4cb839ca1" containerName="payload-processing" Nov 03 01:38:32.856674 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:32.856629 4421 cpu_manager.go:414] "containerMap: removing container" podUID="d3b636b3-5a2e-467d-a364-f8c57824521f" containerName="payload-pre-processing" Nov 03 01:38:32.856674 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:32.856641 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3b636b3-5a2e-467d-a364-f8c57824521f" containerName="payload-pre-processing" Nov 03 01:38:32.856674 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:32.856660 4421 cpu_manager.go:414] "containerMap: removing container" podUID="10544138-8e08-412d-9551-6e31301544ee" containerName="revoke-keys" Nov 03 01:38:32.856674 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:32.856669 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10544138-8e08-412d-9551-6e31301544ee" containerName="revoke-keys" Nov 03 01:38:32.856903 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:32.856885 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3b636b3-5a2e-467d-a364-f8c57824521f" containerName="payload-pre-processing" Nov 03 01:38:32.856958 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:32.856902 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="10544138-8e08-412d-9551-6e31301544ee" containerName="revoke-keys" Nov 03 01:38:32.856958 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:32.856914 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2cd8dba-bbe9-4bdf-9fca-94a4cb839ca1" containerName="payload-processing" Nov 03 01:38:32.864851 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:32.864824 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-ztm5t"] Nov 03 01:38:32.865785 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:32.865705 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-ztm5t" Nov 03 01:38:32.870079 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:32.870049 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-2729f2-openshift-default-dockercfg-9nk2m\"" Nov 03 01:38:32.942398 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:32.942358 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc7f30da-8b1e-42d7-a41a-2398e66263b7-istio-data\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-ztm5t\" (UID: \"dc7f30da-8b1e-42d7-a41a-2398e66263b7\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-ztm5t" Nov 03 01:38:32.942577 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:32.942412 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vx6k\" (UniqueName: \"kubernetes.io/projected/dc7f30da-8b1e-42d7-a41a-2398e66263b7-kube-api-access-4vx6k\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-ztm5t\" (UID: \"dc7f30da-8b1e-42d7-a41a-2398e66263b7\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-ztm5t" Nov 03 01:38:32.942577 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:32.942481 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc7f30da-8b1e-42d7-a41a-2398e66263b7-istio-envoy\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-ztm5t\" (UID: \"dc7f30da-8b1e-42d7-a41a-2398e66263b7\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-ztm5t" Nov 03 01:38:32.942662 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:32.942606 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc7f30da-8b1e-42d7-a41a-2398e66263b7-workload-certs\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-ztm5t\" (UID: \"dc7f30da-8b1e-42d7-a41a-2398e66263b7\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-ztm5t" Nov 03 01:38:32.942751 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:32.942698 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc7f30da-8b1e-42d7-a41a-2398e66263b7-istiod-ca-cert\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-ztm5t\" (UID: \"dc7f30da-8b1e-42d7-a41a-2398e66263b7\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-ztm5t" Nov 03 01:38:32.942751 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:32.942718 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc7f30da-8b1e-42d7-a41a-2398e66263b7-istio-token\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-ztm5t\" (UID: \"dc7f30da-8b1e-42d7-a41a-2398e66263b7\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-ztm5t" Nov 03 01:38:32.942826 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:32.942788 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc7f30da-8b1e-42d7-a41a-2398e66263b7-workload-socket\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-ztm5t\" (UID: \"dc7f30da-8b1e-42d7-a41a-2398e66263b7\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-ztm5t" Nov 03 01:38:32.942863 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:32.942840 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc7f30da-8b1e-42d7-a41a-2398e66263b7-credential-socket\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-ztm5t\" (UID: \"dc7f30da-8b1e-42d7-a41a-2398e66263b7\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-ztm5t" Nov 03 01:38:32.942938 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:32.942915 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc7f30da-8b1e-42d7-a41a-2398e66263b7-istio-podinfo\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-ztm5t\" (UID: \"dc7f30da-8b1e-42d7-a41a-2398e66263b7\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-ztm5t" Nov 03 01:38:33.044506 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:33.044472 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc7f30da-8b1e-42d7-a41a-2398e66263b7-credential-socket\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-ztm5t\" (UID: \"dc7f30da-8b1e-42d7-a41a-2398e66263b7\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-ztm5t" Nov 03 01:38:33.044705 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:33.044527 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc7f30da-8b1e-42d7-a41a-2398e66263b7-istio-podinfo\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-ztm5t\" (UID: \"dc7f30da-8b1e-42d7-a41a-2398e66263b7\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-ztm5t" Nov 03 01:38:33.044705 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:33.044611 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc7f30da-8b1e-42d7-a41a-2398e66263b7-istio-data\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-ztm5t\" (UID: \"dc7f30da-8b1e-42d7-a41a-2398e66263b7\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-ztm5t" Nov 03 01:38:33.044705 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:33.044651 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vx6k\" (UniqueName: \"kubernetes.io/projected/dc7f30da-8b1e-42d7-a41a-2398e66263b7-kube-api-access-4vx6k\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-ztm5t\" (UID: \"dc7f30da-8b1e-42d7-a41a-2398e66263b7\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-ztm5t" Nov 03 01:38:33.044705 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:33.044693 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc7f30da-8b1e-42d7-a41a-2398e66263b7-istio-envoy\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-ztm5t\" (UID: \"dc7f30da-8b1e-42d7-a41a-2398e66263b7\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-ztm5t" Nov 03 01:38:33.044941 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:33.044784 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc7f30da-8b1e-42d7-a41a-2398e66263b7-workload-certs\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-ztm5t\" (UID: \"dc7f30da-8b1e-42d7-a41a-2398e66263b7\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-ztm5t" Nov 03 01:38:33.044941 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:33.044859 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc7f30da-8b1e-42d7-a41a-2398e66263b7-istiod-ca-cert\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-ztm5t\" (UID: \"dc7f30da-8b1e-42d7-a41a-2398e66263b7\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-ztm5t" Nov 03 01:38:33.044941 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:33.044889 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc7f30da-8b1e-42d7-a41a-2398e66263b7-istio-token\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-ztm5t\" (UID: \"dc7f30da-8b1e-42d7-a41a-2398e66263b7\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-ztm5t" Nov 03 01:38:33.045729 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:33.044965 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc7f30da-8b1e-42d7-a41a-2398e66263b7-workload-socket\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-ztm5t\" (UID: \"dc7f30da-8b1e-42d7-a41a-2398e66263b7\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-ztm5t" Nov 03 01:38:33.045729 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:33.044992 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc7f30da-8b1e-42d7-a41a-2398e66263b7-credential-socket\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-ztm5t\" (UID: \"dc7f30da-8b1e-42d7-a41a-2398e66263b7\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-ztm5t" Nov 03 01:38:33.045729 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:33.045052 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc7f30da-8b1e-42d7-a41a-2398e66263b7-istio-data\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-ztm5t\" (UID: \"dc7f30da-8b1e-42d7-a41a-2398e66263b7\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-ztm5t" Nov 03 01:38:33.045729 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:33.045426 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc7f30da-8b1e-42d7-a41a-2398e66263b7-workload-socket\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-ztm5t\" (UID: \"dc7f30da-8b1e-42d7-a41a-2398e66263b7\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-ztm5t" Nov 03 01:38:33.045729 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:33.045528 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc7f30da-8b1e-42d7-a41a-2398e66263b7-workload-certs\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-ztm5t\" (UID: \"dc7f30da-8b1e-42d7-a41a-2398e66263b7\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-ztm5t" Nov 03 01:38:33.045729 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:33.045700 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc7f30da-8b1e-42d7-a41a-2398e66263b7-istiod-ca-cert\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-ztm5t\" (UID: \"dc7f30da-8b1e-42d7-a41a-2398e66263b7\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-ztm5t" Nov 03 01:38:33.047307 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:33.047252 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc7f30da-8b1e-42d7-a41a-2398e66263b7-istio-envoy\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-ztm5t\" (UID: \"dc7f30da-8b1e-42d7-a41a-2398e66263b7\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-ztm5t" Nov 03 01:38:33.047420 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:33.047316 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc7f30da-8b1e-42d7-a41a-2398e66263b7-istio-podinfo\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-ztm5t\" (UID: \"dc7f30da-8b1e-42d7-a41a-2398e66263b7\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-ztm5t" Nov 03 01:38:33.052698 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:33.052671 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc7f30da-8b1e-42d7-a41a-2398e66263b7-istio-token\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-ztm5t\" (UID: \"dc7f30da-8b1e-42d7-a41a-2398e66263b7\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-ztm5t" Nov 03 01:38:33.052817 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:33.052757 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vx6k\" (UniqueName: \"kubernetes.io/projected/dc7f30da-8b1e-42d7-a41a-2398e66263b7-kube-api-access-4vx6k\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-ztm5t\" (UID: \"dc7f30da-8b1e-42d7-a41a-2398e66263b7\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-ztm5t" Nov 03 01:38:33.192245 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:33.192160 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-ztm5t" Nov 03 01:38:33.481749 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:33.481719 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-ztm5t"] Nov 03 01:38:33.482017 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:38:33.481996 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc7f30da_8b1e_42d7_a41a_2398e66263b7.slice/crio-1accb0087949c36c23a1be59170db035e9e116ecffa5475b611828604992f42d WatchSource:0}: Error finding container 1accb0087949c36c23a1be59170db035e9e116ecffa5475b611828604992f42d: Status 404 returned error can't find the container with id 1accb0087949c36c23a1be59170db035e9e116ecffa5475b611828604992f42d Nov 03 01:38:33.485169 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:33.485135 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:38:33.485330 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:33.485209 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:38:33.485330 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:33.485255 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:38:33.778715 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:33.778674 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-ztm5t" event={"ID":"dc7f30da-8b1e-42d7-a41a-2398e66263b7","Type":"ContainerStarted","Data":"ba7b2267ebde4e272760c8646331a1c244e7ca03ad1093dfba7d3523ff45789a"} Nov 03 01:38:33.778900 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:33.778739 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-ztm5t" event={"ID":"dc7f30da-8b1e-42d7-a41a-2398e66263b7","Type":"ContainerStarted","Data":"1accb0087949c36c23a1be59170db035e9e116ecffa5475b611828604992f42d"} Nov 03 01:38:33.807040 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:33.806980 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-ztm5t" podStartSLOduration=1.806966324 podStartE2EDuration="1.806966324s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:38:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:38:33.80199963 +0000 UTC m=+3455.973002728" watchObservedRunningTime="2026-11-03 01:38:33.806966324 +0000 UTC m=+3455.977969420" Nov 03 01:38:34.192358 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:34.192326 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-ztm5t" Nov 03 01:38:34.193786 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:34.193758 4421 patch_prober.go:29] interesting pod/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-ztm5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.110:15021/healthz/ready\": dial tcp 10.134.1.110:15021: connect: connection refused" start-of-body= Nov 03 01:38:34.193928 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:34.193810 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-ztm5t" podUID="dc7f30da-8b1e-42d7-a41a-2398e66263b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.110:15021/healthz/ready\": dial tcp 10.134.1.110:15021: connect: connection refused" Nov 03 01:38:34.293681 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:34.293652 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xmmdd" Nov 03 01:38:34.375727 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:34.375635 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c931c147-75a3-4a38-99ff-26d0aa98a38a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c931c147-75a3-4a38-99ff-26d0aa98a38a-workload-certs\") pod \"c931c147-75a3-4a38-99ff-26d0aa98a38a\" (UID: \"c931c147-75a3-4a38-99ff-26d0aa98a38a\") " Nov 03 01:38:34.375727 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:34.375687 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c931c147-75a3-4a38-99ff-26d0aa98a38a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c931c147-75a3-4a38-99ff-26d0aa98a38a-istio-envoy\") pod \"c931c147-75a3-4a38-99ff-26d0aa98a38a\" (UID: \"c931c147-75a3-4a38-99ff-26d0aa98a38a\") " Nov 03 01:38:34.375727 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:34.375724 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c931c147-75a3-4a38-99ff-26d0aa98a38a-kube-api-access-x94sd\" (UniqueName: \"kubernetes.io/projected/c931c147-75a3-4a38-99ff-26d0aa98a38a-kube-api-access-x94sd\") pod \"c931c147-75a3-4a38-99ff-26d0aa98a38a\" (UID: \"c931c147-75a3-4a38-99ff-26d0aa98a38a\") " Nov 03 01:38:34.376042 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:34.375754 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c931c147-75a3-4a38-99ff-26d0aa98a38a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c931c147-75a3-4a38-99ff-26d0aa98a38a-credential-socket\") pod \"c931c147-75a3-4a38-99ff-26d0aa98a38a\" (UID: \"c931c147-75a3-4a38-99ff-26d0aa98a38a\") " Nov 03 01:38:34.376042 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:34.375827 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c931c147-75a3-4a38-99ff-26d0aa98a38a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c931c147-75a3-4a38-99ff-26d0aa98a38a-istio-podinfo\") pod \"c931c147-75a3-4a38-99ff-26d0aa98a38a\" (UID: \"c931c147-75a3-4a38-99ff-26d0aa98a38a\") " Nov 03 01:38:34.376042 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:34.375943 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c931c147-75a3-4a38-99ff-26d0aa98a38a-workload-certs" pod "c931c147-75a3-4a38-99ff-26d0aa98a38a" (UID: "c931c147-75a3-4a38-99ff-26d0aa98a38a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:38:34.376042 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:34.376000 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c931c147-75a3-4a38-99ff-26d0aa98a38a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c931c147-75a3-4a38-99ff-26d0aa98a38a-workload-socket\") pod \"c931c147-75a3-4a38-99ff-26d0aa98a38a\" (UID: \"c931c147-75a3-4a38-99ff-26d0aa98a38a\") " Nov 03 01:38:34.376337 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:34.376088 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c931c147-75a3-4a38-99ff-26d0aa98a38a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c931c147-75a3-4a38-99ff-26d0aa98a38a-istiod-ca-cert\") pod \"c931c147-75a3-4a38-99ff-26d0aa98a38a\" (UID: \"c931c147-75a3-4a38-99ff-26d0aa98a38a\") " Nov 03 01:38:34.376337 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:34.376182 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c931c147-75a3-4a38-99ff-26d0aa98a38a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c931c147-75a3-4a38-99ff-26d0aa98a38a-istio-data\") pod \"c931c147-75a3-4a38-99ff-26d0aa98a38a\" (UID: \"c931c147-75a3-4a38-99ff-26d0aa98a38a\") " Nov 03 01:38:34.376337 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:34.376197 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c931c147-75a3-4a38-99ff-26d0aa98a38a-credential-socket" pod "c931c147-75a3-4a38-99ff-26d0aa98a38a" (UID: "c931c147-75a3-4a38-99ff-26d0aa98a38a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:38:34.376337 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:34.376239 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c931c147-75a3-4a38-99ff-26d0aa98a38a-istio-token\" (UniqueName: \"kubernetes.io/projected/c931c147-75a3-4a38-99ff-26d0aa98a38a-istio-token\") pod \"c931c147-75a3-4a38-99ff-26d0aa98a38a\" (UID: \"c931c147-75a3-4a38-99ff-26d0aa98a38a\") " Nov 03 01:38:34.376589 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:34.376362 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c931c147-75a3-4a38-99ff-26d0aa98a38a-workload-socket" pod "c931c147-75a3-4a38-99ff-26d0aa98a38a" (UID: "c931c147-75a3-4a38-99ff-26d0aa98a38a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:38:34.376589 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:34.376525 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c931c147-75a3-4a38-99ff-26d0aa98a38a-istiod-ca-cert" pod "c931c147-75a3-4a38-99ff-26d0aa98a38a" (UID: "c931c147-75a3-4a38-99ff-26d0aa98a38a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:38:34.376723 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:34.376682 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c931c147-75a3-4a38-99ff-26d0aa98a38a-istio-data" pod "c931c147-75a3-4a38-99ff-26d0aa98a38a" (UID: "c931c147-75a3-4a38-99ff-26d0aa98a38a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:38:34.377012 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:34.376987 4421 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c931c147-75a3-4a38-99ff-26d0aa98a38a-istiod-ca-cert\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:38:34.377147 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:34.377016 4421 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c931c147-75a3-4a38-99ff-26d0aa98a38a-istio-data\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:38:34.377147 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:34.377032 4421 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c931c147-75a3-4a38-99ff-26d0aa98a38a-workload-certs\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:38:34.377147 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:34.377047 4421 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c931c147-75a3-4a38-99ff-26d0aa98a38a-credential-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:38:34.377147 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:34.377061 4421 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c931c147-75a3-4a38-99ff-26d0aa98a38a-workload-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:38:34.378315 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:34.378262 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c931c147-75a3-4a38-99ff-26d0aa98a38a-istio-podinfo" pod "c931c147-75a3-4a38-99ff-26d0aa98a38a" (UID: "c931c147-75a3-4a38-99ff-26d0aa98a38a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:38:34.378315 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:34.378303 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c931c147-75a3-4a38-99ff-26d0aa98a38a-istio-envoy" pod "c931c147-75a3-4a38-99ff-26d0aa98a38a" (UID: "c931c147-75a3-4a38-99ff-26d0aa98a38a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:38:34.378696 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:34.378673 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c931c147-75a3-4a38-99ff-26d0aa98a38a-kube-api-access-x94sd" pod "c931c147-75a3-4a38-99ff-26d0aa98a38a" (UID: "c931c147-75a3-4a38-99ff-26d0aa98a38a"). InnerVolumeSpecName "kube-api-access-x94sd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:38:34.378854 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:34.378832 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c931c147-75a3-4a38-99ff-26d0aa98a38a-istio-token" pod "c931c147-75a3-4a38-99ff-26d0aa98a38a" (UID: "c931c147-75a3-4a38-99ff-26d0aa98a38a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:38:34.478223 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:34.478185 4421 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c931c147-75a3-4a38-99ff-26d0aa98a38a-istio-token\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:38:34.478223 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:34.478218 4421 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c931c147-75a3-4a38-99ff-26d0aa98a38a-istio-envoy\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:38:34.478223 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:34.478229 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x94sd\" (UniqueName: \"kubernetes.io/projected/c931c147-75a3-4a38-99ff-26d0aa98a38a-kube-api-access-x94sd\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:38:34.478525 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:34.478240 4421 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c931c147-75a3-4a38-99ff-26d0aa98a38a-istio-podinfo\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:38:34.790039 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:34.790004 4421 generic.go:356] "Generic (PLEG): container finished" podID="c931c147-75a3-4a38-99ff-26d0aa98a38a" containerID="27c7b0cb36fd8ff37135e7f6e5af8b67d75d1f1e3376f811659e6845627e8df6" exitCode=0 Nov 03 01:38:34.790216 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:34.790056 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xmmdd" Nov 03 01:38:34.790216 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:34.790100 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xmmdd" event={"ID":"c931c147-75a3-4a38-99ff-26d0aa98a38a","Type":"ContainerDied","Data":"27c7b0cb36fd8ff37135e7f6e5af8b67d75d1f1e3376f811659e6845627e8df6"} Nov 03 01:38:34.790216 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:34.790187 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xmmdd" event={"ID":"c931c147-75a3-4a38-99ff-26d0aa98a38a","Type":"ContainerDied","Data":"ee3770cfb7ab5a227363b25db165c4692f806108681726830aa7ba790fcd06ce"} Nov 03 01:38:34.790394 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:34.790237 4421 scope.go:122] "RemoveContainer" containerID="27c7b0cb36fd8ff37135e7f6e5af8b67d75d1f1e3376f811659e6845627e8df6" Nov 03 01:38:34.807930 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:34.807907 4421 scope.go:122] "RemoveContainer" containerID="27c7b0cb36fd8ff37135e7f6e5af8b67d75d1f1e3376f811659e6845627e8df6" Nov 03 01:38:34.808337 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:38:34.808313 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27c7b0cb36fd8ff37135e7f6e5af8b67d75d1f1e3376f811659e6845627e8df6\": container with ID starting with 27c7b0cb36fd8ff37135e7f6e5af8b67d75d1f1e3376f811659e6845627e8df6 not found: ID does not exist" containerID="27c7b0cb36fd8ff37135e7f6e5af8b67d75d1f1e3376f811659e6845627e8df6" Nov 03 01:38:34.808462 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:34.808349 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27c7b0cb36fd8ff37135e7f6e5af8b67d75d1f1e3376f811659e6845627e8df6"} err="failed to get container status \"27c7b0cb36fd8ff37135e7f6e5af8b67d75d1f1e3376f811659e6845627e8df6\": rpc error: code = NotFound desc = could not find container \"27c7b0cb36fd8ff37135e7f6e5af8b67d75d1f1e3376f811659e6845627e8df6\": container with ID starting with 27c7b0cb36fd8ff37135e7f6e5af8b67d75d1f1e3376f811659e6845627e8df6 not found: ID does not exist" Nov 03 01:38:34.845820 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:34.845781 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xmmdd"] Nov 03 01:38:34.850201 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:34.850171 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-xmmdd"] Nov 03 01:38:35.193515 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:35.193404 4421 patch_prober.go:29] interesting pod/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-ztm5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.110:15021/healthz/ready\": dial tcp 10.134.1.110:15021: connect: connection refused" start-of-body= Nov 03 01:38:35.193515 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:35.193467 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-ztm5t" podUID="dc7f30da-8b1e-42d7-a41a-2398e66263b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.110:15021/healthz/ready\": dial tcp 10.134.1.110:15021: connect: connection refused" Nov 03 01:38:35.981603 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:35.981567 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c931c147-75a3-4a38-99ff-26d0aa98a38a" path="/var/lib/kubelet/pods/c931c147-75a3-4a38-99ff-26d0aa98a38a/volumes" Nov 03 01:38:36.192755 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:36.192714 4421 patch_prober.go:29] interesting pod/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-ztm5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.110:15021/healthz/ready\": dial tcp 10.134.1.110:15021: connect: connection refused" start-of-body= Nov 03 01:38:36.192964 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:36.192783 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-ztm5t" podUID="dc7f30da-8b1e-42d7-a41a-2398e66263b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.110:15021/healthz/ready\": dial tcp 10.134.1.110:15021: connect: connection refused" Nov 03 01:38:37.193560 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:37.193521 4421 patch_prober.go:29] interesting pod/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-ztm5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.110:15021/healthz/ready\": dial tcp 10.134.1.110:15021: connect: connection refused" start-of-body= Nov 03 01:38:37.194073 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:37.193586 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-ztm5t" podUID="dc7f30da-8b1e-42d7-a41a-2398e66263b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.110:15021/healthz/ready\": dial tcp 10.134.1.110:15021: connect: connection refused" Nov 03 01:38:38.193443 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:38.193408 4421 patch_prober.go:29] interesting pod/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-ztm5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.110:15021/healthz/ready\": dial tcp 10.134.1.110:15021: connect: connection refused" start-of-body= Nov 03 01:38:38.193635 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:38.193466 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-ztm5t" podUID="dc7f30da-8b1e-42d7-a41a-2398e66263b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.110:15021/healthz/ready\": dial tcp 10.134.1.110:15021: connect: connection refused" Nov 03 01:38:39.192801 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:39.192759 4421 patch_prober.go:29] interesting pod/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-ztm5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.110:15021/healthz/ready\": dial tcp 10.134.1.110:15021: connect: connection refused" start-of-body= Nov 03 01:38:39.193037 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:39.192851 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-ztm5t" podUID="dc7f30da-8b1e-42d7-a41a-2398e66263b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.110:15021/healthz/ready\": dial tcp 10.134.1.110:15021: connect: connection refused" Nov 03 01:38:40.192664 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:40.192576 4421 patch_prober.go:29] interesting pod/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-ztm5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.110:15021/healthz/ready\": dial tcp 10.134.1.110:15021: connect: connection refused" start-of-body= Nov 03 01:38:40.192664 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:40.192636 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-ztm5t" podUID="dc7f30da-8b1e-42d7-a41a-2398e66263b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.110:15021/healthz/ready\": dial tcp 10.134.1.110:15021: connect: connection refused" Nov 03 01:38:41.197889 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:41.197851 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-ztm5t" Nov 03 01:38:41.198352 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:41.198332 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-ztm5t" Nov 03 01:38:41.199033 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:41.199016 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-ztm5t" Nov 03 01:38:44.845342 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:38:44.842966 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 01:38:48.549290 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:48.549244 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1844b5d-lscvc"] Nov 03 01:38:48.549980 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:48.549965 4421 cpu_manager.go:414] "containerMap: removing container" podUID="c931c147-75a3-4a38-99ff-26d0aa98a38a" containerName="istio-proxy" Nov 03 01:38:48.550031 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:48.549981 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c931c147-75a3-4a38-99ff-26d0aa98a38a" containerName="istio-proxy" Nov 03 01:38:48.550106 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:48.550097 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="c931c147-75a3-4a38-99ff-26d0aa98a38a" containerName="istio-proxy" Nov 03 01:38:48.555237 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:48.555207 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1844b5d-lscvc" Nov 03 01:38:48.559125 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:48.559071 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:38:48.560450 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:48.560425 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6q4xp\"" Nov 03 01:38:48.569852 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:48.561639 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1844b5d-lscvc"] Nov 03 01:38:48.667938 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:48.667903 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxv9s\" (UniqueName: \"kubernetes.io/projected/065a3186-f482-4aa4-be5d-501fc27a6356-kube-api-access-vxv9s\") pod \"maas-api-revoke-sub-a1844b5d-lscvc\" (UID: \"065a3186-f482-4aa4-be5d-501fc27a6356\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1844b5d-lscvc" Nov 03 01:38:48.668315 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:48.668269 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/065a3186-f482-4aa4-be5d-501fc27a6356-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a1844b5d-lscvc\" (UID: \"065a3186-f482-4aa4-be5d-501fc27a6356\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1844b5d-lscvc" Nov 03 01:38:48.668427 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:48.668363 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/065a3186-f482-4aa4-be5d-501fc27a6356-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a1844b5d-lscvc\" (UID: \"065a3186-f482-4aa4-be5d-501fc27a6356\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1844b5d-lscvc" Nov 03 01:38:48.769703 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:48.769671 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/065a3186-f482-4aa4-be5d-501fc27a6356-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a1844b5d-lscvc\" (UID: \"065a3186-f482-4aa4-be5d-501fc27a6356\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1844b5d-lscvc" Nov 03 01:38:48.769894 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:48.769724 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/065a3186-f482-4aa4-be5d-501fc27a6356-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a1844b5d-lscvc\" (UID: \"065a3186-f482-4aa4-be5d-501fc27a6356\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1844b5d-lscvc" Nov 03 01:38:48.769894 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:48.769771 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxv9s\" (UniqueName: \"kubernetes.io/projected/065a3186-f482-4aa4-be5d-501fc27a6356-kube-api-access-vxv9s\") pod \"maas-api-revoke-sub-a1844b5d-lscvc\" (UID: \"065a3186-f482-4aa4-be5d-501fc27a6356\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1844b5d-lscvc" Nov 03 01:38:48.770481 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:48.770457 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/065a3186-f482-4aa4-be5d-501fc27a6356-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a1844b5d-lscvc\" (UID: \"065a3186-f482-4aa4-be5d-501fc27a6356\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1844b5d-lscvc" Nov 03 01:38:48.772180 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:48.772152 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/065a3186-f482-4aa4-be5d-501fc27a6356-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a1844b5d-lscvc\" (UID: \"065a3186-f482-4aa4-be5d-501fc27a6356\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1844b5d-lscvc" Nov 03 01:38:48.777603 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:48.777581 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxv9s\" (UniqueName: \"kubernetes.io/projected/065a3186-f482-4aa4-be5d-501fc27a6356-kube-api-access-vxv9s\") pod \"maas-api-revoke-sub-a1844b5d-lscvc\" (UID: \"065a3186-f482-4aa4-be5d-501fc27a6356\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1844b5d-lscvc" Nov 03 01:38:48.883227 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:48.883150 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1844b5d-lscvc" Nov 03 01:38:49.170875 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:49.170845 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1844b5d-lscvc"] Nov 03 01:38:49.171862 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:38:49.171836 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod065a3186_f482_4aa4_be5d_501fc27a6356.slice/crio-0c1ac8939d28ab11b3c686cab3e7284c5386f90ddff3b6d313d834e752d38373 WatchSource:0}: Error finding container 0c1ac8939d28ab11b3c686cab3e7284c5386f90ddff3b6d313d834e752d38373: Status 404 returned error can't find the container with id 0c1ac8939d28ab11b3c686cab3e7284c5386f90ddff3b6d313d834e752d38373 Nov 03 01:38:49.950532 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:49.950494 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1844b5d-lscvc" event={"ID":"065a3186-f482-4aa4-be5d-501fc27a6356","Type":"ContainerStarted","Data":"19231f3fa2b704a6ce1be1ca774389dd1fa688f5b32f5525fd6404539b483843"} Nov 03 01:38:49.950965 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:49.950554 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1844b5d-lscvc" event={"ID":"065a3186-f482-4aa4-be5d-501fc27a6356","Type":"ContainerStarted","Data":"0c1ac8939d28ab11b3c686cab3e7284c5386f90ddff3b6d313d834e752d38373"} Nov 03 01:38:49.970136 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:49.970085 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1844b5d-lscvc" podStartSLOduration=1.970071049 podStartE2EDuration="1.970071049s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:38:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:38:49.965591009 +0000 UTC m=+3472.136594107" watchObservedRunningTime="2026-11-03 01:38:49.970071049 +0000 UTC m=+3472.141074144" Nov 03 01:38:51.004541 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:51.004501 4421 generic.go:356] "Generic (PLEG): container finished" podID="065a3186-f482-4aa4-be5d-501fc27a6356" containerID="19231f3fa2b704a6ce1be1ca774389dd1fa688f5b32f5525fd6404539b483843" exitCode=0 Nov 03 01:38:51.004992 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:51.004584 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1844b5d-lscvc" event={"ID":"065a3186-f482-4aa4-be5d-501fc27a6356","Type":"ContainerDied","Data":"19231f3fa2b704a6ce1be1ca774389dd1fa688f5b32f5525fd6404539b483843"} Nov 03 01:38:52.203851 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:52.203815 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1844b5d-lscvc" Nov 03 01:38:52.331352 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:52.327355 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/065a3186-f482-4aa4-be5d-501fc27a6356-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/065a3186-f482-4aa4-be5d-501fc27a6356-maas-api-xks-ca\") pod \"065a3186-f482-4aa4-be5d-501fc27a6356\" (UID: \"065a3186-f482-4aa4-be5d-501fc27a6356\") " Nov 03 01:38:52.337765 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:52.337650 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/065a3186-f482-4aa4-be5d-501fc27a6356-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/065a3186-f482-4aa4-be5d-501fc27a6356-maas-api-openshift-ca\") pod \"065a3186-f482-4aa4-be5d-501fc27a6356\" (UID: \"065a3186-f482-4aa4-be5d-501fc27a6356\") " Nov 03 01:38:52.338071 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:52.337898 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/065a3186-f482-4aa4-be5d-501fc27a6356-kube-api-access-vxv9s\" (UniqueName: \"kubernetes.io/projected/065a3186-f482-4aa4-be5d-501fc27a6356-kube-api-access-vxv9s\") pod \"065a3186-f482-4aa4-be5d-501fc27a6356\" (UID: \"065a3186-f482-4aa4-be5d-501fc27a6356\") " Nov 03 01:38:52.342601 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:52.338650 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/065a3186-f482-4aa4-be5d-501fc27a6356-maas-api-openshift-ca" pod "065a3186-f482-4aa4-be5d-501fc27a6356" (UID: "065a3186-f482-4aa4-be5d-501fc27a6356"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:38:52.342601 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:52.338873 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/065a3186-f482-4aa4-be5d-501fc27a6356-maas-api-xks-ca" pod "065a3186-f482-4aa4-be5d-501fc27a6356" (UID: "065a3186-f482-4aa4-be5d-501fc27a6356"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:38:52.353785 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:52.353745 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/065a3186-f482-4aa4-be5d-501fc27a6356-kube-api-access-vxv9s" pod "065a3186-f482-4aa4-be5d-501fc27a6356" (UID: "065a3186-f482-4aa4-be5d-501fc27a6356"). InnerVolumeSpecName "kube-api-access-vxv9s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:38:52.440809 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:52.440769 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vxv9s\" (UniqueName: \"kubernetes.io/projected/065a3186-f482-4aa4-be5d-501fc27a6356-kube-api-access-vxv9s\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:38:52.440809 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:52.440803 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/065a3186-f482-4aa4-be5d-501fc27a6356-maas-api-xks-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:38:52.440809 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:52.440813 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/065a3186-f482-4aa4-be5d-501fc27a6356-maas-api-openshift-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:38:53.020080 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:53.020048 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1844b5d-lscvc" Nov 03 01:38:53.020261 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:53.020073 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1844b5d-lscvc" event={"ID":"065a3186-f482-4aa4-be5d-501fc27a6356","Type":"ContainerDied","Data":"0c1ac8939d28ab11b3c686cab3e7284c5386f90ddff3b6d313d834e752d38373"} Nov 03 01:38:53.020261 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:53.020127 4421 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0c1ac8939d28ab11b3c686cab3e7284c5386f90ddff3b6d313d834e752d38373" Nov 03 01:38:58.754299 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:58.753492 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6fabaa5-pn92q"] Nov 03 01:38:58.754853 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:58.754536 4421 cpu_manager.go:414] "containerMap: removing container" podUID="065a3186-f482-4aa4-be5d-501fc27a6356" containerName="revoke-keys" Nov 03 01:38:58.754853 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:58.754564 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="065a3186-f482-4aa4-be5d-501fc27a6356" containerName="revoke-keys" Nov 03 01:38:58.754853 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:58.754767 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="065a3186-f482-4aa4-be5d-501fc27a6356" containerName="revoke-keys" Nov 03 01:38:58.762856 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:58.759825 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6fabaa5-pn92q" Nov 03 01:38:58.765125 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:58.765087 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:38:58.766189 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:58.766164 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6fabaa5-pn92q"] Nov 03 01:38:58.766482 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:58.766467 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6q4xp\"" Nov 03 01:38:58.858002 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:58.857963 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce04511a-b37f-4f31-abda-2ff55f0afbd1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f6fabaa5-pn92q\" (UID: \"ce04511a-b37f-4f31-abda-2ff55f0afbd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6fabaa5-pn92q" Nov 03 01:38:58.858192 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:58.858124 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce04511a-b37f-4f31-abda-2ff55f0afbd1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f6fabaa5-pn92q\" (UID: \"ce04511a-b37f-4f31-abda-2ff55f0afbd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6fabaa5-pn92q" Nov 03 01:38:58.858192 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:58.858182 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rftwz\" (UniqueName: \"kubernetes.io/projected/ce04511a-b37f-4f31-abda-2ff55f0afbd1-kube-api-access-rftwz\") pod \"maas-api-revoke-sub-f6fabaa5-pn92q\" (UID: \"ce04511a-b37f-4f31-abda-2ff55f0afbd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6fabaa5-pn92q" Nov 03 01:38:58.959508 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:58.959468 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce04511a-b37f-4f31-abda-2ff55f0afbd1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f6fabaa5-pn92q\" (UID: \"ce04511a-b37f-4f31-abda-2ff55f0afbd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6fabaa5-pn92q" Nov 03 01:38:58.959726 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:58.959592 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce04511a-b37f-4f31-abda-2ff55f0afbd1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f6fabaa5-pn92q\" (UID: \"ce04511a-b37f-4f31-abda-2ff55f0afbd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6fabaa5-pn92q" Nov 03 01:38:58.959772 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:58.959729 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rftwz\" (UniqueName: \"kubernetes.io/projected/ce04511a-b37f-4f31-abda-2ff55f0afbd1-kube-api-access-rftwz\") pod \"maas-api-revoke-sub-f6fabaa5-pn92q\" (UID: \"ce04511a-b37f-4f31-abda-2ff55f0afbd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6fabaa5-pn92q" Nov 03 01:38:58.960248 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:58.960222 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce04511a-b37f-4f31-abda-2ff55f0afbd1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f6fabaa5-pn92q\" (UID: \"ce04511a-b37f-4f31-abda-2ff55f0afbd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6fabaa5-pn92q" Nov 03 01:38:58.961954 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:58.961929 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce04511a-b37f-4f31-abda-2ff55f0afbd1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f6fabaa5-pn92q\" (UID: \"ce04511a-b37f-4f31-abda-2ff55f0afbd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6fabaa5-pn92q" Nov 03 01:38:58.967873 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:58.967847 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rftwz\" (UniqueName: \"kubernetes.io/projected/ce04511a-b37f-4f31-abda-2ff55f0afbd1-kube-api-access-rftwz\") pod \"maas-api-revoke-sub-f6fabaa5-pn92q\" (UID: \"ce04511a-b37f-4f31-abda-2ff55f0afbd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6fabaa5-pn92q" Nov 03 01:38:59.078755 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:59.078661 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6fabaa5-pn92q" Nov 03 01:38:59.369128 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:38:59.369096 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6fabaa5-pn92q"] Nov 03 01:38:59.369813 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:38:59.369788 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce04511a_b37f_4f31_abda_2ff55f0afbd1.slice/crio-01c2ec31824c8a194532fba96280b5d460c2806e5b995ec9aa9cc667ac2df5bb WatchSource:0}: Error finding container 01c2ec31824c8a194532fba96280b5d460c2806e5b995ec9aa9cc667ac2df5bb: Status 404 returned error can't find the container with id 01c2ec31824c8a194532fba96280b5d460c2806e5b995ec9aa9cc667ac2df5bb Nov 03 01:39:00.075337 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:00.075301 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6fabaa5-pn92q" event={"ID":"ce04511a-b37f-4f31-abda-2ff55f0afbd1","Type":"ContainerStarted","Data":"f99556a1cd025c25bda9ea18ae669a59fea5a1d716bb9604fed80ba0ad3402e4"} Nov 03 01:39:00.075759 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:00.075356 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6fabaa5-pn92q" event={"ID":"ce04511a-b37f-4f31-abda-2ff55f0afbd1","Type":"ContainerStarted","Data":"01c2ec31824c8a194532fba96280b5d460c2806e5b995ec9aa9cc667ac2df5bb"} Nov 03 01:39:00.104312 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:00.104227 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6fabaa5-pn92q" podStartSLOduration=2.104195371 podStartE2EDuration="2.104195371s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:38:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:39:00.091557401 +0000 UTC m=+3482.262560495" watchObservedRunningTime="2026-11-03 01:39:00.104195371 +0000 UTC m=+3482.275198467" Nov 03 01:39:01.086071 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:01.086034 4421 generic.go:356] "Generic (PLEG): container finished" podID="ce04511a-b37f-4f31-abda-2ff55f0afbd1" containerID="f99556a1cd025c25bda9ea18ae669a59fea5a1d716bb9604fed80ba0ad3402e4" exitCode=0 Nov 03 01:39:01.086674 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:01.086083 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6fabaa5-pn92q" event={"ID":"ce04511a-b37f-4f31-abda-2ff55f0afbd1","Type":"ContainerDied","Data":"f99556a1cd025c25bda9ea18ae669a59fea5a1d716bb9604fed80ba0ad3402e4"} Nov 03 01:39:02.312183 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:02.312155 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6fabaa5-pn92q" Nov 03 01:39:02.432310 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:02.432177 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce04511a-b37f-4f31-abda-2ff55f0afbd1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce04511a-b37f-4f31-abda-2ff55f0afbd1-maas-api-xks-ca\") pod \"ce04511a-b37f-4f31-abda-2ff55f0afbd1\" (UID: \"ce04511a-b37f-4f31-abda-2ff55f0afbd1\") " Nov 03 01:39:02.432475 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:02.432444 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce04511a-b37f-4f31-abda-2ff55f0afbd1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce04511a-b37f-4f31-abda-2ff55f0afbd1-maas-api-openshift-ca\") pod \"ce04511a-b37f-4f31-abda-2ff55f0afbd1\" (UID: \"ce04511a-b37f-4f31-abda-2ff55f0afbd1\") " Nov 03 01:39:02.432531 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:02.432499 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce04511a-b37f-4f31-abda-2ff55f0afbd1-kube-api-access-rftwz\" (UniqueName: \"kubernetes.io/projected/ce04511a-b37f-4f31-abda-2ff55f0afbd1-kube-api-access-rftwz\") pod \"ce04511a-b37f-4f31-abda-2ff55f0afbd1\" (UID: \"ce04511a-b37f-4f31-abda-2ff55f0afbd1\") " Nov 03 01:39:02.432818 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:02.432787 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce04511a-b37f-4f31-abda-2ff55f0afbd1-maas-api-openshift-ca" pod "ce04511a-b37f-4f31-abda-2ff55f0afbd1" (UID: "ce04511a-b37f-4f31-abda-2ff55f0afbd1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:39:02.432966 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:02.432950 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce04511a-b37f-4f31-abda-2ff55f0afbd1-maas-api-openshift-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:39:02.434539 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:02.434516 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce04511a-b37f-4f31-abda-2ff55f0afbd1-kube-api-access-rftwz" pod "ce04511a-b37f-4f31-abda-2ff55f0afbd1" (UID: "ce04511a-b37f-4f31-abda-2ff55f0afbd1"). InnerVolumeSpecName "kube-api-access-rftwz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:39:02.434622 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:02.434564 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce04511a-b37f-4f31-abda-2ff55f0afbd1-maas-api-xks-ca" pod "ce04511a-b37f-4f31-abda-2ff55f0afbd1" (UID: "ce04511a-b37f-4f31-abda-2ff55f0afbd1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:39:02.534692 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:02.534640 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rftwz\" (UniqueName: \"kubernetes.io/projected/ce04511a-b37f-4f31-abda-2ff55f0afbd1-kube-api-access-rftwz\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:39:02.534692 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:02.534686 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce04511a-b37f-4f31-abda-2ff55f0afbd1-maas-api-xks-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:39:03.148192 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:03.148160 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6fabaa5-pn92q" Nov 03 01:39:03.148401 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:03.148193 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6fabaa5-pn92q" event={"ID":"ce04511a-b37f-4f31-abda-2ff55f0afbd1","Type":"ContainerDied","Data":"01c2ec31824c8a194532fba96280b5d460c2806e5b995ec9aa9cc667ac2df5bb"} Nov 03 01:39:03.148401 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:03.148242 4421 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="01c2ec31824c8a194532fba96280b5d460c2806e5b995ec9aa9cc667ac2df5bb" Nov 03 01:39:06.024014 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:06.023976 4421 scope.go:122] "RemoveContainer" containerID="8c6f903b6fcefb1ad9f4b4df0760344808c50204fb9ec5f1af0c28e28fc483e5" Nov 03 01:39:09.413518 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:09.413478 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e18d8a30-h8njr"] Nov 03 01:39:09.414416 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:09.414393 4421 cpu_manager.go:414] "containerMap: removing container" podUID="ce04511a-b37f-4f31-abda-2ff55f0afbd1" containerName="revoke-keys" Nov 03 01:39:09.414416 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:09.414415 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce04511a-b37f-4f31-abda-2ff55f0afbd1" containerName="revoke-keys" Nov 03 01:39:09.414652 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:09.414636 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce04511a-b37f-4f31-abda-2ff55f0afbd1" containerName="revoke-keys" Nov 03 01:39:09.419733 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:09.419707 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e18d8a30-h8njr" Nov 03 01:39:09.424427 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:09.424394 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:39:09.425331 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:09.425299 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e18d8a30-h8njr"] Nov 03 01:39:09.425582 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:09.425564 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6q4xp\"" Nov 03 01:39:09.457516 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:09.457477 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff0057ad-0f4f-44df-9979-a5d97343c69a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e18d8a30-h8njr\" (UID: \"ff0057ad-0f4f-44df-9979-a5d97343c69a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e18d8a30-h8njr" Nov 03 01:39:09.457723 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:09.457697 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtzw6\" (UniqueName: \"kubernetes.io/projected/ff0057ad-0f4f-44df-9979-a5d97343c69a-kube-api-access-gtzw6\") pod \"maas-api-revoke-sub-e18d8a30-h8njr\" (UID: \"ff0057ad-0f4f-44df-9979-a5d97343c69a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e18d8a30-h8njr" Nov 03 01:39:09.457993 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:09.457972 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff0057ad-0f4f-44df-9979-a5d97343c69a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e18d8a30-h8njr\" (UID: \"ff0057ad-0f4f-44df-9979-a5d97343c69a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e18d8a30-h8njr" Nov 03 01:39:09.558911 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:09.558877 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff0057ad-0f4f-44df-9979-a5d97343c69a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e18d8a30-h8njr\" (UID: \"ff0057ad-0f4f-44df-9979-a5d97343c69a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e18d8a30-h8njr" Nov 03 01:39:09.559105 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:09.558999 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtzw6\" (UniqueName: \"kubernetes.io/projected/ff0057ad-0f4f-44df-9979-a5d97343c69a-kube-api-access-gtzw6\") pod \"maas-api-revoke-sub-e18d8a30-h8njr\" (UID: \"ff0057ad-0f4f-44df-9979-a5d97343c69a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e18d8a30-h8njr" Nov 03 01:39:09.559187 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:09.559168 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff0057ad-0f4f-44df-9979-a5d97343c69a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e18d8a30-h8njr\" (UID: \"ff0057ad-0f4f-44df-9979-a5d97343c69a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e18d8a30-h8njr" Nov 03 01:39:09.559726 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:09.559700 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff0057ad-0f4f-44df-9979-a5d97343c69a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e18d8a30-h8njr\" (UID: \"ff0057ad-0f4f-44df-9979-a5d97343c69a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e18d8a30-h8njr" Nov 03 01:39:09.561699 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:09.561679 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff0057ad-0f4f-44df-9979-a5d97343c69a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e18d8a30-h8njr\" (UID: \"ff0057ad-0f4f-44df-9979-a5d97343c69a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e18d8a30-h8njr" Nov 03 01:39:09.567121 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:09.567097 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtzw6\" (UniqueName: \"kubernetes.io/projected/ff0057ad-0f4f-44df-9979-a5d97343c69a-kube-api-access-gtzw6\") pod \"maas-api-revoke-sub-e18d8a30-h8njr\" (UID: \"ff0057ad-0f4f-44df-9979-a5d97343c69a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e18d8a30-h8njr" Nov 03 01:39:09.737025 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:09.736946 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e18d8a30-h8njr" Nov 03 01:39:10.020171 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:10.020138 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e18d8a30-h8njr"] Nov 03 01:39:10.021510 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:39:10.021485 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff0057ad_0f4f_44df_9979_a5d97343c69a.slice/crio-590379288f55fe06b716a85989c134ef0890c6b47488b904b76edab02199f42b WatchSource:0}: Error finding container 590379288f55fe06b716a85989c134ef0890c6b47488b904b76edab02199f42b: Status 404 returned error can't find the container with id 590379288f55fe06b716a85989c134ef0890c6b47488b904b76edab02199f42b Nov 03 01:39:10.209508 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:10.209473 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e18d8a30-h8njr" event={"ID":"ff0057ad-0f4f-44df-9979-a5d97343c69a","Type":"ContainerStarted","Data":"590379288f55fe06b716a85989c134ef0890c6b47488b904b76edab02199f42b"} Nov 03 01:39:11.218762 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:11.218681 4421 generic.go:356] "Generic (PLEG): container finished" podID="ff0057ad-0f4f-44df-9979-a5d97343c69a" containerID="123178896f2b5127cb3e40cab0a939ee81c5f48b0f4d28240f2b8f9064043f29" exitCode=0 Nov 03 01:39:11.219141 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:11.218763 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e18d8a30-h8njr" event={"ID":"ff0057ad-0f4f-44df-9979-a5d97343c69a","Type":"ContainerDied","Data":"123178896f2b5127cb3e40cab0a939ee81c5f48b0f4d28240f2b8f9064043f29"} Nov 03 01:39:12.412759 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:12.412720 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e18d8a30-h8njr" Nov 03 01:39:12.523880 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:12.523848 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff0057ad-0f4f-44df-9979-a5d97343c69a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff0057ad-0f4f-44df-9979-a5d97343c69a-maas-api-xks-ca\") pod \"ff0057ad-0f4f-44df-9979-a5d97343c69a\" (UID: \"ff0057ad-0f4f-44df-9979-a5d97343c69a\") " Nov 03 01:39:12.524099 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:12.523912 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff0057ad-0f4f-44df-9979-a5d97343c69a-kube-api-access-gtzw6\" (UniqueName: \"kubernetes.io/projected/ff0057ad-0f4f-44df-9979-a5d97343c69a-kube-api-access-gtzw6\") pod \"ff0057ad-0f4f-44df-9979-a5d97343c69a\" (UID: \"ff0057ad-0f4f-44df-9979-a5d97343c69a\") " Nov 03 01:39:12.524099 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:12.523964 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff0057ad-0f4f-44df-9979-a5d97343c69a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff0057ad-0f4f-44df-9979-a5d97343c69a-maas-api-openshift-ca\") pod \"ff0057ad-0f4f-44df-9979-a5d97343c69a\" (UID: \"ff0057ad-0f4f-44df-9979-a5d97343c69a\") " Nov 03 01:39:12.524526 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:12.524496 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff0057ad-0f4f-44df-9979-a5d97343c69a-maas-api-openshift-ca" pod "ff0057ad-0f4f-44df-9979-a5d97343c69a" (UID: "ff0057ad-0f4f-44df-9979-a5d97343c69a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:39:12.526220 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:12.526191 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff0057ad-0f4f-44df-9979-a5d97343c69a-maas-api-xks-ca" pod "ff0057ad-0f4f-44df-9979-a5d97343c69a" (UID: "ff0057ad-0f4f-44df-9979-a5d97343c69a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:39:12.526479 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:12.526444 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff0057ad-0f4f-44df-9979-a5d97343c69a-kube-api-access-gtzw6" pod "ff0057ad-0f4f-44df-9979-a5d97343c69a" (UID: "ff0057ad-0f4f-44df-9979-a5d97343c69a"). InnerVolumeSpecName "kube-api-access-gtzw6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:39:12.626750 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:12.626698 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff0057ad-0f4f-44df-9979-a5d97343c69a-maas-api-openshift-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:39:12.626750 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:12.626748 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff0057ad-0f4f-44df-9979-a5d97343c69a-maas-api-xks-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:39:12.627001 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:12.626766 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gtzw6\" (UniqueName: \"kubernetes.io/projected/ff0057ad-0f4f-44df-9979-a5d97343c69a-kube-api-access-gtzw6\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:39:13.266295 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:13.265879 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e18d8a30-h8njr" event={"ID":"ff0057ad-0f4f-44df-9979-a5d97343c69a","Type":"ContainerDied","Data":"590379288f55fe06b716a85989c134ef0890c6b47488b904b76edab02199f42b"} Nov 03 01:39:13.266295 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:13.265969 4421 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="590379288f55fe06b716a85989c134ef0890c6b47488b904b76edab02199f42b" Nov 03 01:39:13.266295 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:13.266194 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e18d8a30-h8njr" Nov 03 01:39:14.670657 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:39:14.670614 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 01:39:17.029003 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:17.028970 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed3dfac7-qrblz"] Nov 03 01:39:17.032866 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:17.032840 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed3dfac7-qrblz"] Nov 03 01:39:17.193546 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:17.193507 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-d2znx"] Nov 03 01:39:17.194471 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:17.194445 4421 cpu_manager.go:414] "containerMap: removing container" podUID="ff0057ad-0f4f-44df-9979-a5d97343c69a" containerName="revoke-keys" Nov 03 01:39:17.194471 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:17.194467 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff0057ad-0f4f-44df-9979-a5d97343c69a" containerName="revoke-keys" Nov 03 01:39:17.194677 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:17.194639 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff0057ad-0f4f-44df-9979-a5d97343c69a" containerName="revoke-keys" Nov 03 01:39:17.198721 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:17.198700 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-d2znx" Nov 03 01:39:17.206039 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:17.206017 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-ef434a-openshift-default-dockercfg-sc8xt\"" Nov 03 01:39:17.209450 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:17.209421 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-d2znx"] Nov 03 01:39:17.310235 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:17.310144 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f9adfd4-8b5b-4a37-b625-9a8727482b7c-workload-socket\") pod \"e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-d2znx\" (UID: \"3f9adfd4-8b5b-4a37-b625-9a8727482b7c\") " pod="openshift-ingress/e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-d2znx" Nov 03 01:39:17.310422 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:17.310344 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f9adfd4-8b5b-4a37-b625-9a8727482b7c-istio-envoy\") pod \"e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-d2znx\" (UID: \"3f9adfd4-8b5b-4a37-b625-9a8727482b7c\") " pod="openshift-ingress/e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-d2znx" Nov 03 01:39:17.310422 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:17.310371 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f9adfd4-8b5b-4a37-b625-9a8727482b7c-istio-data\") pod \"e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-d2znx\" (UID: \"3f9adfd4-8b5b-4a37-b625-9a8727482b7c\") " pod="openshift-ingress/e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-d2znx" Nov 03 01:39:17.310555 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:17.310527 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f9adfd4-8b5b-4a37-b625-9a8727482b7c-workload-certs\") pod \"e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-d2znx\" (UID: \"3f9adfd4-8b5b-4a37-b625-9a8727482b7c\") " pod="openshift-ingress/e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-d2znx" Nov 03 01:39:17.310606 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:17.310561 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5q5cx\" (UniqueName: \"kubernetes.io/projected/3f9adfd4-8b5b-4a37-b625-9a8727482b7c-kube-api-access-5q5cx\") pod \"e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-d2znx\" (UID: \"3f9adfd4-8b5b-4a37-b625-9a8727482b7c\") " pod="openshift-ingress/e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-d2znx" Nov 03 01:39:17.310685 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:17.310655 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f9adfd4-8b5b-4a37-b625-9a8727482b7c-istio-token\") pod \"e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-d2znx\" (UID: \"3f9adfd4-8b5b-4a37-b625-9a8727482b7c\") " pod="openshift-ingress/e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-d2znx" Nov 03 01:39:17.310842 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:17.310695 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f9adfd4-8b5b-4a37-b625-9a8727482b7c-istio-podinfo\") pod \"e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-d2znx\" (UID: \"3f9adfd4-8b5b-4a37-b625-9a8727482b7c\") " pod="openshift-ingress/e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-d2znx" Nov 03 01:39:17.310842 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:17.310798 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f9adfd4-8b5b-4a37-b625-9a8727482b7c-credential-socket\") pod \"e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-d2znx\" (UID: \"3f9adfd4-8b5b-4a37-b625-9a8727482b7c\") " pod="openshift-ingress/e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-d2znx" Nov 03 01:39:17.310959 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:17.310854 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f9adfd4-8b5b-4a37-b625-9a8727482b7c-istiod-ca-cert\") pod \"e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-d2znx\" (UID: \"3f9adfd4-8b5b-4a37-b625-9a8727482b7c\") " pod="openshift-ingress/e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-d2znx" Nov 03 01:39:17.414440 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:17.414394 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f9adfd4-8b5b-4a37-b625-9a8727482b7c-workload-certs\") pod \"e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-d2znx\" (UID: \"3f9adfd4-8b5b-4a37-b625-9a8727482b7c\") " pod="openshift-ingress/e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-d2znx" Nov 03 01:39:17.415363 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:17.415332 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5q5cx\" (UniqueName: \"kubernetes.io/projected/3f9adfd4-8b5b-4a37-b625-9a8727482b7c-kube-api-access-5q5cx\") pod \"e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-d2znx\" (UID: \"3f9adfd4-8b5b-4a37-b625-9a8727482b7c\") " pod="openshift-ingress/e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-d2znx" Nov 03 01:39:17.415681 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:17.415628 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f9adfd4-8b5b-4a37-b625-9a8727482b7c-istio-token\") pod \"e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-d2znx\" (UID: \"3f9adfd4-8b5b-4a37-b625-9a8727482b7c\") " pod="openshift-ingress/e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-d2znx" Nov 03 01:39:17.415846 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:17.415822 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f9adfd4-8b5b-4a37-b625-9a8727482b7c-istio-podinfo\") pod \"e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-d2znx\" (UID: \"3f9adfd4-8b5b-4a37-b625-9a8727482b7c\") " pod="openshift-ingress/e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-d2znx" Nov 03 01:39:17.417868 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:17.417842 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f9adfd4-8b5b-4a37-b625-9a8727482b7c-credential-socket\") pod \"e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-d2znx\" (UID: \"3f9adfd4-8b5b-4a37-b625-9a8727482b7c\") " pod="openshift-ingress/e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-d2znx" Nov 03 01:39:17.418080 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:17.418062 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f9adfd4-8b5b-4a37-b625-9a8727482b7c-istiod-ca-cert\") pod \"e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-d2znx\" (UID: \"3f9adfd4-8b5b-4a37-b625-9a8727482b7c\") " pod="openshift-ingress/e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-d2znx" Nov 03 01:39:17.418225 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:17.418205 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f9adfd4-8b5b-4a37-b625-9a8727482b7c-workload-socket\") pod \"e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-d2znx\" (UID: \"3f9adfd4-8b5b-4a37-b625-9a8727482b7c\") " pod="openshift-ingress/e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-d2znx" Nov 03 01:39:17.418551 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:17.418527 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f9adfd4-8b5b-4a37-b625-9a8727482b7c-istio-envoy\") pod \"e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-d2znx\" (UID: \"3f9adfd4-8b5b-4a37-b625-9a8727482b7c\") " pod="openshift-ingress/e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-d2znx" Nov 03 01:39:17.418677 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:17.418661 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f9adfd4-8b5b-4a37-b625-9a8727482b7c-istio-data\") pod \"e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-d2znx\" (UID: \"3f9adfd4-8b5b-4a37-b625-9a8727482b7c\") " pod="openshift-ingress/e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-d2znx" Nov 03 01:39:17.418985 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:17.415255 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f9adfd4-8b5b-4a37-b625-9a8727482b7c-workload-certs\") pod \"e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-d2znx\" (UID: \"3f9adfd4-8b5b-4a37-b625-9a8727482b7c\") " pod="openshift-ingress/e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-d2znx" Nov 03 01:39:17.419258 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:17.419223 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f9adfd4-8b5b-4a37-b625-9a8727482b7c-istiod-ca-cert\") pod \"e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-d2znx\" (UID: \"3f9adfd4-8b5b-4a37-b625-9a8727482b7c\") " pod="openshift-ingress/e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-d2znx" Nov 03 01:39:17.419390 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:17.418618 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f9adfd4-8b5b-4a37-b625-9a8727482b7c-credential-socket\") pod \"e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-d2znx\" (UID: \"3f9adfd4-8b5b-4a37-b625-9a8727482b7c\") " pod="openshift-ingress/e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-d2znx" Nov 03 01:39:17.420662 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:17.419886 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f9adfd4-8b5b-4a37-b625-9a8727482b7c-istio-data\") pod \"e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-d2znx\" (UID: \"3f9adfd4-8b5b-4a37-b625-9a8727482b7c\") " pod="openshift-ingress/e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-d2znx" Nov 03 01:39:17.421050 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:17.421025 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f9adfd4-8b5b-4a37-b625-9a8727482b7c-workload-socket\") pod \"e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-d2znx\" (UID: \"3f9adfd4-8b5b-4a37-b625-9a8727482b7c\") " pod="openshift-ingress/e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-d2znx" Nov 03 01:39:17.424264 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:17.424241 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f9adfd4-8b5b-4a37-b625-9a8727482b7c-istio-envoy\") pod \"e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-d2znx\" (UID: \"3f9adfd4-8b5b-4a37-b625-9a8727482b7c\") " pod="openshift-ingress/e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-d2znx" Nov 03 01:39:17.425098 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:17.425069 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f9adfd4-8b5b-4a37-b625-9a8727482b7c-istio-podinfo\") pod \"e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-d2znx\" (UID: \"3f9adfd4-8b5b-4a37-b625-9a8727482b7c\") " pod="openshift-ingress/e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-d2znx" Nov 03 01:39:17.425230 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:17.425210 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5q5cx\" (UniqueName: \"kubernetes.io/projected/3f9adfd4-8b5b-4a37-b625-9a8727482b7c-kube-api-access-5q5cx\") pod \"e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-d2znx\" (UID: \"3f9adfd4-8b5b-4a37-b625-9a8727482b7c\") " pod="openshift-ingress/e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-d2znx" Nov 03 01:39:17.426314 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:17.426294 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f9adfd4-8b5b-4a37-b625-9a8727482b7c-istio-token\") pod \"e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-d2znx\" (UID: \"3f9adfd4-8b5b-4a37-b625-9a8727482b7c\") " pod="openshift-ingress/e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-d2znx" Nov 03 01:39:17.515790 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:17.515748 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-d2znx" Nov 03 01:39:17.804466 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:17.804433 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-d2znx"] Nov 03 01:39:17.804626 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:39:17.804587 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f9adfd4_8b5b_4a37_b625_9a8727482b7c.slice/crio-bc404db1142800a81551d059fc6b1c676395f308de4a12e024b007e6521f082f WatchSource:0}: Error finding container bc404db1142800a81551d059fc6b1c676395f308de4a12e024b007e6521f082f: Status 404 returned error can't find the container with id bc404db1142800a81551d059fc6b1c676395f308de4a12e024b007e6521f082f Nov 03 01:39:17.807535 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:17.807500 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:39:17.807645 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:17.807579 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:39:17.807645 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:17.807616 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:39:17.985658 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:17.985621 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fec6be89-9f7a-4db5-aa30-e862be8c53d4" path="/var/lib/kubelet/pods/fec6be89-9f7a-4db5-aa30-e862be8c53d4/volumes" Nov 03 01:39:18.308178 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:18.308145 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-d2znx" event={"ID":"3f9adfd4-8b5b-4a37-b625-9a8727482b7c","Type":"ContainerStarted","Data":"fec0554a23ecf631530529b990ceb185a106d120263b4d3a6253e441fd2359db"} Nov 03 01:39:18.308731 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:18.308210 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-d2znx" event={"ID":"3f9adfd4-8b5b-4a37-b625-9a8727482b7c","Type":"ContainerStarted","Data":"bc404db1142800a81551d059fc6b1c676395f308de4a12e024b007e6521f082f"} Nov 03 01:39:18.330960 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:18.330907 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-d2znx" podStartSLOduration=1.330892819 podStartE2EDuration="1.330892819s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:39:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:39:18.326681277 +0000 UTC m=+3500.497684376" watchObservedRunningTime="2026-11-03 01:39:18.330892819 +0000 UTC m=+3500.501895970" Nov 03 01:39:18.525299 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:18.522619 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-d2znx" Nov 03 01:39:18.525299 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:18.522762 4421 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-d2znx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.114:15021/healthz/ready\": dial tcp 10.134.1.114:15021: connect: connection refused" start-of-body= Nov 03 01:39:18.525299 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:18.522817 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-d2znx" podUID="3f9adfd4-8b5b-4a37-b625-9a8727482b7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.114:15021/healthz/ready\": dial tcp 10.134.1.114:15021: connect: connection refused" Nov 03 01:39:19.530736 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:19.530692 4421 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-d2znx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.114:15021/healthz/ready\": dial tcp 10.134.1.114:15021: connect: connection refused" start-of-body= Nov 03 01:39:19.531326 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:19.530759 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-d2znx" podUID="3f9adfd4-8b5b-4a37-b625-9a8727482b7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.114:15021/healthz/ready\": dial tcp 10.134.1.114:15021: connect: connection refused" Nov 03 01:39:19.874484 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:19.874386 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d8d22e3-th89b"] Nov 03 01:39:19.879532 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:19.879503 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d8d22e3-th89b" Nov 03 01:39:19.883872 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:19.883830 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:39:19.884060 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:19.883941 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6q4xp\"" Nov 03 01:39:19.886062 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:19.886031 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d8d22e3-th89b"] Nov 03 01:39:19.975990 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:19.975742 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2g2sz\" (UniqueName: \"kubernetes.io/projected/6e612246-ef87-4ccb-b32e-ef7dd768afd2-kube-api-access-2g2sz\") pod \"maas-api-revoke-sub-9d8d22e3-th89b\" (UID: \"6e612246-ef87-4ccb-b32e-ef7dd768afd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d8d22e3-th89b" Nov 03 01:39:19.978515 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:19.978484 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e612246-ef87-4ccb-b32e-ef7dd768afd2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9d8d22e3-th89b\" (UID: \"6e612246-ef87-4ccb-b32e-ef7dd768afd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d8d22e3-th89b" Nov 03 01:39:19.978820 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:19.978801 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e612246-ef87-4ccb-b32e-ef7dd768afd2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9d8d22e3-th89b\" (UID: \"6e612246-ef87-4ccb-b32e-ef7dd768afd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d8d22e3-th89b" Nov 03 01:39:20.080427 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:20.080393 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e612246-ef87-4ccb-b32e-ef7dd768afd2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9d8d22e3-th89b\" (UID: \"6e612246-ef87-4ccb-b32e-ef7dd768afd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d8d22e3-th89b" Nov 03 01:39:20.080647 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:20.080479 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e612246-ef87-4ccb-b32e-ef7dd768afd2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9d8d22e3-th89b\" (UID: \"6e612246-ef87-4ccb-b32e-ef7dd768afd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d8d22e3-th89b" Nov 03 01:39:20.080647 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:20.080562 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2g2sz\" (UniqueName: \"kubernetes.io/projected/6e612246-ef87-4ccb-b32e-ef7dd768afd2-kube-api-access-2g2sz\") pod \"maas-api-revoke-sub-9d8d22e3-th89b\" (UID: \"6e612246-ef87-4ccb-b32e-ef7dd768afd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d8d22e3-th89b" Nov 03 01:39:20.081152 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:20.081124 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e612246-ef87-4ccb-b32e-ef7dd768afd2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9d8d22e3-th89b\" (UID: \"6e612246-ef87-4ccb-b32e-ef7dd768afd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d8d22e3-th89b" Nov 03 01:39:20.083016 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:20.082990 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e612246-ef87-4ccb-b32e-ef7dd768afd2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9d8d22e3-th89b\" (UID: \"6e612246-ef87-4ccb-b32e-ef7dd768afd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d8d22e3-th89b" Nov 03 01:39:20.088545 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:20.088523 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2g2sz\" (UniqueName: \"kubernetes.io/projected/6e612246-ef87-4ccb-b32e-ef7dd768afd2-kube-api-access-2g2sz\") pod \"maas-api-revoke-sub-9d8d22e3-th89b\" (UID: \"6e612246-ef87-4ccb-b32e-ef7dd768afd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d8d22e3-th89b" Nov 03 01:39:20.198696 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:20.198608 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d8d22e3-th89b" Nov 03 01:39:20.530303 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:20.529557 4421 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-d2znx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.114:15021/healthz/ready\": dial tcp 10.134.1.114:15021: connect: connection refused" start-of-body= Nov 03 01:39:20.530303 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:20.529628 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-d2znx" podUID="3f9adfd4-8b5b-4a37-b625-9a8727482b7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.114:15021/healthz/ready\": dial tcp 10.134.1.114:15021: connect: connection refused" Nov 03 01:39:20.536037 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:39:20.534524 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e612246_ef87_4ccb_b32e_ef7dd768afd2.slice/crio-f6d12bfe7ca44cd7543044780b22e9d78511e4ef6bc69e35573db18c45b98644 WatchSource:0}: Error finding container f6d12bfe7ca44cd7543044780b22e9d78511e4ef6bc69e35573db18c45b98644: Status 404 returned error can't find the container with id f6d12bfe7ca44cd7543044780b22e9d78511e4ef6bc69e35573db18c45b98644 Nov 03 01:39:20.536037 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:20.534996 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d8d22e3-th89b"] Nov 03 01:39:21.351065 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:21.351025 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d8d22e3-th89b" event={"ID":"6e612246-ef87-4ccb-b32e-ef7dd768afd2","Type":"ContainerStarted","Data":"8f3e7040a2ffc7943e3e31b383e038780a2295232a8eaee7ab3c2abfbfe81b60"} Nov 03 01:39:21.351263 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:21.351084 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d8d22e3-th89b" event={"ID":"6e612246-ef87-4ccb-b32e-ef7dd768afd2","Type":"ContainerStarted","Data":"f6d12bfe7ca44cd7543044780b22e9d78511e4ef6bc69e35573db18c45b98644"} Nov 03 01:39:21.368123 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:21.368062 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d8d22e3-th89b" podStartSLOduration=2.368044992 podStartE2EDuration="2.368044992s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:39:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:39:21.367788692 +0000 UTC m=+3503.538791788" watchObservedRunningTime="2026-11-03 01:39:21.368044992 +0000 UTC m=+3503.539048089" Nov 03 01:39:21.524845 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:21.524805 4421 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-d2znx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.114:15021/healthz/ready\": dial tcp 10.134.1.114:15021: connect: connection refused" start-of-body= Nov 03 01:39:21.525324 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:21.525266 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-d2znx" podUID="3f9adfd4-8b5b-4a37-b625-9a8727482b7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.114:15021/healthz/ready\": dial tcp 10.134.1.114:15021: connect: connection refused" Nov 03 01:39:22.360369 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:22.360333 4421 generic.go:356] "Generic (PLEG): container finished" podID="6e612246-ef87-4ccb-b32e-ef7dd768afd2" containerID="8f3e7040a2ffc7943e3e31b383e038780a2295232a8eaee7ab3c2abfbfe81b60" exitCode=0 Nov 03 01:39:22.360878 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:22.360377 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d8d22e3-th89b" event={"ID":"6e612246-ef87-4ccb-b32e-ef7dd768afd2","Type":"ContainerDied","Data":"8f3e7040a2ffc7943e3e31b383e038780a2295232a8eaee7ab3c2abfbfe81b60"} Nov 03 01:39:22.530304 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:22.523767 4421 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-d2znx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.114:15021/healthz/ready\": dial tcp 10.134.1.114:15021: connect: connection refused" start-of-body= Nov 03 01:39:22.530304 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:22.523843 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-d2znx" podUID="3f9adfd4-8b5b-4a37-b625-9a8727482b7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.114:15021/healthz/ready\": dial tcp 10.134.1.114:15021: connect: connection refused" Nov 03 01:39:23.521394 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:23.516341 4421 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-d2znx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.114:15021/healthz/ready\": dial tcp 10.134.1.114:15021: connect: connection refused" start-of-body= Nov 03 01:39:23.521394 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:23.516408 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-d2znx" podUID="3f9adfd4-8b5b-4a37-b625-9a8727482b7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.114:15021/healthz/ready\": dial tcp 10.134.1.114:15021: connect: connection refused" Nov 03 01:39:23.645694 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:23.645624 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d8d22e3-th89b" Nov 03 01:39:23.684372 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:23.684339 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e612246-ef87-4ccb-b32e-ef7dd768afd2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e612246-ef87-4ccb-b32e-ef7dd768afd2-maas-api-openshift-ca\") pod \"6e612246-ef87-4ccb-b32e-ef7dd768afd2\" (UID: \"6e612246-ef87-4ccb-b32e-ef7dd768afd2\") " Nov 03 01:39:23.684579 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:23.684498 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e612246-ef87-4ccb-b32e-ef7dd768afd2-kube-api-access-2g2sz\" (UniqueName: \"kubernetes.io/projected/6e612246-ef87-4ccb-b32e-ef7dd768afd2-kube-api-access-2g2sz\") pod \"6e612246-ef87-4ccb-b32e-ef7dd768afd2\" (UID: \"6e612246-ef87-4ccb-b32e-ef7dd768afd2\") " Nov 03 01:39:23.684579 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:23.684540 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6e612246-ef87-4ccb-b32e-ef7dd768afd2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e612246-ef87-4ccb-b32e-ef7dd768afd2-maas-api-xks-ca\") pod \"6e612246-ef87-4ccb-b32e-ef7dd768afd2\" (UID: \"6e612246-ef87-4ccb-b32e-ef7dd768afd2\") " Nov 03 01:39:23.684788 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:23.684755 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e612246-ef87-4ccb-b32e-ef7dd768afd2-maas-api-openshift-ca" pod "6e612246-ef87-4ccb-b32e-ef7dd768afd2" (UID: "6e612246-ef87-4ccb-b32e-ef7dd768afd2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:39:23.685396 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:23.685364 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e612246-ef87-4ccb-b32e-ef7dd768afd2-maas-api-openshift-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:39:23.687268 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:23.687244 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e612246-ef87-4ccb-b32e-ef7dd768afd2-maas-api-xks-ca" pod "6e612246-ef87-4ccb-b32e-ef7dd768afd2" (UID: "6e612246-ef87-4ccb-b32e-ef7dd768afd2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:39:23.687403 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:23.687300 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e612246-ef87-4ccb-b32e-ef7dd768afd2-kube-api-access-2g2sz" pod "6e612246-ef87-4ccb-b32e-ef7dd768afd2" (UID: "6e612246-ef87-4ccb-b32e-ef7dd768afd2"). InnerVolumeSpecName "kube-api-access-2g2sz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:39:23.786983 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:23.786947 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2g2sz\" (UniqueName: \"kubernetes.io/projected/6e612246-ef87-4ccb-b32e-ef7dd768afd2-kube-api-access-2g2sz\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:39:23.786983 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:23.786980 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e612246-ef87-4ccb-b32e-ef7dd768afd2-maas-api-xks-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:39:24.379152 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:24.379117 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d8d22e3-th89b" Nov 03 01:39:24.379152 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:24.379134 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d8d22e3-th89b" event={"ID":"6e612246-ef87-4ccb-b32e-ef7dd768afd2","Type":"ContainerDied","Data":"f6d12bfe7ca44cd7543044780b22e9d78511e4ef6bc69e35573db18c45b98644"} Nov 03 01:39:24.379426 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:24.379194 4421 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f6d12bfe7ca44cd7543044780b22e9d78511e4ef6bc69e35573db18c45b98644" Nov 03 01:39:24.549869 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:24.549832 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-d2znx" Nov 03 01:39:24.551670 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:24.551638 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-d2znx" Nov 03 01:39:24.552722 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:24.552701 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-d2znx" Nov 03 01:39:41.031217 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:41.031176 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ad7b8cb-xd298"] Nov 03 01:39:41.035170 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:41.035146 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ad7b8cb-xd298"] Nov 03 01:39:41.981244 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:41.981203 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d2f4a61-1127-4b56-b639-13742aa394b0" path="/var/lib/kubelet/pods/9d2f4a61-1127-4b56-b639-13742aa394b0/volumes" Nov 03 01:39:44.708531 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:39:44.708491 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 01:39:52.022564 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:52.022504 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c05bb0ed-69cvl"] Nov 03 01:39:52.029580 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:52.029547 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c05bb0ed-69cvl"] Nov 03 01:39:53.981866 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:53.981825 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="434cc10e-873d-4d79-a3e2-93f475b37ccf" path="/var/lib/kubelet/pods/434cc10e-873d-4d79-a3e2-93f475b37ccf/volumes" Nov 03 01:39:54.154958 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:54.154923 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c49b28f-6t7dj"] Nov 03 01:39:54.155668 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:54.155652 4421 cpu_manager.go:414] "containerMap: removing container" podUID="6e612246-ef87-4ccb-b32e-ef7dd768afd2" containerName="revoke-keys" Nov 03 01:39:54.155718 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:54.155670 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e612246-ef87-4ccb-b32e-ef7dd768afd2" containerName="revoke-keys" Nov 03 01:39:54.155793 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:54.155783 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e612246-ef87-4ccb-b32e-ef7dd768afd2" containerName="revoke-keys" Nov 03 01:39:54.161101 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:54.161074 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c49b28f-6t7dj" Nov 03 01:39:54.165767 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:54.165737 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c49b28f-6t7dj"] Nov 03 01:39:54.165927 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:54.165870 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:39:54.166098 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:54.166073 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6q4xp\"" Nov 03 01:39:54.258420 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:54.258389 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ced0135-f0a1-4a1a-ae6a-2d5442bdf66a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5c49b28f-6t7dj\" (UID: \"7ced0135-f0a1-4a1a-ae6a-2d5442bdf66a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c49b28f-6t7dj" Nov 03 01:39:54.258723 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:54.258698 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ced0135-f0a1-4a1a-ae6a-2d5442bdf66a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5c49b28f-6t7dj\" (UID: \"7ced0135-f0a1-4a1a-ae6a-2d5442bdf66a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c49b28f-6t7dj" Nov 03 01:39:54.258860 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:54.258841 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l84mr\" (UniqueName: \"kubernetes.io/projected/7ced0135-f0a1-4a1a-ae6a-2d5442bdf66a-kube-api-access-l84mr\") pod \"maas-api-revoke-sub-5c49b28f-6t7dj\" (UID: \"7ced0135-f0a1-4a1a-ae6a-2d5442bdf66a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c49b28f-6t7dj" Nov 03 01:39:54.359874 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:54.359837 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l84mr\" (UniqueName: \"kubernetes.io/projected/7ced0135-f0a1-4a1a-ae6a-2d5442bdf66a-kube-api-access-l84mr\") pod \"maas-api-revoke-sub-5c49b28f-6t7dj\" (UID: \"7ced0135-f0a1-4a1a-ae6a-2d5442bdf66a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c49b28f-6t7dj" Nov 03 01:39:54.360072 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:54.359899 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ced0135-f0a1-4a1a-ae6a-2d5442bdf66a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5c49b28f-6t7dj\" (UID: \"7ced0135-f0a1-4a1a-ae6a-2d5442bdf66a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c49b28f-6t7dj" Nov 03 01:39:54.360118 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:54.360078 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ced0135-f0a1-4a1a-ae6a-2d5442bdf66a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5c49b28f-6t7dj\" (UID: \"7ced0135-f0a1-4a1a-ae6a-2d5442bdf66a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c49b28f-6t7dj" Nov 03 01:39:54.360602 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:54.360580 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ced0135-f0a1-4a1a-ae6a-2d5442bdf66a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5c49b28f-6t7dj\" (UID: \"7ced0135-f0a1-4a1a-ae6a-2d5442bdf66a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c49b28f-6t7dj" Nov 03 01:39:54.362577 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:54.362553 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ced0135-f0a1-4a1a-ae6a-2d5442bdf66a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5c49b28f-6t7dj\" (UID: \"7ced0135-f0a1-4a1a-ae6a-2d5442bdf66a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c49b28f-6t7dj" Nov 03 01:39:54.367714 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:54.367686 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l84mr\" (UniqueName: \"kubernetes.io/projected/7ced0135-f0a1-4a1a-ae6a-2d5442bdf66a-kube-api-access-l84mr\") pod \"maas-api-revoke-sub-5c49b28f-6t7dj\" (UID: \"7ced0135-f0a1-4a1a-ae6a-2d5442bdf66a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c49b28f-6t7dj" Nov 03 01:39:54.477540 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:54.477499 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c49b28f-6t7dj" Nov 03 01:39:54.765753 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:54.765704 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c49b28f-6t7dj"] Nov 03 01:39:55.742066 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:55.742018 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c49b28f-6t7dj" event={"ID":"7ced0135-f0a1-4a1a-ae6a-2d5442bdf66a","Type":"ContainerStarted","Data":"a4696d8a541ee6115527fd8f074035194e4e918484f994b5da8cad4cb42def4f"} Nov 03 01:39:55.742700 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:55.742672 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c49b28f-6t7dj" event={"ID":"7ced0135-f0a1-4a1a-ae6a-2d5442bdf66a","Type":"ContainerStarted","Data":"5adad9da19d1659ebdf29718f85e146c457159fe818cbdc2d9ff20cc0ecdd377"} Nov 03 01:39:55.766812 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:55.766726 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c49b28f-6t7dj" podStartSLOduration=1.76670642 podStartE2EDuration="1.76670642s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:39:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:39:55.766670544 +0000 UTC m=+3537.937673638" watchObservedRunningTime="2026-11-03 01:39:55.76670642 +0000 UTC m=+3537.937709516" Nov 03 01:39:56.759447 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:56.759408 4421 generic.go:356] "Generic (PLEG): container finished" podID="7ced0135-f0a1-4a1a-ae6a-2d5442bdf66a" containerID="a4696d8a541ee6115527fd8f074035194e4e918484f994b5da8cad4cb42def4f" exitCode=0 Nov 03 01:39:56.760063 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:56.759521 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c49b28f-6t7dj" event={"ID":"7ced0135-f0a1-4a1a-ae6a-2d5442bdf66a","Type":"ContainerDied","Data":"a4696d8a541ee6115527fd8f074035194e4e918484f994b5da8cad4cb42def4f"} Nov 03 01:39:57.938685 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:57.938657 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c49b28f-6t7dj" Nov 03 01:39:58.062399 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:58.062256 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ced0135-f0a1-4a1a-ae6a-2d5442bdf66a-kube-api-access-l84mr\" (UniqueName: \"kubernetes.io/projected/7ced0135-f0a1-4a1a-ae6a-2d5442bdf66a-kube-api-access-l84mr\") pod \"7ced0135-f0a1-4a1a-ae6a-2d5442bdf66a\" (UID: \"7ced0135-f0a1-4a1a-ae6a-2d5442bdf66a\") " Nov 03 01:39:58.062646 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:58.062622 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ced0135-f0a1-4a1a-ae6a-2d5442bdf66a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ced0135-f0a1-4a1a-ae6a-2d5442bdf66a-maas-api-openshift-ca\") pod \"7ced0135-f0a1-4a1a-ae6a-2d5442bdf66a\" (UID: \"7ced0135-f0a1-4a1a-ae6a-2d5442bdf66a\") " Nov 03 01:39:58.062747 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:58.062702 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7ced0135-f0a1-4a1a-ae6a-2d5442bdf66a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ced0135-f0a1-4a1a-ae6a-2d5442bdf66a-maas-api-xks-ca\") pod \"7ced0135-f0a1-4a1a-ae6a-2d5442bdf66a\" (UID: \"7ced0135-f0a1-4a1a-ae6a-2d5442bdf66a\") " Nov 03 01:39:58.065091 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:58.065065 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ced0135-f0a1-4a1a-ae6a-2d5442bdf66a-maas-api-openshift-ca" pod "7ced0135-f0a1-4a1a-ae6a-2d5442bdf66a" (UID: "7ced0135-f0a1-4a1a-ae6a-2d5442bdf66a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:39:58.065606 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:58.065578 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ced0135-f0a1-4a1a-ae6a-2d5442bdf66a-maas-api-xks-ca" pod "7ced0135-f0a1-4a1a-ae6a-2d5442bdf66a" (UID: "7ced0135-f0a1-4a1a-ae6a-2d5442bdf66a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:39:58.066688 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:58.066662 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ced0135-f0a1-4a1a-ae6a-2d5442bdf66a-kube-api-access-l84mr" pod "7ced0135-f0a1-4a1a-ae6a-2d5442bdf66a" (UID: "7ced0135-f0a1-4a1a-ae6a-2d5442bdf66a"). InnerVolumeSpecName "kube-api-access-l84mr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:39:58.165012 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:58.164976 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l84mr\" (UniqueName: \"kubernetes.io/projected/7ced0135-f0a1-4a1a-ae6a-2d5442bdf66a-kube-api-access-l84mr\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:39:58.165249 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:58.165217 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ced0135-f0a1-4a1a-ae6a-2d5442bdf66a-maas-api-openshift-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:39:58.165249 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:58.165246 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ced0135-f0a1-4a1a-ae6a-2d5442bdf66a-maas-api-xks-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:39:58.788672 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:58.788638 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c49b28f-6t7dj" Nov 03 01:39:58.788888 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:58.788672 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c49b28f-6t7dj" event={"ID":"7ced0135-f0a1-4a1a-ae6a-2d5442bdf66a","Type":"ContainerDied","Data":"5adad9da19d1659ebdf29718f85e146c457159fe818cbdc2d9ff20cc0ecdd377"} Nov 03 01:39:58.788888 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:39:58.788750 4421 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5adad9da19d1659ebdf29718f85e146c457159fe818cbdc2d9ff20cc0ecdd377" Nov 03 01:40:04.355541 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:04.355501 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-afa07afc-qcn45"] Nov 03 01:40:04.356194 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:04.356179 4421 cpu_manager.go:414] "containerMap: removing container" podUID="7ced0135-f0a1-4a1a-ae6a-2d5442bdf66a" containerName="revoke-keys" Nov 03 01:40:04.356240 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:04.356195 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ced0135-f0a1-4a1a-ae6a-2d5442bdf66a" containerName="revoke-keys" Nov 03 01:40:04.356355 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:04.356345 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ced0135-f0a1-4a1a-ae6a-2d5442bdf66a" containerName="revoke-keys" Nov 03 01:40:04.360245 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:04.360223 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afa07afc-qcn45" Nov 03 01:40:04.364516 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:04.364489 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:40:04.364663 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:04.364586 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6q4xp\"" Nov 03 01:40:04.365908 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:04.365867 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-afa07afc-qcn45"] Nov 03 01:40:04.425225 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:04.425189 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33e4bab9-aa4d-4a30-b787-8986f8460e4a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-afa07afc-qcn45\" (UID: \"33e4bab9-aa4d-4a30-b787-8986f8460e4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afa07afc-qcn45" Nov 03 01:40:04.425472 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:04.425424 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33e4bab9-aa4d-4a30-b787-8986f8460e4a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-afa07afc-qcn45\" (UID: \"33e4bab9-aa4d-4a30-b787-8986f8460e4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afa07afc-qcn45" Nov 03 01:40:04.425636 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:04.425617 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wszdf\" (UniqueName: \"kubernetes.io/projected/33e4bab9-aa4d-4a30-b787-8986f8460e4a-kube-api-access-wszdf\") pod \"maas-api-revoke-sub-afa07afc-qcn45\" (UID: \"33e4bab9-aa4d-4a30-b787-8986f8460e4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afa07afc-qcn45" Nov 03 01:40:04.527541 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:04.527496 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33e4bab9-aa4d-4a30-b787-8986f8460e4a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-afa07afc-qcn45\" (UID: \"33e4bab9-aa4d-4a30-b787-8986f8460e4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afa07afc-qcn45" Nov 03 01:40:04.527757 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:04.527648 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33e4bab9-aa4d-4a30-b787-8986f8460e4a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-afa07afc-qcn45\" (UID: \"33e4bab9-aa4d-4a30-b787-8986f8460e4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afa07afc-qcn45" Nov 03 01:40:04.527848 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:04.527831 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wszdf\" (UniqueName: \"kubernetes.io/projected/33e4bab9-aa4d-4a30-b787-8986f8460e4a-kube-api-access-wszdf\") pod \"maas-api-revoke-sub-afa07afc-qcn45\" (UID: \"33e4bab9-aa4d-4a30-b787-8986f8460e4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afa07afc-qcn45" Nov 03 01:40:04.528178 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:04.528155 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33e4bab9-aa4d-4a30-b787-8986f8460e4a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-afa07afc-qcn45\" (UID: \"33e4bab9-aa4d-4a30-b787-8986f8460e4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afa07afc-qcn45" Nov 03 01:40:04.530087 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:04.530064 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33e4bab9-aa4d-4a30-b787-8986f8460e4a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-afa07afc-qcn45\" (UID: \"33e4bab9-aa4d-4a30-b787-8986f8460e4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afa07afc-qcn45" Nov 03 01:40:04.535805 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:04.535780 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wszdf\" (UniqueName: \"kubernetes.io/projected/33e4bab9-aa4d-4a30-b787-8986f8460e4a-kube-api-access-wszdf\") pod \"maas-api-revoke-sub-afa07afc-qcn45\" (UID: \"33e4bab9-aa4d-4a30-b787-8986f8460e4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afa07afc-qcn45" Nov 03 01:40:04.679041 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:04.678949 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afa07afc-qcn45" Nov 03 01:40:04.961798 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:04.961771 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-afa07afc-qcn45"] Nov 03 01:40:04.968604 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:40:04.968574 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33e4bab9_aa4d_4a30_b787_8986f8460e4a.slice/crio-0c01d7e9045594c154a51f9e7719b0dbc6b184b0dbe53c23ae95f025a5b7fb6d WatchSource:0}: Error finding container 0c01d7e9045594c154a51f9e7719b0dbc6b184b0dbe53c23ae95f025a5b7fb6d: Status 404 returned error can't find the container with id 0c01d7e9045594c154a51f9e7719b0dbc6b184b0dbe53c23ae95f025a5b7fb6d Nov 03 01:40:05.882460 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:05.882417 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afa07afc-qcn45" event={"ID":"33e4bab9-aa4d-4a30-b787-8986f8460e4a","Type":"ContainerStarted","Data":"9b8a8f6590bfacfa17cc3eef68a085accbcba0b83bef0cc1855deb359a2e35d6"} Nov 03 01:40:05.882899 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:05.882495 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afa07afc-qcn45" event={"ID":"33e4bab9-aa4d-4a30-b787-8986f8460e4a","Type":"ContainerStarted","Data":"0c01d7e9045594c154a51f9e7719b0dbc6b184b0dbe53c23ae95f025a5b7fb6d"} Nov 03 01:40:05.899239 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:05.899185 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afa07afc-qcn45" podStartSLOduration=1.89916837 podStartE2EDuration="1.89916837s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:40:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:40:05.897105844 +0000 UTC m=+3548.068108941" watchObservedRunningTime="2026-11-03 01:40:05.89916837 +0000 UTC m=+3548.070171478" Nov 03 01:40:06.071663 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:06.071632 4421 scope.go:122] "RemoveContainer" containerID="50d7c3c76b192320124562f51823595cdbdee54e7d3e2e68426897b2695cb248" Nov 03 01:40:06.892071 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:06.892025 4421 generic.go:356] "Generic (PLEG): container finished" podID="33e4bab9-aa4d-4a30-b787-8986f8460e4a" containerID="9b8a8f6590bfacfa17cc3eef68a085accbcba0b83bef0cc1855deb359a2e35d6" exitCode=0 Nov 03 01:40:06.892551 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:06.892083 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afa07afc-qcn45" event={"ID":"33e4bab9-aa4d-4a30-b787-8986f8460e4a","Type":"ContainerDied","Data":"9b8a8f6590bfacfa17cc3eef68a085accbcba0b83bef0cc1855deb359a2e35d6"} Nov 03 01:40:08.076653 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:08.076623 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afa07afc-qcn45" Nov 03 01:40:08.141825 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:08.141785 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/33e4bab9-aa4d-4a30-b787-8986f8460e4a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33e4bab9-aa4d-4a30-b787-8986f8460e4a-maas-api-xks-ca\") pod \"33e4bab9-aa4d-4a30-b787-8986f8460e4a\" (UID: \"33e4bab9-aa4d-4a30-b787-8986f8460e4a\") " Nov 03 01:40:08.142051 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:08.141909 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33e4bab9-aa4d-4a30-b787-8986f8460e4a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33e4bab9-aa4d-4a30-b787-8986f8460e4a-maas-api-openshift-ca\") pod \"33e4bab9-aa4d-4a30-b787-8986f8460e4a\" (UID: \"33e4bab9-aa4d-4a30-b787-8986f8460e4a\") " Nov 03 01:40:08.142051 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:08.142004 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33e4bab9-aa4d-4a30-b787-8986f8460e4a-kube-api-access-wszdf\" (UniqueName: \"kubernetes.io/projected/33e4bab9-aa4d-4a30-b787-8986f8460e4a-kube-api-access-wszdf\") pod \"33e4bab9-aa4d-4a30-b787-8986f8460e4a\" (UID: \"33e4bab9-aa4d-4a30-b787-8986f8460e4a\") " Nov 03 01:40:08.142299 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:08.142252 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33e4bab9-aa4d-4a30-b787-8986f8460e4a-maas-api-openshift-ca" pod "33e4bab9-aa4d-4a30-b787-8986f8460e4a" (UID: "33e4bab9-aa4d-4a30-b787-8986f8460e4a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:40:08.142478 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:08.142461 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33e4bab9-aa4d-4a30-b787-8986f8460e4a-maas-api-openshift-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:40:08.144101 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:08.144073 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33e4bab9-aa4d-4a30-b787-8986f8460e4a-maas-api-xks-ca" pod "33e4bab9-aa4d-4a30-b787-8986f8460e4a" (UID: "33e4bab9-aa4d-4a30-b787-8986f8460e4a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:40:08.144259 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:08.144216 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33e4bab9-aa4d-4a30-b787-8986f8460e4a-kube-api-access-wszdf" pod "33e4bab9-aa4d-4a30-b787-8986f8460e4a" (UID: "33e4bab9-aa4d-4a30-b787-8986f8460e4a"). InnerVolumeSpecName "kube-api-access-wszdf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:40:08.243904 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:08.243869 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wszdf\" (UniqueName: \"kubernetes.io/projected/33e4bab9-aa4d-4a30-b787-8986f8460e4a-kube-api-access-wszdf\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:40:08.243904 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:08.243902 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33e4bab9-aa4d-4a30-b787-8986f8460e4a-maas-api-xks-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:40:08.917618 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:08.917581 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afa07afc-qcn45" event={"ID":"33e4bab9-aa4d-4a30-b787-8986f8460e4a","Type":"ContainerDied","Data":"0c01d7e9045594c154a51f9e7719b0dbc6b184b0dbe53c23ae95f025a5b7fb6d"} Nov 03 01:40:08.917618 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:08.917606 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afa07afc-qcn45" Nov 03 01:40:08.917854 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:08.917632 4421 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0c01d7e9045594c154a51f9e7719b0dbc6b184b0dbe53c23ae95f025a5b7fb6d" Nov 03 01:40:14.672553 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:40:14.672462 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 01:40:14.969149 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:14.969059 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f20e1ccc-ds5fz"] Nov 03 01:40:14.969787 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:14.969773 4421 cpu_manager.go:414] "containerMap: removing container" podUID="33e4bab9-aa4d-4a30-b787-8986f8460e4a" containerName="revoke-keys" Nov 03 01:40:14.969857 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:14.969788 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33e4bab9-aa4d-4a30-b787-8986f8460e4a" containerName="revoke-keys" Nov 03 01:40:14.969916 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:14.969906 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="33e4bab9-aa4d-4a30-b787-8986f8460e4a" containerName="revoke-keys" Nov 03 01:40:14.973959 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:14.973937 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f20e1ccc-ds5fz" Nov 03 01:40:14.979153 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:14.979128 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:40:14.980004 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:14.979974 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6q4xp\"" Nov 03 01:40:14.981369 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:14.980943 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f20e1ccc-ds5fz"] Nov 03 01:40:15.090464 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:15.090421 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffa952fe-2ab6-4f56-9afd-a9a9008b2f5c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f20e1ccc-ds5fz\" (UID: \"ffa952fe-2ab6-4f56-9afd-a9a9008b2f5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f20e1ccc-ds5fz" Nov 03 01:40:15.090655 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:15.090549 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lfht\" (UniqueName: \"kubernetes.io/projected/ffa952fe-2ab6-4f56-9afd-a9a9008b2f5c-kube-api-access-8lfht\") pod \"maas-api-revoke-sub-f20e1ccc-ds5fz\" (UID: \"ffa952fe-2ab6-4f56-9afd-a9a9008b2f5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f20e1ccc-ds5fz" Nov 03 01:40:15.090655 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:15.090585 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffa952fe-2ab6-4f56-9afd-a9a9008b2f5c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f20e1ccc-ds5fz\" (UID: \"ffa952fe-2ab6-4f56-9afd-a9a9008b2f5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f20e1ccc-ds5fz" Nov 03 01:40:15.191886 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:15.191848 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffa952fe-2ab6-4f56-9afd-a9a9008b2f5c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f20e1ccc-ds5fz\" (UID: \"ffa952fe-2ab6-4f56-9afd-a9a9008b2f5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f20e1ccc-ds5fz" Nov 03 01:40:15.192093 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:15.191924 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lfht\" (UniqueName: \"kubernetes.io/projected/ffa952fe-2ab6-4f56-9afd-a9a9008b2f5c-kube-api-access-8lfht\") pod \"maas-api-revoke-sub-f20e1ccc-ds5fz\" (UID: \"ffa952fe-2ab6-4f56-9afd-a9a9008b2f5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f20e1ccc-ds5fz" Nov 03 01:40:15.192093 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:15.191954 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffa952fe-2ab6-4f56-9afd-a9a9008b2f5c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f20e1ccc-ds5fz\" (UID: \"ffa952fe-2ab6-4f56-9afd-a9a9008b2f5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f20e1ccc-ds5fz" Nov 03 01:40:15.192703 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:15.192677 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffa952fe-2ab6-4f56-9afd-a9a9008b2f5c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f20e1ccc-ds5fz\" (UID: \"ffa952fe-2ab6-4f56-9afd-a9a9008b2f5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f20e1ccc-ds5fz" Nov 03 01:40:15.194460 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:15.194432 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffa952fe-2ab6-4f56-9afd-a9a9008b2f5c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f20e1ccc-ds5fz\" (UID: \"ffa952fe-2ab6-4f56-9afd-a9a9008b2f5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f20e1ccc-ds5fz" Nov 03 01:40:15.200702 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:15.200678 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lfht\" (UniqueName: \"kubernetes.io/projected/ffa952fe-2ab6-4f56-9afd-a9a9008b2f5c-kube-api-access-8lfht\") pod \"maas-api-revoke-sub-f20e1ccc-ds5fz\" (UID: \"ffa952fe-2ab6-4f56-9afd-a9a9008b2f5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f20e1ccc-ds5fz" Nov 03 01:40:15.308259 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:15.308205 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f20e1ccc-ds5fz" Nov 03 01:40:15.595364 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:15.595331 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f20e1ccc-ds5fz"] Nov 03 01:40:15.595885 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:40:15.595857 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffa952fe_2ab6_4f56_9afd_a9a9008b2f5c.slice/crio-2e6de69a99f7e1ffcf0d49b2cbc051394b74d49345f61a492fdeeaeca6e6ad58 WatchSource:0}: Error finding container 2e6de69a99f7e1ffcf0d49b2cbc051394b74d49345f61a492fdeeaeca6e6ad58: Status 404 returned error can't find the container with id 2e6de69a99f7e1ffcf0d49b2cbc051394b74d49345f61a492fdeeaeca6e6ad58 Nov 03 01:40:15.980897 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:15.980854 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f20e1ccc-ds5fz" event={"ID":"ffa952fe-2ab6-4f56-9afd-a9a9008b2f5c","Type":"ContainerStarted","Data":"a64c3be1661b9cfa805d3ff11606b1d435a52e9a367b980020193381259908d5"} Nov 03 01:40:15.980897 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:15.980891 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f20e1ccc-ds5fz" event={"ID":"ffa952fe-2ab6-4f56-9afd-a9a9008b2f5c","Type":"ContainerStarted","Data":"2e6de69a99f7e1ffcf0d49b2cbc051394b74d49345f61a492fdeeaeca6e6ad58"} Nov 03 01:40:15.999304 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:15.999217 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f20e1ccc-ds5fz" podStartSLOduration=1.9992035179999998 podStartE2EDuration="1.999203518s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:40:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:40:15.994487335 +0000 UTC m=+3558.165490432" watchObservedRunningTime="2026-11-03 01:40:15.999203518 +0000 UTC m=+3558.170206611" Nov 03 01:40:16.988923 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:16.988833 4421 generic.go:356] "Generic (PLEG): container finished" podID="ffa952fe-2ab6-4f56-9afd-a9a9008b2f5c" containerID="a64c3be1661b9cfa805d3ff11606b1d435a52e9a367b980020193381259908d5" exitCode=0 Nov 03 01:40:16.989343 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:16.988911 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f20e1ccc-ds5fz" event={"ID":"ffa952fe-2ab6-4f56-9afd-a9a9008b2f5c","Type":"ContainerDied","Data":"a64c3be1661b9cfa805d3ff11606b1d435a52e9a367b980020193381259908d5"} Nov 03 01:40:18.383985 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:18.383951 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f20e1ccc-ds5fz" Nov 03 01:40:18.422809 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:18.422770 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffa952fe-2ab6-4f56-9afd-a9a9008b2f5c-kube-api-access-8lfht\" (UniqueName: \"kubernetes.io/projected/ffa952fe-2ab6-4f56-9afd-a9a9008b2f5c-kube-api-access-8lfht\") pod \"ffa952fe-2ab6-4f56-9afd-a9a9008b2f5c\" (UID: \"ffa952fe-2ab6-4f56-9afd-a9a9008b2f5c\") " Nov 03 01:40:18.423141 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:18.423115 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ffa952fe-2ab6-4f56-9afd-a9a9008b2f5c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffa952fe-2ab6-4f56-9afd-a9a9008b2f5c-maas-api-xks-ca\") pod \"ffa952fe-2ab6-4f56-9afd-a9a9008b2f5c\" (UID: \"ffa952fe-2ab6-4f56-9afd-a9a9008b2f5c\") " Nov 03 01:40:18.423462 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:18.423439 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ffa952fe-2ab6-4f56-9afd-a9a9008b2f5c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffa952fe-2ab6-4f56-9afd-a9a9008b2f5c-maas-api-openshift-ca\") pod \"ffa952fe-2ab6-4f56-9afd-a9a9008b2f5c\" (UID: \"ffa952fe-2ab6-4f56-9afd-a9a9008b2f5c\") " Nov 03 01:40:18.424329 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:18.423788 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffa952fe-2ab6-4f56-9afd-a9a9008b2f5c-maas-api-openshift-ca" pod "ffa952fe-2ab6-4f56-9afd-a9a9008b2f5c" (UID: "ffa952fe-2ab6-4f56-9afd-a9a9008b2f5c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:40:18.424329 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:18.424298 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffa952fe-2ab6-4f56-9afd-a9a9008b2f5c-maas-api-openshift-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:40:18.428766 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:18.428722 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffa952fe-2ab6-4f56-9afd-a9a9008b2f5c-maas-api-xks-ca" pod "ffa952fe-2ab6-4f56-9afd-a9a9008b2f5c" (UID: "ffa952fe-2ab6-4f56-9afd-a9a9008b2f5c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:40:18.430954 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:18.430916 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffa952fe-2ab6-4f56-9afd-a9a9008b2f5c-kube-api-access-8lfht" pod "ffa952fe-2ab6-4f56-9afd-a9a9008b2f5c" (UID: "ffa952fe-2ab6-4f56-9afd-a9a9008b2f5c"). InnerVolumeSpecName "kube-api-access-8lfht". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:40:18.525985 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:18.525943 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8lfht\" (UniqueName: \"kubernetes.io/projected/ffa952fe-2ab6-4f56-9afd-a9a9008b2f5c-kube-api-access-8lfht\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:40:18.525985 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:18.525981 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffa952fe-2ab6-4f56-9afd-a9a9008b2f5c-maas-api-xks-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:40:19.026592 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:19.026556 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6dac96c1-8tdw5"] Nov 03 01:40:19.030516 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:19.030470 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6dac96c1-8tdw5"] Nov 03 01:40:19.032385 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:19.032360 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f20e1ccc-ds5fz" Nov 03 01:40:19.032523 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:19.032406 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f20e1ccc-ds5fz" event={"ID":"ffa952fe-2ab6-4f56-9afd-a9a9008b2f5c","Type":"ContainerDied","Data":"2e6de69a99f7e1ffcf0d49b2cbc051394b74d49345f61a492fdeeaeca6e6ad58"} Nov 03 01:40:19.032523 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:19.032462 4421 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2e6de69a99f7e1ffcf0d49b2cbc051394b74d49345f61a492fdeeaeca6e6ad58" Nov 03 01:40:19.980728 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:19.980684 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03015bfc-aaac-4f5a-b909-790a18fc61f3" path="/var/lib/kubelet/pods/03015bfc-aaac-4f5a-b909-790a18fc61f3/volumes" Nov 03 01:40:19.995876 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:19.995840 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-bx28n"] Nov 03 01:40:19.996822 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:19.996785 4421 cpu_manager.go:414] "containerMap: removing container" podUID="ffa952fe-2ab6-4f56-9afd-a9a9008b2f5c" containerName="revoke-keys" Nov 03 01:40:19.996822 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:19.996813 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffa952fe-2ab6-4f56-9afd-a9a9008b2f5c" containerName="revoke-keys" Nov 03 01:40:19.997025 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:19.997009 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffa952fe-2ab6-4f56-9afd-a9a9008b2f5c" containerName="revoke-keys" Nov 03 01:40:20.001530 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:20.001508 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-bx28n" Nov 03 01:40:20.008051 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:20.008013 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-4a697f-openshift-default-dockercfg-wr6rf\"" Nov 03 01:40:20.009026 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:20.008998 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-bx28n"] Nov 03 01:40:20.049297 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:20.049236 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b44153f-12a2-4b0b-898a-493b372fb7ae-credential-socket\") pod \"e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-bx28n\" (UID: \"7b44153f-12a2-4b0b-898a-493b372fb7ae\") " pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-bx28n" Nov 03 01:40:20.049504 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:20.049304 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l889w\" (UniqueName: \"kubernetes.io/projected/7b44153f-12a2-4b0b-898a-493b372fb7ae-kube-api-access-l889w\") pod \"e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-bx28n\" (UID: \"7b44153f-12a2-4b0b-898a-493b372fb7ae\") " pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-bx28n" Nov 03 01:40:20.049504 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:20.049392 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b44153f-12a2-4b0b-898a-493b372fb7ae-istio-envoy\") pod \"e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-bx28n\" (UID: \"7b44153f-12a2-4b0b-898a-493b372fb7ae\") " pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-bx28n" Nov 03 01:40:20.049504 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:20.049450 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b44153f-12a2-4b0b-898a-493b372fb7ae-istiod-ca-cert\") pod \"e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-bx28n\" (UID: \"7b44153f-12a2-4b0b-898a-493b372fb7ae\") " pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-bx28n" Nov 03 01:40:20.049504 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:20.049477 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b44153f-12a2-4b0b-898a-493b372fb7ae-istio-data\") pod \"e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-bx28n\" (UID: \"7b44153f-12a2-4b0b-898a-493b372fb7ae\") " pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-bx28n" Nov 03 01:40:20.049770 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:20.049541 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b44153f-12a2-4b0b-898a-493b372fb7ae-workload-socket\") pod \"e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-bx28n\" (UID: \"7b44153f-12a2-4b0b-898a-493b372fb7ae\") " pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-bx28n" Nov 03 01:40:20.049770 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:20.049611 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b44153f-12a2-4b0b-898a-493b372fb7ae-istio-podinfo\") pod \"e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-bx28n\" (UID: \"7b44153f-12a2-4b0b-898a-493b372fb7ae\") " pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-bx28n" Nov 03 01:40:20.049770 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:20.049698 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b44153f-12a2-4b0b-898a-493b372fb7ae-istio-token\") pod \"e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-bx28n\" (UID: \"7b44153f-12a2-4b0b-898a-493b372fb7ae\") " pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-bx28n" Nov 03 01:40:20.049957 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:20.049879 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b44153f-12a2-4b0b-898a-493b372fb7ae-workload-certs\") pod \"e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-bx28n\" (UID: \"7b44153f-12a2-4b0b-898a-493b372fb7ae\") " pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-bx28n" Nov 03 01:40:20.151603 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:20.151563 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b44153f-12a2-4b0b-898a-493b372fb7ae-workload-certs\") pod \"e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-bx28n\" (UID: \"7b44153f-12a2-4b0b-898a-493b372fb7ae\") " pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-bx28n" Nov 03 01:40:20.151792 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:20.151713 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b44153f-12a2-4b0b-898a-493b372fb7ae-credential-socket\") pod \"e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-bx28n\" (UID: \"7b44153f-12a2-4b0b-898a-493b372fb7ae\") " pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-bx28n" Nov 03 01:40:20.151792 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:20.151747 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l889w\" (UniqueName: \"kubernetes.io/projected/7b44153f-12a2-4b0b-898a-493b372fb7ae-kube-api-access-l889w\") pod \"e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-bx28n\" (UID: \"7b44153f-12a2-4b0b-898a-493b372fb7ae\") " pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-bx28n" Nov 03 01:40:20.151880 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:20.151793 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b44153f-12a2-4b0b-898a-493b372fb7ae-istio-envoy\") pod \"e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-bx28n\" (UID: \"7b44153f-12a2-4b0b-898a-493b372fb7ae\") " pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-bx28n" Nov 03 01:40:20.151880 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:20.151844 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b44153f-12a2-4b0b-898a-493b372fb7ae-istiod-ca-cert\") pod \"e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-bx28n\" (UID: \"7b44153f-12a2-4b0b-898a-493b372fb7ae\") " pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-bx28n" Nov 03 01:40:20.151880 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:20.151872 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b44153f-12a2-4b0b-898a-493b372fb7ae-istio-data\") pod \"e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-bx28n\" (UID: \"7b44153f-12a2-4b0b-898a-493b372fb7ae\") " pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-bx28n" Nov 03 01:40:20.152024 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:20.151963 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b44153f-12a2-4b0b-898a-493b372fb7ae-workload-socket\") pod \"e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-bx28n\" (UID: \"7b44153f-12a2-4b0b-898a-493b372fb7ae\") " pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-bx28n" Nov 03 01:40:20.152090 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:20.152055 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b44153f-12a2-4b0b-898a-493b372fb7ae-istio-podinfo\") pod \"e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-bx28n\" (UID: \"7b44153f-12a2-4b0b-898a-493b372fb7ae\") " pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-bx28n" Nov 03 01:40:20.152220 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:20.152181 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b44153f-12a2-4b0b-898a-493b372fb7ae-istio-token\") pod \"e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-bx28n\" (UID: \"7b44153f-12a2-4b0b-898a-493b372fb7ae\") " pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-bx28n" Nov 03 01:40:20.152312 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:20.152056 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b44153f-12a2-4b0b-898a-493b372fb7ae-workload-certs\") pod \"e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-bx28n\" (UID: \"7b44153f-12a2-4b0b-898a-493b372fb7ae\") " pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-bx28n" Nov 03 01:40:20.152312 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:20.152261 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b44153f-12a2-4b0b-898a-493b372fb7ae-istio-data\") pod \"e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-bx28n\" (UID: \"7b44153f-12a2-4b0b-898a-493b372fb7ae\") " pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-bx28n" Nov 03 01:40:20.152437 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:20.152359 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b44153f-12a2-4b0b-898a-493b372fb7ae-workload-socket\") pod \"e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-bx28n\" (UID: \"7b44153f-12a2-4b0b-898a-493b372fb7ae\") " pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-bx28n" Nov 03 01:40:20.152437 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:20.152407 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b44153f-12a2-4b0b-898a-493b372fb7ae-credential-socket\") pod \"e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-bx28n\" (UID: \"7b44153f-12a2-4b0b-898a-493b372fb7ae\") " pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-bx28n" Nov 03 01:40:20.152657 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:20.152637 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b44153f-12a2-4b0b-898a-493b372fb7ae-istiod-ca-cert\") pod \"e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-bx28n\" (UID: \"7b44153f-12a2-4b0b-898a-493b372fb7ae\") " pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-bx28n" Nov 03 01:40:20.154304 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:20.154225 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b44153f-12a2-4b0b-898a-493b372fb7ae-istio-envoy\") pod \"e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-bx28n\" (UID: \"7b44153f-12a2-4b0b-898a-493b372fb7ae\") " pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-bx28n" Nov 03 01:40:20.154780 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:20.154759 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b44153f-12a2-4b0b-898a-493b372fb7ae-istio-podinfo\") pod \"e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-bx28n\" (UID: \"7b44153f-12a2-4b0b-898a-493b372fb7ae\") " pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-bx28n" Nov 03 01:40:20.159588 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:20.159565 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b44153f-12a2-4b0b-898a-493b372fb7ae-istio-token\") pod \"e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-bx28n\" (UID: \"7b44153f-12a2-4b0b-898a-493b372fb7ae\") " pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-bx28n" Nov 03 01:40:20.159832 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:20.159808 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l889w\" (UniqueName: \"kubernetes.io/projected/7b44153f-12a2-4b0b-898a-493b372fb7ae-kube-api-access-l889w\") pod \"e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-bx28n\" (UID: \"7b44153f-12a2-4b0b-898a-493b372fb7ae\") " pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-bx28n" Nov 03 01:40:20.321567 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:20.321528 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-bx28n" Nov 03 01:40:20.600349 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:20.600247 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2c966aa-vjmw5"] Nov 03 01:40:20.605875 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:20.605855 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2c966aa-vjmw5" Nov 03 01:40:20.610891 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:20.610860 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:40:20.611028 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:20.610985 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6q4xp\"" Nov 03 01:40:20.613350 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:20.612845 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-bx28n"] Nov 03 01:40:20.617404 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:20.617350 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:40:20.617517 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:20.617447 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:40:20.617517 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:20.617488 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:40:20.617897 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:20.617491 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2c966aa-vjmw5"] Nov 03 01:40:20.668795 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:20.668765 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e0718b6-a990-4b70-a0a4-873956e942b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2c966aa-vjmw5\" (UID: \"6e0718b6-a990-4b70-a0a4-873956e942b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2c966aa-vjmw5" Nov 03 01:40:20.669157 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:20.669135 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e0718b6-a990-4b70-a0a4-873956e942b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2c966aa-vjmw5\" (UID: \"6e0718b6-a990-4b70-a0a4-873956e942b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2c966aa-vjmw5" Nov 03 01:40:20.669528 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:20.669494 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfmsf\" (UniqueName: \"kubernetes.io/projected/6e0718b6-a990-4b70-a0a4-873956e942b0-kube-api-access-bfmsf\") pod \"maas-api-revoke-sub-d2c966aa-vjmw5\" (UID: \"6e0718b6-a990-4b70-a0a4-873956e942b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2c966aa-vjmw5" Nov 03 01:40:20.771241 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:20.771202 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e0718b6-a990-4b70-a0a4-873956e942b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2c966aa-vjmw5\" (UID: \"6e0718b6-a990-4b70-a0a4-873956e942b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2c966aa-vjmw5" Nov 03 01:40:20.771470 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:20.771351 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfmsf\" (UniqueName: \"kubernetes.io/projected/6e0718b6-a990-4b70-a0a4-873956e942b0-kube-api-access-bfmsf\") pod \"maas-api-revoke-sub-d2c966aa-vjmw5\" (UID: \"6e0718b6-a990-4b70-a0a4-873956e942b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2c966aa-vjmw5" Nov 03 01:40:20.771547 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:20.771473 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e0718b6-a990-4b70-a0a4-873956e942b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2c966aa-vjmw5\" (UID: \"6e0718b6-a990-4b70-a0a4-873956e942b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2c966aa-vjmw5" Nov 03 01:40:20.772001 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:20.771975 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e0718b6-a990-4b70-a0a4-873956e942b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2c966aa-vjmw5\" (UID: \"6e0718b6-a990-4b70-a0a4-873956e942b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2c966aa-vjmw5" Nov 03 01:40:20.773903 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:20.773884 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e0718b6-a990-4b70-a0a4-873956e942b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2c966aa-vjmw5\" (UID: \"6e0718b6-a990-4b70-a0a4-873956e942b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2c966aa-vjmw5" Nov 03 01:40:20.780921 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:20.780896 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfmsf\" (UniqueName: \"kubernetes.io/projected/6e0718b6-a990-4b70-a0a4-873956e942b0-kube-api-access-bfmsf\") pod \"maas-api-revoke-sub-d2c966aa-vjmw5\" (UID: \"6e0718b6-a990-4b70-a0a4-873956e942b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2c966aa-vjmw5" Nov 03 01:40:20.954224 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:20.954135 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2c966aa-vjmw5" Nov 03 01:40:21.057455 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:21.057413 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-bx28n" event={"ID":"7b44153f-12a2-4b0b-898a-493b372fb7ae","Type":"ContainerStarted","Data":"20ca046e7a0306502f4df0894925f2fa0d360f924086bf218ce935538bab71e1"} Nov 03 01:40:21.058029 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:21.057997 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-bx28n" event={"ID":"7b44153f-12a2-4b0b-898a-493b372fb7ae","Type":"ContainerStarted","Data":"ee1c4f85b490e97e004538c8718093d0bf0e1c38df53b7cadd35bcf567c4643e"} Nov 03 01:40:21.087428 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:21.087370 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-bx28n" podStartSLOduration=2.087355118 podStartE2EDuration="2.087355118s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:40:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:40:21.083708048 +0000 UTC m=+3563.254711148" watchObservedRunningTime="2026-11-03 01:40:21.087355118 +0000 UTC m=+3563.258358214" Nov 03 01:40:21.269970 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:21.269943 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2c966aa-vjmw5"] Nov 03 01:40:21.271561 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:40:21.271528 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e0718b6_a990_4b70_a0a4_873956e942b0.slice/crio-fd20f85d3f964ef6067ff66cdfaf8f9b054a9657d9a2e644b3b71cffb7df3afb WatchSource:0}: Error finding container fd20f85d3f964ef6067ff66cdfaf8f9b054a9657d9a2e644b3b71cffb7df3afb: Status 404 returned error can't find the container with id fd20f85d3f964ef6067ff66cdfaf8f9b054a9657d9a2e644b3b71cffb7df3afb Nov 03 01:40:21.329380 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:21.326477 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-bx28n" Nov 03 01:40:21.329380 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:21.328395 4421 patch_prober.go:29] interesting pod/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-bx28n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.119:15021/healthz/ready\": dial tcp 10.134.1.119:15021: connect: connection refused" start-of-body= Nov 03 01:40:21.329380 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:21.328452 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-bx28n" podUID="7b44153f-12a2-4b0b-898a-493b372fb7ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.119:15021/healthz/ready\": dial tcp 10.134.1.119:15021: connect: connection refused" Nov 03 01:40:22.014299 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:22.005528 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-fqqx7"] Nov 03 01:40:22.037295 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:22.032527 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-fqqx7" Nov 03 01:40:22.050710 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:22.050678 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-556c14-openshift-default-dockercfg-bpvll\"" Nov 03 01:40:22.090797 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:22.073174 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-fqqx7"] Nov 03 01:40:22.102931 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:22.099514 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/392e8058-14f9-4dfb-b20e-3e209eb3c625-credential-socket\") pod \"e2e-shared-b-w3-556c14-openshift-default-754b5bc799-fqqx7\" (UID: \"392e8058-14f9-4dfb-b20e-3e209eb3c625\") " pod="openshift-ingress/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-fqqx7" Nov 03 01:40:22.102931 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:22.099583 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/392e8058-14f9-4dfb-b20e-3e209eb3c625-istio-podinfo\") pod \"e2e-shared-b-w3-556c14-openshift-default-754b5bc799-fqqx7\" (UID: \"392e8058-14f9-4dfb-b20e-3e209eb3c625\") " pod="openshift-ingress/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-fqqx7" Nov 03 01:40:22.102931 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:22.099639 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/392e8058-14f9-4dfb-b20e-3e209eb3c625-workload-socket\") pod \"e2e-shared-b-w3-556c14-openshift-default-754b5bc799-fqqx7\" (UID: \"392e8058-14f9-4dfb-b20e-3e209eb3c625\") " pod="openshift-ingress/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-fqqx7" Nov 03 01:40:22.102931 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:22.099676 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/392e8058-14f9-4dfb-b20e-3e209eb3c625-istio-data\") pod \"e2e-shared-b-w3-556c14-openshift-default-754b5bc799-fqqx7\" (UID: \"392e8058-14f9-4dfb-b20e-3e209eb3c625\") " pod="openshift-ingress/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-fqqx7" Nov 03 01:40:22.102931 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:22.099723 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5qmx\" (UniqueName: \"kubernetes.io/projected/392e8058-14f9-4dfb-b20e-3e209eb3c625-kube-api-access-g5qmx\") pod \"e2e-shared-b-w3-556c14-openshift-default-754b5bc799-fqqx7\" (UID: \"392e8058-14f9-4dfb-b20e-3e209eb3c625\") " pod="openshift-ingress/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-fqqx7" Nov 03 01:40:22.102931 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:22.099782 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/392e8058-14f9-4dfb-b20e-3e209eb3c625-workload-certs\") pod \"e2e-shared-b-w3-556c14-openshift-default-754b5bc799-fqqx7\" (UID: \"392e8058-14f9-4dfb-b20e-3e209eb3c625\") " pod="openshift-ingress/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-fqqx7" Nov 03 01:40:22.102931 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:22.099859 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/392e8058-14f9-4dfb-b20e-3e209eb3c625-istio-envoy\") pod \"e2e-shared-b-w3-556c14-openshift-default-754b5bc799-fqqx7\" (UID: \"392e8058-14f9-4dfb-b20e-3e209eb3c625\") " pod="openshift-ingress/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-fqqx7" Nov 03 01:40:22.102931 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:22.099885 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/392e8058-14f9-4dfb-b20e-3e209eb3c625-istiod-ca-cert\") pod \"e2e-shared-b-w3-556c14-openshift-default-754b5bc799-fqqx7\" (UID: \"392e8058-14f9-4dfb-b20e-3e209eb3c625\") " pod="openshift-ingress/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-fqqx7" Nov 03 01:40:22.102931 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:22.099920 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/392e8058-14f9-4dfb-b20e-3e209eb3c625-istio-token\") pod \"e2e-shared-b-w3-556c14-openshift-default-754b5bc799-fqqx7\" (UID: \"392e8058-14f9-4dfb-b20e-3e209eb3c625\") " pod="openshift-ingress/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-fqqx7" Nov 03 01:40:22.139758 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:22.139393 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2c966aa-vjmw5" event={"ID":"6e0718b6-a990-4b70-a0a4-873956e942b0","Type":"ContainerStarted","Data":"c754d9a944e5d2010843a3ee1e546e66b2cf643cb59c62ee8fdc7617c6381932"} Nov 03 01:40:22.139758 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:22.139462 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2c966aa-vjmw5" event={"ID":"6e0718b6-a990-4b70-a0a4-873956e942b0","Type":"ContainerStarted","Data":"fd20f85d3f964ef6067ff66cdfaf8f9b054a9657d9a2e644b3b71cffb7df3afb"} Nov 03 01:40:22.157894 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:22.157841 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2c966aa-vjmw5" podStartSLOduration=2.157827119 podStartE2EDuration="2.157827119s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:40:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:40:22.154846125 +0000 UTC m=+3564.325849222" watchObservedRunningTime="2026-11-03 01:40:22.157827119 +0000 UTC m=+3564.328830211" Nov 03 01:40:22.201149 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:22.201108 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/392e8058-14f9-4dfb-b20e-3e209eb3c625-istio-envoy\") pod \"e2e-shared-b-w3-556c14-openshift-default-754b5bc799-fqqx7\" (UID: \"392e8058-14f9-4dfb-b20e-3e209eb3c625\") " pod="openshift-ingress/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-fqqx7" Nov 03 01:40:22.201149 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:22.201151 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/392e8058-14f9-4dfb-b20e-3e209eb3c625-istiod-ca-cert\") pod \"e2e-shared-b-w3-556c14-openshift-default-754b5bc799-fqqx7\" (UID: \"392e8058-14f9-4dfb-b20e-3e209eb3c625\") " pod="openshift-ingress/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-fqqx7" Nov 03 01:40:22.201486 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:22.201216 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/392e8058-14f9-4dfb-b20e-3e209eb3c625-istio-token\") pod \"e2e-shared-b-w3-556c14-openshift-default-754b5bc799-fqqx7\" (UID: \"392e8058-14f9-4dfb-b20e-3e209eb3c625\") " pod="openshift-ingress/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-fqqx7" Nov 03 01:40:22.201551 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:22.201514 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/392e8058-14f9-4dfb-b20e-3e209eb3c625-credential-socket\") pod \"e2e-shared-b-w3-556c14-openshift-default-754b5bc799-fqqx7\" (UID: \"392e8058-14f9-4dfb-b20e-3e209eb3c625\") " pod="openshift-ingress/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-fqqx7" Nov 03 01:40:22.201637 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:22.201610 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/392e8058-14f9-4dfb-b20e-3e209eb3c625-istio-podinfo\") pod \"e2e-shared-b-w3-556c14-openshift-default-754b5bc799-fqqx7\" (UID: \"392e8058-14f9-4dfb-b20e-3e209eb3c625\") " pod="openshift-ingress/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-fqqx7" Nov 03 01:40:22.201727 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:22.201705 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/392e8058-14f9-4dfb-b20e-3e209eb3c625-workload-socket\") pod \"e2e-shared-b-w3-556c14-openshift-default-754b5bc799-fqqx7\" (UID: \"392e8058-14f9-4dfb-b20e-3e209eb3c625\") " pod="openshift-ingress/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-fqqx7" Nov 03 01:40:22.201819 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:22.201794 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/392e8058-14f9-4dfb-b20e-3e209eb3c625-credential-socket\") pod \"e2e-shared-b-w3-556c14-openshift-default-754b5bc799-fqqx7\" (UID: \"392e8058-14f9-4dfb-b20e-3e209eb3c625\") " pod="openshift-ingress/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-fqqx7" Nov 03 01:40:22.202013 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:22.201992 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/392e8058-14f9-4dfb-b20e-3e209eb3c625-workload-socket\") pod \"e2e-shared-b-w3-556c14-openshift-default-754b5bc799-fqqx7\" (UID: \"392e8058-14f9-4dfb-b20e-3e209eb3c625\") " pod="openshift-ingress/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-fqqx7" Nov 03 01:40:22.202092 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:22.202064 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/392e8058-14f9-4dfb-b20e-3e209eb3c625-istio-data\") pod \"e2e-shared-b-w3-556c14-openshift-default-754b5bc799-fqqx7\" (UID: \"392e8058-14f9-4dfb-b20e-3e209eb3c625\") " pod="openshift-ingress/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-fqqx7" Nov 03 01:40:22.202161 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:22.202086 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/392e8058-14f9-4dfb-b20e-3e209eb3c625-istiod-ca-cert\") pod \"e2e-shared-b-w3-556c14-openshift-default-754b5bc799-fqqx7\" (UID: \"392e8058-14f9-4dfb-b20e-3e209eb3c625\") " pod="openshift-ingress/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-fqqx7" Nov 03 01:40:22.202161 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:22.202117 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5qmx\" (UniqueName: \"kubernetes.io/projected/392e8058-14f9-4dfb-b20e-3e209eb3c625-kube-api-access-g5qmx\") pod \"e2e-shared-b-w3-556c14-openshift-default-754b5bc799-fqqx7\" (UID: \"392e8058-14f9-4dfb-b20e-3e209eb3c625\") " pod="openshift-ingress/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-fqqx7" Nov 03 01:40:22.202312 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:22.202216 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/392e8058-14f9-4dfb-b20e-3e209eb3c625-workload-certs\") pod \"e2e-shared-b-w3-556c14-openshift-default-754b5bc799-fqqx7\" (UID: \"392e8058-14f9-4dfb-b20e-3e209eb3c625\") " pod="openshift-ingress/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-fqqx7" Nov 03 01:40:22.202480 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:22.202448 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/392e8058-14f9-4dfb-b20e-3e209eb3c625-istio-data\") pod \"e2e-shared-b-w3-556c14-openshift-default-754b5bc799-fqqx7\" (UID: \"392e8058-14f9-4dfb-b20e-3e209eb3c625\") " pod="openshift-ingress/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-fqqx7" Nov 03 01:40:22.202535 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:22.202507 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/392e8058-14f9-4dfb-b20e-3e209eb3c625-workload-certs\") pod \"e2e-shared-b-w3-556c14-openshift-default-754b5bc799-fqqx7\" (UID: \"392e8058-14f9-4dfb-b20e-3e209eb3c625\") " pod="openshift-ingress/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-fqqx7" Nov 03 01:40:22.203770 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:22.203749 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/392e8058-14f9-4dfb-b20e-3e209eb3c625-istio-envoy\") pod \"e2e-shared-b-w3-556c14-openshift-default-754b5bc799-fqqx7\" (UID: \"392e8058-14f9-4dfb-b20e-3e209eb3c625\") " pod="openshift-ingress/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-fqqx7" Nov 03 01:40:22.204239 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:22.204222 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/392e8058-14f9-4dfb-b20e-3e209eb3c625-istio-podinfo\") pod \"e2e-shared-b-w3-556c14-openshift-default-754b5bc799-fqqx7\" (UID: \"392e8058-14f9-4dfb-b20e-3e209eb3c625\") " pod="openshift-ingress/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-fqqx7" Nov 03 01:40:22.236377 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:22.236163 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5qmx\" (UniqueName: \"kubernetes.io/projected/392e8058-14f9-4dfb-b20e-3e209eb3c625-kube-api-access-g5qmx\") pod \"e2e-shared-b-w3-556c14-openshift-default-754b5bc799-fqqx7\" (UID: \"392e8058-14f9-4dfb-b20e-3e209eb3c625\") " pod="openshift-ingress/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-fqqx7" Nov 03 01:40:22.237837 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:22.237808 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/392e8058-14f9-4dfb-b20e-3e209eb3c625-istio-token\") pod \"e2e-shared-b-w3-556c14-openshift-default-754b5bc799-fqqx7\" (UID: \"392e8058-14f9-4dfb-b20e-3e209eb3c625\") " pod="openshift-ingress/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-fqqx7" Nov 03 01:40:22.331834 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:22.331787 4421 patch_prober.go:29] interesting pod/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-bx28n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.119:15021/healthz/ready\": dial tcp 10.134.1.119:15021: connect: connection refused" start-of-body= Nov 03 01:40:22.332072 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:22.331859 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-bx28n" podUID="7b44153f-12a2-4b0b-898a-493b372fb7ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.119:15021/healthz/ready\": dial tcp 10.134.1.119:15021: connect: connection refused" Nov 03 01:40:22.452498 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:22.452405 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-fqqx7" Nov 03 01:40:22.767602 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:22.767572 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-fqqx7"] Nov 03 01:40:22.768403 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:40:22.768374 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod392e8058_14f9_4dfb_b20e_3e209eb3c625.slice/crio-ad90ee00d3d6134f4e88d41e8b409c8c053a527c7f113aaf4b7b5f08f07bd669 WatchSource:0}: Error finding container ad90ee00d3d6134f4e88d41e8b409c8c053a527c7f113aaf4b7b5f08f07bd669: Status 404 returned error can't find the container with id ad90ee00d3d6134f4e88d41e8b409c8c053a527c7f113aaf4b7b5f08f07bd669 Nov 03 01:40:22.771587 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:22.771546 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:40:22.771673 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:22.771639 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:40:22.771756 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:22.771674 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:40:23.160769 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:23.160682 4421 generic.go:356] "Generic (PLEG): container finished" podID="6e0718b6-a990-4b70-a0a4-873956e942b0" containerID="c754d9a944e5d2010843a3ee1e546e66b2cf643cb59c62ee8fdc7617c6381932" exitCode=0 Nov 03 01:40:23.161160 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:23.160780 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2c966aa-vjmw5" event={"ID":"6e0718b6-a990-4b70-a0a4-873956e942b0","Type":"ContainerDied","Data":"c754d9a944e5d2010843a3ee1e546e66b2cf643cb59c62ee8fdc7617c6381932"} Nov 03 01:40:23.162777 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:23.162748 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-fqqx7" event={"ID":"392e8058-14f9-4dfb-b20e-3e209eb3c625","Type":"ContainerStarted","Data":"9c0f1c5153a027730dca45af068c6bab21690e517eee27a99c60c527f90ccc9a"} Nov 03 01:40:23.162918 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:23.162799 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-fqqx7" event={"ID":"392e8058-14f9-4dfb-b20e-3e209eb3c625","Type":"ContainerStarted","Data":"ad90ee00d3d6134f4e88d41e8b409c8c053a527c7f113aaf4b7b5f08f07bd669"} Nov 03 01:40:23.200483 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:23.198367 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-fqqx7" podStartSLOduration=2.1983473829999998 podStartE2EDuration="2.198347383s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:40:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:40:23.195119714 +0000 UTC m=+3565.366122802" watchObservedRunningTime="2026-11-03 01:40:23.198347383 +0000 UTC m=+3565.369350480" Nov 03 01:40:23.337316 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:23.335591 4421 patch_prober.go:29] interesting pod/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-bx28n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.119:15021/healthz/ready\": dial tcp 10.134.1.119:15021: connect: connection refused" start-of-body= Nov 03 01:40:23.337316 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:23.335654 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-bx28n" podUID="7b44153f-12a2-4b0b-898a-493b372fb7ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.119:15021/healthz/ready\": dial tcp 10.134.1.119:15021: connect: connection refused" Nov 03 01:40:23.453679 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:23.453587 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-fqqx7" Nov 03 01:40:23.455036 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:23.455004 4421 patch_prober.go:29] interesting pod/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-fqqx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.121:15021/healthz/ready\": dial tcp 10.134.1.121:15021: connect: connection refused" start-of-body= Nov 03 01:40:23.455176 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:23.455065 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-fqqx7" podUID="392e8058-14f9-4dfb-b20e-3e209eb3c625" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.121:15021/healthz/ready\": dial tcp 10.134.1.121:15021: connect: connection refused" Nov 03 01:40:24.365362 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:24.361222 4421 patch_prober.go:29] interesting pod/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-bx28n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.119:15021/healthz/ready\": dial tcp 10.134.1.119:15021: connect: connection refused" start-of-body= Nov 03 01:40:24.365362 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:24.361320 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-bx28n" podUID="7b44153f-12a2-4b0b-898a-493b372fb7ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.119:15021/healthz/ready\": dial tcp 10.134.1.119:15021: connect: connection refused" Nov 03 01:40:24.375013 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:24.374493 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-531447-667d774bfc-9shwq"] Nov 03 01:40:24.381161 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:24.381125 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-531447-7cf7bc4dcf-xt4bm"] Nov 03 01:40:24.464591 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:24.464488 4421 patch_prober.go:29] interesting pod/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-fqqx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.121:15021/healthz/ready\": dial tcp 10.134.1.121:15021: connect: connection refused" start-of-body= Nov 03 01:40:24.470487 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:24.464843 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-fqqx7" podUID="392e8058-14f9-4dfb-b20e-3e209eb3c625" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.121:15021/healthz/ready\": dial tcp 10.134.1.121:15021: connect: connection refused" Nov 03 01:40:24.570732 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:24.570698 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2c966aa-vjmw5" Nov 03 01:40:24.681553 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:24.681520 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e0718b6-a990-4b70-a0a4-873956e942b0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e0718b6-a990-4b70-a0a4-873956e942b0-maas-api-openshift-ca\") pod \"6e0718b6-a990-4b70-a0a4-873956e942b0\" (UID: \"6e0718b6-a990-4b70-a0a4-873956e942b0\") " Nov 03 01:40:24.681769 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:24.681687 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e0718b6-a990-4b70-a0a4-873956e942b0-kube-api-access-bfmsf\" (UniqueName: \"kubernetes.io/projected/6e0718b6-a990-4b70-a0a4-873956e942b0-kube-api-access-bfmsf\") pod \"6e0718b6-a990-4b70-a0a4-873956e942b0\" (UID: \"6e0718b6-a990-4b70-a0a4-873956e942b0\") " Nov 03 01:40:24.681927 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:24.681901 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e0718b6-a990-4b70-a0a4-873956e942b0-maas-api-openshift-ca" pod "6e0718b6-a990-4b70-a0a4-873956e942b0" (UID: "6e0718b6-a990-4b70-a0a4-873956e942b0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:40:24.682009 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:24.681907 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6e0718b6-a990-4b70-a0a4-873956e942b0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e0718b6-a990-4b70-a0a4-873956e942b0-maas-api-xks-ca\") pod \"6e0718b6-a990-4b70-a0a4-873956e942b0\" (UID: \"6e0718b6-a990-4b70-a0a4-873956e942b0\") " Nov 03 01:40:24.682584 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:24.682560 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e0718b6-a990-4b70-a0a4-873956e942b0-maas-api-openshift-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:40:24.684210 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:24.684175 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e0718b6-a990-4b70-a0a4-873956e942b0-kube-api-access-bfmsf" pod "6e0718b6-a990-4b70-a0a4-873956e942b0" (UID: "6e0718b6-a990-4b70-a0a4-873956e942b0"). InnerVolumeSpecName "kube-api-access-bfmsf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:40:24.687929 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:24.687883 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e0718b6-a990-4b70-a0a4-873956e942b0-maas-api-xks-ca" pod "6e0718b6-a990-4b70-a0a4-873956e942b0" (UID: "6e0718b6-a990-4b70-a0a4-873956e942b0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:40:24.815298 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:24.814970 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e0718b6-a990-4b70-a0a4-873956e942b0-maas-api-xks-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:40:24.815298 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:24.815009 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfmsf\" (UniqueName: \"kubernetes.io/projected/6e0718b6-a990-4b70-a0a4-873956e942b0-kube-api-access-bfmsf\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:40:25.182988 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:25.182902 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2c966aa-vjmw5" Nov 03 01:40:25.182988 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:25.182939 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2c966aa-vjmw5" event={"ID":"6e0718b6-a990-4b70-a0a4-873956e942b0","Type":"ContainerDied","Data":"fd20f85d3f964ef6067ff66cdfaf8f9b054a9657d9a2e644b3b71cffb7df3afb"} Nov 03 01:40:25.183191 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:25.182990 4421 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fd20f85d3f964ef6067ff66cdfaf8f9b054a9657d9a2e644b3b71cffb7df3afb" Nov 03 01:40:25.329299 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:25.328809 4421 patch_prober.go:29] interesting pod/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-bx28n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.119:15021/healthz/ready\": dial tcp 10.134.1.119:15021: connect: connection refused" start-of-body= Nov 03 01:40:25.329299 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:25.328882 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-bx28n" podUID="7b44153f-12a2-4b0b-898a-493b372fb7ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.119:15021/healthz/ready\": dial tcp 10.134.1.119:15021: connect: connection refused" Nov 03 01:40:25.455783 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:25.455690 4421 patch_prober.go:29] interesting pod/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-fqqx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.121:15021/healthz/ready\": dial tcp 10.134.1.121:15021: connect: connection refused" start-of-body= Nov 03 01:40:25.456242 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:25.456107 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-fqqx7" podUID="392e8058-14f9-4dfb-b20e-3e209eb3c625" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.121:15021/healthz/ready\": dial tcp 10.134.1.121:15021: connect: connection refused" Nov 03 01:40:25.566303 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:25.561381 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ff331d9-tdxww"] Nov 03 01:40:25.566303 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:25.562509 4421 cpu_manager.go:414] "containerMap: removing container" podUID="6e0718b6-a990-4b70-a0a4-873956e942b0" containerName="revoke-keys" Nov 03 01:40:25.566303 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:25.562533 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e0718b6-a990-4b70-a0a4-873956e942b0" containerName="revoke-keys" Nov 03 01:40:25.566303 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:25.562762 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e0718b6-a990-4b70-a0a4-873956e942b0" containerName="revoke-keys" Nov 03 01:40:25.569952 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:25.569924 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ff331d9-tdxww"] Nov 03 01:40:25.570233 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:25.570218 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ff331d9-tdxww" Nov 03 01:40:25.574506 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:25.574469 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:40:25.574624 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:25.574609 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6q4xp\"" Nov 03 01:40:25.668378 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:25.668337 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qn2gp\" (UniqueName: \"kubernetes.io/projected/dd3fd1e0-4557-429f-8ba6-da6007ffe32d-kube-api-access-qn2gp\") pod \"maas-api-revoke-sub-9ff331d9-tdxww\" (UID: \"dd3fd1e0-4557-429f-8ba6-da6007ffe32d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ff331d9-tdxww" Nov 03 01:40:25.668875 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:25.668842 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd3fd1e0-4557-429f-8ba6-da6007ffe32d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9ff331d9-tdxww\" (UID: \"dd3fd1e0-4557-429f-8ba6-da6007ffe32d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ff331d9-tdxww" Nov 03 01:40:25.669197 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:25.669174 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd3fd1e0-4557-429f-8ba6-da6007ffe32d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9ff331d9-tdxww\" (UID: \"dd3fd1e0-4557-429f-8ba6-da6007ffe32d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ff331d9-tdxww" Nov 03 01:40:25.777983 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:25.777941 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qn2gp\" (UniqueName: \"kubernetes.io/projected/dd3fd1e0-4557-429f-8ba6-da6007ffe32d-kube-api-access-qn2gp\") pod \"maas-api-revoke-sub-9ff331d9-tdxww\" (UID: \"dd3fd1e0-4557-429f-8ba6-da6007ffe32d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ff331d9-tdxww" Nov 03 01:40:25.778905 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:25.778872 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd3fd1e0-4557-429f-8ba6-da6007ffe32d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9ff331d9-tdxww\" (UID: \"dd3fd1e0-4557-429f-8ba6-da6007ffe32d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ff331d9-tdxww" Nov 03 01:40:25.779821 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:25.779806 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd3fd1e0-4557-429f-8ba6-da6007ffe32d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9ff331d9-tdxww\" (UID: \"dd3fd1e0-4557-429f-8ba6-da6007ffe32d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ff331d9-tdxww" Nov 03 01:40:25.780394 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:25.779679 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd3fd1e0-4557-429f-8ba6-da6007ffe32d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9ff331d9-tdxww\" (UID: \"dd3fd1e0-4557-429f-8ba6-da6007ffe32d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ff331d9-tdxww" Nov 03 01:40:25.782829 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:25.782803 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd3fd1e0-4557-429f-8ba6-da6007ffe32d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9ff331d9-tdxww\" (UID: \"dd3fd1e0-4557-429f-8ba6-da6007ffe32d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ff331d9-tdxww" Nov 03 01:40:25.787007 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:25.786981 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qn2gp\" (UniqueName: \"kubernetes.io/projected/dd3fd1e0-4557-429f-8ba6-da6007ffe32d-kube-api-access-qn2gp\") pod \"maas-api-revoke-sub-9ff331d9-tdxww\" (UID: \"dd3fd1e0-4557-429f-8ba6-da6007ffe32d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ff331d9-tdxww" Nov 03 01:40:25.889555 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:25.889513 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ff331d9-tdxww" Nov 03 01:40:26.366615 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:26.362359 4421 patch_prober.go:29] interesting pod/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-bx28n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.119:15021/healthz/ready\": dial tcp 10.134.1.119:15021: connect: connection refused" start-of-body= Nov 03 01:40:26.366615 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:26.362447 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-bx28n" podUID="7b44153f-12a2-4b0b-898a-493b372fb7ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.119:15021/healthz/ready\": dial tcp 10.134.1.119:15021: connect: connection refused" Nov 03 01:40:26.368421 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:26.368392 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ff331d9-tdxww"] Nov 03 01:40:26.456405 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:26.455898 4421 patch_prober.go:29] interesting pod/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-fqqx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.121:15021/healthz/ready\": dial tcp 10.134.1.121:15021: connect: connection refused" start-of-body= Nov 03 01:40:26.456405 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:26.455962 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-fqqx7" podUID="392e8058-14f9-4dfb-b20e-3e209eb3c625" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.121:15021/healthz/ready\": dial tcp 10.134.1.121:15021: connect: connection refused" Nov 03 01:40:27.203721 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:27.203626 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ff331d9-tdxww" event={"ID":"dd3fd1e0-4557-429f-8ba6-da6007ffe32d","Type":"ContainerStarted","Data":"5dd50e8fb524cd50b6d72695142bca9d8de5afcdadf6381351ca5d7bb0f29323"} Nov 03 01:40:27.203721 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:27.203690 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ff331d9-tdxww" event={"ID":"dd3fd1e0-4557-429f-8ba6-da6007ffe32d","Type":"ContainerStarted","Data":"e853bc09106812293fbdb0b8ae1b24bccea4dcb9a546f5374974244e1382696c"} Nov 03 01:40:27.237188 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:27.237128 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ff331d9-tdxww" podStartSLOduration=2.23711174 podStartE2EDuration="2.23711174s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:40:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:40:27.233038523 +0000 UTC m=+3569.404041625" watchObservedRunningTime="2026-11-03 01:40:27.23711174 +0000 UTC m=+3569.408114863" Nov 03 01:40:27.326298 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:27.325646 4421 patch_prober.go:29] interesting pod/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-bx28n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.119:15021/healthz/ready\": dial tcp 10.134.1.119:15021: connect: connection refused" start-of-body= Nov 03 01:40:27.326298 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:27.325710 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-bx28n" podUID="7b44153f-12a2-4b0b-898a-493b372fb7ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.119:15021/healthz/ready\": dial tcp 10.134.1.119:15021: connect: connection refused" Nov 03 01:40:27.452988 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:27.452943 4421 patch_prober.go:29] interesting pod/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-fqqx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.121:15021/healthz/ready\": dial tcp 10.134.1.121:15021: connect: connection refused" start-of-body= Nov 03 01:40:27.453177 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:27.453025 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-fqqx7" podUID="392e8058-14f9-4dfb-b20e-3e209eb3c625" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.121:15021/healthz/ready\": dial tcp 10.134.1.121:15021: connect: connection refused" Nov 03 01:40:28.218818 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:28.218770 4421 generic.go:356] "Generic (PLEG): container finished" podID="dd3fd1e0-4557-429f-8ba6-da6007ffe32d" containerID="5dd50e8fb524cd50b6d72695142bca9d8de5afcdadf6381351ca5d7bb0f29323" exitCode=0 Nov 03 01:40:28.223582 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:28.220212 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ff331d9-tdxww" event={"ID":"dd3fd1e0-4557-429f-8ba6-da6007ffe32d","Type":"ContainerDied","Data":"5dd50e8fb524cd50b6d72695142bca9d8de5afcdadf6381351ca5d7bb0f29323"} Nov 03 01:40:28.335260 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:28.335234 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-bx28n" Nov 03 01:40:28.335920 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:28.335880 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-bx28n" Nov 03 01:40:28.336673 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:28.336651 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-bx28n" Nov 03 01:40:28.453597 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:28.453564 4421 patch_prober.go:29] interesting pod/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-fqqx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.121:15021/healthz/ready\": dial tcp 10.134.1.121:15021: connect: connection refused" start-of-body= Nov 03 01:40:28.453772 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:28.453618 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-fqqx7" podUID="392e8058-14f9-4dfb-b20e-3e209eb3c625" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.121:15021/healthz/ready\": dial tcp 10.134.1.121:15021: connect: connection refused" Nov 03 01:40:29.034786 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:29.034743 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a78aa3c2-hk7dm"] Nov 03 01:40:29.039663 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:29.039633 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a78aa3c2-hk7dm"] Nov 03 01:40:29.375570 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:29.375396 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-531447-667d774bfc-9shwq" podUID="9f997097-9fbb-4cf2-b370-f48154901810" containerName="payload-processing" containerID="cri-o://338e3b01c784321a516c487e5b6c5637fb1b9a172718345ee311f93a221ef9ec" gracePeriod=25 Nov 03 01:40:29.382189 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:29.381573 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-531447-7cf7bc4dcf-xt4bm" podUID="8a3e520f-1e5b-4ae4-b263-c99521d89d9e" containerName="payload-pre-processing" containerID="cri-o://b74c3fdc92d31740b8e8a002c49dba79a27ab542c30b0e2649687605e7824252" gracePeriod=25 Nov 03 01:40:29.453511 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:29.453419 4421 patch_prober.go:29] interesting pod/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-fqqx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.121:15021/healthz/ready\": dial tcp 10.134.1.121:15021: connect: connection refused" start-of-body= Nov 03 01:40:29.453511 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:29.453476 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-fqqx7" podUID="392e8058-14f9-4dfb-b20e-3e209eb3c625" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.121:15021/healthz/ready\": dial tcp 10.134.1.121:15021: connect: connection refused" Nov 03 01:40:29.490425 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:29.490399 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ff331d9-tdxww" Nov 03 01:40:29.537881 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:29.537836 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dd3fd1e0-4557-429f-8ba6-da6007ffe32d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd3fd1e0-4557-429f-8ba6-da6007ffe32d-maas-api-xks-ca\") pod \"dd3fd1e0-4557-429f-8ba6-da6007ffe32d\" (UID: \"dd3fd1e0-4557-429f-8ba6-da6007ffe32d\") " Nov 03 01:40:29.538106 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:29.538068 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd3fd1e0-4557-429f-8ba6-da6007ffe32d-kube-api-access-qn2gp\" (UniqueName: \"kubernetes.io/projected/dd3fd1e0-4557-429f-8ba6-da6007ffe32d-kube-api-access-qn2gp\") pod \"dd3fd1e0-4557-429f-8ba6-da6007ffe32d\" (UID: \"dd3fd1e0-4557-429f-8ba6-da6007ffe32d\") " Nov 03 01:40:29.538183 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:29.538138 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd3fd1e0-4557-429f-8ba6-da6007ffe32d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd3fd1e0-4557-429f-8ba6-da6007ffe32d-maas-api-openshift-ca\") pod \"dd3fd1e0-4557-429f-8ba6-da6007ffe32d\" (UID: \"dd3fd1e0-4557-429f-8ba6-da6007ffe32d\") " Nov 03 01:40:29.539253 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:29.539219 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd3fd1e0-4557-429f-8ba6-da6007ffe32d-maas-api-openshift-ca" pod "dd3fd1e0-4557-429f-8ba6-da6007ffe32d" (UID: "dd3fd1e0-4557-429f-8ba6-da6007ffe32d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:40:29.541499 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:29.541473 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd3fd1e0-4557-429f-8ba6-da6007ffe32d-maas-api-xks-ca" pod "dd3fd1e0-4557-429f-8ba6-da6007ffe32d" (UID: "dd3fd1e0-4557-429f-8ba6-da6007ffe32d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:40:29.541723 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:29.541544 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd3fd1e0-4557-429f-8ba6-da6007ffe32d-kube-api-access-qn2gp" pod "dd3fd1e0-4557-429f-8ba6-da6007ffe32d" (UID: "dd3fd1e0-4557-429f-8ba6-da6007ffe32d"). InnerVolumeSpecName "kube-api-access-qn2gp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:40:29.640889 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:29.640799 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd3fd1e0-4557-429f-8ba6-da6007ffe32d-maas-api-xks-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:40:29.640889 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:29.640830 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qn2gp\" (UniqueName: \"kubernetes.io/projected/dd3fd1e0-4557-429f-8ba6-da6007ffe32d-kube-api-access-qn2gp\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:40:29.640889 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:29.640842 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd3fd1e0-4557-429f-8ba6-da6007ffe32d-maas-api-openshift-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:40:30.011193 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:30.011160 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad30bab7-985e-4174-b75b-26d590a6acfe" path="/var/lib/kubelet/pods/ad30bab7-985e-4174-b75b-26d590a6acfe/volumes" Nov 03 01:40:30.267821 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:30.267729 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ff331d9-tdxww" Nov 03 01:40:30.267821 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:30.267756 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ff331d9-tdxww" event={"ID":"dd3fd1e0-4557-429f-8ba6-da6007ffe32d","Type":"ContainerDied","Data":"e853bc09106812293fbdb0b8ae1b24bccea4dcb9a546f5374974244e1382696c"} Nov 03 01:40:30.267821 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:30.267813 4421 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e853bc09106812293fbdb0b8ae1b24bccea4dcb9a546f5374974244e1382696c" Nov 03 01:40:30.458324 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:30.458292 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-fqqx7" Nov 03 01:40:30.458754 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:30.458737 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-fqqx7" Nov 03 01:40:30.459708 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:30.459686 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-fqqx7" Nov 03 01:40:30.830783 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:30.830740 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e23907d5-fdfjd"] Nov 03 01:40:30.831830 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:30.831808 4421 cpu_manager.go:414] "containerMap: removing container" podUID="dd3fd1e0-4557-429f-8ba6-da6007ffe32d" containerName="revoke-keys" Nov 03 01:40:30.831939 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:30.831830 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd3fd1e0-4557-429f-8ba6-da6007ffe32d" containerName="revoke-keys" Nov 03 01:40:30.832064 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:30.832052 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd3fd1e0-4557-429f-8ba6-da6007ffe32d" containerName="revoke-keys" Nov 03 01:40:30.836880 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:30.836853 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e23907d5-fdfjd" Nov 03 01:40:30.840456 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:30.840420 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:40:30.840456 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:30.840433 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6q4xp\"" Nov 03 01:40:30.841158 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:30.841130 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e23907d5-fdfjd"] Nov 03 01:40:30.969411 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:30.969365 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tql7f\" (UniqueName: \"kubernetes.io/projected/e2b6d01d-9fcb-4164-a217-2a596801ce5b-kube-api-access-tql7f\") pod \"maas-api-revoke-sub-e23907d5-fdfjd\" (UID: \"e2b6d01d-9fcb-4164-a217-2a596801ce5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e23907d5-fdfjd" Nov 03 01:40:30.969611 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:30.969586 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2b6d01d-9fcb-4164-a217-2a596801ce5b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e23907d5-fdfjd\" (UID: \"e2b6d01d-9fcb-4164-a217-2a596801ce5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e23907d5-fdfjd" Nov 03 01:40:30.969710 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:30.969645 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2b6d01d-9fcb-4164-a217-2a596801ce5b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e23907d5-fdfjd\" (UID: \"e2b6d01d-9fcb-4164-a217-2a596801ce5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e23907d5-fdfjd" Nov 03 01:40:31.071410 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:31.071375 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tql7f\" (UniqueName: \"kubernetes.io/projected/e2b6d01d-9fcb-4164-a217-2a596801ce5b-kube-api-access-tql7f\") pod \"maas-api-revoke-sub-e23907d5-fdfjd\" (UID: \"e2b6d01d-9fcb-4164-a217-2a596801ce5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e23907d5-fdfjd" Nov 03 01:40:31.071634 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:31.071502 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2b6d01d-9fcb-4164-a217-2a596801ce5b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e23907d5-fdfjd\" (UID: \"e2b6d01d-9fcb-4164-a217-2a596801ce5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e23907d5-fdfjd" Nov 03 01:40:31.071634 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:31.071536 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2b6d01d-9fcb-4164-a217-2a596801ce5b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e23907d5-fdfjd\" (UID: \"e2b6d01d-9fcb-4164-a217-2a596801ce5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e23907d5-fdfjd" Nov 03 01:40:31.072342 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:31.072311 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2b6d01d-9fcb-4164-a217-2a596801ce5b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e23907d5-fdfjd\" (UID: \"e2b6d01d-9fcb-4164-a217-2a596801ce5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e23907d5-fdfjd" Nov 03 01:40:31.074266 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:31.074239 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2b6d01d-9fcb-4164-a217-2a596801ce5b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e23907d5-fdfjd\" (UID: \"e2b6d01d-9fcb-4164-a217-2a596801ce5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e23907d5-fdfjd" Nov 03 01:40:31.079038 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:31.079016 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tql7f\" (UniqueName: \"kubernetes.io/projected/e2b6d01d-9fcb-4164-a217-2a596801ce5b-kube-api-access-tql7f\") pod \"maas-api-revoke-sub-e23907d5-fdfjd\" (UID: \"e2b6d01d-9fcb-4164-a217-2a596801ce5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e23907d5-fdfjd" Nov 03 01:40:31.154468 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:31.154374 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e23907d5-fdfjd" Nov 03 01:40:31.471336 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:31.471301 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e23907d5-fdfjd"] Nov 03 01:40:31.471845 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:40:31.471819 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2b6d01d_9fcb_4164_a217_2a596801ce5b.slice/crio-16001daed466fef4c4cede2d3f4a88d6a4c86206fe17da458cc85e27a93cd927 WatchSource:0}: Error finding container 16001daed466fef4c4cede2d3f4a88d6a4c86206fe17da458cc85e27a93cd927: Status 404 returned error can't find the container with id 16001daed466fef4c4cede2d3f4a88d6a4c86206fe17da458cc85e27a93cd927 Nov 03 01:40:32.290566 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:32.290530 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e23907d5-fdfjd" event={"ID":"e2b6d01d-9fcb-4164-a217-2a596801ce5b","Type":"ContainerStarted","Data":"5634b34e211f4611085fbbb1e56530b401cb4df39db4c80d7d6025f140fdf7bd"} Nov 03 01:40:32.290761 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:32.290583 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e23907d5-fdfjd" event={"ID":"e2b6d01d-9fcb-4164-a217-2a596801ce5b","Type":"ContainerStarted","Data":"16001daed466fef4c4cede2d3f4a88d6a4c86206fe17da458cc85e27a93cd927"} Nov 03 01:40:32.309269 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:32.309207 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e23907d5-fdfjd" podStartSLOduration=2.309192009 podStartE2EDuration="2.309192009s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:40:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:40:32.306517806 +0000 UTC m=+3574.477520902" watchObservedRunningTime="2026-11-03 01:40:32.309192009 +0000 UTC m=+3574.480195106" Nov 03 01:40:33.300738 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:33.300704 4421 generic.go:356] "Generic (PLEG): container finished" podID="e2b6d01d-9fcb-4164-a217-2a596801ce5b" containerID="5634b34e211f4611085fbbb1e56530b401cb4df39db4c80d7d6025f140fdf7bd" exitCode=0 Nov 03 01:40:33.301129 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:33.300751 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e23907d5-fdfjd" event={"ID":"e2b6d01d-9fcb-4164-a217-2a596801ce5b","Type":"ContainerDied","Data":"5634b34e211f4611085fbbb1e56530b401cb4df39db4c80d7d6025f140fdf7bd"} Nov 03 01:40:34.513314 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:34.513269 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e23907d5-fdfjd" Nov 03 01:40:34.658496 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:34.658386 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2b6d01d-9fcb-4164-a217-2a596801ce5b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2b6d01d-9fcb-4164-a217-2a596801ce5b-maas-api-openshift-ca\") pod \"e2b6d01d-9fcb-4164-a217-2a596801ce5b\" (UID: \"e2b6d01d-9fcb-4164-a217-2a596801ce5b\") " Nov 03 01:40:34.658687 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:34.658539 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e2b6d01d-9fcb-4164-a217-2a596801ce5b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2b6d01d-9fcb-4164-a217-2a596801ce5b-maas-api-xks-ca\") pod \"e2b6d01d-9fcb-4164-a217-2a596801ce5b\" (UID: \"e2b6d01d-9fcb-4164-a217-2a596801ce5b\") " Nov 03 01:40:34.658687 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:34.658610 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2b6d01d-9fcb-4164-a217-2a596801ce5b-kube-api-access-tql7f\" (UniqueName: \"kubernetes.io/projected/e2b6d01d-9fcb-4164-a217-2a596801ce5b-kube-api-access-tql7f\") pod \"e2b6d01d-9fcb-4164-a217-2a596801ce5b\" (UID: \"e2b6d01d-9fcb-4164-a217-2a596801ce5b\") " Nov 03 01:40:34.658916 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:34.658829 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2b6d01d-9fcb-4164-a217-2a596801ce5b-maas-api-openshift-ca" pod "e2b6d01d-9fcb-4164-a217-2a596801ce5b" (UID: "e2b6d01d-9fcb-4164-a217-2a596801ce5b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:40:34.659386 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:34.659360 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2b6d01d-9fcb-4164-a217-2a596801ce5b-maas-api-openshift-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:40:34.661522 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:34.661491 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2b6d01d-9fcb-4164-a217-2a596801ce5b-maas-api-xks-ca" pod "e2b6d01d-9fcb-4164-a217-2a596801ce5b" (UID: "e2b6d01d-9fcb-4164-a217-2a596801ce5b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:40:34.662445 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:34.662416 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2b6d01d-9fcb-4164-a217-2a596801ce5b-kube-api-access-tql7f" pod "e2b6d01d-9fcb-4164-a217-2a596801ce5b" (UID: "e2b6d01d-9fcb-4164-a217-2a596801ce5b"). InnerVolumeSpecName "kube-api-access-tql7f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:40:34.761351 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:34.761320 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2b6d01d-9fcb-4164-a217-2a596801ce5b-maas-api-xks-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:40:34.761351 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:34.761347 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tql7f\" (UniqueName: \"kubernetes.io/projected/e2b6d01d-9fcb-4164-a217-2a596801ce5b-kube-api-access-tql7f\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:40:35.352814 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:35.352763 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e23907d5-fdfjd" event={"ID":"e2b6d01d-9fcb-4164-a217-2a596801ce5b","Type":"ContainerDied","Data":"16001daed466fef4c4cede2d3f4a88d6a4c86206fe17da458cc85e27a93cd927"} Nov 03 01:40:35.353034 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:35.352830 4421 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="16001daed466fef4c4cede2d3f4a88d6a4c86206fe17da458cc85e27a93cd927" Nov 03 01:40:35.353034 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:35.352898 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e23907d5-fdfjd" Nov 03 01:40:36.066008 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:36.065970 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-6c0ec0/sub-test-model-6c0ec0-kserve-666b858575-xpltp"] Nov 03 01:40:37.599094 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:37.599052 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-531447-7cf7bc4dcf-xt4bm" podUID="8a3e520f-1e5b-4ae4-b263-c99521d89d9e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:40:38.607992 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:38.607952 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-531447-667d774bfc-9shwq" podUID="9f997097-9fbb-4cf2-b370-f48154901810" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:40:38.710668 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:38.709400 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-5ws4g"] Nov 03 01:40:38.710668 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:38.709716 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-5ws4g" podUID="05c74cad-b107-403b-bffc-af74dea66076" containerName="istio-proxy" containerID="cri-o://35d21581f8d905ac294fb78104bcd18118fbf08cebe90ca75f0d9833b8c127e5" gracePeriod=30 Nov 03 01:40:38.721330 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:38.721082 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-c9sjv"] Nov 03 01:40:38.721529 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:38.721372 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-c9sjv" podUID="73bae713-0108-4120-8486-a35dbf68b0c9" containerName="istio-proxy" containerID="cri-o://a5f6e226d649f869467b1020f02969e268556d71065246f2002bf534df9c178a" gracePeriod=30 Nov 03 01:40:38.728231 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:38.728203 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-tlggp"] Nov 03 01:40:38.728518 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:38.728469 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-tlggp" podUID="8e32924b-f5e1-4ecc-8a7d-08647aebf8fd" containerName="istio-proxy" containerID="cri-o://1c547b60d1d0b1ad263ddb29c394402893bbbeb36320a99b4367ca055962796b" gracePeriod=30 Nov 03 01:40:38.733493 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:38.733465 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-jcsn2"] Nov 03 01:40:38.733692 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:38.733667 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-jcsn2" podUID="2233c8fb-b2bc-4987-82de-c06a99a7d396" containerName="istio-proxy" containerID="cri-o://0fdac1b8e75bb68607bd72e909890c6790f7ac9fe81871b6e7c0931a585e1c34" gracePeriod=30 Nov 03 01:40:39.256659 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:39.256623 4421 patch_prober.go:29] interesting pod/e2e-shared-a-w2-531447-openshift-default-7995d46767-jcsn2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 01:40:39.256822 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:39.256687 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-jcsn2" podUID="2233c8fb-b2bc-4987-82de-c06a99a7d396" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 01:40:39.374476 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:39.374441 4421 patch_prober.go:29] interesting pod/e2e-shared-a-w2-531447-openshift-default-7995d46767-tlggp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 01:40:39.374642 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:39.374504 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-tlggp" podUID="8e32924b-f5e1-4ecc-8a7d-08647aebf8fd" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 01:40:40.201517 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:40.201478 4421 patch_prober.go:29] interesting pod/e2e-shared-a-w2-531447-openshift-default-7995d46767-c9sjv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 01:40:40.202027 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:40.201543 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-c9sjv" podUID="73bae713-0108-4120-8486-a35dbf68b0c9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 01:40:41.172841 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:41.172795 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-4a697f/sub-test-model-4a697f-kserve-588b784c4c-j5qfn"] Nov 03 01:40:43.159516 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:43.159472 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-717217/rate-test-model-717217-kserve-74ddd7bbcc-4kxbf"] Nov 03 01:40:44.086040 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:44.086004 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-5ws4g" Nov 03 01:40:44.153676 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:44.153640 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-jcsn2" Nov 03 01:40:44.158787 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:44.158757 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-c9sjv" Nov 03 01:40:44.163861 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:44.163831 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-tlggp" Nov 03 01:40:44.174600 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:44.173946 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/05c74cad-b107-403b-bffc-af74dea66076-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05c74cad-b107-403b-bffc-af74dea66076-istio-podinfo\") pod \"05c74cad-b107-403b-bffc-af74dea66076\" (UID: \"05c74cad-b107-403b-bffc-af74dea66076\") " Nov 03 01:40:44.174600 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:44.174035 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05c74cad-b107-403b-bffc-af74dea66076-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05c74cad-b107-403b-bffc-af74dea66076-istio-data\") pod \"05c74cad-b107-403b-bffc-af74dea66076\" (UID: \"05c74cad-b107-403b-bffc-af74dea66076\") " Nov 03 01:40:44.174600 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:44.174071 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05c74cad-b107-403b-bffc-af74dea66076-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05c74cad-b107-403b-bffc-af74dea66076-credential-socket\") pod \"05c74cad-b107-403b-bffc-af74dea66076\" (UID: \"05c74cad-b107-403b-bffc-af74dea66076\") " Nov 03 01:40:44.174600 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:44.174107 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05c74cad-b107-403b-bffc-af74dea66076-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05c74cad-b107-403b-bffc-af74dea66076-workload-certs\") pod \"05c74cad-b107-403b-bffc-af74dea66076\" (UID: \"05c74cad-b107-403b-bffc-af74dea66076\") " Nov 03 01:40:44.174600 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:44.174232 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05c74cad-b107-403b-bffc-af74dea66076-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05c74cad-b107-403b-bffc-af74dea66076-workload-socket\") pod \"05c74cad-b107-403b-bffc-af74dea66076\" (UID: \"05c74cad-b107-403b-bffc-af74dea66076\") " Nov 03 01:40:44.174600 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:44.174255 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05c74cad-b107-403b-bffc-af74dea66076-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05c74cad-b107-403b-bffc-af74dea66076-istio-envoy\") pod \"05c74cad-b107-403b-bffc-af74dea66076\" (UID: \"05c74cad-b107-403b-bffc-af74dea66076\") " Nov 03 01:40:44.174600 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:44.174300 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05c74cad-b107-403b-bffc-af74dea66076-kube-api-access-pmlpq\" (UniqueName: \"kubernetes.io/projected/05c74cad-b107-403b-bffc-af74dea66076-kube-api-access-pmlpq\") pod \"05c74cad-b107-403b-bffc-af74dea66076\" (UID: \"05c74cad-b107-403b-bffc-af74dea66076\") " Nov 03 01:40:44.174600 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:44.174328 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05c74cad-b107-403b-bffc-af74dea66076-istio-token\" (UniqueName: \"kubernetes.io/projected/05c74cad-b107-403b-bffc-af74dea66076-istio-token\") pod \"05c74cad-b107-403b-bffc-af74dea66076\" (UID: \"05c74cad-b107-403b-bffc-af74dea66076\") " Nov 03 01:40:44.174600 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:44.174363 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05c74cad-b107-403b-bffc-af74dea66076-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05c74cad-b107-403b-bffc-af74dea66076-istiod-ca-cert\") pod \"05c74cad-b107-403b-bffc-af74dea66076\" (UID: \"05c74cad-b107-403b-bffc-af74dea66076\") " Nov 03 01:40:44.175153 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:44.174928 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05c74cad-b107-403b-bffc-af74dea66076-istio-data" pod "05c74cad-b107-403b-bffc-af74dea66076" (UID: "05c74cad-b107-403b-bffc-af74dea66076"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:40:44.175153 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:44.175113 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05c74cad-b107-403b-bffc-af74dea66076-workload-socket" pod "05c74cad-b107-403b-bffc-af74dea66076" (UID: "05c74cad-b107-403b-bffc-af74dea66076"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:40:44.175303 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:44.175214 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05c74cad-b107-403b-bffc-af74dea66076-istiod-ca-cert" pod "05c74cad-b107-403b-bffc-af74dea66076" (UID: "05c74cad-b107-403b-bffc-af74dea66076"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:40:44.175381 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:44.175365 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05c74cad-b107-403b-bffc-af74dea66076-credential-socket" pod "05c74cad-b107-403b-bffc-af74dea66076" (UID: "05c74cad-b107-403b-bffc-af74dea66076"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:40:44.175474 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:44.175450 4421 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05c74cad-b107-403b-bffc-af74dea66076-istio-data\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:40:44.175542 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:44.175482 4421 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05c74cad-b107-403b-bffc-af74dea66076-workload-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:40:44.175647 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:44.175625 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05c74cad-b107-403b-bffc-af74dea66076-workload-certs" pod "05c74cad-b107-403b-bffc-af74dea66076" (UID: "05c74cad-b107-403b-bffc-af74dea66076"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:40:44.177949 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:44.177919 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05c74cad-b107-403b-bffc-af74dea66076-istio-envoy" pod "05c74cad-b107-403b-bffc-af74dea66076" (UID: "05c74cad-b107-403b-bffc-af74dea66076"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:40:44.178246 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:44.178216 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05c74cad-b107-403b-bffc-af74dea66076-istio-token" pod "05c74cad-b107-403b-bffc-af74dea66076" (UID: "05c74cad-b107-403b-bffc-af74dea66076"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:40:44.178512 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:44.178491 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/05c74cad-b107-403b-bffc-af74dea66076-istio-podinfo" pod "05c74cad-b107-403b-bffc-af74dea66076" (UID: "05c74cad-b107-403b-bffc-af74dea66076"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:40:44.178643 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:44.178622 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05c74cad-b107-403b-bffc-af74dea66076-kube-api-access-pmlpq" pod "05c74cad-b107-403b-bffc-af74dea66076" (UID: "05c74cad-b107-403b-bffc-af74dea66076"). InnerVolumeSpecName "kube-api-access-pmlpq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:40:44.276673 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:44.276636 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73bae713-0108-4120-8486-a35dbf68b0c9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73bae713-0108-4120-8486-a35dbf68b0c9-workload-socket\") pod \"73bae713-0108-4120-8486-a35dbf68b0c9\" (UID: \"73bae713-0108-4120-8486-a35dbf68b0c9\") " Nov 03 01:40:44.276673 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:44.276674 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2233c8fb-b2bc-4987-82de-c06a99a7d396-istio-token\" (UniqueName: \"kubernetes.io/projected/2233c8fb-b2bc-4987-82de-c06a99a7d396-istio-token\") pod \"2233c8fb-b2bc-4987-82de-c06a99a7d396\" (UID: \"2233c8fb-b2bc-4987-82de-c06a99a7d396\") " Nov 03 01:40:44.276936 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:44.276710 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73bae713-0108-4120-8486-a35dbf68b0c9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73bae713-0108-4120-8486-a35dbf68b0c9-istio-envoy\") pod \"73bae713-0108-4120-8486-a35dbf68b0c9\" (UID: \"73bae713-0108-4120-8486-a35dbf68b0c9\") " Nov 03 01:40:44.276936 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:44.276735 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8e32924b-f5e1-4ecc-8a7d-08647aebf8fd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e32924b-f5e1-4ecc-8a7d-08647aebf8fd-istio-podinfo\") pod \"8e32924b-f5e1-4ecc-8a7d-08647aebf8fd\" (UID: \"8e32924b-f5e1-4ecc-8a7d-08647aebf8fd\") " Nov 03 01:40:44.276936 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:44.276760 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e32924b-f5e1-4ecc-8a7d-08647aebf8fd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e32924b-f5e1-4ecc-8a7d-08647aebf8fd-istiod-ca-cert\") pod \"8e32924b-f5e1-4ecc-8a7d-08647aebf8fd\" (UID: \"8e32924b-f5e1-4ecc-8a7d-08647aebf8fd\") " Nov 03 01:40:44.276936 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:44.276788 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e32924b-f5e1-4ecc-8a7d-08647aebf8fd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e32924b-f5e1-4ecc-8a7d-08647aebf8fd-istio-envoy\") pod \"8e32924b-f5e1-4ecc-8a7d-08647aebf8fd\" (UID: \"8e32924b-f5e1-4ecc-8a7d-08647aebf8fd\") " Nov 03 01:40:44.276936 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:44.276865 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2233c8fb-b2bc-4987-82de-c06a99a7d396-kube-api-access-g2x8c\" (UniqueName: \"kubernetes.io/projected/2233c8fb-b2bc-4987-82de-c06a99a7d396-kube-api-access-g2x8c\") pod \"2233c8fb-b2bc-4987-82de-c06a99a7d396\" (UID: \"2233c8fb-b2bc-4987-82de-c06a99a7d396\") " Nov 03 01:40:44.276936 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:44.276905 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73bae713-0108-4120-8486-a35dbf68b0c9-kube-api-access-pzbp8\" (UniqueName: \"kubernetes.io/projected/73bae713-0108-4120-8486-a35dbf68b0c9-kube-api-access-pzbp8\") pod \"73bae713-0108-4120-8486-a35dbf68b0c9\" (UID: \"73bae713-0108-4120-8486-a35dbf68b0c9\") " Nov 03 01:40:44.276936 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:44.276933 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2233c8fb-b2bc-4987-82de-c06a99a7d396-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2233c8fb-b2bc-4987-82de-c06a99a7d396-istio-podinfo\") pod \"2233c8fb-b2bc-4987-82de-c06a99a7d396\" (UID: \"2233c8fb-b2bc-4987-82de-c06a99a7d396\") " Nov 03 01:40:44.277379 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:44.276969 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e32924b-f5e1-4ecc-8a7d-08647aebf8fd-kube-api-access-tghxw\" (UniqueName: \"kubernetes.io/projected/8e32924b-f5e1-4ecc-8a7d-08647aebf8fd-kube-api-access-tghxw\") pod \"8e32924b-f5e1-4ecc-8a7d-08647aebf8fd\" (UID: \"8e32924b-f5e1-4ecc-8a7d-08647aebf8fd\") " Nov 03 01:40:44.277379 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:44.276978 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73bae713-0108-4120-8486-a35dbf68b0c9-workload-socket" pod "73bae713-0108-4120-8486-a35dbf68b0c9" (UID: "73bae713-0108-4120-8486-a35dbf68b0c9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:40:44.277379 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:44.277022 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73bae713-0108-4120-8486-a35dbf68b0c9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73bae713-0108-4120-8486-a35dbf68b0c9-istiod-ca-cert\") pod \"73bae713-0108-4120-8486-a35dbf68b0c9\" (UID: \"73bae713-0108-4120-8486-a35dbf68b0c9\") " Nov 03 01:40:44.277379 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:44.277095 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e32924b-f5e1-4ecc-8a7d-08647aebf8fd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e32924b-f5e1-4ecc-8a7d-08647aebf8fd-workload-certs\") pod \"8e32924b-f5e1-4ecc-8a7d-08647aebf8fd\" (UID: \"8e32924b-f5e1-4ecc-8a7d-08647aebf8fd\") " Nov 03 01:40:44.277379 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:44.277153 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2233c8fb-b2bc-4987-82de-c06a99a7d396-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2233c8fb-b2bc-4987-82de-c06a99a7d396-credential-socket\") pod \"2233c8fb-b2bc-4987-82de-c06a99a7d396\" (UID: \"2233c8fb-b2bc-4987-82de-c06a99a7d396\") " Nov 03 01:40:44.277379 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:44.277202 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73bae713-0108-4120-8486-a35dbf68b0c9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73bae713-0108-4120-8486-a35dbf68b0c9-workload-certs\") pod \"73bae713-0108-4120-8486-a35dbf68b0c9\" (UID: \"73bae713-0108-4120-8486-a35dbf68b0c9\") " Nov 03 01:40:44.277379 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:44.277264 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e32924b-f5e1-4ecc-8a7d-08647aebf8fd-istiod-ca-cert" pod "8e32924b-f5e1-4ecc-8a7d-08647aebf8fd" (UID: "8e32924b-f5e1-4ecc-8a7d-08647aebf8fd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:40:44.277379 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:44.277364 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2233c8fb-b2bc-4987-82de-c06a99a7d396-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2233c8fb-b2bc-4987-82de-c06a99a7d396-istio-data\") pod \"2233c8fb-b2bc-4987-82de-c06a99a7d396\" (UID: \"2233c8fb-b2bc-4987-82de-c06a99a7d396\") " Nov 03 01:40:44.277853 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:44.277399 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73bae713-0108-4120-8486-a35dbf68b0c9-istiod-ca-cert" pod "73bae713-0108-4120-8486-a35dbf68b0c9" (UID: "73bae713-0108-4120-8486-a35dbf68b0c9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:40:44.277853 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:44.277410 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e32924b-f5e1-4ecc-8a7d-08647aebf8fd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e32924b-f5e1-4ecc-8a7d-08647aebf8fd-istio-data\") pod \"8e32924b-f5e1-4ecc-8a7d-08647aebf8fd\" (UID: \"8e32924b-f5e1-4ecc-8a7d-08647aebf8fd\") " Nov 03 01:40:44.277853 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:44.277455 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2233c8fb-b2bc-4987-82de-c06a99a7d396-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2233c8fb-b2bc-4987-82de-c06a99a7d396-workload-certs\") pod \"2233c8fb-b2bc-4987-82de-c06a99a7d396\" (UID: \"2233c8fb-b2bc-4987-82de-c06a99a7d396\") " Nov 03 01:40:44.277853 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:44.277490 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e32924b-f5e1-4ecc-8a7d-08647aebf8fd-workload-certs" pod "8e32924b-f5e1-4ecc-8a7d-08647aebf8fd" (UID: "8e32924b-f5e1-4ecc-8a7d-08647aebf8fd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:40:44.277853 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:44.277516 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2233c8fb-b2bc-4987-82de-c06a99a7d396-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2233c8fb-b2bc-4987-82de-c06a99a7d396-workload-socket\") pod \"2233c8fb-b2bc-4987-82de-c06a99a7d396\" (UID: \"2233c8fb-b2bc-4987-82de-c06a99a7d396\") " Nov 03 01:40:44.277853 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:44.277552 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/73bae713-0108-4120-8486-a35dbf68b0c9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73bae713-0108-4120-8486-a35dbf68b0c9-istio-podinfo\") pod \"73bae713-0108-4120-8486-a35dbf68b0c9\" (UID: \"73bae713-0108-4120-8486-a35dbf68b0c9\") " Nov 03 01:40:44.277853 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:44.277621 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2233c8fb-b2bc-4987-82de-c06a99a7d396-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2233c8fb-b2bc-4987-82de-c06a99a7d396-istiod-ca-cert\") pod \"2233c8fb-b2bc-4987-82de-c06a99a7d396\" (UID: \"2233c8fb-b2bc-4987-82de-c06a99a7d396\") " Nov 03 01:40:44.277853 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:44.277657 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e32924b-f5e1-4ecc-8a7d-08647aebf8fd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e32924b-f5e1-4ecc-8a7d-08647aebf8fd-workload-socket\") pod \"8e32924b-f5e1-4ecc-8a7d-08647aebf8fd\" (UID: \"8e32924b-f5e1-4ecc-8a7d-08647aebf8fd\") " Nov 03 01:40:44.277853 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:44.277720 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e32924b-f5e1-4ecc-8a7d-08647aebf8fd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e32924b-f5e1-4ecc-8a7d-08647aebf8fd-credential-socket\") pod \"8e32924b-f5e1-4ecc-8a7d-08647aebf8fd\" (UID: \"8e32924b-f5e1-4ecc-8a7d-08647aebf8fd\") " Nov 03 01:40:44.277853 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:44.277779 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73bae713-0108-4120-8486-a35dbf68b0c9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73bae713-0108-4120-8486-a35dbf68b0c9-istio-data\") pod \"73bae713-0108-4120-8486-a35dbf68b0c9\" (UID: \"73bae713-0108-4120-8486-a35dbf68b0c9\") " Nov 03 01:40:44.277853 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:44.277822 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73bae713-0108-4120-8486-a35dbf68b0c9-istio-token\" (UniqueName: \"kubernetes.io/projected/73bae713-0108-4120-8486-a35dbf68b0c9-istio-token\") pod \"73bae713-0108-4120-8486-a35dbf68b0c9\" (UID: \"73bae713-0108-4120-8486-a35dbf68b0c9\") " Nov 03 01:40:44.279187 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:44.277877 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e32924b-f5e1-4ecc-8a7d-08647aebf8fd-istio-token\" (UniqueName: \"kubernetes.io/projected/8e32924b-f5e1-4ecc-8a7d-08647aebf8fd-istio-token\") pod \"8e32924b-f5e1-4ecc-8a7d-08647aebf8fd\" (UID: \"8e32924b-f5e1-4ecc-8a7d-08647aebf8fd\") " Nov 03 01:40:44.279187 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:44.277933 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73bae713-0108-4120-8486-a35dbf68b0c9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73bae713-0108-4120-8486-a35dbf68b0c9-credential-socket\") pod \"73bae713-0108-4120-8486-a35dbf68b0c9\" (UID: \"73bae713-0108-4120-8486-a35dbf68b0c9\") " Nov 03 01:40:44.279187 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:44.277974 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2233c8fb-b2bc-4987-82de-c06a99a7d396-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2233c8fb-b2bc-4987-82de-c06a99a7d396-istio-envoy\") pod \"2233c8fb-b2bc-4987-82de-c06a99a7d396\" (UID: \"2233c8fb-b2bc-4987-82de-c06a99a7d396\") " Nov 03 01:40:44.279187 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:44.278991 4421 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73bae713-0108-4120-8486-a35dbf68b0c9-workload-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:40:44.279187 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:44.279018 4421 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e32924b-f5e1-4ecc-8a7d-08647aebf8fd-istiod-ca-cert\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:40:44.279187 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:44.279032 4421 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05c74cad-b107-403b-bffc-af74dea66076-istio-envoy\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:40:44.279187 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:44.279045 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmlpq\" (UniqueName: \"kubernetes.io/projected/05c74cad-b107-403b-bffc-af74dea66076-kube-api-access-pmlpq\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:40:44.279187 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:44.279058 4421 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05c74cad-b107-403b-bffc-af74dea66076-istio-token\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:40:44.279187 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:44.279071 4421 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05c74cad-b107-403b-bffc-af74dea66076-istiod-ca-cert\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:40:44.279187 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:44.279085 4421 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73bae713-0108-4120-8486-a35dbf68b0c9-istiod-ca-cert\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:40:44.279187 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:44.279100 4421 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e32924b-f5e1-4ecc-8a7d-08647aebf8fd-workload-certs\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:40:44.279187 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:44.279115 4421 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05c74cad-b107-403b-bffc-af74dea66076-istio-podinfo\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:40:44.279187 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:44.279129 4421 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05c74cad-b107-403b-bffc-af74dea66076-credential-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:40:44.279187 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:44.279142 4421 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05c74cad-b107-403b-bffc-af74dea66076-workload-certs\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:40:44.280048 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:44.279539 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73bae713-0108-4120-8486-a35dbf68b0c9-istio-envoy" pod "73bae713-0108-4120-8486-a35dbf68b0c9" (UID: "73bae713-0108-4120-8486-a35dbf68b0c9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:40:44.280048 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:44.279560 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8e32924b-f5e1-4ecc-8a7d-08647aebf8fd-istio-podinfo" pod "8e32924b-f5e1-4ecc-8a7d-08647aebf8fd" (UID: "8e32924b-f5e1-4ecc-8a7d-08647aebf8fd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:40:44.280048 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:44.279610 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2233c8fb-b2bc-4987-82de-c06a99a7d396-istio-token" pod "2233c8fb-b2bc-4987-82de-c06a99a7d396" (UID: "2233c8fb-b2bc-4987-82de-c06a99a7d396"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:40:44.280048 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:44.280000 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73bae713-0108-4120-8486-a35dbf68b0c9-istio-data" pod "73bae713-0108-4120-8486-a35dbf68b0c9" (UID: "73bae713-0108-4120-8486-a35dbf68b0c9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:40:44.280326 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:44.280198 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2233c8fb-b2bc-4987-82de-c06a99a7d396-istio-podinfo" pod "2233c8fb-b2bc-4987-82de-c06a99a7d396" (UID: "2233c8fb-b2bc-4987-82de-c06a99a7d396"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:40:44.280431 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:44.280321 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2233c8fb-b2bc-4987-82de-c06a99a7d396-istiod-ca-cert" pod "2233c8fb-b2bc-4987-82de-c06a99a7d396" (UID: "2233c8fb-b2bc-4987-82de-c06a99a7d396"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:40:44.280503 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:44.280429 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e32924b-f5e1-4ecc-8a7d-08647aebf8fd-workload-socket" pod "8e32924b-f5e1-4ecc-8a7d-08647aebf8fd" (UID: "8e32924b-f5e1-4ecc-8a7d-08647aebf8fd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:40:44.280574 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:44.280497 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2233c8fb-b2bc-4987-82de-c06a99a7d396-kube-api-access-g2x8c" pod "2233c8fb-b2bc-4987-82de-c06a99a7d396" (UID: "2233c8fb-b2bc-4987-82de-c06a99a7d396"). InnerVolumeSpecName "kube-api-access-g2x8c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:40:44.280664 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:44.280642 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2233c8fb-b2bc-4987-82de-c06a99a7d396-istio-data" pod "2233c8fb-b2bc-4987-82de-c06a99a7d396" (UID: "2233c8fb-b2bc-4987-82de-c06a99a7d396"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:40:44.280664 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:44.280657 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e32924b-f5e1-4ecc-8a7d-08647aebf8fd-credential-socket" pod "8e32924b-f5e1-4ecc-8a7d-08647aebf8fd" (UID: "8e32924b-f5e1-4ecc-8a7d-08647aebf8fd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:40:44.280873 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:44.280852 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2233c8fb-b2bc-4987-82de-c06a99a7d396-credential-socket" pod "2233c8fb-b2bc-4987-82de-c06a99a7d396" (UID: "2233c8fb-b2bc-4987-82de-c06a99a7d396"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:40:44.280873 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:44.280861 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73bae713-0108-4120-8486-a35dbf68b0c9-workload-certs" pod "73bae713-0108-4120-8486-a35dbf68b0c9" (UID: "73bae713-0108-4120-8486-a35dbf68b0c9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:40:44.281057 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:44.281039 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2233c8fb-b2bc-4987-82de-c06a99a7d396-workload-certs" pod "2233c8fb-b2bc-4987-82de-c06a99a7d396" (UID: "2233c8fb-b2bc-4987-82de-c06a99a7d396"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:40:44.281057 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:44.281042 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73bae713-0108-4120-8486-a35dbf68b0c9-credential-socket" pod "73bae713-0108-4120-8486-a35dbf68b0c9" (UID: "73bae713-0108-4120-8486-a35dbf68b0c9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:40:44.281361 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:44.281221 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2233c8fb-b2bc-4987-82de-c06a99a7d396-workload-socket" pod "2233c8fb-b2bc-4987-82de-c06a99a7d396" (UID: "2233c8fb-b2bc-4987-82de-c06a99a7d396"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:40:44.281361 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:44.281229 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e32924b-f5e1-4ecc-8a7d-08647aebf8fd-istio-data" pod "8e32924b-f5e1-4ecc-8a7d-08647aebf8fd" (UID: "8e32924b-f5e1-4ecc-8a7d-08647aebf8fd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:40:44.281945 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:44.281918 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2233c8fb-b2bc-4987-82de-c06a99a7d396-istio-envoy" pod "2233c8fb-b2bc-4987-82de-c06a99a7d396" (UID: "2233c8fb-b2bc-4987-82de-c06a99a7d396"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:40:44.282109 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:44.281920 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e32924b-f5e1-4ecc-8a7d-08647aebf8fd-istio-envoy" pod "8e32924b-f5e1-4ecc-8a7d-08647aebf8fd" (UID: "8e32924b-f5e1-4ecc-8a7d-08647aebf8fd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:40:44.282109 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:44.282082 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/73bae713-0108-4120-8486-a35dbf68b0c9-istio-podinfo" pod "73bae713-0108-4120-8486-a35dbf68b0c9" (UID: "73bae713-0108-4120-8486-a35dbf68b0c9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:40:44.282186 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:44.282166 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73bae713-0108-4120-8486-a35dbf68b0c9-kube-api-access-pzbp8" pod "73bae713-0108-4120-8486-a35dbf68b0c9" (UID: "73bae713-0108-4120-8486-a35dbf68b0c9"). InnerVolumeSpecName "kube-api-access-pzbp8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:40:44.282470 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:44.282436 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e32924b-f5e1-4ecc-8a7d-08647aebf8fd-kube-api-access-tghxw" pod "8e32924b-f5e1-4ecc-8a7d-08647aebf8fd" (UID: "8e32924b-f5e1-4ecc-8a7d-08647aebf8fd"). InnerVolumeSpecName "kube-api-access-tghxw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:40:44.282565 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:44.282502 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e32924b-f5e1-4ecc-8a7d-08647aebf8fd-istio-token" pod "8e32924b-f5e1-4ecc-8a7d-08647aebf8fd" (UID: "8e32924b-f5e1-4ecc-8a7d-08647aebf8fd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:40:44.283418 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:44.283402 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73bae713-0108-4120-8486-a35dbf68b0c9-istio-token" pod "73bae713-0108-4120-8486-a35dbf68b0c9" (UID: "73bae713-0108-4120-8486-a35dbf68b0c9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:40:44.380409 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:44.380378 4421 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e32924b-f5e1-4ecc-8a7d-08647aebf8fd-istio-envoy\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:40:44.380409 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:44.380409 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2x8c\" (UniqueName: \"kubernetes.io/projected/2233c8fb-b2bc-4987-82de-c06a99a7d396-kube-api-access-g2x8c\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:40:44.380409 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:44.380423 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzbp8\" (UniqueName: \"kubernetes.io/projected/73bae713-0108-4120-8486-a35dbf68b0c9-kube-api-access-pzbp8\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:40:44.380707 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:44.380437 4421 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2233c8fb-b2bc-4987-82de-c06a99a7d396-istio-podinfo\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:40:44.380707 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:44.380462 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tghxw\" (UniqueName: \"kubernetes.io/projected/8e32924b-f5e1-4ecc-8a7d-08647aebf8fd-kube-api-access-tghxw\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:40:44.380707 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:44.380473 4421 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2233c8fb-b2bc-4987-82de-c06a99a7d396-credential-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:40:44.380707 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:44.380485 4421 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73bae713-0108-4120-8486-a35dbf68b0c9-workload-certs\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:40:44.380707 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:44.380497 4421 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2233c8fb-b2bc-4987-82de-c06a99a7d396-istio-data\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:40:44.380707 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:44.380509 4421 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e32924b-f5e1-4ecc-8a7d-08647aebf8fd-istio-data\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:40:44.380707 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:44.380522 4421 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2233c8fb-b2bc-4987-82de-c06a99a7d396-workload-certs\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:40:44.380707 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:44.380534 4421 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2233c8fb-b2bc-4987-82de-c06a99a7d396-workload-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:40:44.380707 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:44.380547 4421 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73bae713-0108-4120-8486-a35dbf68b0c9-istio-podinfo\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:40:44.380707 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:44.380560 4421 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2233c8fb-b2bc-4987-82de-c06a99a7d396-istiod-ca-cert\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:40:44.380707 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:44.380572 4421 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e32924b-f5e1-4ecc-8a7d-08647aebf8fd-workload-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:40:44.380707 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:44.380584 4421 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e32924b-f5e1-4ecc-8a7d-08647aebf8fd-credential-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:40:44.380707 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:44.380596 4421 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73bae713-0108-4120-8486-a35dbf68b0c9-istio-data\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:40:44.380707 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:44.380609 4421 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73bae713-0108-4120-8486-a35dbf68b0c9-istio-token\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:40:44.380707 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:44.380624 4421 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e32924b-f5e1-4ecc-8a7d-08647aebf8fd-istio-token\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:40:44.380707 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:44.380637 4421 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73bae713-0108-4120-8486-a35dbf68b0c9-credential-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:40:44.380707 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:44.380651 4421 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2233c8fb-b2bc-4987-82de-c06a99a7d396-istio-envoy\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:40:44.380707 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:44.380664 4421 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2233c8fb-b2bc-4987-82de-c06a99a7d396-istio-token\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:40:44.380707 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:44.380677 4421 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73bae713-0108-4120-8486-a35dbf68b0c9-istio-envoy\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:40:44.380707 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:44.380691 4421 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e32924b-f5e1-4ecc-8a7d-08647aebf8fd-istio-podinfo\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:40:44.480324 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:44.480288 4421 generic.go:356] "Generic (PLEG): container finished" podID="8e32924b-f5e1-4ecc-8a7d-08647aebf8fd" containerID="1c547b60d1d0b1ad263ddb29c394402893bbbeb36320a99b4367ca055962796b" exitCode=0 Nov 03 01:40:44.480498 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:44.480338 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-tlggp" Nov 03 01:40:44.480498 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:44.480369 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-tlggp" event={"ID":"8e32924b-f5e1-4ecc-8a7d-08647aebf8fd","Type":"ContainerDied","Data":"1c547b60d1d0b1ad263ddb29c394402893bbbeb36320a99b4367ca055962796b"} Nov 03 01:40:44.480498 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:44.480425 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-tlggp" event={"ID":"8e32924b-f5e1-4ecc-8a7d-08647aebf8fd","Type":"ContainerDied","Data":"d64967260ae9b5601b7e241b83941d160d41a777f4c5f13b4bded6bca78f8f04"} Nov 03 01:40:44.480498 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:44.480454 4421 scope.go:122] "RemoveContainer" containerID="1c547b60d1d0b1ad263ddb29c394402893bbbeb36320a99b4367ca055962796b" Nov 03 01:40:44.482214 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:44.482190 4421 generic.go:356] "Generic (PLEG): container finished" podID="73bae713-0108-4120-8486-a35dbf68b0c9" containerID="a5f6e226d649f869467b1020f02969e268556d71065246f2002bf534df9c178a" exitCode=0 Nov 03 01:40:44.482433 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:44.482293 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-c9sjv" event={"ID":"73bae713-0108-4120-8486-a35dbf68b0c9","Type":"ContainerDied","Data":"a5f6e226d649f869467b1020f02969e268556d71065246f2002bf534df9c178a"} Nov 03 01:40:44.482433 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:44.482311 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-c9sjv" Nov 03 01:40:44.482433 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:44.482333 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-c9sjv" event={"ID":"73bae713-0108-4120-8486-a35dbf68b0c9","Type":"ContainerDied","Data":"62027605b2b3ba4b1731586e5f45a6f58042c8bb194468bb2ecaac84df7b95fb"} Nov 03 01:40:44.484901 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:44.484388 4421 generic.go:356] "Generic (PLEG): container finished" podID="05c74cad-b107-403b-bffc-af74dea66076" containerID="35d21581f8d905ac294fb78104bcd18118fbf08cebe90ca75f0d9833b8c127e5" exitCode=0 Nov 03 01:40:44.484901 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:44.484460 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-5ws4g" event={"ID":"05c74cad-b107-403b-bffc-af74dea66076","Type":"ContainerDied","Data":"35d21581f8d905ac294fb78104bcd18118fbf08cebe90ca75f0d9833b8c127e5"} Nov 03 01:40:44.484901 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:44.484493 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-5ws4g" event={"ID":"05c74cad-b107-403b-bffc-af74dea66076","Type":"ContainerDied","Data":"2998d8d4c7d49fafdd2fa0383cd1a109d148edbd9edf8800baf6b2d775ea484d"} Nov 03 01:40:44.484901 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:44.484680 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-5ws4g" Nov 03 01:40:44.487337 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:44.486571 4421 generic.go:356] "Generic (PLEG): container finished" podID="2233c8fb-b2bc-4987-82de-c06a99a7d396" containerID="0fdac1b8e75bb68607bd72e909890c6790f7ac9fe81871b6e7c0931a585e1c34" exitCode=0 Nov 03 01:40:44.487337 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:44.486647 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-jcsn2" event={"ID":"2233c8fb-b2bc-4987-82de-c06a99a7d396","Type":"ContainerDied","Data":"0fdac1b8e75bb68607bd72e909890c6790f7ac9fe81871b6e7c0931a585e1c34"} Nov 03 01:40:44.487337 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:44.486685 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-jcsn2" event={"ID":"2233c8fb-b2bc-4987-82de-c06a99a7d396","Type":"ContainerDied","Data":"b5ff7e40da5f9d7ff6e98b918152392a64c534a8d5d1648bbd3cda08310e360a"} Nov 03 01:40:44.487337 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:44.486751 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-jcsn2" Nov 03 01:40:44.500005 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:44.499903 4421 scope.go:122] "RemoveContainer" containerID="1c547b60d1d0b1ad263ddb29c394402893bbbeb36320a99b4367ca055962796b" Nov 03 01:40:44.500353 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:40:44.500268 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c547b60d1d0b1ad263ddb29c394402893bbbeb36320a99b4367ca055962796b\": container with ID starting with 1c547b60d1d0b1ad263ddb29c394402893bbbeb36320a99b4367ca055962796b not found: ID does not exist" containerID="1c547b60d1d0b1ad263ddb29c394402893bbbeb36320a99b4367ca055962796b" Nov 03 01:40:44.500448 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:44.500367 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c547b60d1d0b1ad263ddb29c394402893bbbeb36320a99b4367ca055962796b"} err="failed to get container status \"1c547b60d1d0b1ad263ddb29c394402893bbbeb36320a99b4367ca055962796b\": rpc error: code = NotFound desc = could not find container \"1c547b60d1d0b1ad263ddb29c394402893bbbeb36320a99b4367ca055962796b\": container with ID starting with 1c547b60d1d0b1ad263ddb29c394402893bbbeb36320a99b4367ca055962796b not found: ID does not exist" Nov 03 01:40:44.500448 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:44.500395 4421 scope.go:122] "RemoveContainer" containerID="a5f6e226d649f869467b1020f02969e268556d71065246f2002bf534df9c178a" Nov 03 01:40:44.512905 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:44.512881 4421 scope.go:122] "RemoveContainer" containerID="a5f6e226d649f869467b1020f02969e268556d71065246f2002bf534df9c178a" Nov 03 01:40:44.513169 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:40:44.513147 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5f6e226d649f869467b1020f02969e268556d71065246f2002bf534df9c178a\": container with ID starting with a5f6e226d649f869467b1020f02969e268556d71065246f2002bf534df9c178a not found: ID does not exist" containerID="a5f6e226d649f869467b1020f02969e268556d71065246f2002bf534df9c178a" Nov 03 01:40:44.513212 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:44.513176 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5f6e226d649f869467b1020f02969e268556d71065246f2002bf534df9c178a"} err="failed to get container status \"a5f6e226d649f869467b1020f02969e268556d71065246f2002bf534df9c178a\": rpc error: code = NotFound desc = could not find container \"a5f6e226d649f869467b1020f02969e268556d71065246f2002bf534df9c178a\": container with ID starting with a5f6e226d649f869467b1020f02969e268556d71065246f2002bf534df9c178a not found: ID does not exist" Nov 03 01:40:44.513212 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:44.513193 4421 scope.go:122] "RemoveContainer" containerID="35d21581f8d905ac294fb78104bcd18118fbf08cebe90ca75f0d9833b8c127e5" Nov 03 01:40:44.525054 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:44.525029 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-tlggp"] Nov 03 01:40:44.525988 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:44.525967 4421 scope.go:122] "RemoveContainer" containerID="52f533bb4c8a15e2597427f81efc7c95ff1857307d03a59375f513687b8a55a0" Nov 03 01:40:44.529957 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:44.529932 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-tlggp"] Nov 03 01:40:44.538491 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:44.538453 4421 scope.go:122] "RemoveContainer" containerID="35d21581f8d905ac294fb78104bcd18118fbf08cebe90ca75f0d9833b8c127e5" Nov 03 01:40:44.538803 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:40:44.538775 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35d21581f8d905ac294fb78104bcd18118fbf08cebe90ca75f0d9833b8c127e5\": container with ID starting with 35d21581f8d905ac294fb78104bcd18118fbf08cebe90ca75f0d9833b8c127e5 not found: ID does not exist" containerID="35d21581f8d905ac294fb78104bcd18118fbf08cebe90ca75f0d9833b8c127e5" Nov 03 01:40:44.538871 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:44.538814 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35d21581f8d905ac294fb78104bcd18118fbf08cebe90ca75f0d9833b8c127e5"} err="failed to get container status \"35d21581f8d905ac294fb78104bcd18118fbf08cebe90ca75f0d9833b8c127e5\": rpc error: code = NotFound desc = could not find container \"35d21581f8d905ac294fb78104bcd18118fbf08cebe90ca75f0d9833b8c127e5\": container with ID starting with 35d21581f8d905ac294fb78104bcd18118fbf08cebe90ca75f0d9833b8c127e5 not found: ID does not exist" Nov 03 01:40:44.538871 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:44.538837 4421 scope.go:122] "RemoveContainer" containerID="52f533bb4c8a15e2597427f81efc7c95ff1857307d03a59375f513687b8a55a0" Nov 03 01:40:44.539161 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:40:44.539144 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52f533bb4c8a15e2597427f81efc7c95ff1857307d03a59375f513687b8a55a0\": container with ID starting with 52f533bb4c8a15e2597427f81efc7c95ff1857307d03a59375f513687b8a55a0 not found: ID does not exist" containerID="52f533bb4c8a15e2597427f81efc7c95ff1857307d03a59375f513687b8a55a0" Nov 03 01:40:44.539207 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:44.539169 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52f533bb4c8a15e2597427f81efc7c95ff1857307d03a59375f513687b8a55a0"} err="failed to get container status \"52f533bb4c8a15e2597427f81efc7c95ff1857307d03a59375f513687b8a55a0\": rpc error: code = NotFound desc = could not find container \"52f533bb4c8a15e2597427f81efc7c95ff1857307d03a59375f513687b8a55a0\": container with ID starting with 52f533bb4c8a15e2597427f81efc7c95ff1857307d03a59375f513687b8a55a0 not found: ID does not exist" Nov 03 01:40:44.539207 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:44.539188 4421 scope.go:122] "RemoveContainer" containerID="0fdac1b8e75bb68607bd72e909890c6790f7ac9fe81871b6e7c0931a585e1c34" Nov 03 01:40:44.539854 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:44.539834 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-c9sjv"] Nov 03 01:40:44.544165 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:44.544142 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-c9sjv"] Nov 03 01:40:44.550413 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:44.550397 4421 scope.go:122] "RemoveContainer" containerID="0fdac1b8e75bb68607bd72e909890c6790f7ac9fe81871b6e7c0931a585e1c34" Nov 03 01:40:44.550763 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:40:44.550744 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0fdac1b8e75bb68607bd72e909890c6790f7ac9fe81871b6e7c0931a585e1c34\": container with ID starting with 0fdac1b8e75bb68607bd72e909890c6790f7ac9fe81871b6e7c0931a585e1c34 not found: ID does not exist" containerID="0fdac1b8e75bb68607bd72e909890c6790f7ac9fe81871b6e7c0931a585e1c34" Nov 03 01:40:44.550842 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:44.550770 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fdac1b8e75bb68607bd72e909890c6790f7ac9fe81871b6e7c0931a585e1c34"} err="failed to get container status \"0fdac1b8e75bb68607bd72e909890c6790f7ac9fe81871b6e7c0931a585e1c34\": rpc error: code = NotFound desc = could not find container \"0fdac1b8e75bb68607bd72e909890c6790f7ac9fe81871b6e7c0931a585e1c34\": container with ID starting with 0fdac1b8e75bb68607bd72e909890c6790f7ac9fe81871b6e7c0931a585e1c34 not found: ID does not exist" Nov 03 01:40:44.554869 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:44.554846 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-5ws4g"] Nov 03 01:40:44.560080 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:44.560060 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-5ws4g"] Nov 03 01:40:44.571835 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:44.571814 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-jcsn2"] Nov 03 01:40:44.577065 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:44.577040 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-jcsn2"] Nov 03 01:40:44.670444 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:40:44.670410 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 01:40:45.982827 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:45.982783 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05c74cad-b107-403b-bffc-af74dea66076" path="/var/lib/kubelet/pods/05c74cad-b107-403b-bffc-af74dea66076/volumes" Nov 03 01:40:45.983674 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:45.983648 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2233c8fb-b2bc-4987-82de-c06a99a7d396" path="/var/lib/kubelet/pods/2233c8fb-b2bc-4987-82de-c06a99a7d396/volumes" Nov 03 01:40:45.984424 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:45.984400 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73bae713-0108-4120-8486-a35dbf68b0c9" path="/var/lib/kubelet/pods/73bae713-0108-4120-8486-a35dbf68b0c9/volumes" Nov 03 01:40:45.986514 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:45.986494 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e32924b-f5e1-4ecc-8a7d-08647aebf8fd" path="/var/lib/kubelet/pods/8e32924b-f5e1-4ecc-8a7d-08647aebf8fd/volumes" Nov 03 01:40:47.599260 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:47.599209 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-531447-7cf7bc4dcf-xt4bm" podUID="8a3e520f-1e5b-4ae4-b263-c99521d89d9e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:40:48.608687 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:48.608644 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-531447-667d774bfc-9shwq" podUID="9f997097-9fbb-4cf2-b370-f48154901810" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:40:49.456182 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:49.456142 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-556c14/rate-test-model-556c14-kserve-5ccd9784cc-s8fq2"] Nov 03 01:40:49.660330 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:49.657312 4421 generic.go:356] "Generic (PLEG): container finished" podID="8a3e520f-1e5b-4ae4-b263-c99521d89d9e" containerID="b74c3fdc92d31740b8e8a002c49dba79a27ab542c30b0e2649687605e7824252" exitCode=0 Nov 03 01:40:49.660330 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:49.657554 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-531447-7cf7bc4dcf-xt4bm" event={"ID":"8a3e520f-1e5b-4ae4-b263-c99521d89d9e","Type":"ContainerDied","Data":"b74c3fdc92d31740b8e8a002c49dba79a27ab542c30b0e2649687605e7824252"} Nov 03 01:40:49.664331 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:49.661287 4421 generic.go:356] "Generic (PLEG): container finished" podID="9f997097-9fbb-4cf2-b370-f48154901810" containerID="338e3b01c784321a516c487e5b6c5637fb1b9a172718345ee311f93a221ef9ec" exitCode=0 Nov 03 01:40:49.664331 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:49.662396 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-531447-667d774bfc-9shwq" event={"ID":"9f997097-9fbb-4cf2-b370-f48154901810","Type":"ContainerDied","Data":"338e3b01c784321a516c487e5b6c5637fb1b9a172718345ee311f93a221ef9ec"} Nov 03 01:40:49.784261 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:49.784225 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-531447-667d774bfc-9shwq" Nov 03 01:40:49.801222 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:49.801177 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f997097-9fbb-4cf2-b370-f48154901810-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f997097-9fbb-4cf2-b370-f48154901810-plugins-config-volume\") pod \"9f997097-9fbb-4cf2-b370-f48154901810\" (UID: \"9f997097-9fbb-4cf2-b370-f48154901810\") " Nov 03 01:40:49.801486 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:49.801331 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f997097-9fbb-4cf2-b370-f48154901810-kube-api-access-bdq2n\" (UniqueName: \"kubernetes.io/projected/9f997097-9fbb-4cf2-b370-f48154901810-kube-api-access-bdq2n\") pod \"9f997097-9fbb-4cf2-b370-f48154901810\" (UID: \"9f997097-9fbb-4cf2-b370-f48154901810\") " Nov 03 01:40:49.802073 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:49.802011 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f997097-9fbb-4cf2-b370-f48154901810-plugins-config-volume" pod "9f997097-9fbb-4cf2-b370-f48154901810" (UID: "9f997097-9fbb-4cf2-b370-f48154901810"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:40:49.805451 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:49.804495 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f997097-9fbb-4cf2-b370-f48154901810-kube-api-access-bdq2n" pod "9f997097-9fbb-4cf2-b370-f48154901810" (UID: "9f997097-9fbb-4cf2-b370-f48154901810"). InnerVolumeSpecName "kube-api-access-bdq2n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:40:49.917680 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:49.916720 4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f997097-9fbb-4cf2-b370-f48154901810-plugins-config-volume\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:40:49.917680 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:49.916760 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bdq2n\" (UniqueName: \"kubernetes.io/projected/9f997097-9fbb-4cf2-b370-f48154901810-kube-api-access-bdq2n\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:40:50.111471 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:50.111437 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-531447-7cf7bc4dcf-xt4bm" Nov 03 01:40:50.128236 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:50.127078 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a3e520f-1e5b-4ae4-b263-c99521d89d9e-kube-api-access-868qq\" (UniqueName: \"kubernetes.io/projected/8a3e520f-1e5b-4ae4-b263-c99521d89d9e-kube-api-access-868qq\") pod \"8a3e520f-1e5b-4ae4-b263-c99521d89d9e\" (UID: \"8a3e520f-1e5b-4ae4-b263-c99521d89d9e\") " Nov 03 01:40:50.128236 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:50.127227 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a3e520f-1e5b-4ae4-b263-c99521d89d9e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a3e520f-1e5b-4ae4-b263-c99521d89d9e-plugins-config-volume\") pod \"8a3e520f-1e5b-4ae4-b263-c99521d89d9e\" (UID: \"8a3e520f-1e5b-4ae4-b263-c99521d89d9e\") " Nov 03 01:40:50.128715 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:50.128671 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a3e520f-1e5b-4ae4-b263-c99521d89d9e-plugins-config-volume" pod "8a3e520f-1e5b-4ae4-b263-c99521d89d9e" (UID: "8a3e520f-1e5b-4ae4-b263-c99521d89d9e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:40:50.135246 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:50.135219 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a3e520f-1e5b-4ae4-b263-c99521d89d9e-kube-api-access-868qq" pod "8a3e520f-1e5b-4ae4-b263-c99521d89d9e" (UID: "8a3e520f-1e5b-4ae4-b263-c99521d89d9e"). InnerVolumeSpecName "kube-api-access-868qq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:40:50.229400 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:50.229316 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-868qq\" (UniqueName: \"kubernetes.io/projected/8a3e520f-1e5b-4ae4-b263-c99521d89d9e-kube-api-access-868qq\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:40:50.229400 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:50.229353 4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a3e520f-1e5b-4ae4-b263-c99521d89d9e-plugins-config-volume\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:40:50.671973 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:50.671937 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-531447-7cf7bc4dcf-xt4bm" Nov 03 01:40:50.671973 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:50.671967 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-531447-7cf7bc4dcf-xt4bm" event={"ID":"8a3e520f-1e5b-4ae4-b263-c99521d89d9e","Type":"ContainerDied","Data":"785dddfd78f2f79b36b88062848563ed07db86e8dac2918b72dc906238242813"} Nov 03 01:40:50.672556 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:50.672032 4421 scope.go:122] "RemoveContainer" containerID="b74c3fdc92d31740b8e8a002c49dba79a27ab542c30b0e2649687605e7824252" Nov 03 01:40:50.675165 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:50.675129 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-531447-667d774bfc-9shwq" event={"ID":"9f997097-9fbb-4cf2-b370-f48154901810","Type":"ContainerDied","Data":"8dd52f52e7e22dc9353573de40dae33ffaa21d2013eceeb850f7e2e1a62420d6"} Nov 03 01:40:50.675337 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:50.675156 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-531447-667d774bfc-9shwq" Nov 03 01:40:50.686146 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:50.686126 4421 scope.go:122] "RemoveContainer" containerID="338e3b01c784321a516c487e5b6c5637fb1b9a172718345ee311f93a221ef9ec" Nov 03 01:40:50.697523 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:50.697486 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-531447-667d774bfc-9shwq"] Nov 03 01:40:50.706023 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:50.702460 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-531447-667d774bfc-9shwq"] Nov 03 01:40:50.720170 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:50.720134 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-531447-7cf7bc4dcf-xt4bm"] Nov 03 01:40:50.724164 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:50.724139 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-531447-7cf7bc4dcf-xt4bm"] Nov 03 01:40:51.117519 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:51.117477 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w4-6c0ec0/sub-test-model-6c0ec0-kserve-666b858575-xpltp" podUID="bc3c2641-8192-4c01-a52c-29e3cd6de1f8" containerName="main" containerID="cri-o://9a41b6c191fbbb6b19d8629ecb0cf5fe41dffc624c50b5563b93aa83c2641175" gracePeriod=45 Nov 03 01:40:51.123658 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:51.123619 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w4-6c0ec0/sub-test-model-6c0ec0-kserve-666b858575-xpltp" podUID="bc3c2641-8192-4c01-a52c-29e3cd6de1f8" containerName="main" probeResult="failure" output="Get \"https://10.134.1.100:8000/ready\": EOF" Nov 03 01:40:51.500316 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:51.500009 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-6c0ec0/sub-test-model-6c0ec0-kserve-666b858575-xpltp" Nov 03 01:40:51.575868 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:51.575818 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bc3c2641-8192-4c01-a52c-29e3cd6de1f8-tls-certs\" (UniqueName: \"kubernetes.io/secret/bc3c2641-8192-4c01-a52c-29e3cd6de1f8-tls-certs\") pod \"bc3c2641-8192-4c01-a52c-29e3cd6de1f8\" (UID: \"bc3c2641-8192-4c01-a52c-29e3cd6de1f8\") " Nov 03 01:40:51.576088 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:51.576072 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc3c2641-8192-4c01-a52c-29e3cd6de1f8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc3c2641-8192-4c01-a52c-29e3cd6de1f8-model-cache\") pod \"bc3c2641-8192-4c01-a52c-29e3cd6de1f8\" (UID: \"bc3c2641-8192-4c01-a52c-29e3cd6de1f8\") " Nov 03 01:40:51.576188 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:51.576146 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc3c2641-8192-4c01-a52c-29e3cd6de1f8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc3c2641-8192-4c01-a52c-29e3cd6de1f8-dshm\") pod \"bc3c2641-8192-4c01-a52c-29e3cd6de1f8\" (UID: \"bc3c2641-8192-4c01-a52c-29e3cd6de1f8\") " Nov 03 01:40:51.576266 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:51.576238 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc3c2641-8192-4c01-a52c-29e3cd6de1f8-home\" (UniqueName: \"kubernetes.io/empty-dir/bc3c2641-8192-4c01-a52c-29e3cd6de1f8-home\") pod \"bc3c2641-8192-4c01-a52c-29e3cd6de1f8\" (UID: \"bc3c2641-8192-4c01-a52c-29e3cd6de1f8\") " Nov 03 01:40:51.576400 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:51.576359 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc3c2641-8192-4c01-a52c-29e3cd6de1f8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc3c2641-8192-4c01-a52c-29e3cd6de1f8-tmp-dir\") pod \"bc3c2641-8192-4c01-a52c-29e3cd6de1f8\" (UID: \"bc3c2641-8192-4c01-a52c-29e3cd6de1f8\") " Nov 03 01:40:51.576481 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:51.576410 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc3c2641-8192-4c01-a52c-29e3cd6de1f8-kube-api-access-kxbmt\" (UniqueName: \"kubernetes.io/projected/bc3c2641-8192-4c01-a52c-29e3cd6de1f8-kube-api-access-kxbmt\") pod \"bc3c2641-8192-4c01-a52c-29e3cd6de1f8\" (UID: \"bc3c2641-8192-4c01-a52c-29e3cd6de1f8\") " Nov 03 01:40:51.580175 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:51.579877 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc3c2641-8192-4c01-a52c-29e3cd6de1f8-tmp-dir" pod "bc3c2641-8192-4c01-a52c-29e3cd6de1f8" (UID: "bc3c2641-8192-4c01-a52c-29e3cd6de1f8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:40:51.580726 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:51.580701 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc3c2641-8192-4c01-a52c-29e3cd6de1f8-model-cache" pod "bc3c2641-8192-4c01-a52c-29e3cd6de1f8" (UID: "bc3c2641-8192-4c01-a52c-29e3cd6de1f8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:40:51.581052 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:51.581036 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc3c2641-8192-4c01-a52c-29e3cd6de1f8-home" pod "bc3c2641-8192-4c01-a52c-29e3cd6de1f8" (UID: "bc3c2641-8192-4c01-a52c-29e3cd6de1f8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:40:51.581897 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:51.581866 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc3c2641-8192-4c01-a52c-29e3cd6de1f8-kube-api-access-kxbmt" pod "bc3c2641-8192-4c01-a52c-29e3cd6de1f8" (UID: "bc3c2641-8192-4c01-a52c-29e3cd6de1f8"). InnerVolumeSpecName "kube-api-access-kxbmt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:40:51.582364 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:51.582331 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc3c2641-8192-4c01-a52c-29e3cd6de1f8-dshm" pod "bc3c2641-8192-4c01-a52c-29e3cd6de1f8" (UID: "bc3c2641-8192-4c01-a52c-29e3cd6de1f8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:40:51.582659 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:51.582632 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc3c2641-8192-4c01-a52c-29e3cd6de1f8-tls-certs" pod "bc3c2641-8192-4c01-a52c-29e3cd6de1f8" (UID: "bc3c2641-8192-4c01-a52c-29e3cd6de1f8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:40:51.678113 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:51.678073 4421 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc3c2641-8192-4c01-a52c-29e3cd6de1f8-model-cache\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:40:51.678113 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:51.678109 4421 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc3c2641-8192-4c01-a52c-29e3cd6de1f8-dshm\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:40:51.678113 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:51.678119 4421 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc3c2641-8192-4c01-a52c-29e3cd6de1f8-home\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:40:51.678652 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:51.678128 4421 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc3c2641-8192-4c01-a52c-29e3cd6de1f8-tmp-dir\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:40:51.678652 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:51.678138 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kxbmt\" (UniqueName: \"kubernetes.io/projected/bc3c2641-8192-4c01-a52c-29e3cd6de1f8-kube-api-access-kxbmt\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:40:51.678652 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:51.678148 4421 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc3c2641-8192-4c01-a52c-29e3cd6de1f8-tls-certs\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:40:51.683944 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:51.683918 4421 generic.go:356] "Generic (PLEG): container finished" podID="bc3c2641-8192-4c01-a52c-29e3cd6de1f8" containerID="9a41b6c191fbbb6b19d8629ecb0cf5fe41dffc624c50b5563b93aa83c2641175" exitCode=0 Nov 03 01:40:51.684090 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:51.683965 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-6c0ec0/sub-test-model-6c0ec0-kserve-666b858575-xpltp" Nov 03 01:40:51.684090 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:51.684012 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-6c0ec0/sub-test-model-6c0ec0-kserve-666b858575-xpltp" event={"ID":"bc3c2641-8192-4c01-a52c-29e3cd6de1f8","Type":"ContainerDied","Data":"9a41b6c191fbbb6b19d8629ecb0cf5fe41dffc624c50b5563b93aa83c2641175"} Nov 03 01:40:51.684090 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:51.684061 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-6c0ec0/sub-test-model-6c0ec0-kserve-666b858575-xpltp" event={"ID":"bc3c2641-8192-4c01-a52c-29e3cd6de1f8","Type":"ContainerDied","Data":"6cedbc0fd939b1bbc3bfe96b059bb92c1f070041cd30061d9568ca129ad9ffc8"} Nov 03 01:40:51.684090 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:51.684084 4421 scope.go:122] "RemoveContainer" containerID="9a41b6c191fbbb6b19d8629ecb0cf5fe41dffc624c50b5563b93aa83c2641175" Nov 03 01:40:51.701478 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:51.701455 4421 scope.go:122] "RemoveContainer" containerID="9a41b6c191fbbb6b19d8629ecb0cf5fe41dffc624c50b5563b93aa83c2641175" Nov 03 01:40:51.701806 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:40:51.701785 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a41b6c191fbbb6b19d8629ecb0cf5fe41dffc624c50b5563b93aa83c2641175\": container with ID starting with 9a41b6c191fbbb6b19d8629ecb0cf5fe41dffc624c50b5563b93aa83c2641175 not found: ID does not exist" containerID="9a41b6c191fbbb6b19d8629ecb0cf5fe41dffc624c50b5563b93aa83c2641175" Nov 03 01:40:51.701880 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:51.701819 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a41b6c191fbbb6b19d8629ecb0cf5fe41dffc624c50b5563b93aa83c2641175"} err="failed to get container status \"9a41b6c191fbbb6b19d8629ecb0cf5fe41dffc624c50b5563b93aa83c2641175\": rpc error: code = NotFound desc = could not find container \"9a41b6c191fbbb6b19d8629ecb0cf5fe41dffc624c50b5563b93aa83c2641175\": container with ID starting with 9a41b6c191fbbb6b19d8629ecb0cf5fe41dffc624c50b5563b93aa83c2641175 not found: ID does not exist" Nov 03 01:40:51.709360 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:51.709331 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-6c0ec0/sub-test-model-6c0ec0-kserve-666b858575-xpltp"] Nov 03 01:40:51.713121 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:51.713098 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w4-6c0ec0/sub-test-model-6c0ec0-kserve-666b858575-xpltp"] Nov 03 01:40:51.981126 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:51.981043 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a3e520f-1e5b-4ae4-b263-c99521d89d9e" path="/var/lib/kubelet/pods/8a3e520f-1e5b-4ae4-b263-c99521d89d9e/volumes" Nov 03 01:40:51.981648 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:51.981627 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f997097-9fbb-4cf2-b370-f48154901810" path="/var/lib/kubelet/pods/9f997097-9fbb-4cf2-b370-f48154901810/volumes" Nov 03 01:40:51.982049 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:51.982034 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc3c2641-8192-4c01-a52c-29e3cd6de1f8" path="/var/lib/kubelet/pods/bc3c2641-8192-4c01-a52c-29e3cd6de1f8/volumes" Nov 03 01:40:55.959846 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:55.954447 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-d25759-857574b47-bpmv4"] Nov 03 01:40:56.227617 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:56.227522 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w4-4a697f/sub-test-model-4a697f-kserve-588b784c4c-j5qfn" podUID="0494e957-033d-42d6-8b56-ebca9f26b75b" containerName="main" containerID="cri-o://a724d2c9a38ae879b14a60f1249832a5ce6f76d26eb69f7482827d1053d18901" gracePeriod=45 Nov 03 01:40:56.303296 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:56.301140 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-4a697f/sub-test-model-4a697f-kserve-588b784c4c-j5qfn" podUID="0494e957-033d-42d6-8b56-ebca9f26b75b" containerName="main" probeResult="failure" output="Get \"https://10.134.1.103:8000/ready\": dial tcp 10.134.1.103:8000: connect: connection refused" Nov 03 01:40:56.570561 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:56.570534 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-4a697f/sub-test-model-4a697f-kserve-588b784c4c-j5qfn" Nov 03 01:40:56.675966 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:56.675935 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0494e957-033d-42d6-8b56-ebca9f26b75b-home\" (UniqueName: \"kubernetes.io/empty-dir/0494e957-033d-42d6-8b56-ebca9f26b75b-home\") pod \"0494e957-033d-42d6-8b56-ebca9f26b75b\" (UID: \"0494e957-033d-42d6-8b56-ebca9f26b75b\") " Nov 03 01:40:56.676153 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:56.675993 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0494e957-033d-42d6-8b56-ebca9f26b75b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0494e957-033d-42d6-8b56-ebca9f26b75b-tmp-dir\") pod \"0494e957-033d-42d6-8b56-ebca9f26b75b\" (UID: \"0494e957-033d-42d6-8b56-ebca9f26b75b\") " Nov 03 01:40:56.676153 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:56.676101 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0494e957-033d-42d6-8b56-ebca9f26b75b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0494e957-033d-42d6-8b56-ebca9f26b75b-model-cache\") pod \"0494e957-033d-42d6-8b56-ebca9f26b75b\" (UID: \"0494e957-033d-42d6-8b56-ebca9f26b75b\") " Nov 03 01:40:56.676298 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:56.676178 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0494e957-033d-42d6-8b56-ebca9f26b75b-tls-certs\" (UniqueName: \"kubernetes.io/secret/0494e957-033d-42d6-8b56-ebca9f26b75b-tls-certs\") pod \"0494e957-033d-42d6-8b56-ebca9f26b75b\" (UID: \"0494e957-033d-42d6-8b56-ebca9f26b75b\") " Nov 03 01:40:56.676298 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:56.676247 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0494e957-033d-42d6-8b56-ebca9f26b75b-home" pod "0494e957-033d-42d6-8b56-ebca9f26b75b" (UID: "0494e957-033d-42d6-8b56-ebca9f26b75b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:40:56.676423 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:56.676310 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0494e957-033d-42d6-8b56-ebca9f26b75b-tmp-dir" pod "0494e957-033d-42d6-8b56-ebca9f26b75b" (UID: "0494e957-033d-42d6-8b56-ebca9f26b75b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:40:56.676423 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:56.676340 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0494e957-033d-42d6-8b56-ebca9f26b75b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0494e957-033d-42d6-8b56-ebca9f26b75b-dshm\") pod \"0494e957-033d-42d6-8b56-ebca9f26b75b\" (UID: \"0494e957-033d-42d6-8b56-ebca9f26b75b\") " Nov 03 01:40:56.676520 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:56.676425 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0494e957-033d-42d6-8b56-ebca9f26b75b-kube-api-access-r2jxc\" (UniqueName: \"kubernetes.io/projected/0494e957-033d-42d6-8b56-ebca9f26b75b-kube-api-access-r2jxc\") pod \"0494e957-033d-42d6-8b56-ebca9f26b75b\" (UID: \"0494e957-033d-42d6-8b56-ebca9f26b75b\") " Nov 03 01:40:56.676520 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:56.676437 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0494e957-033d-42d6-8b56-ebca9f26b75b-model-cache" pod "0494e957-033d-42d6-8b56-ebca9f26b75b" (UID: "0494e957-033d-42d6-8b56-ebca9f26b75b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:40:56.676984 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:56.676961 4421 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0494e957-033d-42d6-8b56-ebca9f26b75b-home\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:40:56.676984 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:56.676985 4421 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0494e957-033d-42d6-8b56-ebca9f26b75b-tmp-dir\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:40:56.677178 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:56.676997 4421 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0494e957-033d-42d6-8b56-ebca9f26b75b-model-cache\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:40:56.678608 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:56.678586 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0494e957-033d-42d6-8b56-ebca9f26b75b-kube-api-access-r2jxc" pod "0494e957-033d-42d6-8b56-ebca9f26b75b" (UID: "0494e957-033d-42d6-8b56-ebca9f26b75b"). InnerVolumeSpecName "kube-api-access-r2jxc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:40:56.678698 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:56.678639 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0494e957-033d-42d6-8b56-ebca9f26b75b-dshm" pod "0494e957-033d-42d6-8b56-ebca9f26b75b" (UID: "0494e957-033d-42d6-8b56-ebca9f26b75b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:40:56.678738 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:56.678702 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0494e957-033d-42d6-8b56-ebca9f26b75b-tls-certs" pod "0494e957-033d-42d6-8b56-ebca9f26b75b" (UID: "0494e957-033d-42d6-8b56-ebca9f26b75b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:40:56.738947 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:56.736056 4421 generic.go:356] "Generic (PLEG): container finished" podID="0494e957-033d-42d6-8b56-ebca9f26b75b" containerID="a724d2c9a38ae879b14a60f1249832a5ce6f76d26eb69f7482827d1053d18901" exitCode=0 Nov 03 01:40:56.738947 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:56.736292 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-4a697f/sub-test-model-4a697f-kserve-588b784c4c-j5qfn" event={"ID":"0494e957-033d-42d6-8b56-ebca9f26b75b","Type":"ContainerDied","Data":"a724d2c9a38ae879b14a60f1249832a5ce6f76d26eb69f7482827d1053d18901"} Nov 03 01:40:56.738947 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:56.736331 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-4a697f/sub-test-model-4a697f-kserve-588b784c4c-j5qfn" event={"ID":"0494e957-033d-42d6-8b56-ebca9f26b75b","Type":"ContainerDied","Data":"e038a6c2cab1da5bcd977a7b1637d72e3922bbd0dc37a6eb6834a7d88eaba461"} Nov 03 01:40:56.738947 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:56.736355 4421 scope.go:122] "RemoveContainer" containerID="a724d2c9a38ae879b14a60f1249832a5ce6f76d26eb69f7482827d1053d18901" Nov 03 01:40:56.738947 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:56.736512 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-4a697f/sub-test-model-4a697f-kserve-588b784c4c-j5qfn" Nov 03 01:40:56.754533 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:56.753649 4421 scope.go:122] "RemoveContainer" containerID="a724d2c9a38ae879b14a60f1249832a5ce6f76d26eb69f7482827d1053d18901" Nov 03 01:40:56.754786 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:40:56.754598 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a724d2c9a38ae879b14a60f1249832a5ce6f76d26eb69f7482827d1053d18901\": container with ID starting with a724d2c9a38ae879b14a60f1249832a5ce6f76d26eb69f7482827d1053d18901 not found: ID does not exist" containerID="a724d2c9a38ae879b14a60f1249832a5ce6f76d26eb69f7482827d1053d18901" Nov 03 01:40:56.754786 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:56.754629 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a724d2c9a38ae879b14a60f1249832a5ce6f76d26eb69f7482827d1053d18901"} err="failed to get container status \"a724d2c9a38ae879b14a60f1249832a5ce6f76d26eb69f7482827d1053d18901\": rpc error: code = NotFound desc = could not find container \"a724d2c9a38ae879b14a60f1249832a5ce6f76d26eb69f7482827d1053d18901\": container with ID starting with a724d2c9a38ae879b14a60f1249832a5ce6f76d26eb69f7482827d1053d18901 not found: ID does not exist" Nov 03 01:40:56.776775 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:56.776739 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-4a697f/sub-test-model-4a697f-kserve-588b784c4c-j5qfn"] Nov 03 01:40:56.778721 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:56.778696 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w4-4a697f/sub-test-model-4a697f-kserve-588b784c4c-j5qfn"] Nov 03 01:40:56.778791 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:56.778757 4421 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0494e957-033d-42d6-8b56-ebca9f26b75b-tls-certs\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:40:56.778791 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:56.778775 4421 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0494e957-033d-42d6-8b56-ebca9f26b75b-dshm\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:40:56.778791 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:56.778786 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2jxc\" (UniqueName: \"kubernetes.io/projected/0494e957-033d-42d6-8b56-ebca9f26b75b-kube-api-access-r2jxc\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:40:57.023460 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:57.023420 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1230bd0-qx57v"] Nov 03 01:40:57.027400 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:57.027369 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1230bd0-qx57v"] Nov 03 01:40:57.982250 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:57.982218 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0494e957-033d-42d6-8b56-ebca9f26b75b" path="/var/lib/kubelet/pods/0494e957-033d-42d6-8b56-ebca9f26b75b/volumes" Nov 03 01:40:57.982750 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:57.982736 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="845cc619-fae7-4de0-a6a9-33d2e4f6ce87" path="/var/lib/kubelet/pods/845cc619-fae7-4de0-a6a9-33d2e4f6ce87/volumes" Nov 03 01:40:58.187078 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:58.187016 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-717217/rate-test-model-717217-kserve-74ddd7bbcc-4kxbf" podUID="aee71e38-f4ab-40e6-a3ad-1fe79bf0c63f" containerName="main" containerID="cri-o://f395b119afd548c8c73c11058798f5d2d4381f429f3d07ea06d6ae29a02e5275" gracePeriod=45 Nov 03 01:40:58.489677 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:58.489654 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-717217/rate-test-model-717217-kserve-74ddd7bbcc-4kxbf" Nov 03 01:40:58.639134 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:58.639058 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aee71e38-f4ab-40e6-a3ad-1fe79bf0c63f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aee71e38-f4ab-40e6-a3ad-1fe79bf0c63f-model-cache\") pod \"aee71e38-f4ab-40e6-a3ad-1fe79bf0c63f\" (UID: \"aee71e38-f4ab-40e6-a3ad-1fe79bf0c63f\") " Nov 03 01:40:58.639311 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:58.639160 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aee71e38-f4ab-40e6-a3ad-1fe79bf0c63f-home\" (UniqueName: \"kubernetes.io/empty-dir/aee71e38-f4ab-40e6-a3ad-1fe79bf0c63f-home\") pod \"aee71e38-f4ab-40e6-a3ad-1fe79bf0c63f\" (UID: \"aee71e38-f4ab-40e6-a3ad-1fe79bf0c63f\") " Nov 03 01:40:58.639311 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:58.639221 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aee71e38-f4ab-40e6-a3ad-1fe79bf0c63f-tls-certs\" (UniqueName: \"kubernetes.io/secret/aee71e38-f4ab-40e6-a3ad-1fe79bf0c63f-tls-certs\") pod \"aee71e38-f4ab-40e6-a3ad-1fe79bf0c63f\" (UID: \"aee71e38-f4ab-40e6-a3ad-1fe79bf0c63f\") " Nov 03 01:40:58.639397 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:58.639350 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aee71e38-f4ab-40e6-a3ad-1fe79bf0c63f-kube-api-access-j9bn9\" (UniqueName: \"kubernetes.io/projected/aee71e38-f4ab-40e6-a3ad-1fe79bf0c63f-kube-api-access-j9bn9\") pod \"aee71e38-f4ab-40e6-a3ad-1fe79bf0c63f\" (UID: \"aee71e38-f4ab-40e6-a3ad-1fe79bf0c63f\") " Nov 03 01:40:58.639397 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:58.639378 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aee71e38-f4ab-40e6-a3ad-1fe79bf0c63f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aee71e38-f4ab-40e6-a3ad-1fe79bf0c63f-tmp-dir\") pod \"aee71e38-f4ab-40e6-a3ad-1fe79bf0c63f\" (UID: \"aee71e38-f4ab-40e6-a3ad-1fe79bf0c63f\") " Nov 03 01:40:58.639476 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:58.639347 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aee71e38-f4ab-40e6-a3ad-1fe79bf0c63f-model-cache" pod "aee71e38-f4ab-40e6-a3ad-1fe79bf0c63f" (UID: "aee71e38-f4ab-40e6-a3ad-1fe79bf0c63f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:40:58.639476 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:58.639425 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aee71e38-f4ab-40e6-a3ad-1fe79bf0c63f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/aee71e38-f4ab-40e6-a3ad-1fe79bf0c63f-dshm\") pod \"aee71e38-f4ab-40e6-a3ad-1fe79bf0c63f\" (UID: \"aee71e38-f4ab-40e6-a3ad-1fe79bf0c63f\") " Nov 03 01:40:58.639562 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:58.639509 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aee71e38-f4ab-40e6-a3ad-1fe79bf0c63f-home" pod "aee71e38-f4ab-40e6-a3ad-1fe79bf0c63f" (UID: "aee71e38-f4ab-40e6-a3ad-1fe79bf0c63f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:40:58.639691 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:58.639669 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aee71e38-f4ab-40e6-a3ad-1fe79bf0c63f-tmp-dir" pod "aee71e38-f4ab-40e6-a3ad-1fe79bf0c63f" (UID: "aee71e38-f4ab-40e6-a3ad-1fe79bf0c63f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:40:58.640153 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:58.640132 4421 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aee71e38-f4ab-40e6-a3ad-1fe79bf0c63f-tmp-dir\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:40:58.640153 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:58.640151 4421 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aee71e38-f4ab-40e6-a3ad-1fe79bf0c63f-model-cache\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:40:58.640390 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:58.640162 4421 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aee71e38-f4ab-40e6-a3ad-1fe79bf0c63f-home\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:40:58.641567 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:58.641541 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aee71e38-f4ab-40e6-a3ad-1fe79bf0c63f-dshm" pod "aee71e38-f4ab-40e6-a3ad-1fe79bf0c63f" (UID: "aee71e38-f4ab-40e6-a3ad-1fe79bf0c63f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:40:58.641675 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:58.641658 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aee71e38-f4ab-40e6-a3ad-1fe79bf0c63f-kube-api-access-j9bn9" pod "aee71e38-f4ab-40e6-a3ad-1fe79bf0c63f" (UID: "aee71e38-f4ab-40e6-a3ad-1fe79bf0c63f"). InnerVolumeSpecName "kube-api-access-j9bn9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:40:58.641734 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:58.641657 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aee71e38-f4ab-40e6-a3ad-1fe79bf0c63f-tls-certs" pod "aee71e38-f4ab-40e6-a3ad-1fe79bf0c63f" (UID: "aee71e38-f4ab-40e6-a3ad-1fe79bf0c63f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:40:58.748668 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:58.744792 4421 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aee71e38-f4ab-40e6-a3ad-1fe79bf0c63f-tls-certs\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:40:58.748668 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:58.744832 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9bn9\" (UniqueName: \"kubernetes.io/projected/aee71e38-f4ab-40e6-a3ad-1fe79bf0c63f-kube-api-access-j9bn9\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:40:58.748668 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:58.744848 4421 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aee71e38-f4ab-40e6-a3ad-1fe79bf0c63f-dshm\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:40:58.775457 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:58.769551 4421 generic.go:356] "Generic (PLEG): container finished" podID="aee71e38-f4ab-40e6-a3ad-1fe79bf0c63f" containerID="f395b119afd548c8c73c11058798f5d2d4381f429f3d07ea06d6ae29a02e5275" exitCode=0 Nov 03 01:40:58.775457 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:58.769678 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-717217/rate-test-model-717217-kserve-74ddd7bbcc-4kxbf" event={"ID":"aee71e38-f4ab-40e6-a3ad-1fe79bf0c63f","Type":"ContainerDied","Data":"f395b119afd548c8c73c11058798f5d2d4381f429f3d07ea06d6ae29a02e5275"} Nov 03 01:40:58.775457 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:58.769727 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-717217/rate-test-model-717217-kserve-74ddd7bbcc-4kxbf" event={"ID":"aee71e38-f4ab-40e6-a3ad-1fe79bf0c63f","Type":"ContainerDied","Data":"685ff6abf86b1a6f6d83b45bded6df02fa25a01aca8cc733a27cb1053499e1ef"} Nov 03 01:40:58.775457 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:58.769753 4421 scope.go:122] "RemoveContainer" containerID="f395b119afd548c8c73c11058798f5d2d4381f429f3d07ea06d6ae29a02e5275" Nov 03 01:40:58.775457 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:58.769890 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-717217/rate-test-model-717217-kserve-74ddd7bbcc-4kxbf" Nov 03 01:40:58.788583 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:58.788561 4421 scope.go:122] "RemoveContainer" containerID="f395b119afd548c8c73c11058798f5d2d4381f429f3d07ea06d6ae29a02e5275" Nov 03 01:40:58.788904 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:40:58.788883 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f395b119afd548c8c73c11058798f5d2d4381f429f3d07ea06d6ae29a02e5275\": container with ID starting with f395b119afd548c8c73c11058798f5d2d4381f429f3d07ea06d6ae29a02e5275 not found: ID does not exist" containerID="f395b119afd548c8c73c11058798f5d2d4381f429f3d07ea06d6ae29a02e5275" Nov 03 01:40:58.788975 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:58.788913 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f395b119afd548c8c73c11058798f5d2d4381f429f3d07ea06d6ae29a02e5275"} err="failed to get container status \"f395b119afd548c8c73c11058798f5d2d4381f429f3d07ea06d6ae29a02e5275\": rpc error: code = NotFound desc = could not find container \"f395b119afd548c8c73c11058798f5d2d4381f429f3d07ea06d6ae29a02e5275\": container with ID starting with f395b119afd548c8c73c11058798f5d2d4381f429f3d07ea06d6ae29a02e5275 not found: ID does not exist" Nov 03 01:40:58.805130 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:58.805087 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-717217/rate-test-model-717217-kserve-74ddd7bbcc-4kxbf"] Nov 03 01:40:58.815942 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:58.815902 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-717217/rate-test-model-717217-kserve-74ddd7bbcc-4kxbf"] Nov 03 01:40:59.394842 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:59.394805 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-d2znx"] Nov 03 01:40:59.395341 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:59.395084 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-d2znx" podUID="3f9adfd4-8b5b-4a37-b625-9a8727482b7c" containerName="istio-proxy" containerID="cri-o://fec0554a23ecf631530529b990ceb185a106d120263b4d3a6253e441fd2359db" gracePeriod=30 Nov 03 01:40:59.980747 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:40:59.980704 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aee71e38-f4ab-40e6-a3ad-1fe79bf0c63f" path="/var/lib/kubelet/pods/aee71e38-f4ab-40e6-a3ad-1fe79bf0c63f/volumes" Nov 03 01:41:00.219665 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:00.211262 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-d25759-5c4fc584b4-xldtp"] Nov 03 01:41:00.219665 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:00.211538 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d25759-5c4fc584b4-xldtp" podUID="4f9b330f-a209-4b16-a14a-6441781d4889" containerName="maas-api" containerID="cri-o://9ba2c56d8a8c68719f768359b827e5b00b3c4c86b500534989047631adf023b6" gracePeriod=30 Nov 03 01:41:00.327464 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:00.327428 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d925fdb-fbmp5"] Nov 03 01:41:00.328068 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:00.328054 4421 cpu_manager.go:414] "containerMap: removing container" podUID="aee71e38-f4ab-40e6-a3ad-1fe79bf0c63f" containerName="main" Nov 03 01:41:00.328114 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:00.328069 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aee71e38-f4ab-40e6-a3ad-1fe79bf0c63f" containerName="main" Nov 03 01:41:00.328114 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:00.328087 4421 cpu_manager.go:414] "containerMap: removing container" podUID="05c74cad-b107-403b-bffc-af74dea66076" containerName="istio-proxy" Nov 03 01:41:00.328114 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:00.328093 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05c74cad-b107-403b-bffc-af74dea66076" containerName="istio-proxy" Nov 03 01:41:00.328114 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:00.328102 4421 cpu_manager.go:414] "containerMap: removing container" podUID="0494e957-033d-42d6-8b56-ebca9f26b75b" containerName="main" Nov 03 01:41:00.328114 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:00.328107 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0494e957-033d-42d6-8b56-ebca9f26b75b" containerName="main" Nov 03 01:41:00.329402 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:00.328119 4421 cpu_manager.go:414] "containerMap: removing container" podUID="9f997097-9fbb-4cf2-b370-f48154901810" containerName="payload-processing" Nov 03 01:41:00.329402 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:00.328125 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f997097-9fbb-4cf2-b370-f48154901810" containerName="payload-processing" Nov 03 01:41:00.329402 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:00.328140 4421 cpu_manager.go:414] "containerMap: removing container" podUID="8a3e520f-1e5b-4ae4-b263-c99521d89d9e" containerName="payload-pre-processing" Nov 03 01:41:00.329402 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:00.328145 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a3e520f-1e5b-4ae4-b263-c99521d89d9e" containerName="payload-pre-processing" Nov 03 01:41:00.329402 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:00.328174 4421 cpu_manager.go:414] "containerMap: removing container" podUID="bc3c2641-8192-4c01-a52c-29e3cd6de1f8" containerName="main" Nov 03 01:41:00.329402 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:00.328180 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc3c2641-8192-4c01-a52c-29e3cd6de1f8" containerName="main" Nov 03 01:41:00.329402 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:00.328191 4421 cpu_manager.go:414] "containerMap: removing container" podUID="e2b6d01d-9fcb-4164-a217-2a596801ce5b" containerName="revoke-keys" Nov 03 01:41:00.329402 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:00.328196 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2b6d01d-9fcb-4164-a217-2a596801ce5b" containerName="revoke-keys" Nov 03 01:41:00.329402 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:00.328206 4421 cpu_manager.go:414] "containerMap: removing container" podUID="2233c8fb-b2bc-4987-82de-c06a99a7d396" containerName="istio-proxy" Nov 03 01:41:00.329402 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:00.328212 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2233c8fb-b2bc-4987-82de-c06a99a7d396" containerName="istio-proxy" Nov 03 01:41:00.329402 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:00.328220 4421 cpu_manager.go:414] "containerMap: removing container" podUID="8e32924b-f5e1-4ecc-8a7d-08647aebf8fd" containerName="istio-proxy" Nov 03 01:41:00.329402 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:00.328225 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e32924b-f5e1-4ecc-8a7d-08647aebf8fd" containerName="istio-proxy" Nov 03 01:41:00.329402 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:00.328233 4421 cpu_manager.go:414] "containerMap: removing container" podUID="73bae713-0108-4120-8486-a35dbf68b0c9" containerName="istio-proxy" Nov 03 01:41:00.329402 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:00.328239 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73bae713-0108-4120-8486-a35dbf68b0c9" containerName="istio-proxy" Nov 03 01:41:00.329402 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:00.328375 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="05c74cad-b107-403b-bffc-af74dea66076" containerName="istio-proxy" Nov 03 01:41:00.329402 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:00.328386 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="aee71e38-f4ab-40e6-a3ad-1fe79bf0c63f" containerName="main" Nov 03 01:41:00.329402 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:00.328395 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="05c74cad-b107-403b-bffc-af74dea66076" containerName="istio-proxy" Nov 03 01:41:00.329402 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:00.328409 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="0494e957-033d-42d6-8b56-ebca9f26b75b" containerName="main" Nov 03 01:41:00.329402 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:00.328416 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f997097-9fbb-4cf2-b370-f48154901810" containerName="payload-processing" Nov 03 01:41:00.329402 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:00.328423 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a3e520f-1e5b-4ae4-b263-c99521d89d9e" containerName="payload-pre-processing" Nov 03 01:41:00.329402 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:00.328431 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc3c2641-8192-4c01-a52c-29e3cd6de1f8" containerName="main" Nov 03 01:41:00.329402 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:00.328437 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2b6d01d-9fcb-4164-a217-2a596801ce5b" containerName="revoke-keys" Nov 03 01:41:00.329402 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:00.328443 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="2233c8fb-b2bc-4987-82de-c06a99a7d396" containerName="istio-proxy" Nov 03 01:41:00.329402 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:00.328448 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e32924b-f5e1-4ecc-8a7d-08647aebf8fd" containerName="istio-proxy" Nov 03 01:41:00.329402 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:00.328453 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="73bae713-0108-4120-8486-a35dbf68b0c9" containerName="istio-proxy" Nov 03 01:41:00.331902 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:00.331886 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d925fdb-fbmp5" Nov 03 01:41:00.337732 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:00.337710 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6q4xp\"" Nov 03 01:41:00.338005 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:00.337981 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:41:00.343044 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:00.343001 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d925fdb-fbmp5"] Nov 03 01:41:00.372031 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:00.371998 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6dbc64a9-42b0-407c-b1d5-f2651acc523b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6d925fdb-fbmp5\" (UID: \"6dbc64a9-42b0-407c-b1d5-f2651acc523b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d925fdb-fbmp5" Nov 03 01:41:00.372340 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:00.372312 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-699dk\" (UniqueName: \"kubernetes.io/projected/6dbc64a9-42b0-407c-b1d5-f2651acc523b-kube-api-access-699dk\") pod \"maas-api-revoke-sub-6d925fdb-fbmp5\" (UID: \"6dbc64a9-42b0-407c-b1d5-f2651acc523b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d925fdb-fbmp5" Nov 03 01:41:00.372427 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:00.372411 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6dbc64a9-42b0-407c-b1d5-f2651acc523b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6d925fdb-fbmp5\" (UID: \"6dbc64a9-42b0-407c-b1d5-f2651acc523b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d925fdb-fbmp5" Nov 03 01:41:00.486347 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:00.474227 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-699dk\" (UniqueName: \"kubernetes.io/projected/6dbc64a9-42b0-407c-b1d5-f2651acc523b-kube-api-access-699dk\") pod \"maas-api-revoke-sub-6d925fdb-fbmp5\" (UID: \"6dbc64a9-42b0-407c-b1d5-f2651acc523b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d925fdb-fbmp5" Nov 03 01:41:00.486347 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:00.474324 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6dbc64a9-42b0-407c-b1d5-f2651acc523b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6d925fdb-fbmp5\" (UID: \"6dbc64a9-42b0-407c-b1d5-f2651acc523b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d925fdb-fbmp5" Nov 03 01:41:00.486347 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:00.474449 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6dbc64a9-42b0-407c-b1d5-f2651acc523b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6d925fdb-fbmp5\" (UID: \"6dbc64a9-42b0-407c-b1d5-f2651acc523b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d925fdb-fbmp5" Nov 03 01:41:00.486347 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:00.485102 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6dbc64a9-42b0-407c-b1d5-f2651acc523b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6d925fdb-fbmp5\" (UID: \"6dbc64a9-42b0-407c-b1d5-f2651acc523b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d925fdb-fbmp5" Nov 03 01:41:00.530130 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:00.503626 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6dbc64a9-42b0-407c-b1d5-f2651acc523b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6d925fdb-fbmp5\" (UID: \"6dbc64a9-42b0-407c-b1d5-f2651acc523b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d925fdb-fbmp5" Nov 03 01:41:00.530130 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:00.504658 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-699dk\" (UniqueName: \"kubernetes.io/projected/6dbc64a9-42b0-407c-b1d5-f2651acc523b-kube-api-access-699dk\") pod \"maas-api-revoke-sub-6d925fdb-fbmp5\" (UID: \"6dbc64a9-42b0-407c-b1d5-f2651acc523b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d925fdb-fbmp5" Nov 03 01:41:00.655874 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:00.655846 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d925fdb-fbmp5" Nov 03 01:41:00.672805 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:00.672784 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d25759-5c4fc584b4-xldtp" Nov 03 01:41:00.780199 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:00.778751 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f9b330f-a209-4b16-a14a-6441781d4889-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f9b330f-a209-4b16-a14a-6441781d4889-openshift-service-ca\") pod \"4f9b330f-a209-4b16-a14a-6441781d4889\" (UID: \"4f9b330f-a209-4b16-a14a-6441781d4889\") " Nov 03 01:41:00.780199 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:00.778799 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f9b330f-a209-4b16-a14a-6441781d4889-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f9b330f-a209-4b16-a14a-6441781d4889-maas-api-tls\") pod \"4f9b330f-a209-4b16-a14a-6441781d4889\" (UID: \"4f9b330f-a209-4b16-a14a-6441781d4889\") " Nov 03 01:41:00.780199 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:00.778818 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f9b330f-a209-4b16-a14a-6441781d4889-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f9b330f-a209-4b16-a14a-6441781d4889-metrics-certs\") pod \"4f9b330f-a209-4b16-a14a-6441781d4889\" (UID: \"4f9b330f-a209-4b16-a14a-6441781d4889\") " Nov 03 01:41:00.780199 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:00.778850 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f9b330f-a209-4b16-a14a-6441781d4889-kube-api-access-5fvpx\" (UniqueName: \"kubernetes.io/projected/4f9b330f-a209-4b16-a14a-6441781d4889-kube-api-access-5fvpx\") pod \"4f9b330f-a209-4b16-a14a-6441781d4889\" (UID: \"4f9b330f-a209-4b16-a14a-6441781d4889\") " Nov 03 01:41:00.780199 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:00.779785 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f9b330f-a209-4b16-a14a-6441781d4889-openshift-service-ca" pod "4f9b330f-a209-4b16-a14a-6441781d4889" (UID: "4f9b330f-a209-4b16-a14a-6441781d4889"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:41:00.785335 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:00.785236 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f9b330f-a209-4b16-a14a-6441781d4889-kube-api-access-5fvpx" pod "4f9b330f-a209-4b16-a14a-6441781d4889" (UID: "4f9b330f-a209-4b16-a14a-6441781d4889"). InnerVolumeSpecName "kube-api-access-5fvpx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:41:00.787161 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:00.787128 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f9b330f-a209-4b16-a14a-6441781d4889-metrics-certs" pod "4f9b330f-a209-4b16-a14a-6441781d4889" (UID: "4f9b330f-a209-4b16-a14a-6441781d4889"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:41:00.787646 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:00.787615 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f9b330f-a209-4b16-a14a-6441781d4889-maas-api-tls" pod "4f9b330f-a209-4b16-a14a-6441781d4889" (UID: "4f9b330f-a209-4b16-a14a-6441781d4889"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:41:00.791438 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:00.791409 4421 generic.go:356] "Generic (PLEG): container finished" podID="4f9b330f-a209-4b16-a14a-6441781d4889" containerID="9ba2c56d8a8c68719f768359b827e5b00b3c4c86b500534989047631adf023b6" exitCode=0 Nov 03 01:41:00.791556 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:00.791478 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d25759-5c4fc584b4-xldtp" Nov 03 01:41:00.791556 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:00.791493 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d25759-5c4fc584b4-xldtp" event={"ID":"4f9b330f-a209-4b16-a14a-6441781d4889","Type":"ContainerDied","Data":"9ba2c56d8a8c68719f768359b827e5b00b3c4c86b500534989047631adf023b6"} Nov 03 01:41:00.791686 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:00.791557 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d25759-5c4fc584b4-xldtp" event={"ID":"4f9b330f-a209-4b16-a14a-6441781d4889","Type":"ContainerDied","Data":"5853776eb5fcd897175b7810ed2d1a5b67ff81055cbb66399a510750a3260141"} Nov 03 01:41:00.791686 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:00.791588 4421 scope.go:122] "RemoveContainer" containerID="9ba2c56d8a8c68719f768359b827e5b00b3c4c86b500534989047631adf023b6" Nov 03 01:41:00.818375 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:00.818349 4421 scope.go:122] "RemoveContainer" containerID="9ba2c56d8a8c68719f768359b827e5b00b3c4c86b500534989047631adf023b6" Nov 03 01:41:00.818794 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:41:00.818766 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ba2c56d8a8c68719f768359b827e5b00b3c4c86b500534989047631adf023b6\": container with ID starting with 9ba2c56d8a8c68719f768359b827e5b00b3c4c86b500534989047631adf023b6 not found: ID does not exist" containerID="9ba2c56d8a8c68719f768359b827e5b00b3c4c86b500534989047631adf023b6" Nov 03 01:41:00.818880 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:00.818809 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ba2c56d8a8c68719f768359b827e5b00b3c4c86b500534989047631adf023b6"} err="failed to get container status \"9ba2c56d8a8c68719f768359b827e5b00b3c4c86b500534989047631adf023b6\": rpc error: code = NotFound desc = could not find container \"9ba2c56d8a8c68719f768359b827e5b00b3c4c86b500534989047631adf023b6\": container with ID starting with 9ba2c56d8a8c68719f768359b827e5b00b3c4c86b500534989047631adf023b6 not found: ID does not exist" Nov 03 01:41:00.832321 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:00.825539 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-d25759-5c4fc584b4-xldtp"] Nov 03 01:41:00.832321 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:00.831601 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-d25759-5c4fc584b4-xldtp"] Nov 03 01:41:00.880999 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:00.880953 4421 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f9b330f-a209-4b16-a14a-6441781d4889-openshift-service-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:00.880999 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:00.880991 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f9b330f-a209-4b16-a14a-6441781d4889-maas-api-tls\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:00.880999 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:00.881005 4421 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f9b330f-a209-4b16-a14a-6441781d4889-metrics-certs\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:00.881265 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:00.881017 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5fvpx\" (UniqueName: \"kubernetes.io/projected/4f9b330f-a209-4b16-a14a-6441781d4889-kube-api-access-5fvpx\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:00.955189 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:00.955146 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d25759-857574b47-bpmv4" podUID="d5576371-83ce-4e3e-b021-f0de09a49d6e" containerName="payload-pre-processing" containerID="cri-o://28f81a1d05bfcf330a3f478d3b32e252f1a8ecc18e619bef8e09c3c8c3c7e3dd" gracePeriod=25 Nov 03 01:41:00.956590 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:00.956560 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d925fdb-fbmp5"] Nov 03 01:41:00.957443 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:41:00.957420 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6dbc64a9_42b0_407c_b1d5_f2651acc523b.slice/crio-ab3ec0bd772de3bbc361d8b01af8751a0b4f3de1c65c6d268aa90990646d34b4 WatchSource:0}: Error finding container ab3ec0bd772de3bbc361d8b01af8751a0b4f3de1c65c6d268aa90990646d34b4: Status 404 returned error can't find the container with id ab3ec0bd772de3bbc361d8b01af8751a0b4f3de1c65c6d268aa90990646d34b4 Nov 03 01:41:01.801906 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:01.801862 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d925fdb-fbmp5" event={"ID":"6dbc64a9-42b0-407c-b1d5-f2651acc523b","Type":"ContainerStarted","Data":"f83b31037917694bbc17199dff8d6feaf83050d17244445c4ed549e1106a5c60"} Nov 03 01:41:01.802451 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:01.801923 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d925fdb-fbmp5" event={"ID":"6dbc64a9-42b0-407c-b1d5-f2651acc523b","Type":"ContainerStarted","Data":"ab3ec0bd772de3bbc361d8b01af8751a0b4f3de1c65c6d268aa90990646d34b4"} Nov 03 01:41:01.818364 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:01.818305 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d925fdb-fbmp5" podStartSLOduration=1.81828539 podStartE2EDuration="1.81828539s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:41:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:41:01.817250218 +0000 UTC m=+3603.988253316" watchObservedRunningTime="2026-11-03 01:41:01.81828539 +0000 UTC m=+3603.989288490" Nov 03 01:41:02.025006 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:02.024966 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f9b330f-a209-4b16-a14a-6441781d4889" path="/var/lib/kubelet/pods/4f9b330f-a209-4b16-a14a-6441781d4889/volumes" Nov 03 01:41:02.811111 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:02.811069 4421 generic.go:356] "Generic (PLEG): container finished" podID="6dbc64a9-42b0-407c-b1d5-f2651acc523b" containerID="f83b31037917694bbc17199dff8d6feaf83050d17244445c4ed549e1106a5c60" exitCode=0 Nov 03 01:41:02.811645 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:02.811145 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d925fdb-fbmp5" event={"ID":"6dbc64a9-42b0-407c-b1d5-f2651acc523b","Type":"ContainerDied","Data":"f83b31037917694bbc17199dff8d6feaf83050d17244445c4ed549e1106a5c60"} Nov 03 01:41:03.190104 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:03.189995 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d25759-857574b47-bpmv4" podUID="d5576371-83ce-4e3e-b021-f0de09a49d6e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:41:03.258810 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:03.258761 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-67sww"] Nov 03 01:41:03.259560 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:03.259536 4421 cpu_manager.go:414] "containerMap: removing container" podUID="05c74cad-b107-403b-bffc-af74dea66076" containerName="istio-proxy" Nov 03 01:41:03.259560 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:03.259558 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05c74cad-b107-403b-bffc-af74dea66076" containerName="istio-proxy" Nov 03 01:41:03.259762 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:03.259587 4421 cpu_manager.go:414] "containerMap: removing container" podUID="4f9b330f-a209-4b16-a14a-6441781d4889" containerName="maas-api" Nov 03 01:41:03.259762 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:03.259596 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f9b330f-a209-4b16-a14a-6441781d4889" containerName="maas-api" Nov 03 01:41:03.259868 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:03.259835 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f9b330f-a209-4b16-a14a-6441781d4889" containerName="maas-api" Nov 03 01:41:03.263442 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:03.263422 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-67sww" Nov 03 01:41:03.272516 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:03.272480 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-4af1f7-openshift-default-dockercfg-dmrp9\"" Nov 03 01:41:03.276498 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:03.276467 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-67sww"] Nov 03 01:41:03.426097 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:03.419266 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35687f7e-87be-465b-ab66-ec518b459edc-credential-socket\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-67sww\" (UID: \"35687f7e-87be-465b-ab66-ec518b459edc\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-67sww" Nov 03 01:41:03.426097 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:03.419352 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35687f7e-87be-465b-ab66-ec518b459edc-istio-token\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-67sww\" (UID: \"35687f7e-87be-465b-ab66-ec518b459edc\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-67sww" Nov 03 01:41:03.426097 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:03.419391 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35687f7e-87be-465b-ab66-ec518b459edc-workload-socket\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-67sww\" (UID: \"35687f7e-87be-465b-ab66-ec518b459edc\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-67sww" Nov 03 01:41:03.426097 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:03.419455 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35687f7e-87be-465b-ab66-ec518b459edc-istiod-ca-cert\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-67sww\" (UID: \"35687f7e-87be-465b-ab66-ec518b459edc\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-67sww" Nov 03 01:41:03.426097 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:03.419497 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35687f7e-87be-465b-ab66-ec518b459edc-istio-envoy\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-67sww\" (UID: \"35687f7e-87be-465b-ab66-ec518b459edc\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-67sww" Nov 03 01:41:03.426097 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:03.419668 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35687f7e-87be-465b-ab66-ec518b459edc-istio-data\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-67sww\" (UID: \"35687f7e-87be-465b-ab66-ec518b459edc\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-67sww" Nov 03 01:41:03.426097 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:03.419778 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35687f7e-87be-465b-ab66-ec518b459edc-workload-certs\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-67sww\" (UID: \"35687f7e-87be-465b-ab66-ec518b459edc\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-67sww" Nov 03 01:41:03.426097 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:03.419927 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hg25\" (UniqueName: \"kubernetes.io/projected/35687f7e-87be-465b-ab66-ec518b459edc-kube-api-access-2hg25\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-67sww\" (UID: \"35687f7e-87be-465b-ab66-ec518b459edc\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-67sww" Nov 03 01:41:03.426097 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:03.420018 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35687f7e-87be-465b-ab66-ec518b459edc-istio-podinfo\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-67sww\" (UID: \"35687f7e-87be-465b-ab66-ec518b459edc\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-67sww" Nov 03 01:41:03.521859 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:03.521822 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35687f7e-87be-465b-ab66-ec518b459edc-workload-certs\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-67sww\" (UID: \"35687f7e-87be-465b-ab66-ec518b459edc\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-67sww" Nov 03 01:41:03.522087 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:03.521886 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hg25\" (UniqueName: \"kubernetes.io/projected/35687f7e-87be-465b-ab66-ec518b459edc-kube-api-access-2hg25\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-67sww\" (UID: \"35687f7e-87be-465b-ab66-ec518b459edc\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-67sww" Nov 03 01:41:03.522087 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:03.521923 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35687f7e-87be-465b-ab66-ec518b459edc-istio-podinfo\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-67sww\" (UID: \"35687f7e-87be-465b-ab66-ec518b459edc\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-67sww" Nov 03 01:41:03.522087 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:03.521948 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35687f7e-87be-465b-ab66-ec518b459edc-credential-socket\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-67sww\" (UID: \"35687f7e-87be-465b-ab66-ec518b459edc\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-67sww" Nov 03 01:41:03.522087 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:03.521977 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35687f7e-87be-465b-ab66-ec518b459edc-istio-token\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-67sww\" (UID: \"35687f7e-87be-465b-ab66-ec518b459edc\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-67sww" Nov 03 01:41:03.522087 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:03.521997 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35687f7e-87be-465b-ab66-ec518b459edc-workload-socket\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-67sww\" (UID: \"35687f7e-87be-465b-ab66-ec518b459edc\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-67sww" Nov 03 01:41:03.522087 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:03.522051 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35687f7e-87be-465b-ab66-ec518b459edc-istiod-ca-cert\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-67sww\" (UID: \"35687f7e-87be-465b-ab66-ec518b459edc\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-67sww" Nov 03 01:41:03.522593 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:03.522093 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35687f7e-87be-465b-ab66-ec518b459edc-istio-envoy\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-67sww\" (UID: \"35687f7e-87be-465b-ab66-ec518b459edc\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-67sww" Nov 03 01:41:03.522593 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:03.522261 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35687f7e-87be-465b-ab66-ec518b459edc-istio-data\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-67sww\" (UID: \"35687f7e-87be-465b-ab66-ec518b459edc\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-67sww" Nov 03 01:41:03.522593 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:03.522358 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35687f7e-87be-465b-ab66-ec518b459edc-workload-certs\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-67sww\" (UID: \"35687f7e-87be-465b-ab66-ec518b459edc\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-67sww" Nov 03 01:41:03.522593 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:03.522445 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35687f7e-87be-465b-ab66-ec518b459edc-credential-socket\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-67sww\" (UID: \"35687f7e-87be-465b-ab66-ec518b459edc\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-67sww" Nov 03 01:41:03.522593 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:03.522550 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35687f7e-87be-465b-ab66-ec518b459edc-workload-socket\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-67sww\" (UID: \"35687f7e-87be-465b-ab66-ec518b459edc\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-67sww" Nov 03 01:41:03.522887 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:03.522763 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35687f7e-87be-465b-ab66-ec518b459edc-istio-data\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-67sww\" (UID: \"35687f7e-87be-465b-ab66-ec518b459edc\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-67sww" Nov 03 01:41:03.522887 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:03.522836 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35687f7e-87be-465b-ab66-ec518b459edc-istiod-ca-cert\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-67sww\" (UID: \"35687f7e-87be-465b-ab66-ec518b459edc\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-67sww" Nov 03 01:41:03.524578 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:03.524556 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35687f7e-87be-465b-ab66-ec518b459edc-istio-envoy\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-67sww\" (UID: \"35687f7e-87be-465b-ab66-ec518b459edc\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-67sww" Nov 03 01:41:03.524686 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:03.524597 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35687f7e-87be-465b-ab66-ec518b459edc-istio-podinfo\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-67sww\" (UID: \"35687f7e-87be-465b-ab66-ec518b459edc\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-67sww" Nov 03 01:41:03.529761 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:03.529732 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hg25\" (UniqueName: \"kubernetes.io/projected/35687f7e-87be-465b-ab66-ec518b459edc-kube-api-access-2hg25\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-67sww\" (UID: \"35687f7e-87be-465b-ab66-ec518b459edc\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-67sww" Nov 03 01:41:03.529761 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:03.529753 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35687f7e-87be-465b-ab66-ec518b459edc-istio-token\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-67sww\" (UID: \"35687f7e-87be-465b-ab66-ec518b459edc\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-67sww" Nov 03 01:41:03.585669 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:03.585624 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-67sww" Nov 03 01:41:04.013869 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:04.013840 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-67sww"] Nov 03 01:41:04.014725 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:41:04.014698 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35687f7e_87be_465b_ab66_ec518b459edc.slice/crio-69956387b6fd5667d4dc9c7d18be5c787ca0667741fc584cfbad8a2d4840802c WatchSource:0}: Error finding container 69956387b6fd5667d4dc9c7d18be5c787ca0667741fc584cfbad8a2d4840802c: Status 404 returned error can't find the container with id 69956387b6fd5667d4dc9c7d18be5c787ca0667741fc584cfbad8a2d4840802c Nov 03 01:41:04.018203 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:04.018164 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:41:04.018293 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:04.018250 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:41:04.018335 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:04.018312 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:41:04.053024 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:04.052999 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d925fdb-fbmp5" Nov 03 01:41:04.169134 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:04.169099 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6dbc64a9-42b0-407c-b1d5-f2651acc523b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6dbc64a9-42b0-407c-b1d5-f2651acc523b-maas-api-openshift-ca\") pod \"6dbc64a9-42b0-407c-b1d5-f2651acc523b\" (UID: \"6dbc64a9-42b0-407c-b1d5-f2651acc523b\") " Nov 03 01:41:04.170085 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:04.170050 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6dbc64a9-42b0-407c-b1d5-f2651acc523b-kube-api-access-699dk\" (UniqueName: \"kubernetes.io/projected/6dbc64a9-42b0-407c-b1d5-f2651acc523b-kube-api-access-699dk\") pod \"6dbc64a9-42b0-407c-b1d5-f2651acc523b\" (UID: \"6dbc64a9-42b0-407c-b1d5-f2651acc523b\") " Nov 03 01:41:04.170247 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:04.170171 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6dbc64a9-42b0-407c-b1d5-f2651acc523b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6dbc64a9-42b0-407c-b1d5-f2651acc523b-maas-api-xks-ca\") pod \"6dbc64a9-42b0-407c-b1d5-f2651acc523b\" (UID: \"6dbc64a9-42b0-407c-b1d5-f2651acc523b\") " Nov 03 01:41:04.171418 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:04.169826 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6dbc64a9-42b0-407c-b1d5-f2651acc523b-maas-api-openshift-ca" pod "6dbc64a9-42b0-407c-b1d5-f2651acc523b" (UID: "6dbc64a9-42b0-407c-b1d5-f2651acc523b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:41:04.204637 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:04.204560 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6dbc64a9-42b0-407c-b1d5-f2651acc523b-kube-api-access-699dk" pod "6dbc64a9-42b0-407c-b1d5-f2651acc523b" (UID: "6dbc64a9-42b0-407c-b1d5-f2651acc523b"). InnerVolumeSpecName "kube-api-access-699dk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:41:04.206232 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:04.206200 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6dbc64a9-42b0-407c-b1d5-f2651acc523b-maas-api-xks-ca" pod "6dbc64a9-42b0-407c-b1d5-f2651acc523b" (UID: "6dbc64a9-42b0-407c-b1d5-f2651acc523b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:41:04.210492 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:04.210380 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-zxs8d"] Nov 03 01:41:04.210633 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:04.210608 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-zxs8d" podUID="6e5cb5e3-4c37-4ae7-9c6b-ef145cb87eb3" containerName="istio-proxy" containerID="cri-o://00666c657a79b88b3f1f97e09cdcc20c75e999c4caa98f003675fa116d6d4a56" gracePeriod=30 Nov 03 01:41:04.272048 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:04.272016 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6dbc64a9-42b0-407c-b1d5-f2651acc523b-maas-api-openshift-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:04.272048 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:04.272046 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-699dk\" (UniqueName: \"kubernetes.io/projected/6dbc64a9-42b0-407c-b1d5-f2651acc523b-kube-api-access-699dk\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:04.272048 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:04.272057 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6dbc64a9-42b0-407c-b1d5-f2651acc523b-maas-api-xks-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:04.480568 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:04.480524 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-556c14/rate-test-model-556c14-kserve-5ccd9784cc-s8fq2" podUID="141cb342-025b-46eb-83b8-cbd2a60df0c0" containerName="main" containerID="cri-o://01092830cf39cda527bba975b19fae5025e0ad65e1f647a3dc67810494b8c5b7" gracePeriod=45 Nov 03 01:41:04.504948 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:04.504915 4421 patch_prober.go:29] interesting pod/e2e-extmod-a-d25759-openshift-default-d47fd9785-zxs8d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 01:41:04.505109 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:04.504970 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-zxs8d" podUID="6e5cb5e3-4c37-4ae7-9c6b-ef145cb87eb3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 01:41:04.861123 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:04.861088 4421 generic.go:356] "Generic (PLEG): container finished" podID="141cb342-025b-46eb-83b8-cbd2a60df0c0" containerID="01092830cf39cda527bba975b19fae5025e0ad65e1f647a3dc67810494b8c5b7" exitCode=0 Nov 03 01:41:04.861341 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:04.861148 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-556c14/rate-test-model-556c14-kserve-5ccd9784cc-s8fq2" event={"ID":"141cb342-025b-46eb-83b8-cbd2a60df0c0","Type":"ContainerDied","Data":"01092830cf39cda527bba975b19fae5025e0ad65e1f647a3dc67810494b8c5b7"} Nov 03 01:41:04.863672 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:04.863642 4421 generic.go:356] "Generic (PLEG): container finished" podID="3f9adfd4-8b5b-4a37-b625-9a8727482b7c" containerID="fec0554a23ecf631530529b990ceb185a106d120263b4d3a6253e441fd2359db" exitCode=0 Nov 03 01:41:04.863920 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:04.863892 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-d2znx" event={"ID":"3f9adfd4-8b5b-4a37-b625-9a8727482b7c","Type":"ContainerDied","Data":"fec0554a23ecf631530529b990ceb185a106d120263b4d3a6253e441fd2359db"} Nov 03 01:41:04.864024 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:04.863957 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-d2znx" event={"ID":"3f9adfd4-8b5b-4a37-b625-9a8727482b7c","Type":"ContainerDied","Data":"bc404db1142800a81551d059fc6b1c676395f308de4a12e024b007e6521f082f"} Nov 03 01:41:04.864024 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:04.863982 4421 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bc404db1142800a81551d059fc6b1c676395f308de4a12e024b007e6521f082f" Nov 03 01:41:04.865678 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:04.865652 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d925fdb-fbmp5" Nov 03 01:41:04.865678 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:04.865660 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d925fdb-fbmp5" event={"ID":"6dbc64a9-42b0-407c-b1d5-f2651acc523b","Type":"ContainerDied","Data":"ab3ec0bd772de3bbc361d8b01af8751a0b4f3de1c65c6d268aa90990646d34b4"} Nov 03 01:41:04.865879 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:04.865705 4421 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ab3ec0bd772de3bbc361d8b01af8751a0b4f3de1c65c6d268aa90990646d34b4" Nov 03 01:41:04.866190 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:04.866166 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-d2znx" Nov 03 01:41:04.869124 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:04.867931 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-67sww" event={"ID":"35687f7e-87be-465b-ab66-ec518b459edc","Type":"ContainerStarted","Data":"1de77136bb6bdb59b4401f8aaffa55e621774945ac035c09c55473d1d1c2042e"} Nov 03 01:41:04.869124 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:04.867985 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-67sww" event={"ID":"35687f7e-87be-465b-ab66-ec518b459edc","Type":"ContainerStarted","Data":"69956387b6fd5667d4dc9c7d18be5c787ca0667741fc584cfbad8a2d4840802c"} Nov 03 01:41:04.887363 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:04.887329 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f9adfd4-8b5b-4a37-b625-9a8727482b7c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f9adfd4-8b5b-4a37-b625-9a8727482b7c-workload-socket\") pod \"3f9adfd4-8b5b-4a37-b625-9a8727482b7c\" (UID: \"3f9adfd4-8b5b-4a37-b625-9a8727482b7c\") " Nov 03 01:41:04.888678 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:04.887368 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f9adfd4-8b5b-4a37-b625-9a8727482b7c-istio-token\" (UniqueName: \"kubernetes.io/projected/3f9adfd4-8b5b-4a37-b625-9a8727482b7c-istio-token\") pod \"3f9adfd4-8b5b-4a37-b625-9a8727482b7c\" (UID: \"3f9adfd4-8b5b-4a37-b625-9a8727482b7c\") " Nov 03 01:41:04.888834 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:04.888790 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f9adfd4-8b5b-4a37-b625-9a8727482b7c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f9adfd4-8b5b-4a37-b625-9a8727482b7c-istiod-ca-cert\") pod \"3f9adfd4-8b5b-4a37-b625-9a8727482b7c\" (UID: \"3f9adfd4-8b5b-4a37-b625-9a8727482b7c\") " Nov 03 01:41:04.888921 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:04.888899 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f9adfd4-8b5b-4a37-b625-9a8727482b7c-kube-api-access-5q5cx\" (UniqueName: \"kubernetes.io/projected/3f9adfd4-8b5b-4a37-b625-9a8727482b7c-kube-api-access-5q5cx\") pod \"3f9adfd4-8b5b-4a37-b625-9a8727482b7c\" (UID: \"3f9adfd4-8b5b-4a37-b625-9a8727482b7c\") " Nov 03 01:41:04.889001 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:04.888979 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f9adfd4-8b5b-4a37-b625-9a8727482b7c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f9adfd4-8b5b-4a37-b625-9a8727482b7c-credential-socket\") pod \"3f9adfd4-8b5b-4a37-b625-9a8727482b7c\" (UID: \"3f9adfd4-8b5b-4a37-b625-9a8727482b7c\") " Nov 03 01:41:04.889073 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:04.889023 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3f9adfd4-8b5b-4a37-b625-9a8727482b7c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f9adfd4-8b5b-4a37-b625-9a8727482b7c-istio-podinfo\") pod \"3f9adfd4-8b5b-4a37-b625-9a8727482b7c\" (UID: \"3f9adfd4-8b5b-4a37-b625-9a8727482b7c\") " Nov 03 01:41:04.889146 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:04.889121 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f9adfd4-8b5b-4a37-b625-9a8727482b7c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f9adfd4-8b5b-4a37-b625-9a8727482b7c-istio-data\") pod \"3f9adfd4-8b5b-4a37-b625-9a8727482b7c\" (UID: \"3f9adfd4-8b5b-4a37-b625-9a8727482b7c\") " Nov 03 01:41:04.889207 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:04.889165 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f9adfd4-8b5b-4a37-b625-9a8727482b7c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f9adfd4-8b5b-4a37-b625-9a8727482b7c-istio-envoy\") pod \"3f9adfd4-8b5b-4a37-b625-9a8727482b7c\" (UID: \"3f9adfd4-8b5b-4a37-b625-9a8727482b7c\") " Nov 03 01:41:04.889265 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:04.889243 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f9adfd4-8b5b-4a37-b625-9a8727482b7c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f9adfd4-8b5b-4a37-b625-9a8727482b7c-workload-certs\") pod \"3f9adfd4-8b5b-4a37-b625-9a8727482b7c\" (UID: \"3f9adfd4-8b5b-4a37-b625-9a8727482b7c\") " Nov 03 01:41:04.889495 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:04.889475 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f9adfd4-8b5b-4a37-b625-9a8727482b7c-istio-data" pod "3f9adfd4-8b5b-4a37-b625-9a8727482b7c" (UID: "3f9adfd4-8b5b-4a37-b625-9a8727482b7c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:41:04.893337 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:04.890379 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f9adfd4-8b5b-4a37-b625-9a8727482b7c-workload-socket" pod "3f9adfd4-8b5b-4a37-b625-9a8727482b7c" (UID: "3f9adfd4-8b5b-4a37-b625-9a8727482b7c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:41:04.893337 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:04.890576 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f9adfd4-8b5b-4a37-b625-9a8727482b7c-istiod-ca-cert" pod "3f9adfd4-8b5b-4a37-b625-9a8727482b7c" (UID: "3f9adfd4-8b5b-4a37-b625-9a8727482b7c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:41:04.893337 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:04.891029 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f9adfd4-8b5b-4a37-b625-9a8727482b7c-workload-certs" pod "3f9adfd4-8b5b-4a37-b625-9a8727482b7c" (UID: "3f9adfd4-8b5b-4a37-b625-9a8727482b7c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:41:04.893337 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:04.891902 4421 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f9adfd4-8b5b-4a37-b625-9a8727482b7c-istio-data\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:04.893337 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:04.891922 4421 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f9adfd4-8b5b-4a37-b625-9a8727482b7c-workload-certs\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:04.893337 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:04.891937 4421 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f9adfd4-8b5b-4a37-b625-9a8727482b7c-workload-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:04.893337 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:04.891951 4421 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f9adfd4-8b5b-4a37-b625-9a8727482b7c-istiod-ca-cert\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:04.894399 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:04.894350 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f9adfd4-8b5b-4a37-b625-9a8727482b7c-credential-socket" pod "3f9adfd4-8b5b-4a37-b625-9a8727482b7c" (UID: "3f9adfd4-8b5b-4a37-b625-9a8727482b7c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:41:04.904740 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:04.904706 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3f9adfd4-8b5b-4a37-b625-9a8727482b7c-istio-podinfo" pod "3f9adfd4-8b5b-4a37-b625-9a8727482b7c" (UID: "3f9adfd4-8b5b-4a37-b625-9a8727482b7c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:41:04.941325 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:04.941240 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f9adfd4-8b5b-4a37-b625-9a8727482b7c-istio-envoy" pod "3f9adfd4-8b5b-4a37-b625-9a8727482b7c" (UID: "3f9adfd4-8b5b-4a37-b625-9a8727482b7c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:41:04.951298 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:04.950074 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-67sww" podStartSLOduration=1.950052054 podStartE2EDuration="1.950052054s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:41:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:41:04.927784009 +0000 UTC m=+3607.098787106" watchObservedRunningTime="2026-11-03 01:41:04.950052054 +0000 UTC m=+3607.121055149" Nov 03 01:41:04.957616 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:04.956820 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f9adfd4-8b5b-4a37-b625-9a8727482b7c-kube-api-access-5q5cx" pod "3f9adfd4-8b5b-4a37-b625-9a8727482b7c" (UID: "3f9adfd4-8b5b-4a37-b625-9a8727482b7c"). InnerVolumeSpecName "kube-api-access-5q5cx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:41:04.957616 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:04.956919 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f9adfd4-8b5b-4a37-b625-9a8727482b7c-istio-token" pod "3f9adfd4-8b5b-4a37-b625-9a8727482b7c" (UID: "3f9adfd4-8b5b-4a37-b625-9a8727482b7c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:41:04.959962 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:04.959935 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-f7680d/simulator-w2-f7680d-kserve-85bdc86cd6-8gxrg"] Nov 03 01:41:04.971435 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:04.971410 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-556c14/rate-test-model-556c14-kserve-5ccd9784cc-s8fq2" Nov 03 01:41:04.992998 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:04.992964 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/141cb342-025b-46eb-83b8-cbd2a60df0c0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/141cb342-025b-46eb-83b8-cbd2a60df0c0-model-cache\") pod \"141cb342-025b-46eb-83b8-cbd2a60df0c0\" (UID: \"141cb342-025b-46eb-83b8-cbd2a60df0c0\") " Nov 03 01:41:04.993174 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:04.993055 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/141cb342-025b-46eb-83b8-cbd2a60df0c0-home\" (UniqueName: \"kubernetes.io/empty-dir/141cb342-025b-46eb-83b8-cbd2a60df0c0-home\") pod \"141cb342-025b-46eb-83b8-cbd2a60df0c0\" (UID: \"141cb342-025b-46eb-83b8-cbd2a60df0c0\") " Nov 03 01:41:04.993174 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:04.993105 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/141cb342-025b-46eb-83b8-cbd2a60df0c0-kube-api-access-m48rr\" (UniqueName: \"kubernetes.io/projected/141cb342-025b-46eb-83b8-cbd2a60df0c0-kube-api-access-m48rr\") pod \"141cb342-025b-46eb-83b8-cbd2a60df0c0\" (UID: \"141cb342-025b-46eb-83b8-cbd2a60df0c0\") " Nov 03 01:41:04.993174 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:04.993157 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/141cb342-025b-46eb-83b8-cbd2a60df0c0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/141cb342-025b-46eb-83b8-cbd2a60df0c0-tmp-dir\") pod \"141cb342-025b-46eb-83b8-cbd2a60df0c0\" (UID: \"141cb342-025b-46eb-83b8-cbd2a60df0c0\") " Nov 03 01:41:04.993411 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:04.993255 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/141cb342-025b-46eb-83b8-cbd2a60df0c0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/141cb342-025b-46eb-83b8-cbd2a60df0c0-dshm\") pod \"141cb342-025b-46eb-83b8-cbd2a60df0c0\" (UID: \"141cb342-025b-46eb-83b8-cbd2a60df0c0\") " Nov 03 01:41:04.993411 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:04.993301 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/141cb342-025b-46eb-83b8-cbd2a60df0c0-model-cache" pod "141cb342-025b-46eb-83b8-cbd2a60df0c0" (UID: "141cb342-025b-46eb-83b8-cbd2a60df0c0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:41:04.993411 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:04.993381 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/141cb342-025b-46eb-83b8-cbd2a60df0c0-tls-certs\" (UniqueName: \"kubernetes.io/secret/141cb342-025b-46eb-83b8-cbd2a60df0c0-tls-certs\") pod \"141cb342-025b-46eb-83b8-cbd2a60df0c0\" (UID: \"141cb342-025b-46eb-83b8-cbd2a60df0c0\") " Nov 03 01:41:04.993411 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:04.993391 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/141cb342-025b-46eb-83b8-cbd2a60df0c0-home" pod "141cb342-025b-46eb-83b8-cbd2a60df0c0" (UID: "141cb342-025b-46eb-83b8-cbd2a60df0c0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:41:04.993639 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:04.993501 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/141cb342-025b-46eb-83b8-cbd2a60df0c0-tmp-dir" pod "141cb342-025b-46eb-83b8-cbd2a60df0c0" (UID: "141cb342-025b-46eb-83b8-cbd2a60df0c0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:41:04.993946 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:04.993926 4421 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f9adfd4-8b5b-4a37-b625-9a8727482b7c-istio-envoy\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:04.993946 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:04.993945 4421 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/141cb342-025b-46eb-83b8-cbd2a60df0c0-tmp-dir\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:04.994136 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:04.993955 4421 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f9adfd4-8b5b-4a37-b625-9a8727482b7c-istio-token\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:04.994136 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:04.993965 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5q5cx\" (UniqueName: \"kubernetes.io/projected/3f9adfd4-8b5b-4a37-b625-9a8727482b7c-kube-api-access-5q5cx\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:04.994136 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:04.993975 4421 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/141cb342-025b-46eb-83b8-cbd2a60df0c0-model-cache\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:04.994136 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:04.993984 4421 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f9adfd4-8b5b-4a37-b625-9a8727482b7c-credential-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:04.994136 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:04.993994 4421 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f9adfd4-8b5b-4a37-b625-9a8727482b7c-istio-podinfo\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:04.994136 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:04.994004 4421 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/141cb342-025b-46eb-83b8-cbd2a60df0c0-home\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:04.995767 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:04.995743 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/141cb342-025b-46eb-83b8-cbd2a60df0c0-dshm" pod "141cb342-025b-46eb-83b8-cbd2a60df0c0" (UID: "141cb342-025b-46eb-83b8-cbd2a60df0c0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:41:04.995873 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:04.995837 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/141cb342-025b-46eb-83b8-cbd2a60df0c0-kube-api-access-m48rr" pod "141cb342-025b-46eb-83b8-cbd2a60df0c0" (UID: "141cb342-025b-46eb-83b8-cbd2a60df0c0"). InnerVolumeSpecName "kube-api-access-m48rr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:41:04.997018 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:04.996988 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/141cb342-025b-46eb-83b8-cbd2a60df0c0-tls-certs" pod "141cb342-025b-46eb-83b8-cbd2a60df0c0" (UID: "141cb342-025b-46eb-83b8-cbd2a60df0c0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:41:05.095185 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:05.095099 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m48rr\" (UniqueName: \"kubernetes.io/projected/141cb342-025b-46eb-83b8-cbd2a60df0c0-kube-api-access-m48rr\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:05.095185 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:05.095131 4421 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/141cb342-025b-46eb-83b8-cbd2a60df0c0-dshm\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:05.095185 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:05.095141 4421 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/141cb342-025b-46eb-83b8-cbd2a60df0c0-tls-certs\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:05.589334 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:05.588508 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-67sww" Nov 03 01:41:05.591167 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:05.590955 4421 patch_prober.go:29] interesting pod/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-67sww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.125:15021/healthz/ready\": dial tcp 10.134.1.125:15021: connect: connection refused" start-of-body= Nov 03 01:41:05.591372 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:05.591253 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-67sww" podUID="35687f7e-87be-465b-ab66-ec518b459edc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.125:15021/healthz/ready\": dial tcp 10.134.1.125:15021: connect: connection refused" Nov 03 01:41:05.877094 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:05.877010 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-556c14/rate-test-model-556c14-kserve-5ccd9784cc-s8fq2" Nov 03 01:41:05.877094 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:05.877032 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-556c14/rate-test-model-556c14-kserve-5ccd9784cc-s8fq2" event={"ID":"141cb342-025b-46eb-83b8-cbd2a60df0c0","Type":"ContainerDied","Data":"6304b5fc34cb38cea1c3828049e655f48a53cfdfcddfd0dd6ff75597f1fe16cc"} Nov 03 01:41:05.877094 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:05.877084 4421 scope.go:122] "RemoveContainer" containerID="01092830cf39cda527bba975b19fae5025e0ad65e1f647a3dc67810494b8c5b7" Nov 03 01:41:05.877401 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:05.877268 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-d2znx" Nov 03 01:41:05.925301 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:05.905257 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-556c14/rate-test-model-556c14-kserve-5ccd9784cc-s8fq2"] Nov 03 01:41:05.934245 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:05.934207 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-556c14/rate-test-model-556c14-kserve-5ccd9784cc-s8fq2"] Nov 03 01:41:05.942483 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:05.942434 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-d2znx"] Nov 03 01:41:05.946124 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:05.946103 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-d2znx"] Nov 03 01:41:05.980528 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:05.980495 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="141cb342-025b-46eb-83b8-cbd2a60df0c0" path="/var/lib/kubelet/pods/141cb342-025b-46eb-83b8-cbd2a60df0c0/volumes" Nov 03 01:41:05.981120 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:05.981104 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f9adfd4-8b5b-4a37-b625-9a8727482b7c" path="/var/lib/kubelet/pods/3f9adfd4-8b5b-4a37-b625-9a8727482b7c/volumes" Nov 03 01:41:06.126299 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:06.117383 4421 scope.go:122] "RemoveContainer" containerID="426d3623a95d23577d85b059b270ae8c86282c77aa9586917f76a24c49ab9861" Nov 03 01:41:06.145730 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:06.145709 4421 scope.go:122] "RemoveContainer" containerID="b590451a28a979db1b8c7570c6d79145ac726ca815310f08d2cf9ef25a083b13" Nov 03 01:41:06.155700 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:06.155676 4421 scope.go:122] "RemoveContainer" containerID="ac321584e82282af33485c24ee8379fdb77edee07abd643508c5b845fd39082b" Nov 03 01:41:06.168163 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:06.168139 4421 scope.go:122] "RemoveContainer" containerID="0d6bf2d0d19470b6bda9530f344af79ea8233019d657ec96eb95074152b18bf4" Nov 03 01:41:06.586712 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:06.586672 4421 patch_prober.go:29] interesting pod/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-67sww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.125:15021/healthz/ready\": dial tcp 10.134.1.125:15021: connect: connection refused" start-of-body= Nov 03 01:41:06.587011 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:06.586734 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-67sww" podUID="35687f7e-87be-465b-ab66-ec518b459edc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.125:15021/healthz/ready\": dial tcp 10.134.1.125:15021: connect: connection refused" Nov 03 01:41:07.586465 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:07.586428 4421 patch_prober.go:29] interesting pod/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-67sww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.125:15021/healthz/ready\": dial tcp 10.134.1.125:15021: connect: connection refused" start-of-body= Nov 03 01:41:07.586894 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:07.586493 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-67sww" podUID="35687f7e-87be-465b-ab66-ec518b459edc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.125:15021/healthz/ready\": dial tcp 10.134.1.125:15021: connect: connection refused" Nov 03 01:41:08.586549 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:08.586514 4421 patch_prober.go:29] interesting pod/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-67sww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.125:15021/healthz/ready\": dial tcp 10.134.1.125:15021: connect: connection refused" start-of-body= Nov 03 01:41:08.588998 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:08.586567 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-67sww" podUID="35687f7e-87be-465b-ab66-ec518b459edc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.125:15021/healthz/ready\": dial tcp 10.134.1.125:15021: connect: connection refused" Nov 03 01:41:09.571594 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:09.571566 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-zxs8d" Nov 03 01:41:09.586681 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:09.586651 4421 patch_prober.go:29] interesting pod/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-67sww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.125:15021/healthz/ready\": dial tcp 10.134.1.125:15021: connect: connection refused" start-of-body= Nov 03 01:41:09.587147 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:09.587087 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-67sww" podUID="35687f7e-87be-465b-ab66-ec518b459edc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.125:15021/healthz/ready\": dial tcp 10.134.1.125:15021: connect: connection refused" Nov 03 01:41:09.700821 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:09.700786 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e5cb5e3-4c37-4ae7-9c6b-ef145cb87eb3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e5cb5e3-4c37-4ae7-9c6b-ef145cb87eb3-istiod-ca-cert\") pod \"6e5cb5e3-4c37-4ae7-9c6b-ef145cb87eb3\" (UID: \"6e5cb5e3-4c37-4ae7-9c6b-ef145cb87eb3\") " Nov 03 01:41:09.701025 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:09.700877 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e5cb5e3-4c37-4ae7-9c6b-ef145cb87eb3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e5cb5e3-4c37-4ae7-9c6b-ef145cb87eb3-istio-data\") pod \"6e5cb5e3-4c37-4ae7-9c6b-ef145cb87eb3\" (UID: \"6e5cb5e3-4c37-4ae7-9c6b-ef145cb87eb3\") " Nov 03 01:41:09.701025 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:09.700915 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e5cb5e3-4c37-4ae7-9c6b-ef145cb87eb3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e5cb5e3-4c37-4ae7-9c6b-ef145cb87eb3-credential-socket\") pod \"6e5cb5e3-4c37-4ae7-9c6b-ef145cb87eb3\" (UID: \"6e5cb5e3-4c37-4ae7-9c6b-ef145cb87eb3\") " Nov 03 01:41:09.701025 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:09.700957 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e5cb5e3-4c37-4ae7-9c6b-ef145cb87eb3-kube-api-access-xd59p\" (UniqueName: \"kubernetes.io/projected/6e5cb5e3-4c37-4ae7-9c6b-ef145cb87eb3-kube-api-access-xd59p\") pod \"6e5cb5e3-4c37-4ae7-9c6b-ef145cb87eb3\" (UID: \"6e5cb5e3-4c37-4ae7-9c6b-ef145cb87eb3\") " Nov 03 01:41:09.701025 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:09.700989 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e5cb5e3-4c37-4ae7-9c6b-ef145cb87eb3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e5cb5e3-4c37-4ae7-9c6b-ef145cb87eb3-istio-envoy\") pod \"6e5cb5e3-4c37-4ae7-9c6b-ef145cb87eb3\" (UID: \"6e5cb5e3-4c37-4ae7-9c6b-ef145cb87eb3\") " Nov 03 01:41:09.701025 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:09.701022 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e5cb5e3-4c37-4ae7-9c6b-ef145cb87eb3-istio-token\" (UniqueName: \"kubernetes.io/projected/6e5cb5e3-4c37-4ae7-9c6b-ef145cb87eb3-istio-token\") pod \"6e5cb5e3-4c37-4ae7-9c6b-ef145cb87eb3\" (UID: \"6e5cb5e3-4c37-4ae7-9c6b-ef145cb87eb3\") " Nov 03 01:41:09.701329 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:09.701214 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e5cb5e3-4c37-4ae7-9c6b-ef145cb87eb3-istiod-ca-cert" pod "6e5cb5e3-4c37-4ae7-9c6b-ef145cb87eb3" (UID: "6e5cb5e3-4c37-4ae7-9c6b-ef145cb87eb3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:41:09.701444 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:09.701416 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e5cb5e3-4c37-4ae7-9c6b-ef145cb87eb3-istio-data" pod "6e5cb5e3-4c37-4ae7-9c6b-ef145cb87eb3" (UID: "6e5cb5e3-4c37-4ae7-9c6b-ef145cb87eb3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:41:09.701654 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:09.701624 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e5cb5e3-4c37-4ae7-9c6b-ef145cb87eb3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e5cb5e3-4c37-4ae7-9c6b-ef145cb87eb3-workload-certs\") pod \"6e5cb5e3-4c37-4ae7-9c6b-ef145cb87eb3\" (UID: \"6e5cb5e3-4c37-4ae7-9c6b-ef145cb87eb3\") " Nov 03 01:41:09.701748 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:09.701727 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e5cb5e3-4c37-4ae7-9c6b-ef145cb87eb3-credential-socket" pod "6e5cb5e3-4c37-4ae7-9c6b-ef145cb87eb3" (UID: "6e5cb5e3-4c37-4ae7-9c6b-ef145cb87eb3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:41:09.701843 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:09.701826 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e5cb5e3-4c37-4ae7-9c6b-ef145cb87eb3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e5cb5e3-4c37-4ae7-9c6b-ef145cb87eb3-workload-socket\") pod \"6e5cb5e3-4c37-4ae7-9c6b-ef145cb87eb3\" (UID: \"6e5cb5e3-4c37-4ae7-9c6b-ef145cb87eb3\") " Nov 03 01:41:09.701892 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:09.701860 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6e5cb5e3-4c37-4ae7-9c6b-ef145cb87eb3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e5cb5e3-4c37-4ae7-9c6b-ef145cb87eb3-istio-podinfo\") pod \"6e5cb5e3-4c37-4ae7-9c6b-ef145cb87eb3\" (UID: \"6e5cb5e3-4c37-4ae7-9c6b-ef145cb87eb3\") " Nov 03 01:41:09.701966 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:09.701884 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e5cb5e3-4c37-4ae7-9c6b-ef145cb87eb3-workload-certs" pod "6e5cb5e3-4c37-4ae7-9c6b-ef145cb87eb3" (UID: "6e5cb5e3-4c37-4ae7-9c6b-ef145cb87eb3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:41:09.702131 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:09.702095 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e5cb5e3-4c37-4ae7-9c6b-ef145cb87eb3-workload-socket" pod "6e5cb5e3-4c37-4ae7-9c6b-ef145cb87eb3" (UID: "6e5cb5e3-4c37-4ae7-9c6b-ef145cb87eb3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:41:09.702691 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:09.702660 4421 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e5cb5e3-4c37-4ae7-9c6b-ef145cb87eb3-workload-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:09.702691 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:09.702692 4421 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e5cb5e3-4c37-4ae7-9c6b-ef145cb87eb3-istiod-ca-cert\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:09.702881 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:09.702707 4421 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e5cb5e3-4c37-4ae7-9c6b-ef145cb87eb3-istio-data\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:09.702881 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:09.702722 4421 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e5cb5e3-4c37-4ae7-9c6b-ef145cb87eb3-credential-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:09.702881 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:09.702736 4421 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e5cb5e3-4c37-4ae7-9c6b-ef145cb87eb3-workload-certs\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:09.703623 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:09.703599 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e5cb5e3-4c37-4ae7-9c6b-ef145cb87eb3-kube-api-access-xd59p" pod "6e5cb5e3-4c37-4ae7-9c6b-ef145cb87eb3" (UID: "6e5cb5e3-4c37-4ae7-9c6b-ef145cb87eb3"). InnerVolumeSpecName "kube-api-access-xd59p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:41:09.703799 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:09.703777 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e5cb5e3-4c37-4ae7-9c6b-ef145cb87eb3-istio-envoy" pod "6e5cb5e3-4c37-4ae7-9c6b-ef145cb87eb3" (UID: "6e5cb5e3-4c37-4ae7-9c6b-ef145cb87eb3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:41:09.703799 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:09.703790 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e5cb5e3-4c37-4ae7-9c6b-ef145cb87eb3-istio-token" pod "6e5cb5e3-4c37-4ae7-9c6b-ef145cb87eb3" (UID: "6e5cb5e3-4c37-4ae7-9c6b-ef145cb87eb3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:41:09.704324 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:09.704296 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6e5cb5e3-4c37-4ae7-9c6b-ef145cb87eb3-istio-podinfo" pod "6e5cb5e3-4c37-4ae7-9c6b-ef145cb87eb3" (UID: "6e5cb5e3-4c37-4ae7-9c6b-ef145cb87eb3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:41:09.804204 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:09.804110 4421 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e5cb5e3-4c37-4ae7-9c6b-ef145cb87eb3-istio-podinfo\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:09.804204 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:09.804143 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xd59p\" (UniqueName: \"kubernetes.io/projected/6e5cb5e3-4c37-4ae7-9c6b-ef145cb87eb3-kube-api-access-xd59p\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:09.804204 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:09.804156 4421 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e5cb5e3-4c37-4ae7-9c6b-ef145cb87eb3-istio-envoy\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:09.804204 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:09.804165 4421 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e5cb5e3-4c37-4ae7-9c6b-ef145cb87eb3-istio-token\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:09.935626 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:09.935591 4421 generic.go:356] "Generic (PLEG): container finished" podID="6e5cb5e3-4c37-4ae7-9c6b-ef145cb87eb3" containerID="00666c657a79b88b3f1f97e09cdcc20c75e999c4caa98f003675fa116d6d4a56" exitCode=0 Nov 03 01:41:09.935846 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:09.935643 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-zxs8d" event={"ID":"6e5cb5e3-4c37-4ae7-9c6b-ef145cb87eb3","Type":"ContainerDied","Data":"00666c657a79b88b3f1f97e09cdcc20c75e999c4caa98f003675fa116d6d4a56"} Nov 03 01:41:09.935846 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:09.935730 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-zxs8d" event={"ID":"6e5cb5e3-4c37-4ae7-9c6b-ef145cb87eb3","Type":"ContainerDied","Data":"a5d285de4bbcecc1e22a9e8d80eb805514374c126a9e96896163e4ea55db8f3a"} Nov 03 01:41:09.935846 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:09.935743 4421 scope.go:122] "RemoveContainer" containerID="00666c657a79b88b3f1f97e09cdcc20c75e999c4caa98f003675fa116d6d4a56" Nov 03 01:41:09.935846 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:09.935652 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-zxs8d" Nov 03 01:41:09.947491 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:09.947471 4421 scope.go:122] "RemoveContainer" containerID="00666c657a79b88b3f1f97e09cdcc20c75e999c4caa98f003675fa116d6d4a56" Nov 03 01:41:09.947771 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:41:09.947750 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00666c657a79b88b3f1f97e09cdcc20c75e999c4caa98f003675fa116d6d4a56\": container with ID starting with 00666c657a79b88b3f1f97e09cdcc20c75e999c4caa98f003675fa116d6d4a56 not found: ID does not exist" containerID="00666c657a79b88b3f1f97e09cdcc20c75e999c4caa98f003675fa116d6d4a56" Nov 03 01:41:09.947819 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:09.947780 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00666c657a79b88b3f1f97e09cdcc20c75e999c4caa98f003675fa116d6d4a56"} err="failed to get container status \"00666c657a79b88b3f1f97e09cdcc20c75e999c4caa98f003675fa116d6d4a56\": rpc error: code = NotFound desc = could not find container \"00666c657a79b88b3f1f97e09cdcc20c75e999c4caa98f003675fa116d6d4a56\": container with ID starting with 00666c657a79b88b3f1f97e09cdcc20c75e999c4caa98f003675fa116d6d4a56 not found: ID does not exist" Nov 03 01:41:09.961757 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:09.961715 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-zxs8d"] Nov 03 01:41:09.965608 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:09.965578 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-zxs8d"] Nov 03 01:41:09.982556 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:09.982518 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e5cb5e3-4c37-4ae7-9c6b-ef145cb87eb3" path="/var/lib/kubelet/pods/6e5cb5e3-4c37-4ae7-9c6b-ef145cb87eb3/volumes" Nov 03 01:41:10.740253 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:10.740220 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-67sww" Nov 03 01:41:10.741176 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:10.741149 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-67sww" Nov 03 01:41:10.742232 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:10.742209 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-67sww" Nov 03 01:41:10.999562 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:10.999531 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-e1f08d-f5fb4c64f-mmvp6"] Nov 03 01:41:11.005736 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:11.005699 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-e1f08d-597cb67847-6rjh6"] Nov 03 01:41:11.369347 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:11.368299 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f3bbab0-lnccm"] Nov 03 01:41:11.369548 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:11.369406 4421 cpu_manager.go:414] "containerMap: removing container" podUID="3f9adfd4-8b5b-4a37-b625-9a8727482b7c" containerName="istio-proxy" Nov 03 01:41:11.369548 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:11.369429 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f9adfd4-8b5b-4a37-b625-9a8727482b7c" containerName="istio-proxy" Nov 03 01:41:11.369548 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:11.369453 4421 cpu_manager.go:414] "containerMap: removing container" podUID="6dbc64a9-42b0-407c-b1d5-f2651acc523b" containerName="revoke-keys" Nov 03 01:41:11.369548 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:11.369461 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6dbc64a9-42b0-407c-b1d5-f2651acc523b" containerName="revoke-keys" Nov 03 01:41:11.369548 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:11.369475 4421 cpu_manager.go:414] "containerMap: removing container" podUID="141cb342-025b-46eb-83b8-cbd2a60df0c0" containerName="main" Nov 03 01:41:11.369548 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:11.369482 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="141cb342-025b-46eb-83b8-cbd2a60df0c0" containerName="main" Nov 03 01:41:11.369548 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:11.369519 4421 cpu_manager.go:414] "containerMap: removing container" podUID="6e5cb5e3-4c37-4ae7-9c6b-ef145cb87eb3" containerName="istio-proxy" Nov 03 01:41:11.369548 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:11.369527 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e5cb5e3-4c37-4ae7-9c6b-ef145cb87eb3" containerName="istio-proxy" Nov 03 01:41:11.369982 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:11.369654 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e5cb5e3-4c37-4ae7-9c6b-ef145cb87eb3" containerName="istio-proxy" Nov 03 01:41:11.369982 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:11.369672 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f9adfd4-8b5b-4a37-b625-9a8727482b7c" containerName="istio-proxy" Nov 03 01:41:11.369982 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:11.369683 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="6dbc64a9-42b0-407c-b1d5-f2651acc523b" containerName="revoke-keys" Nov 03 01:41:11.369982 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:11.369695 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="141cb342-025b-46eb-83b8-cbd2a60df0c0" containerName="main" Nov 03 01:41:11.377305 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:11.376347 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f3bbab0-lnccm" Nov 03 01:41:11.391633 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:11.391597 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6q4xp\"" Nov 03 01:41:11.391848 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:11.391681 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:41:11.439639 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:11.439547 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f3bbab0-lnccm"] Nov 03 01:41:11.455179 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:11.455145 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a18e15c-77bb-4660-ab81-e731b3a5938f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1f3bbab0-lnccm\" (UID: \"4a18e15c-77bb-4660-ab81-e731b3a5938f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f3bbab0-lnccm" Nov 03 01:41:11.455389 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:11.455196 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a18e15c-77bb-4660-ab81-e731b3a5938f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1f3bbab0-lnccm\" (UID: \"4a18e15c-77bb-4660-ab81-e731b3a5938f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f3bbab0-lnccm" Nov 03 01:41:11.455460 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:11.455403 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztqvs\" (UniqueName: \"kubernetes.io/projected/4a18e15c-77bb-4660-ab81-e731b3a5938f-kube-api-access-ztqvs\") pod \"maas-api-revoke-sub-1f3bbab0-lnccm\" (UID: \"4a18e15c-77bb-4660-ab81-e731b3a5938f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f3bbab0-lnccm" Nov 03 01:41:11.556320 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:11.556261 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a18e15c-77bb-4660-ab81-e731b3a5938f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1f3bbab0-lnccm\" (UID: \"4a18e15c-77bb-4660-ab81-e731b3a5938f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f3bbab0-lnccm" Nov 03 01:41:11.556643 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:11.556332 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a18e15c-77bb-4660-ab81-e731b3a5938f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1f3bbab0-lnccm\" (UID: \"4a18e15c-77bb-4660-ab81-e731b3a5938f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f3bbab0-lnccm" Nov 03 01:41:11.556643 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:11.556438 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztqvs\" (UniqueName: \"kubernetes.io/projected/4a18e15c-77bb-4660-ab81-e731b3a5938f-kube-api-access-ztqvs\") pod \"maas-api-revoke-sub-1f3bbab0-lnccm\" (UID: \"4a18e15c-77bb-4660-ab81-e731b3a5938f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f3bbab0-lnccm" Nov 03 01:41:11.557005 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:11.556978 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a18e15c-77bb-4660-ab81-e731b3a5938f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1f3bbab0-lnccm\" (UID: \"4a18e15c-77bb-4660-ab81-e731b3a5938f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f3bbab0-lnccm" Nov 03 01:41:11.558859 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:11.558835 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a18e15c-77bb-4660-ab81-e731b3a5938f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1f3bbab0-lnccm\" (UID: \"4a18e15c-77bb-4660-ab81-e731b3a5938f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f3bbab0-lnccm" Nov 03 01:41:11.565006 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:11.564978 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztqvs\" (UniqueName: \"kubernetes.io/projected/4a18e15c-77bb-4660-ab81-e731b3a5938f-kube-api-access-ztqvs\") pod \"maas-api-revoke-sub-1f3bbab0-lnccm\" (UID: \"4a18e15c-77bb-4660-ab81-e731b3a5938f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f3bbab0-lnccm" Nov 03 01:41:11.752919 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:11.752873 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f3bbab0-lnccm" Nov 03 01:41:12.034849 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:12.034817 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f3bbab0-lnccm"] Nov 03 01:41:12.035882 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:41:12.035839 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a18e15c_77bb_4660_ab81_e731b3a5938f.slice/crio-79d05509eccd7322a829ba37b436e2121b0617262ece6d12552d1e0697114514 WatchSource:0}: Error finding container 79d05509eccd7322a829ba37b436e2121b0617262ece6d12552d1e0697114514: Status 404 returned error can't find the container with id 79d05509eccd7322a829ba37b436e2121b0617262ece6d12552d1e0697114514 Nov 03 01:41:12.065727 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:12.065701 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-f7680d-59ddf48c45-8xgwl"] Nov 03 01:41:12.075173 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:12.075139 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-f7680d-67845fdfd7-r56nd"] Nov 03 01:41:12.960432 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:12.960331 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f3bbab0-lnccm" event={"ID":"4a18e15c-77bb-4660-ab81-e731b3a5938f","Type":"ContainerStarted","Data":"6fb8ee4b10e193db76ebcb9a1d1c21d27b583b5d9f95f7f1d6063d181b8f5019"} Nov 03 01:41:12.960432 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:12.960399 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f3bbab0-lnccm" event={"ID":"4a18e15c-77bb-4660-ab81-e731b3a5938f","Type":"ContainerStarted","Data":"79d05509eccd7322a829ba37b436e2121b0617262ece6d12552d1e0697114514"} Nov 03 01:41:12.984085 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:12.984024 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f3bbab0-lnccm" podStartSLOduration=1.984008312 podStartE2EDuration="1.984008312s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:41:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:41:12.979534879 +0000 UTC m=+3615.150537977" watchObservedRunningTime="2026-11-03 01:41:12.984008312 +0000 UTC m=+3615.155011408" Nov 03 01:41:13.189662 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:13.189624 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d25759-857574b47-bpmv4" podUID="d5576371-83ce-4e3e-b021-f0de09a49d6e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:41:13.976874 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:13.976837 4421 generic.go:356] "Generic (PLEG): container finished" podID="4a18e15c-77bb-4660-ab81-e731b3a5938f" containerID="6fb8ee4b10e193db76ebcb9a1d1c21d27b583b5d9f95f7f1d6063d181b8f5019" exitCode=0 Nov 03 01:41:14.004492 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:14.004447 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f3bbab0-lnccm" event={"ID":"4a18e15c-77bb-4660-ab81-e731b3a5938f","Type":"ContainerDied","Data":"6fb8ee4b10e193db76ebcb9a1d1c21d27b583b5d9f95f7f1d6063d181b8f5019"} Nov 03 01:41:14.672910 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:41:14.672868 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 01:41:15.217868 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:15.216640 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f3bbab0-lnccm" Nov 03 01:41:15.300556 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:15.300518 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4a18e15c-77bb-4660-ab81-e731b3a5938f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a18e15c-77bb-4660-ab81-e731b3a5938f-maas-api-xks-ca\") pod \"4a18e15c-77bb-4660-ab81-e731b3a5938f\" (UID: \"4a18e15c-77bb-4660-ab81-e731b3a5938f\") " Nov 03 01:41:15.300556 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:15.300559 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a18e15c-77bb-4660-ab81-e731b3a5938f-kube-api-access-ztqvs\" (UniqueName: \"kubernetes.io/projected/4a18e15c-77bb-4660-ab81-e731b3a5938f-kube-api-access-ztqvs\") pod \"4a18e15c-77bb-4660-ab81-e731b3a5938f\" (UID: \"4a18e15c-77bb-4660-ab81-e731b3a5938f\") " Nov 03 01:41:15.300870 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:15.300633 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a18e15c-77bb-4660-ab81-e731b3a5938f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a18e15c-77bb-4660-ab81-e731b3a5938f-maas-api-openshift-ca\") pod \"4a18e15c-77bb-4660-ab81-e731b3a5938f\" (UID: \"4a18e15c-77bb-4660-ab81-e731b3a5938f\") " Nov 03 01:41:15.301065 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:15.301036 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a18e15c-77bb-4660-ab81-e731b3a5938f-maas-api-openshift-ca" pod "4a18e15c-77bb-4660-ab81-e731b3a5938f" (UID: "4a18e15c-77bb-4660-ab81-e731b3a5938f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:41:15.301350 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:15.301332 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a18e15c-77bb-4660-ab81-e731b3a5938f-maas-api-openshift-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:15.302758 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:15.302729 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a18e15c-77bb-4660-ab81-e731b3a5938f-maas-api-xks-ca" pod "4a18e15c-77bb-4660-ab81-e731b3a5938f" (UID: "4a18e15c-77bb-4660-ab81-e731b3a5938f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:41:15.302850 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:15.302792 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a18e15c-77bb-4660-ab81-e731b3a5938f-kube-api-access-ztqvs" pod "4a18e15c-77bb-4660-ab81-e731b3a5938f" (UID: "4a18e15c-77bb-4660-ab81-e731b3a5938f"). InnerVolumeSpecName "kube-api-access-ztqvs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:41:15.403185 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:15.403149 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a18e15c-77bb-4660-ab81-e731b3a5938f-maas-api-xks-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:15.403185 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:15.403182 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ztqvs\" (UniqueName: \"kubernetes.io/projected/4a18e15c-77bb-4660-ab81-e731b3a5938f-kube-api-access-ztqvs\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:15.999699 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:15.999655 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f3bbab0-lnccm" event={"ID":"4a18e15c-77bb-4660-ab81-e731b3a5938f","Type":"ContainerDied","Data":"79d05509eccd7322a829ba37b436e2121b0617262ece6d12552d1e0697114514"} Nov 03 01:41:15.999884 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:15.999727 4421 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="79d05509eccd7322a829ba37b436e2121b0617262ece6d12552d1e0697114514" Nov 03 01:41:15.999884 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:15.999807 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f3bbab0-lnccm" Nov 03 01:41:16.002001 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:16.001962 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-e1f08d-f5fb4c64f-mmvp6" podUID="e60b0573-2ad3-40d9-898e-9c644f41dbc9" containerName="payload-processing" containerID="cri-o://4f2d454b26668b3c077e1a50e676853a63d03300af6ff64682ea5dd916a39fae" gracePeriod=25 Nov 03 01:41:16.008125 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:16.008069 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e1f08d-597cb67847-6rjh6" podUID="f4e62211-7bb6-42d2-8e73-c95d43c2737c" containerName="payload-pre-processing" containerID="cri-o://478ff80e0d5bd352e46475f00afacd38b6bd4378d7eb0aff9a2223fb40a44aa2" gracePeriod=25 Nov 03 01:41:17.066190 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:17.066119 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-f7680d-59ddf48c45-8xgwl" podUID="a5e0534c-5a31-4081-a049-5777d11925e1" containerName="payload-processing" containerID="cri-o://b31d098b3dd0a05021850986a7657443452ccc5d2d5863072ecd1b3cf0881c0d" gracePeriod=25 Nov 03 01:41:17.076213 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:17.076170 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-f7680d-67845fdfd7-r56nd" podUID="08dc3816-9d73-4dcf-abff-cef7f40b0009" containerName="payload-pre-processing" containerID="cri-o://ac6c24b89e688aaa135a985a9e843b8e06ba6aa6dc64ff236fd40f3ee8c4cdfb" gracePeriod=25 Nov 03 01:41:17.759531 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:17.759469 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-e1f08d-f5fb4c64f-mmvp6" podUID="e60b0573-2ad3-40d9-898e-9c644f41dbc9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:41:17.883302 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:17.860196 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-e1f08d-67989d5bdc-8k4qm"] Nov 03 01:41:17.899048 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:17.899000 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e1f08d-67989d5bdc-8k4qm" podUID="d748c793-a88a-43c7-ab53-1ff9242d53d3" containerName="maas-api" containerID="cri-o://fd49fb070443f3e2208b05f6999c758dde0ff220265133572835ec4b2e935692" gracePeriod=30 Nov 03 01:41:18.040707 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:18.040651 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-f7680d-67845fdfd7-r56nd" podUID="08dc3816-9d73-4dcf-abff-cef7f40b0009" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:41:18.041403 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:18.041365 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-f7680d-59ddf48c45-8xgwl" podUID="a5e0534c-5a31-4081-a049-5777d11925e1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:41:18.392194 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:18.392166 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e1f08d-67989d5bdc-8k4qm" Nov 03 01:41:18.486698 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:18.486653 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d748c793-a88a-43c7-ab53-1ff9242d53d3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d748c793-a88a-43c7-ab53-1ff9242d53d3-metrics-certs\") pod \"d748c793-a88a-43c7-ab53-1ff9242d53d3\" (UID: \"d748c793-a88a-43c7-ab53-1ff9242d53d3\") " Nov 03 01:41:18.486864 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:18.486786 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d748c793-a88a-43c7-ab53-1ff9242d53d3-kube-api-access-xxq8q\" (UniqueName: \"kubernetes.io/projected/d748c793-a88a-43c7-ab53-1ff9242d53d3-kube-api-access-xxq8q\") pod \"d748c793-a88a-43c7-ab53-1ff9242d53d3\" (UID: \"d748c793-a88a-43c7-ab53-1ff9242d53d3\") " Nov 03 01:41:18.486864 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:18.486813 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d748c793-a88a-43c7-ab53-1ff9242d53d3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d748c793-a88a-43c7-ab53-1ff9242d53d3-openshift-service-ca\") pod \"d748c793-a88a-43c7-ab53-1ff9242d53d3\" (UID: \"d748c793-a88a-43c7-ab53-1ff9242d53d3\") " Nov 03 01:41:18.487007 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:18.486946 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d748c793-a88a-43c7-ab53-1ff9242d53d3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d748c793-a88a-43c7-ab53-1ff9242d53d3-maas-api-tls\") pod \"d748c793-a88a-43c7-ab53-1ff9242d53d3\" (UID: \"d748c793-a88a-43c7-ab53-1ff9242d53d3\") " Nov 03 01:41:18.487340 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:18.487303 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d748c793-a88a-43c7-ab53-1ff9242d53d3-openshift-service-ca" pod "d748c793-a88a-43c7-ab53-1ff9242d53d3" (UID: "d748c793-a88a-43c7-ab53-1ff9242d53d3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:41:18.487845 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:18.487823 4421 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d748c793-a88a-43c7-ab53-1ff9242d53d3-openshift-service-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:18.489131 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:18.489110 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d748c793-a88a-43c7-ab53-1ff9242d53d3-maas-api-tls" pod "d748c793-a88a-43c7-ab53-1ff9242d53d3" (UID: "d748c793-a88a-43c7-ab53-1ff9242d53d3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:41:18.489242 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:18.489226 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d748c793-a88a-43c7-ab53-1ff9242d53d3-kube-api-access-xxq8q" pod "d748c793-a88a-43c7-ab53-1ff9242d53d3" (UID: "d748c793-a88a-43c7-ab53-1ff9242d53d3"). InnerVolumeSpecName "kube-api-access-xxq8q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:41:18.489436 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:18.489404 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d748c793-a88a-43c7-ab53-1ff9242d53d3-metrics-certs" pod "d748c793-a88a-43c7-ab53-1ff9242d53d3" (UID: "d748c793-a88a-43c7-ab53-1ff9242d53d3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:41:18.589026 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:18.588991 4421 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d748c793-a88a-43c7-ab53-1ff9242d53d3-metrics-certs\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:18.589026 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:18.589021 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxq8q\" (UniqueName: \"kubernetes.io/projected/d748c793-a88a-43c7-ab53-1ff9242d53d3-kube-api-access-xxq8q\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:18.589026 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:18.589032 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d748c793-a88a-43c7-ab53-1ff9242d53d3-maas-api-tls\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:19.027110 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:19.027076 4421 generic.go:356] "Generic (PLEG): container finished" podID="d748c793-a88a-43c7-ab53-1ff9242d53d3" containerID="fd49fb070443f3e2208b05f6999c758dde0ff220265133572835ec4b2e935692" exitCode=0 Nov 03 01:41:19.027377 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:19.027149 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e1f08d-67989d5bdc-8k4qm" Nov 03 01:41:19.027377 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:19.027172 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e1f08d-67989d5bdc-8k4qm" event={"ID":"d748c793-a88a-43c7-ab53-1ff9242d53d3","Type":"ContainerDied","Data":"fd49fb070443f3e2208b05f6999c758dde0ff220265133572835ec4b2e935692"} Nov 03 01:41:19.027377 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:19.027242 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e1f08d-67989d5bdc-8k4qm" event={"ID":"d748c793-a88a-43c7-ab53-1ff9242d53d3","Type":"ContainerDied","Data":"89addcab389e6a6abb0b5ed4b46dc53f21313014ec573933307d96d729e3d600"} Nov 03 01:41:19.027377 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:19.027312 4421 scope.go:122] "RemoveContainer" containerID="fd49fb070443f3e2208b05f6999c758dde0ff220265133572835ec4b2e935692" Nov 03 01:41:19.048843 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:19.048732 4421 scope.go:122] "RemoveContainer" containerID="fd49fb070443f3e2208b05f6999c758dde0ff220265133572835ec4b2e935692" Nov 03 01:41:19.051679 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:41:19.051639 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd49fb070443f3e2208b05f6999c758dde0ff220265133572835ec4b2e935692\": container with ID starting with fd49fb070443f3e2208b05f6999c758dde0ff220265133572835ec4b2e935692 not found: ID does not exist" containerID="fd49fb070443f3e2208b05f6999c758dde0ff220265133572835ec4b2e935692" Nov 03 01:41:19.051795 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:19.051690 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd49fb070443f3e2208b05f6999c758dde0ff220265133572835ec4b2e935692"} err="failed to get container status \"fd49fb070443f3e2208b05f6999c758dde0ff220265133572835ec4b2e935692\": rpc error: code = NotFound desc = could not find container \"fd49fb070443f3e2208b05f6999c758dde0ff220265133572835ec4b2e935692\": container with ID starting with fd49fb070443f3e2208b05f6999c758dde0ff220265133572835ec4b2e935692 not found: ID does not exist" Nov 03 01:41:19.076946 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:19.076912 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-e1f08d-67989d5bdc-8k4qm"] Nov 03 01:41:19.095998 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:19.095963 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-e1f08d-67989d5bdc-8k4qm"] Nov 03 01:41:19.661468 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:19.661421 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-dg98g"] Nov 03 01:41:19.661906 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:19.661827 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-dg98g" podUID="d7edd679-63c6-4949-b3e5-e4e09274d522" containerName="istio-proxy" containerID="cri-o://dc2c5fd1d391f1fc4ff87efd006b69a9cf60f5724b64cbd4ae1c64e60006617d" gracePeriod=30 Nov 03 01:41:19.982010 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:19.981916 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d748c793-a88a-43c7-ab53-1ff9242d53d3" path="/var/lib/kubelet/pods/d748c793-a88a-43c7-ab53-1ff9242d53d3/volumes" Nov 03 01:41:19.985093 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:19.985063 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-f7680d/simulator-w2-f7680d-kserve-85bdc86cd6-8gxrg" podUID="2bdcf521-cebd-4d6b-8554-1b60db8b0f3c" containerName="main" containerID="cri-o://e1d27bc4e9bea20398f7045149f5ec0e5093a5044e55cd165135ea66390f613e" gracePeriod=45 Nov 03 01:41:20.302496 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:20.302457 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-f7680d/simulator-w2-f7680d-kserve-85bdc86cd6-8gxrg" Nov 03 01:41:20.321167 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:20.321127 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bdcf521-cebd-4d6b-8554-1b60db8b0f3c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2bdcf521-cebd-4d6b-8554-1b60db8b0f3c-model-cache\") pod \"2bdcf521-cebd-4d6b-8554-1b60db8b0f3c\" (UID: \"2bdcf521-cebd-4d6b-8554-1b60db8b0f3c\") " Nov 03 01:41:20.321562 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:20.321207 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bdcf521-cebd-4d6b-8554-1b60db8b0f3c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2bdcf521-cebd-4d6b-8554-1b60db8b0f3c-dshm\") pod \"2bdcf521-cebd-4d6b-8554-1b60db8b0f3c\" (UID: \"2bdcf521-cebd-4d6b-8554-1b60db8b0f3c\") " Nov 03 01:41:20.321562 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:20.321486 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bdcf521-cebd-4d6b-8554-1b60db8b0f3c-home\" (UniqueName: \"kubernetes.io/empty-dir/2bdcf521-cebd-4d6b-8554-1b60db8b0f3c-home\") pod \"2bdcf521-cebd-4d6b-8554-1b60db8b0f3c\" (UID: \"2bdcf521-cebd-4d6b-8554-1b60db8b0f3c\") " Nov 03 01:41:20.321562 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:20.321484 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bdcf521-cebd-4d6b-8554-1b60db8b0f3c-model-cache" pod "2bdcf521-cebd-4d6b-8554-1b60db8b0f3c" (UID: "2bdcf521-cebd-4d6b-8554-1b60db8b0f3c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:41:20.322093 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:20.321588 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2bdcf521-cebd-4d6b-8554-1b60db8b0f3c-tls-certs\" (UniqueName: \"kubernetes.io/secret/2bdcf521-cebd-4d6b-8554-1b60db8b0f3c-tls-certs\") pod \"2bdcf521-cebd-4d6b-8554-1b60db8b0f3c\" (UID: \"2bdcf521-cebd-4d6b-8554-1b60db8b0f3c\") " Nov 03 01:41:20.322093 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:20.321663 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bdcf521-cebd-4d6b-8554-1b60db8b0f3c-kube-api-access-rr5fm\" (UniqueName: \"kubernetes.io/projected/2bdcf521-cebd-4d6b-8554-1b60db8b0f3c-kube-api-access-rr5fm\") pod \"2bdcf521-cebd-4d6b-8554-1b60db8b0f3c\" (UID: \"2bdcf521-cebd-4d6b-8554-1b60db8b0f3c\") " Nov 03 01:41:20.322093 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:20.321758 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bdcf521-cebd-4d6b-8554-1b60db8b0f3c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2bdcf521-cebd-4d6b-8554-1b60db8b0f3c-tmp-dir\") pod \"2bdcf521-cebd-4d6b-8554-1b60db8b0f3c\" (UID: \"2bdcf521-cebd-4d6b-8554-1b60db8b0f3c\") " Nov 03 01:41:20.322093 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:20.321918 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bdcf521-cebd-4d6b-8554-1b60db8b0f3c-home" pod "2bdcf521-cebd-4d6b-8554-1b60db8b0f3c" (UID: "2bdcf521-cebd-4d6b-8554-1b60db8b0f3c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:41:20.322386 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:20.322227 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bdcf521-cebd-4d6b-8554-1b60db8b0f3c-tmp-dir" pod "2bdcf521-cebd-4d6b-8554-1b60db8b0f3c" (UID: "2bdcf521-cebd-4d6b-8554-1b60db8b0f3c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:41:20.322975 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:20.322902 4421 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2bdcf521-cebd-4d6b-8554-1b60db8b0f3c-home\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:20.322975 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:20.322931 4421 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2bdcf521-cebd-4d6b-8554-1b60db8b0f3c-tmp-dir\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:20.322975 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:20.322944 4421 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2bdcf521-cebd-4d6b-8554-1b60db8b0f3c-model-cache\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:20.325672 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:20.325639 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bdcf521-cebd-4d6b-8554-1b60db8b0f3c-dshm" pod "2bdcf521-cebd-4d6b-8554-1b60db8b0f3c" (UID: "2bdcf521-cebd-4d6b-8554-1b60db8b0f3c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:41:20.326495 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:20.326454 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2bdcf521-cebd-4d6b-8554-1b60db8b0f3c-tls-certs" pod "2bdcf521-cebd-4d6b-8554-1b60db8b0f3c" (UID: "2bdcf521-cebd-4d6b-8554-1b60db8b0f3c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:41:20.327135 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:20.327110 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bdcf521-cebd-4d6b-8554-1b60db8b0f3c-kube-api-access-rr5fm" pod "2bdcf521-cebd-4d6b-8554-1b60db8b0f3c" (UID: "2bdcf521-cebd-4d6b-8554-1b60db8b0f3c"). InnerVolumeSpecName "kube-api-access-rr5fm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:41:20.424785 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:20.424748 4421 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2bdcf521-cebd-4d6b-8554-1b60db8b0f3c-dshm\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:20.424785 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:20.424788 4421 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2bdcf521-cebd-4d6b-8554-1b60db8b0f3c-tls-certs\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:20.425008 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:20.424804 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rr5fm\" (UniqueName: \"kubernetes.io/projected/2bdcf521-cebd-4d6b-8554-1b60db8b0f3c-kube-api-access-rr5fm\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:20.450989 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:20.450952 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b98b0ec-x74kl"] Nov 03 01:41:20.451619 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:20.451602 4421 cpu_manager.go:414] "containerMap: removing container" podUID="2bdcf521-cebd-4d6b-8554-1b60db8b0f3c" containerName="main" Nov 03 01:41:20.451672 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:20.451621 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bdcf521-cebd-4d6b-8554-1b60db8b0f3c" containerName="main" Nov 03 01:41:20.451672 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:20.451635 4421 cpu_manager.go:414] "containerMap: removing container" podUID="d748c793-a88a-43c7-ab53-1ff9242d53d3" containerName="maas-api" Nov 03 01:41:20.451672 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:20.451640 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d748c793-a88a-43c7-ab53-1ff9242d53d3" containerName="maas-api" Nov 03 01:41:20.451672 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:20.451669 4421 cpu_manager.go:414] "containerMap: removing container" podUID="4a18e15c-77bb-4660-ab81-e731b3a5938f" containerName="revoke-keys" Nov 03 01:41:20.451816 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:20.451675 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a18e15c-77bb-4660-ab81-e731b3a5938f" containerName="revoke-keys" Nov 03 01:41:20.451816 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:20.451769 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a18e15c-77bb-4660-ab81-e731b3a5938f" containerName="revoke-keys" Nov 03 01:41:20.451816 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:20.451782 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="2bdcf521-cebd-4d6b-8554-1b60db8b0f3c" containerName="main" Nov 03 01:41:20.451816 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:20.451790 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="d748c793-a88a-43c7-ab53-1ff9242d53d3" containerName="maas-api" Nov 03 01:41:20.455552 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:20.455528 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b98b0ec-x74kl" Nov 03 01:41:20.462015 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:20.461859 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 01:41:20.462015 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:20.461922 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:41:20.463024 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:20.462997 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b98b0ec-x74kl"] Nov 03 01:41:20.463240 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:20.463224 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6q4xp\"" Nov 03 01:41:20.463331 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:20.463301 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 01:41:20.525832 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:20.525734 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/126cb5d9-27c6-4f14-8612-45c82f71946c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7b98b0ec-x74kl\" (UID: \"126cb5d9-27c6-4f14-8612-45c82f71946c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b98b0ec-x74kl" Nov 03 01:41:20.526020 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:20.525900 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngdhs\" (UniqueName: \"kubernetes.io/projected/126cb5d9-27c6-4f14-8612-45c82f71946c-kube-api-access-ngdhs\") pod \"maas-api-revoke-sub-7b98b0ec-x74kl\" (UID: \"126cb5d9-27c6-4f14-8612-45c82f71946c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b98b0ec-x74kl" Nov 03 01:41:20.526091 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:20.526038 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/126cb5d9-27c6-4f14-8612-45c82f71946c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7b98b0ec-x74kl\" (UID: \"126cb5d9-27c6-4f14-8612-45c82f71946c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b98b0ec-x74kl" Nov 03 01:41:20.627888 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:20.627842 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/126cb5d9-27c6-4f14-8612-45c82f71946c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7b98b0ec-x74kl\" (UID: \"126cb5d9-27c6-4f14-8612-45c82f71946c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b98b0ec-x74kl" Nov 03 01:41:20.628078 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:20.627976 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngdhs\" (UniqueName: \"kubernetes.io/projected/126cb5d9-27c6-4f14-8612-45c82f71946c-kube-api-access-ngdhs\") pod \"maas-api-revoke-sub-7b98b0ec-x74kl\" (UID: \"126cb5d9-27c6-4f14-8612-45c82f71946c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b98b0ec-x74kl" Nov 03 01:41:20.628078 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:20.628029 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/126cb5d9-27c6-4f14-8612-45c82f71946c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7b98b0ec-x74kl\" (UID: \"126cb5d9-27c6-4f14-8612-45c82f71946c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b98b0ec-x74kl" Nov 03 01:41:20.628894 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:20.628864 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/126cb5d9-27c6-4f14-8612-45c82f71946c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7b98b0ec-x74kl\" (UID: \"126cb5d9-27c6-4f14-8612-45c82f71946c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b98b0ec-x74kl" Nov 03 01:41:20.630309 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:20.630285 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/126cb5d9-27c6-4f14-8612-45c82f71946c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7b98b0ec-x74kl\" (UID: \"126cb5d9-27c6-4f14-8612-45c82f71946c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b98b0ec-x74kl" Nov 03 01:41:20.636193 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:20.636169 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngdhs\" (UniqueName: \"kubernetes.io/projected/126cb5d9-27c6-4f14-8612-45c82f71946c-kube-api-access-ngdhs\") pod \"maas-api-revoke-sub-7b98b0ec-x74kl\" (UID: \"126cb5d9-27c6-4f14-8612-45c82f71946c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b98b0ec-x74kl" Nov 03 01:41:20.792348 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:20.782513 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b98b0ec-x74kl" Nov 03 01:41:21.044909 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:21.044819 4421 generic.go:356] "Generic (PLEG): container finished" podID="2bdcf521-cebd-4d6b-8554-1b60db8b0f3c" containerID="e1d27bc4e9bea20398f7045149f5ec0e5093a5044e55cd165135ea66390f613e" exitCode=0 Nov 03 01:41:21.044909 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:21.044888 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-f7680d/simulator-w2-f7680d-kserve-85bdc86cd6-8gxrg" Nov 03 01:41:21.045153 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:21.044910 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-f7680d/simulator-w2-f7680d-kserve-85bdc86cd6-8gxrg" event={"ID":"2bdcf521-cebd-4d6b-8554-1b60db8b0f3c","Type":"ContainerDied","Data":"e1d27bc4e9bea20398f7045149f5ec0e5093a5044e55cd165135ea66390f613e"} Nov 03 01:41:21.045153 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:21.044962 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-f7680d/simulator-w2-f7680d-kserve-85bdc86cd6-8gxrg" event={"ID":"2bdcf521-cebd-4d6b-8554-1b60db8b0f3c","Type":"ContainerDied","Data":"7488f92228758641ac38f77b2498265f2ddddf5000f83ef4b196bf6d52af02cd"} Nov 03 01:41:21.045153 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:21.044990 4421 scope.go:122] "RemoveContainer" containerID="e1d27bc4e9bea20398f7045149f5ec0e5093a5044e55cd165135ea66390f613e" Nov 03 01:41:21.070285 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:21.070198 4421 scope.go:122] "RemoveContainer" containerID="e1d27bc4e9bea20398f7045149f5ec0e5093a5044e55cd165135ea66390f613e" Nov 03 01:41:21.070627 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:41:21.070598 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1d27bc4e9bea20398f7045149f5ec0e5093a5044e55cd165135ea66390f613e\": container with ID starting with e1d27bc4e9bea20398f7045149f5ec0e5093a5044e55cd165135ea66390f613e not found: ID does not exist" containerID="e1d27bc4e9bea20398f7045149f5ec0e5093a5044e55cd165135ea66390f613e" Nov 03 01:41:21.070776 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:21.070647 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1d27bc4e9bea20398f7045149f5ec0e5093a5044e55cd165135ea66390f613e"} err="failed to get container status \"e1d27bc4e9bea20398f7045149f5ec0e5093a5044e55cd165135ea66390f613e\": rpc error: code = NotFound desc = could not find container \"e1d27bc4e9bea20398f7045149f5ec0e5093a5044e55cd165135ea66390f613e\": container with ID starting with e1d27bc4e9bea20398f7045149f5ec0e5093a5044e55cd165135ea66390f613e not found: ID does not exist" Nov 03 01:41:21.071414 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:21.071393 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-f7680d/simulator-w2-f7680d-kserve-85bdc86cd6-8gxrg"] Nov 03 01:41:21.075196 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:21.075173 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-f7680d/simulator-w2-f7680d-kserve-85bdc86cd6-8gxrg"] Nov 03 01:41:21.157142 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:21.157111 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b98b0ec-x74kl"] Nov 03 01:41:21.297916 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:21.297844 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d25759-857574b47-bpmv4" Nov 03 01:41:21.361980 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:21.361944 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5576371-83ce-4e3e-b021-f0de09a49d6e-kube-api-access-6g8z9\" (UniqueName: \"kubernetes.io/projected/d5576371-83ce-4e3e-b021-f0de09a49d6e-kube-api-access-6g8z9\") pod \"d5576371-83ce-4e3e-b021-f0de09a49d6e\" (UID: \"d5576371-83ce-4e3e-b021-f0de09a49d6e\") " Nov 03 01:41:21.362198 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:21.362038 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5576371-83ce-4e3e-b021-f0de09a49d6e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5576371-83ce-4e3e-b021-f0de09a49d6e-plugins-config-volume\") pod \"d5576371-83ce-4e3e-b021-f0de09a49d6e\" (UID: \"d5576371-83ce-4e3e-b021-f0de09a49d6e\") " Nov 03 01:41:21.362547 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:21.362517 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5576371-83ce-4e3e-b021-f0de09a49d6e-plugins-config-volume" pod "d5576371-83ce-4e3e-b021-f0de09a49d6e" (UID: "d5576371-83ce-4e3e-b021-f0de09a49d6e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:41:21.362714 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:21.362696 4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5576371-83ce-4e3e-b021-f0de09a49d6e-plugins-config-volume\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:21.364329 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:21.364305 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5576371-83ce-4e3e-b021-f0de09a49d6e-kube-api-access-6g8z9" pod "d5576371-83ce-4e3e-b021-f0de09a49d6e" (UID: "d5576371-83ce-4e3e-b021-f0de09a49d6e"). InnerVolumeSpecName "kube-api-access-6g8z9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:41:21.464368 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:21.464315 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6g8z9\" (UniqueName: \"kubernetes.io/projected/d5576371-83ce-4e3e-b021-f0de09a49d6e-kube-api-access-6g8z9\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:21.981365 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:21.981326 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2bdcf521-cebd-4d6b-8554-1b60db8b0f3c" path="/var/lib/kubelet/pods/2bdcf521-cebd-4d6b-8554-1b60db8b0f3c/volumes" Nov 03 01:41:22.054641 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:22.054611 4421 generic.go:356] "Generic (PLEG): container finished" podID="d5576371-83ce-4e3e-b021-f0de09a49d6e" containerID="28f81a1d05bfcf330a3f478d3b32e252f1a8ecc18e619bef8e09c3c8c3c7e3dd" exitCode=0 Nov 03 01:41:22.054822 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:22.054665 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d25759-857574b47-bpmv4" Nov 03 01:41:22.054822 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:22.054697 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d25759-857574b47-bpmv4" event={"ID":"d5576371-83ce-4e3e-b021-f0de09a49d6e","Type":"ContainerDied","Data":"28f81a1d05bfcf330a3f478d3b32e252f1a8ecc18e619bef8e09c3c8c3c7e3dd"} Nov 03 01:41:22.054822 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:22.054756 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d25759-857574b47-bpmv4" event={"ID":"d5576371-83ce-4e3e-b021-f0de09a49d6e","Type":"ContainerDied","Data":"7be4139f8a63280d178cd0ade1485046ab552e0c907f32cf192f65243b723054"} Nov 03 01:41:22.054822 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:22.054772 4421 scope.go:122] "RemoveContainer" containerID="28f81a1d05bfcf330a3f478d3b32e252f1a8ecc18e619bef8e09c3c8c3c7e3dd" Nov 03 01:41:22.056790 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:22.056766 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b98b0ec-x74kl" event={"ID":"126cb5d9-27c6-4f14-8612-45c82f71946c","Type":"ContainerStarted","Data":"f3b13ada8b5b3a8795ee0297827dfcdefa4aaf8317dc1a3f54d0e2d7fb8b88a1"} Nov 03 01:41:22.056919 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:22.056811 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b98b0ec-x74kl" event={"ID":"126cb5d9-27c6-4f14-8612-45c82f71946c","Type":"ContainerStarted","Data":"b76a470d26b94c2e1dc8ae5731c3aec6971b90fa523963d3fbf742efb7172f8d"} Nov 03 01:41:22.066609 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:22.066585 4421 scope.go:122] "RemoveContainer" containerID="28f81a1d05bfcf330a3f478d3b32e252f1a8ecc18e619bef8e09c3c8c3c7e3dd" Nov 03 01:41:22.066911 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:41:22.066886 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28f81a1d05bfcf330a3f478d3b32e252f1a8ecc18e619bef8e09c3c8c3c7e3dd\": container with ID starting with 28f81a1d05bfcf330a3f478d3b32e252f1a8ecc18e619bef8e09c3c8c3c7e3dd not found: ID does not exist" containerID="28f81a1d05bfcf330a3f478d3b32e252f1a8ecc18e619bef8e09c3c8c3c7e3dd" Nov 03 01:41:22.066995 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:22.066922 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28f81a1d05bfcf330a3f478d3b32e252f1a8ecc18e619bef8e09c3c8c3c7e3dd"} err="failed to get container status \"28f81a1d05bfcf330a3f478d3b32e252f1a8ecc18e619bef8e09c3c8c3c7e3dd\": rpc error: code = NotFound desc = could not find container \"28f81a1d05bfcf330a3f478d3b32e252f1a8ecc18e619bef8e09c3c8c3c7e3dd\": container with ID starting with 28f81a1d05bfcf330a3f478d3b32e252f1a8ecc18e619bef8e09c3c8c3c7e3dd not found: ID does not exist" Nov 03 01:41:22.074761 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:22.074733 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-d25759-857574b47-bpmv4"] Nov 03 01:41:22.078466 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:22.078444 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-d25759-857574b47-bpmv4"] Nov 03 01:41:22.090014 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:22.089958 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b98b0ec-x74kl" podStartSLOduration=2.089941274 podStartE2EDuration="2.089941274s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:41:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:41:22.087396324 +0000 UTC m=+3624.258399422" watchObservedRunningTime="2026-11-03 01:41:22.089941274 +0000 UTC m=+3624.260944370" Nov 03 01:41:23.066501 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:23.066460 4421 generic.go:356] "Generic (PLEG): container finished" podID="126cb5d9-27c6-4f14-8612-45c82f71946c" containerID="f3b13ada8b5b3a8795ee0297827dfcdefa4aaf8317dc1a3f54d0e2d7fb8b88a1" exitCode=0 Nov 03 01:41:23.066980 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:23.066542 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b98b0ec-x74kl" event={"ID":"126cb5d9-27c6-4f14-8612-45c82f71946c","Type":"ContainerDied","Data":"f3b13ada8b5b3a8795ee0297827dfcdefa4aaf8317dc1a3f54d0e2d7fb8b88a1"} Nov 03 01:41:23.981632 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:23.981593 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5576371-83ce-4e3e-b021-f0de09a49d6e" path="/var/lib/kubelet/pods/d5576371-83ce-4e3e-b021-f0de09a49d6e/volumes" Nov 03 01:41:24.272221 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:24.272188 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b98b0ec-x74kl" Nov 03 01:41:24.322555 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:24.322517 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/126cb5d9-27c6-4f14-8612-45c82f71946c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/126cb5d9-27c6-4f14-8612-45c82f71946c-maas-api-xks-ca\") pod \"126cb5d9-27c6-4f14-8612-45c82f71946c\" (UID: \"126cb5d9-27c6-4f14-8612-45c82f71946c\") " Nov 03 01:41:24.322730 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:24.322591 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/126cb5d9-27c6-4f14-8612-45c82f71946c-kube-api-access-ngdhs\" (UniqueName: \"kubernetes.io/projected/126cb5d9-27c6-4f14-8612-45c82f71946c-kube-api-access-ngdhs\") pod \"126cb5d9-27c6-4f14-8612-45c82f71946c\" (UID: \"126cb5d9-27c6-4f14-8612-45c82f71946c\") " Nov 03 01:41:24.322730 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:24.322654 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/126cb5d9-27c6-4f14-8612-45c82f71946c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/126cb5d9-27c6-4f14-8612-45c82f71946c-maas-api-openshift-ca\") pod \"126cb5d9-27c6-4f14-8612-45c82f71946c\" (UID: \"126cb5d9-27c6-4f14-8612-45c82f71946c\") " Nov 03 01:41:24.323117 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:24.323078 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/126cb5d9-27c6-4f14-8612-45c82f71946c-maas-api-openshift-ca" pod "126cb5d9-27c6-4f14-8612-45c82f71946c" (UID: "126cb5d9-27c6-4f14-8612-45c82f71946c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:41:24.323430 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:24.323405 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/126cb5d9-27c6-4f14-8612-45c82f71946c-maas-api-openshift-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:24.324985 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:24.324958 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/126cb5d9-27c6-4f14-8612-45c82f71946c-kube-api-access-ngdhs" pod "126cb5d9-27c6-4f14-8612-45c82f71946c" (UID: "126cb5d9-27c6-4f14-8612-45c82f71946c"). InnerVolumeSpecName "kube-api-access-ngdhs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:41:24.325217 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:24.325199 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/126cb5d9-27c6-4f14-8612-45c82f71946c-maas-api-xks-ca" pod "126cb5d9-27c6-4f14-8612-45c82f71946c" (UID: "126cb5d9-27c6-4f14-8612-45c82f71946c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:41:24.424574 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:24.424533 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/126cb5d9-27c6-4f14-8612-45c82f71946c-maas-api-xks-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:24.424574 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:24.424569 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ngdhs\" (UniqueName: \"kubernetes.io/projected/126cb5d9-27c6-4f14-8612-45c82f71946c-kube-api-access-ngdhs\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:24.504690 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:24.504653 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e1f08d-597cb67847-6rjh6" podUID="f4e62211-7bb6-42d2-8e73-c95d43c2737c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:41:24.980427 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:24.980398 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-dg98g" Nov 03 01:41:25.039588 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:25.039552 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7edd679-63c6-4949-b3e5-e4e09274d522-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7edd679-63c6-4949-b3e5-e4e09274d522-credential-socket\") pod \"d7edd679-63c6-4949-b3e5-e4e09274d522\" (UID: \"d7edd679-63c6-4949-b3e5-e4e09274d522\") " Nov 03 01:41:25.040672 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:25.039601 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7edd679-63c6-4949-b3e5-e4e09274d522-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7edd679-63c6-4949-b3e5-e4e09274d522-istio-data\") pod \"d7edd679-63c6-4949-b3e5-e4e09274d522\" (UID: \"d7edd679-63c6-4949-b3e5-e4e09274d522\") " Nov 03 01:41:25.040672 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:25.039669 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7edd679-63c6-4949-b3e5-e4e09274d522-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7edd679-63c6-4949-b3e5-e4e09274d522-istio-envoy\") pod \"d7edd679-63c6-4949-b3e5-e4e09274d522\" (UID: \"d7edd679-63c6-4949-b3e5-e4e09274d522\") " Nov 03 01:41:25.040672 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:25.039710 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7edd679-63c6-4949-b3e5-e4e09274d522-kube-api-access-jwkh7\" (UniqueName: \"kubernetes.io/projected/d7edd679-63c6-4949-b3e5-e4e09274d522-kube-api-access-jwkh7\") pod \"d7edd679-63c6-4949-b3e5-e4e09274d522\" (UID: \"d7edd679-63c6-4949-b3e5-e4e09274d522\") " Nov 03 01:41:25.040672 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:25.039770 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7edd679-63c6-4949-b3e5-e4e09274d522-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7edd679-63c6-4949-b3e5-e4e09274d522-istiod-ca-cert\") pod \"d7edd679-63c6-4949-b3e5-e4e09274d522\" (UID: \"d7edd679-63c6-4949-b3e5-e4e09274d522\") " Nov 03 01:41:25.040672 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:25.039805 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d7edd679-63c6-4949-b3e5-e4e09274d522-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7edd679-63c6-4949-b3e5-e4e09274d522-istio-podinfo\") pod \"d7edd679-63c6-4949-b3e5-e4e09274d522\" (UID: \"d7edd679-63c6-4949-b3e5-e4e09274d522\") " Nov 03 01:41:25.040672 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:25.039841 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7edd679-63c6-4949-b3e5-e4e09274d522-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7edd679-63c6-4949-b3e5-e4e09274d522-workload-socket\") pod \"d7edd679-63c6-4949-b3e5-e4e09274d522\" (UID: \"d7edd679-63c6-4949-b3e5-e4e09274d522\") " Nov 03 01:41:25.040672 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:25.039997 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7edd679-63c6-4949-b3e5-e4e09274d522-istio-token\" (UniqueName: \"kubernetes.io/projected/d7edd679-63c6-4949-b3e5-e4e09274d522-istio-token\") pod \"d7edd679-63c6-4949-b3e5-e4e09274d522\" (UID: \"d7edd679-63c6-4949-b3e5-e4e09274d522\") " Nov 03 01:41:25.040672 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:25.039995 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7edd679-63c6-4949-b3e5-e4e09274d522-credential-socket" pod "d7edd679-63c6-4949-b3e5-e4e09274d522" (UID: "d7edd679-63c6-4949-b3e5-e4e09274d522"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:41:25.040672 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:25.040039 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7edd679-63c6-4949-b3e5-e4e09274d522-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7edd679-63c6-4949-b3e5-e4e09274d522-workload-certs\") pod \"d7edd679-63c6-4949-b3e5-e4e09274d522\" (UID: \"d7edd679-63c6-4949-b3e5-e4e09274d522\") " Nov 03 01:41:25.040672 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:25.040488 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7edd679-63c6-4949-b3e5-e4e09274d522-istiod-ca-cert" pod "d7edd679-63c6-4949-b3e5-e4e09274d522" (UID: "d7edd679-63c6-4949-b3e5-e4e09274d522"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:41:25.040672 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:25.040571 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7edd679-63c6-4949-b3e5-e4e09274d522-workload-socket" pod "d7edd679-63c6-4949-b3e5-e4e09274d522" (UID: "d7edd679-63c6-4949-b3e5-e4e09274d522"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:41:25.041457 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:25.040583 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7edd679-63c6-4949-b3e5-e4e09274d522-istio-data" pod "d7edd679-63c6-4949-b3e5-e4e09274d522" (UID: "d7edd679-63c6-4949-b3e5-e4e09274d522"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:41:25.041457 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:25.040752 4421 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7edd679-63c6-4949-b3e5-e4e09274d522-credential-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:25.041457 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:25.040766 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7edd679-63c6-4949-b3e5-e4e09274d522-workload-certs" pod "d7edd679-63c6-4949-b3e5-e4e09274d522" (UID: "d7edd679-63c6-4949-b3e5-e4e09274d522"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:41:25.041457 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:25.040774 4421 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7edd679-63c6-4949-b3e5-e4e09274d522-istiod-ca-cert\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:25.041457 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:25.040799 4421 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7edd679-63c6-4949-b3e5-e4e09274d522-workload-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:25.043466 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:25.043401 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d7edd679-63c6-4949-b3e5-e4e09274d522-istio-podinfo" pod "d7edd679-63c6-4949-b3e5-e4e09274d522" (UID: "d7edd679-63c6-4949-b3e5-e4e09274d522"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:41:25.043592 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:25.043556 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7edd679-63c6-4949-b3e5-e4e09274d522-istio-envoy" pod "d7edd679-63c6-4949-b3e5-e4e09274d522" (UID: "d7edd679-63c6-4949-b3e5-e4e09274d522"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:41:25.043736 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:25.043704 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7edd679-63c6-4949-b3e5-e4e09274d522-kube-api-access-jwkh7" pod "d7edd679-63c6-4949-b3e5-e4e09274d522" (UID: "d7edd679-63c6-4949-b3e5-e4e09274d522"). InnerVolumeSpecName "kube-api-access-jwkh7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:41:25.043870 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:25.043846 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7edd679-63c6-4949-b3e5-e4e09274d522-istio-token" pod "d7edd679-63c6-4949-b3e5-e4e09274d522" (UID: "d7edd679-63c6-4949-b3e5-e4e09274d522"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:41:25.083560 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:25.083521 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b98b0ec-x74kl" Nov 03 01:41:25.083560 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:25.083554 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b98b0ec-x74kl" event={"ID":"126cb5d9-27c6-4f14-8612-45c82f71946c","Type":"ContainerDied","Data":"b76a470d26b94c2e1dc8ae5731c3aec6971b90fa523963d3fbf742efb7172f8d"} Nov 03 01:41:25.083793 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:25.083606 4421 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b76a470d26b94c2e1dc8ae5731c3aec6971b90fa523963d3fbf742efb7172f8d" Nov 03 01:41:25.085509 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:25.085479 4421 generic.go:356] "Generic (PLEG): container finished" podID="d7edd679-63c6-4949-b3e5-e4e09274d522" containerID="dc2c5fd1d391f1fc4ff87efd006b69a9cf60f5724b64cbd4ae1c64e60006617d" exitCode=0 Nov 03 01:41:25.085657 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:25.085520 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-dg98g" Nov 03 01:41:25.085657 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:25.085547 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-dg98g" event={"ID":"d7edd679-63c6-4949-b3e5-e4e09274d522","Type":"ContainerDied","Data":"dc2c5fd1d391f1fc4ff87efd006b69a9cf60f5724b64cbd4ae1c64e60006617d"} Nov 03 01:41:25.085657 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:25.085598 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-dg98g" event={"ID":"d7edd679-63c6-4949-b3e5-e4e09274d522","Type":"ContainerDied","Data":"892cd0cb29ffba4246bd60daf0780c582832cbc32d34f4d7c49297772f53f984"} Nov 03 01:41:25.085657 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:25.085626 4421 scope.go:122] "RemoveContainer" containerID="dc2c5fd1d391f1fc4ff87efd006b69a9cf60f5724b64cbd4ae1c64e60006617d" Nov 03 01:41:25.101376 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:25.101355 4421 scope.go:122] "RemoveContainer" containerID="3e48bdcc23749d41fff58606c10e327e7b4b81376eb8d44e28bb4d33eee1db46" Nov 03 01:41:25.117193 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:25.116450 4421 scope.go:122] "RemoveContainer" containerID="dc2c5fd1d391f1fc4ff87efd006b69a9cf60f5724b64cbd4ae1c64e60006617d" Nov 03 01:41:25.117193 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:41:25.116966 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc2c5fd1d391f1fc4ff87efd006b69a9cf60f5724b64cbd4ae1c64e60006617d\": container with ID starting with dc2c5fd1d391f1fc4ff87efd006b69a9cf60f5724b64cbd4ae1c64e60006617d not found: ID does not exist" containerID="dc2c5fd1d391f1fc4ff87efd006b69a9cf60f5724b64cbd4ae1c64e60006617d" Nov 03 01:41:25.117193 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:25.117002 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc2c5fd1d391f1fc4ff87efd006b69a9cf60f5724b64cbd4ae1c64e60006617d"} err="failed to get container status \"dc2c5fd1d391f1fc4ff87efd006b69a9cf60f5724b64cbd4ae1c64e60006617d\": rpc error: code = NotFound desc = could not find container \"dc2c5fd1d391f1fc4ff87efd006b69a9cf60f5724b64cbd4ae1c64e60006617d\": container with ID starting with dc2c5fd1d391f1fc4ff87efd006b69a9cf60f5724b64cbd4ae1c64e60006617d not found: ID does not exist" Nov 03 01:41:25.117193 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:25.117030 4421 scope.go:122] "RemoveContainer" containerID="3e48bdcc23749d41fff58606c10e327e7b4b81376eb8d44e28bb4d33eee1db46" Nov 03 01:41:25.117545 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:41:25.117456 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e48bdcc23749d41fff58606c10e327e7b4b81376eb8d44e28bb4d33eee1db46\": container with ID starting with 3e48bdcc23749d41fff58606c10e327e7b4b81376eb8d44e28bb4d33eee1db46 not found: ID does not exist" containerID="3e48bdcc23749d41fff58606c10e327e7b4b81376eb8d44e28bb4d33eee1db46" Nov 03 01:41:25.117545 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:25.117492 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e48bdcc23749d41fff58606c10e327e7b4b81376eb8d44e28bb4d33eee1db46"} err="failed to get container status \"3e48bdcc23749d41fff58606c10e327e7b4b81376eb8d44e28bb4d33eee1db46\": rpc error: code = NotFound desc = could not find container \"3e48bdcc23749d41fff58606c10e327e7b4b81376eb8d44e28bb4d33eee1db46\": container with ID starting with 3e48bdcc23749d41fff58606c10e327e7b4b81376eb8d44e28bb4d33eee1db46 not found: ID does not exist" Nov 03 01:41:25.119312 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:25.119287 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-dg98g"] Nov 03 01:41:25.123025 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:25.122999 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-dg98g"] Nov 03 01:41:25.141623 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:25.141591 4421 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7edd679-63c6-4949-b3e5-e4e09274d522-istio-token\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:25.141623 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:25.141620 4421 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7edd679-63c6-4949-b3e5-e4e09274d522-workload-certs\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:25.141785 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:25.141639 4421 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7edd679-63c6-4949-b3e5-e4e09274d522-istio-data\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:25.141785 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:25.141652 4421 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7edd679-63c6-4949-b3e5-e4e09274d522-istio-envoy\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:25.141785 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:25.141666 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwkh7\" (UniqueName: \"kubernetes.io/projected/d7edd679-63c6-4949-b3e5-e4e09274d522-kube-api-access-jwkh7\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:25.141785 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:25.141680 4421 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7edd679-63c6-4949-b3e5-e4e09274d522-istio-podinfo\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:25.637408 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:25.637373 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-717217-5f86cd4989-6685p"] Nov 03 01:41:25.662641 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:25.662606 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-717217-54d8c4bcf-v8j62"] Nov 03 01:41:25.980608 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:25.980523 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7edd679-63c6-4949-b3e5-e4e09274d522" path="/var/lib/kubelet/pods/d7edd679-63c6-4949-b3e5-e4e09274d522/volumes" Nov 03 01:41:27.742795 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:27.742758 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-e1f08d-f5fb4c64f-mmvp6" podUID="e60b0573-2ad3-40d9-898e-9c644f41dbc9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:41:28.039334 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:28.039262 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-f7680d-67845fdfd7-r56nd" podUID="08dc3816-9d73-4dcf-abff-cef7f40b0009" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:41:28.039752 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:28.039723 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-f7680d-59ddf48c45-8xgwl" podUID="a5e0534c-5a31-4081-a049-5777d11925e1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:41:28.854784 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:28.854748 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-gh5gs"] Nov 03 01:41:28.855374 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:28.855009 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-gh5gs" podUID="5c6e134c-a6fe-4794-9c80-b52ebb11abb2" containerName="istio-proxy" containerID="cri-o://16a01b50082bb8e9803e2536274a12f92a4e69e3e8742e9fc97aad2d2fbe60ed" gracePeriod=30 Nov 03 01:41:30.638395 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:30.638343 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-717217-5f86cd4989-6685p" podUID="f0f55109-ee26-4eab-b6ca-b610c783906d" containerName="payload-processing" containerID="cri-o://53f45c8b8b8508e4304cfb541ef35380dfdc7dcf9d19189c7b0173207a1fa436" gracePeriod=25 Nov 03 01:41:30.656959 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:30.656926 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5c3d8bc-hsgcz"] Nov 03 01:41:30.657592 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:30.657576 4421 cpu_manager.go:414] "containerMap: removing container" podUID="d7edd679-63c6-4949-b3e5-e4e09274d522" containerName="istio-proxy" Nov 03 01:41:30.657658 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:30.657593 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7edd679-63c6-4949-b3e5-e4e09274d522" containerName="istio-proxy" Nov 03 01:41:30.657658 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:30.657606 4421 cpu_manager.go:414] "containerMap: removing container" podUID="126cb5d9-27c6-4f14-8612-45c82f71946c" containerName="revoke-keys" Nov 03 01:41:30.657658 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:30.657611 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="126cb5d9-27c6-4f14-8612-45c82f71946c" containerName="revoke-keys" Nov 03 01:41:30.657658 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:30.657621 4421 cpu_manager.go:414] "containerMap: removing container" podUID="d5576371-83ce-4e3e-b021-f0de09a49d6e" containerName="payload-pre-processing" Nov 03 01:41:30.657658 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:30.657626 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5576371-83ce-4e3e-b021-f0de09a49d6e" containerName="payload-pre-processing" Nov 03 01:41:30.657658 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:30.657638 4421 cpu_manager.go:414] "containerMap: removing container" podUID="d7edd679-63c6-4949-b3e5-e4e09274d522" containerName="istio-proxy" Nov 03 01:41:30.657658 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:30.657643 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7edd679-63c6-4949-b3e5-e4e09274d522" containerName="istio-proxy" Nov 03 01:41:30.657893 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:30.657743 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7edd679-63c6-4949-b3e5-e4e09274d522" containerName="istio-proxy" Nov 03 01:41:30.657893 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:30.657753 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="126cb5d9-27c6-4f14-8612-45c82f71946c" containerName="revoke-keys" Nov 03 01:41:30.657893 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:30.657759 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5576371-83ce-4e3e-b021-f0de09a49d6e" containerName="payload-pre-processing" Nov 03 01:41:30.657893 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:30.657765 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7edd679-63c6-4949-b3e5-e4e09274d522" containerName="istio-proxy" Nov 03 01:41:30.662933 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:30.662913 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5c3d8bc-hsgcz" Nov 03 01:41:30.664050 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:30.663740 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-717217-54d8c4bcf-v8j62" podUID="38f1d8ba-2f55-4d82-b5e6-0491c36a8009" containerName="payload-pre-processing" containerID="cri-o://045a5208c7c98ef9a29aaebb16150c29be541504e6561b1ae71b8a056031c398" gracePeriod=25 Nov 03 01:41:30.668604 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:30.668579 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 01:41:30.669398 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:30.669377 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5c3d8bc-hsgcz"] Nov 03 01:41:30.669904 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:30.669884 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:41:30.669904 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:30.669877 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 01:41:30.669904 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:30.669911 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6q4xp\"" Nov 03 01:41:30.751284 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:30.751250 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d08194d0-b707-48e0-8a62-6c04c6cdc1e7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c5c3d8bc-hsgcz\" (UID: \"d08194d0-b707-48e0-8a62-6c04c6cdc1e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5c3d8bc-hsgcz" Nov 03 01:41:30.751465 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:30.751311 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d08194d0-b707-48e0-8a62-6c04c6cdc1e7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c5c3d8bc-hsgcz\" (UID: \"d08194d0-b707-48e0-8a62-6c04c6cdc1e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5c3d8bc-hsgcz" Nov 03 01:41:30.751610 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:30.751591 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6p2m\" (UniqueName: \"kubernetes.io/projected/d08194d0-b707-48e0-8a62-6c04c6cdc1e7-kube-api-access-g6p2m\") pod \"maas-api-revoke-sub-c5c3d8bc-hsgcz\" (UID: \"d08194d0-b707-48e0-8a62-6c04c6cdc1e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5c3d8bc-hsgcz" Nov 03 01:41:30.852867 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:30.852834 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6p2m\" (UniqueName: \"kubernetes.io/projected/d08194d0-b707-48e0-8a62-6c04c6cdc1e7-kube-api-access-g6p2m\") pod \"maas-api-revoke-sub-c5c3d8bc-hsgcz\" (UID: \"d08194d0-b707-48e0-8a62-6c04c6cdc1e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5c3d8bc-hsgcz" Nov 03 01:41:30.853034 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:30.852906 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d08194d0-b707-48e0-8a62-6c04c6cdc1e7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c5c3d8bc-hsgcz\" (UID: \"d08194d0-b707-48e0-8a62-6c04c6cdc1e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5c3d8bc-hsgcz" Nov 03 01:41:30.853034 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:30.852934 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d08194d0-b707-48e0-8a62-6c04c6cdc1e7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c5c3d8bc-hsgcz\" (UID: \"d08194d0-b707-48e0-8a62-6c04c6cdc1e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5c3d8bc-hsgcz" Nov 03 01:41:30.853751 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:30.853730 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d08194d0-b707-48e0-8a62-6c04c6cdc1e7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c5c3d8bc-hsgcz\" (UID: \"d08194d0-b707-48e0-8a62-6c04c6cdc1e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5c3d8bc-hsgcz" Nov 03 01:41:30.855420 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:30.855397 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d08194d0-b707-48e0-8a62-6c04c6cdc1e7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c5c3d8bc-hsgcz\" (UID: \"d08194d0-b707-48e0-8a62-6c04c6cdc1e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5c3d8bc-hsgcz" Nov 03 01:41:30.861676 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:30.861647 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6p2m\" (UniqueName: \"kubernetes.io/projected/d08194d0-b707-48e0-8a62-6c04c6cdc1e7-kube-api-access-g6p2m\") pod \"maas-api-revoke-sub-c5c3d8bc-hsgcz\" (UID: \"d08194d0-b707-48e0-8a62-6c04c6cdc1e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5c3d8bc-hsgcz" Nov 03 01:41:30.984827 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:30.984728 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5c3d8bc-hsgcz" Nov 03 01:41:31.270997 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:31.270959 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5c3d8bc-hsgcz"] Nov 03 01:41:31.272192 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:41:31.272164 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd08194d0_b707_48e0_8a62_6c04c6cdc1e7.slice/crio-e9fd52bd2610eb4ec81b5bc00a5c632525fa26f075ad99ded439641531bd4e04 WatchSource:0}: Error finding container e9fd52bd2610eb4ec81b5bc00a5c632525fa26f075ad99ded439641531bd4e04: Status 404 returned error can't find the container with id e9fd52bd2610eb4ec81b5bc00a5c632525fa26f075ad99ded439641531bd4e04 Nov 03 01:41:32.241086 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:32.241044 4421 generic.go:356] "Generic (PLEG): container finished" podID="d08194d0-b707-48e0-8a62-6c04c6cdc1e7" containerID="4665afafb4dfa9894c5f0af3f636e0a18b9cad983a6f0973af2e06a395fde60d" exitCode=0 Nov 03 01:41:32.241598 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:32.241130 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5c3d8bc-hsgcz" event={"ID":"d08194d0-b707-48e0-8a62-6c04c6cdc1e7","Type":"ContainerDied","Data":"4665afafb4dfa9894c5f0af3f636e0a18b9cad983a6f0973af2e06a395fde60d"} Nov 03 01:41:32.241598 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:32.241202 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5c3d8bc-hsgcz" event={"ID":"d08194d0-b707-48e0-8a62-6c04c6cdc1e7","Type":"ContainerStarted","Data":"e9fd52bd2610eb4ec81b5bc00a5c632525fa26f075ad99ded439641531bd4e04"} Nov 03 01:41:33.442593 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:33.442561 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5c3d8bc-hsgcz" Nov 03 01:41:33.504952 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:33.504870 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d08194d0-b707-48e0-8a62-6c04c6cdc1e7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d08194d0-b707-48e0-8a62-6c04c6cdc1e7-maas-api-xks-ca\") pod \"d08194d0-b707-48e0-8a62-6c04c6cdc1e7\" (UID: \"d08194d0-b707-48e0-8a62-6c04c6cdc1e7\") " Nov 03 01:41:33.504952 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:33.504941 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d08194d0-b707-48e0-8a62-6c04c6cdc1e7-kube-api-access-g6p2m\" (UniqueName: \"kubernetes.io/projected/d08194d0-b707-48e0-8a62-6c04c6cdc1e7-kube-api-access-g6p2m\") pod \"d08194d0-b707-48e0-8a62-6c04c6cdc1e7\" (UID: \"d08194d0-b707-48e0-8a62-6c04c6cdc1e7\") " Nov 03 01:41:33.505205 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:33.505053 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d08194d0-b707-48e0-8a62-6c04c6cdc1e7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d08194d0-b707-48e0-8a62-6c04c6cdc1e7-maas-api-openshift-ca\") pod \"d08194d0-b707-48e0-8a62-6c04c6cdc1e7\" (UID: \"d08194d0-b707-48e0-8a62-6c04c6cdc1e7\") " Nov 03 01:41:33.505767 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:33.505498 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d08194d0-b707-48e0-8a62-6c04c6cdc1e7-maas-api-openshift-ca" pod "d08194d0-b707-48e0-8a62-6c04c6cdc1e7" (UID: "d08194d0-b707-48e0-8a62-6c04c6cdc1e7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:41:33.505767 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:33.505583 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d08194d0-b707-48e0-8a62-6c04c6cdc1e7-maas-api-openshift-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:33.508442 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:33.508415 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d08194d0-b707-48e0-8a62-6c04c6cdc1e7-maas-api-xks-ca" pod "d08194d0-b707-48e0-8a62-6c04c6cdc1e7" (UID: "d08194d0-b707-48e0-8a62-6c04c6cdc1e7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:41:33.508558 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:33.508444 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d08194d0-b707-48e0-8a62-6c04c6cdc1e7-kube-api-access-g6p2m" pod "d08194d0-b707-48e0-8a62-6c04c6cdc1e7" (UID: "d08194d0-b707-48e0-8a62-6c04c6cdc1e7"). InnerVolumeSpecName "kube-api-access-g6p2m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:41:33.606441 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:33.606404 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d08194d0-b707-48e0-8a62-6c04c6cdc1e7-maas-api-xks-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:33.606441 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:33.606435 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g6p2m\" (UniqueName: \"kubernetes.io/projected/d08194d0-b707-48e0-8a62-6c04c6cdc1e7-kube-api-access-g6p2m\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:34.247379 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:34.247346 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-gh5gs" Nov 03 01:41:34.262230 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:34.262196 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5c3d8bc-hsgcz" Nov 03 01:41:34.266507 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:34.266460 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5c3d8bc-hsgcz" event={"ID":"d08194d0-b707-48e0-8a62-6c04c6cdc1e7","Type":"ContainerDied","Data":"e9fd52bd2610eb4ec81b5bc00a5c632525fa26f075ad99ded439641531bd4e04"} Nov 03 01:41:34.266723 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:34.266705 4421 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e9fd52bd2610eb4ec81b5bc00a5c632525fa26f075ad99ded439641531bd4e04" Nov 03 01:41:34.269910 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:34.269878 4421 generic.go:356] "Generic (PLEG): container finished" podID="5c6e134c-a6fe-4794-9c80-b52ebb11abb2" containerID="16a01b50082bb8e9803e2536274a12f92a4e69e3e8742e9fc97aad2d2fbe60ed" exitCode=0 Nov 03 01:41:34.270042 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:34.269928 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-gh5gs" event={"ID":"5c6e134c-a6fe-4794-9c80-b52ebb11abb2","Type":"ContainerDied","Data":"16a01b50082bb8e9803e2536274a12f92a4e69e3e8742e9fc97aad2d2fbe60ed"} Nov 03 01:41:34.270042 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:34.269959 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-gh5gs" event={"ID":"5c6e134c-a6fe-4794-9c80-b52ebb11abb2","Type":"ContainerDied","Data":"47d0f45950f762349d90ea749b03a069d22a14dbaf0312fdedade3f0f0168ad3"} Nov 03 01:41:34.270042 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:34.269979 4421 scope.go:122] "RemoveContainer" containerID="16a01b50082bb8e9803e2536274a12f92a4e69e3e8742e9fc97aad2d2fbe60ed" Nov 03 01:41:34.270231 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:34.270111 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-gh5gs" Nov 03 01:41:34.289573 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:34.289548 4421 scope.go:122] "RemoveContainer" containerID="16a01b50082bb8e9803e2536274a12f92a4e69e3e8742e9fc97aad2d2fbe60ed" Nov 03 01:41:34.289872 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:41:34.289855 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16a01b50082bb8e9803e2536274a12f92a4e69e3e8742e9fc97aad2d2fbe60ed\": container with ID starting with 16a01b50082bb8e9803e2536274a12f92a4e69e3e8742e9fc97aad2d2fbe60ed not found: ID does not exist" containerID="16a01b50082bb8e9803e2536274a12f92a4e69e3e8742e9fc97aad2d2fbe60ed" Nov 03 01:41:34.289967 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:34.289882 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16a01b50082bb8e9803e2536274a12f92a4e69e3e8742e9fc97aad2d2fbe60ed"} err="failed to get container status \"16a01b50082bb8e9803e2536274a12f92a4e69e3e8742e9fc97aad2d2fbe60ed\": rpc error: code = NotFound desc = could not find container \"16a01b50082bb8e9803e2536274a12f92a4e69e3e8742e9fc97aad2d2fbe60ed\": container with ID starting with 16a01b50082bb8e9803e2536274a12f92a4e69e3e8742e9fc97aad2d2fbe60ed not found: ID does not exist" Nov 03 01:41:34.319698 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:34.319665 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c6e134c-a6fe-4794-9c80-b52ebb11abb2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c6e134c-a6fe-4794-9c80-b52ebb11abb2-credential-socket\") pod \"5c6e134c-a6fe-4794-9c80-b52ebb11abb2\" (UID: \"5c6e134c-a6fe-4794-9c80-b52ebb11abb2\") " Nov 03 01:41:34.319841 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:34.319775 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c6e134c-a6fe-4794-9c80-b52ebb11abb2-istio-token\" (UniqueName: \"kubernetes.io/projected/5c6e134c-a6fe-4794-9c80-b52ebb11abb2-istio-token\") pod \"5c6e134c-a6fe-4794-9c80-b52ebb11abb2\" (UID: \"5c6e134c-a6fe-4794-9c80-b52ebb11abb2\") " Nov 03 01:41:34.319898 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:34.319843 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c6e134c-a6fe-4794-9c80-b52ebb11abb2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c6e134c-a6fe-4794-9c80-b52ebb11abb2-workload-certs\") pod \"5c6e134c-a6fe-4794-9c80-b52ebb11abb2\" (UID: \"5c6e134c-a6fe-4794-9c80-b52ebb11abb2\") " Nov 03 01:41:34.319898 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:34.319878 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c6e134c-a6fe-4794-9c80-b52ebb11abb2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c6e134c-a6fe-4794-9c80-b52ebb11abb2-istio-envoy\") pod \"5c6e134c-a6fe-4794-9c80-b52ebb11abb2\" (UID: \"5c6e134c-a6fe-4794-9c80-b52ebb11abb2\") " Nov 03 01:41:34.319968 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:34.319933 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c6e134c-a6fe-4794-9c80-b52ebb11abb2-kube-api-access-j48wr\" (UniqueName: \"kubernetes.io/projected/5c6e134c-a6fe-4794-9c80-b52ebb11abb2-kube-api-access-j48wr\") pod \"5c6e134c-a6fe-4794-9c80-b52ebb11abb2\" (UID: \"5c6e134c-a6fe-4794-9c80-b52ebb11abb2\") " Nov 03 01:41:34.320002 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:34.319967 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5c6e134c-a6fe-4794-9c80-b52ebb11abb2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c6e134c-a6fe-4794-9c80-b52ebb11abb2-istio-podinfo\") pod \"5c6e134c-a6fe-4794-9c80-b52ebb11abb2\" (UID: \"5c6e134c-a6fe-4794-9c80-b52ebb11abb2\") " Nov 03 01:41:34.320038 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:34.319999 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c6e134c-a6fe-4794-9c80-b52ebb11abb2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c6e134c-a6fe-4794-9c80-b52ebb11abb2-istio-data\") pod \"5c6e134c-a6fe-4794-9c80-b52ebb11abb2\" (UID: \"5c6e134c-a6fe-4794-9c80-b52ebb11abb2\") " Nov 03 01:41:34.320089 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:34.320055 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c6e134c-a6fe-4794-9c80-b52ebb11abb2-credential-socket" pod "5c6e134c-a6fe-4794-9c80-b52ebb11abb2" (UID: "5c6e134c-a6fe-4794-9c80-b52ebb11abb2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:41:34.320135 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:34.320092 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c6e134c-a6fe-4794-9c80-b52ebb11abb2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c6e134c-a6fe-4794-9c80-b52ebb11abb2-workload-socket\") pod \"5c6e134c-a6fe-4794-9c80-b52ebb11abb2\" (UID: \"5c6e134c-a6fe-4794-9c80-b52ebb11abb2\") " Nov 03 01:41:34.320179 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:34.320164 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c6e134c-a6fe-4794-9c80-b52ebb11abb2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c6e134c-a6fe-4794-9c80-b52ebb11abb2-istiod-ca-cert\") pod \"5c6e134c-a6fe-4794-9c80-b52ebb11abb2\" (UID: \"5c6e134c-a6fe-4794-9c80-b52ebb11abb2\") " Nov 03 01:41:34.320618 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:34.320474 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c6e134c-a6fe-4794-9c80-b52ebb11abb2-istio-data" pod "5c6e134c-a6fe-4794-9c80-b52ebb11abb2" (UID: "5c6e134c-a6fe-4794-9c80-b52ebb11abb2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:41:34.320618 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:34.320563 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c6e134c-a6fe-4794-9c80-b52ebb11abb2-workload-certs" pod "5c6e134c-a6fe-4794-9c80-b52ebb11abb2" (UID: "5c6e134c-a6fe-4794-9c80-b52ebb11abb2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:41:34.320796 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:34.320742 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c6e134c-a6fe-4794-9c80-b52ebb11abb2-workload-socket" pod "5c6e134c-a6fe-4794-9c80-b52ebb11abb2" (UID: "5c6e134c-a6fe-4794-9c80-b52ebb11abb2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:41:34.321337 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:34.320935 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c6e134c-a6fe-4794-9c80-b52ebb11abb2-istiod-ca-cert" pod "5c6e134c-a6fe-4794-9c80-b52ebb11abb2" (UID: "5c6e134c-a6fe-4794-9c80-b52ebb11abb2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:41:34.321337 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:34.321230 4421 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c6e134c-a6fe-4794-9c80-b52ebb11abb2-workload-certs\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:34.321337 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:34.321252 4421 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c6e134c-a6fe-4794-9c80-b52ebb11abb2-istio-data\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:34.321337 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:34.321266 4421 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c6e134c-a6fe-4794-9c80-b52ebb11abb2-workload-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:34.321337 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:34.321298 4421 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c6e134c-a6fe-4794-9c80-b52ebb11abb2-istiod-ca-cert\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:34.321337 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:34.321313 4421 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c6e134c-a6fe-4794-9c80-b52ebb11abb2-credential-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:34.323378 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:34.323335 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c6e134c-a6fe-4794-9c80-b52ebb11abb2-istio-envoy" pod "5c6e134c-a6fe-4794-9c80-b52ebb11abb2" (UID: "5c6e134c-a6fe-4794-9c80-b52ebb11abb2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:41:34.323759 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:34.323701 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5c6e134c-a6fe-4794-9c80-b52ebb11abb2-istio-podinfo" pod "5c6e134c-a6fe-4794-9c80-b52ebb11abb2" (UID: "5c6e134c-a6fe-4794-9c80-b52ebb11abb2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:41:34.323759 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:34.323704 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c6e134c-a6fe-4794-9c80-b52ebb11abb2-kube-api-access-j48wr" pod "5c6e134c-a6fe-4794-9c80-b52ebb11abb2" (UID: "5c6e134c-a6fe-4794-9c80-b52ebb11abb2"). InnerVolumeSpecName "kube-api-access-j48wr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:41:34.324191 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:34.324171 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c6e134c-a6fe-4794-9c80-b52ebb11abb2-istio-token" pod "5c6e134c-a6fe-4794-9c80-b52ebb11abb2" (UID: "5c6e134c-a6fe-4794-9c80-b52ebb11abb2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:41:34.422850 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:34.422814 4421 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c6e134c-a6fe-4794-9c80-b52ebb11abb2-istio-token\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:34.422850 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:34.422847 4421 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c6e134c-a6fe-4794-9c80-b52ebb11abb2-istio-envoy\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:34.423024 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:34.422857 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j48wr\" (UniqueName: \"kubernetes.io/projected/5c6e134c-a6fe-4794-9c80-b52ebb11abb2-kube-api-access-j48wr\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:34.423024 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:34.422868 4421 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c6e134c-a6fe-4794-9c80-b52ebb11abb2-istio-podinfo\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:34.504964 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:34.504925 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e1f08d-597cb67847-6rjh6" podUID="f4e62211-7bb6-42d2-8e73-c95d43c2737c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:41:34.595799 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:34.595752 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-gh5gs"] Nov 03 01:41:34.600860 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:34.600838 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-gh5gs"] Nov 03 01:41:35.980187 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:35.980147 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c6e134c-a6fe-4794-9c80-b52ebb11abb2" path="/var/lib/kubelet/pods/5c6e134c-a6fe-4794-9c80-b52ebb11abb2/volumes" Nov 03 01:41:36.311426 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:36.310629 4421 generic.go:356] "Generic (PLEG): container finished" podID="e60b0573-2ad3-40d9-898e-9c644f41dbc9" containerID="4f2d454b26668b3c077e1a50e676853a63d03300af6ff64682ea5dd916a39fae" exitCode=0 Nov 03 01:41:36.311426 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:36.310819 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-e1f08d-f5fb4c64f-mmvp6" event={"ID":"e60b0573-2ad3-40d9-898e-9c644f41dbc9","Type":"ContainerDied","Data":"4f2d454b26668b3c077e1a50e676853a63d03300af6ff64682ea5dd916a39fae"} Nov 03 01:41:36.312974 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:36.312945 4421 generic.go:356] "Generic (PLEG): container finished" podID="f4e62211-7bb6-42d2-8e73-c95d43c2737c" containerID="478ff80e0d5bd352e46475f00afacd38b6bd4378d7eb0aff9a2223fb40a44aa2" exitCode=0 Nov 03 01:41:36.313156 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:36.313133 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e1f08d-597cb67847-6rjh6" event={"ID":"f4e62211-7bb6-42d2-8e73-c95d43c2737c","Type":"ContainerDied","Data":"478ff80e0d5bd352e46475f00afacd38b6bd4378d7eb0aff9a2223fb40a44aa2"} Nov 03 01:41:36.388414 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:36.388375 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-e1f08d-f5fb4c64f-mmvp6" Nov 03 01:41:36.438782 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:36.438746 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e1f08d-597cb67847-6rjh6" Nov 03 01:41:36.513422 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:36.513385 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4e62211-7bb6-42d2-8e73-c95d43c2737c-kube-api-access-rbqcd\" (UniqueName: \"kubernetes.io/projected/f4e62211-7bb6-42d2-8e73-c95d43c2737c-kube-api-access-rbqcd\") pod \"f4e62211-7bb6-42d2-8e73-c95d43c2737c\" (UID: \"f4e62211-7bb6-42d2-8e73-c95d43c2737c\") " Nov 03 01:41:36.513591 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:36.513541 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e60b0573-2ad3-40d9-898e-9c644f41dbc9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e60b0573-2ad3-40d9-898e-9c644f41dbc9-plugins-config-volume\") pod \"e60b0573-2ad3-40d9-898e-9c644f41dbc9\" (UID: \"e60b0573-2ad3-40d9-898e-9c644f41dbc9\") " Nov 03 01:41:36.513669 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:36.513611 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e60b0573-2ad3-40d9-898e-9c644f41dbc9-kube-api-access-57js7\" (UniqueName: \"kubernetes.io/projected/e60b0573-2ad3-40d9-898e-9c644f41dbc9-kube-api-access-57js7\") pod \"e60b0573-2ad3-40d9-898e-9c644f41dbc9\" (UID: \"e60b0573-2ad3-40d9-898e-9c644f41dbc9\") " Nov 03 01:41:36.513669 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:36.513657 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4e62211-7bb6-42d2-8e73-c95d43c2737c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4e62211-7bb6-42d2-8e73-c95d43c2737c-plugins-config-volume\") pod \"f4e62211-7bb6-42d2-8e73-c95d43c2737c\" (UID: \"f4e62211-7bb6-42d2-8e73-c95d43c2737c\") " Nov 03 01:41:36.514195 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:36.514158 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e60b0573-2ad3-40d9-898e-9c644f41dbc9-plugins-config-volume" pod "e60b0573-2ad3-40d9-898e-9c644f41dbc9" (UID: "e60b0573-2ad3-40d9-898e-9c644f41dbc9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:41:36.514499 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:36.514464 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4e62211-7bb6-42d2-8e73-c95d43c2737c-plugins-config-volume" pod "f4e62211-7bb6-42d2-8e73-c95d43c2737c" (UID: "f4e62211-7bb6-42d2-8e73-c95d43c2737c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:41:36.515793 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:36.515761 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4e62211-7bb6-42d2-8e73-c95d43c2737c-kube-api-access-rbqcd" pod "f4e62211-7bb6-42d2-8e73-c95d43c2737c" (UID: "f4e62211-7bb6-42d2-8e73-c95d43c2737c"). InnerVolumeSpecName "kube-api-access-rbqcd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:41:36.515950 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:36.515934 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e60b0573-2ad3-40d9-898e-9c644f41dbc9-kube-api-access-57js7" pod "e60b0573-2ad3-40d9-898e-9c644f41dbc9" (UID: "e60b0573-2ad3-40d9-898e-9c644f41dbc9"). InnerVolumeSpecName "kube-api-access-57js7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:41:36.615145 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:36.615057 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57js7\" (UniqueName: \"kubernetes.io/projected/e60b0573-2ad3-40d9-898e-9c644f41dbc9-kube-api-access-57js7\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:36.615145 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:36.615090 4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4e62211-7bb6-42d2-8e73-c95d43c2737c-plugins-config-volume\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:36.615145 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:36.615104 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbqcd\" (UniqueName: \"kubernetes.io/projected/f4e62211-7bb6-42d2-8e73-c95d43c2737c-kube-api-access-rbqcd\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:36.615145 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:36.615114 4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e60b0573-2ad3-40d9-898e-9c644f41dbc9-plugins-config-volume\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:37.356315 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:37.353946 4421 generic.go:356] "Generic (PLEG): container finished" podID="a5e0534c-5a31-4081-a049-5777d11925e1" containerID="b31d098b3dd0a05021850986a7657443452ccc5d2d5863072ecd1b3cf0881c0d" exitCode=0 Nov 03 01:41:37.356315 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:37.354016 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-f7680d-59ddf48c45-8xgwl" event={"ID":"a5e0534c-5a31-4081-a049-5777d11925e1","Type":"ContainerDied","Data":"b31d098b3dd0a05021850986a7657443452ccc5d2d5863072ecd1b3cf0881c0d"} Nov 03 01:41:37.356315 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:37.355806 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-e1f08d-f5fb4c64f-mmvp6" event={"ID":"e60b0573-2ad3-40d9-898e-9c644f41dbc9","Type":"ContainerDied","Data":"10fdc578bbe9186f314c3b276a87a7bc0f70a128b08a8b765a3dd3f0816e7642"} Nov 03 01:41:37.356315 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:37.355821 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-e1f08d-f5fb4c64f-mmvp6" Nov 03 01:41:37.356315 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:37.355859 4421 scope.go:122] "RemoveContainer" containerID="4f2d454b26668b3c077e1a50e676853a63d03300af6ff64682ea5dd916a39fae" Nov 03 01:41:37.357745 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:37.357723 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e1f08d-597cb67847-6rjh6" Nov 03 01:41:37.357966 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:37.357949 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e1f08d-597cb67847-6rjh6" event={"ID":"f4e62211-7bb6-42d2-8e73-c95d43c2737c","Type":"ContainerDied","Data":"d82e62ce7f215da719e8e43425737a12336fab57ad8975f51d150ec74f76d586"} Nov 03 01:41:37.367319 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:37.359701 4421 generic.go:356] "Generic (PLEG): container finished" podID="08dc3816-9d73-4dcf-abff-cef7f40b0009" containerID="ac6c24b89e688aaa135a985a9e843b8e06ba6aa6dc64ff236fd40f3ee8c4cdfb" exitCode=0 Nov 03 01:41:37.367319 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:37.359760 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-f7680d-67845fdfd7-r56nd" event={"ID":"08dc3816-9d73-4dcf-abff-cef7f40b0009","Type":"ContainerDied","Data":"ac6c24b89e688aaa135a985a9e843b8e06ba6aa6dc64ff236fd40f3ee8c4cdfb"} Nov 03 01:41:37.374050 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:37.374027 4421 scope.go:122] "RemoveContainer" containerID="478ff80e0d5bd352e46475f00afacd38b6bd4378d7eb0aff9a2223fb40a44aa2" Nov 03 01:41:37.392235 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:37.391873 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-e1f08d-f5fb4c64f-mmvp6"] Nov 03 01:41:37.396637 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:37.396613 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-e1f08d-f5fb4c64f-mmvp6"] Nov 03 01:41:37.409748 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:37.409685 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-e1f08d-597cb67847-6rjh6"] Nov 03 01:41:37.415987 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:37.415955 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-e1f08d-597cb67847-6rjh6"] Nov 03 01:41:37.438003 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:37.437972 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-f7680d-67845fdfd7-r56nd" Nov 03 01:41:37.479159 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:37.479129 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-f7680d-59ddf48c45-8xgwl" Nov 03 01:41:37.543024 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:37.542988 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08dc3816-9d73-4dcf-abff-cef7f40b0009-kube-api-access-qhb8b\" (UniqueName: \"kubernetes.io/projected/08dc3816-9d73-4dcf-abff-cef7f40b0009-kube-api-access-qhb8b\") pod \"08dc3816-9d73-4dcf-abff-cef7f40b0009\" (UID: \"08dc3816-9d73-4dcf-abff-cef7f40b0009\") " Nov 03 01:41:37.543221 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:37.543182 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5e0534c-5a31-4081-a049-5777d11925e1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5e0534c-5a31-4081-a049-5777d11925e1-plugins-config-volume\") pod \"a5e0534c-5a31-4081-a049-5777d11925e1\" (UID: \"a5e0534c-5a31-4081-a049-5777d11925e1\") " Nov 03 01:41:37.543314 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:37.543224 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5e0534c-5a31-4081-a049-5777d11925e1-kube-api-access-z96wd\" (UniqueName: \"kubernetes.io/projected/a5e0534c-5a31-4081-a049-5777d11925e1-kube-api-access-z96wd\") pod \"a5e0534c-5a31-4081-a049-5777d11925e1\" (UID: \"a5e0534c-5a31-4081-a049-5777d11925e1\") " Nov 03 01:41:37.543314 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:37.543255 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08dc3816-9d73-4dcf-abff-cef7f40b0009-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08dc3816-9d73-4dcf-abff-cef7f40b0009-plugins-config-volume\") pod \"08dc3816-9d73-4dcf-abff-cef7f40b0009\" (UID: \"08dc3816-9d73-4dcf-abff-cef7f40b0009\") " Nov 03 01:41:37.544145 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:37.544114 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08dc3816-9d73-4dcf-abff-cef7f40b0009-plugins-config-volume" pod "08dc3816-9d73-4dcf-abff-cef7f40b0009" (UID: "08dc3816-9d73-4dcf-abff-cef7f40b0009"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:41:37.544268 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:37.544183 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5e0534c-5a31-4081-a049-5777d11925e1-plugins-config-volume" pod "a5e0534c-5a31-4081-a049-5777d11925e1" (UID: "a5e0534c-5a31-4081-a049-5777d11925e1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:41:37.545965 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:37.545943 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08dc3816-9d73-4dcf-abff-cef7f40b0009-kube-api-access-qhb8b" pod "08dc3816-9d73-4dcf-abff-cef7f40b0009" (UID: "08dc3816-9d73-4dcf-abff-cef7f40b0009"). InnerVolumeSpecName "kube-api-access-qhb8b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:41:37.546062 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:37.545976 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5e0534c-5a31-4081-a049-5777d11925e1-kube-api-access-z96wd" pod "a5e0534c-5a31-4081-a049-5777d11925e1" (UID: "a5e0534c-5a31-4081-a049-5777d11925e1"). InnerVolumeSpecName "kube-api-access-z96wd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:41:37.645295 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:37.645205 4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5e0534c-5a31-4081-a049-5777d11925e1-plugins-config-volume\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:37.645295 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:37.645232 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z96wd\" (UniqueName: \"kubernetes.io/projected/a5e0534c-5a31-4081-a049-5777d11925e1-kube-api-access-z96wd\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:37.645295 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:37.645242 4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08dc3816-9d73-4dcf-abff-cef7f40b0009-plugins-config-volume\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:37.645295 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:37.645252 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhb8b\" (UniqueName: \"kubernetes.io/projected/08dc3816-9d73-4dcf-abff-cef7f40b0009-kube-api-access-qhb8b\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:37.981632 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:37.981551 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e60b0573-2ad3-40d9-898e-9c644f41dbc9" path="/var/lib/kubelet/pods/e60b0573-2ad3-40d9-898e-9c644f41dbc9/volumes" Nov 03 01:41:37.982021 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:37.982006 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4e62211-7bb6-42d2-8e73-c95d43c2737c" path="/var/lib/kubelet/pods/f4e62211-7bb6-42d2-8e73-c95d43c2737c/volumes" Nov 03 01:41:38.382420 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:38.382382 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-f7680d-67845fdfd7-r56nd" event={"ID":"08dc3816-9d73-4dcf-abff-cef7f40b0009","Type":"ContainerDied","Data":"2846f83e2ef2deb7fc8dcd8b5409f8391345a56e6dbb64918be1d5e3f06e13cd"} Nov 03 01:41:38.383288 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:38.383027 4421 scope.go:122] "RemoveContainer" containerID="ac6c24b89e688aaa135a985a9e843b8e06ba6aa6dc64ff236fd40f3ee8c4cdfb" Nov 03 01:41:38.383288 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:38.383162 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-f7680d-67845fdfd7-r56nd" Nov 03 01:41:38.388435 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:38.388411 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-f7680d-59ddf48c45-8xgwl" Nov 03 01:41:38.391287 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:38.391252 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-f7680d-59ddf48c45-8xgwl" event={"ID":"a5e0534c-5a31-4081-a049-5777d11925e1","Type":"ContainerDied","Data":"08e453e4b63892619dfe14e1415e8c0bb83f43cbc77d2b43f33281e78e6fca57"} Nov 03 01:41:38.400295 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:38.400246 4421 scope.go:122] "RemoveContainer" containerID="b31d098b3dd0a05021850986a7657443452ccc5d2d5863072ecd1b3cf0881c0d" Nov 03 01:41:38.421649 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:38.412643 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-f7680d-67845fdfd7-r56nd"] Nov 03 01:41:38.421649 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:38.418537 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-f7680d-67845fdfd7-r56nd"] Nov 03 01:41:38.439300 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:38.438445 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-f7680d-59ddf48c45-8xgwl"] Nov 03 01:41:38.448386 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:38.448352 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-f7680d-59ddf48c45-8xgwl"] Nov 03 01:41:39.391548 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:39.391509 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-717217-54d8c4bcf-v8j62" podUID="38f1d8ba-2f55-4d82-b5e6-0491c36a8009" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:41:39.984598 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:39.984555 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08dc3816-9d73-4dcf-abff-cef7f40b0009" path="/var/lib/kubelet/pods/08dc3816-9d73-4dcf-abff-cef7f40b0009/volumes" Nov 03 01:41:39.985173 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:39.985149 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5e0534c-5a31-4081-a049-5777d11925e1" path="/var/lib/kubelet/pods/a5e0534c-5a31-4081-a049-5777d11925e1/volumes" Nov 03 01:41:40.427294 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:40.419545 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-717217-5f86cd4989-6685p" podUID="f0f55109-ee26-4eab-b6ca-b610c783906d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:41:44.194151 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:44.194112 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-bx28n"] Nov 03 01:41:44.194825 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:44.194421 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-bx28n" podUID="7b44153f-12a2-4b0b-898a-493b372fb7ae" containerName="istio-proxy" containerID="cri-o://20ca046e7a0306502f4df0894925f2fa0d360f924086bf218ce935538bab71e1" gracePeriod=30 Nov 03 01:41:44.672602 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:41:44.672556 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 01:41:45.097192 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:45.097146 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57bcd798-bkqcf"] Nov 03 01:41:45.097744 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:45.097728 4421 cpu_manager.go:414] "containerMap: removing container" podUID="5c6e134c-a6fe-4794-9c80-b52ebb11abb2" containerName="istio-proxy" Nov 03 01:41:45.097792 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:45.097745 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c6e134c-a6fe-4794-9c80-b52ebb11abb2" containerName="istio-proxy" Nov 03 01:41:45.097792 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:45.097765 4421 cpu_manager.go:414] "containerMap: removing container" podUID="e60b0573-2ad3-40d9-898e-9c644f41dbc9" containerName="payload-processing" Nov 03 01:41:45.097792 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:45.097770 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e60b0573-2ad3-40d9-898e-9c644f41dbc9" containerName="payload-processing" Nov 03 01:41:45.097792 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:45.097788 4421 cpu_manager.go:414] "containerMap: removing container" podUID="08dc3816-9d73-4dcf-abff-cef7f40b0009" containerName="payload-pre-processing" Nov 03 01:41:45.097931 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:45.097793 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08dc3816-9d73-4dcf-abff-cef7f40b0009" containerName="payload-pre-processing" Nov 03 01:41:45.097931 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:45.097816 4421 cpu_manager.go:414] "containerMap: removing container" podUID="f4e62211-7bb6-42d2-8e73-c95d43c2737c" containerName="payload-pre-processing" Nov 03 01:41:45.097931 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:45.097822 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4e62211-7bb6-42d2-8e73-c95d43c2737c" containerName="payload-pre-processing" Nov 03 01:41:45.097931 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:45.097830 4421 cpu_manager.go:414] "containerMap: removing container" podUID="d08194d0-b707-48e0-8a62-6c04c6cdc1e7" containerName="revoke-keys" Nov 03 01:41:45.097931 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:45.097834 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d08194d0-b707-48e0-8a62-6c04c6cdc1e7" containerName="revoke-keys" Nov 03 01:41:45.097931 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:45.097845 4421 cpu_manager.go:414] "containerMap: removing container" podUID="a5e0534c-5a31-4081-a049-5777d11925e1" containerName="payload-processing" Nov 03 01:41:45.097931 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:45.097850 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5e0534c-5a31-4081-a049-5777d11925e1" containerName="payload-processing" Nov 03 01:41:45.097931 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:45.097929 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4e62211-7bb6-42d2-8e73-c95d43c2737c" containerName="payload-pre-processing" Nov 03 01:41:45.098621 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:45.097936 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="d08194d0-b707-48e0-8a62-6c04c6cdc1e7" containerName="revoke-keys" Nov 03 01:41:45.098621 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:45.097944 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5e0534c-5a31-4081-a049-5777d11925e1" containerName="payload-processing" Nov 03 01:41:45.098621 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:45.097951 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c6e134c-a6fe-4794-9c80-b52ebb11abb2" containerName="istio-proxy" Nov 03 01:41:45.098621 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:45.097961 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="e60b0573-2ad3-40d9-898e-9c644f41dbc9" containerName="payload-processing" Nov 03 01:41:45.098621 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:45.097969 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="08dc3816-9d73-4dcf-abff-cef7f40b0009" containerName="payload-pre-processing" Nov 03 01:41:45.103237 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:45.103219 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57bcd798-bkqcf" Nov 03 01:41:45.107501 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:45.107477 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:41:45.108908 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:45.108888 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6q4xp\"" Nov 03 01:41:45.110040 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:45.110014 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57bcd798-bkqcf"] Nov 03 01:41:45.197696 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:45.197661 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e88c376d-a5b0-4d87-b5d8-cb14fa46b8db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-57bcd798-bkqcf\" (UID: \"e88c376d-a5b0-4d87-b5d8-cb14fa46b8db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57bcd798-bkqcf" Nov 03 01:41:45.198115 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:45.197753 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e88c376d-a5b0-4d87-b5d8-cb14fa46b8db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-57bcd798-bkqcf\" (UID: \"e88c376d-a5b0-4d87-b5d8-cb14fa46b8db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57bcd798-bkqcf" Nov 03 01:41:45.198115 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:45.197826 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6qmv\" (UniqueName: \"kubernetes.io/projected/e88c376d-a5b0-4d87-b5d8-cb14fa46b8db-kube-api-access-h6qmv\") pod \"maas-api-revoke-sub-57bcd798-bkqcf\" (UID: \"e88c376d-a5b0-4d87-b5d8-cb14fa46b8db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57bcd798-bkqcf" Nov 03 01:41:45.199457 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:45.199438 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 01:41:45.199543 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:45.199490 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 01:41:45.299476 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:45.299440 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e88c376d-a5b0-4d87-b5d8-cb14fa46b8db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-57bcd798-bkqcf\" (UID: \"e88c376d-a5b0-4d87-b5d8-cb14fa46b8db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57bcd798-bkqcf" Nov 03 01:41:45.299476 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:45.299480 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e88c376d-a5b0-4d87-b5d8-cb14fa46b8db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-57bcd798-bkqcf\" (UID: \"e88c376d-a5b0-4d87-b5d8-cb14fa46b8db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57bcd798-bkqcf" Nov 03 01:41:45.299702 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:45.299563 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6qmv\" (UniqueName: \"kubernetes.io/projected/e88c376d-a5b0-4d87-b5d8-cb14fa46b8db-kube-api-access-h6qmv\") pod \"maas-api-revoke-sub-57bcd798-bkqcf\" (UID: \"e88c376d-a5b0-4d87-b5d8-cb14fa46b8db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57bcd798-bkqcf" Nov 03 01:41:45.300228 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:45.300198 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e88c376d-a5b0-4d87-b5d8-cb14fa46b8db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-57bcd798-bkqcf\" (UID: \"e88c376d-a5b0-4d87-b5d8-cb14fa46b8db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57bcd798-bkqcf" Nov 03 01:41:45.302051 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:45.302028 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e88c376d-a5b0-4d87-b5d8-cb14fa46b8db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-57bcd798-bkqcf\" (UID: \"e88c376d-a5b0-4d87-b5d8-cb14fa46b8db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57bcd798-bkqcf" Nov 03 01:41:45.308114 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:45.308096 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6qmv\" (UniqueName: \"kubernetes.io/projected/e88c376d-a5b0-4d87-b5d8-cb14fa46b8db-kube-api-access-h6qmv\") pod \"maas-api-revoke-sub-57bcd798-bkqcf\" (UID: \"e88c376d-a5b0-4d87-b5d8-cb14fa46b8db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57bcd798-bkqcf" Nov 03 01:41:45.424385 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:45.424269 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57bcd798-bkqcf" Nov 03 01:41:45.705489 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:45.705456 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57bcd798-bkqcf"] Nov 03 01:41:45.707795 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:41:45.707764 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode88c376d_a5b0_4d87_b5d8_cb14fa46b8db.slice/crio-629e8f93e128aa5059a4c6ff4f0cf1fc28392a05bda7908891bb7b438b03fdcf WatchSource:0}: Error finding container 629e8f93e128aa5059a4c6ff4f0cf1fc28392a05bda7908891bb7b438b03fdcf: Status 404 returned error can't find the container with id 629e8f93e128aa5059a4c6ff4f0cf1fc28392a05bda7908891bb7b438b03fdcf Nov 03 01:41:46.334506 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:46.332152 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-2fkgc"] Nov 03 01:41:46.334506 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:46.332483 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-2fkgc" podUID="f0c98b4a-9b13-4ad5-9e0b-ac50ae14fa5d" containerName="istio-proxy" containerID="cri-o://deb7249f7de228dd52e91ac11967498578fdb606c5f6b720428814bb6f8fb0da" gracePeriod=30 Nov 03 01:41:46.525268 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:46.525222 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57bcd798-bkqcf" event={"ID":"e88c376d-a5b0-4d87-b5d8-cb14fa46b8db","Type":"ContainerStarted","Data":"089638156edf368cf15cbddaf0799fcb496dda8db1fbe0050bf39deef21c5690"} Nov 03 01:41:46.525712 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:46.525585 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57bcd798-bkqcf" event={"ID":"e88c376d-a5b0-4d87-b5d8-cb14fa46b8db","Type":"ContainerStarted","Data":"629e8f93e128aa5059a4c6ff4f0cf1fc28392a05bda7908891bb7b438b03fdcf"} Nov 03 01:41:46.545885 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:46.545829 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57bcd798-bkqcf" podStartSLOduration=1.545814757 podStartE2EDuration="1.545814757s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:41:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:41:46.543745716 +0000 UTC m=+3648.714748814" watchObservedRunningTime="2026-11-03 01:41:46.545814757 +0000 UTC m=+3648.716817996" Nov 03 01:41:46.639261 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:46.639168 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-4af1f7/rate-test-model-4af1f7-kserve-6b4d9677cc-6pz6m"] Nov 03 01:41:47.533885 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:47.533849 4421 generic.go:356] "Generic (PLEG): container finished" podID="e88c376d-a5b0-4d87-b5d8-cb14fa46b8db" containerID="089638156edf368cf15cbddaf0799fcb496dda8db1fbe0050bf39deef21c5690" exitCode=0 Nov 03 01:41:47.534423 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:47.533938 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57bcd798-bkqcf" event={"ID":"e88c376d-a5b0-4d87-b5d8-cb14fa46b8db","Type":"ContainerDied","Data":"089638156edf368cf15cbddaf0799fcb496dda8db1fbe0050bf39deef21c5690"} Nov 03 01:41:48.719184 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:48.719150 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57bcd798-bkqcf" Nov 03 01:41:48.781934 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:48.781838 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e88c376d-a5b0-4d87-b5d8-cb14fa46b8db-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e88c376d-a5b0-4d87-b5d8-cb14fa46b8db-maas-api-xks-ca\") pod \"e88c376d-a5b0-4d87-b5d8-cb14fa46b8db\" (UID: \"e88c376d-a5b0-4d87-b5d8-cb14fa46b8db\") " Nov 03 01:41:48.781934 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:48.781928 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e88c376d-a5b0-4d87-b5d8-cb14fa46b8db-kube-api-access-h6qmv\" (UniqueName: \"kubernetes.io/projected/e88c376d-a5b0-4d87-b5d8-cb14fa46b8db-kube-api-access-h6qmv\") pod \"e88c376d-a5b0-4d87-b5d8-cb14fa46b8db\" (UID: \"e88c376d-a5b0-4d87-b5d8-cb14fa46b8db\") " Nov 03 01:41:48.782207 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:48.781957 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e88c376d-a5b0-4d87-b5d8-cb14fa46b8db-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e88c376d-a5b0-4d87-b5d8-cb14fa46b8db-maas-api-openshift-ca\") pod \"e88c376d-a5b0-4d87-b5d8-cb14fa46b8db\" (UID: \"e88c376d-a5b0-4d87-b5d8-cb14fa46b8db\") " Nov 03 01:41:48.783250 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:48.783210 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e88c376d-a5b0-4d87-b5d8-cb14fa46b8db-maas-api-openshift-ca" pod "e88c376d-a5b0-4d87-b5d8-cb14fa46b8db" (UID: "e88c376d-a5b0-4d87-b5d8-cb14fa46b8db"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:41:48.809895 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:48.803537 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e88c376d-a5b0-4d87-b5d8-cb14fa46b8db-maas-api-xks-ca" pod "e88c376d-a5b0-4d87-b5d8-cb14fa46b8db" (UID: "e88c376d-a5b0-4d87-b5d8-cb14fa46b8db"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:41:48.817983 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:48.817943 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e88c376d-a5b0-4d87-b5d8-cb14fa46b8db-kube-api-access-h6qmv" pod "e88c376d-a5b0-4d87-b5d8-cb14fa46b8db" (UID: "e88c376d-a5b0-4d87-b5d8-cb14fa46b8db"). InnerVolumeSpecName "kube-api-access-h6qmv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:41:48.883850 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:48.883806 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e88c376d-a5b0-4d87-b5d8-cb14fa46b8db-maas-api-xks-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:48.883850 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:48.883849 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6qmv\" (UniqueName: \"kubernetes.io/projected/e88c376d-a5b0-4d87-b5d8-cb14fa46b8db-kube-api-access-h6qmv\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:48.884145 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:48.883869 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e88c376d-a5b0-4d87-b5d8-cb14fa46b8db-maas-api-openshift-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:49.392204 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:49.392162 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-717217-54d8c4bcf-v8j62" podUID="38f1d8ba-2f55-4d82-b5e6-0491c36a8009" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:41:49.438224 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:49.438186 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-bx28n" Nov 03 01:41:49.509747 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:49.508542 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b44153f-12a2-4b0b-898a-493b372fb7ae-kube-api-access-l889w\" (UniqueName: \"kubernetes.io/projected/7b44153f-12a2-4b0b-898a-493b372fb7ae-kube-api-access-l889w\") pod \"7b44153f-12a2-4b0b-898a-493b372fb7ae\" (UID: \"7b44153f-12a2-4b0b-898a-493b372fb7ae\") " Nov 03 01:41:49.509747 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:49.508644 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b44153f-12a2-4b0b-898a-493b372fb7ae-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b44153f-12a2-4b0b-898a-493b372fb7ae-istio-data\") pod \"7b44153f-12a2-4b0b-898a-493b372fb7ae\" (UID: \"7b44153f-12a2-4b0b-898a-493b372fb7ae\") " Nov 03 01:41:49.509747 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:49.508706 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b44153f-12a2-4b0b-898a-493b372fb7ae-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b44153f-12a2-4b0b-898a-493b372fb7ae-istio-envoy\") pod \"7b44153f-12a2-4b0b-898a-493b372fb7ae\" (UID: \"7b44153f-12a2-4b0b-898a-493b372fb7ae\") " Nov 03 01:41:49.509747 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:49.508760 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b44153f-12a2-4b0b-898a-493b372fb7ae-istio-token\" (UniqueName: \"kubernetes.io/projected/7b44153f-12a2-4b0b-898a-493b372fb7ae-istio-token\") pod \"7b44153f-12a2-4b0b-898a-493b372fb7ae\" (UID: \"7b44153f-12a2-4b0b-898a-493b372fb7ae\") " Nov 03 01:41:49.509747 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:49.508795 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b44153f-12a2-4b0b-898a-493b372fb7ae-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b44153f-12a2-4b0b-898a-493b372fb7ae-istiod-ca-cert\") pod \"7b44153f-12a2-4b0b-898a-493b372fb7ae\" (UID: \"7b44153f-12a2-4b0b-898a-493b372fb7ae\") " Nov 03 01:41:49.509747 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:49.508842 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b44153f-12a2-4b0b-898a-493b372fb7ae-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b44153f-12a2-4b0b-898a-493b372fb7ae-workload-socket\") pod \"7b44153f-12a2-4b0b-898a-493b372fb7ae\" (UID: \"7b44153f-12a2-4b0b-898a-493b372fb7ae\") " Nov 03 01:41:49.509747 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:49.508907 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7b44153f-12a2-4b0b-898a-493b372fb7ae-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b44153f-12a2-4b0b-898a-493b372fb7ae-istio-podinfo\") pod \"7b44153f-12a2-4b0b-898a-493b372fb7ae\" (UID: \"7b44153f-12a2-4b0b-898a-493b372fb7ae\") " Nov 03 01:41:49.509747 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:49.508947 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b44153f-12a2-4b0b-898a-493b372fb7ae-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b44153f-12a2-4b0b-898a-493b372fb7ae-credential-socket\") pod \"7b44153f-12a2-4b0b-898a-493b372fb7ae\" (UID: \"7b44153f-12a2-4b0b-898a-493b372fb7ae\") " Nov 03 01:41:49.509747 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:49.508976 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b44153f-12a2-4b0b-898a-493b372fb7ae-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b44153f-12a2-4b0b-898a-493b372fb7ae-workload-certs\") pod \"7b44153f-12a2-4b0b-898a-493b372fb7ae\" (UID: \"7b44153f-12a2-4b0b-898a-493b372fb7ae\") " Nov 03 01:41:49.510520 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:49.509759 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b44153f-12a2-4b0b-898a-493b372fb7ae-istio-data" pod "7b44153f-12a2-4b0b-898a-493b372fb7ae" (UID: "7b44153f-12a2-4b0b-898a-493b372fb7ae"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:41:49.510520 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:49.509977 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b44153f-12a2-4b0b-898a-493b372fb7ae-istiod-ca-cert" pod "7b44153f-12a2-4b0b-898a-493b372fb7ae" (UID: "7b44153f-12a2-4b0b-898a-493b372fb7ae"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:41:49.510520 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:49.510214 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b44153f-12a2-4b0b-898a-493b372fb7ae-workload-socket" pod "7b44153f-12a2-4b0b-898a-493b372fb7ae" (UID: "7b44153f-12a2-4b0b-898a-493b372fb7ae"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:41:49.510520 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:49.510268 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b44153f-12a2-4b0b-898a-493b372fb7ae-credential-socket" pod "7b44153f-12a2-4b0b-898a-493b372fb7ae" (UID: "7b44153f-12a2-4b0b-898a-493b372fb7ae"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:41:49.510819 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:49.510789 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b44153f-12a2-4b0b-898a-493b372fb7ae-workload-certs" pod "7b44153f-12a2-4b0b-898a-493b372fb7ae" (UID: "7b44153f-12a2-4b0b-898a-493b372fb7ae"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:41:49.512508 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:49.512482 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b44153f-12a2-4b0b-898a-493b372fb7ae-istio-token" pod "7b44153f-12a2-4b0b-898a-493b372fb7ae" (UID: "7b44153f-12a2-4b0b-898a-493b372fb7ae"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:41:49.512673 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:49.512646 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7b44153f-12a2-4b0b-898a-493b372fb7ae-istio-podinfo" pod "7b44153f-12a2-4b0b-898a-493b372fb7ae" (UID: "7b44153f-12a2-4b0b-898a-493b372fb7ae"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:41:49.512755 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:49.512684 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b44153f-12a2-4b0b-898a-493b372fb7ae-kube-api-access-l889w" pod "7b44153f-12a2-4b0b-898a-493b372fb7ae" (UID: "7b44153f-12a2-4b0b-898a-493b372fb7ae"). InnerVolumeSpecName "kube-api-access-l889w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:41:49.513484 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:49.513459 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b44153f-12a2-4b0b-898a-493b372fb7ae-istio-envoy" pod "7b44153f-12a2-4b0b-898a-493b372fb7ae" (UID: "7b44153f-12a2-4b0b-898a-493b372fb7ae"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:41:49.557092 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:49.557055 4421 generic.go:356] "Generic (PLEG): container finished" podID="7b44153f-12a2-4b0b-898a-493b372fb7ae" containerID="20ca046e7a0306502f4df0894925f2fa0d360f924086bf218ce935538bab71e1" exitCode=0 Nov 03 01:41:49.557316 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:49.557173 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-bx28n" Nov 03 01:41:49.557316 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:49.557193 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-bx28n" event={"ID":"7b44153f-12a2-4b0b-898a-493b372fb7ae","Type":"ContainerDied","Data":"20ca046e7a0306502f4df0894925f2fa0d360f924086bf218ce935538bab71e1"} Nov 03 01:41:49.557316 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:49.557260 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-bx28n" event={"ID":"7b44153f-12a2-4b0b-898a-493b372fb7ae","Type":"ContainerDied","Data":"ee1c4f85b490e97e004538c8718093d0bf0e1c38df53b7cadd35bcf567c4643e"} Nov 03 01:41:49.557486 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:49.557329 4421 scope.go:122] "RemoveContainer" containerID="20ca046e7a0306502f4df0894925f2fa0d360f924086bf218ce935538bab71e1" Nov 03 01:41:49.563536 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:49.563500 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57bcd798-bkqcf" Nov 03 01:41:49.563695 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:49.563533 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57bcd798-bkqcf" event={"ID":"e88c376d-a5b0-4d87-b5d8-cb14fa46b8db","Type":"ContainerDied","Data":"629e8f93e128aa5059a4c6ff4f0cf1fc28392a05bda7908891bb7b438b03fdcf"} Nov 03 01:41:49.563695 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:49.563593 4421 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="629e8f93e128aa5059a4c6ff4f0cf1fc28392a05bda7908891bb7b438b03fdcf" Nov 03 01:41:49.574830 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:49.574794 4421 scope.go:122] "RemoveContainer" containerID="20ca046e7a0306502f4df0894925f2fa0d360f924086bf218ce935538bab71e1" Nov 03 01:41:49.578539 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:41:49.578500 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20ca046e7a0306502f4df0894925f2fa0d360f924086bf218ce935538bab71e1\": container with ID starting with 20ca046e7a0306502f4df0894925f2fa0d360f924086bf218ce935538bab71e1 not found: ID does not exist" containerID="20ca046e7a0306502f4df0894925f2fa0d360f924086bf218ce935538bab71e1" Nov 03 01:41:49.578689 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:49.578551 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20ca046e7a0306502f4df0894925f2fa0d360f924086bf218ce935538bab71e1"} err="failed to get container status \"20ca046e7a0306502f4df0894925f2fa0d360f924086bf218ce935538bab71e1\": rpc error: code = NotFound desc = could not find container \"20ca046e7a0306502f4df0894925f2fa0d360f924086bf218ce935538bab71e1\": container with ID starting with 20ca046e7a0306502f4df0894925f2fa0d360f924086bf218ce935538bab71e1 not found: ID does not exist" Nov 03 01:41:49.592792 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:49.592757 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-bx28n"] Nov 03 01:41:49.596497 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:49.596465 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-bx28n"] Nov 03 01:41:49.610476 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:49.610431 4421 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b44153f-12a2-4b0b-898a-493b372fb7ae-istio-podinfo\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:49.610476 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:49.610470 4421 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b44153f-12a2-4b0b-898a-493b372fb7ae-credential-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:49.610713 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:49.610490 4421 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b44153f-12a2-4b0b-898a-493b372fb7ae-workload-certs\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:49.610713 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:49.610504 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l889w\" (UniqueName: \"kubernetes.io/projected/7b44153f-12a2-4b0b-898a-493b372fb7ae-kube-api-access-l889w\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:49.610713 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:49.610520 4421 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b44153f-12a2-4b0b-898a-493b372fb7ae-istio-data\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:49.610713 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:49.610533 4421 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b44153f-12a2-4b0b-898a-493b372fb7ae-istio-envoy\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:49.610713 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:49.610545 4421 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b44153f-12a2-4b0b-898a-493b372fb7ae-istio-token\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:49.610713 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:49.610557 4421 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b44153f-12a2-4b0b-898a-493b372fb7ae-istiod-ca-cert\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:49.610713 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:49.610571 4421 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b44153f-12a2-4b0b-898a-493b372fb7ae-workload-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:49.710073 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:49.710032 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-650be0/premium-w4-650be0-kserve-7bf8555dfb-z97n5"] Nov 03 01:41:49.722208 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:49.722176 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-650be0/simulator-w4-650be0-kserve-7bc588774f-gvz2k"] Nov 03 01:41:49.781342 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:49.781235 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-fqqx7"] Nov 03 01:41:49.781550 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:49.781524 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-fqqx7" podUID="392e8058-14f9-4dfb-b20e-3e209eb3c625" containerName="istio-proxy" containerID="cri-o://9c0f1c5153a027730dca45af068c6bab21690e517eee27a99c60c527f90ccc9a" gracePeriod=30 Nov 03 01:41:49.980324 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:49.980267 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b44153f-12a2-4b0b-898a-493b372fb7ae" path="/var/lib/kubelet/pods/7b44153f-12a2-4b0b-898a-493b372fb7ae/volumes" Nov 03 01:41:50.401077 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:50.401037 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-717217-5f86cd4989-6685p" podUID="f0f55109-ee26-4eab-b6ca-b610c783906d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:41:50.534242 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:50.534208 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-66775a6a-6kwtx"] Nov 03 01:41:50.534920 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:50.534900 4421 cpu_manager.go:414] "containerMap: removing container" podUID="7b44153f-12a2-4b0b-898a-493b372fb7ae" containerName="istio-proxy" Nov 03 01:41:50.534973 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:50.534922 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b44153f-12a2-4b0b-898a-493b372fb7ae" containerName="istio-proxy" Nov 03 01:41:50.534973 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:50.534964 4421 cpu_manager.go:414] "containerMap: removing container" podUID="e88c376d-a5b0-4d87-b5d8-cb14fa46b8db" containerName="revoke-keys" Nov 03 01:41:50.534973 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:50.534970 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e88c376d-a5b0-4d87-b5d8-cb14fa46b8db" containerName="revoke-keys" Nov 03 01:41:50.535083 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:50.535050 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="e88c376d-a5b0-4d87-b5d8-cb14fa46b8db" containerName="revoke-keys" Nov 03 01:41:50.535083 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:50.535061 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b44153f-12a2-4b0b-898a-493b372fb7ae" containerName="istio-proxy" Nov 03 01:41:50.540002 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:50.539982 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66775a6a-6kwtx" Nov 03 01:41:50.543658 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:50.543609 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:41:50.544383 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:50.544358 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 01:41:50.544796 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:50.544774 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 01:41:50.545134 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:50.545112 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6q4xp\"" Nov 03 01:41:50.546295 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:50.546252 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-66775a6a-6kwtx"] Nov 03 01:41:50.632455 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:50.632424 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxh4v\" (UniqueName: \"kubernetes.io/projected/24a9d347-bb35-4a9c-b489-726098a59823-kube-api-access-kxh4v\") pod \"maas-api-revoke-sub-66775a6a-6kwtx\" (UID: \"24a9d347-bb35-4a9c-b489-726098a59823\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66775a6a-6kwtx" Nov 03 01:41:50.632627 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:50.632549 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24a9d347-bb35-4a9c-b489-726098a59823-maas-api-xks-ca\") pod \"maas-api-revoke-sub-66775a6a-6kwtx\" (UID: \"24a9d347-bb35-4a9c-b489-726098a59823\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66775a6a-6kwtx" Nov 03 01:41:50.632627 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:50.632602 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24a9d347-bb35-4a9c-b489-726098a59823-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-66775a6a-6kwtx\" (UID: \"24a9d347-bb35-4a9c-b489-726098a59823\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66775a6a-6kwtx" Nov 03 01:41:50.734012 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:50.733983 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24a9d347-bb35-4a9c-b489-726098a59823-maas-api-xks-ca\") pod \"maas-api-revoke-sub-66775a6a-6kwtx\" (UID: \"24a9d347-bb35-4a9c-b489-726098a59823\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66775a6a-6kwtx" Nov 03 01:41:50.734427 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:50.734079 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24a9d347-bb35-4a9c-b489-726098a59823-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-66775a6a-6kwtx\" (UID: \"24a9d347-bb35-4a9c-b489-726098a59823\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66775a6a-6kwtx" Nov 03 01:41:50.734427 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:50.734230 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxh4v\" (UniqueName: \"kubernetes.io/projected/24a9d347-bb35-4a9c-b489-726098a59823-kube-api-access-kxh4v\") pod \"maas-api-revoke-sub-66775a6a-6kwtx\" (UID: \"24a9d347-bb35-4a9c-b489-726098a59823\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66775a6a-6kwtx" Nov 03 01:41:50.734849 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:50.734831 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24a9d347-bb35-4a9c-b489-726098a59823-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-66775a6a-6kwtx\" (UID: \"24a9d347-bb35-4a9c-b489-726098a59823\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66775a6a-6kwtx" Nov 03 01:41:50.736313 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:50.736292 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24a9d347-bb35-4a9c-b489-726098a59823-maas-api-xks-ca\") pod \"maas-api-revoke-sub-66775a6a-6kwtx\" (UID: \"24a9d347-bb35-4a9c-b489-726098a59823\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66775a6a-6kwtx" Nov 03 01:41:50.742248 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:50.742222 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxh4v\" (UniqueName: \"kubernetes.io/projected/24a9d347-bb35-4a9c-b489-726098a59823-kube-api-access-kxh4v\") pod \"maas-api-revoke-sub-66775a6a-6kwtx\" (UID: \"24a9d347-bb35-4a9c-b489-726098a59823\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66775a6a-6kwtx" Nov 03 01:41:50.857970 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:50.857889 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66775a6a-6kwtx" Nov 03 01:41:50.985147 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:50.984775 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-717217-5f86cd4989-6685p" Nov 03 01:41:51.021370 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:51.021343 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-717217-54d8c4bcf-v8j62" Nov 03 01:41:51.061680 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:51.061643 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0f55109-ee26-4eab-b6ca-b610c783906d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0f55109-ee26-4eab-b6ca-b610c783906d-plugins-config-volume\") pod \"f0f55109-ee26-4eab-b6ca-b610c783906d\" (UID: \"f0f55109-ee26-4eab-b6ca-b610c783906d\") " Nov 03 01:41:51.061937 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:51.061911 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0f55109-ee26-4eab-b6ca-b610c783906d-kube-api-access-gv8mp\" (UniqueName: \"kubernetes.io/projected/f0f55109-ee26-4eab-b6ca-b610c783906d-kube-api-access-gv8mp\") pod \"f0f55109-ee26-4eab-b6ca-b610c783906d\" (UID: \"f0f55109-ee26-4eab-b6ca-b610c783906d\") " Nov 03 01:41:51.062127 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:51.062096 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0f55109-ee26-4eab-b6ca-b610c783906d-plugins-config-volume" pod "f0f55109-ee26-4eab-b6ca-b610c783906d" (UID: "f0f55109-ee26-4eab-b6ca-b610c783906d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:41:51.062976 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:51.062946 4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0f55109-ee26-4eab-b6ca-b610c783906d-plugins-config-volume\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:51.064249 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:51.064211 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0f55109-ee26-4eab-b6ca-b610c783906d-kube-api-access-gv8mp" pod "f0f55109-ee26-4eab-b6ca-b610c783906d" (UID: "f0f55109-ee26-4eab-b6ca-b610c783906d"). InnerVolumeSpecName "kube-api-access-gv8mp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:41:51.164161 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:51.164078 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38f1d8ba-2f55-4d82-b5e6-0491c36a8009-kube-api-access-r4kth\" (UniqueName: \"kubernetes.io/projected/38f1d8ba-2f55-4d82-b5e6-0491c36a8009-kube-api-access-r4kth\") pod \"38f1d8ba-2f55-4d82-b5e6-0491c36a8009\" (UID: \"38f1d8ba-2f55-4d82-b5e6-0491c36a8009\") " Nov 03 01:41:51.164343 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:51.164167 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38f1d8ba-2f55-4d82-b5e6-0491c36a8009-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38f1d8ba-2f55-4d82-b5e6-0491c36a8009-plugins-config-volume\") pod \"38f1d8ba-2f55-4d82-b5e6-0491c36a8009\" (UID: \"38f1d8ba-2f55-4d82-b5e6-0491c36a8009\") " Nov 03 01:41:51.164568 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:51.164545 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38f1d8ba-2f55-4d82-b5e6-0491c36a8009-plugins-config-volume" pod "38f1d8ba-2f55-4d82-b5e6-0491c36a8009" (UID: "38f1d8ba-2f55-4d82-b5e6-0491c36a8009"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:41:51.164817 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:51.164792 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gv8mp\" (UniqueName: \"kubernetes.io/projected/f0f55109-ee26-4eab-b6ca-b610c783906d-kube-api-access-gv8mp\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:51.164817 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:51.164816 4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38f1d8ba-2f55-4d82-b5e6-0491c36a8009-plugins-config-volume\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:51.166315 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:51.166296 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38f1d8ba-2f55-4d82-b5e6-0491c36a8009-kube-api-access-r4kth" pod "38f1d8ba-2f55-4d82-b5e6-0491c36a8009" (UID: "38f1d8ba-2f55-4d82-b5e6-0491c36a8009"). InnerVolumeSpecName "kube-api-access-r4kth". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:41:51.198996 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:51.198971 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-66775a6a-6kwtx"] Nov 03 01:41:51.200314 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:41:51.200263 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24a9d347_bb35_4a9c_b489_726098a59823.slice/crio-a1fbd250c0b423283f9c0c68c64452d9fd7808368f760a47d2754b966389668a WatchSource:0}: Error finding container a1fbd250c0b423283f9c0c68c64452d9fd7808368f760a47d2754b966389668a: Status 404 returned error can't find the container with id a1fbd250c0b423283f9c0c68c64452d9fd7808368f760a47d2754b966389668a Nov 03 01:41:51.265940 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:51.265909 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4kth\" (UniqueName: \"kubernetes.io/projected/38f1d8ba-2f55-4d82-b5e6-0491c36a8009-kube-api-access-r4kth\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:51.587104 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:51.587067 4421 generic.go:356] "Generic (PLEG): container finished" podID="f0f55109-ee26-4eab-b6ca-b610c783906d" containerID="53f45c8b8b8508e4304cfb541ef35380dfdc7dcf9d19189c7b0173207a1fa436" exitCode=0 Nov 03 01:41:51.587374 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:51.587117 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-717217-5f86cd4989-6685p" Nov 03 01:41:51.587374 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:51.587156 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-717217-5f86cd4989-6685p" event={"ID":"f0f55109-ee26-4eab-b6ca-b610c783906d","Type":"ContainerDied","Data":"53f45c8b8b8508e4304cfb541ef35380dfdc7dcf9d19189c7b0173207a1fa436"} Nov 03 01:41:51.587374 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:51.587211 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-717217-5f86cd4989-6685p" event={"ID":"f0f55109-ee26-4eab-b6ca-b610c783906d","Type":"ContainerDied","Data":"2685c07cc40073e5f18c8da383c403cab84c214becfb0282c6a842e68d27ebf4"} Nov 03 01:41:51.587374 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:51.587239 4421 scope.go:122] "RemoveContainer" containerID="53f45c8b8b8508e4304cfb541ef35380dfdc7dcf9d19189c7b0173207a1fa436" Nov 03 01:41:51.589893 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:51.589866 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66775a6a-6kwtx" event={"ID":"24a9d347-bb35-4a9c-b489-726098a59823","Type":"ContainerStarted","Data":"a1fbd250c0b423283f9c0c68c64452d9fd7808368f760a47d2754b966389668a"} Nov 03 01:41:51.591515 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:51.591488 4421 generic.go:356] "Generic (PLEG): container finished" podID="38f1d8ba-2f55-4d82-b5e6-0491c36a8009" containerID="045a5208c7c98ef9a29aaebb16150c29be541504e6561b1ae71b8a056031c398" exitCode=0 Nov 03 01:41:51.591658 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:51.591537 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-717217-54d8c4bcf-v8j62" Nov 03 01:41:51.591658 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:51.591557 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-717217-54d8c4bcf-v8j62" event={"ID":"38f1d8ba-2f55-4d82-b5e6-0491c36a8009","Type":"ContainerDied","Data":"045a5208c7c98ef9a29aaebb16150c29be541504e6561b1ae71b8a056031c398"} Nov 03 01:41:51.591658 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:51.591619 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-717217-54d8c4bcf-v8j62" event={"ID":"38f1d8ba-2f55-4d82-b5e6-0491c36a8009","Type":"ContainerDied","Data":"df214b099585ec039e612fd2f531c4b79dc0a88e0c52e742e21dead93d7edad9"} Nov 03 01:41:51.594618 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:51.594506 4421 generic.go:356] "Generic (PLEG): container finished" podID="f0c98b4a-9b13-4ad5-9e0b-ac50ae14fa5d" containerID="deb7249f7de228dd52e91ac11967498578fdb606c5f6b720428814bb6f8fb0da" exitCode=0 Nov 03 01:41:51.594618 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:51.594537 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-2fkgc" event={"ID":"f0c98b4a-9b13-4ad5-9e0b-ac50ae14fa5d","Type":"ContainerDied","Data":"deb7249f7de228dd52e91ac11967498578fdb606c5f6b720428814bb6f8fb0da"} Nov 03 01:41:51.607119 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:51.607103 4421 scope.go:122] "RemoveContainer" containerID="53f45c8b8b8508e4304cfb541ef35380dfdc7dcf9d19189c7b0173207a1fa436" Nov 03 01:41:51.607411 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:41:51.607386 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53f45c8b8b8508e4304cfb541ef35380dfdc7dcf9d19189c7b0173207a1fa436\": container with ID starting with 53f45c8b8b8508e4304cfb541ef35380dfdc7dcf9d19189c7b0173207a1fa436 not found: ID does not exist" containerID="53f45c8b8b8508e4304cfb541ef35380dfdc7dcf9d19189c7b0173207a1fa436" Nov 03 01:41:51.607485 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:51.607420 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53f45c8b8b8508e4304cfb541ef35380dfdc7dcf9d19189c7b0173207a1fa436"} err="failed to get container status \"53f45c8b8b8508e4304cfb541ef35380dfdc7dcf9d19189c7b0173207a1fa436\": rpc error: code = NotFound desc = could not find container \"53f45c8b8b8508e4304cfb541ef35380dfdc7dcf9d19189c7b0173207a1fa436\": container with ID starting with 53f45c8b8b8508e4304cfb541ef35380dfdc7dcf9d19189c7b0173207a1fa436 not found: ID does not exist" Nov 03 01:41:51.607485 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:51.607442 4421 scope.go:122] "RemoveContainer" containerID="045a5208c7c98ef9a29aaebb16150c29be541504e6561b1ae71b8a056031c398" Nov 03 01:41:51.612582 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:51.612559 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-2fkgc" Nov 03 01:41:51.615728 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:51.615708 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-717217-5f86cd4989-6685p"] Nov 03 01:41:51.619414 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:51.619394 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-717217-5f86cd4989-6685p"] Nov 03 01:41:51.620440 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:51.620199 4421 scope.go:122] "RemoveContainer" containerID="045a5208c7c98ef9a29aaebb16150c29be541504e6561b1ae71b8a056031c398" Nov 03 01:41:51.620934 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:41:51.620911 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"045a5208c7c98ef9a29aaebb16150c29be541504e6561b1ae71b8a056031c398\": container with ID starting with 045a5208c7c98ef9a29aaebb16150c29be541504e6561b1ae71b8a056031c398 not found: ID does not exist" containerID="045a5208c7c98ef9a29aaebb16150c29be541504e6561b1ae71b8a056031c398" Nov 03 01:41:51.621108 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:51.620942 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"045a5208c7c98ef9a29aaebb16150c29be541504e6561b1ae71b8a056031c398"} err="failed to get container status \"045a5208c7c98ef9a29aaebb16150c29be541504e6561b1ae71b8a056031c398\": rpc error: code = NotFound desc = could not find container \"045a5208c7c98ef9a29aaebb16150c29be541504e6561b1ae71b8a056031c398\": container with ID starting with 045a5208c7c98ef9a29aaebb16150c29be541504e6561b1ae71b8a056031c398 not found: ID does not exist" Nov 03 01:41:51.621108 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:51.620969 4421 scope.go:122] "RemoveContainer" containerID="9ef83ec7d0dc2496a4493b5009f4a307086c31c8e36533834180854f6ae4a2ad" Nov 03 01:41:51.642958 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:51.642895 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-717217-54d8c4bcf-v8j62"] Nov 03 01:41:51.647305 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:51.647260 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-717217-54d8c4bcf-v8j62"] Nov 03 01:41:51.771793 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:51.771761 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0c98b4a-9b13-4ad5-9e0b-ac50ae14fa5d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0c98b4a-9b13-4ad5-9e0b-ac50ae14fa5d-workload-certs\") pod \"f0c98b4a-9b13-4ad5-9e0b-ac50ae14fa5d\" (UID: \"f0c98b4a-9b13-4ad5-9e0b-ac50ae14fa5d\") " Nov 03 01:41:51.772258 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:51.771807 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0c98b4a-9b13-4ad5-9e0b-ac50ae14fa5d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0c98b4a-9b13-4ad5-9e0b-ac50ae14fa5d-istio-data\") pod \"f0c98b4a-9b13-4ad5-9e0b-ac50ae14fa5d\" (UID: \"f0c98b4a-9b13-4ad5-9e0b-ac50ae14fa5d\") " Nov 03 01:41:51.772258 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:51.771876 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0c98b4a-9b13-4ad5-9e0b-ac50ae14fa5d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0c98b4a-9b13-4ad5-9e0b-ac50ae14fa5d-credential-socket\") pod \"f0c98b4a-9b13-4ad5-9e0b-ac50ae14fa5d\" (UID: \"f0c98b4a-9b13-4ad5-9e0b-ac50ae14fa5d\") " Nov 03 01:41:51.772258 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:51.771932 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0c98b4a-9b13-4ad5-9e0b-ac50ae14fa5d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0c98b4a-9b13-4ad5-9e0b-ac50ae14fa5d-istio-envoy\") pod \"f0c98b4a-9b13-4ad5-9e0b-ac50ae14fa5d\" (UID: \"f0c98b4a-9b13-4ad5-9e0b-ac50ae14fa5d\") " Nov 03 01:41:51.772258 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:51.771957 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0c98b4a-9b13-4ad5-9e0b-ac50ae14fa5d-kube-api-access-zvnjx\" (UniqueName: \"kubernetes.io/projected/f0c98b4a-9b13-4ad5-9e0b-ac50ae14fa5d-kube-api-access-zvnjx\") pod \"f0c98b4a-9b13-4ad5-9e0b-ac50ae14fa5d\" (UID: \"f0c98b4a-9b13-4ad5-9e0b-ac50ae14fa5d\") " Nov 03 01:41:51.772258 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:51.771981 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0c98b4a-9b13-4ad5-9e0b-ac50ae14fa5d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0c98b4a-9b13-4ad5-9e0b-ac50ae14fa5d-workload-socket\") pod \"f0c98b4a-9b13-4ad5-9e0b-ac50ae14fa5d\" (UID: \"f0c98b4a-9b13-4ad5-9e0b-ac50ae14fa5d\") " Nov 03 01:41:51.772258 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:51.772040 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0c98b4a-9b13-4ad5-9e0b-ac50ae14fa5d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0c98b4a-9b13-4ad5-9e0b-ac50ae14fa5d-istiod-ca-cert\") pod \"f0c98b4a-9b13-4ad5-9e0b-ac50ae14fa5d\" (UID: \"f0c98b4a-9b13-4ad5-9e0b-ac50ae14fa5d\") " Nov 03 01:41:51.772258 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:51.772068 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f0c98b4a-9b13-4ad5-9e0b-ac50ae14fa5d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0c98b4a-9b13-4ad5-9e0b-ac50ae14fa5d-istio-podinfo\") pod \"f0c98b4a-9b13-4ad5-9e0b-ac50ae14fa5d\" (UID: \"f0c98b4a-9b13-4ad5-9e0b-ac50ae14fa5d\") " Nov 03 01:41:51.772258 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:51.772157 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0c98b4a-9b13-4ad5-9e0b-ac50ae14fa5d-istio-token\" (UniqueName: \"kubernetes.io/projected/f0c98b4a-9b13-4ad5-9e0b-ac50ae14fa5d-istio-token\") pod \"f0c98b4a-9b13-4ad5-9e0b-ac50ae14fa5d\" (UID: \"f0c98b4a-9b13-4ad5-9e0b-ac50ae14fa5d\") " Nov 03 01:41:51.772784 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:51.772758 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0c98b4a-9b13-4ad5-9e0b-ac50ae14fa5d-istio-data" pod "f0c98b4a-9b13-4ad5-9e0b-ac50ae14fa5d" (UID: "f0c98b4a-9b13-4ad5-9e0b-ac50ae14fa5d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:41:51.772894 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:51.772867 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0c98b4a-9b13-4ad5-9e0b-ac50ae14fa5d-workload-certs" pod "f0c98b4a-9b13-4ad5-9e0b-ac50ae14fa5d" (UID: "f0c98b4a-9b13-4ad5-9e0b-ac50ae14fa5d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:41:51.773001 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:51.772974 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0c98b4a-9b13-4ad5-9e0b-ac50ae14fa5d-istiod-ca-cert" pod "f0c98b4a-9b13-4ad5-9e0b-ac50ae14fa5d" (UID: "f0c98b4a-9b13-4ad5-9e0b-ac50ae14fa5d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:41:51.773246 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:51.773203 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0c98b4a-9b13-4ad5-9e0b-ac50ae14fa5d-credential-socket" pod "f0c98b4a-9b13-4ad5-9e0b-ac50ae14fa5d" (UID: "f0c98b4a-9b13-4ad5-9e0b-ac50ae14fa5d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:41:51.773387 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:51.773285 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0c98b4a-9b13-4ad5-9e0b-ac50ae14fa5d-workload-socket" pod "f0c98b4a-9b13-4ad5-9e0b-ac50ae14fa5d" (UID: "f0c98b4a-9b13-4ad5-9e0b-ac50ae14fa5d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:41:51.774877 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:51.774848 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0c98b4a-9b13-4ad5-9e0b-ac50ae14fa5d-istio-envoy" pod "f0c98b4a-9b13-4ad5-9e0b-ac50ae14fa5d" (UID: "f0c98b4a-9b13-4ad5-9e0b-ac50ae14fa5d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:41:51.775000 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:51.774911 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0c98b4a-9b13-4ad5-9e0b-ac50ae14fa5d-kube-api-access-zvnjx" pod "f0c98b4a-9b13-4ad5-9e0b-ac50ae14fa5d" (UID: "f0c98b4a-9b13-4ad5-9e0b-ac50ae14fa5d"). InnerVolumeSpecName "kube-api-access-zvnjx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:41:51.775000 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:51.774965 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0c98b4a-9b13-4ad5-9e0b-ac50ae14fa5d-istio-token" pod "f0c98b4a-9b13-4ad5-9e0b-ac50ae14fa5d" (UID: "f0c98b4a-9b13-4ad5-9e0b-ac50ae14fa5d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:41:51.775127 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:51.775109 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f0c98b4a-9b13-4ad5-9e0b-ac50ae14fa5d-istio-podinfo" pod "f0c98b4a-9b13-4ad5-9e0b-ac50ae14fa5d" (UID: "f0c98b4a-9b13-4ad5-9e0b-ac50ae14fa5d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:41:51.873707 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:51.873669 4421 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0c98b4a-9b13-4ad5-9e0b-ac50ae14fa5d-istio-token\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:51.873707 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:51.873701 4421 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0c98b4a-9b13-4ad5-9e0b-ac50ae14fa5d-workload-certs\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:51.873707 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:51.873710 4421 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0c98b4a-9b13-4ad5-9e0b-ac50ae14fa5d-istio-data\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:51.873961 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:51.873720 4421 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0c98b4a-9b13-4ad5-9e0b-ac50ae14fa5d-credential-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:51.873961 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:51.873730 4421 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0c98b4a-9b13-4ad5-9e0b-ac50ae14fa5d-istio-envoy\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:51.873961 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:51.873739 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvnjx\" (UniqueName: \"kubernetes.io/projected/f0c98b4a-9b13-4ad5-9e0b-ac50ae14fa5d-kube-api-access-zvnjx\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:51.873961 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:51.873749 4421 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0c98b4a-9b13-4ad5-9e0b-ac50ae14fa5d-workload-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:51.873961 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:51.873760 4421 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0c98b4a-9b13-4ad5-9e0b-ac50ae14fa5d-istiod-ca-cert\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:51.873961 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:51.873768 4421 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0c98b4a-9b13-4ad5-9e0b-ac50ae14fa5d-istio-podinfo\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:51.979811 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:51.979719 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38f1d8ba-2f55-4d82-b5e6-0491c36a8009" path="/var/lib/kubelet/pods/38f1d8ba-2f55-4d82-b5e6-0491c36a8009/volumes" Nov 03 01:41:51.980226 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:51.980210 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0f55109-ee26-4eab-b6ca-b610c783906d" path="/var/lib/kubelet/pods/f0f55109-ee26-4eab-b6ca-b610c783906d/volumes" Nov 03 01:41:52.602988 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:52.602955 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-2fkgc" Nov 03 01:41:52.603220 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:52.602990 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-2fkgc" event={"ID":"f0c98b4a-9b13-4ad5-9e0b-ac50ae14fa5d","Type":"ContainerDied","Data":"f235caf7ea49d70e7fb0e1e6da53c7f90c7466ddd68d12d6604cc5fc72510e69"} Nov 03 01:41:52.603220 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:52.603073 4421 scope.go:122] "RemoveContainer" containerID="deb7249f7de228dd52e91ac11967498578fdb606c5f6b720428814bb6f8fb0da" Nov 03 01:41:52.606245 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:52.606218 4421 generic.go:356] "Generic (PLEG): container finished" podID="24a9d347-bb35-4a9c-b489-726098a59823" containerID="a51e848cbff6bcad6b457200ebc9a9d2ca36b5123f1e31645dfa7a9c4d888e86" exitCode=0 Nov 03 01:41:52.606366 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:52.606309 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66775a6a-6kwtx" event={"ID":"24a9d347-bb35-4a9c-b489-726098a59823","Type":"ContainerDied","Data":"a51e848cbff6bcad6b457200ebc9a9d2ca36b5123f1e31645dfa7a9c4d888e86"} Nov 03 01:41:52.663013 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:52.662976 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-2fkgc"] Nov 03 01:41:52.666928 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:52.666899 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-2fkgc"] Nov 03 01:41:53.169329 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:53.169269 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-821f5a-7655cbb57c-wgbsw"] Nov 03 01:41:53.180506 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:53.180464 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-821f5a-84858c594c-pszhq"] Nov 03 01:41:53.812983 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:53.812954 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66775a6a-6kwtx" Nov 03 01:41:53.911611 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:53.911534 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24a9d347-bb35-4a9c-b489-726098a59823-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24a9d347-bb35-4a9c-b489-726098a59823-maas-api-openshift-ca\") pod \"24a9d347-bb35-4a9c-b489-726098a59823\" (UID: \"24a9d347-bb35-4a9c-b489-726098a59823\") " Nov 03 01:41:53.911611 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:53.911576 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/24a9d347-bb35-4a9c-b489-726098a59823-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24a9d347-bb35-4a9c-b489-726098a59823-maas-api-xks-ca\") pod \"24a9d347-bb35-4a9c-b489-726098a59823\" (UID: \"24a9d347-bb35-4a9c-b489-726098a59823\") " Nov 03 01:41:53.911805 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:53.911755 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24a9d347-bb35-4a9c-b489-726098a59823-kube-api-access-kxh4v\" (UniqueName: \"kubernetes.io/projected/24a9d347-bb35-4a9c-b489-726098a59823-kube-api-access-kxh4v\") pod \"24a9d347-bb35-4a9c-b489-726098a59823\" (UID: \"24a9d347-bb35-4a9c-b489-726098a59823\") " Nov 03 01:41:53.911894 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:53.911870 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24a9d347-bb35-4a9c-b489-726098a59823-maas-api-openshift-ca" pod "24a9d347-bb35-4a9c-b489-726098a59823" (UID: "24a9d347-bb35-4a9c-b489-726098a59823"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:41:53.912466 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:53.912179 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24a9d347-bb35-4a9c-b489-726098a59823-maas-api-openshift-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:53.913624 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:53.913602 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24a9d347-bb35-4a9c-b489-726098a59823-maas-api-xks-ca" pod "24a9d347-bb35-4a9c-b489-726098a59823" (UID: "24a9d347-bb35-4a9c-b489-726098a59823"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:41:53.913854 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:53.913833 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24a9d347-bb35-4a9c-b489-726098a59823-kube-api-access-kxh4v" pod "24a9d347-bb35-4a9c-b489-726098a59823" (UID: "24a9d347-bb35-4a9c-b489-726098a59823"). InnerVolumeSpecName "kube-api-access-kxh4v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:41:53.979021 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:53.978985 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0c98b4a-9b13-4ad5-9e0b-ac50ae14fa5d" path="/var/lib/kubelet/pods/f0c98b4a-9b13-4ad5-9e0b-ac50ae14fa5d/volumes" Nov 03 01:41:54.013674 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:54.013634 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kxh4v\" (UniqueName: \"kubernetes.io/projected/24a9d347-bb35-4a9c-b489-726098a59823-kube-api-access-kxh4v\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:54.013674 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:54.013682 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24a9d347-bb35-4a9c-b489-726098a59823-maas-api-xks-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:54.625515 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:54.625485 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66775a6a-6kwtx" Nov 03 01:41:54.625982 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:54.625513 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66775a6a-6kwtx" event={"ID":"24a9d347-bb35-4a9c-b489-726098a59823","Type":"ContainerDied","Data":"a1fbd250c0b423283f9c0c68c64452d9fd7808368f760a47d2754b966389668a"} Nov 03 01:41:54.625982 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:54.625561 4421 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a1fbd250c0b423283f9c0c68c64452d9fd7808368f760a47d2754b966389668a" Nov 03 01:41:55.092330 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:55.092300 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-fqqx7" Nov 03 01:41:55.237714 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:55.237616 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-83755a/distinct-w3-83755a-kserve-86bdbbdd85-2cnnl"] Nov 03 01:41:55.242808 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:55.242746 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/392e8058-14f9-4dfb-b20e-3e209eb3c625-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/392e8058-14f9-4dfb-b20e-3e209eb3c625-istio-envoy\") pod \"392e8058-14f9-4dfb-b20e-3e209eb3c625\" (UID: \"392e8058-14f9-4dfb-b20e-3e209eb3c625\") " Nov 03 01:41:55.243174 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:55.242857 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/392e8058-14f9-4dfb-b20e-3e209eb3c625-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/392e8058-14f9-4dfb-b20e-3e209eb3c625-workload-socket\") pod \"392e8058-14f9-4dfb-b20e-3e209eb3c625\" (UID: \"392e8058-14f9-4dfb-b20e-3e209eb3c625\") " Nov 03 01:41:55.243341 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:55.243247 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/392e8058-14f9-4dfb-b20e-3e209eb3c625-workload-socket" pod "392e8058-14f9-4dfb-b20e-3e209eb3c625" (UID: "392e8058-14f9-4dfb-b20e-3e209eb3c625"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:41:55.243446 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:55.243423 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/392e8058-14f9-4dfb-b20e-3e209eb3c625-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/392e8058-14f9-4dfb-b20e-3e209eb3c625-istio-podinfo\") pod \"392e8058-14f9-4dfb-b20e-3e209eb3c625\" (UID: \"392e8058-14f9-4dfb-b20e-3e209eb3c625\") " Nov 03 01:41:55.243533 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:55.243492 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/392e8058-14f9-4dfb-b20e-3e209eb3c625-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/392e8058-14f9-4dfb-b20e-3e209eb3c625-istio-data\") pod \"392e8058-14f9-4dfb-b20e-3e209eb3c625\" (UID: \"392e8058-14f9-4dfb-b20e-3e209eb3c625\") " Nov 03 01:41:55.244022 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:55.243998 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/392e8058-14f9-4dfb-b20e-3e209eb3c625-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/392e8058-14f9-4dfb-b20e-3e209eb3c625-workload-certs\") pod \"392e8058-14f9-4dfb-b20e-3e209eb3c625\" (UID: \"392e8058-14f9-4dfb-b20e-3e209eb3c625\") " Nov 03 01:41:55.244145 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:55.244055 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/392e8058-14f9-4dfb-b20e-3e209eb3c625-kube-api-access-g5qmx\" (UniqueName: \"kubernetes.io/projected/392e8058-14f9-4dfb-b20e-3e209eb3c625-kube-api-access-g5qmx\") pod \"392e8058-14f9-4dfb-b20e-3e209eb3c625\" (UID: \"392e8058-14f9-4dfb-b20e-3e209eb3c625\") " Nov 03 01:41:55.244145 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:55.244122 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/392e8058-14f9-4dfb-b20e-3e209eb3c625-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/392e8058-14f9-4dfb-b20e-3e209eb3c625-credential-socket\") pod \"392e8058-14f9-4dfb-b20e-3e209eb3c625\" (UID: \"392e8058-14f9-4dfb-b20e-3e209eb3c625\") " Nov 03 01:41:55.244300 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:55.244177 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/392e8058-14f9-4dfb-b20e-3e209eb3c625-istio-token\" (UniqueName: \"kubernetes.io/projected/392e8058-14f9-4dfb-b20e-3e209eb3c625-istio-token\") pod \"392e8058-14f9-4dfb-b20e-3e209eb3c625\" (UID: \"392e8058-14f9-4dfb-b20e-3e209eb3c625\") " Nov 03 01:41:55.244300 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:55.244205 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/392e8058-14f9-4dfb-b20e-3e209eb3c625-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/392e8058-14f9-4dfb-b20e-3e209eb3c625-istiod-ca-cert\") pod \"392e8058-14f9-4dfb-b20e-3e209eb3c625\" (UID: \"392e8058-14f9-4dfb-b20e-3e209eb3c625\") " Nov 03 01:41:55.244442 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:55.243911 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/392e8058-14f9-4dfb-b20e-3e209eb3c625-istio-data" pod "392e8058-14f9-4dfb-b20e-3e209eb3c625" (UID: "392e8058-14f9-4dfb-b20e-3e209eb3c625"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:41:55.244442 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:55.244409 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/392e8058-14f9-4dfb-b20e-3e209eb3c625-workload-certs" pod "392e8058-14f9-4dfb-b20e-3e209eb3c625" (UID: "392e8058-14f9-4dfb-b20e-3e209eb3c625"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:41:55.244643 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:55.244621 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/392e8058-14f9-4dfb-b20e-3e209eb3c625-credential-socket" pod "392e8058-14f9-4dfb-b20e-3e209eb3c625" (UID: "392e8058-14f9-4dfb-b20e-3e209eb3c625"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:41:55.244806 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:55.244784 4421 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/392e8058-14f9-4dfb-b20e-3e209eb3c625-workload-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:55.244881 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:55.244812 4421 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/392e8058-14f9-4dfb-b20e-3e209eb3c625-istio-data\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:55.244881 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:55.244828 4421 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/392e8058-14f9-4dfb-b20e-3e209eb3c625-workload-certs\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:55.244881 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:55.244842 4421 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/392e8058-14f9-4dfb-b20e-3e209eb3c625-credential-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:55.245245 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:55.245226 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/392e8058-14f9-4dfb-b20e-3e209eb3c625-istiod-ca-cert" pod "392e8058-14f9-4dfb-b20e-3e209eb3c625" (UID: "392e8058-14f9-4dfb-b20e-3e209eb3c625"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:41:55.245878 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:55.245851 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/392e8058-14f9-4dfb-b20e-3e209eb3c625-istio-envoy" pod "392e8058-14f9-4dfb-b20e-3e209eb3c625" (UID: "392e8058-14f9-4dfb-b20e-3e209eb3c625"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:41:55.246121 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:55.246099 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/392e8058-14f9-4dfb-b20e-3e209eb3c625-istio-podinfo" pod "392e8058-14f9-4dfb-b20e-3e209eb3c625" (UID: "392e8058-14f9-4dfb-b20e-3e209eb3c625"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:41:55.247111 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:55.247089 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/392e8058-14f9-4dfb-b20e-3e209eb3c625-kube-api-access-g5qmx" pod "392e8058-14f9-4dfb-b20e-3e209eb3c625" (UID: "392e8058-14f9-4dfb-b20e-3e209eb3c625"). InnerVolumeSpecName "kube-api-access-g5qmx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:41:55.247459 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:55.247440 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/392e8058-14f9-4dfb-b20e-3e209eb3c625-istio-token" pod "392e8058-14f9-4dfb-b20e-3e209eb3c625" (UID: "392e8058-14f9-4dfb-b20e-3e209eb3c625"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:41:55.248596 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:55.248576 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-83755a/premium-w3-83755a-kserve-7c9d66988b-lt4xd"] Nov 03 01:41:55.261408 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:55.261374 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-83755a/simulator-w3-83755a-kserve-77d6664559-psqtg"] Nov 03 01:41:55.346142 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:55.346099 4421 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/392e8058-14f9-4dfb-b20e-3e209eb3c625-istio-envoy\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:55.346142 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:55.346136 4421 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/392e8058-14f9-4dfb-b20e-3e209eb3c625-istio-podinfo\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:55.346142 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:55.346146 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5qmx\" (UniqueName: \"kubernetes.io/projected/392e8058-14f9-4dfb-b20e-3e209eb3c625-kube-api-access-g5qmx\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:55.346142 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:55.346156 4421 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/392e8058-14f9-4dfb-b20e-3e209eb3c625-istio-token\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:55.346473 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:55.346165 4421 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/392e8058-14f9-4dfb-b20e-3e209eb3c625-istiod-ca-cert\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:55.634572 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:55.634539 4421 generic.go:356] "Generic (PLEG): container finished" podID="392e8058-14f9-4dfb-b20e-3e209eb3c625" containerID="9c0f1c5153a027730dca45af068c6bab21690e517eee27a99c60c527f90ccc9a" exitCode=0 Nov 03 01:41:55.635051 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:55.634591 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-fqqx7" Nov 03 01:41:55.635051 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:55.634629 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-fqqx7" event={"ID":"392e8058-14f9-4dfb-b20e-3e209eb3c625","Type":"ContainerDied","Data":"9c0f1c5153a027730dca45af068c6bab21690e517eee27a99c60c527f90ccc9a"} Nov 03 01:41:55.635051 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:55.634688 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-fqqx7" event={"ID":"392e8058-14f9-4dfb-b20e-3e209eb3c625","Type":"ContainerDied","Data":"ad90ee00d3d6134f4e88d41e8b409c8c053a527c7f113aaf4b7b5f08f07bd669"} Nov 03 01:41:55.635051 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:55.634711 4421 scope.go:122] "RemoveContainer" containerID="9c0f1c5153a027730dca45af068c6bab21690e517eee27a99c60c527f90ccc9a" Nov 03 01:41:55.646912 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:55.646895 4421 scope.go:122] "RemoveContainer" containerID="9c0f1c5153a027730dca45af068c6bab21690e517eee27a99c60c527f90ccc9a" Nov 03 01:41:55.647205 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:41:55.647181 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c0f1c5153a027730dca45af068c6bab21690e517eee27a99c60c527f90ccc9a\": container with ID starting with 9c0f1c5153a027730dca45af068c6bab21690e517eee27a99c60c527f90ccc9a not found: ID does not exist" containerID="9c0f1c5153a027730dca45af068c6bab21690e517eee27a99c60c527f90ccc9a" Nov 03 01:41:55.647254 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:55.647213 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c0f1c5153a027730dca45af068c6bab21690e517eee27a99c60c527f90ccc9a"} err="failed to get container status \"9c0f1c5153a027730dca45af068c6bab21690e517eee27a99c60c527f90ccc9a\": rpc error: code = NotFound desc = could not find container \"9c0f1c5153a027730dca45af068c6bab21690e517eee27a99c60c527f90ccc9a\": container with ID starting with 9c0f1c5153a027730dca45af068c6bab21690e517eee27a99c60c527f90ccc9a not found: ID does not exist" Nov 03 01:41:55.660485 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:55.660394 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-fqqx7"] Nov 03 01:41:55.664221 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:55.664200 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-fqqx7"] Nov 03 01:41:55.979235 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:55.979134 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="392e8058-14f9-4dfb-b20e-3e209eb3c625" path="/var/lib/kubelet/pods/392e8058-14f9-4dfb-b20e-3e209eb3c625/volumes" Nov 03 01:41:58.170680 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:58.170629 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-821f5a-7655cbb57c-wgbsw" podUID="61568421-54f2-4ba7-973f-8efa5ab1e805" containerName="payload-processing" containerID="cri-o://97b45b47e745ec0e2a4a7e0edf92413d611bb2ab85559d64835d2cbe6e726fcd" gracePeriod=25 Nov 03 01:41:58.181314 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:58.181232 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-821f5a-84858c594c-pszhq" podUID="542b6309-1f43-413e-a448-b604a68a5047" containerName="payload-pre-processing" containerID="cri-o://4ed7957e2ccc19fddec9b2dc34613e524e0d7cb91be6a03bc5b59c895350bcf5" gracePeriod=25 Nov 03 01:41:59.850678 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:59.850638 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-821f5a-openshift-default-7bfc657d9-bv95q"] Nov 03 01:41:59.851179 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:41:59.850914 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-821f5a-openshift-default-7bfc657d9-bv95q" podUID="ff2ff304-350a-4e21-8415-1c28dcb90a7d" containerName="istio-proxy" containerID="cri-o://9d6092a60568bf598c4fbde5716d6486743852949a4adfa6b39d6adaafcdcb95" gracePeriod=30 Nov 03 01:42:00.157724 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:00.157624 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-821f5a-7655cbb57c-wgbsw" podUID="61568421-54f2-4ba7-973f-8efa5ab1e805" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:42:00.157899 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:00.157870 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-821f5a-84858c594c-pszhq" podUID="542b6309-1f43-413e-a448-b604a68a5047" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:42:01.662964 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:01.662898 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w1-4af1f7/rate-test-model-4af1f7-kserve-6b4d9677cc-6pz6m" podUID="6a57251c-8bd7-449e-ac32-563fe0f99711" containerName="main" containerID="cri-o://d21569a00857a12296071ac8e1289a495e00f991f7d3ca8cf15ece2a0739934a" gracePeriod=45 Nov 03 01:42:01.939054 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:01.939022 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-4af1f7/rate-test-model-4af1f7-kserve-6b4d9677cc-6pz6m" Nov 03 01:42:02.043244 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:02.043212 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a57251c-8bd7-449e-ac32-563fe0f99711-tls-certs\" (UniqueName: \"kubernetes.io/secret/6a57251c-8bd7-449e-ac32-563fe0f99711-tls-certs\") pod \"6a57251c-8bd7-449e-ac32-563fe0f99711\" (UID: \"6a57251c-8bd7-449e-ac32-563fe0f99711\") " Nov 03 01:42:02.043459 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:02.043348 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a57251c-8bd7-449e-ac32-563fe0f99711-home\" (UniqueName: \"kubernetes.io/empty-dir/6a57251c-8bd7-449e-ac32-563fe0f99711-home\") pod \"6a57251c-8bd7-449e-ac32-563fe0f99711\" (UID: \"6a57251c-8bd7-449e-ac32-563fe0f99711\") " Nov 03 01:42:02.043459 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:02.043376 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a57251c-8bd7-449e-ac32-563fe0f99711-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a57251c-8bd7-449e-ac32-563fe0f99711-dshm\") pod \"6a57251c-8bd7-449e-ac32-563fe0f99711\" (UID: \"6a57251c-8bd7-449e-ac32-563fe0f99711\") " Nov 03 01:42:02.043584 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:02.043514 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a57251c-8bd7-449e-ac32-563fe0f99711-kube-api-access-5lstp\" (UniqueName: \"kubernetes.io/projected/6a57251c-8bd7-449e-ac32-563fe0f99711-kube-api-access-5lstp\") pod \"6a57251c-8bd7-449e-ac32-563fe0f99711\" (UID: \"6a57251c-8bd7-449e-ac32-563fe0f99711\") " Nov 03 01:42:02.043584 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:02.043550 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a57251c-8bd7-449e-ac32-563fe0f99711-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a57251c-8bd7-449e-ac32-563fe0f99711-tmp-dir\") pod \"6a57251c-8bd7-449e-ac32-563fe0f99711\" (UID: \"6a57251c-8bd7-449e-ac32-563fe0f99711\") " Nov 03 01:42:02.043704 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:02.043598 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a57251c-8bd7-449e-ac32-563fe0f99711-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a57251c-8bd7-449e-ac32-563fe0f99711-model-cache\") pod \"6a57251c-8bd7-449e-ac32-563fe0f99711\" (UID: \"6a57251c-8bd7-449e-ac32-563fe0f99711\") " Nov 03 01:42:02.043704 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:02.043607 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a57251c-8bd7-449e-ac32-563fe0f99711-home" pod "6a57251c-8bd7-449e-ac32-563fe0f99711" (UID: "6a57251c-8bd7-449e-ac32-563fe0f99711"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:42:02.043878 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:02.043852 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a57251c-8bd7-449e-ac32-563fe0f99711-tmp-dir" pod "6a57251c-8bd7-449e-ac32-563fe0f99711" (UID: "6a57251c-8bd7-449e-ac32-563fe0f99711"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:42:02.043878 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:02.043866 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a57251c-8bd7-449e-ac32-563fe0f99711-model-cache" pod "6a57251c-8bd7-449e-ac32-563fe0f99711" (UID: "6a57251c-8bd7-449e-ac32-563fe0f99711"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:42:02.044138 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:02.044123 4421 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a57251c-8bd7-449e-ac32-563fe0f99711-home\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:02.044189 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:02.044159 4421 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a57251c-8bd7-449e-ac32-563fe0f99711-tmp-dir\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:02.044189 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:02.044171 4421 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a57251c-8bd7-449e-ac32-563fe0f99711-model-cache\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:02.045671 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:02.045641 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a57251c-8bd7-449e-ac32-563fe0f99711-dshm" pod "6a57251c-8bd7-449e-ac32-563fe0f99711" (UID: "6a57251c-8bd7-449e-ac32-563fe0f99711"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:42:02.045948 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:02.045920 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a57251c-8bd7-449e-ac32-563fe0f99711-kube-api-access-5lstp" pod "6a57251c-8bd7-449e-ac32-563fe0f99711" (UID: "6a57251c-8bd7-449e-ac32-563fe0f99711"). InnerVolumeSpecName "kube-api-access-5lstp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:42:02.046033 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:02.045925 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a57251c-8bd7-449e-ac32-563fe0f99711-tls-certs" pod "6a57251c-8bd7-449e-ac32-563fe0f99711" (UID: "6a57251c-8bd7-449e-ac32-563fe0f99711"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:42:02.145127 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:02.145083 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5lstp\" (UniqueName: \"kubernetes.io/projected/6a57251c-8bd7-449e-ac32-563fe0f99711-kube-api-access-5lstp\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:02.145127 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:02.145122 4421 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a57251c-8bd7-449e-ac32-563fe0f99711-tls-certs\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:02.145127 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:02.145133 4421 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a57251c-8bd7-449e-ac32-563fe0f99711-dshm\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:02.266143 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:02.266105 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-200ed79c-fcghv"] Nov 03 01:42:02.266751 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:02.266730 4421 cpu_manager.go:414] "containerMap: removing container" podUID="f0c98b4a-9b13-4ad5-9e0b-ac50ae14fa5d" containerName="istio-proxy" Nov 03 01:42:02.266866 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:02.266751 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0c98b4a-9b13-4ad5-9e0b-ac50ae14fa5d" containerName="istio-proxy" Nov 03 01:42:02.266866 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:02.266774 4421 cpu_manager.go:414] "containerMap: removing container" podUID="392e8058-14f9-4dfb-b20e-3e209eb3c625" containerName="istio-proxy" Nov 03 01:42:02.266866 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:02.266781 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="392e8058-14f9-4dfb-b20e-3e209eb3c625" containerName="istio-proxy" Nov 03 01:42:02.266866 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:02.266793 4421 cpu_manager.go:414] "containerMap: removing container" podUID="38f1d8ba-2f55-4d82-b5e6-0491c36a8009" containerName="payload-pre-processing" Nov 03 01:42:02.266866 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:02.266801 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38f1d8ba-2f55-4d82-b5e6-0491c36a8009" containerName="payload-pre-processing" Nov 03 01:42:02.266866 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:02.266816 4421 cpu_manager.go:414] "containerMap: removing container" podUID="24a9d347-bb35-4a9c-b489-726098a59823" containerName="revoke-keys" Nov 03 01:42:02.266866 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:02.266823 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24a9d347-bb35-4a9c-b489-726098a59823" containerName="revoke-keys" Nov 03 01:42:02.266866 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:02.266831 4421 cpu_manager.go:414] "containerMap: removing container" podUID="f0c98b4a-9b13-4ad5-9e0b-ac50ae14fa5d" containerName="istio-proxy" Nov 03 01:42:02.266866 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:02.266838 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0c98b4a-9b13-4ad5-9e0b-ac50ae14fa5d" containerName="istio-proxy" Nov 03 01:42:02.266866 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:02.266861 4421 cpu_manager.go:414] "containerMap: removing container" podUID="6a57251c-8bd7-449e-ac32-563fe0f99711" containerName="main" Nov 03 01:42:02.266866 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:02.266868 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a57251c-8bd7-449e-ac32-563fe0f99711" containerName="main" Nov 03 01:42:02.267831 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:02.266885 4421 cpu_manager.go:414] "containerMap: removing container" podUID="f0f55109-ee26-4eab-b6ca-b610c783906d" containerName="payload-processing" Nov 03 01:42:02.267831 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:02.266894 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0f55109-ee26-4eab-b6ca-b610c783906d" containerName="payload-processing" Nov 03 01:42:02.267831 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:02.267054 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a57251c-8bd7-449e-ac32-563fe0f99711" containerName="main" Nov 03 01:42:02.267831 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:02.267072 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0f55109-ee26-4eab-b6ca-b610c783906d" containerName="payload-processing" Nov 03 01:42:02.267831 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:02.267089 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0c98b4a-9b13-4ad5-9e0b-ac50ae14fa5d" containerName="istio-proxy" Nov 03 01:42:02.267831 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:02.267100 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="392e8058-14f9-4dfb-b20e-3e209eb3c625" containerName="istio-proxy" Nov 03 01:42:02.267831 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:02.267110 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="38f1d8ba-2f55-4d82-b5e6-0491c36a8009" containerName="payload-pre-processing" Nov 03 01:42:02.267831 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:02.267119 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="24a9d347-bb35-4a9c-b489-726098a59823" containerName="revoke-keys" Nov 03 01:42:02.267831 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:02.267128 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0c98b4a-9b13-4ad5-9e0b-ac50ae14fa5d" containerName="istio-proxy" Nov 03 01:42:02.271037 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:02.271017 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-200ed79c-fcghv" Nov 03 01:42:02.275820 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:02.275794 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 01:42:02.276189 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:02.276164 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6q4xp\"" Nov 03 01:42:02.276336 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:02.276199 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:42:02.276430 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:02.276374 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 01:42:02.280455 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:02.280429 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-200ed79c-fcghv"] Nov 03 01:42:02.347096 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:02.347060 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zc8sr\" (UniqueName: \"kubernetes.io/projected/9c753d73-2ccb-410f-901d-8d19b175c510-kube-api-access-zc8sr\") pod \"maas-api-revoke-sub-200ed79c-fcghv\" (UID: \"9c753d73-2ccb-410f-901d-8d19b175c510\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-200ed79c-fcghv" Nov 03 01:42:02.347317 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:02.347141 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c753d73-2ccb-410f-901d-8d19b175c510-maas-api-xks-ca\") pod \"maas-api-revoke-sub-200ed79c-fcghv\" (UID: \"9c753d73-2ccb-410f-901d-8d19b175c510\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-200ed79c-fcghv" Nov 03 01:42:02.347317 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:02.347257 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c753d73-2ccb-410f-901d-8d19b175c510-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-200ed79c-fcghv\" (UID: \"9c753d73-2ccb-410f-901d-8d19b175c510\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-200ed79c-fcghv" Nov 03 01:42:02.448669 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:02.448626 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c753d73-2ccb-410f-901d-8d19b175c510-maas-api-xks-ca\") pod \"maas-api-revoke-sub-200ed79c-fcghv\" (UID: \"9c753d73-2ccb-410f-901d-8d19b175c510\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-200ed79c-fcghv" Nov 03 01:42:02.448845 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:02.448696 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c753d73-2ccb-410f-901d-8d19b175c510-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-200ed79c-fcghv\" (UID: \"9c753d73-2ccb-410f-901d-8d19b175c510\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-200ed79c-fcghv" Nov 03 01:42:02.448845 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:02.448802 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zc8sr\" (UniqueName: \"kubernetes.io/projected/9c753d73-2ccb-410f-901d-8d19b175c510-kube-api-access-zc8sr\") pod \"maas-api-revoke-sub-200ed79c-fcghv\" (UID: \"9c753d73-2ccb-410f-901d-8d19b175c510\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-200ed79c-fcghv" Nov 03 01:42:02.449510 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:02.449489 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c753d73-2ccb-410f-901d-8d19b175c510-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-200ed79c-fcghv\" (UID: \"9c753d73-2ccb-410f-901d-8d19b175c510\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-200ed79c-fcghv" Nov 03 01:42:02.451164 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:02.451144 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c753d73-2ccb-410f-901d-8d19b175c510-maas-api-xks-ca\") pod \"maas-api-revoke-sub-200ed79c-fcghv\" (UID: \"9c753d73-2ccb-410f-901d-8d19b175c510\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-200ed79c-fcghv" Nov 03 01:42:02.456535 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:02.456510 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zc8sr\" (UniqueName: \"kubernetes.io/projected/9c753d73-2ccb-410f-901d-8d19b175c510-kube-api-access-zc8sr\") pod \"maas-api-revoke-sub-200ed79c-fcghv\" (UID: \"9c753d73-2ccb-410f-901d-8d19b175c510\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-200ed79c-fcghv" Nov 03 01:42:02.588536 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:02.588134 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-200ed79c-fcghv" Nov 03 01:42:02.694318 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:02.694250 4421 generic.go:356] "Generic (PLEG): container finished" podID="6a57251c-8bd7-449e-ac32-563fe0f99711" containerID="d21569a00857a12296071ac8e1289a495e00f991f7d3ca8cf15ece2a0739934a" exitCode=0 Nov 03 01:42:02.694819 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:02.694311 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-4af1f7/rate-test-model-4af1f7-kserve-6b4d9677cc-6pz6m" event={"ID":"6a57251c-8bd7-449e-ac32-563fe0f99711","Type":"ContainerDied","Data":"d21569a00857a12296071ac8e1289a495e00f991f7d3ca8cf15ece2a0739934a"} Nov 03 01:42:02.694819 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:02.694331 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-4af1f7/rate-test-model-4af1f7-kserve-6b4d9677cc-6pz6m" Nov 03 01:42:02.694819 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:02.694367 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-4af1f7/rate-test-model-4af1f7-kserve-6b4d9677cc-6pz6m" event={"ID":"6a57251c-8bd7-449e-ac32-563fe0f99711","Type":"ContainerDied","Data":"dadabe136be9daf89ca24bcd994fea9f410befca57443bbf066fe568fca43540"} Nov 03 01:42:02.694819 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:02.694388 4421 scope.go:122] "RemoveContainer" containerID="d21569a00857a12296071ac8e1289a495e00f991f7d3ca8cf15ece2a0739934a" Nov 03 01:42:02.707686 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:02.707663 4421 scope.go:122] "RemoveContainer" containerID="d21569a00857a12296071ac8e1289a495e00f991f7d3ca8cf15ece2a0739934a" Nov 03 01:42:02.708074 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:42:02.708046 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d21569a00857a12296071ac8e1289a495e00f991f7d3ca8cf15ece2a0739934a\": container with ID starting with d21569a00857a12296071ac8e1289a495e00f991f7d3ca8cf15ece2a0739934a not found: ID does not exist" containerID="d21569a00857a12296071ac8e1289a495e00f991f7d3ca8cf15ece2a0739934a" Nov 03 01:42:02.708133 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:02.708091 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d21569a00857a12296071ac8e1289a495e00f991f7d3ca8cf15ece2a0739934a"} err="failed to get container status \"d21569a00857a12296071ac8e1289a495e00f991f7d3ca8cf15ece2a0739934a\": rpc error: code = NotFound desc = could not find container \"d21569a00857a12296071ac8e1289a495e00f991f7d3ca8cf15ece2a0739934a\": container with ID starting with d21569a00857a12296071ac8e1289a495e00f991f7d3ca8cf15ece2a0739934a not found: ID does not exist" Nov 03 01:42:02.722785 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:02.722755 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-4af1f7/rate-test-model-4af1f7-kserve-6b4d9677cc-6pz6m"] Nov 03 01:42:02.725964 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:02.725939 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w1-4af1f7/rate-test-model-4af1f7-kserve-6b4d9677cc-6pz6m"] Nov 03 01:42:02.872693 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:02.872666 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-200ed79c-fcghv"] Nov 03 01:42:02.873891 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:42:02.873858 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c753d73_2ccb_410f_901d_8d19b175c510.slice/crio-c39a972f9370dda86f4005f51c78d45a85adaa0c8ea57967ce6ebf78efa27a1d WatchSource:0}: Error finding container c39a972f9370dda86f4005f51c78d45a85adaa0c8ea57967ce6ebf78efa27a1d: Status 404 returned error can't find the container with id c39a972f9370dda86f4005f51c78d45a85adaa0c8ea57967ce6ebf78efa27a1d Nov 03 01:42:03.702847 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:03.702804 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-200ed79c-fcghv" event={"ID":"9c753d73-2ccb-410f-901d-8d19b175c510","Type":"ContainerStarted","Data":"fe07e5ecf5cce2ef4a8039ec70ccefba7d8e2c79c8a7313f99a7c0509f203719"} Nov 03 01:42:03.703355 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:03.702871 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-200ed79c-fcghv" event={"ID":"9c753d73-2ccb-410f-901d-8d19b175c510","Type":"ContainerStarted","Data":"c39a972f9370dda86f4005f51c78d45a85adaa0c8ea57967ce6ebf78efa27a1d"} Nov 03 01:42:03.718332 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:03.718250 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-200ed79c-fcghv" podStartSLOduration=1.718231926 podStartE2EDuration="1.718231926s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:42:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:42:03.717908519 +0000 UTC m=+3665.888911616" watchObservedRunningTime="2026-11-03 01:42:03.718231926 +0000 UTC m=+3665.889235024" Nov 03 01:42:03.979968 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:03.979932 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a57251c-8bd7-449e-ac32-563fe0f99711" path="/var/lib/kubelet/pods/6a57251c-8bd7-449e-ac32-563fe0f99711/volumes" Nov 03 01:42:04.712701 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:04.712663 4421 generic.go:356] "Generic (PLEG): container finished" podID="9c753d73-2ccb-410f-901d-8d19b175c510" containerID="fe07e5ecf5cce2ef4a8039ec70ccefba7d8e2c79c8a7313f99a7c0509f203719" exitCode=0 Nov 03 01:42:04.713130 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:04.712750 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-200ed79c-fcghv" event={"ID":"9c753d73-2ccb-410f-901d-8d19b175c510","Type":"ContainerDied","Data":"fe07e5ecf5cce2ef4a8039ec70ccefba7d8e2c79c8a7313f99a7c0509f203719"} Nov 03 01:42:04.741223 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:04.741158 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-650be0/premium-w4-650be0-kserve-7bf8555dfb-z97n5" podUID="40601dc5-808f-456f-826f-6e0c35966ada" containerName="main" containerID="cri-o://0e06adc10ca35db139d1d5f7954e16ef1af7e2c4f7d1c6a891f85baaf4de03fb" gracePeriod=45 Nov 03 01:42:04.744673 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:04.744605 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-650be0/simulator-w4-650be0-kserve-7bc588774f-gvz2k" podUID="a827fa88-f078-4312-8f12-2c799c11b461" containerName="main" containerID="cri-o://ad7a4e337194ba2f2ac79ba4bc0d9251212a38a57d514c25b56b24fd41f233a2" gracePeriod=45 Nov 03 01:42:04.810734 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:04.810690 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-650be0/premium-w4-650be0-kserve-7bf8555dfb-z97n5" podUID="40601dc5-808f-456f-826f-6e0c35966ada" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": dial tcp 10.134.0.65:8000: connect: connection refused" Nov 03 01:42:05.029154 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:05.029115 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-650be0/premium-w4-650be0-kserve-7bf8555dfb-z97n5" Nov 03 01:42:05.044411 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:05.044382 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-650be0/simulator-w4-650be0-kserve-7bc588774f-gvz2k" Nov 03 01:42:05.090885 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:05.090851 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a827fa88-f078-4312-8f12-2c799c11b461-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a827fa88-f078-4312-8f12-2c799c11b461-dshm\") pod \"a827fa88-f078-4312-8f12-2c799c11b461\" (UID: \"a827fa88-f078-4312-8f12-2c799c11b461\") " Nov 03 01:42:05.090885 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:05.090893 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a827fa88-f078-4312-8f12-2c799c11b461-tls-certs\" (UniqueName: \"kubernetes.io/secret/a827fa88-f078-4312-8f12-2c799c11b461-tls-certs\") pod \"a827fa88-f078-4312-8f12-2c799c11b461\" (UID: \"a827fa88-f078-4312-8f12-2c799c11b461\") " Nov 03 01:42:05.091141 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:05.090931 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/40601dc5-808f-456f-826f-6e0c35966ada-tls-certs\" (UniqueName: \"kubernetes.io/secret/40601dc5-808f-456f-826f-6e0c35966ada-tls-certs\") pod \"40601dc5-808f-456f-826f-6e0c35966ada\" (UID: \"40601dc5-808f-456f-826f-6e0c35966ada\") " Nov 03 01:42:05.091141 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:05.090990 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40601dc5-808f-456f-826f-6e0c35966ada-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40601dc5-808f-456f-826f-6e0c35966ada-tmp-dir\") pod \"40601dc5-808f-456f-826f-6e0c35966ada\" (UID: \"40601dc5-808f-456f-826f-6e0c35966ada\") " Nov 03 01:42:05.091294 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:05.091249 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40601dc5-808f-456f-826f-6e0c35966ada-tmp-dir" pod "40601dc5-808f-456f-826f-6e0c35966ada" (UID: "40601dc5-808f-456f-826f-6e0c35966ada"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:42:05.091516 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:05.091494 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a827fa88-f078-4312-8f12-2c799c11b461-kube-api-access-r99xx\" (UniqueName: \"kubernetes.io/projected/a827fa88-f078-4312-8f12-2c799c11b461-kube-api-access-r99xx\") pod \"a827fa88-f078-4312-8f12-2c799c11b461\" (UID: \"a827fa88-f078-4312-8f12-2c799c11b461\") " Nov 03 01:42:05.091609 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:05.091588 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a827fa88-f078-4312-8f12-2c799c11b461-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a827fa88-f078-4312-8f12-2c799c11b461-tmp-dir\") pod \"a827fa88-f078-4312-8f12-2c799c11b461\" (UID: \"a827fa88-f078-4312-8f12-2c799c11b461\") " Nov 03 01:42:05.091685 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:05.091639 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40601dc5-808f-456f-826f-6e0c35966ada-home\" (UniqueName: \"kubernetes.io/empty-dir/40601dc5-808f-456f-826f-6e0c35966ada-home\") pod \"40601dc5-808f-456f-826f-6e0c35966ada\" (UID: \"40601dc5-808f-456f-826f-6e0c35966ada\") " Nov 03 01:42:05.091685 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:05.091675 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40601dc5-808f-456f-826f-6e0c35966ada-dshm\" (UniqueName: \"kubernetes.io/empty-dir/40601dc5-808f-456f-826f-6e0c35966ada-dshm\") pod \"40601dc5-808f-456f-826f-6e0c35966ada\" (UID: \"40601dc5-808f-456f-826f-6e0c35966ada\") " Nov 03 01:42:05.091810 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:05.091722 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a827fa88-f078-4312-8f12-2c799c11b461-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a827fa88-f078-4312-8f12-2c799c11b461-model-cache\") pod \"a827fa88-f078-4312-8f12-2c799c11b461\" (UID: \"a827fa88-f078-4312-8f12-2c799c11b461\") " Nov 03 01:42:05.091875 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:05.091859 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40601dc5-808f-456f-826f-6e0c35966ada-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40601dc5-808f-456f-826f-6e0c35966ada-model-cache\") pod \"40601dc5-808f-456f-826f-6e0c35966ada\" (UID: \"40601dc5-808f-456f-826f-6e0c35966ada\") " Nov 03 01:42:05.091938 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:05.091907 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a827fa88-f078-4312-8f12-2c799c11b461-home\" (UniqueName: \"kubernetes.io/empty-dir/a827fa88-f078-4312-8f12-2c799c11b461-home\") pod \"a827fa88-f078-4312-8f12-2c799c11b461\" (UID: \"a827fa88-f078-4312-8f12-2c799c11b461\") " Nov 03 01:42:05.092001 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:05.091955 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40601dc5-808f-456f-826f-6e0c35966ada-kube-api-access-7f4hv\" (UniqueName: \"kubernetes.io/projected/40601dc5-808f-456f-826f-6e0c35966ada-kube-api-access-7f4hv\") pod \"40601dc5-808f-456f-826f-6e0c35966ada\" (UID: \"40601dc5-808f-456f-826f-6e0c35966ada\") " Nov 03 01:42:05.092508 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:05.092473 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a827fa88-f078-4312-8f12-2c799c11b461-model-cache" pod "a827fa88-f078-4312-8f12-2c799c11b461" (UID: "a827fa88-f078-4312-8f12-2c799c11b461"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:42:05.092774 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:05.092747 4421 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40601dc5-808f-456f-826f-6e0c35966ada-tmp-dir\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:05.092876 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:05.092781 4421 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a827fa88-f078-4312-8f12-2c799c11b461-model-cache\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:05.092995 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:05.092969 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40601dc5-808f-456f-826f-6e0c35966ada-model-cache" pod "40601dc5-808f-456f-826f-6e0c35966ada" (UID: "40601dc5-808f-456f-826f-6e0c35966ada"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:42:05.093217 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:05.093195 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a827fa88-f078-4312-8f12-2c799c11b461-home" pod "a827fa88-f078-4312-8f12-2c799c11b461" (UID: "a827fa88-f078-4312-8f12-2c799c11b461"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:42:05.093428 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:05.093400 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a827fa88-f078-4312-8f12-2c799c11b461-tmp-dir" pod "a827fa88-f078-4312-8f12-2c799c11b461" (UID: "a827fa88-f078-4312-8f12-2c799c11b461"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:42:05.093951 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:05.093922 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a827fa88-f078-4312-8f12-2c799c11b461-tls-certs" pod "a827fa88-f078-4312-8f12-2c799c11b461" (UID: "a827fa88-f078-4312-8f12-2c799c11b461"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:42:05.094472 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:05.094434 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40601dc5-808f-456f-826f-6e0c35966ada-home" pod "40601dc5-808f-456f-826f-6e0c35966ada" (UID: "40601dc5-808f-456f-826f-6e0c35966ada"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:42:05.095234 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:05.095119 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a827fa88-f078-4312-8f12-2c799c11b461-kube-api-access-r99xx" pod "a827fa88-f078-4312-8f12-2c799c11b461" (UID: "a827fa88-f078-4312-8f12-2c799c11b461"). InnerVolumeSpecName "kube-api-access-r99xx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:42:05.095234 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:05.095212 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40601dc5-808f-456f-826f-6e0c35966ada-tls-certs" pod "40601dc5-808f-456f-826f-6e0c35966ada" (UID: "40601dc5-808f-456f-826f-6e0c35966ada"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:42:05.095516 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:05.095490 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a827fa88-f078-4312-8f12-2c799c11b461-dshm" pod "a827fa88-f078-4312-8f12-2c799c11b461" (UID: "a827fa88-f078-4312-8f12-2c799c11b461"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:42:05.096655 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:05.096629 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40601dc5-808f-456f-826f-6e0c35966ada-dshm" pod "40601dc5-808f-456f-826f-6e0c35966ada" (UID: "40601dc5-808f-456f-826f-6e0c35966ada"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:42:05.096779 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:05.096758 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40601dc5-808f-456f-826f-6e0c35966ada-kube-api-access-7f4hv" pod "40601dc5-808f-456f-826f-6e0c35966ada" (UID: "40601dc5-808f-456f-826f-6e0c35966ada"). InnerVolumeSpecName "kube-api-access-7f4hv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:42:05.179132 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:05.179103 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-821f5a-openshift-default-7bfc657d9-bv95q" Nov 03 01:42:05.193512 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:05.193481 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r99xx\" (UniqueName: \"kubernetes.io/projected/a827fa88-f078-4312-8f12-2c799c11b461-kube-api-access-r99xx\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:05.193512 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:05.193512 4421 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a827fa88-f078-4312-8f12-2c799c11b461-tmp-dir\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:05.193679 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:05.193527 4421 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40601dc5-808f-456f-826f-6e0c35966ada-home\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:05.193679 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:05.193536 4421 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40601dc5-808f-456f-826f-6e0c35966ada-dshm\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:05.193679 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:05.193544 4421 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40601dc5-808f-456f-826f-6e0c35966ada-model-cache\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:05.193679 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:05.193554 4421 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a827fa88-f078-4312-8f12-2c799c11b461-home\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:05.193679 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:05.193564 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7f4hv\" (UniqueName: \"kubernetes.io/projected/40601dc5-808f-456f-826f-6e0c35966ada-kube-api-access-7f4hv\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:05.193679 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:05.193573 4421 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a827fa88-f078-4312-8f12-2c799c11b461-dshm\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:05.193679 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:05.193581 4421 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a827fa88-f078-4312-8f12-2c799c11b461-tls-certs\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:05.193679 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:05.193590 4421 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40601dc5-808f-456f-826f-6e0c35966ada-tls-certs\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:05.294643 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:05.294607 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff2ff304-350a-4e21-8415-1c28dcb90a7d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff2ff304-350a-4e21-8415-1c28dcb90a7d-credential-socket\") pod \"ff2ff304-350a-4e21-8415-1c28dcb90a7d\" (UID: \"ff2ff304-350a-4e21-8415-1c28dcb90a7d\") " Nov 03 01:42:05.294843 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:05.294695 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff2ff304-350a-4e21-8415-1c28dcb90a7d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff2ff304-350a-4e21-8415-1c28dcb90a7d-istio-data\") pod \"ff2ff304-350a-4e21-8415-1c28dcb90a7d\" (UID: \"ff2ff304-350a-4e21-8415-1c28dcb90a7d\") " Nov 03 01:42:05.294843 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:05.294718 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff2ff304-350a-4e21-8415-1c28dcb90a7d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff2ff304-350a-4e21-8415-1c28dcb90a7d-workload-socket\") pod \"ff2ff304-350a-4e21-8415-1c28dcb90a7d\" (UID: \"ff2ff304-350a-4e21-8415-1c28dcb90a7d\") " Nov 03 01:42:05.294843 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:05.294779 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff2ff304-350a-4e21-8415-1c28dcb90a7d-kube-api-access-pj4gx\" (UniqueName: \"kubernetes.io/projected/ff2ff304-350a-4e21-8415-1c28dcb90a7d-kube-api-access-pj4gx\") pod \"ff2ff304-350a-4e21-8415-1c28dcb90a7d\" (UID: \"ff2ff304-350a-4e21-8415-1c28dcb90a7d\") " Nov 03 01:42:05.294843 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:05.294808 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ff2ff304-350a-4e21-8415-1c28dcb90a7d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff2ff304-350a-4e21-8415-1c28dcb90a7d-istio-podinfo\") pod \"ff2ff304-350a-4e21-8415-1c28dcb90a7d\" (UID: \"ff2ff304-350a-4e21-8415-1c28dcb90a7d\") " Nov 03 01:42:05.295082 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:05.294843 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff2ff304-350a-4e21-8415-1c28dcb90a7d-istio-token\" (UniqueName: \"kubernetes.io/projected/ff2ff304-350a-4e21-8415-1c28dcb90a7d-istio-token\") pod \"ff2ff304-350a-4e21-8415-1c28dcb90a7d\" (UID: \"ff2ff304-350a-4e21-8415-1c28dcb90a7d\") " Nov 03 01:42:05.295082 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:05.294969 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff2ff304-350a-4e21-8415-1c28dcb90a7d-credential-socket" pod "ff2ff304-350a-4e21-8415-1c28dcb90a7d" (UID: "ff2ff304-350a-4e21-8415-1c28dcb90a7d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:42:05.295082 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:05.295030 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff2ff304-350a-4e21-8415-1c28dcb90a7d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff2ff304-350a-4e21-8415-1c28dcb90a7d-istiod-ca-cert\") pod \"ff2ff304-350a-4e21-8415-1c28dcb90a7d\" (UID: \"ff2ff304-350a-4e21-8415-1c28dcb90a7d\") " Nov 03 01:42:05.295249 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:05.295130 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff2ff304-350a-4e21-8415-1c28dcb90a7d-workload-socket" pod "ff2ff304-350a-4e21-8415-1c28dcb90a7d" (UID: "ff2ff304-350a-4e21-8415-1c28dcb90a7d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:42:05.295249 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:05.295156 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff2ff304-350a-4e21-8415-1c28dcb90a7d-istio-data" pod "ff2ff304-350a-4e21-8415-1c28dcb90a7d" (UID: "ff2ff304-350a-4e21-8415-1c28dcb90a7d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:42:05.295249 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:05.295173 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff2ff304-350a-4e21-8415-1c28dcb90a7d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff2ff304-350a-4e21-8415-1c28dcb90a7d-workload-certs\") pod \"ff2ff304-350a-4e21-8415-1c28dcb90a7d\" (UID: \"ff2ff304-350a-4e21-8415-1c28dcb90a7d\") " Nov 03 01:42:05.295249 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:05.295237 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff2ff304-350a-4e21-8415-1c28dcb90a7d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff2ff304-350a-4e21-8415-1c28dcb90a7d-istio-envoy\") pod \"ff2ff304-350a-4e21-8415-1c28dcb90a7d\" (UID: \"ff2ff304-350a-4e21-8415-1c28dcb90a7d\") " Nov 03 01:42:05.295459 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:05.295433 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff2ff304-350a-4e21-8415-1c28dcb90a7d-workload-certs" pod "ff2ff304-350a-4e21-8415-1c28dcb90a7d" (UID: "ff2ff304-350a-4e21-8415-1c28dcb90a7d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:42:05.295522 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:05.295480 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff2ff304-350a-4e21-8415-1c28dcb90a7d-istiod-ca-cert" pod "ff2ff304-350a-4e21-8415-1c28dcb90a7d" (UID: "ff2ff304-350a-4e21-8415-1c28dcb90a7d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:42:05.295800 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:05.295765 4421 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff2ff304-350a-4e21-8415-1c28dcb90a7d-istio-data\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:05.295800 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:05.295801 4421 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff2ff304-350a-4e21-8415-1c28dcb90a7d-workload-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:05.295800 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:05.295813 4421 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff2ff304-350a-4e21-8415-1c28dcb90a7d-istiod-ca-cert\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:05.296041 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:05.295824 4421 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff2ff304-350a-4e21-8415-1c28dcb90a7d-workload-certs\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:05.296041 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:05.295839 4421 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff2ff304-350a-4e21-8415-1c28dcb90a7d-credential-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:05.297287 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:05.297244 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff2ff304-350a-4e21-8415-1c28dcb90a7d-kube-api-access-pj4gx" pod "ff2ff304-350a-4e21-8415-1c28dcb90a7d" (UID: "ff2ff304-350a-4e21-8415-1c28dcb90a7d"). InnerVolumeSpecName "kube-api-access-pj4gx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:42:05.297381 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:05.297330 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff2ff304-350a-4e21-8415-1c28dcb90a7d-istio-envoy" pod "ff2ff304-350a-4e21-8415-1c28dcb90a7d" (UID: "ff2ff304-350a-4e21-8415-1c28dcb90a7d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:42:05.297617 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:05.297583 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ff2ff304-350a-4e21-8415-1c28dcb90a7d-istio-podinfo" pod "ff2ff304-350a-4e21-8415-1c28dcb90a7d" (UID: "ff2ff304-350a-4e21-8415-1c28dcb90a7d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:42:05.297617 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:05.297601 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff2ff304-350a-4e21-8415-1c28dcb90a7d-istio-token" pod "ff2ff304-350a-4e21-8415-1c28dcb90a7d" (UID: "ff2ff304-350a-4e21-8415-1c28dcb90a7d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:42:05.397024 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:05.396985 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pj4gx\" (UniqueName: \"kubernetes.io/projected/ff2ff304-350a-4e21-8415-1c28dcb90a7d-kube-api-access-pj4gx\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:05.397024 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:05.397019 4421 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff2ff304-350a-4e21-8415-1c28dcb90a7d-istio-podinfo\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:05.397024 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:05.397030 4421 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff2ff304-350a-4e21-8415-1c28dcb90a7d-istio-token\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:05.397312 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:05.397041 4421 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff2ff304-350a-4e21-8415-1c28dcb90a7d-istio-envoy\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:05.721512 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:05.721417 4421 generic.go:356] "Generic (PLEG): container finished" podID="40601dc5-808f-456f-826f-6e0c35966ada" containerID="0e06adc10ca35db139d1d5f7954e16ef1af7e2c4f7d1c6a891f85baaf4de03fb" exitCode=0 Nov 03 01:42:05.721512 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:05.721470 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-650be0/premium-w4-650be0-kserve-7bf8555dfb-z97n5" Nov 03 01:42:05.722029 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:05.721508 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-650be0/premium-w4-650be0-kserve-7bf8555dfb-z97n5" event={"ID":"40601dc5-808f-456f-826f-6e0c35966ada","Type":"ContainerDied","Data":"0e06adc10ca35db139d1d5f7954e16ef1af7e2c4f7d1c6a891f85baaf4de03fb"} Nov 03 01:42:05.722029 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:05.721555 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-650be0/premium-w4-650be0-kserve-7bf8555dfb-z97n5" event={"ID":"40601dc5-808f-456f-826f-6e0c35966ada","Type":"ContainerDied","Data":"87d84786c0e81f9cc4000674d9160fbc1492c6ec2b42588206fae48b26988092"} Nov 03 01:42:05.722029 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:05.721576 4421 scope.go:122] "RemoveContainer" containerID="0e06adc10ca35db139d1d5f7954e16ef1af7e2c4f7d1c6a891f85baaf4de03fb" Nov 03 01:42:05.723408 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:05.723357 4421 generic.go:356] "Generic (PLEG): container finished" podID="a827fa88-f078-4312-8f12-2c799c11b461" containerID="ad7a4e337194ba2f2ac79ba4bc0d9251212a38a57d514c25b56b24fd41f233a2" exitCode=0 Nov 03 01:42:05.723547 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:05.723412 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-650be0/simulator-w4-650be0-kserve-7bc588774f-gvz2k" Nov 03 01:42:05.723547 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:05.723446 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-650be0/simulator-w4-650be0-kserve-7bc588774f-gvz2k" event={"ID":"a827fa88-f078-4312-8f12-2c799c11b461","Type":"ContainerDied","Data":"ad7a4e337194ba2f2ac79ba4bc0d9251212a38a57d514c25b56b24fd41f233a2"} Nov 03 01:42:05.723547 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:05.723506 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-650be0/simulator-w4-650be0-kserve-7bc588774f-gvz2k" event={"ID":"a827fa88-f078-4312-8f12-2c799c11b461","Type":"ContainerDied","Data":"295ac683966dd8ccb6c1db5f9ca9e5fbc9f2506375901e3723514c614957a636"} Nov 03 01:42:05.725569 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:05.725545 4421 generic.go:356] "Generic (PLEG): container finished" podID="ff2ff304-350a-4e21-8415-1c28dcb90a7d" containerID="9d6092a60568bf598c4fbde5716d6486743852949a4adfa6b39d6adaafcdcb95" exitCode=0 Nov 03 01:42:05.725705 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:05.725592 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-821f5a-openshift-default-7bfc657d9-bv95q" Nov 03 01:42:05.725705 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:05.725625 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-821f5a-openshift-default-7bfc657d9-bv95q" event={"ID":"ff2ff304-350a-4e21-8415-1c28dcb90a7d","Type":"ContainerDied","Data":"9d6092a60568bf598c4fbde5716d6486743852949a4adfa6b39d6adaafcdcb95"} Nov 03 01:42:05.725705 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:05.725689 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-821f5a-openshift-default-7bfc657d9-bv95q" event={"ID":"ff2ff304-350a-4e21-8415-1c28dcb90a7d","Type":"ContainerDied","Data":"c08ab3d5b43e3348801137be9f4f578011e53c020cbf1309aac421a75e352225"} Nov 03 01:42:05.735117 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:05.735099 4421 scope.go:122] "RemoveContainer" containerID="0e06adc10ca35db139d1d5f7954e16ef1af7e2c4f7d1c6a891f85baaf4de03fb" Nov 03 01:42:05.736035 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:42:05.735519 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e06adc10ca35db139d1d5f7954e16ef1af7e2c4f7d1c6a891f85baaf4de03fb\": container with ID starting with 0e06adc10ca35db139d1d5f7954e16ef1af7e2c4f7d1c6a891f85baaf4de03fb not found: ID does not exist" containerID="0e06adc10ca35db139d1d5f7954e16ef1af7e2c4f7d1c6a891f85baaf4de03fb" Nov 03 01:42:05.736035 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:05.735572 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e06adc10ca35db139d1d5f7954e16ef1af7e2c4f7d1c6a891f85baaf4de03fb"} err="failed to get container status \"0e06adc10ca35db139d1d5f7954e16ef1af7e2c4f7d1c6a891f85baaf4de03fb\": rpc error: code = NotFound desc = could not find container \"0e06adc10ca35db139d1d5f7954e16ef1af7e2c4f7d1c6a891f85baaf4de03fb\": container with ID starting with 0e06adc10ca35db139d1d5f7954e16ef1af7e2c4f7d1c6a891f85baaf4de03fb not found: ID does not exist" Nov 03 01:42:05.736035 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:05.735602 4421 scope.go:122] "RemoveContainer" containerID="ad7a4e337194ba2f2ac79ba4bc0d9251212a38a57d514c25b56b24fd41f233a2" Nov 03 01:42:05.752223 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:05.752136 4421 scope.go:122] "RemoveContainer" containerID="ad7a4e337194ba2f2ac79ba4bc0d9251212a38a57d514c25b56b24fd41f233a2" Nov 03 01:42:05.752842 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:42:05.752814 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad7a4e337194ba2f2ac79ba4bc0d9251212a38a57d514c25b56b24fd41f233a2\": container with ID starting with ad7a4e337194ba2f2ac79ba4bc0d9251212a38a57d514c25b56b24fd41f233a2 not found: ID does not exist" containerID="ad7a4e337194ba2f2ac79ba4bc0d9251212a38a57d514c25b56b24fd41f233a2" Nov 03 01:42:05.752968 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:05.752850 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad7a4e337194ba2f2ac79ba4bc0d9251212a38a57d514c25b56b24fd41f233a2"} err="failed to get container status \"ad7a4e337194ba2f2ac79ba4bc0d9251212a38a57d514c25b56b24fd41f233a2\": rpc error: code = NotFound desc = could not find container \"ad7a4e337194ba2f2ac79ba4bc0d9251212a38a57d514c25b56b24fd41f233a2\": container with ID starting with ad7a4e337194ba2f2ac79ba4bc0d9251212a38a57d514c25b56b24fd41f233a2 not found: ID does not exist" Nov 03 01:42:05.752968 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:05.752868 4421 scope.go:122] "RemoveContainer" containerID="9d6092a60568bf598c4fbde5716d6486743852949a4adfa6b39d6adaafcdcb95" Nov 03 01:42:05.753938 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:05.753918 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-650be0/premium-w4-650be0-kserve-7bf8555dfb-z97n5"] Nov 03 01:42:05.757268 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:05.757244 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-650be0/premium-w4-650be0-kserve-7bf8555dfb-z97n5"] Nov 03 01:42:05.763223 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:05.763196 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-821f5a-openshift-default-7bfc657d9-bv95q"] Nov 03 01:42:05.769787 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:05.769764 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-821f5a-openshift-default-7bfc657d9-bv95q"] Nov 03 01:42:05.769865 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:05.769804 4421 scope.go:122] "RemoveContainer" containerID="06d83afdaa519e9974b129def6ada0b1fa89a128b24d2597a074e887c7956f04" Nov 03 01:42:05.783481 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:05.783457 4421 scope.go:122] "RemoveContainer" containerID="9d6092a60568bf598c4fbde5716d6486743852949a4adfa6b39d6adaafcdcb95" Nov 03 01:42:05.783782 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:42:05.783764 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d6092a60568bf598c4fbde5716d6486743852949a4adfa6b39d6adaafcdcb95\": container with ID starting with 9d6092a60568bf598c4fbde5716d6486743852949a4adfa6b39d6adaafcdcb95 not found: ID does not exist" containerID="9d6092a60568bf598c4fbde5716d6486743852949a4adfa6b39d6adaafcdcb95" Nov 03 01:42:05.783846 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:05.783791 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d6092a60568bf598c4fbde5716d6486743852949a4adfa6b39d6adaafcdcb95"} err="failed to get container status \"9d6092a60568bf598c4fbde5716d6486743852949a4adfa6b39d6adaafcdcb95\": rpc error: code = NotFound desc = could not find container \"9d6092a60568bf598c4fbde5716d6486743852949a4adfa6b39d6adaafcdcb95\": container with ID starting with 9d6092a60568bf598c4fbde5716d6486743852949a4adfa6b39d6adaafcdcb95 not found: ID does not exist" Nov 03 01:42:05.783846 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:05.783824 4421 scope.go:122] "RemoveContainer" containerID="06d83afdaa519e9974b129def6ada0b1fa89a128b24d2597a074e887c7956f04" Nov 03 01:42:05.784056 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:42:05.784038 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06d83afdaa519e9974b129def6ada0b1fa89a128b24d2597a074e887c7956f04\": container with ID starting with 06d83afdaa519e9974b129def6ada0b1fa89a128b24d2597a074e887c7956f04 not found: ID does not exist" containerID="06d83afdaa519e9974b129def6ada0b1fa89a128b24d2597a074e887c7956f04" Nov 03 01:42:05.784119 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:05.784060 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06d83afdaa519e9974b129def6ada0b1fa89a128b24d2597a074e887c7956f04"} err="failed to get container status \"06d83afdaa519e9974b129def6ada0b1fa89a128b24d2597a074e887c7956f04\": rpc error: code = NotFound desc = could not find container \"06d83afdaa519e9974b129def6ada0b1fa89a128b24d2597a074e887c7956f04\": container with ID starting with 06d83afdaa519e9974b129def6ada0b1fa89a128b24d2597a074e887c7956f04 not found: ID does not exist" Nov 03 01:42:05.786612 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:05.786590 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-650be0/simulator-w4-650be0-kserve-7bc588774f-gvz2k"] Nov 03 01:42:05.789878 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:05.789856 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-650be0/simulator-w4-650be0-kserve-7bc588774f-gvz2k"] Nov 03 01:42:05.879555 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:05.879532 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-200ed79c-fcghv" Nov 03 01:42:05.978786 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:05.978695 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40601dc5-808f-456f-826f-6e0c35966ada" path="/var/lib/kubelet/pods/40601dc5-808f-456f-826f-6e0c35966ada/volumes" Nov 03 01:42:05.979224 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:05.979205 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a827fa88-f078-4312-8f12-2c799c11b461" path="/var/lib/kubelet/pods/a827fa88-f078-4312-8f12-2c799c11b461/volumes" Nov 03 01:42:05.979657 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:05.979643 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff2ff304-350a-4e21-8415-1c28dcb90a7d" path="/var/lib/kubelet/pods/ff2ff304-350a-4e21-8415-1c28dcb90a7d/volumes" Nov 03 01:42:06.003666 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:06.003639 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c753d73-2ccb-410f-901d-8d19b175c510-kube-api-access-zc8sr\" (UniqueName: \"kubernetes.io/projected/9c753d73-2ccb-410f-901d-8d19b175c510-kube-api-access-zc8sr\") pod \"9c753d73-2ccb-410f-901d-8d19b175c510\" (UID: \"9c753d73-2ccb-410f-901d-8d19b175c510\") " Nov 03 01:42:06.003787 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:06.003756 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c753d73-2ccb-410f-901d-8d19b175c510-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c753d73-2ccb-410f-901d-8d19b175c510-maas-api-xks-ca\") pod \"9c753d73-2ccb-410f-901d-8d19b175c510\" (UID: \"9c753d73-2ccb-410f-901d-8d19b175c510\") " Nov 03 01:42:06.003881 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:06.003862 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c753d73-2ccb-410f-901d-8d19b175c510-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c753d73-2ccb-410f-901d-8d19b175c510-maas-api-openshift-ca\") pod \"9c753d73-2ccb-410f-901d-8d19b175c510\" (UID: \"9c753d73-2ccb-410f-901d-8d19b175c510\") " Nov 03 01:42:06.004331 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:06.004304 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c753d73-2ccb-410f-901d-8d19b175c510-maas-api-openshift-ca" pod "9c753d73-2ccb-410f-901d-8d19b175c510" (UID: "9c753d73-2ccb-410f-901d-8d19b175c510"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:42:06.006144 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:06.006116 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c753d73-2ccb-410f-901d-8d19b175c510-kube-api-access-zc8sr" pod "9c753d73-2ccb-410f-901d-8d19b175c510" (UID: "9c753d73-2ccb-410f-901d-8d19b175c510"). InnerVolumeSpecName "kube-api-access-zc8sr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:42:06.006365 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:06.006334 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c753d73-2ccb-410f-901d-8d19b175c510-maas-api-xks-ca" pod "9c753d73-2ccb-410f-901d-8d19b175c510" (UID: "9c753d73-2ccb-410f-901d-8d19b175c510"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:42:06.105106 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:06.105069 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zc8sr\" (UniqueName: \"kubernetes.io/projected/9c753d73-2ccb-410f-901d-8d19b175c510-kube-api-access-zc8sr\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:06.105106 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:06.105105 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c753d73-2ccb-410f-901d-8d19b175c510-maas-api-xks-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:06.105106 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:06.105119 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c753d73-2ccb-410f-901d-8d19b175c510-maas-api-openshift-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:06.306247 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:06.306207 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d5a9312-s5zth"] Nov 03 01:42:06.306871 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:06.306848 4421 cpu_manager.go:414] "containerMap: removing container" podUID="ff2ff304-350a-4e21-8415-1c28dcb90a7d" containerName="istio-proxy" Nov 03 01:42:06.306996 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:06.306873 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff2ff304-350a-4e21-8415-1c28dcb90a7d" containerName="istio-proxy" Nov 03 01:42:06.306996 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:06.306894 4421 cpu_manager.go:414] "containerMap: removing container" podUID="ff2ff304-350a-4e21-8415-1c28dcb90a7d" containerName="istio-proxy" Nov 03 01:42:06.306996 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:06.306904 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff2ff304-350a-4e21-8415-1c28dcb90a7d" containerName="istio-proxy" Nov 03 01:42:06.306996 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:06.306928 4421 cpu_manager.go:414] "containerMap: removing container" podUID="40601dc5-808f-456f-826f-6e0c35966ada" containerName="main" Nov 03 01:42:06.306996 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:06.306936 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40601dc5-808f-456f-826f-6e0c35966ada" containerName="main" Nov 03 01:42:06.306996 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:06.306960 4421 cpu_manager.go:414] "containerMap: removing container" podUID="a827fa88-f078-4312-8f12-2c799c11b461" containerName="main" Nov 03 01:42:06.306996 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:06.306967 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a827fa88-f078-4312-8f12-2c799c11b461" containerName="main" Nov 03 01:42:06.307862 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:06.307008 4421 cpu_manager.go:414] "containerMap: removing container" podUID="9c753d73-2ccb-410f-901d-8d19b175c510" containerName="revoke-keys" Nov 03 01:42:06.307862 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:06.307015 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c753d73-2ccb-410f-901d-8d19b175c510" containerName="revoke-keys" Nov 03 01:42:06.307862 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:06.307129 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c753d73-2ccb-410f-901d-8d19b175c510" containerName="revoke-keys" Nov 03 01:42:06.307862 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:06.307140 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff2ff304-350a-4e21-8415-1c28dcb90a7d" containerName="istio-proxy" Nov 03 01:42:06.307862 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:06.307147 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff2ff304-350a-4e21-8415-1c28dcb90a7d" containerName="istio-proxy" Nov 03 01:42:06.307862 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:06.307155 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="40601dc5-808f-456f-826f-6e0c35966ada" containerName="main" Nov 03 01:42:06.307862 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:06.307162 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="a827fa88-f078-4312-8f12-2c799c11b461" containerName="main" Nov 03 01:42:06.312082 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:06.312059 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d5a9312-s5zth" Nov 03 01:42:06.319032 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:06.319005 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d5a9312-s5zth"] Nov 03 01:42:06.321193 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:06.321166 4421 scope.go:122] "RemoveContainer" containerID="65214d143721edd3fb04d3c1fb1fc69ba46c5937217121e32162451088bf159a" Nov 03 01:42:06.336488 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:06.336459 4421 scope.go:122] "RemoveContainer" containerID="d0a44dc039c752dabfdfb9cab8ebcd92a27ea27138a85f86a13d61695b8dff4b" Nov 03 01:42:06.349365 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:06.349336 4421 scope.go:122] "RemoveContainer" containerID="1aa0aa1962028996bc494bc47f39e170cc90bc0396bbaef258ad31ab6fc6fb3f" Nov 03 01:42:06.360683 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:06.360656 4421 scope.go:122] "RemoveContainer" containerID="b53c38ff45bbf00536acbb28cdf51ca62bb7a00e6280f941ea73f4f37bb8e649" Nov 03 01:42:06.430412 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:06.430017 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpvpb\" (UniqueName: \"kubernetes.io/projected/157a7bd1-4ae4-4460-8ddd-15c02cae279d-kube-api-access-dpvpb\") pod \"maas-api-revoke-sub-1d5a9312-s5zth\" (UID: \"157a7bd1-4ae4-4460-8ddd-15c02cae279d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d5a9312-s5zth" Nov 03 01:42:06.430412 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:06.430081 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/157a7bd1-4ae4-4460-8ddd-15c02cae279d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1d5a9312-s5zth\" (UID: \"157a7bd1-4ae4-4460-8ddd-15c02cae279d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d5a9312-s5zth" Nov 03 01:42:06.430412 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:06.430163 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/157a7bd1-4ae4-4460-8ddd-15c02cae279d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1d5a9312-s5zth\" (UID: \"157a7bd1-4ae4-4460-8ddd-15c02cae279d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d5a9312-s5zth" Nov 03 01:42:06.551312 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:06.540800 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/157a7bd1-4ae4-4460-8ddd-15c02cae279d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1d5a9312-s5zth\" (UID: \"157a7bd1-4ae4-4460-8ddd-15c02cae279d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d5a9312-s5zth" Nov 03 01:42:06.551312 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:06.541045 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpvpb\" (UniqueName: \"kubernetes.io/projected/157a7bd1-4ae4-4460-8ddd-15c02cae279d-kube-api-access-dpvpb\") pod \"maas-api-revoke-sub-1d5a9312-s5zth\" (UID: \"157a7bd1-4ae4-4460-8ddd-15c02cae279d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d5a9312-s5zth" Nov 03 01:42:06.551312 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:06.541082 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/157a7bd1-4ae4-4460-8ddd-15c02cae279d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1d5a9312-s5zth\" (UID: \"157a7bd1-4ae4-4460-8ddd-15c02cae279d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d5a9312-s5zth" Nov 03 01:42:06.551312 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:06.543066 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/157a7bd1-4ae4-4460-8ddd-15c02cae279d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1d5a9312-s5zth\" (UID: \"157a7bd1-4ae4-4460-8ddd-15c02cae279d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d5a9312-s5zth" Nov 03 01:42:06.574982 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:06.574909 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/157a7bd1-4ae4-4460-8ddd-15c02cae279d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1d5a9312-s5zth\" (UID: \"157a7bd1-4ae4-4460-8ddd-15c02cae279d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d5a9312-s5zth" Nov 03 01:42:06.588975 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:06.588928 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpvpb\" (UniqueName: \"kubernetes.io/projected/157a7bd1-4ae4-4460-8ddd-15c02cae279d-kube-api-access-dpvpb\") pod \"maas-api-revoke-sub-1d5a9312-s5zth\" (UID: \"157a7bd1-4ae4-4460-8ddd-15c02cae279d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d5a9312-s5zth" Nov 03 01:42:06.630150 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:06.630121 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d5a9312-s5zth" Nov 03 01:42:06.740264 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:06.737899 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-200ed79c-fcghv" event={"ID":"9c753d73-2ccb-410f-901d-8d19b175c510","Type":"ContainerDied","Data":"c39a972f9370dda86f4005f51c78d45a85adaa0c8ea57967ce6ebf78efa27a1d"} Nov 03 01:42:06.740264 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:06.737965 4421 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c39a972f9370dda86f4005f51c78d45a85adaa0c8ea57967ce6ebf78efa27a1d" Nov 03 01:42:06.740264 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:06.738038 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-200ed79c-fcghv" Nov 03 01:42:06.926267 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:06.926240 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d5a9312-s5zth"] Nov 03 01:42:06.927982 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:42:06.927954 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod157a7bd1_4ae4_4460_8ddd_15c02cae279d.slice/crio-e5a54f24b683061a88eb3dc4dadc5ff29b3765e3137d86f7b6c41cf6468eea14 WatchSource:0}: Error finding container e5a54f24b683061a88eb3dc4dadc5ff29b3765e3137d86f7b6c41cf6468eea14: Status 404 returned error can't find the container with id e5a54f24b683061a88eb3dc4dadc5ff29b3765e3137d86f7b6c41cf6468eea14 Nov 03 01:42:07.747990 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:07.747906 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d5a9312-s5zth" event={"ID":"157a7bd1-4ae4-4460-8ddd-15c02cae279d","Type":"ContainerStarted","Data":"7ad169dd1d7c974c53009b56cea55ee9158dfd0c5d75a8f92be67cb39c0b13d4"} Nov 03 01:42:07.748609 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:07.748579 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d5a9312-s5zth" event={"ID":"157a7bd1-4ae4-4460-8ddd-15c02cae279d","Type":"ContainerStarted","Data":"e5a54f24b683061a88eb3dc4dadc5ff29b3765e3137d86f7b6c41cf6468eea14"} Nov 03 01:42:07.770070 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:07.770003 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d5a9312-s5zth" podStartSLOduration=1.769987558 podStartE2EDuration="1.769987558s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:42:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:42:07.766382721 +0000 UTC m=+3669.937385838" watchObservedRunningTime="2026-11-03 01:42:07.769987558 +0000 UTC m=+3669.940990653" Nov 03 01:42:08.304054 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:08.304012 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-83755a-67c54f67f5-mxhfv"] Nov 03 01:42:08.322342 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:08.322296 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-83755a-7dfdb9c56c-l78nm"] Nov 03 01:42:08.756105 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:08.756074 4421 generic.go:356] "Generic (PLEG): container finished" podID="157a7bd1-4ae4-4460-8ddd-15c02cae279d" containerID="7ad169dd1d7c974c53009b56cea55ee9158dfd0c5d75a8f92be67cb39c0b13d4" exitCode=0 Nov 03 01:42:08.756562 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:08.756169 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d5a9312-s5zth" event={"ID":"157a7bd1-4ae4-4460-8ddd-15c02cae279d","Type":"ContainerDied","Data":"7ad169dd1d7c974c53009b56cea55ee9158dfd0c5d75a8f92be67cb39c0b13d4"} Nov 03 01:42:09.917060 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:09.917031 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d5a9312-s5zth" Nov 03 01:42:09.994820 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:09.994793 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/157a7bd1-4ae4-4460-8ddd-15c02cae279d-kube-api-access-dpvpb\" (UniqueName: \"kubernetes.io/projected/157a7bd1-4ae4-4460-8ddd-15c02cae279d-kube-api-access-dpvpb\") pod \"157a7bd1-4ae4-4460-8ddd-15c02cae279d\" (UID: \"157a7bd1-4ae4-4460-8ddd-15c02cae279d\") " Nov 03 01:42:09.994985 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:09.994878 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/157a7bd1-4ae4-4460-8ddd-15c02cae279d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/157a7bd1-4ae4-4460-8ddd-15c02cae279d-maas-api-xks-ca\") pod \"157a7bd1-4ae4-4460-8ddd-15c02cae279d\" (UID: \"157a7bd1-4ae4-4460-8ddd-15c02cae279d\") " Nov 03 01:42:09.994985 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:09.994942 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/157a7bd1-4ae4-4460-8ddd-15c02cae279d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/157a7bd1-4ae4-4460-8ddd-15c02cae279d-maas-api-openshift-ca\") pod \"157a7bd1-4ae4-4460-8ddd-15c02cae279d\" (UID: \"157a7bd1-4ae4-4460-8ddd-15c02cae279d\") " Nov 03 01:42:09.995297 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:09.995261 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/157a7bd1-4ae4-4460-8ddd-15c02cae279d-maas-api-openshift-ca" pod "157a7bd1-4ae4-4460-8ddd-15c02cae279d" (UID: "157a7bd1-4ae4-4460-8ddd-15c02cae279d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:42:09.995474 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:09.995453 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/157a7bd1-4ae4-4460-8ddd-15c02cae279d-maas-api-openshift-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:09.996974 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:09.996958 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/157a7bd1-4ae4-4460-8ddd-15c02cae279d-kube-api-access-dpvpb" pod "157a7bd1-4ae4-4460-8ddd-15c02cae279d" (UID: "157a7bd1-4ae4-4460-8ddd-15c02cae279d"). InnerVolumeSpecName "kube-api-access-dpvpb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:42:09.997054 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:09.997034 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/157a7bd1-4ae4-4460-8ddd-15c02cae279d-maas-api-xks-ca" pod "157a7bd1-4ae4-4460-8ddd-15c02cae279d" (UID: "157a7bd1-4ae4-4460-8ddd-15c02cae279d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:42:10.096688 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:10.096580 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dpvpb\" (UniqueName: \"kubernetes.io/projected/157a7bd1-4ae4-4460-8ddd-15c02cae279d-kube-api-access-dpvpb\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:10.096688 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:10.096622 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/157a7bd1-4ae4-4460-8ddd-15c02cae279d-maas-api-xks-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:10.154127 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:10.154077 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-821f5a-84858c594c-pszhq" podUID="542b6309-1f43-413e-a448-b604a68a5047" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:42:10.154578 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:10.154549 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-821f5a-7655cbb57c-wgbsw" podUID="61568421-54f2-4ba7-973f-8efa5ab1e805" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:42:10.261454 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:10.261416 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-83755a/distinct-w3-83755a-kserve-86bdbbdd85-2cnnl" podUID="f4ae9624-d127-4a55-837d-7c9ff0267b37" containerName="main" containerID="cri-o://2633a787e3178961d8b1866a31c0a48176dc9947574a2796d272c4ead3e28c59" gracePeriod=45 Nov 03 01:42:10.272735 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:10.272702 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-83755a/premium-w3-83755a-kserve-7c9d66988b-lt4xd" podUID="c0c6b02e-2ecb-428e-a13d-be6d7cd1ba39" containerName="main" containerID="cri-o://7b75b22b2c90201f0aadd6d4faee52c88b19a60528330311b41922eb76153cae" gracePeriod=45 Nov 03 01:42:10.283571 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:10.283537 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-83755a/simulator-w3-83755a-kserve-77d6664559-psqtg" podUID="b0a4c234-af58-4da0-a1a4-e1f167953f6a" containerName="main" containerID="cri-o://c7df4530e82e735f7a56e2000452563afb60bde4657ba112acb41f449d0dbc27" gracePeriod=45 Nov 03 01:42:10.542974 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:10.542936 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-83755a/distinct-w3-83755a-kserve-86bdbbdd85-2cnnl" Nov 03 01:42:10.591859 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:10.591817 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-83755a/simulator-w3-83755a-kserve-77d6664559-psqtg" Nov 03 01:42:10.615414 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:10.615379 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-83755a/premium-w3-83755a-kserve-7c9d66988b-lt4xd" Nov 03 01:42:10.702372 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:10.702291 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0c6b02e-2ecb-428e-a13d-be6d7cd1ba39-home\" (UniqueName: \"kubernetes.io/empty-dir/c0c6b02e-2ecb-428e-a13d-be6d7cd1ba39-home\") pod \"c0c6b02e-2ecb-428e-a13d-be6d7cd1ba39\" (UID: \"c0c6b02e-2ecb-428e-a13d-be6d7cd1ba39\") " Nov 03 01:42:10.702372 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:10.702334 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0a4c234-af58-4da0-a1a4-e1f167953f6a-kube-api-access-qks2d\" (UniqueName: \"kubernetes.io/projected/b0a4c234-af58-4da0-a1a4-e1f167953f6a-kube-api-access-qks2d\") pod \"b0a4c234-af58-4da0-a1a4-e1f167953f6a\" (UID: \"b0a4c234-af58-4da0-a1a4-e1f167953f6a\") " Nov 03 01:42:10.702599 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:10.702375 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4ae9624-d127-4a55-837d-7c9ff0267b37-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4ae9624-d127-4a55-837d-7c9ff0267b37-dshm\") pod \"f4ae9624-d127-4a55-837d-7c9ff0267b37\" (UID: \"f4ae9624-d127-4a55-837d-7c9ff0267b37\") " Nov 03 01:42:10.702599 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:10.702400 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4ae9624-d127-4a55-837d-7c9ff0267b37-kube-api-access-9cwld\" (UniqueName: \"kubernetes.io/projected/f4ae9624-d127-4a55-837d-7c9ff0267b37-kube-api-access-9cwld\") pod \"f4ae9624-d127-4a55-837d-7c9ff0267b37\" (UID: \"f4ae9624-d127-4a55-837d-7c9ff0267b37\") " Nov 03 01:42:10.702599 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:10.702435 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f4ae9624-d127-4a55-837d-7c9ff0267b37-tls-certs\" (UniqueName: \"kubernetes.io/secret/f4ae9624-d127-4a55-837d-7c9ff0267b37-tls-certs\") pod \"f4ae9624-d127-4a55-837d-7c9ff0267b37\" (UID: \"f4ae9624-d127-4a55-837d-7c9ff0267b37\") " Nov 03 01:42:10.702599 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:10.702472 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0a4c234-af58-4da0-a1a4-e1f167953f6a-home\" (UniqueName: \"kubernetes.io/empty-dir/b0a4c234-af58-4da0-a1a4-e1f167953f6a-home\") pod \"b0a4c234-af58-4da0-a1a4-e1f167953f6a\" (UID: \"b0a4c234-af58-4da0-a1a4-e1f167953f6a\") " Nov 03 01:42:10.702599 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:10.702518 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0c6b02e-2ecb-428e-a13d-be6d7cd1ba39-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0c6b02e-2ecb-428e-a13d-be6d7cd1ba39-model-cache\") pod \"c0c6b02e-2ecb-428e-a13d-be6d7cd1ba39\" (UID: \"c0c6b02e-2ecb-428e-a13d-be6d7cd1ba39\") " Nov 03 01:42:10.702599 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:10.702574 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0c6b02e-2ecb-428e-a13d-be6d7cd1ba39-tls-certs\" (UniqueName: \"kubernetes.io/secret/c0c6b02e-2ecb-428e-a13d-be6d7cd1ba39-tls-certs\") pod \"c0c6b02e-2ecb-428e-a13d-be6d7cd1ba39\" (UID: \"c0c6b02e-2ecb-428e-a13d-be6d7cd1ba39\") " Nov 03 01:42:10.702953 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:10.702610 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b0a4c234-af58-4da0-a1a4-e1f167953f6a-tls-certs\" (UniqueName: \"kubernetes.io/secret/b0a4c234-af58-4da0-a1a4-e1f167953f6a-tls-certs\") pod \"b0a4c234-af58-4da0-a1a4-e1f167953f6a\" (UID: \"b0a4c234-af58-4da0-a1a4-e1f167953f6a\") " Nov 03 01:42:10.702953 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:10.702685 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0c6b02e-2ecb-428e-a13d-be6d7cd1ba39-home" pod "c0c6b02e-2ecb-428e-a13d-be6d7cd1ba39" (UID: "c0c6b02e-2ecb-428e-a13d-be6d7cd1ba39"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:42:10.702953 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:10.702748 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0a4c234-af58-4da0-a1a4-e1f167953f6a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0a4c234-af58-4da0-a1a4-e1f167953f6a-tmp-dir\") pod \"b0a4c234-af58-4da0-a1a4-e1f167953f6a\" (UID: \"b0a4c234-af58-4da0-a1a4-e1f167953f6a\") " Nov 03 01:42:10.702953 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:10.702776 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0c6b02e-2ecb-428e-a13d-be6d7cd1ba39-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0c6b02e-2ecb-428e-a13d-be6d7cd1ba39-dshm\") pod \"c0c6b02e-2ecb-428e-a13d-be6d7cd1ba39\" (UID: \"c0c6b02e-2ecb-428e-a13d-be6d7cd1ba39\") " Nov 03 01:42:10.702953 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:10.702813 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0c6b02e-2ecb-428e-a13d-be6d7cd1ba39-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0c6b02e-2ecb-428e-a13d-be6d7cd1ba39-tmp-dir\") pod \"c0c6b02e-2ecb-428e-a13d-be6d7cd1ba39\" (UID: \"c0c6b02e-2ecb-428e-a13d-be6d7cd1ba39\") " Nov 03 01:42:10.702953 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:10.702845 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0c6b02e-2ecb-428e-a13d-be6d7cd1ba39-kube-api-access-jvl9h\" (UniqueName: \"kubernetes.io/projected/c0c6b02e-2ecb-428e-a13d-be6d7cd1ba39-kube-api-access-jvl9h\") pod \"c0c6b02e-2ecb-428e-a13d-be6d7cd1ba39\" (UID: \"c0c6b02e-2ecb-428e-a13d-be6d7cd1ba39\") " Nov 03 01:42:10.702953 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:10.702857 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0a4c234-af58-4da0-a1a4-e1f167953f6a-home" pod "b0a4c234-af58-4da0-a1a4-e1f167953f6a" (UID: "b0a4c234-af58-4da0-a1a4-e1f167953f6a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:42:10.702953 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:10.702883 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0c6b02e-2ecb-428e-a13d-be6d7cd1ba39-model-cache" pod "c0c6b02e-2ecb-428e-a13d-be6d7cd1ba39" (UID: "c0c6b02e-2ecb-428e-a13d-be6d7cd1ba39"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:42:10.702953 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:10.702899 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4ae9624-d127-4a55-837d-7c9ff0267b37-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4ae9624-d127-4a55-837d-7c9ff0267b37-tmp-dir\") pod \"f4ae9624-d127-4a55-837d-7c9ff0267b37\" (UID: \"f4ae9624-d127-4a55-837d-7c9ff0267b37\") " Nov 03 01:42:10.702953 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:10.702934 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0a4c234-af58-4da0-a1a4-e1f167953f6a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0a4c234-af58-4da0-a1a4-e1f167953f6a-model-cache\") pod \"b0a4c234-af58-4da0-a1a4-e1f167953f6a\" (UID: \"b0a4c234-af58-4da0-a1a4-e1f167953f6a\") " Nov 03 01:42:10.703588 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:10.702965 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0a4c234-af58-4da0-a1a4-e1f167953f6a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0a4c234-af58-4da0-a1a4-e1f167953f6a-dshm\") pod \"b0a4c234-af58-4da0-a1a4-e1f167953f6a\" (UID: \"b0a4c234-af58-4da0-a1a4-e1f167953f6a\") " Nov 03 01:42:10.703588 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:10.702992 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4ae9624-d127-4a55-837d-7c9ff0267b37-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4ae9624-d127-4a55-837d-7c9ff0267b37-model-cache\") pod \"f4ae9624-d127-4a55-837d-7c9ff0267b37\" (UID: \"f4ae9624-d127-4a55-837d-7c9ff0267b37\") " Nov 03 01:42:10.703588 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:10.703047 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4ae9624-d127-4a55-837d-7c9ff0267b37-home\" (UniqueName: \"kubernetes.io/empty-dir/f4ae9624-d127-4a55-837d-7c9ff0267b37-home\") pod \"f4ae9624-d127-4a55-837d-7c9ff0267b37\" (UID: \"f4ae9624-d127-4a55-837d-7c9ff0267b37\") " Nov 03 01:42:10.703588 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:10.703217 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4ae9624-d127-4a55-837d-7c9ff0267b37-tmp-dir" pod "f4ae9624-d127-4a55-837d-7c9ff0267b37" (UID: "f4ae9624-d127-4a55-837d-7c9ff0267b37"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:42:10.703588 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:10.703521 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4ae9624-d127-4a55-837d-7c9ff0267b37-home" pod "f4ae9624-d127-4a55-837d-7c9ff0267b37" (UID: "f4ae9624-d127-4a55-837d-7c9ff0267b37"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:42:10.703867 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:10.703639 4421 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4ae9624-d127-4a55-837d-7c9ff0267b37-tmp-dir\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:10.703867 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:10.703663 4421 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4ae9624-d127-4a55-837d-7c9ff0267b37-home\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:10.703867 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:10.703677 4421 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0c6b02e-2ecb-428e-a13d-be6d7cd1ba39-home\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:10.703867 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:10.703691 4421 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0a4c234-af58-4da0-a1a4-e1f167953f6a-home\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:10.703867 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:10.703705 4421 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0c6b02e-2ecb-428e-a13d-be6d7cd1ba39-model-cache\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:10.704141 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:10.703909 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0c6b02e-2ecb-428e-a13d-be6d7cd1ba39-tmp-dir" pod "c0c6b02e-2ecb-428e-a13d-be6d7cd1ba39" (UID: "c0c6b02e-2ecb-428e-a13d-be6d7cd1ba39"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:42:10.704486 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:10.704460 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0a4c234-af58-4da0-a1a4-e1f167953f6a-model-cache" pod "b0a4c234-af58-4da0-a1a4-e1f167953f6a" (UID: "b0a4c234-af58-4da0-a1a4-e1f167953f6a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:42:10.705908 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:10.705879 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4ae9624-d127-4a55-837d-7c9ff0267b37-kube-api-access-9cwld" pod "f4ae9624-d127-4a55-837d-7c9ff0267b37" (UID: "f4ae9624-d127-4a55-837d-7c9ff0267b37"). InnerVolumeSpecName "kube-api-access-9cwld". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:42:10.706057 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:10.705901 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0c6b02e-2ecb-428e-a13d-be6d7cd1ba39-tls-certs" pod "c0c6b02e-2ecb-428e-a13d-be6d7cd1ba39" (UID: "c0c6b02e-2ecb-428e-a13d-be6d7cd1ba39"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:42:10.706057 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:10.705948 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4ae9624-d127-4a55-837d-7c9ff0267b37-tls-certs" pod "f4ae9624-d127-4a55-837d-7c9ff0267b37" (UID: "f4ae9624-d127-4a55-837d-7c9ff0267b37"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:42:10.706212 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:10.706186 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0a4c234-af58-4da0-a1a4-e1f167953f6a-tmp-dir" pod "b0a4c234-af58-4da0-a1a4-e1f167953f6a" (UID: "b0a4c234-af58-4da0-a1a4-e1f167953f6a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:42:10.706212 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:10.706199 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4ae9624-d127-4a55-837d-7c9ff0267b37-model-cache" pod "f4ae9624-d127-4a55-837d-7c9ff0267b37" (UID: "f4ae9624-d127-4a55-837d-7c9ff0267b37"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:42:10.706359 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:10.706227 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4ae9624-d127-4a55-837d-7c9ff0267b37-dshm" pod "f4ae9624-d127-4a55-837d-7c9ff0267b37" (UID: "f4ae9624-d127-4a55-837d-7c9ff0267b37"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:42:10.706579 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:10.706553 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0a4c234-af58-4da0-a1a4-e1f167953f6a-tls-certs" pod "b0a4c234-af58-4da0-a1a4-e1f167953f6a" (UID: "b0a4c234-af58-4da0-a1a4-e1f167953f6a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:42:10.707524 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:10.707493 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0c6b02e-2ecb-428e-a13d-be6d7cd1ba39-kube-api-access-jvl9h" pod "c0c6b02e-2ecb-428e-a13d-be6d7cd1ba39" (UID: "c0c6b02e-2ecb-428e-a13d-be6d7cd1ba39"). InnerVolumeSpecName "kube-api-access-jvl9h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:42:10.707636 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:10.707499 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0a4c234-af58-4da0-a1a4-e1f167953f6a-dshm" pod "b0a4c234-af58-4da0-a1a4-e1f167953f6a" (UID: "b0a4c234-af58-4da0-a1a4-e1f167953f6a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:42:10.707701 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:10.707680 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0c6b02e-2ecb-428e-a13d-be6d7cd1ba39-dshm" pod "c0c6b02e-2ecb-428e-a13d-be6d7cd1ba39" (UID: "c0c6b02e-2ecb-428e-a13d-be6d7cd1ba39"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:42:10.708178 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:10.708159 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0a4c234-af58-4da0-a1a4-e1f167953f6a-kube-api-access-qks2d" pod "b0a4c234-af58-4da0-a1a4-e1f167953f6a" (UID: "b0a4c234-af58-4da0-a1a4-e1f167953f6a"). InnerVolumeSpecName "kube-api-access-qks2d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:42:10.770179 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:10.770141 4421 generic.go:356] "Generic (PLEG): container finished" podID="b0a4c234-af58-4da0-a1a4-e1f167953f6a" containerID="c7df4530e82e735f7a56e2000452563afb60bde4657ba112acb41f449d0dbc27" exitCode=0 Nov 03 01:42:10.770356 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:10.770189 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-83755a/simulator-w3-83755a-kserve-77d6664559-psqtg" Nov 03 01:42:10.770356 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:10.770187 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-83755a/simulator-w3-83755a-kserve-77d6664559-psqtg" event={"ID":"b0a4c234-af58-4da0-a1a4-e1f167953f6a","Type":"ContainerDied","Data":"c7df4530e82e735f7a56e2000452563afb60bde4657ba112acb41f449d0dbc27"} Nov 03 01:42:10.770356 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:10.770234 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-83755a/simulator-w3-83755a-kserve-77d6664559-psqtg" event={"ID":"b0a4c234-af58-4da0-a1a4-e1f167953f6a","Type":"ContainerDied","Data":"dc39bf25a53fecfe9cb1dcbbc512b0ae9c0e98d93e3a90dfb39c60eadf39f114"} Nov 03 01:42:10.770356 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:10.770259 4421 scope.go:122] "RemoveContainer" containerID="c7df4530e82e735f7a56e2000452563afb60bde4657ba112acb41f449d0dbc27" Nov 03 01:42:10.771959 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:10.771934 4421 generic.go:356] "Generic (PLEG): container finished" podID="f4ae9624-d127-4a55-837d-7c9ff0267b37" containerID="2633a787e3178961d8b1866a31c0a48176dc9947574a2796d272c4ead3e28c59" exitCode=0 Nov 03 01:42:10.772054 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:10.771967 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-83755a/distinct-w3-83755a-kserve-86bdbbdd85-2cnnl" event={"ID":"f4ae9624-d127-4a55-837d-7c9ff0267b37","Type":"ContainerDied","Data":"2633a787e3178961d8b1866a31c0a48176dc9947574a2796d272c4ead3e28c59"} Nov 03 01:42:10.772054 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:10.772009 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-83755a/distinct-w3-83755a-kserve-86bdbbdd85-2cnnl" event={"ID":"f4ae9624-d127-4a55-837d-7c9ff0267b37","Type":"ContainerDied","Data":"c77461ae653c3831bfa0320e184b2f0498d17a87b7dcd57deace9f8a06f504f4"} Nov 03 01:42:10.772054 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:10.771981 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-83755a/distinct-w3-83755a-kserve-86bdbbdd85-2cnnl" Nov 03 01:42:10.773771 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:10.773742 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d5a9312-s5zth" event={"ID":"157a7bd1-4ae4-4460-8ddd-15c02cae279d","Type":"ContainerDied","Data":"e5a54f24b683061a88eb3dc4dadc5ff29b3765e3137d86f7b6c41cf6468eea14"} Nov 03 01:42:10.773879 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:10.773792 4421 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e5a54f24b683061a88eb3dc4dadc5ff29b3765e3137d86f7b6c41cf6468eea14" Nov 03 01:42:10.773879 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:10.773845 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d5a9312-s5zth" Nov 03 01:42:10.775663 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:10.775640 4421 generic.go:356] "Generic (PLEG): container finished" podID="c0c6b02e-2ecb-428e-a13d-be6d7cd1ba39" containerID="7b75b22b2c90201f0aadd6d4faee52c88b19a60528330311b41922eb76153cae" exitCode=0 Nov 03 01:42:10.775760 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:10.775688 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-83755a/premium-w3-83755a-kserve-7c9d66988b-lt4xd" event={"ID":"c0c6b02e-2ecb-428e-a13d-be6d7cd1ba39","Type":"ContainerDied","Data":"7b75b22b2c90201f0aadd6d4faee52c88b19a60528330311b41922eb76153cae"} Nov 03 01:42:10.775760 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:10.775717 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-83755a/premium-w3-83755a-kserve-7c9d66988b-lt4xd" event={"ID":"c0c6b02e-2ecb-428e-a13d-be6d7cd1ba39","Type":"ContainerDied","Data":"a83c0396d408d40610706e5bdfd7ca4a0fdbf2fd9a7c96f054803050cd4b16d4"} Nov 03 01:42:10.775863 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:10.775769 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-83755a/premium-w3-83755a-kserve-7c9d66988b-lt4xd" Nov 03 01:42:10.789660 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:10.789644 4421 scope.go:122] "RemoveContainer" containerID="c7df4530e82e735f7a56e2000452563afb60bde4657ba112acb41f449d0dbc27" Nov 03 01:42:10.789962 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:42:10.789939 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7df4530e82e735f7a56e2000452563afb60bde4657ba112acb41f449d0dbc27\": container with ID starting with c7df4530e82e735f7a56e2000452563afb60bde4657ba112acb41f449d0dbc27 not found: ID does not exist" containerID="c7df4530e82e735f7a56e2000452563afb60bde4657ba112acb41f449d0dbc27" Nov 03 01:42:10.790065 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:10.789973 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7df4530e82e735f7a56e2000452563afb60bde4657ba112acb41f449d0dbc27"} err="failed to get container status \"c7df4530e82e735f7a56e2000452563afb60bde4657ba112acb41f449d0dbc27\": rpc error: code = NotFound desc = could not find container \"c7df4530e82e735f7a56e2000452563afb60bde4657ba112acb41f449d0dbc27\": container with ID starting with c7df4530e82e735f7a56e2000452563afb60bde4657ba112acb41f449d0dbc27 not found: ID does not exist" Nov 03 01:42:10.790065 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:10.789995 4421 scope.go:122] "RemoveContainer" containerID="2633a787e3178961d8b1866a31c0a48176dc9947574a2796d272c4ead3e28c59" Nov 03 01:42:10.799037 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:10.799014 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-83755a/distinct-w3-83755a-kserve-86bdbbdd85-2cnnl"] Nov 03 01:42:10.801617 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:10.801593 4421 scope.go:122] "RemoveContainer" containerID="2633a787e3178961d8b1866a31c0a48176dc9947574a2796d272c4ead3e28c59" Nov 03 01:42:10.801889 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:42:10.801870 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2633a787e3178961d8b1866a31c0a48176dc9947574a2796d272c4ead3e28c59\": container with ID starting with 2633a787e3178961d8b1866a31c0a48176dc9947574a2796d272c4ead3e28c59 not found: ID does not exist" containerID="2633a787e3178961d8b1866a31c0a48176dc9947574a2796d272c4ead3e28c59" Nov 03 01:42:10.801961 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:10.801902 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2633a787e3178961d8b1866a31c0a48176dc9947574a2796d272c4ead3e28c59"} err="failed to get container status \"2633a787e3178961d8b1866a31c0a48176dc9947574a2796d272c4ead3e28c59\": rpc error: code = NotFound desc = could not find container \"2633a787e3178961d8b1866a31c0a48176dc9947574a2796d272c4ead3e28c59\": container with ID starting with 2633a787e3178961d8b1866a31c0a48176dc9947574a2796d272c4ead3e28c59 not found: ID does not exist" Nov 03 01:42:10.801961 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:10.801928 4421 scope.go:122] "RemoveContainer" containerID="7b75b22b2c90201f0aadd6d4faee52c88b19a60528330311b41922eb76153cae" Nov 03 01:42:10.804741 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:10.804714 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-83755a/distinct-w3-83755a-kserve-86bdbbdd85-2cnnl"] Nov 03 01:42:10.804820 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:10.804768 4421 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0a4c234-af58-4da0-a1a4-e1f167953f6a-tmp-dir\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:10.804820 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:10.804786 4421 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0c6b02e-2ecb-428e-a13d-be6d7cd1ba39-dshm\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:10.804820 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:10.804794 4421 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0c6b02e-2ecb-428e-a13d-be6d7cd1ba39-tmp-dir\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:10.804820 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:10.804803 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvl9h\" (UniqueName: \"kubernetes.io/projected/c0c6b02e-2ecb-428e-a13d-be6d7cd1ba39-kube-api-access-jvl9h\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:10.804820 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:10.804821 4421 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0a4c234-af58-4da0-a1a4-e1f167953f6a-model-cache\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:10.805006 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:10.804830 4421 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0a4c234-af58-4da0-a1a4-e1f167953f6a-dshm\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:10.805006 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:10.804839 4421 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4ae9624-d127-4a55-837d-7c9ff0267b37-model-cache\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:10.805006 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:10.804847 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qks2d\" (UniqueName: \"kubernetes.io/projected/b0a4c234-af58-4da0-a1a4-e1f167953f6a-kube-api-access-qks2d\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:10.805006 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:10.804855 4421 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4ae9624-d127-4a55-837d-7c9ff0267b37-dshm\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:10.805006 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:10.804863 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9cwld\" (UniqueName: \"kubernetes.io/projected/f4ae9624-d127-4a55-837d-7c9ff0267b37-kube-api-access-9cwld\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:10.805006 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:10.804871 4421 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4ae9624-d127-4a55-837d-7c9ff0267b37-tls-certs\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:10.805006 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:10.804879 4421 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0c6b02e-2ecb-428e-a13d-be6d7cd1ba39-tls-certs\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:10.805006 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:10.804887 4421 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0a4c234-af58-4da0-a1a4-e1f167953f6a-tls-certs\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:10.811866 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:10.811847 4421 scope.go:122] "RemoveContainer" containerID="7b75b22b2c90201f0aadd6d4faee52c88b19a60528330311b41922eb76153cae" Nov 03 01:42:10.812151 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:42:10.812130 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b75b22b2c90201f0aadd6d4faee52c88b19a60528330311b41922eb76153cae\": container with ID starting with 7b75b22b2c90201f0aadd6d4faee52c88b19a60528330311b41922eb76153cae not found: ID does not exist" containerID="7b75b22b2c90201f0aadd6d4faee52c88b19a60528330311b41922eb76153cae" Nov 03 01:42:10.812213 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:10.812158 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b75b22b2c90201f0aadd6d4faee52c88b19a60528330311b41922eb76153cae"} err="failed to get container status \"7b75b22b2c90201f0aadd6d4faee52c88b19a60528330311b41922eb76153cae\": rpc error: code = NotFound desc = could not find container \"7b75b22b2c90201f0aadd6d4faee52c88b19a60528330311b41922eb76153cae\": container with ID starting with 7b75b22b2c90201f0aadd6d4faee52c88b19a60528330311b41922eb76153cae not found: ID does not exist" Nov 03 01:42:10.816423 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:10.816403 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-83755a/simulator-w3-83755a-kserve-77d6664559-psqtg"] Nov 03 01:42:10.822433 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:10.822411 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-83755a/simulator-w3-83755a-kserve-77d6664559-psqtg"] Nov 03 01:42:10.837331 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:10.836073 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-83755a/premium-w3-83755a-kserve-7c9d66988b-lt4xd"] Nov 03 01:42:10.841157 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:10.841121 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-83755a/premium-w3-83755a-kserve-7c9d66988b-lt4xd"] Nov 03 01:42:11.979090 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:11.979053 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0a4c234-af58-4da0-a1a4-e1f167953f6a" path="/var/lib/kubelet/pods/b0a4c234-af58-4da0-a1a4-e1f167953f6a/volumes" Nov 03 01:42:11.979546 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:11.979531 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0c6b02e-2ecb-428e-a13d-be6d7cd1ba39" path="/var/lib/kubelet/pods/c0c6b02e-2ecb-428e-a13d-be6d7cd1ba39/volumes" Nov 03 01:42:11.979914 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:11.979902 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4ae9624-d127-4a55-837d-7c9ff0267b37" path="/var/lib/kubelet/pods/f4ae9624-d127-4a55-837d-7c9ff0267b37/volumes" Nov 03 01:42:13.304890 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:13.304843 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-83755a-67c54f67f5-mxhfv" podUID="2fb9b19f-9286-4d24-813c-be4055c5e8e2" containerName="payload-processing" containerID="cri-o://56488b5383deb40352cd71f0573db3fb1712d8045cc1f2d914bbcdcd990d3144" gracePeriod=25 Nov 03 01:42:13.323374 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:13.323330 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-83755a-7dfdb9c56c-l78nm" podUID="f7b858b0-7dd7-42f7-bb5f-22920b707d4c" containerName="payload-pre-processing" containerID="cri-o://5a0b873bf7ce4f774466861e5d89a026203e92953c2cee27156f614ed729ea95" gracePeriod=25 Nov 03 01:42:14.672012 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:42:14.671962 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 01:42:18.039602 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:18.039564 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-83755a-7dfdb9c56c-l78nm" podUID="f7b858b0-7dd7-42f7-bb5f-22920b707d4c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:42:18.477722 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:18.477693 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-821f5a-7655cbb57c-wgbsw" Nov 03 01:42:18.512872 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:18.512834 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-821f5a-84858c594c-pszhq" Nov 03 01:42:18.605027 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:18.604729 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61568421-54f2-4ba7-973f-8efa5ab1e805-kube-api-access-tcff7\" (UniqueName: \"kubernetes.io/projected/61568421-54f2-4ba7-973f-8efa5ab1e805-kube-api-access-tcff7\") pod \"61568421-54f2-4ba7-973f-8efa5ab1e805\" (UID: \"61568421-54f2-4ba7-973f-8efa5ab1e805\") " Nov 03 01:42:18.605027 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:18.604818 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61568421-54f2-4ba7-973f-8efa5ab1e805-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61568421-54f2-4ba7-973f-8efa5ab1e805-plugins-config-volume\") pod \"61568421-54f2-4ba7-973f-8efa5ab1e805\" (UID: \"61568421-54f2-4ba7-973f-8efa5ab1e805\") " Nov 03 01:42:18.607050 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:18.607009 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61568421-54f2-4ba7-973f-8efa5ab1e805-plugins-config-volume" pod "61568421-54f2-4ba7-973f-8efa5ab1e805" (UID: "61568421-54f2-4ba7-973f-8efa5ab1e805"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:42:18.614187 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:18.614149 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61568421-54f2-4ba7-973f-8efa5ab1e805-kube-api-access-tcff7" pod "61568421-54f2-4ba7-973f-8efa5ab1e805" (UID: "61568421-54f2-4ba7-973f-8efa5ab1e805"). InnerVolumeSpecName "kube-api-access-tcff7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:42:18.706445 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:18.706407 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/542b6309-1f43-413e-a448-b604a68a5047-kube-api-access-bkrjq\" (UniqueName: \"kubernetes.io/projected/542b6309-1f43-413e-a448-b604a68a5047-kube-api-access-bkrjq\") pod \"542b6309-1f43-413e-a448-b604a68a5047\" (UID: \"542b6309-1f43-413e-a448-b604a68a5047\") " Nov 03 01:42:18.706620 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:18.706604 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/542b6309-1f43-413e-a448-b604a68a5047-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/542b6309-1f43-413e-a448-b604a68a5047-plugins-config-volume\") pod \"542b6309-1f43-413e-a448-b604a68a5047\" (UID: \"542b6309-1f43-413e-a448-b604a68a5047\") " Nov 03 01:42:18.706961 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:18.706935 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/542b6309-1f43-413e-a448-b604a68a5047-plugins-config-volume" pod "542b6309-1f43-413e-a448-b604a68a5047" (UID: "542b6309-1f43-413e-a448-b604a68a5047"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:42:18.707198 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:18.707175 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tcff7\" (UniqueName: \"kubernetes.io/projected/61568421-54f2-4ba7-973f-8efa5ab1e805-kube-api-access-tcff7\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:18.707297 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:18.707204 4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61568421-54f2-4ba7-973f-8efa5ab1e805-plugins-config-volume\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:18.707297 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:18.707220 4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/542b6309-1f43-413e-a448-b604a68a5047-plugins-config-volume\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:18.708677 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:18.708649 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/542b6309-1f43-413e-a448-b604a68a5047-kube-api-access-bkrjq" pod "542b6309-1f43-413e-a448-b604a68a5047" (UID: "542b6309-1f43-413e-a448-b604a68a5047"). InnerVolumeSpecName "kube-api-access-bkrjq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:42:18.808179 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:18.808142 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bkrjq\" (UniqueName: \"kubernetes.io/projected/542b6309-1f43-413e-a448-b604a68a5047-kube-api-access-bkrjq\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:18.850325 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:18.846717 4421 generic.go:356] "Generic (PLEG): container finished" podID="61568421-54f2-4ba7-973f-8efa5ab1e805" containerID="97b45b47e745ec0e2a4a7e0edf92413d611bb2ab85559d64835d2cbe6e726fcd" exitCode=0 Nov 03 01:42:18.850325 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:18.846906 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-821f5a-7655cbb57c-wgbsw" event={"ID":"61568421-54f2-4ba7-973f-8efa5ab1e805","Type":"ContainerDied","Data":"97b45b47e745ec0e2a4a7e0edf92413d611bb2ab85559d64835d2cbe6e726fcd"} Nov 03 01:42:18.850325 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:18.846962 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-821f5a-7655cbb57c-wgbsw" event={"ID":"61568421-54f2-4ba7-973f-8efa5ab1e805","Type":"ContainerDied","Data":"fc1b5528c63670d6f2e2a33df763c6ba60686b0a6bc5a3604c8a34846018cfef"} Nov 03 01:42:18.850325 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:18.846984 4421 scope.go:122] "RemoveContainer" containerID="97b45b47e745ec0e2a4a7e0edf92413d611bb2ab85559d64835d2cbe6e726fcd" Nov 03 01:42:18.850325 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:18.847105 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-821f5a-7655cbb57c-wgbsw" Nov 03 01:42:18.853971 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:18.853934 4421 generic.go:356] "Generic (PLEG): container finished" podID="542b6309-1f43-413e-a448-b604a68a5047" containerID="4ed7957e2ccc19fddec9b2dc34613e524e0d7cb91be6a03bc5b59c895350bcf5" exitCode=0 Nov 03 01:42:18.854140 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:18.853973 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-821f5a-84858c594c-pszhq" event={"ID":"542b6309-1f43-413e-a448-b604a68a5047","Type":"ContainerDied","Data":"4ed7957e2ccc19fddec9b2dc34613e524e0d7cb91be6a03bc5b59c895350bcf5"} Nov 03 01:42:18.854140 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:18.854006 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-821f5a-84858c594c-pszhq" Nov 03 01:42:18.854140 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:18.854041 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-821f5a-84858c594c-pszhq" event={"ID":"542b6309-1f43-413e-a448-b604a68a5047","Type":"ContainerDied","Data":"f81ddb6f19713e21176140a10b46565bae8a88267aee6d8c928df1f525db761b"} Nov 03 01:42:18.862614 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:18.862586 4421 scope.go:122] "RemoveContainer" containerID="97b45b47e745ec0e2a4a7e0edf92413d611bb2ab85559d64835d2cbe6e726fcd" Nov 03 01:42:18.862938 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:42:18.862916 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97b45b47e745ec0e2a4a7e0edf92413d611bb2ab85559d64835d2cbe6e726fcd\": container with ID starting with 97b45b47e745ec0e2a4a7e0edf92413d611bb2ab85559d64835d2cbe6e726fcd not found: ID does not exist" containerID="97b45b47e745ec0e2a4a7e0edf92413d611bb2ab85559d64835d2cbe6e726fcd" Nov 03 01:42:18.863046 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:18.862945 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97b45b47e745ec0e2a4a7e0edf92413d611bb2ab85559d64835d2cbe6e726fcd"} err="failed to get container status \"97b45b47e745ec0e2a4a7e0edf92413d611bb2ab85559d64835d2cbe6e726fcd\": rpc error: code = NotFound desc = could not find container \"97b45b47e745ec0e2a4a7e0edf92413d611bb2ab85559d64835d2cbe6e726fcd\": container with ID starting with 97b45b47e745ec0e2a4a7e0edf92413d611bb2ab85559d64835d2cbe6e726fcd not found: ID does not exist" Nov 03 01:42:18.863046 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:18.862968 4421 scope.go:122] "RemoveContainer" containerID="4ed7957e2ccc19fddec9b2dc34613e524e0d7cb91be6a03bc5b59c895350bcf5" Nov 03 01:42:18.876956 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:18.876929 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-821f5a-7655cbb57c-wgbsw"] Nov 03 01:42:18.877102 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:18.877085 4421 scope.go:122] "RemoveContainer" containerID="4ed7957e2ccc19fddec9b2dc34613e524e0d7cb91be6a03bc5b59c895350bcf5" Nov 03 01:42:18.877481 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:42:18.877455 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ed7957e2ccc19fddec9b2dc34613e524e0d7cb91be6a03bc5b59c895350bcf5\": container with ID starting with 4ed7957e2ccc19fddec9b2dc34613e524e0d7cb91be6a03bc5b59c895350bcf5 not found: ID does not exist" containerID="4ed7957e2ccc19fddec9b2dc34613e524e0d7cb91be6a03bc5b59c895350bcf5" Nov 03 01:42:18.877571 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:18.877488 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ed7957e2ccc19fddec9b2dc34613e524e0d7cb91be6a03bc5b59c895350bcf5"} err="failed to get container status \"4ed7957e2ccc19fddec9b2dc34613e524e0d7cb91be6a03bc5b59c895350bcf5\": rpc error: code = NotFound desc = could not find container \"4ed7957e2ccc19fddec9b2dc34613e524e0d7cb91be6a03bc5b59c895350bcf5\": container with ID starting with 4ed7957e2ccc19fddec9b2dc34613e524e0d7cb91be6a03bc5b59c895350bcf5 not found: ID does not exist" Nov 03 01:42:18.881843 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:18.881824 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-821f5a-7655cbb57c-wgbsw"] Nov 03 01:42:18.895765 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:18.895732 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-821f5a-84858c594c-pszhq"] Nov 03 01:42:18.899838 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:18.899816 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-821f5a-84858c594c-pszhq"] Nov 03 01:42:19.047036 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:19.046994 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-83755a-67c54f67f5-mxhfv" podUID="2fb9b19f-9286-4d24-813c-be4055c5e8e2" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:42:19.979286 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:19.979247 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="542b6309-1f43-413e-a448-b604a68a5047" path="/var/lib/kubelet/pods/542b6309-1f43-413e-a448-b604a68a5047/volumes" Nov 03 01:42:19.979794 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:19.979773 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61568421-54f2-4ba7-973f-8efa5ab1e805" path="/var/lib/kubelet/pods/61568421-54f2-4ba7-973f-8efa5ab1e805/volumes" Nov 03 01:42:28.039763 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:28.039722 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-83755a-7dfdb9c56c-l78nm" podUID="f7b858b0-7dd7-42f7-bb5f-22920b707d4c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:42:29.045991 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:29.045950 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-83755a-67c54f67f5-mxhfv" podUID="2fb9b19f-9286-4d24-813c-be4055c5e8e2" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:42:30.392640 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:30.392605 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-ztm5t"] Nov 03 01:42:30.393143 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:30.392818 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-ztm5t" podUID="dc7f30da-8b1e-42d7-a41a-2398e66263b7" containerName="istio-proxy" containerID="cri-o://ba7b2267ebde4e272760c8646331a1c244e7ca03ad1093dfba7d3523ff45789a" gracePeriod=30 Nov 03 01:42:33.588818 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:33.588786 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-83755a-67c54f67f5-mxhfv" Nov 03 01:42:33.650590 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:33.650554 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-83755a-7dfdb9c56c-l78nm" Nov 03 01:42:33.671116 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:33.671085 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fb9b19f-9286-4d24-813c-be4055c5e8e2-kube-api-access-m5b4f\" (UniqueName: \"kubernetes.io/projected/2fb9b19f-9286-4d24-813c-be4055c5e8e2-kube-api-access-m5b4f\") pod \"2fb9b19f-9286-4d24-813c-be4055c5e8e2\" (UID: \"2fb9b19f-9286-4d24-813c-be4055c5e8e2\") " Nov 03 01:42:33.671341 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:33.671144 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2fb9b19f-9286-4d24-813c-be4055c5e8e2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2fb9b19f-9286-4d24-813c-be4055c5e8e2-plugins-config-volume\") pod \"2fb9b19f-9286-4d24-813c-be4055c5e8e2\" (UID: \"2fb9b19f-9286-4d24-813c-be4055c5e8e2\") " Nov 03 01:42:33.672044 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:33.672017 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fb9b19f-9286-4d24-813c-be4055c5e8e2-plugins-config-volume" pod "2fb9b19f-9286-4d24-813c-be4055c5e8e2" (UID: "2fb9b19f-9286-4d24-813c-be4055c5e8e2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:42:33.674363 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:33.674258 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fb9b19f-9286-4d24-813c-be4055c5e8e2-kube-api-access-m5b4f" pod "2fb9b19f-9286-4d24-813c-be4055c5e8e2" (UID: "2fb9b19f-9286-4d24-813c-be4055c5e8e2"). InnerVolumeSpecName "kube-api-access-m5b4f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:42:33.772007 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:33.771969 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7b858b0-7dd7-42f7-bb5f-22920b707d4c-kube-api-access-scszf\" (UniqueName: \"kubernetes.io/projected/f7b858b0-7dd7-42f7-bb5f-22920b707d4c-kube-api-access-scszf\") pod \"f7b858b0-7dd7-42f7-bb5f-22920b707d4c\" (UID: \"f7b858b0-7dd7-42f7-bb5f-22920b707d4c\") " Nov 03 01:42:33.772199 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:33.772108 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7b858b0-7dd7-42f7-bb5f-22920b707d4c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7b858b0-7dd7-42f7-bb5f-22920b707d4c-plugins-config-volume\") pod \"f7b858b0-7dd7-42f7-bb5f-22920b707d4c\" (UID: \"f7b858b0-7dd7-42f7-bb5f-22920b707d4c\") " Nov 03 01:42:33.772433 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:33.772411 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5b4f\" (UniqueName: \"kubernetes.io/projected/2fb9b19f-9286-4d24-813c-be4055c5e8e2-kube-api-access-m5b4f\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:33.772433 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:33.772432 4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2fb9b19f-9286-4d24-813c-be4055c5e8e2-plugins-config-volume\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:33.772617 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:33.772588 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7b858b0-7dd7-42f7-bb5f-22920b707d4c-plugins-config-volume" pod "f7b858b0-7dd7-42f7-bb5f-22920b707d4c" (UID: "f7b858b0-7dd7-42f7-bb5f-22920b707d4c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:42:33.774073 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:33.774050 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7b858b0-7dd7-42f7-bb5f-22920b707d4c-kube-api-access-scszf" pod "f7b858b0-7dd7-42f7-bb5f-22920b707d4c" (UID: "f7b858b0-7dd7-42f7-bb5f-22920b707d4c"). InnerVolumeSpecName "kube-api-access-scszf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:42:33.873188 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:33.873099 4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7b858b0-7dd7-42f7-bb5f-22920b707d4c-plugins-config-volume\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:33.873188 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:33.873129 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-scszf\" (UniqueName: \"kubernetes.io/projected/f7b858b0-7dd7-42f7-bb5f-22920b707d4c-kube-api-access-scszf\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:33.958409 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:33.958372 4421 generic.go:356] "Generic (PLEG): container finished" podID="2fb9b19f-9286-4d24-813c-be4055c5e8e2" containerID="56488b5383deb40352cd71f0573db3fb1712d8045cc1f2d914bbcdcd990d3144" exitCode=0 Nov 03 01:42:33.958611 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:33.958428 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-83755a-67c54f67f5-mxhfv" Nov 03 01:42:33.958611 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:33.958466 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-83755a-67c54f67f5-mxhfv" event={"ID":"2fb9b19f-9286-4d24-813c-be4055c5e8e2","Type":"ContainerDied","Data":"56488b5383deb40352cd71f0573db3fb1712d8045cc1f2d914bbcdcd990d3144"} Nov 03 01:42:33.958611 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:33.958568 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-83755a-67c54f67f5-mxhfv" event={"ID":"2fb9b19f-9286-4d24-813c-be4055c5e8e2","Type":"ContainerDied","Data":"29f2ab9f42c1c95f9b34794cdaf292a86811aed821bc8448063bfb25dec2e929"} Nov 03 01:42:33.958611 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:33.958606 4421 scope.go:122] "RemoveContainer" containerID="56488b5383deb40352cd71f0573db3fb1712d8045cc1f2d914bbcdcd990d3144" Nov 03 01:42:33.960124 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:33.960102 4421 generic.go:356] "Generic (PLEG): container finished" podID="f7b858b0-7dd7-42f7-bb5f-22920b707d4c" containerID="5a0b873bf7ce4f774466861e5d89a026203e92953c2cee27156f614ed729ea95" exitCode=0 Nov 03 01:42:33.960235 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:33.960147 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-83755a-7dfdb9c56c-l78nm" Nov 03 01:42:33.960235 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:33.960220 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-83755a-7dfdb9c56c-l78nm" event={"ID":"f7b858b0-7dd7-42f7-bb5f-22920b707d4c","Type":"ContainerDied","Data":"5a0b873bf7ce4f774466861e5d89a026203e92953c2cee27156f614ed729ea95"} Nov 03 01:42:33.960416 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:33.960266 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-83755a-7dfdb9c56c-l78nm" event={"ID":"f7b858b0-7dd7-42f7-bb5f-22920b707d4c","Type":"ContainerDied","Data":"aa34f1f761d93823e852c656579c3be69fc113d8ffe5f6e9caae0c0a2acf5a0f"} Nov 03 01:42:33.969562 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:33.969541 4421 scope.go:122] "RemoveContainer" containerID="56488b5383deb40352cd71f0573db3fb1712d8045cc1f2d914bbcdcd990d3144" Nov 03 01:42:33.969923 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:42:33.969887 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56488b5383deb40352cd71f0573db3fb1712d8045cc1f2d914bbcdcd990d3144\": container with ID starting with 56488b5383deb40352cd71f0573db3fb1712d8045cc1f2d914bbcdcd990d3144 not found: ID does not exist" containerID="56488b5383deb40352cd71f0573db3fb1712d8045cc1f2d914bbcdcd990d3144" Nov 03 01:42:33.970012 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:33.969934 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56488b5383deb40352cd71f0573db3fb1712d8045cc1f2d914bbcdcd990d3144"} err="failed to get container status \"56488b5383deb40352cd71f0573db3fb1712d8045cc1f2d914bbcdcd990d3144\": rpc error: code = NotFound desc = could not find container \"56488b5383deb40352cd71f0573db3fb1712d8045cc1f2d914bbcdcd990d3144\": container with ID starting with 56488b5383deb40352cd71f0573db3fb1712d8045cc1f2d914bbcdcd990d3144 not found: ID does not exist" Nov 03 01:42:33.970012 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:33.969958 4421 scope.go:122] "RemoveContainer" containerID="5a0b873bf7ce4f774466861e5d89a026203e92953c2cee27156f614ed729ea95" Nov 03 01:42:33.982660 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:33.982632 4421 scope.go:122] "RemoveContainer" containerID="5a0b873bf7ce4f774466861e5d89a026203e92953c2cee27156f614ed729ea95" Nov 03 01:42:33.983148 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:42:33.983122 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a0b873bf7ce4f774466861e5d89a026203e92953c2cee27156f614ed729ea95\": container with ID starting with 5a0b873bf7ce4f774466861e5d89a026203e92953c2cee27156f614ed729ea95 not found: ID does not exist" containerID="5a0b873bf7ce4f774466861e5d89a026203e92953c2cee27156f614ed729ea95" Nov 03 01:42:33.983308 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:33.983155 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a0b873bf7ce4f774466861e5d89a026203e92953c2cee27156f614ed729ea95"} err="failed to get container status \"5a0b873bf7ce4f774466861e5d89a026203e92953c2cee27156f614ed729ea95\": rpc error: code = NotFound desc = could not find container \"5a0b873bf7ce4f774466861e5d89a026203e92953c2cee27156f614ed729ea95\": container with ID starting with 5a0b873bf7ce4f774466861e5d89a026203e92953c2cee27156f614ed729ea95 not found: ID does not exist" Nov 03 01:42:33.988436 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:33.988405 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-83755a-67c54f67f5-mxhfv"] Nov 03 01:42:33.991736 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:33.991709 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-83755a-67c54f67f5-mxhfv"] Nov 03 01:42:34.001197 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:34.001168 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-83755a-7dfdb9c56c-l78nm"] Nov 03 01:42:34.018253 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:34.018059 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-83755a-7dfdb9c56c-l78nm"] Nov 03 01:42:35.623946 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:35.622254 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-mlxvq"] Nov 03 01:42:35.623946 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:35.622554 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-mlxvq" podUID="dcf063ce-88a5-4341-ae0f-605522bb91d5" containerName="istio-proxy" containerID="cri-o://dcf8a1ffba7dd0036d36c7d4b74904648e76bc6c8bdbd457c757c4fc064e6b68" gracePeriod=30 Nov 03 01:42:35.667805 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:35.667736 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-ztm5t" Nov 03 01:42:35.802609 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:35.802560 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc7f30da-8b1e-42d7-a41a-2398e66263b7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc7f30da-8b1e-42d7-a41a-2398e66263b7-credential-socket\") pod \"dc7f30da-8b1e-42d7-a41a-2398e66263b7\" (UID: \"dc7f30da-8b1e-42d7-a41a-2398e66263b7\") " Nov 03 01:42:35.802837 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:35.802684 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc7f30da-8b1e-42d7-a41a-2398e66263b7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc7f30da-8b1e-42d7-a41a-2398e66263b7-workload-socket\") pod \"dc7f30da-8b1e-42d7-a41a-2398e66263b7\" (UID: \"dc7f30da-8b1e-42d7-a41a-2398e66263b7\") " Nov 03 01:42:35.802837 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:35.802719 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc7f30da-8b1e-42d7-a41a-2398e66263b7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc7f30da-8b1e-42d7-a41a-2398e66263b7-workload-certs\") pod \"dc7f30da-8b1e-42d7-a41a-2398e66263b7\" (UID: \"dc7f30da-8b1e-42d7-a41a-2398e66263b7\") " Nov 03 01:42:35.802837 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:35.802775 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc7f30da-8b1e-42d7-a41a-2398e66263b7-istio-token\" (UniqueName: \"kubernetes.io/projected/dc7f30da-8b1e-42d7-a41a-2398e66263b7-istio-token\") pod \"dc7f30da-8b1e-42d7-a41a-2398e66263b7\" (UID: \"dc7f30da-8b1e-42d7-a41a-2398e66263b7\") " Nov 03 01:42:35.802837 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:35.802827 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc7f30da-8b1e-42d7-a41a-2398e66263b7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc7f30da-8b1e-42d7-a41a-2398e66263b7-istio-data\") pod \"dc7f30da-8b1e-42d7-a41a-2398e66263b7\" (UID: \"dc7f30da-8b1e-42d7-a41a-2398e66263b7\") " Nov 03 01:42:35.803132 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:35.802869 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc7f30da-8b1e-42d7-a41a-2398e66263b7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc7f30da-8b1e-42d7-a41a-2398e66263b7-istio-envoy\") pod \"dc7f30da-8b1e-42d7-a41a-2398e66263b7\" (UID: \"dc7f30da-8b1e-42d7-a41a-2398e66263b7\") " Nov 03 01:42:35.803132 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:35.802883 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc7f30da-8b1e-42d7-a41a-2398e66263b7-credential-socket" pod "dc7f30da-8b1e-42d7-a41a-2398e66263b7" (UID: "dc7f30da-8b1e-42d7-a41a-2398e66263b7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:42:35.803132 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:35.802909 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc7f30da-8b1e-42d7-a41a-2398e66263b7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc7f30da-8b1e-42d7-a41a-2398e66263b7-istiod-ca-cert\") pod \"dc7f30da-8b1e-42d7-a41a-2398e66263b7\" (UID: \"dc7f30da-8b1e-42d7-a41a-2398e66263b7\") " Nov 03 01:42:35.803132 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:35.802946 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc7f30da-8b1e-42d7-a41a-2398e66263b7-kube-api-access-4vx6k\" (UniqueName: \"kubernetes.io/projected/dc7f30da-8b1e-42d7-a41a-2398e66263b7-kube-api-access-4vx6k\") pod \"dc7f30da-8b1e-42d7-a41a-2398e66263b7\" (UID: \"dc7f30da-8b1e-42d7-a41a-2398e66263b7\") " Nov 03 01:42:35.803132 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:35.802996 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dc7f30da-8b1e-42d7-a41a-2398e66263b7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc7f30da-8b1e-42d7-a41a-2398e66263b7-istio-podinfo\") pod \"dc7f30da-8b1e-42d7-a41a-2398e66263b7\" (UID: \"dc7f30da-8b1e-42d7-a41a-2398e66263b7\") " Nov 03 01:42:35.803132 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:35.803009 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc7f30da-8b1e-42d7-a41a-2398e66263b7-workload-socket" pod "dc7f30da-8b1e-42d7-a41a-2398e66263b7" (UID: "dc7f30da-8b1e-42d7-a41a-2398e66263b7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:42:35.803132 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:35.803114 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc7f30da-8b1e-42d7-a41a-2398e66263b7-istio-data" pod "dc7f30da-8b1e-42d7-a41a-2398e66263b7" (UID: "dc7f30da-8b1e-42d7-a41a-2398e66263b7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:42:35.803729 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:35.803693 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc7f30da-8b1e-42d7-a41a-2398e66263b7-workload-certs" pod "dc7f30da-8b1e-42d7-a41a-2398e66263b7" (UID: "dc7f30da-8b1e-42d7-a41a-2398e66263b7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:42:35.803855 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:35.803761 4421 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc7f30da-8b1e-42d7-a41a-2398e66263b7-istio-data\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:35.803855 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:35.803786 4421 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc7f30da-8b1e-42d7-a41a-2398e66263b7-credential-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:35.803855 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:35.803797 4421 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc7f30da-8b1e-42d7-a41a-2398e66263b7-workload-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:35.803855 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:35.803803 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc7f30da-8b1e-42d7-a41a-2398e66263b7-istiod-ca-cert" pod "dc7f30da-8b1e-42d7-a41a-2398e66263b7" (UID: "dc7f30da-8b1e-42d7-a41a-2398e66263b7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:42:35.805642 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:35.805612 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc7f30da-8b1e-42d7-a41a-2398e66263b7-istio-token" pod "dc7f30da-8b1e-42d7-a41a-2398e66263b7" (UID: "dc7f30da-8b1e-42d7-a41a-2398e66263b7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:42:35.805942 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:35.805908 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc7f30da-8b1e-42d7-a41a-2398e66263b7-kube-api-access-4vx6k" pod "dc7f30da-8b1e-42d7-a41a-2398e66263b7" (UID: "dc7f30da-8b1e-42d7-a41a-2398e66263b7"). InnerVolumeSpecName "kube-api-access-4vx6k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:42:35.806071 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:35.805990 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dc7f30da-8b1e-42d7-a41a-2398e66263b7-istio-podinfo" pod "dc7f30da-8b1e-42d7-a41a-2398e66263b7" (UID: "dc7f30da-8b1e-42d7-a41a-2398e66263b7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:42:35.806481 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:35.806460 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc7f30da-8b1e-42d7-a41a-2398e66263b7-istio-envoy" pod "dc7f30da-8b1e-42d7-a41a-2398e66263b7" (UID: "dc7f30da-8b1e-42d7-a41a-2398e66263b7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:42:35.904698 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:35.904662 4421 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc7f30da-8b1e-42d7-a41a-2398e66263b7-workload-certs\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:35.904698 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:35.904694 4421 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc7f30da-8b1e-42d7-a41a-2398e66263b7-istio-token\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:35.904698 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:35.904704 4421 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc7f30da-8b1e-42d7-a41a-2398e66263b7-istio-envoy\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:35.904960 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:35.904713 4421 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc7f30da-8b1e-42d7-a41a-2398e66263b7-istiod-ca-cert\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:35.904960 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:35.904723 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vx6k\" (UniqueName: \"kubernetes.io/projected/dc7f30da-8b1e-42d7-a41a-2398e66263b7-kube-api-access-4vx6k\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:35.904960 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:35.904733 4421 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc7f30da-8b1e-42d7-a41a-2398e66263b7-istio-podinfo\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:35.978834 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:35.978744 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2fb9b19f-9286-4d24-813c-be4055c5e8e2" path="/var/lib/kubelet/pods/2fb9b19f-9286-4d24-813c-be4055c5e8e2/volumes" Nov 03 01:42:35.979030 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:35.979007 4421 generic.go:356] "Generic (PLEG): container finished" podID="dc7f30da-8b1e-42d7-a41a-2398e66263b7" containerID="ba7b2267ebde4e272760c8646331a1c244e7ca03ad1093dfba7d3523ff45789a" exitCode=0 Nov 03 01:42:35.979114 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:35.979081 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-ztm5t" Nov 03 01:42:35.982418 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:35.979789 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7b858b0-7dd7-42f7-bb5f-22920b707d4c" path="/var/lib/kubelet/pods/f7b858b0-7dd7-42f7-bb5f-22920b707d4c/volumes" Nov 03 01:42:35.982418 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:35.980598 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-ztm5t" event={"ID":"dc7f30da-8b1e-42d7-a41a-2398e66263b7","Type":"ContainerDied","Data":"ba7b2267ebde4e272760c8646331a1c244e7ca03ad1093dfba7d3523ff45789a"} Nov 03 01:42:35.982418 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:35.980665 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-ztm5t" event={"ID":"dc7f30da-8b1e-42d7-a41a-2398e66263b7","Type":"ContainerDied","Data":"1accb0087949c36c23a1be59170db035e9e116ecffa5475b611828604992f42d"} Nov 03 01:42:35.982418 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:35.980711 4421 scope.go:122] "RemoveContainer" containerID="ba7b2267ebde4e272760c8646331a1c244e7ca03ad1093dfba7d3523ff45789a" Nov 03 01:42:36.003517 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:36.003497 4421 scope.go:122] "RemoveContainer" containerID="ba7b2267ebde4e272760c8646331a1c244e7ca03ad1093dfba7d3523ff45789a" Nov 03 01:42:36.003799 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:42:36.003777 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba7b2267ebde4e272760c8646331a1c244e7ca03ad1093dfba7d3523ff45789a\": container with ID starting with ba7b2267ebde4e272760c8646331a1c244e7ca03ad1093dfba7d3523ff45789a not found: ID does not exist" containerID="ba7b2267ebde4e272760c8646331a1c244e7ca03ad1093dfba7d3523ff45789a" Nov 03 01:42:36.003867 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:36.003810 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba7b2267ebde4e272760c8646331a1c244e7ca03ad1093dfba7d3523ff45789a"} err="failed to get container status \"ba7b2267ebde4e272760c8646331a1c244e7ca03ad1093dfba7d3523ff45789a\": rpc error: code = NotFound desc = could not find container \"ba7b2267ebde4e272760c8646331a1c244e7ca03ad1093dfba7d3523ff45789a\": container with ID starting with ba7b2267ebde4e272760c8646331a1c244e7ca03ad1093dfba7d3523ff45789a not found: ID does not exist" Nov 03 01:42:36.011583 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:36.011555 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-ztm5t"] Nov 03 01:42:36.019552 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:36.019518 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-ztm5t"] Nov 03 01:42:36.419975 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:36.419935 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-4af1f7-7fc8d84d47-mp2bh"] Nov 03 01:42:36.435504 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:36.435470 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-4af1f7-7cc74c488c-tsnxd"] Nov 03 01:42:37.978161 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:37.978126 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc7f30da-8b1e-42d7-a41a-2398e66263b7" path="/var/lib/kubelet/pods/dc7f30da-8b1e-42d7-a41a-2398e66263b7/volumes" Nov 03 01:42:40.910900 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:40.910831 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-mlxvq" Nov 03 01:42:41.017742 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:41.017711 4421 generic.go:356] "Generic (PLEG): container finished" podID="dcf063ce-88a5-4341-ae0f-605522bb91d5" containerID="dcf8a1ffba7dd0036d36c7d4b74904648e76bc6c8bdbd457c757c4fc064e6b68" exitCode=0 Nov 03 01:42:41.017914 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:41.017771 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-mlxvq" Nov 03 01:42:41.017914 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:41.017804 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-mlxvq" event={"ID":"dcf063ce-88a5-4341-ae0f-605522bb91d5","Type":"ContainerDied","Data":"dcf8a1ffba7dd0036d36c7d4b74904648e76bc6c8bdbd457c757c4fc064e6b68"} Nov 03 01:42:41.017914 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:41.017861 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-mlxvq" event={"ID":"dcf063ce-88a5-4341-ae0f-605522bb91d5","Type":"ContainerDied","Data":"4e0b940675d53c6f1e5283d220d817cb8b929b76b12d6c3e416bd7fdacd5e60d"} Nov 03 01:42:41.017914 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:41.017894 4421 scope.go:122] "RemoveContainer" containerID="dcf8a1ffba7dd0036d36c7d4b74904648e76bc6c8bdbd457c757c4fc064e6b68" Nov 03 01:42:41.028413 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:41.028388 4421 scope.go:122] "RemoveContainer" containerID="dcf8a1ffba7dd0036d36c7d4b74904648e76bc6c8bdbd457c757c4fc064e6b68" Nov 03 01:42:41.028795 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:42:41.028771 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dcf8a1ffba7dd0036d36c7d4b74904648e76bc6c8bdbd457c757c4fc064e6b68\": container with ID starting with dcf8a1ffba7dd0036d36c7d4b74904648e76bc6c8bdbd457c757c4fc064e6b68 not found: ID does not exist" containerID="dcf8a1ffba7dd0036d36c7d4b74904648e76bc6c8bdbd457c757c4fc064e6b68" Nov 03 01:42:41.028873 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:41.028803 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dcf8a1ffba7dd0036d36c7d4b74904648e76bc6c8bdbd457c757c4fc064e6b68"} err="failed to get container status \"dcf8a1ffba7dd0036d36c7d4b74904648e76bc6c8bdbd457c757c4fc064e6b68\": rpc error: code = NotFound desc = could not find container \"dcf8a1ffba7dd0036d36c7d4b74904648e76bc6c8bdbd457c757c4fc064e6b68\": container with ID starting with dcf8a1ffba7dd0036d36c7d4b74904648e76bc6c8bdbd457c757c4fc064e6b68 not found: ID does not exist" Nov 03 01:42:41.064996 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:41.064960 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dcf063ce-88a5-4341-ae0f-605522bb91d5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcf063ce-88a5-4341-ae0f-605522bb91d5-istio-podinfo\") pod \"dcf063ce-88a5-4341-ae0f-605522bb91d5\" (UID: \"dcf063ce-88a5-4341-ae0f-605522bb91d5\") " Nov 03 01:42:41.065178 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:41.065011 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dcf063ce-88a5-4341-ae0f-605522bb91d5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcf063ce-88a5-4341-ae0f-605522bb91d5-istiod-ca-cert\") pod \"dcf063ce-88a5-4341-ae0f-605522bb91d5\" (UID: \"dcf063ce-88a5-4341-ae0f-605522bb91d5\") " Nov 03 01:42:41.065178 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:41.065057 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcf063ce-88a5-4341-ae0f-605522bb91d5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcf063ce-88a5-4341-ae0f-605522bb91d5-credential-socket\") pod \"dcf063ce-88a5-4341-ae0f-605522bb91d5\" (UID: \"dcf063ce-88a5-4341-ae0f-605522bb91d5\") " Nov 03 01:42:41.065178 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:41.065077 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcf063ce-88a5-4341-ae0f-605522bb91d5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcf063ce-88a5-4341-ae0f-605522bb91d5-istio-envoy\") pod \"dcf063ce-88a5-4341-ae0f-605522bb91d5\" (UID: \"dcf063ce-88a5-4341-ae0f-605522bb91d5\") " Nov 03 01:42:41.065178 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:41.065107 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcf063ce-88a5-4341-ae0f-605522bb91d5-kube-api-access-fmhp8\" (UniqueName: \"kubernetes.io/projected/dcf063ce-88a5-4341-ae0f-605522bb91d5-kube-api-access-fmhp8\") pod \"dcf063ce-88a5-4341-ae0f-605522bb91d5\" (UID: \"dcf063ce-88a5-4341-ae0f-605522bb91d5\") " Nov 03 01:42:41.065453 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:41.065190 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcf063ce-88a5-4341-ae0f-605522bb91d5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcf063ce-88a5-4341-ae0f-605522bb91d5-istio-data\") pod \"dcf063ce-88a5-4341-ae0f-605522bb91d5\" (UID: \"dcf063ce-88a5-4341-ae0f-605522bb91d5\") " Nov 03 01:42:41.065453 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:41.065219 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcf063ce-88a5-4341-ae0f-605522bb91d5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcf063ce-88a5-4341-ae0f-605522bb91d5-workload-socket\") pod \"dcf063ce-88a5-4341-ae0f-605522bb91d5\" (UID: \"dcf063ce-88a5-4341-ae0f-605522bb91d5\") " Nov 03 01:42:41.065453 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:41.065253 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcf063ce-88a5-4341-ae0f-605522bb91d5-istio-token\" (UniqueName: \"kubernetes.io/projected/dcf063ce-88a5-4341-ae0f-605522bb91d5-istio-token\") pod \"dcf063ce-88a5-4341-ae0f-605522bb91d5\" (UID: \"dcf063ce-88a5-4341-ae0f-605522bb91d5\") " Nov 03 01:42:41.065453 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:41.065318 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcf063ce-88a5-4341-ae0f-605522bb91d5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcf063ce-88a5-4341-ae0f-605522bb91d5-workload-certs\") pod \"dcf063ce-88a5-4341-ae0f-605522bb91d5\" (UID: \"dcf063ce-88a5-4341-ae0f-605522bb91d5\") " Nov 03 01:42:41.065676 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:41.065521 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcf063ce-88a5-4341-ae0f-605522bb91d5-credential-socket" pod "dcf063ce-88a5-4341-ae0f-605522bb91d5" (UID: "dcf063ce-88a5-4341-ae0f-605522bb91d5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:42:41.065864 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:41.065835 4421 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcf063ce-88a5-4341-ae0f-605522bb91d5-credential-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:41.066125 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:41.066083 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcf063ce-88a5-4341-ae0f-605522bb91d5-istiod-ca-cert" pod "dcf063ce-88a5-4341-ae0f-605522bb91d5" (UID: "dcf063ce-88a5-4341-ae0f-605522bb91d5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:42:41.066744 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:41.066710 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcf063ce-88a5-4341-ae0f-605522bb91d5-workload-certs" pod "dcf063ce-88a5-4341-ae0f-605522bb91d5" (UID: "dcf063ce-88a5-4341-ae0f-605522bb91d5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:42:41.066861 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:41.066751 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcf063ce-88a5-4341-ae0f-605522bb91d5-workload-socket" pod "dcf063ce-88a5-4341-ae0f-605522bb91d5" (UID: "dcf063ce-88a5-4341-ae0f-605522bb91d5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:42:41.066861 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:41.066767 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcf063ce-88a5-4341-ae0f-605522bb91d5-istio-data" pod "dcf063ce-88a5-4341-ae0f-605522bb91d5" (UID: "dcf063ce-88a5-4341-ae0f-605522bb91d5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:42:41.067599 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:41.067576 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcf063ce-88a5-4341-ae0f-605522bb91d5-kube-api-access-fmhp8" pod "dcf063ce-88a5-4341-ae0f-605522bb91d5" (UID: "dcf063ce-88a5-4341-ae0f-605522bb91d5"). InnerVolumeSpecName "kube-api-access-fmhp8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:42:41.067696 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:41.067609 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dcf063ce-88a5-4341-ae0f-605522bb91d5-istio-podinfo" pod "dcf063ce-88a5-4341-ae0f-605522bb91d5" (UID: "dcf063ce-88a5-4341-ae0f-605522bb91d5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:42:41.067696 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:41.067660 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcf063ce-88a5-4341-ae0f-605522bb91d5-istio-envoy" pod "dcf063ce-88a5-4341-ae0f-605522bb91d5" (UID: "dcf063ce-88a5-4341-ae0f-605522bb91d5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:42:41.068155 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:41.068131 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcf063ce-88a5-4341-ae0f-605522bb91d5-istio-token" pod "dcf063ce-88a5-4341-ae0f-605522bb91d5" (UID: "dcf063ce-88a5-4341-ae0f-605522bb91d5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:42:41.166947 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:41.166856 4421 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcf063ce-88a5-4341-ae0f-605522bb91d5-istiod-ca-cert\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:41.166947 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:41.166889 4421 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcf063ce-88a5-4341-ae0f-605522bb91d5-istio-envoy\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:41.166947 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:41.166902 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmhp8\" (UniqueName: \"kubernetes.io/projected/dcf063ce-88a5-4341-ae0f-605522bb91d5-kube-api-access-fmhp8\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:41.166947 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:41.166913 4421 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcf063ce-88a5-4341-ae0f-605522bb91d5-istio-data\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:41.166947 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:41.166922 4421 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcf063ce-88a5-4341-ae0f-605522bb91d5-workload-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:41.166947 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:41.166930 4421 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcf063ce-88a5-4341-ae0f-605522bb91d5-istio-token\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:41.166947 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:41.166939 4421 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcf063ce-88a5-4341-ae0f-605522bb91d5-workload-certs\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:41.166947 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:41.166949 4421 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcf063ce-88a5-4341-ae0f-605522bb91d5-istio-podinfo\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:41.342804 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:41.342769 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-mlxvq"] Nov 03 01:42:41.348779 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:41.348750 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-mlxvq"] Nov 03 01:42:41.423873 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:41.423306 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-4af1f7-7fc8d84d47-mp2bh" podUID="af3bcba4-1005-480a-859f-8fc73f0cbe53" containerName="payload-processing" containerID="cri-o://74547c4d0d974eb572b4856b9a95a03223bdfbaa4f775e5000f3306df8c788be" gracePeriod=25 Nov 03 01:42:41.435968 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:41.435923 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-4af1f7-7cc74c488c-tsnxd" podUID="6ff46e7d-ec61-4e17-ae76-0dc25edd95b4" containerName="payload-pre-processing" containerID="cri-o://64e2268ac33a34235b706162fd388297fc7e09136e6c290a7eb4b036e78812dc" gracePeriod=25 Nov 03 01:42:41.978482 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:41.978447 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dcf063ce-88a5-4341-ae0f-605522bb91d5" path="/var/lib/kubelet/pods/dcf063ce-88a5-4341-ae0f-605522bb91d5/volumes" Nov 03 01:42:42.768455 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:42.768412 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-4af1f7-7cc74c488c-tsnxd" podUID="6ff46e7d-ec61-4e17-ae76-0dc25edd95b4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:42:42.768618 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:42.768420 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-4af1f7-7fc8d84d47-mp2bh" podUID="af3bcba4-1005-480a-859f-8fc73f0cbe53" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:42:44.024535 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:44.024487 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-031e492b-hpfg6"] Nov 03 01:42:44.027473 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:44.027447 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-031e492b-hpfg6"] Nov 03 01:42:44.669370 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:42:44.669331 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 01:42:44.743661 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:44.743624 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-67sww"] Nov 03 01:42:44.743937 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:44.743884 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-67sww" podUID="35687f7e-87be-465b-ab66-ec518b459edc" containerName="istio-proxy" containerID="cri-o://1de77136bb6bdb59b4401f8aaffa55e621774945ac035c09c55473d1d1c2042e" gracePeriod=30 Nov 03 01:42:45.978224 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:45.978180 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="617b1b92-68e6-47cd-ac46-2542031c7efc" path="/var/lib/kubelet/pods/617b1b92-68e6-47cd-ac46-2542031c7efc/volumes" Nov 03 01:42:50.004229 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:50.004199 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-67sww" Nov 03 01:42:50.083573 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:50.083474 4421 generic.go:356] "Generic (PLEG): container finished" podID="35687f7e-87be-465b-ab66-ec518b459edc" containerID="1de77136bb6bdb59b4401f8aaffa55e621774945ac035c09c55473d1d1c2042e" exitCode=0 Nov 03 01:42:50.083573 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:50.083538 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-67sww" Nov 03 01:42:50.083573 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:50.083542 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-67sww" event={"ID":"35687f7e-87be-465b-ab66-ec518b459edc","Type":"ContainerDied","Data":"1de77136bb6bdb59b4401f8aaffa55e621774945ac035c09c55473d1d1c2042e"} Nov 03 01:42:50.083882 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:50.083598 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-67sww" event={"ID":"35687f7e-87be-465b-ab66-ec518b459edc","Type":"ContainerDied","Data":"69956387b6fd5667d4dc9c7d18be5c787ca0667741fc584cfbad8a2d4840802c"} Nov 03 01:42:50.083882 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:50.083621 4421 scope.go:122] "RemoveContainer" containerID="1de77136bb6bdb59b4401f8aaffa55e621774945ac035c09c55473d1d1c2042e" Nov 03 01:42:50.093441 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:50.093414 4421 scope.go:122] "RemoveContainer" containerID="1de77136bb6bdb59b4401f8aaffa55e621774945ac035c09c55473d1d1c2042e" Nov 03 01:42:50.093757 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:42:50.093737 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1de77136bb6bdb59b4401f8aaffa55e621774945ac035c09c55473d1d1c2042e\": container with ID starting with 1de77136bb6bdb59b4401f8aaffa55e621774945ac035c09c55473d1d1c2042e not found: ID does not exist" containerID="1de77136bb6bdb59b4401f8aaffa55e621774945ac035c09c55473d1d1c2042e" Nov 03 01:42:50.093813 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:50.093767 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1de77136bb6bdb59b4401f8aaffa55e621774945ac035c09c55473d1d1c2042e"} err="failed to get container status \"1de77136bb6bdb59b4401f8aaffa55e621774945ac035c09c55473d1d1c2042e\": rpc error: code = NotFound desc = could not find container \"1de77136bb6bdb59b4401f8aaffa55e621774945ac035c09c55473d1d1c2042e\": container with ID starting with 1de77136bb6bdb59b4401f8aaffa55e621774945ac035c09c55473d1d1c2042e not found: ID does not exist" Nov 03 01:42:50.170560 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:50.170502 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35687f7e-87be-465b-ab66-ec518b459edc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35687f7e-87be-465b-ab66-ec518b459edc-workload-certs\") pod \"35687f7e-87be-465b-ab66-ec518b459edc\" (UID: \"35687f7e-87be-465b-ab66-ec518b459edc\") " Nov 03 01:42:50.170753 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:50.170589 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35687f7e-87be-465b-ab66-ec518b459edc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35687f7e-87be-465b-ab66-ec518b459edc-workload-socket\") pod \"35687f7e-87be-465b-ab66-ec518b459edc\" (UID: \"35687f7e-87be-465b-ab66-ec518b459edc\") " Nov 03 01:42:50.170753 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:50.170637 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35687f7e-87be-465b-ab66-ec518b459edc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35687f7e-87be-465b-ab66-ec518b459edc-credential-socket\") pod \"35687f7e-87be-465b-ab66-ec518b459edc\" (UID: \"35687f7e-87be-465b-ab66-ec518b459edc\") " Nov 03 01:42:50.170753 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:50.170677 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35687f7e-87be-465b-ab66-ec518b459edc-kube-api-access-2hg25\" (UniqueName: \"kubernetes.io/projected/35687f7e-87be-465b-ab66-ec518b459edc-kube-api-access-2hg25\") pod \"35687f7e-87be-465b-ab66-ec518b459edc\" (UID: \"35687f7e-87be-465b-ab66-ec518b459edc\") " Nov 03 01:42:50.170753 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:50.170701 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/35687f7e-87be-465b-ab66-ec518b459edc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35687f7e-87be-465b-ab66-ec518b459edc-istio-podinfo\") pod \"35687f7e-87be-465b-ab66-ec518b459edc\" (UID: \"35687f7e-87be-465b-ab66-ec518b459edc\") " Nov 03 01:42:50.170753 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:50.170737 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35687f7e-87be-465b-ab66-ec518b459edc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35687f7e-87be-465b-ab66-ec518b459edc-istio-envoy\") pod \"35687f7e-87be-465b-ab66-ec518b459edc\" (UID: \"35687f7e-87be-465b-ab66-ec518b459edc\") " Nov 03 01:42:50.171057 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:50.170771 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35687f7e-87be-465b-ab66-ec518b459edc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35687f7e-87be-465b-ab66-ec518b459edc-istiod-ca-cert\") pod \"35687f7e-87be-465b-ab66-ec518b459edc\" (UID: \"35687f7e-87be-465b-ab66-ec518b459edc\") " Nov 03 01:42:50.171057 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:50.170786 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35687f7e-87be-465b-ab66-ec518b459edc-workload-certs" pod "35687f7e-87be-465b-ab66-ec518b459edc" (UID: "35687f7e-87be-465b-ab66-ec518b459edc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:42:50.171057 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:50.170849 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35687f7e-87be-465b-ab66-ec518b459edc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35687f7e-87be-465b-ab66-ec518b459edc-istio-data\") pod \"35687f7e-87be-465b-ab66-ec518b459edc\" (UID: \"35687f7e-87be-465b-ab66-ec518b459edc\") " Nov 03 01:42:50.171057 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:50.170931 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35687f7e-87be-465b-ab66-ec518b459edc-istio-token\" (UniqueName: \"kubernetes.io/projected/35687f7e-87be-465b-ab66-ec518b459edc-istio-token\") pod \"35687f7e-87be-465b-ab66-ec518b459edc\" (UID: \"35687f7e-87be-465b-ab66-ec518b459edc\") " Nov 03 01:42:50.171057 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:50.170946 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35687f7e-87be-465b-ab66-ec518b459edc-workload-socket" pod "35687f7e-87be-465b-ab66-ec518b459edc" (UID: "35687f7e-87be-465b-ab66-ec518b459edc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:42:50.171057 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:50.171025 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35687f7e-87be-465b-ab66-ec518b459edc-credential-socket" pod "35687f7e-87be-465b-ab66-ec518b459edc" (UID: "35687f7e-87be-465b-ab66-ec518b459edc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:42:50.171379 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:50.171263 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35687f7e-87be-465b-ab66-ec518b459edc-istiod-ca-cert" pod "35687f7e-87be-465b-ab66-ec518b459edc" (UID: "35687f7e-87be-465b-ab66-ec518b459edc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:42:50.171466 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:50.171436 4421 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35687f7e-87be-465b-ab66-ec518b459edc-workload-certs\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:50.171560 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:50.171473 4421 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35687f7e-87be-465b-ab66-ec518b459edc-workload-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:50.171560 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:50.171491 4421 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35687f7e-87be-465b-ab66-ec518b459edc-credential-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:50.171560 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:50.171505 4421 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35687f7e-87be-465b-ab66-ec518b459edc-istiod-ca-cert\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:50.171691 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:50.171572 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35687f7e-87be-465b-ab66-ec518b459edc-istio-data" pod "35687f7e-87be-465b-ab66-ec518b459edc" (UID: "35687f7e-87be-465b-ab66-ec518b459edc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:42:50.173122 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:50.173098 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/35687f7e-87be-465b-ab66-ec518b459edc-istio-podinfo" pod "35687f7e-87be-465b-ab66-ec518b459edc" (UID: "35687f7e-87be-465b-ab66-ec518b459edc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:42:50.173239 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:50.173101 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35687f7e-87be-465b-ab66-ec518b459edc-kube-api-access-2hg25" pod "35687f7e-87be-465b-ab66-ec518b459edc" (UID: "35687f7e-87be-465b-ab66-ec518b459edc"). InnerVolumeSpecName "kube-api-access-2hg25". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:42:50.173411 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:50.173392 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35687f7e-87be-465b-ab66-ec518b459edc-istio-token" pod "35687f7e-87be-465b-ab66-ec518b459edc" (UID: "35687f7e-87be-465b-ab66-ec518b459edc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:42:50.173411 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:50.173400 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35687f7e-87be-465b-ab66-ec518b459edc-istio-envoy" pod "35687f7e-87be-465b-ab66-ec518b459edc" (UID: "35687f7e-87be-465b-ab66-ec518b459edc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:42:50.272779 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:50.272745 4421 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35687f7e-87be-465b-ab66-ec518b459edc-istio-token\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:50.272779 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:50.272777 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hg25\" (UniqueName: \"kubernetes.io/projected/35687f7e-87be-465b-ab66-ec518b459edc-kube-api-access-2hg25\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:50.273004 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:50.272788 4421 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35687f7e-87be-465b-ab66-ec518b459edc-istio-podinfo\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:50.273004 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:50.272797 4421 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35687f7e-87be-465b-ab66-ec518b459edc-istio-envoy\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:50.273004 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:50.272806 4421 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35687f7e-87be-465b-ab66-ec518b459edc-istio-data\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:50.408154 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:50.408087 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-67sww"] Nov 03 01:42:50.411458 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:50.411427 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-67sww"] Nov 03 01:42:51.977980 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:51.977946 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35687f7e-87be-465b-ab66-ec518b459edc" path="/var/lib/kubelet/pods/35687f7e-87be-465b-ab66-ec518b459edc/volumes" Nov 03 01:42:52.769145 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:52.769106 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-4af1f7-7cc74c488c-tsnxd" podUID="6ff46e7d-ec61-4e17-ae76-0dc25edd95b4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:42:52.769145 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:52.769138 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-4af1f7-7fc8d84d47-mp2bh" podUID="af3bcba4-1005-480a-859f-8fc73f0cbe53" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:42:56.670340 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:56.670304 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-88bbf7-78b86696f9-x7mgb"] Nov 03 01:42:56.697087 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:42:56.697054 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-88bbf7-554c5df5-dxctw"] Nov 03 01:43:01.671037 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:01.670941 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-88bbf7-78b86696f9-x7mgb" podUID="a454ee55-ade7-4715-bf20-8cc451458d99" containerName="payload-processing" containerID="cri-o://1d8ffb67f7937117e6bd70d56767fe923a236863c7c970cecb5a9b92948570d3" gracePeriod=25 Nov 03 01:43:01.703841 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:01.703146 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-88bbf7-554c5df5-dxctw" podUID="4f2cede9-84b3-4d23-8456-38016e4ff7fb" containerName="payload-pre-processing" containerID="cri-o://0ff9ab7becf756d32b71503c6f764e90d53a670eec8139025660a683194927a7" gracePeriod=25 Nov 03 01:43:01.754499 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:01.754471 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-4af1f7-7fc8d84d47-mp2bh" Nov 03 01:43:01.757701 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:01.757679 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-4af1f7-7cc74c488c-tsnxd" Nov 03 01:43:01.805009 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:01.804976 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ff46e7d-ec61-4e17-ae76-0dc25edd95b4-kube-api-access-qqxp5\" (UniqueName: \"kubernetes.io/projected/6ff46e7d-ec61-4e17-ae76-0dc25edd95b4-kube-api-access-qqxp5\") pod \"6ff46e7d-ec61-4e17-ae76-0dc25edd95b4\" (UID: \"6ff46e7d-ec61-4e17-ae76-0dc25edd95b4\") " Nov 03 01:43:01.805213 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:01.805062 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ff46e7d-ec61-4e17-ae76-0dc25edd95b4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ff46e7d-ec61-4e17-ae76-0dc25edd95b4-plugins-config-volume\") pod \"6ff46e7d-ec61-4e17-ae76-0dc25edd95b4\" (UID: \"6ff46e7d-ec61-4e17-ae76-0dc25edd95b4\") " Nov 03 01:43:01.805213 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:01.805127 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af3bcba4-1005-480a-859f-8fc73f0cbe53-kube-api-access-5rqtq\" (UniqueName: \"kubernetes.io/projected/af3bcba4-1005-480a-859f-8fc73f0cbe53-kube-api-access-5rqtq\") pod \"af3bcba4-1005-480a-859f-8fc73f0cbe53\" (UID: \"af3bcba4-1005-480a-859f-8fc73f0cbe53\") " Nov 03 01:43:01.805213 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:01.805153 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af3bcba4-1005-480a-859f-8fc73f0cbe53-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af3bcba4-1005-480a-859f-8fc73f0cbe53-plugins-config-volume\") pod \"af3bcba4-1005-480a-859f-8fc73f0cbe53\" (UID: \"af3bcba4-1005-480a-859f-8fc73f0cbe53\") " Nov 03 01:43:01.805513 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:01.805485 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ff46e7d-ec61-4e17-ae76-0dc25edd95b4-plugins-config-volume" pod "6ff46e7d-ec61-4e17-ae76-0dc25edd95b4" (UID: "6ff46e7d-ec61-4e17-ae76-0dc25edd95b4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:43:01.805627 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:01.805566 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af3bcba4-1005-480a-859f-8fc73f0cbe53-plugins-config-volume" pod "af3bcba4-1005-480a-859f-8fc73f0cbe53" (UID: "af3bcba4-1005-480a-859f-8fc73f0cbe53"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:43:01.807391 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:01.807358 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af3bcba4-1005-480a-859f-8fc73f0cbe53-kube-api-access-5rqtq" pod "af3bcba4-1005-480a-859f-8fc73f0cbe53" (UID: "af3bcba4-1005-480a-859f-8fc73f0cbe53"). InnerVolumeSpecName "kube-api-access-5rqtq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:43:01.807521 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:01.807385 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ff46e7d-ec61-4e17-ae76-0dc25edd95b4-kube-api-access-qqxp5" pod "6ff46e7d-ec61-4e17-ae76-0dc25edd95b4" (UID: "6ff46e7d-ec61-4e17-ae76-0dc25edd95b4"). InnerVolumeSpecName "kube-api-access-qqxp5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:43:01.906123 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:01.906080 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqxp5\" (UniqueName: \"kubernetes.io/projected/6ff46e7d-ec61-4e17-ae76-0dc25edd95b4-kube-api-access-qqxp5\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:43:01.906123 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:01.906115 4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ff46e7d-ec61-4e17-ae76-0dc25edd95b4-plugins-config-volume\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:43:01.906123 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:01.906129 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rqtq\" (UniqueName: \"kubernetes.io/projected/af3bcba4-1005-480a-859f-8fc73f0cbe53-kube-api-access-5rqtq\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:43:01.906410 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:01.906143 4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af3bcba4-1005-480a-859f-8fc73f0cbe53-plugins-config-volume\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:43:02.029639 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:02.029594 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9fa9240-slh8k"] Nov 03 01:43:02.032765 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:02.032736 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9fa9240-slh8k"] Nov 03 01:43:02.163235 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:02.163204 4421 generic.go:356] "Generic (PLEG): container finished" podID="af3bcba4-1005-480a-859f-8fc73f0cbe53" containerID="74547c4d0d974eb572b4856b9a95a03223bdfbaa4f775e5000f3306df8c788be" exitCode=0 Nov 03 01:43:02.163453 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:02.163268 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-4af1f7-7fc8d84d47-mp2bh" Nov 03 01:43:02.163453 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:02.163295 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-4af1f7-7fc8d84d47-mp2bh" event={"ID":"af3bcba4-1005-480a-859f-8fc73f0cbe53","Type":"ContainerDied","Data":"74547c4d0d974eb572b4856b9a95a03223bdfbaa4f775e5000f3306df8c788be"} Nov 03 01:43:02.163453 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:02.163354 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-4af1f7-7fc8d84d47-mp2bh" event={"ID":"af3bcba4-1005-480a-859f-8fc73f0cbe53","Type":"ContainerDied","Data":"168ea17e1d3f76f3d1ef595095e6f820a94ad533f5971be7dacd45d1a31f42b6"} Nov 03 01:43:02.163453 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:02.163378 4421 scope.go:122] "RemoveContainer" containerID="74547c4d0d974eb572b4856b9a95a03223bdfbaa4f775e5000f3306df8c788be" Nov 03 01:43:02.164907 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:02.164888 4421 generic.go:356] "Generic (PLEG): container finished" podID="6ff46e7d-ec61-4e17-ae76-0dc25edd95b4" containerID="64e2268ac33a34235b706162fd388297fc7e09136e6c290a7eb4b036e78812dc" exitCode=0 Nov 03 01:43:02.165056 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:02.164930 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-4af1f7-7cc74c488c-tsnxd" Nov 03 01:43:02.165056 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:02.164982 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-4af1f7-7cc74c488c-tsnxd" event={"ID":"6ff46e7d-ec61-4e17-ae76-0dc25edd95b4","Type":"ContainerDied","Data":"64e2268ac33a34235b706162fd388297fc7e09136e6c290a7eb4b036e78812dc"} Nov 03 01:43:02.165056 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:02.165039 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-4af1f7-7cc74c488c-tsnxd" event={"ID":"6ff46e7d-ec61-4e17-ae76-0dc25edd95b4","Type":"ContainerDied","Data":"50dcd91fbeec832bd266e47e22c842128bcde4aaca98a3dae4d1f10b1e0b74d0"} Nov 03 01:43:02.174487 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:02.174470 4421 scope.go:122] "RemoveContainer" containerID="74547c4d0d974eb572b4856b9a95a03223bdfbaa4f775e5000f3306df8c788be" Nov 03 01:43:02.174802 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:43:02.174782 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74547c4d0d974eb572b4856b9a95a03223bdfbaa4f775e5000f3306df8c788be\": container with ID starting with 74547c4d0d974eb572b4856b9a95a03223bdfbaa4f775e5000f3306df8c788be not found: ID does not exist" containerID="74547c4d0d974eb572b4856b9a95a03223bdfbaa4f775e5000f3306df8c788be" Nov 03 01:43:02.174884 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:02.174812 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74547c4d0d974eb572b4856b9a95a03223bdfbaa4f775e5000f3306df8c788be"} err="failed to get container status \"74547c4d0d974eb572b4856b9a95a03223bdfbaa4f775e5000f3306df8c788be\": rpc error: code = NotFound desc = could not find container \"74547c4d0d974eb572b4856b9a95a03223bdfbaa4f775e5000f3306df8c788be\": container with ID starting with 74547c4d0d974eb572b4856b9a95a03223bdfbaa4f775e5000f3306df8c788be not found: ID does not exist" Nov 03 01:43:02.174884 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:02.174830 4421 scope.go:122] "RemoveContainer" containerID="64e2268ac33a34235b706162fd388297fc7e09136e6c290a7eb4b036e78812dc" Nov 03 01:43:02.184571 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:02.184242 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-4af1f7-7cc74c488c-tsnxd"] Nov 03 01:43:02.187364 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:02.187335 4421 scope.go:122] "RemoveContainer" containerID="64e2268ac33a34235b706162fd388297fc7e09136e6c290a7eb4b036e78812dc" Nov 03 01:43:02.187735 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:43:02.187710 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64e2268ac33a34235b706162fd388297fc7e09136e6c290a7eb4b036e78812dc\": container with ID starting with 64e2268ac33a34235b706162fd388297fc7e09136e6c290a7eb4b036e78812dc not found: ID does not exist" containerID="64e2268ac33a34235b706162fd388297fc7e09136e6c290a7eb4b036e78812dc" Nov 03 01:43:02.187804 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:02.187742 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64e2268ac33a34235b706162fd388297fc7e09136e6c290a7eb4b036e78812dc"} err="failed to get container status \"64e2268ac33a34235b706162fd388297fc7e09136e6c290a7eb4b036e78812dc\": rpc error: code = NotFound desc = could not find container \"64e2268ac33a34235b706162fd388297fc7e09136e6c290a7eb4b036e78812dc\": container with ID starting with 64e2268ac33a34235b706162fd388297fc7e09136e6c290a7eb4b036e78812dc not found: ID does not exist" Nov 03 01:43:02.188701 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:02.188680 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-4af1f7-7cc74c488c-tsnxd"] Nov 03 01:43:02.198762 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:02.198734 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-4af1f7-7fc8d84d47-mp2bh"] Nov 03 01:43:02.201586 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:02.201563 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-4af1f7-7fc8d84d47-mp2bh"] Nov 03 01:43:03.977696 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:03.977659 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10544138-8e08-412d-9551-6e31301544ee" path="/var/lib/kubelet/pods/10544138-8e08-412d-9551-6e31301544ee/volumes" Nov 03 01:43:03.978140 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:03.978124 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ff46e7d-ec61-4e17-ae76-0dc25edd95b4" path="/var/lib/kubelet/pods/6ff46e7d-ec61-4e17-ae76-0dc25edd95b4/volumes" Nov 03 01:43:03.978550 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:03.978527 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af3bcba4-1005-480a-859f-8fc73f0cbe53" path="/var/lib/kubelet/pods/af3bcba4-1005-480a-859f-8fc73f0cbe53/volumes" Nov 03 01:43:06.097525 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:06.097481 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-88bbf7-78b86696f9-x7mgb" podUID="a454ee55-ade7-4715-bf20-8cc451458d99" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:43:06.097960 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:06.097490 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-88bbf7-554c5df5-dxctw" podUID="4f2cede9-84b3-4d23-8456-38016e4ff7fb" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:43:14.669335 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:43:14.669236 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 01:43:16.097445 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:16.097406 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-88bbf7-78b86696f9-x7mgb" podUID="a454ee55-ade7-4715-bf20-8cc451458d99" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:43:16.097851 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:16.097472 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-88bbf7-554c5df5-dxctw" podUID="4f2cede9-84b3-4d23-8456-38016e4ff7fb" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:43:21.927683 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:21.927614 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-88bbf7-78b86696f9-x7mgb" Nov 03 01:43:21.965358 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:21.965327 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-88bbf7-554c5df5-dxctw" Nov 03 01:43:21.994333 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:21.994299 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a454ee55-ade7-4715-bf20-8cc451458d99-kube-api-access-plh56\" (UniqueName: \"kubernetes.io/projected/a454ee55-ade7-4715-bf20-8cc451458d99-kube-api-access-plh56\") pod \"a454ee55-ade7-4715-bf20-8cc451458d99\" (UID: \"a454ee55-ade7-4715-bf20-8cc451458d99\") " Nov 03 01:43:21.994521 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:21.994420 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f2cede9-84b3-4d23-8456-38016e4ff7fb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f2cede9-84b3-4d23-8456-38016e4ff7fb-plugins-config-volume\") pod \"4f2cede9-84b3-4d23-8456-38016e4ff7fb\" (UID: \"4f2cede9-84b3-4d23-8456-38016e4ff7fb\") " Nov 03 01:43:21.994521 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:21.994473 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f2cede9-84b3-4d23-8456-38016e4ff7fb-kube-api-access-knf9j\" (UniqueName: \"kubernetes.io/projected/4f2cede9-84b3-4d23-8456-38016e4ff7fb-kube-api-access-knf9j\") pod \"4f2cede9-84b3-4d23-8456-38016e4ff7fb\" (UID: \"4f2cede9-84b3-4d23-8456-38016e4ff7fb\") " Nov 03 01:43:21.994666 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:21.994590 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a454ee55-ade7-4715-bf20-8cc451458d99-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a454ee55-ade7-4715-bf20-8cc451458d99-plugins-config-volume\") pod \"a454ee55-ade7-4715-bf20-8cc451458d99\" (UID: \"a454ee55-ade7-4715-bf20-8cc451458d99\") " Nov 03 01:43:21.994965 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:21.994925 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f2cede9-84b3-4d23-8456-38016e4ff7fb-plugins-config-volume" pod "4f2cede9-84b3-4d23-8456-38016e4ff7fb" (UID: "4f2cede9-84b3-4d23-8456-38016e4ff7fb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:43:21.994965 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:21.994944 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a454ee55-ade7-4715-bf20-8cc451458d99-plugins-config-volume" pod "a454ee55-ade7-4715-bf20-8cc451458d99" (UID: "a454ee55-ade7-4715-bf20-8cc451458d99"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:43:21.996595 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:21.996568 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f2cede9-84b3-4d23-8456-38016e4ff7fb-kube-api-access-knf9j" pod "4f2cede9-84b3-4d23-8456-38016e4ff7fb" (UID: "4f2cede9-84b3-4d23-8456-38016e4ff7fb"). InnerVolumeSpecName "kube-api-access-knf9j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:43:21.996709 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:21.996595 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a454ee55-ade7-4715-bf20-8cc451458d99-kube-api-access-plh56" pod "a454ee55-ade7-4715-bf20-8cc451458d99" (UID: "a454ee55-ade7-4715-bf20-8cc451458d99"). InnerVolumeSpecName "kube-api-access-plh56". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:43:22.096483 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:22.096448 4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a454ee55-ade7-4715-bf20-8cc451458d99-plugins-config-volume\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:43:22.096483 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:22.096477 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-plh56\" (UniqueName: \"kubernetes.io/projected/a454ee55-ade7-4715-bf20-8cc451458d99-kube-api-access-plh56\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:43:22.096483 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:22.096487 4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f2cede9-84b3-4d23-8456-38016e4ff7fb-plugins-config-volume\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:43:22.096734 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:22.096497 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-knf9j\" (UniqueName: \"kubernetes.io/projected/4f2cede9-84b3-4d23-8456-38016e4ff7fb-kube-api-access-knf9j\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:43:22.282519 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:22.282482 4421 generic.go:356] "Generic (PLEG): container finished" podID="4f2cede9-84b3-4d23-8456-38016e4ff7fb" containerID="0ff9ab7becf756d32b71503c6f764e90d53a670eec8139025660a683194927a7" exitCode=0 Nov 03 01:43:22.282746 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:22.282539 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-88bbf7-554c5df5-dxctw" Nov 03 01:43:22.282746 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:22.282569 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-88bbf7-554c5df5-dxctw" event={"ID":"4f2cede9-84b3-4d23-8456-38016e4ff7fb","Type":"ContainerDied","Data":"0ff9ab7becf756d32b71503c6f764e90d53a670eec8139025660a683194927a7"} Nov 03 01:43:22.282746 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:22.282631 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-88bbf7-554c5df5-dxctw" event={"ID":"4f2cede9-84b3-4d23-8456-38016e4ff7fb","Type":"ContainerDied","Data":"a2911f66a6a8f3cfb35b1e461a88733ec65462629d7c5b85c904cf1ba1531971"} Nov 03 01:43:22.282746 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:22.282655 4421 scope.go:122] "RemoveContainer" containerID="0ff9ab7becf756d32b71503c6f764e90d53a670eec8139025660a683194927a7" Nov 03 01:43:22.284232 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:22.284212 4421 generic.go:356] "Generic (PLEG): container finished" podID="a454ee55-ade7-4715-bf20-8cc451458d99" containerID="1d8ffb67f7937117e6bd70d56767fe923a236863c7c970cecb5a9b92948570d3" exitCode=0 Nov 03 01:43:22.284404 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:22.284260 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-88bbf7-78b86696f9-x7mgb" event={"ID":"a454ee55-ade7-4715-bf20-8cc451458d99","Type":"ContainerDied","Data":"1d8ffb67f7937117e6bd70d56767fe923a236863c7c970cecb5a9b92948570d3"} Nov 03 01:43:22.284404 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:22.284313 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-88bbf7-78b86696f9-x7mgb" event={"ID":"a454ee55-ade7-4715-bf20-8cc451458d99","Type":"ContainerDied","Data":"ac7fdeb1c87c6eee91d403b70bf45e60abb903f9ba8f6b9b0ca8dafcf5e438c6"} Nov 03 01:43:22.284404 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:22.284340 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-88bbf7-78b86696f9-x7mgb" Nov 03 01:43:22.294062 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:22.294042 4421 scope.go:122] "RemoveContainer" containerID="0ff9ab7becf756d32b71503c6f764e90d53a670eec8139025660a683194927a7" Nov 03 01:43:22.294603 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:43:22.294376 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ff9ab7becf756d32b71503c6f764e90d53a670eec8139025660a683194927a7\": container with ID starting with 0ff9ab7becf756d32b71503c6f764e90d53a670eec8139025660a683194927a7 not found: ID does not exist" containerID="0ff9ab7becf756d32b71503c6f764e90d53a670eec8139025660a683194927a7" Nov 03 01:43:22.294603 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:22.294405 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ff9ab7becf756d32b71503c6f764e90d53a670eec8139025660a683194927a7"} err="failed to get container status \"0ff9ab7becf756d32b71503c6f764e90d53a670eec8139025660a683194927a7\": rpc error: code = NotFound desc = could not find container \"0ff9ab7becf756d32b71503c6f764e90d53a670eec8139025660a683194927a7\": container with ID starting with 0ff9ab7becf756d32b71503c6f764e90d53a670eec8139025660a683194927a7 not found: ID does not exist" Nov 03 01:43:22.294603 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:22.294422 4421 scope.go:122] "RemoveContainer" containerID="1d8ffb67f7937117e6bd70d56767fe923a236863c7c970cecb5a9b92948570d3" Nov 03 01:43:22.305188 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:22.305161 4421 scope.go:122] "RemoveContainer" containerID="1d8ffb67f7937117e6bd70d56767fe923a236863c7c970cecb5a9b92948570d3" Nov 03 01:43:22.305544 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:43:22.305522 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d8ffb67f7937117e6bd70d56767fe923a236863c7c970cecb5a9b92948570d3\": container with ID starting with 1d8ffb67f7937117e6bd70d56767fe923a236863c7c970cecb5a9b92948570d3 not found: ID does not exist" containerID="1d8ffb67f7937117e6bd70d56767fe923a236863c7c970cecb5a9b92948570d3" Nov 03 01:43:22.305700 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:22.305552 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d8ffb67f7937117e6bd70d56767fe923a236863c7c970cecb5a9b92948570d3"} err="failed to get container status \"1d8ffb67f7937117e6bd70d56767fe923a236863c7c970cecb5a9b92948570d3\": rpc error: code = NotFound desc = could not find container \"1d8ffb67f7937117e6bd70d56767fe923a236863c7c970cecb5a9b92948570d3\": container with ID starting with 1d8ffb67f7937117e6bd70d56767fe923a236863c7c970cecb5a9b92948570d3 not found: ID does not exist" Nov 03 01:43:22.307031 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:22.307006 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-88bbf7-554c5df5-dxctw"] Nov 03 01:43:22.311727 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:22.311703 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-88bbf7-554c5df5-dxctw"] Nov 03 01:43:22.320549 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:22.320518 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-88bbf7-78b86696f9-x7mgb"] Nov 03 01:43:22.325611 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:22.325581 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-88bbf7-78b86696f9-x7mgb"] Nov 03 01:43:23.979203 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:23.979170 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f2cede9-84b3-4d23-8456-38016e4ff7fb" path="/var/lib/kubelet/pods/4f2cede9-84b3-4d23-8456-38016e4ff7fb/volumes" Nov 03 01:43:23.984363 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:23.984340 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a454ee55-ade7-4715-bf20-8cc451458d99" path="/var/lib/kubelet/pods/a454ee55-ade7-4715-bf20-8cc451458d99/volumes" Nov 03 01:43:29.239122 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:29.239084 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-h9mjc"] Nov 03 01:43:29.239735 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:29.239377 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-h9mjc" podUID="dd5f7acb-d880-4f98-8b64-8501641747b6" containerName="istio-proxy" containerID="cri-o://72ec5e73ad4ad6c8646fb93943166bf654cf890599ac54ec59e7d4e072ccfd7c" gracePeriod=30 Nov 03 01:43:30.338763 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:30.338707 4421 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-597cdd5495-wq4hg_5f9699ce-0019-4989-a2df-3043231dd93e/authorino/0.log" Nov 03 01:43:30.952589 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:30.952560 4421 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-4222l_c2040caa-0454-46bb-bf81-a7063657d4ad/kuadrant-console-plugin/0.log" Nov 03 01:43:31.638515 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:31.638480 4421 patch_prober.go:29] interesting pod/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-h9mjc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 01:43:31.638984 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:31.638535 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-h9mjc" podUID="dd5f7acb-d880-4f98-8b64-8501641747b6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 01:43:34.364864 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:34.364817 4421 generic.go:356] "Generic (PLEG): container finished" podID="dd5f7acb-d880-4f98-8b64-8501641747b6" containerID="72ec5e73ad4ad6c8646fb93943166bf654cf890599ac54ec59e7d4e072ccfd7c" exitCode=0 Nov 03 01:43:34.365247 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:34.364889 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-h9mjc" event={"ID":"dd5f7acb-d880-4f98-8b64-8501641747b6","Type":"ContainerDied","Data":"72ec5e73ad4ad6c8646fb93943166bf654cf890599ac54ec59e7d4e072ccfd7c"} Nov 03 01:43:34.502529 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:34.502505 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-h9mjc" Nov 03 01:43:34.630664 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:34.630567 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd5f7acb-d880-4f98-8b64-8501641747b6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd5f7acb-d880-4f98-8b64-8501641747b6-istio-envoy\") pod \"dd5f7acb-d880-4f98-8b64-8501641747b6\" (UID: \"dd5f7acb-d880-4f98-8b64-8501641747b6\") " Nov 03 01:43:34.630664 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:34.630626 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd5f7acb-d880-4f98-8b64-8501641747b6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd5f7acb-d880-4f98-8b64-8501641747b6-workload-certs\") pod \"dd5f7acb-d880-4f98-8b64-8501641747b6\" (UID: \"dd5f7acb-d880-4f98-8b64-8501641747b6\") " Nov 03 01:43:34.630664 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:34.630659 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd5f7acb-d880-4f98-8b64-8501641747b6-istio-token\" (UniqueName: \"kubernetes.io/projected/dd5f7acb-d880-4f98-8b64-8501641747b6-istio-token\") pod \"dd5f7acb-d880-4f98-8b64-8501641747b6\" (UID: \"dd5f7acb-d880-4f98-8b64-8501641747b6\") " Nov 03 01:43:34.630959 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:34.630703 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd5f7acb-d880-4f98-8b64-8501641747b6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd5f7acb-d880-4f98-8b64-8501641747b6-workload-socket\") pod \"dd5f7acb-d880-4f98-8b64-8501641747b6\" (UID: \"dd5f7acb-d880-4f98-8b64-8501641747b6\") " Nov 03 01:43:34.630959 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:34.630731 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd5f7acb-d880-4f98-8b64-8501641747b6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd5f7acb-d880-4f98-8b64-8501641747b6-istio-data\") pod \"dd5f7acb-d880-4f98-8b64-8501641747b6\" (UID: \"dd5f7acb-d880-4f98-8b64-8501641747b6\") " Nov 03 01:43:34.630959 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:34.630773 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dd5f7acb-d880-4f98-8b64-8501641747b6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd5f7acb-d880-4f98-8b64-8501641747b6-istio-podinfo\") pod \"dd5f7acb-d880-4f98-8b64-8501641747b6\" (UID: \"dd5f7acb-d880-4f98-8b64-8501641747b6\") " Nov 03 01:43:34.630959 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:34.630806 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd5f7acb-d880-4f98-8b64-8501641747b6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd5f7acb-d880-4f98-8b64-8501641747b6-istiod-ca-cert\") pod \"dd5f7acb-d880-4f98-8b64-8501641747b6\" (UID: \"dd5f7acb-d880-4f98-8b64-8501641747b6\") " Nov 03 01:43:34.630959 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:34.630898 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd5f7acb-d880-4f98-8b64-8501641747b6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd5f7acb-d880-4f98-8b64-8501641747b6-credential-socket\") pod \"dd5f7acb-d880-4f98-8b64-8501641747b6\" (UID: \"dd5f7acb-d880-4f98-8b64-8501641747b6\") " Nov 03 01:43:34.631436 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:34.630978 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd5f7acb-d880-4f98-8b64-8501641747b6-kube-api-access-dj7rf\" (UniqueName: \"kubernetes.io/projected/dd5f7acb-d880-4f98-8b64-8501641747b6-kube-api-access-dj7rf\") pod \"dd5f7acb-d880-4f98-8b64-8501641747b6\" (UID: \"dd5f7acb-d880-4f98-8b64-8501641747b6\") " Nov 03 01:43:34.631436 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:34.631094 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd5f7acb-d880-4f98-8b64-8501641747b6-workload-socket" pod "dd5f7acb-d880-4f98-8b64-8501641747b6" (UID: "dd5f7acb-d880-4f98-8b64-8501641747b6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:43:34.631436 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:34.631032 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd5f7acb-d880-4f98-8b64-8501641747b6-workload-certs" pod "dd5f7acb-d880-4f98-8b64-8501641747b6" (UID: "dd5f7acb-d880-4f98-8b64-8501641747b6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:43:34.631436 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:34.631350 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd5f7acb-d880-4f98-8b64-8501641747b6-istiod-ca-cert" pod "dd5f7acb-d880-4f98-8b64-8501641747b6" (UID: "dd5f7acb-d880-4f98-8b64-8501641747b6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:43:34.631436 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:34.631384 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd5f7acb-d880-4f98-8b64-8501641747b6-credential-socket" pod "dd5f7acb-d880-4f98-8b64-8501641747b6" (UID: "dd5f7acb-d880-4f98-8b64-8501641747b6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:43:34.631724 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:34.631474 4421 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd5f7acb-d880-4f98-8b64-8501641747b6-workload-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:43:34.631724 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:34.631499 4421 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd5f7acb-d880-4f98-8b64-8501641747b6-workload-certs\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:43:34.631724 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:34.631527 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd5f7acb-d880-4f98-8b64-8501641747b6-istio-data" pod "dd5f7acb-d880-4f98-8b64-8501641747b6" (UID: "dd5f7acb-d880-4f98-8b64-8501641747b6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:43:34.633071 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:34.633045 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd5f7acb-d880-4f98-8b64-8501641747b6-istio-envoy" pod "dd5f7acb-d880-4f98-8b64-8501641747b6" (UID: "dd5f7acb-d880-4f98-8b64-8501641747b6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:43:34.633198 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:34.633171 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd5f7acb-d880-4f98-8b64-8501641747b6-istio-token" pod "dd5f7acb-d880-4f98-8b64-8501641747b6" (UID: "dd5f7acb-d880-4f98-8b64-8501641747b6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:43:34.633311 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:34.633208 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dd5f7acb-d880-4f98-8b64-8501641747b6-istio-podinfo" pod "dd5f7acb-d880-4f98-8b64-8501641747b6" (UID: "dd5f7acb-d880-4f98-8b64-8501641747b6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:43:34.633311 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:34.633292 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd5f7acb-d880-4f98-8b64-8501641747b6-kube-api-access-dj7rf" pod "dd5f7acb-d880-4f98-8b64-8501641747b6" (UID: "dd5f7acb-d880-4f98-8b64-8501641747b6"). InnerVolumeSpecName "kube-api-access-dj7rf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:43:34.732313 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:34.732250 4421 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd5f7acb-d880-4f98-8b64-8501641747b6-istio-data\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:43:34.732313 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:34.732307 4421 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd5f7acb-d880-4f98-8b64-8501641747b6-istio-podinfo\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:43:34.732313 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:34.732319 4421 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd5f7acb-d880-4f98-8b64-8501641747b6-istiod-ca-cert\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:43:34.732566 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:34.732328 4421 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd5f7acb-d880-4f98-8b64-8501641747b6-credential-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:43:34.732566 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:34.732336 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dj7rf\" (UniqueName: \"kubernetes.io/projected/dd5f7acb-d880-4f98-8b64-8501641747b6-kube-api-access-dj7rf\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:43:34.732566 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:34.732348 4421 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd5f7acb-d880-4f98-8b64-8501641747b6-istio-envoy\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:43:34.732566 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:34.732356 4421 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd5f7acb-d880-4f98-8b64-8501641747b6-istio-token\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:43:35.371485 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:35.371445 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-h9mjc" event={"ID":"dd5f7acb-d880-4f98-8b64-8501641747b6","Type":"ContainerDied","Data":"7863271c03fb1a86f972bb25580e780b8dca28d98cb0e4436ffaa60cf1dbbdce"} Nov 03 01:43:35.371485 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:35.371490 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-h9mjc" Nov 03 01:43:35.371982 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:35.371525 4421 scope.go:122] "RemoveContainer" containerID="72ec5e73ad4ad6c8646fb93943166bf654cf890599ac54ec59e7d4e072ccfd7c" Nov 03 01:43:35.394802 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:35.394751 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-h9mjc"] Nov 03 01:43:35.398465 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:35.398436 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-h9mjc"] Nov 03 01:43:35.977661 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:35.977627 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd5f7acb-d880-4f98-8b64-8501641747b6" path="/var/lib/kubelet/pods/dd5f7acb-d880-4f98-8b64-8501641747b6/volumes" Nov 03 01:43:44.669418 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:43:44.669378 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 01:43:52.021649 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:52.021613 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1844b5d-lscvc"] Nov 03 01:43:52.023990 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:52.023968 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1844b5d-lscvc"] Nov 03 01:43:53.977739 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:53.977706 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="065a3186-f482-4aa4-be5d-501fc27a6356" path="/var/lib/kubelet/pods/065a3186-f482-4aa4-be5d-501fc27a6356/volumes" Nov 03 01:43:57.750570 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:57.750531 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a932d7e1-zm74g"] Nov 03 01:43:57.750978 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:57.750913 4421 cpu_manager.go:414] "containerMap: removing container" podUID="61568421-54f2-4ba7-973f-8efa5ab1e805" containerName="payload-processing" Nov 03 01:43:57.750978 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:57.750923 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61568421-54f2-4ba7-973f-8efa5ab1e805" containerName="payload-processing" Nov 03 01:43:57.750978 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:57.750939 4421 cpu_manager.go:414] "containerMap: removing container" podUID="c0c6b02e-2ecb-428e-a13d-be6d7cd1ba39" containerName="main" Nov 03 01:43:57.750978 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:57.750944 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0c6b02e-2ecb-428e-a13d-be6d7cd1ba39" containerName="main" Nov 03 01:43:57.750978 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:57.750950 4421 cpu_manager.go:414] "containerMap: removing container" podUID="f7b858b0-7dd7-42f7-bb5f-22920b707d4c" containerName="payload-pre-processing" Nov 03 01:43:57.750978 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:57.750955 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7b858b0-7dd7-42f7-bb5f-22920b707d4c" containerName="payload-pre-processing" Nov 03 01:43:57.750978 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:57.750965 4421 cpu_manager.go:414] "containerMap: removing container" podUID="4f2cede9-84b3-4d23-8456-38016e4ff7fb" containerName="payload-pre-processing" Nov 03 01:43:57.750978 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:57.750969 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f2cede9-84b3-4d23-8456-38016e4ff7fb" containerName="payload-pre-processing" Nov 03 01:43:57.750978 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:57.750976 4421 cpu_manager.go:414] "containerMap: removing container" podUID="35687f7e-87be-465b-ab66-ec518b459edc" containerName="istio-proxy" Nov 03 01:43:57.750978 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:57.750980 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35687f7e-87be-465b-ab66-ec518b459edc" containerName="istio-proxy" Nov 03 01:43:57.751356 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:57.750988 4421 cpu_manager.go:414] "containerMap: removing container" podUID="f4ae9624-d127-4a55-837d-7c9ff0267b37" containerName="main" Nov 03 01:43:57.751356 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:57.750992 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4ae9624-d127-4a55-837d-7c9ff0267b37" containerName="main" Nov 03 01:43:57.751356 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:57.751001 4421 cpu_manager.go:414] "containerMap: removing container" podUID="b0a4c234-af58-4da0-a1a4-e1f167953f6a" containerName="main" Nov 03 01:43:57.751356 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:57.751005 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0a4c234-af58-4da0-a1a4-e1f167953f6a" containerName="main" Nov 03 01:43:57.751356 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:57.751011 4421 cpu_manager.go:414] "containerMap: removing container" podUID="dd5f7acb-d880-4f98-8b64-8501641747b6" containerName="istio-proxy" Nov 03 01:43:57.751356 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:57.751015 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd5f7acb-d880-4f98-8b64-8501641747b6" containerName="istio-proxy" Nov 03 01:43:57.751356 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:57.751022 4421 cpu_manager.go:414] "containerMap: removing container" podUID="542b6309-1f43-413e-a448-b604a68a5047" containerName="payload-pre-processing" Nov 03 01:43:57.751356 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:57.751026 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="542b6309-1f43-413e-a448-b604a68a5047" containerName="payload-pre-processing" Nov 03 01:43:57.751356 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:57.751045 4421 cpu_manager.go:414] "containerMap: removing container" podUID="6ff46e7d-ec61-4e17-ae76-0dc25edd95b4" containerName="payload-pre-processing" Nov 03 01:43:57.751356 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:57.751050 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ff46e7d-ec61-4e17-ae76-0dc25edd95b4" containerName="payload-pre-processing" Nov 03 01:43:57.751356 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:57.751058 4421 cpu_manager.go:414] "containerMap: removing container" podUID="dc7f30da-8b1e-42d7-a41a-2398e66263b7" containerName="istio-proxy" Nov 03 01:43:57.751356 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:57.751062 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc7f30da-8b1e-42d7-a41a-2398e66263b7" containerName="istio-proxy" Nov 03 01:43:57.751356 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:57.751067 4421 cpu_manager.go:414] "containerMap: removing container" podUID="157a7bd1-4ae4-4460-8ddd-15c02cae279d" containerName="revoke-keys" Nov 03 01:43:57.751356 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:57.751071 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="157a7bd1-4ae4-4460-8ddd-15c02cae279d" containerName="revoke-keys" Nov 03 01:43:57.751356 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:57.751079 4421 cpu_manager.go:414] "containerMap: removing container" podUID="a454ee55-ade7-4715-bf20-8cc451458d99" containerName="payload-processing" Nov 03 01:43:57.751356 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:57.751083 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a454ee55-ade7-4715-bf20-8cc451458d99" containerName="payload-processing" Nov 03 01:43:57.751356 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:57.751090 4421 cpu_manager.go:414] "containerMap: removing container" podUID="af3bcba4-1005-480a-859f-8fc73f0cbe53" containerName="payload-processing" Nov 03 01:43:57.751356 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:57.751094 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af3bcba4-1005-480a-859f-8fc73f0cbe53" containerName="payload-processing" Nov 03 01:43:57.751356 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:57.751101 4421 cpu_manager.go:414] "containerMap: removing container" podUID="dcf063ce-88a5-4341-ae0f-605522bb91d5" containerName="istio-proxy" Nov 03 01:43:57.751356 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:57.751105 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dcf063ce-88a5-4341-ae0f-605522bb91d5" containerName="istio-proxy" Nov 03 01:43:57.751356 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:57.751117 4421 cpu_manager.go:414] "containerMap: removing container" podUID="2fb9b19f-9286-4d24-813c-be4055c5e8e2" containerName="payload-processing" Nov 03 01:43:57.751356 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:57.751122 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fb9b19f-9286-4d24-813c-be4055c5e8e2" containerName="payload-processing" Nov 03 01:43:57.751356 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:57.751173 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="a454ee55-ade7-4715-bf20-8cc451458d99" containerName="payload-processing" Nov 03 01:43:57.751356 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:57.751183 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="af3bcba4-1005-480a-859f-8fc73f0cbe53" containerName="payload-processing" Nov 03 01:43:57.751356 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:57.751191 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="dcf063ce-88a5-4341-ae0f-605522bb91d5" containerName="istio-proxy" Nov 03 01:43:57.751356 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:57.751198 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fb9b19f-9286-4d24-813c-be4055c5e8e2" containerName="payload-processing" Nov 03 01:43:57.751356 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:57.751203 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="61568421-54f2-4ba7-973f-8efa5ab1e805" containerName="payload-processing" Nov 03 01:43:57.751356 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:57.751213 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0c6b02e-2ecb-428e-a13d-be6d7cd1ba39" containerName="main" Nov 03 01:43:57.751356 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:57.751218 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7b858b0-7dd7-42f7-bb5f-22920b707d4c" containerName="payload-pre-processing" Nov 03 01:43:57.751356 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:57.751223 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f2cede9-84b3-4d23-8456-38016e4ff7fb" containerName="payload-pre-processing" Nov 03 01:43:57.751356 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:57.751229 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="35687f7e-87be-465b-ab66-ec518b459edc" containerName="istio-proxy" Nov 03 01:43:57.751356 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:57.751234 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4ae9624-d127-4a55-837d-7c9ff0267b37" containerName="main" Nov 03 01:43:57.751356 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:57.751239 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0a4c234-af58-4da0-a1a4-e1f167953f6a" containerName="main" Nov 03 01:43:57.751356 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:57.751244 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd5f7acb-d880-4f98-8b64-8501641747b6" containerName="istio-proxy" Nov 03 01:43:57.751356 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:57.751250 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="542b6309-1f43-413e-a448-b604a68a5047" containerName="payload-pre-processing" Nov 03 01:43:57.751356 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:57.751258 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ff46e7d-ec61-4e17-ae76-0dc25edd95b4" containerName="payload-pre-processing" Nov 03 01:43:57.751356 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:57.751264 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc7f30da-8b1e-42d7-a41a-2398e66263b7" containerName="istio-proxy" Nov 03 01:43:57.751356 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:57.751269 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="157a7bd1-4ae4-4460-8ddd-15c02cae279d" containerName="revoke-keys" Nov 03 01:43:57.755750 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:57.755725 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a932d7e1-zm74g" Nov 03 01:43:57.760745 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:57.760714 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 01:43:57.760961 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:57.760866 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6q4xp\"" Nov 03 01:43:57.760961 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:57.760930 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 01:43:57.760961 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:57.760950 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:43:57.762044 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:57.762013 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a932d7e1-zm74g"] Nov 03 01:43:57.853361 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:57.853328 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2afa68b-b6b3-4599-b29c-89a62b4a267b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-a932d7e1-zm74g\" (UID: \"f2afa68b-b6b3-4599-b29c-89a62b4a267b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a932d7e1-zm74g" Nov 03 01:43:57.853554 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:57.853374 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bhtl\" (UniqueName: \"kubernetes.io/projected/f2afa68b-b6b3-4599-b29c-89a62b4a267b-kube-api-access-7bhtl\") pod \"maas-api-revoke-keys-e2e-ait-a932d7e1-zm74g\" (UID: \"f2afa68b-b6b3-4599-b29c-89a62b4a267b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a932d7e1-zm74g" Nov 03 01:43:57.853554 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:57.853446 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2afa68b-b6b3-4599-b29c-89a62b4a267b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-a932d7e1-zm74g\" (UID: \"f2afa68b-b6b3-4599-b29c-89a62b4a267b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a932d7e1-zm74g" Nov 03 01:43:57.954080 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:57.954047 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2afa68b-b6b3-4599-b29c-89a62b4a267b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-a932d7e1-zm74g\" (UID: \"f2afa68b-b6b3-4599-b29c-89a62b4a267b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a932d7e1-zm74g" Nov 03 01:43:57.954296 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:57.954090 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bhtl\" (UniqueName: \"kubernetes.io/projected/f2afa68b-b6b3-4599-b29c-89a62b4a267b-kube-api-access-7bhtl\") pod \"maas-api-revoke-keys-e2e-ait-a932d7e1-zm74g\" (UID: \"f2afa68b-b6b3-4599-b29c-89a62b4a267b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a932d7e1-zm74g" Nov 03 01:43:57.954296 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:57.954223 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2afa68b-b6b3-4599-b29c-89a62b4a267b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-a932d7e1-zm74g\" (UID: \"f2afa68b-b6b3-4599-b29c-89a62b4a267b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a932d7e1-zm74g" Nov 03 01:43:57.954791 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:57.954770 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2afa68b-b6b3-4599-b29c-89a62b4a267b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-a932d7e1-zm74g\" (UID: \"f2afa68b-b6b3-4599-b29c-89a62b4a267b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a932d7e1-zm74g" Nov 03 01:43:57.956620 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:57.956599 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2afa68b-b6b3-4599-b29c-89a62b4a267b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-a932d7e1-zm74g\" (UID: \"f2afa68b-b6b3-4599-b29c-89a62b4a267b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a932d7e1-zm74g" Nov 03 01:43:57.962285 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:57.962248 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bhtl\" (UniqueName: \"kubernetes.io/projected/f2afa68b-b6b3-4599-b29c-89a62b4a267b-kube-api-access-7bhtl\") pod \"maas-api-revoke-keys-e2e-ait-a932d7e1-zm74g\" (UID: \"f2afa68b-b6b3-4599-b29c-89a62b4a267b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a932d7e1-zm74g" Nov 03 01:43:58.074353 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:58.074240 4421 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a932d7e1-zm74g" secret="" err="secret \"maas-api-cleanup-dockercfg-6q4xp\" not found" Nov 03 01:43:58.074353 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:58.074350 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a932d7e1-zm74g" Nov 03 01:43:58.351651 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:58.351623 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a932d7e1-zm74g"] Nov 03 01:43:58.354623 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:43:58.354594 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2afa68b_b6b3_4599_b29c_89a62b4a267b.slice/crio-a0733e1fd029134256a1a4b06ae13ab70ce3f1a0d13310d3ab36239ff76fedf1 WatchSource:0}: Error finding container a0733e1fd029134256a1a4b06ae13ab70ce3f1a0d13310d3ab36239ff76fedf1: Status 404 returned error can't find the container with id a0733e1fd029134256a1a4b06ae13ab70ce3f1a0d13310d3ab36239ff76fedf1 Nov 03 01:43:58.513684 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:58.513649 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a932d7e1-zm74g" event={"ID":"f2afa68b-b6b3-4599-b29c-89a62b4a267b","Type":"ContainerStarted","Data":"a0733e1fd029134256a1a4b06ae13ab70ce3f1a0d13310d3ab36239ff76fedf1"} Nov 03 01:43:59.521691 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:59.521658 4421 generic.go:356] "Generic (PLEG): container finished" podID="f2afa68b-b6b3-4599-b29c-89a62b4a267b" containerID="02775679b7026bdbe1cd47d9fb5106c6c5233da4ed159ab3215b092a841a762c" exitCode=0 Nov 03 01:43:59.522121 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:43:59.521711 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a932d7e1-zm74g" event={"ID":"f2afa68b-b6b3-4599-b29c-89a62b4a267b","Type":"ContainerDied","Data":"02775679b7026bdbe1cd47d9fb5106c6c5233da4ed159ab3215b092a841a762c"} Nov 03 01:44:00.665180 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:44:00.665151 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a932d7e1-zm74g" Nov 03 01:44:00.783699 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:44:00.783614 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2afa68b-b6b3-4599-b29c-89a62b4a267b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2afa68b-b6b3-4599-b29c-89a62b4a267b-maas-api-openshift-ca\") pod \"f2afa68b-b6b3-4599-b29c-89a62b4a267b\" (UID: \"f2afa68b-b6b3-4599-b29c-89a62b4a267b\") " Nov 03 01:44:00.783699 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:44:00.783698 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2afa68b-b6b3-4599-b29c-89a62b4a267b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2afa68b-b6b3-4599-b29c-89a62b4a267b-maas-api-xks-ca\") pod \"f2afa68b-b6b3-4599-b29c-89a62b4a267b\" (UID: \"f2afa68b-b6b3-4599-b29c-89a62b4a267b\") " Nov 03 01:44:00.783950 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:44:00.783868 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2afa68b-b6b3-4599-b29c-89a62b4a267b-kube-api-access-7bhtl\" (UniqueName: \"kubernetes.io/projected/f2afa68b-b6b3-4599-b29c-89a62b4a267b-kube-api-access-7bhtl\") pod \"f2afa68b-b6b3-4599-b29c-89a62b4a267b\" (UID: \"f2afa68b-b6b3-4599-b29c-89a62b4a267b\") " Nov 03 01:44:00.784020 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:44:00.783985 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2afa68b-b6b3-4599-b29c-89a62b4a267b-maas-api-openshift-ca" pod "f2afa68b-b6b3-4599-b29c-89a62b4a267b" (UID: "f2afa68b-b6b3-4599-b29c-89a62b4a267b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:44:00.784265 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:44:00.784246 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2afa68b-b6b3-4599-b29c-89a62b4a267b-maas-api-openshift-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:44:00.785836 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:44:00.785811 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2afa68b-b6b3-4599-b29c-89a62b4a267b-kube-api-access-7bhtl" pod "f2afa68b-b6b3-4599-b29c-89a62b4a267b" (UID: "f2afa68b-b6b3-4599-b29c-89a62b4a267b"). InnerVolumeSpecName "kube-api-access-7bhtl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:44:00.785904 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:44:00.785889 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2afa68b-b6b3-4599-b29c-89a62b4a267b-maas-api-xks-ca" pod "f2afa68b-b6b3-4599-b29c-89a62b4a267b" (UID: "f2afa68b-b6b3-4599-b29c-89a62b4a267b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:44:00.885415 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:44:00.885364 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2afa68b-b6b3-4599-b29c-89a62b4a267b-maas-api-xks-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:44:00.885415 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:44:00.885402 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7bhtl\" (UniqueName: \"kubernetes.io/projected/f2afa68b-b6b3-4599-b29c-89a62b4a267b-kube-api-access-7bhtl\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:44:01.534531 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:44:01.534500 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a932d7e1-zm74g" Nov 03 01:44:01.534763 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:44:01.534538 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a932d7e1-zm74g" event={"ID":"f2afa68b-b6b3-4599-b29c-89a62b4a267b","Type":"ContainerDied","Data":"a0733e1fd029134256a1a4b06ae13ab70ce3f1a0d13310d3ab36239ff76fedf1"} Nov 03 01:44:01.534763 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:44:01.534593 4421 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a0733e1fd029134256a1a4b06ae13ab70ce3f1a0d13310d3ab36239ff76fedf1" Nov 03 01:44:03.021148 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:44:03.021107 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6fabaa5-pn92q"] Nov 03 01:44:03.023740 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:44:03.023707 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6fabaa5-pn92q"] Nov 03 01:44:03.977693 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:44:03.977657 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce04511a-b37f-4f31-abda-2ff55f0afbd1" path="/var/lib/kubelet/pods/ce04511a-b37f-4f31-abda-2ff55f0afbd1/volumes" Nov 03 01:44:06.696367 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:44:06.696336 4421 scope.go:122] "RemoveContainer" containerID="88c028d3c4647c20692797a7273d01a7ea3fc7ad08b3678980326c23faca4651" Nov 03 01:44:06.705779 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:44:06.705755 4421 scope.go:122] "RemoveContainer" containerID="d665d84272233cca08640f0873e58eba761cb71da4ed66269713a193e85bc5d2" Nov 03 01:44:09.066522 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:44:09.066482 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a932d7e1-zm74g"] Nov 03 01:44:09.068961 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:44:09.068926 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a932d7e1-zm74g"] Nov 03 01:44:09.976692 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:44:09.976657 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2afa68b-b6b3-4599-b29c-89a62b4a267b" path="/var/lib/kubelet/pods/f2afa68b-b6b3-4599-b29c-89a62b4a267b/volumes" Nov 03 01:44:13.028339 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:44:13.028300 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e18d8a30-h8njr"] Nov 03 01:44:13.030625 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:44:13.030603 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e18d8a30-h8njr"] Nov 03 01:44:13.978386 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:44:13.978348 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff0057ad-0f4f-44df-9979-a5d97343c69a" path="/var/lib/kubelet/pods/ff0057ad-0f4f-44df-9979-a5d97343c69a/volumes" Nov 03 01:44:14.668376 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:44:14.668332 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 01:44:24.023541 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:44:24.023495 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d8d22e3-th89b"] Nov 03 01:44:24.025749 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:44:24.025723 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d8d22e3-th89b"] Nov 03 01:44:25.977780 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:44:25.977743 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e612246-ef87-4ccb-b32e-ef7dd768afd2" path="/var/lib/kubelet/pods/6e612246-ef87-4ccb-b32e-ef7dd768afd2/volumes" Nov 03 01:44:44.669106 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:44:44.669016 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 01:44:58.020176 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:44:58.020136 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c49b28f-6t7dj"] Nov 03 01:44:58.025487 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:44:58.025453 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c49b28f-6t7dj"] Nov 03 01:44:59.977709 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:44:59.977679 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ced0135-f0a1-4a1a-ae6a-2d5442bdf66a" path="/var/lib/kubelet/pods/7ced0135-f0a1-4a1a-ae6a-2d5442bdf66a/volumes" Nov 03 01:45:06.748626 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:45:06.748588 4421 scope.go:122] "RemoveContainer" containerID="19231f3fa2b704a6ce1be1ca774389dd1fa688f5b32f5525fd6404539b483843" Nov 03 01:45:06.758416 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:45:06.758388 4421 scope.go:122] "RemoveContainer" containerID="f99556a1cd025c25bda9ea18ae669a59fea5a1d716bb9604fed80ba0ad3402e4" Nov 03 01:45:08.027571 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:45:08.027538 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-afa07afc-qcn45"] Nov 03 01:45:08.029785 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:45:08.029759 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-afa07afc-qcn45"] Nov 03 01:45:09.977181 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:45:09.977141 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33e4bab9-aa4d-4a30-b787-8986f8460e4a" path="/var/lib/kubelet/pods/33e4bab9-aa4d-4a30-b787-8986f8460e4a/volumes" Nov 03 01:45:14.674929 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:45:14.674879 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 01:45:19.020876 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:45:19.020819 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f20e1ccc-ds5fz"] Nov 03 01:45:19.023194 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:45:19.023162 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f20e1ccc-ds5fz"] Nov 03 01:45:19.977129 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:45:19.977090 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ffa952fe-2ab6-4f56-9afd-a9a9008b2f5c" path="/var/lib/kubelet/pods/ffa952fe-2ab6-4f56-9afd-a9a9008b2f5c/volumes" Nov 03 01:45:25.023088 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:45:25.023036 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2c966aa-vjmw5"] Nov 03 01:45:25.025470 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:45:25.025436 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2c966aa-vjmw5"] Nov 03 01:45:25.977926 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:45:25.977887 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e0718b6-a990-4b70-a0a4-873956e942b0" path="/var/lib/kubelet/pods/6e0718b6-a990-4b70-a0a4-873956e942b0/volumes" Nov 03 01:45:30.021084 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:45:30.021041 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ff331d9-tdxww"] Nov 03 01:45:30.025449 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:45:30.025424 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ff331d9-tdxww"] Nov 03 01:45:31.987195 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:45:31.987155 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd3fd1e0-4557-429f-8ba6-da6007ffe32d" path="/var/lib/kubelet/pods/dd3fd1e0-4557-429f-8ba6-da6007ffe32d/volumes" Nov 03 01:45:35.022562 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:45:35.022493 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e23907d5-fdfjd"] Nov 03 01:45:35.024776 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:45:35.024743 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e23907d5-fdfjd"] Nov 03 01:45:35.977251 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:45:35.977215 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2b6d01d-9fcb-4164-a217-2a596801ce5b" path="/var/lib/kubelet/pods/e2b6d01d-9fcb-4164-a217-2a596801ce5b/volumes" Nov 03 01:45:38.935299 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:45:38.934407 4421 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-s4dc8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 01:45:38.935299 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:45:38.934517 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-s4dc8" podUID="ab5414b5-9d47-4c9f-9d0e-82242b9d6db2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 01:45:44.669791 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:45:44.669745 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 01:46:04.022085 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:46:04.021874 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d925fdb-fbmp5"] Nov 03 01:46:04.024790 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:46:04.024756 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d925fdb-fbmp5"] Nov 03 01:46:05.977548 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:46:05.977515 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6dbc64a9-42b0-407c-b1d5-f2651acc523b" path="/var/lib/kubelet/pods/6dbc64a9-42b0-407c-b1d5-f2651acc523b/volumes" Nov 03 01:46:06.783975 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:46:06.783944 4421 scope.go:122] "RemoveContainer" containerID="9b8a8f6590bfacfa17cc3eef68a085accbcba0b83bef0cc1855deb359a2e35d6" Nov 03 01:46:06.793236 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:46:06.793215 4421 scope.go:122] "RemoveContainer" containerID="123178896f2b5127cb3e40cab0a939ee81c5f48b0f4d28240f2b8f9064043f29" Nov 03 01:46:06.802396 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:46:06.802368 4421 scope.go:122] "RemoveContainer" containerID="a4696d8a541ee6115527fd8f074035194e4e918484f994b5da8cad4cb42def4f" Nov 03 01:46:06.813596 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:46:06.813573 4421 scope.go:122] "RemoveContainer" containerID="8f3e7040a2ffc7943e3e31b383e038780a2295232a8eaee7ab3c2abfbfe81b60" Nov 03 01:46:06.823462 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:46:06.823440 4421 scope.go:122] "RemoveContainer" containerID="fec0554a23ecf631530529b990ceb185a106d120263b4d3a6253e441fd2359db" Nov 03 01:46:14.668240 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:46:14.668157 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 01:46:16.020963 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:46:16.020928 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f3bbab0-lnccm"] Nov 03 01:46:16.023513 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:46:16.023484 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f3bbab0-lnccm"] Nov 03 01:46:17.977049 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:46:17.977019 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a18e15c-77bb-4660-ab81-e731b3a5938f" path="/var/lib/kubelet/pods/4a18e15c-77bb-4660-ab81-e731b3a5938f/volumes" Nov 03 01:46:23.930453 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:46:23.930405 4421 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-s4dc8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 01:46:23.931213 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:46:23.931161 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-s4dc8" podUID="ab5414b5-9d47-4c9f-9d0e-82242b9d6db2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 01:46:25.020286 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:46:25.020247 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b98b0ec-x74kl"] Nov 03 01:46:25.022018 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:46:25.021995 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b98b0ec-x74kl"] Nov 03 01:46:25.977861 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:46:25.977821 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="126cb5d9-27c6-4f14-8612-45c82f71946c" path="/var/lib/kubelet/pods/126cb5d9-27c6-4f14-8612-45c82f71946c/volumes" Nov 03 01:46:34.019822 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:46:34.019783 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5c3d8bc-hsgcz"] Nov 03 01:46:34.022053 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:46:34.022027 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5c3d8bc-hsgcz"] Nov 03 01:46:35.977091 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:46:35.977051 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d08194d0-b707-48e0-8a62-6c04c6cdc1e7" path="/var/lib/kubelet/pods/d08194d0-b707-48e0-8a62-6c04c6cdc1e7/volumes" Nov 03 01:46:44.668683 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:46:44.668630 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 01:46:49.020387 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:46:49.020347 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57bcd798-bkqcf"] Nov 03 01:46:49.024083 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:46:49.024049 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57bcd798-bkqcf"] Nov 03 01:46:49.977427 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:46:49.977393 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e88c376d-a5b0-4d87-b5d8-cb14fa46b8db" path="/var/lib/kubelet/pods/e88c376d-a5b0-4d87-b5d8-cb14fa46b8db/volumes" Nov 03 01:46:54.021343 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:46:54.021301 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-66775a6a-6kwtx"] Nov 03 01:46:54.023867 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:46:54.023835 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-66775a6a-6kwtx"] Nov 03 01:46:55.978595 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:46:55.978555 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24a9d347-bb35-4a9c-b489-726098a59823" path="/var/lib/kubelet/pods/24a9d347-bb35-4a9c-b489-726098a59823/volumes" Nov 03 01:47:06.020891 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:06.020821 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-200ed79c-fcghv"] Nov 03 01:47:06.023453 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:06.023428 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-200ed79c-fcghv"] Nov 03 01:47:06.868213 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:06.868179 4421 scope.go:122] "RemoveContainer" containerID="f83b31037917694bbc17199dff8d6feaf83050d17244445c4ed549e1106a5c60" Nov 03 01:47:06.877149 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:06.877127 4421 scope.go:122] "RemoveContainer" containerID="5634b34e211f4611085fbbb1e56530b401cb4df39db4c80d7d6025f140fdf7bd" Nov 03 01:47:06.885345 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:06.885325 4421 scope.go:122] "RemoveContainer" containerID="5dd50e8fb524cd50b6d72695142bca9d8de5afcdadf6381351ca5d7bb0f29323" Nov 03 01:47:06.893532 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:06.893512 4421 scope.go:122] "RemoveContainer" containerID="c754d9a944e5d2010843a3ee1e546e66b2cf643cb59c62ee8fdc7617c6381932" Nov 03 01:47:06.900964 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:06.900938 4421 scope.go:122] "RemoveContainer" containerID="a64c3be1661b9cfa805d3ff11606b1d435a52e9a367b980020193381259908d5" Nov 03 01:47:07.977897 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:07.977858 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c753d73-2ccb-410f-901d-8d19b175c510" path="/var/lib/kubelet/pods/9c753d73-2ccb-410f-901d-8d19b175c510/volumes" Nov 03 01:47:09.943203 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:09.943159 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fabdcaee-grtwc"] Nov 03 01:47:09.943860 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:09.943835 4421 cpu_manager.go:414] "containerMap: removing container" podUID="f2afa68b-b6b3-4599-b29c-89a62b4a267b" containerName="revoke-keys" Nov 03 01:47:09.943922 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:09.943866 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2afa68b-b6b3-4599-b29c-89a62b4a267b" containerName="revoke-keys" Nov 03 01:47:09.943972 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:09.943959 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2afa68b-b6b3-4599-b29c-89a62b4a267b" containerName="revoke-keys" Nov 03 01:47:09.948904 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:09.948882 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fabdcaee-grtwc" Nov 03 01:47:09.953731 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:09.953668 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:47:09.953883 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:09.953732 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 01:47:09.953883 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:09.953787 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-swfcr\"" Nov 03 01:47:09.953883 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:09.953808 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 01:47:09.955737 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:09.955709 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fabdcaee-grtwc"] Nov 03 01:47:10.026971 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:10.026930 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d5a9312-s5zth"] Nov 03 01:47:10.030762 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:10.030735 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d5a9312-s5zth"] Nov 03 01:47:10.085641 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:10.085604 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a94bb737-6938-44c2-8949-794c4702a4cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fabdcaee-grtwc\" (UID: \"a94bb737-6938-44c2-8949-794c4702a4cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fabdcaee-grtwc" Nov 03 01:47:10.085840 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:10.085679 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a94bb737-6938-44c2-8949-794c4702a4cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fabdcaee-grtwc\" (UID: \"a94bb737-6938-44c2-8949-794c4702a4cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fabdcaee-grtwc" Nov 03 01:47:10.085840 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:10.085786 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knjjs\" (UniqueName: \"kubernetes.io/projected/a94bb737-6938-44c2-8949-794c4702a4cf-kube-api-access-knjjs\") pod \"maas-api-revoke-sub-fabdcaee-grtwc\" (UID: \"a94bb737-6938-44c2-8949-794c4702a4cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fabdcaee-grtwc" Nov 03 01:47:10.186343 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:10.186301 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a94bb737-6938-44c2-8949-794c4702a4cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fabdcaee-grtwc\" (UID: \"a94bb737-6938-44c2-8949-794c4702a4cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fabdcaee-grtwc" Nov 03 01:47:10.186595 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:10.186385 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a94bb737-6938-44c2-8949-794c4702a4cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fabdcaee-grtwc\" (UID: \"a94bb737-6938-44c2-8949-794c4702a4cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fabdcaee-grtwc" Nov 03 01:47:10.186595 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:10.186421 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knjjs\" (UniqueName: \"kubernetes.io/projected/a94bb737-6938-44c2-8949-794c4702a4cf-kube-api-access-knjjs\") pod \"maas-api-revoke-sub-fabdcaee-grtwc\" (UID: \"a94bb737-6938-44c2-8949-794c4702a4cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fabdcaee-grtwc" Nov 03 01:47:10.187118 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:10.187095 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a94bb737-6938-44c2-8949-794c4702a4cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fabdcaee-grtwc\" (UID: \"a94bb737-6938-44c2-8949-794c4702a4cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fabdcaee-grtwc" Nov 03 01:47:10.188674 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:10.188647 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a94bb737-6938-44c2-8949-794c4702a4cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fabdcaee-grtwc\" (UID: \"a94bb737-6938-44c2-8949-794c4702a4cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fabdcaee-grtwc" Nov 03 01:47:10.194584 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:10.194525 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-knjjs\" (UniqueName: \"kubernetes.io/projected/a94bb737-6938-44c2-8949-794c4702a4cf-kube-api-access-knjjs\") pod \"maas-api-revoke-sub-fabdcaee-grtwc\" (UID: \"a94bb737-6938-44c2-8949-794c4702a4cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fabdcaee-grtwc" Nov 03 01:47:10.263834 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:10.263789 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fabdcaee-grtwc" Nov 03 01:47:10.537897 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:10.537832 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fabdcaee-grtwc"] Nov 03 01:47:10.540876 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:47:10.540846 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda94bb737_6938_44c2_8949_794c4702a4cf.slice/crio-9d7a0a447b39b4f1bf2421898ae1446f0a7cf88e85b829c532df0a00b09ac523 WatchSource:0}: Error finding container 9d7a0a447b39b4f1bf2421898ae1446f0a7cf88e85b829c532df0a00b09ac523: Status 404 returned error can't find the container with id 9d7a0a447b39b4f1bf2421898ae1446f0a7cf88e85b829c532df0a00b09ac523 Nov 03 01:47:10.721916 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:10.721885 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fabdcaee-grtwc" event={"ID":"a94bb737-6938-44c2-8949-794c4702a4cf","Type":"ContainerStarted","Data":"9d7a0a447b39b4f1bf2421898ae1446f0a7cf88e85b829c532df0a00b09ac523"} Nov 03 01:47:11.728783 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:11.728687 4421 generic.go:356] "Generic (PLEG): container finished" podID="a94bb737-6938-44c2-8949-794c4702a4cf" containerID="b27e94badf4be2dada2012713ae3ba095938cbc4f420e9160cd630235bcc2458" exitCode=0 Nov 03 01:47:11.728783 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:11.728763 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fabdcaee-grtwc" event={"ID":"a94bb737-6938-44c2-8949-794c4702a4cf","Type":"ContainerDied","Data":"b27e94badf4be2dada2012713ae3ba095938cbc4f420e9160cd630235bcc2458"} Nov 03 01:47:11.977204 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:11.977173 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="157a7bd1-4ae4-4460-8ddd-15c02cae279d" path="/var/lib/kubelet/pods/157a7bd1-4ae4-4460-8ddd-15c02cae279d/volumes" Nov 03 01:47:12.879106 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:12.879075 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fabdcaee-grtwc" Nov 03 01:47:13.013355 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:13.013310 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a94bb737-6938-44c2-8949-794c4702a4cf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a94bb737-6938-44c2-8949-794c4702a4cf-maas-api-openshift-ca\") pod \"a94bb737-6938-44c2-8949-794c4702a4cf\" (UID: \"a94bb737-6938-44c2-8949-794c4702a4cf\") " Nov 03 01:47:13.013567 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:13.013475 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a94bb737-6938-44c2-8949-794c4702a4cf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a94bb737-6938-44c2-8949-794c4702a4cf-maas-api-xks-ca\") pod \"a94bb737-6938-44c2-8949-794c4702a4cf\" (UID: \"a94bb737-6938-44c2-8949-794c4702a4cf\") " Nov 03 01:47:13.013567 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:13.013546 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a94bb737-6938-44c2-8949-794c4702a4cf-kube-api-access-knjjs\" (UniqueName: \"kubernetes.io/projected/a94bb737-6938-44c2-8949-794c4702a4cf-kube-api-access-knjjs\") pod \"a94bb737-6938-44c2-8949-794c4702a4cf\" (UID: \"a94bb737-6938-44c2-8949-794c4702a4cf\") " Nov 03 01:47:13.013768 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:13.013741 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a94bb737-6938-44c2-8949-794c4702a4cf-maas-api-openshift-ca" pod "a94bb737-6938-44c2-8949-794c4702a4cf" (UID: "a94bb737-6938-44c2-8949-794c4702a4cf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:47:13.013905 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:13.013884 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a94bb737-6938-44c2-8949-794c4702a4cf-maas-api-openshift-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:47:13.015630 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:13.015608 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a94bb737-6938-44c2-8949-794c4702a4cf-maas-api-xks-ca" pod "a94bb737-6938-44c2-8949-794c4702a4cf" (UID: "a94bb737-6938-44c2-8949-794c4702a4cf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:47:13.015775 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:13.015756 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a94bb737-6938-44c2-8949-794c4702a4cf-kube-api-access-knjjs" pod "a94bb737-6938-44c2-8949-794c4702a4cf" (UID: "a94bb737-6938-44c2-8949-794c4702a4cf"). InnerVolumeSpecName "kube-api-access-knjjs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:47:13.115354 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:13.115317 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-knjjs\" (UniqueName: \"kubernetes.io/projected/a94bb737-6938-44c2-8949-794c4702a4cf-kube-api-access-knjjs\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:47:13.115354 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:13.115357 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a94bb737-6938-44c2-8949-794c4702a4cf-maas-api-xks-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:47:13.740612 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:13.740557 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fabdcaee-grtwc" event={"ID":"a94bb737-6938-44c2-8949-794c4702a4cf","Type":"ContainerDied","Data":"9d7a0a447b39b4f1bf2421898ae1446f0a7cf88e85b829c532df0a00b09ac523"} Nov 03 01:47:13.740805 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:13.740638 4421 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9d7a0a447b39b4f1bf2421898ae1446f0a7cf88e85b829c532df0a00b09ac523" Nov 03 01:47:13.740805 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:13.740575 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fabdcaee-grtwc" Nov 03 01:47:14.668793 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:47:14.668755 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 01:47:25.209963 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:25.209922 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2539e5d6-scqdz"] Nov 03 01:47:25.210522 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:25.210494 4421 cpu_manager.go:414] "containerMap: removing container" podUID="a94bb737-6938-44c2-8949-794c4702a4cf" containerName="revoke-keys" Nov 03 01:47:25.210613 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:25.210524 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a94bb737-6938-44c2-8949-794c4702a4cf" containerName="revoke-keys" Nov 03 01:47:25.210690 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:25.210675 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="a94bb737-6938-44c2-8949-794c4702a4cf" containerName="revoke-keys" Nov 03 01:47:25.214787 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:25.214764 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2539e5d6-scqdz" Nov 03 01:47:25.219565 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:25.219487 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 01:47:25.219963 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:25.219931 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:47:25.220103 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:25.219991 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-swfcr\"" Nov 03 01:47:25.220103 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:25.219992 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 01:47:25.220547 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:25.220469 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2539e5d6-scqdz"] Nov 03 01:47:25.383555 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:25.383520 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0143cfca-8bd8-4380-9eda-ff603b205172-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2539e5d6-scqdz\" (UID: \"0143cfca-8bd8-4380-9eda-ff603b205172\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2539e5d6-scqdz" Nov 03 01:47:25.383555 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:25.383557 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s99bw\" (UniqueName: \"kubernetes.io/projected/0143cfca-8bd8-4380-9eda-ff603b205172-kube-api-access-s99bw\") pod \"maas-api-revoke-sub-2539e5d6-scqdz\" (UID: \"0143cfca-8bd8-4380-9eda-ff603b205172\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2539e5d6-scqdz" Nov 03 01:47:25.383808 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:25.383700 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0143cfca-8bd8-4380-9eda-ff603b205172-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2539e5d6-scqdz\" (UID: \"0143cfca-8bd8-4380-9eda-ff603b205172\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2539e5d6-scqdz" Nov 03 01:47:25.484756 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:25.484640 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0143cfca-8bd8-4380-9eda-ff603b205172-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2539e5d6-scqdz\" (UID: \"0143cfca-8bd8-4380-9eda-ff603b205172\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2539e5d6-scqdz" Nov 03 01:47:25.484974 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:25.484787 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0143cfca-8bd8-4380-9eda-ff603b205172-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2539e5d6-scqdz\" (UID: \"0143cfca-8bd8-4380-9eda-ff603b205172\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2539e5d6-scqdz" Nov 03 01:47:25.484974 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:25.484819 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s99bw\" (UniqueName: \"kubernetes.io/projected/0143cfca-8bd8-4380-9eda-ff603b205172-kube-api-access-s99bw\") pod \"maas-api-revoke-sub-2539e5d6-scqdz\" (UID: \"0143cfca-8bd8-4380-9eda-ff603b205172\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2539e5d6-scqdz" Nov 03 01:47:25.485469 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:25.485446 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0143cfca-8bd8-4380-9eda-ff603b205172-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2539e5d6-scqdz\" (UID: \"0143cfca-8bd8-4380-9eda-ff603b205172\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2539e5d6-scqdz" Nov 03 01:47:25.487155 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:25.487135 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0143cfca-8bd8-4380-9eda-ff603b205172-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2539e5d6-scqdz\" (UID: \"0143cfca-8bd8-4380-9eda-ff603b205172\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2539e5d6-scqdz" Nov 03 01:47:25.492918 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:25.492898 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s99bw\" (UniqueName: \"kubernetes.io/projected/0143cfca-8bd8-4380-9eda-ff603b205172-kube-api-access-s99bw\") pod \"maas-api-revoke-sub-2539e5d6-scqdz\" (UID: \"0143cfca-8bd8-4380-9eda-ff603b205172\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2539e5d6-scqdz" Nov 03 01:47:25.530474 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:25.530448 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2539e5d6-scqdz" Nov 03 01:47:25.810322 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:25.810269 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2539e5d6-scqdz"] Nov 03 01:47:25.815384 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:47:25.815338 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0143cfca_8bd8_4380_9eda_ff603b205172.slice/crio-ef51168ec0e86fbe7984748ad2f2457663dbb90a70382657a4ea54d1a3edcc2f WatchSource:0}: Error finding container ef51168ec0e86fbe7984748ad2f2457663dbb90a70382657a4ea54d1a3edcc2f: Status 404 returned error can't find the container with id ef51168ec0e86fbe7984748ad2f2457663dbb90a70382657a4ea54d1a3edcc2f Nov 03 01:47:26.855298 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:26.854671 4421 generic.go:356] "Generic (PLEG): container finished" podID="0143cfca-8bd8-4380-9eda-ff603b205172" containerID="834980a7c4244550679703c544e140f1873d945934b08bb1446ff5a7369c591b" exitCode=0 Nov 03 01:47:26.855298 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:26.854745 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2539e5d6-scqdz" event={"ID":"0143cfca-8bd8-4380-9eda-ff603b205172","Type":"ContainerDied","Data":"834980a7c4244550679703c544e140f1873d945934b08bb1446ff5a7369c591b"} Nov 03 01:47:26.855298 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:26.854823 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2539e5d6-scqdz" event={"ID":"0143cfca-8bd8-4380-9eda-ff603b205172","Type":"ContainerStarted","Data":"ef51168ec0e86fbe7984748ad2f2457663dbb90a70382657a4ea54d1a3edcc2f"} Nov 03 01:47:28.229297 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:28.224654 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2539e5d6-scqdz" Nov 03 01:47:28.371394 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:28.371353 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0143cfca-8bd8-4380-9eda-ff603b205172-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0143cfca-8bd8-4380-9eda-ff603b205172-maas-api-xks-ca\") pod \"0143cfca-8bd8-4380-9eda-ff603b205172\" (UID: \"0143cfca-8bd8-4380-9eda-ff603b205172\") " Nov 03 01:47:28.371620 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:28.371508 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0143cfca-8bd8-4380-9eda-ff603b205172-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0143cfca-8bd8-4380-9eda-ff603b205172-maas-api-openshift-ca\") pod \"0143cfca-8bd8-4380-9eda-ff603b205172\" (UID: \"0143cfca-8bd8-4380-9eda-ff603b205172\") " Nov 03 01:47:28.371620 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:28.371549 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0143cfca-8bd8-4380-9eda-ff603b205172-kube-api-access-s99bw\" (UniqueName: \"kubernetes.io/projected/0143cfca-8bd8-4380-9eda-ff603b205172-kube-api-access-s99bw\") pod \"0143cfca-8bd8-4380-9eda-ff603b205172\" (UID: \"0143cfca-8bd8-4380-9eda-ff603b205172\") " Nov 03 01:47:28.371828 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:28.371794 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0143cfca-8bd8-4380-9eda-ff603b205172-maas-api-openshift-ca" pod "0143cfca-8bd8-4380-9eda-ff603b205172" (UID: "0143cfca-8bd8-4380-9eda-ff603b205172"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:47:28.372001 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:28.371962 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0143cfca-8bd8-4380-9eda-ff603b205172-maas-api-openshift-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:47:28.373730 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:28.373704 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0143cfca-8bd8-4380-9eda-ff603b205172-maas-api-xks-ca" pod "0143cfca-8bd8-4380-9eda-ff603b205172" (UID: "0143cfca-8bd8-4380-9eda-ff603b205172"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:47:28.373890 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:28.373874 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0143cfca-8bd8-4380-9eda-ff603b205172-kube-api-access-s99bw" pod "0143cfca-8bd8-4380-9eda-ff603b205172" (UID: "0143cfca-8bd8-4380-9eda-ff603b205172"). InnerVolumeSpecName "kube-api-access-s99bw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:47:28.473311 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:28.473214 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s99bw\" (UniqueName: \"kubernetes.io/projected/0143cfca-8bd8-4380-9eda-ff603b205172-kube-api-access-s99bw\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:47:28.473311 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:28.473244 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0143cfca-8bd8-4380-9eda-ff603b205172-maas-api-xks-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:47:28.880521 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:28.880490 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2539e5d6-scqdz" Nov 03 01:47:28.880735 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:28.880485 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2539e5d6-scqdz" event={"ID":"0143cfca-8bd8-4380-9eda-ff603b205172","Type":"ContainerDied","Data":"ef51168ec0e86fbe7984748ad2f2457663dbb90a70382657a4ea54d1a3edcc2f"} Nov 03 01:47:28.880735 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:28.880611 4421 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ef51168ec0e86fbe7984748ad2f2457663dbb90a70382657a4ea54d1a3edcc2f" Nov 03 01:47:37.290863 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:37.290818 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-515c8f-openshift-default-67c66d4ffcdz4nw"] Nov 03 01:47:37.291359 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:37.291288 4421 cpu_manager.go:414] "containerMap: removing container" podUID="0143cfca-8bd8-4380-9eda-ff603b205172" containerName="revoke-keys" Nov 03 01:47:37.291359 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:37.291302 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0143cfca-8bd8-4380-9eda-ff603b205172" containerName="revoke-keys" Nov 03 01:47:37.291450 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:37.291386 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="0143cfca-8bd8-4380-9eda-ff603b205172" containerName="revoke-keys" Nov 03 01:47:37.294547 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:37.294528 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-515c8f-openshift-default-67c66d4ffcdz4nw" Nov 03 01:47:37.299523 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:37.299489 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-515c8f-openshift-default-dockercfg-nbbk6\"" Nov 03 01:47:37.307200 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:37.307175 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-515c8f-openshift-default-67c66d4ffcdz4nw"] Nov 03 01:47:37.357838 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:37.357807 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17cc6835-b616-49f8-8dd6-e2cd0ede70a9-istio-data\") pod \"e2e-worker-single-main-515c8f-openshift-default-67c66d4ffcdz4nw\" (UID: \"17cc6835-b616-49f8-8dd6-e2cd0ede70a9\") " pod="openshift-ingress/e2e-worker-single-main-515c8f-openshift-default-67c66d4ffcdz4nw" Nov 03 01:47:37.358313 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:37.357857 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17cc6835-b616-49f8-8dd6-e2cd0ede70a9-istiod-ca-cert\") pod \"e2e-worker-single-main-515c8f-openshift-default-67c66d4ffcdz4nw\" (UID: \"17cc6835-b616-49f8-8dd6-e2cd0ede70a9\") " pod="openshift-ingress/e2e-worker-single-main-515c8f-openshift-default-67c66d4ffcdz4nw" Nov 03 01:47:37.358313 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:37.357930 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17cc6835-b616-49f8-8dd6-e2cd0ede70a9-istio-podinfo\") pod \"e2e-worker-single-main-515c8f-openshift-default-67c66d4ffcdz4nw\" (UID: \"17cc6835-b616-49f8-8dd6-e2cd0ede70a9\") " pod="openshift-ingress/e2e-worker-single-main-515c8f-openshift-default-67c66d4ffcdz4nw" Nov 03 01:47:37.358313 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:37.357983 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17cc6835-b616-49f8-8dd6-e2cd0ede70a9-workload-certs\") pod \"e2e-worker-single-main-515c8f-openshift-default-67c66d4ffcdz4nw\" (UID: \"17cc6835-b616-49f8-8dd6-e2cd0ede70a9\") " pod="openshift-ingress/e2e-worker-single-main-515c8f-openshift-default-67c66d4ffcdz4nw" Nov 03 01:47:37.358313 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:37.358022 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17cc6835-b616-49f8-8dd6-e2cd0ede70a9-workload-socket\") pod \"e2e-worker-single-main-515c8f-openshift-default-67c66d4ffcdz4nw\" (UID: \"17cc6835-b616-49f8-8dd6-e2cd0ede70a9\") " pod="openshift-ingress/e2e-worker-single-main-515c8f-openshift-default-67c66d4ffcdz4nw" Nov 03 01:47:37.358313 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:37.358121 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qf6d8\" (UniqueName: \"kubernetes.io/projected/17cc6835-b616-49f8-8dd6-e2cd0ede70a9-kube-api-access-qf6d8\") pod \"e2e-worker-single-main-515c8f-openshift-default-67c66d4ffcdz4nw\" (UID: \"17cc6835-b616-49f8-8dd6-e2cd0ede70a9\") " pod="openshift-ingress/e2e-worker-single-main-515c8f-openshift-default-67c66d4ffcdz4nw" Nov 03 01:47:37.358313 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:37.358158 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17cc6835-b616-49f8-8dd6-e2cd0ede70a9-istio-envoy\") pod \"e2e-worker-single-main-515c8f-openshift-default-67c66d4ffcdz4nw\" (UID: \"17cc6835-b616-49f8-8dd6-e2cd0ede70a9\") " pod="openshift-ingress/e2e-worker-single-main-515c8f-openshift-default-67c66d4ffcdz4nw" Nov 03 01:47:37.358313 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:37.358184 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17cc6835-b616-49f8-8dd6-e2cd0ede70a9-istio-token\") pod \"e2e-worker-single-main-515c8f-openshift-default-67c66d4ffcdz4nw\" (UID: \"17cc6835-b616-49f8-8dd6-e2cd0ede70a9\") " pod="openshift-ingress/e2e-worker-single-main-515c8f-openshift-default-67c66d4ffcdz4nw" Nov 03 01:47:37.358313 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:37.358224 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17cc6835-b616-49f8-8dd6-e2cd0ede70a9-credential-socket\") pod \"e2e-worker-single-main-515c8f-openshift-default-67c66d4ffcdz4nw\" (UID: \"17cc6835-b616-49f8-8dd6-e2cd0ede70a9\") " pod="openshift-ingress/e2e-worker-single-main-515c8f-openshift-default-67c66d4ffcdz4nw" Nov 03 01:47:37.458882 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:37.458842 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17cc6835-b616-49f8-8dd6-e2cd0ede70a9-workload-certs\") pod \"e2e-worker-single-main-515c8f-openshift-default-67c66d4ffcdz4nw\" (UID: \"17cc6835-b616-49f8-8dd6-e2cd0ede70a9\") " pod="openshift-ingress/e2e-worker-single-main-515c8f-openshift-default-67c66d4ffcdz4nw" Nov 03 01:47:37.459069 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:37.458906 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17cc6835-b616-49f8-8dd6-e2cd0ede70a9-workload-socket\") pod \"e2e-worker-single-main-515c8f-openshift-default-67c66d4ffcdz4nw\" (UID: \"17cc6835-b616-49f8-8dd6-e2cd0ede70a9\") " pod="openshift-ingress/e2e-worker-single-main-515c8f-openshift-default-67c66d4ffcdz4nw" Nov 03 01:47:37.459069 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:37.458939 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qf6d8\" (UniqueName: \"kubernetes.io/projected/17cc6835-b616-49f8-8dd6-e2cd0ede70a9-kube-api-access-qf6d8\") pod \"e2e-worker-single-main-515c8f-openshift-default-67c66d4ffcdz4nw\" (UID: \"17cc6835-b616-49f8-8dd6-e2cd0ede70a9\") " pod="openshift-ingress/e2e-worker-single-main-515c8f-openshift-default-67c66d4ffcdz4nw" Nov 03 01:47:37.459069 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:37.458961 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17cc6835-b616-49f8-8dd6-e2cd0ede70a9-istio-envoy\") pod \"e2e-worker-single-main-515c8f-openshift-default-67c66d4ffcdz4nw\" (UID: \"17cc6835-b616-49f8-8dd6-e2cd0ede70a9\") " pod="openshift-ingress/e2e-worker-single-main-515c8f-openshift-default-67c66d4ffcdz4nw" Nov 03 01:47:37.459069 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:37.458978 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17cc6835-b616-49f8-8dd6-e2cd0ede70a9-istio-token\") pod \"e2e-worker-single-main-515c8f-openshift-default-67c66d4ffcdz4nw\" (UID: \"17cc6835-b616-49f8-8dd6-e2cd0ede70a9\") " pod="openshift-ingress/e2e-worker-single-main-515c8f-openshift-default-67c66d4ffcdz4nw" Nov 03 01:47:37.459069 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:37.459020 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17cc6835-b616-49f8-8dd6-e2cd0ede70a9-credential-socket\") pod \"e2e-worker-single-main-515c8f-openshift-default-67c66d4ffcdz4nw\" (UID: \"17cc6835-b616-49f8-8dd6-e2cd0ede70a9\") " pod="openshift-ingress/e2e-worker-single-main-515c8f-openshift-default-67c66d4ffcdz4nw" Nov 03 01:47:37.459069 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:37.459054 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17cc6835-b616-49f8-8dd6-e2cd0ede70a9-istio-data\") pod \"e2e-worker-single-main-515c8f-openshift-default-67c66d4ffcdz4nw\" (UID: \"17cc6835-b616-49f8-8dd6-e2cd0ede70a9\") " pod="openshift-ingress/e2e-worker-single-main-515c8f-openshift-default-67c66d4ffcdz4nw" Nov 03 01:47:37.459492 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:37.459112 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17cc6835-b616-49f8-8dd6-e2cd0ede70a9-istiod-ca-cert\") pod \"e2e-worker-single-main-515c8f-openshift-default-67c66d4ffcdz4nw\" (UID: \"17cc6835-b616-49f8-8dd6-e2cd0ede70a9\") " pod="openshift-ingress/e2e-worker-single-main-515c8f-openshift-default-67c66d4ffcdz4nw" Nov 03 01:47:37.459492 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:37.459143 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17cc6835-b616-49f8-8dd6-e2cd0ede70a9-istio-podinfo\") pod \"e2e-worker-single-main-515c8f-openshift-default-67c66d4ffcdz4nw\" (UID: \"17cc6835-b616-49f8-8dd6-e2cd0ede70a9\") " pod="openshift-ingress/e2e-worker-single-main-515c8f-openshift-default-67c66d4ffcdz4nw" Nov 03 01:47:37.459492 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:37.459343 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17cc6835-b616-49f8-8dd6-e2cd0ede70a9-workload-certs\") pod \"e2e-worker-single-main-515c8f-openshift-default-67c66d4ffcdz4nw\" (UID: \"17cc6835-b616-49f8-8dd6-e2cd0ede70a9\") " pod="openshift-ingress/e2e-worker-single-main-515c8f-openshift-default-67c66d4ffcdz4nw" Nov 03 01:47:37.459492 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:37.459410 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17cc6835-b616-49f8-8dd6-e2cd0ede70a9-workload-socket\") pod \"e2e-worker-single-main-515c8f-openshift-default-67c66d4ffcdz4nw\" (UID: \"17cc6835-b616-49f8-8dd6-e2cd0ede70a9\") " pod="openshift-ingress/e2e-worker-single-main-515c8f-openshift-default-67c66d4ffcdz4nw" Nov 03 01:47:37.459719 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:37.459605 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17cc6835-b616-49f8-8dd6-e2cd0ede70a9-credential-socket\") pod \"e2e-worker-single-main-515c8f-openshift-default-67c66d4ffcdz4nw\" (UID: \"17cc6835-b616-49f8-8dd6-e2cd0ede70a9\") " pod="openshift-ingress/e2e-worker-single-main-515c8f-openshift-default-67c66d4ffcdz4nw" Nov 03 01:47:37.459975 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:37.459951 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17cc6835-b616-49f8-8dd6-e2cd0ede70a9-istio-data\") pod \"e2e-worker-single-main-515c8f-openshift-default-67c66d4ffcdz4nw\" (UID: \"17cc6835-b616-49f8-8dd6-e2cd0ede70a9\") " pod="openshift-ingress/e2e-worker-single-main-515c8f-openshift-default-67c66d4ffcdz4nw" Nov 03 01:47:37.460138 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:37.460111 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17cc6835-b616-49f8-8dd6-e2cd0ede70a9-istiod-ca-cert\") pod \"e2e-worker-single-main-515c8f-openshift-default-67c66d4ffcdz4nw\" (UID: \"17cc6835-b616-49f8-8dd6-e2cd0ede70a9\") " pod="openshift-ingress/e2e-worker-single-main-515c8f-openshift-default-67c66d4ffcdz4nw" Nov 03 01:47:37.461627 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:37.461599 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17cc6835-b616-49f8-8dd6-e2cd0ede70a9-istio-envoy\") pod \"e2e-worker-single-main-515c8f-openshift-default-67c66d4ffcdz4nw\" (UID: \"17cc6835-b616-49f8-8dd6-e2cd0ede70a9\") " pod="openshift-ingress/e2e-worker-single-main-515c8f-openshift-default-67c66d4ffcdz4nw" Nov 03 01:47:37.461831 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:37.461813 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17cc6835-b616-49f8-8dd6-e2cd0ede70a9-istio-podinfo\") pod \"e2e-worker-single-main-515c8f-openshift-default-67c66d4ffcdz4nw\" (UID: \"17cc6835-b616-49f8-8dd6-e2cd0ede70a9\") " pod="openshift-ingress/e2e-worker-single-main-515c8f-openshift-default-67c66d4ffcdz4nw" Nov 03 01:47:37.467074 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:37.467049 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17cc6835-b616-49f8-8dd6-e2cd0ede70a9-istio-token\") pod \"e2e-worker-single-main-515c8f-openshift-default-67c66d4ffcdz4nw\" (UID: \"17cc6835-b616-49f8-8dd6-e2cd0ede70a9\") " pod="openshift-ingress/e2e-worker-single-main-515c8f-openshift-default-67c66d4ffcdz4nw" Nov 03 01:47:37.467176 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:37.467085 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qf6d8\" (UniqueName: \"kubernetes.io/projected/17cc6835-b616-49f8-8dd6-e2cd0ede70a9-kube-api-access-qf6d8\") pod \"e2e-worker-single-main-515c8f-openshift-default-67c66d4ffcdz4nw\" (UID: \"17cc6835-b616-49f8-8dd6-e2cd0ede70a9\") " pod="openshift-ingress/e2e-worker-single-main-515c8f-openshift-default-67c66d4ffcdz4nw" Nov 03 01:47:37.611727 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:37.611645 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-515c8f-openshift-default-67c66d4ffcdz4nw" Nov 03 01:47:37.895089 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:37.895058 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-515c8f-openshift-default-67c66d4ffcdz4nw"] Nov 03 01:47:37.897972 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:47:37.897942 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17cc6835_b616_49f8_8dd6_e2cd0ede70a9.slice/crio-5ea56e18e0091ad54ff16278893b0c809cbb9a657aa537753708bd42e1b64b8c WatchSource:0}: Error finding container 5ea56e18e0091ad54ff16278893b0c809cbb9a657aa537753708bd42e1b64b8c: Status 404 returned error can't find the container with id 5ea56e18e0091ad54ff16278893b0c809cbb9a657aa537753708bd42e1b64b8c Nov 03 01:47:37.900407 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:37.900372 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:47:37.900500 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:37.900442 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:47:37.900500 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:37.900474 4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:47:37.926537 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:37.926503 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-515c8f-openshift-default-67c66d4ffcdz4nw" event={"ID":"17cc6835-b616-49f8-8dd6-e2cd0ede70a9","Type":"ContainerStarted","Data":"5ea56e18e0091ad54ff16278893b0c809cbb9a657aa537753708bd42e1b64b8c"} Nov 03 01:47:38.934203 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:38.934162 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-515c8f-openshift-default-67c66d4ffcdz4nw" event={"ID":"17cc6835-b616-49f8-8dd6-e2cd0ede70a9","Type":"ContainerStarted","Data":"dbc8635100eb25d057488716f308494a7e48d73fb6e52172758669fa23eaeed7"} Nov 03 01:47:38.956266 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:38.956214 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-515c8f-openshift-default-67c66d4ffcdz4nw" podStartSLOduration=1.956197784 podStartE2EDuration="1.956197784s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:47:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:47:38.954800827 +0000 UTC m=+4001.125803922" watchObservedRunningTime="2026-11-03 01:47:38.956197784 +0000 UTC m=+4001.127200879" Nov 03 01:47:39.612743 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:39.612708 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-515c8f-openshift-default-67c66d4ffcdz4nw" Nov 03 01:47:39.614100 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:39.614071 4421 patch_prober.go:29] interesting pod/e2e-worker-single-main-515c8f-openshift-default-67c66d4ffcdz4nw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.136:15021/healthz/ready\": dial tcp 10.134.1.136:15021: connect: connection refused" start-of-body= Nov 03 01:47:39.614241 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:39.614127 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-515c8f-openshift-default-67c66d4ffcdz4nw" podUID="17cc6835-b616-49f8-8dd6-e2cd0ede70a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.136:15021/healthz/ready\": dial tcp 10.134.1.136:15021: connect: connection refused" Nov 03 01:47:40.612545 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:40.612442 4421 patch_prober.go:29] interesting pod/e2e-worker-single-main-515c8f-openshift-default-67c66d4ffcdz4nw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.136:15021/healthz/ready\": dial tcp 10.134.1.136:15021: connect: connection refused" start-of-body= Nov 03 01:47:40.612545 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:40.612522 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-515c8f-openshift-default-67c66d4ffcdz4nw" podUID="17cc6835-b616-49f8-8dd6-e2cd0ede70a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.136:15021/healthz/ready\": dial tcp 10.134.1.136:15021: connect: connection refused" Nov 03 01:47:41.612906 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:41.612869 4421 patch_prober.go:29] interesting pod/e2e-worker-single-main-515c8f-openshift-default-67c66d4ffcdz4nw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.136:15021/healthz/ready\": dial tcp 10.134.1.136:15021: connect: connection refused" start-of-body= Nov 03 01:47:41.613366 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:41.612937 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-515c8f-openshift-default-67c66d4ffcdz4nw" podUID="17cc6835-b616-49f8-8dd6-e2cd0ede70a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.136:15021/healthz/ready\": dial tcp 10.134.1.136:15021: connect: connection refused" Nov 03 01:47:42.452898 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:42.452858 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-515c8f-78658vzmt5"] Nov 03 01:47:42.456655 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:42.456634 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-515c8f-78658vzmt5" Nov 03 01:47:42.470804 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:42.470740 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-515c8f\"" Nov 03 01:47:42.471135 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:42.471097 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-7c3b72c5-dockercfg-bvllr\"" Nov 03 01:47:42.472895 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:42.472871 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-515c8f-78658vzmt5"] Nov 03 01:47:42.480867 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:42.480841 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-515c8f-5bc5b95bcx7mrg"] Nov 03 01:47:42.484424 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:42.484409 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-515c8f-5bc5b95bcx7mrg" Nov 03 01:47:42.492995 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:42.492971 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-515c8f-5bc5b95bcx7mrg"] Nov 03 01:47:42.508663 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:42.508633 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jp9c\" (UniqueName: \"kubernetes.io/projected/111bfc1a-cee1-4a78-99ea-25c769ed7c2e-kube-api-access-2jp9c\") pod \"payload-pre-processing-e2e-worker-single-main-515c8f-78658vzmt5\" (UID: \"111bfc1a-cee1-4a78-99ea-25c769ed7c2e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-515c8f-78658vzmt5" Nov 03 01:47:42.508663 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:42.508667 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43919454-2a18-49a8-9582-5c0bebc68666-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-515c8f-5bc5b95bcx7mrg\" (UID: \"43919454-2a18-49a8-9582-5c0bebc68666\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-515c8f-5bc5b95bcx7mrg" Nov 03 01:47:42.508935 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:42.508906 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/111bfc1a-cee1-4a78-99ea-25c769ed7c2e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-515c8f-78658vzmt5\" (UID: \"111bfc1a-cee1-4a78-99ea-25c769ed7c2e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-515c8f-78658vzmt5" Nov 03 01:47:42.509000 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:42.508955 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pl55j\" (UniqueName: \"kubernetes.io/projected/43919454-2a18-49a8-9582-5c0bebc68666-kube-api-access-pl55j\") pod \"payload-processing-e2e-worker-single-main-515c8f-5bc5b95bcx7mrg\" (UID: \"43919454-2a18-49a8-9582-5c0bebc68666\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-515c8f-5bc5b95bcx7mrg" Nov 03 01:47:42.610174 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:42.610140 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jp9c\" (UniqueName: \"kubernetes.io/projected/111bfc1a-cee1-4a78-99ea-25c769ed7c2e-kube-api-access-2jp9c\") pod \"payload-pre-processing-e2e-worker-single-main-515c8f-78658vzmt5\" (UID: \"111bfc1a-cee1-4a78-99ea-25c769ed7c2e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-515c8f-78658vzmt5" Nov 03 01:47:42.610174 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:42.610175 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43919454-2a18-49a8-9582-5c0bebc68666-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-515c8f-5bc5b95bcx7mrg\" (UID: \"43919454-2a18-49a8-9582-5c0bebc68666\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-515c8f-5bc5b95bcx7mrg" Nov 03 01:47:42.610480 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:42.610264 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/111bfc1a-cee1-4a78-99ea-25c769ed7c2e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-515c8f-78658vzmt5\" (UID: \"111bfc1a-cee1-4a78-99ea-25c769ed7c2e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-515c8f-78658vzmt5" Nov 03 01:47:42.610480 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:42.610319 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pl55j\" (UniqueName: \"kubernetes.io/projected/43919454-2a18-49a8-9582-5c0bebc68666-kube-api-access-pl55j\") pod \"payload-processing-e2e-worker-single-main-515c8f-5bc5b95bcx7mrg\" (UID: \"43919454-2a18-49a8-9582-5c0bebc68666\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-515c8f-5bc5b95bcx7mrg" Nov 03 01:47:42.611048 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:42.611021 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43919454-2a18-49a8-9582-5c0bebc68666-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-515c8f-5bc5b95bcx7mrg\" (UID: \"43919454-2a18-49a8-9582-5c0bebc68666\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-515c8f-5bc5b95bcx7mrg" Nov 03 01:47:42.611148 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:42.611049 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/111bfc1a-cee1-4a78-99ea-25c769ed7c2e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-515c8f-78658vzmt5\" (UID: \"111bfc1a-cee1-4a78-99ea-25c769ed7c2e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-515c8f-78658vzmt5" Nov 03 01:47:42.612162 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:42.612142 4421 patch_prober.go:29] interesting pod/e2e-worker-single-main-515c8f-openshift-default-67c66d4ffcdz4nw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.136:15021/healthz/ready\": dial tcp 10.134.1.136:15021: connect: connection refused" start-of-body= Nov 03 01:47:42.612231 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:42.612183 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-515c8f-openshift-default-67c66d4ffcdz4nw" podUID="17cc6835-b616-49f8-8dd6-e2cd0ede70a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.136:15021/healthz/ready\": dial tcp 10.134.1.136:15021: connect: connection refused" Nov 03 01:47:42.621479 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:42.621457 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pl55j\" (UniqueName: \"kubernetes.io/projected/43919454-2a18-49a8-9582-5c0bebc68666-kube-api-access-pl55j\") pod \"payload-processing-e2e-worker-single-main-515c8f-5bc5b95bcx7mrg\" (UID: \"43919454-2a18-49a8-9582-5c0bebc68666\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-515c8f-5bc5b95bcx7mrg" Nov 03 01:47:42.621785 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:42.621534 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jp9c\" (UniqueName: \"kubernetes.io/projected/111bfc1a-cee1-4a78-99ea-25c769ed7c2e-kube-api-access-2jp9c\") pod \"payload-pre-processing-e2e-worker-single-main-515c8f-78658vzmt5\" (UID: \"111bfc1a-cee1-4a78-99ea-25c769ed7c2e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-515c8f-78658vzmt5" Nov 03 01:47:42.769770 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:42.769729 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-515c8f-78658vzmt5" Nov 03 01:47:42.802408 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:42.802371 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-515c8f-5bc5b95bcx7mrg" Nov 03 01:47:43.050478 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:43.050437 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-515c8f-78658vzmt5"] Nov 03 01:47:43.053005 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:47:43.052963 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod111bfc1a_cee1_4a78_99ea_25c769ed7c2e.slice/crio-1d2c8d84001ddbe6a2f03a0b39f6a99b09b51a7a48bcdf5f887f9c8acc955091 WatchSource:0}: Error finding container 1d2c8d84001ddbe6a2f03a0b39f6a99b09b51a7a48bcdf5f887f9c8acc955091: Status 404 returned error can't find the container with id 1d2c8d84001ddbe6a2f03a0b39f6a99b09b51a7a48bcdf5f887f9c8acc955091 Nov 03 01:47:43.094891 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:43.094860 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-515c8f-5bc5b95bcx7mrg"] Nov 03 01:47:43.100047 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:47:43.100014 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43919454_2a18_49a8_9582_5c0bebc68666.slice/crio-7b7237c5c4733044c958371ad0877eea92f3fa527432103798d5b0948da3f3fb WatchSource:0}: Error finding container 7b7237c5c4733044c958371ad0877eea92f3fa527432103798d5b0948da3f3fb: Status 404 returned error can't find the container with id 7b7237c5c4733044c958371ad0877eea92f3fa527432103798d5b0948da3f3fb Nov 03 01:47:43.612488 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:43.612453 4421 patch_prober.go:29] interesting pod/e2e-worker-single-main-515c8f-openshift-default-67c66d4ffcdz4nw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.136:15021/healthz/ready\": dial tcp 10.134.1.136:15021: connect: connection refused" start-of-body= Nov 03 01:47:43.612710 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:43.612512 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-515c8f-openshift-default-67c66d4ffcdz4nw" podUID="17cc6835-b616-49f8-8dd6-e2cd0ede70a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.136:15021/healthz/ready\": dial tcp 10.134.1.136:15021: connect: connection refused" Nov 03 01:47:43.960416 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:43.960307 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-515c8f-5bc5b95bcx7mrg" event={"ID":"43919454-2a18-49a8-9582-5c0bebc68666","Type":"ContainerStarted","Data":"385b7d8c7677acc7eada0ca323ae312f210551c48b63dd2930f5eff4f56de8dc"} Nov 03 01:47:43.960416 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:43.960375 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-515c8f-5bc5b95bcx7mrg" event={"ID":"43919454-2a18-49a8-9582-5c0bebc68666","Type":"ContainerStarted","Data":"7b7237c5c4733044c958371ad0877eea92f3fa527432103798d5b0948da3f3fb"} Nov 03 01:47:43.960941 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:43.960526 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-515c8f-5bc5b95bcx7mrg" Nov 03 01:47:43.961936 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:43.961910 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-515c8f-78658vzmt5" event={"ID":"111bfc1a-cee1-4a78-99ea-25c769ed7c2e","Type":"ContainerStarted","Data":"ee4e1792eecd889a95c389697017cc55ae6ac2b74e1b803ac4f071ab35c4a3fa"} Nov 03 01:47:43.962051 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:43.961954 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-515c8f-78658vzmt5" event={"ID":"111bfc1a-cee1-4a78-99ea-25c769ed7c2e","Type":"ContainerStarted","Data":"1d2c8d84001ddbe6a2f03a0b39f6a99b09b51a7a48bcdf5f887f9c8acc955091"} Nov 03 01:47:43.962051 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:43.961997 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-515c8f-78658vzmt5" Nov 03 01:47:43.977510 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:43.977463 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-515c8f-5bc5b95bcx7mrg" podStartSLOduration=1.977447586 podStartE2EDuration="1.977447586s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:47:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:47:43.975496398 +0000 UTC m=+4006.146499493" watchObservedRunningTime="2026-11-03 01:47:43.977447586 +0000 UTC m=+4006.148450689" Nov 03 01:47:43.991622 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:43.991577 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-515c8f-78658vzmt5" podStartSLOduration=1.991564229 podStartE2EDuration="1.991564229s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:47:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:47:43.989673903 +0000 UTC m=+4006.160676998" watchObservedRunningTime="2026-11-03 01:47:43.991564229 +0000 UTC m=+4006.162567325" Nov 03 01:47:44.612208 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:44.612170 4421 patch_prober.go:29] interesting pod/e2e-worker-single-main-515c8f-openshift-default-67c66d4ffcdz4nw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.136:15021/healthz/ready\": dial tcp 10.134.1.136:15021: connect: connection refused" start-of-body= Nov 03 01:47:44.612462 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:44.612235 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-515c8f-openshift-default-67c66d4ffcdz4nw" podUID="17cc6835-b616-49f8-8dd6-e2cd0ede70a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.136:15021/healthz/ready\": dial tcp 10.134.1.136:15021: connect: connection refused" Nov 03 01:47:44.668645 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:47:44.668609 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 01:47:44.853977 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:44.853936 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gw8mh"] Nov 03 01:47:44.854201 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:44.854128 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gw8mh" podUID="8e670068-38e4-4a0f-9313-4757438f9727" containerName="discovery" containerID="cri-o://be251286daa2cd0b2e68643a924b3c1cc424c165c6fdc84393671e7991f4d817" gracePeriod=30 Nov 03 01:47:45.147083 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:45.147015 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gw8mh" Nov 03 01:47:45.237687 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:45.237648 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e670068-38e4-4a0f-9313-4757438f9727-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8e670068-38e4-4a0f-9313-4757438f9727-istio-kubeconfig\") pod \"8e670068-38e4-4a0f-9313-4757438f9727\" (UID: \"8e670068-38e4-4a0f-9313-4757438f9727\") " Nov 03 01:47:45.237862 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:45.237714 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e670068-38e4-4a0f-9313-4757438f9727-istio-token\" (UniqueName: \"kubernetes.io/projected/8e670068-38e4-4a0f-9313-4757438f9727-istio-token\") pod \"8e670068-38e4-4a0f-9313-4757438f9727\" (UID: \"8e670068-38e4-4a0f-9313-4757438f9727\") " Nov 03 01:47:45.237862 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:45.237768 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e670068-38e4-4a0f-9313-4757438f9727-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e670068-38e4-4a0f-9313-4757438f9727-local-certs\") pod \"8e670068-38e4-4a0f-9313-4757438f9727\" (UID: \"8e670068-38e4-4a0f-9313-4757438f9727\") " Nov 03 01:47:45.237862 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:45.237801 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e670068-38e4-4a0f-9313-4757438f9727-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/8e670068-38e4-4a0f-9313-4757438f9727-istio-csr-dns-cert\") pod \"8e670068-38e4-4a0f-9313-4757438f9727\" (UID: \"8e670068-38e4-4a0f-9313-4757438f9727\") " Nov 03 01:47:45.238030 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:45.237875 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e670068-38e4-4a0f-9313-4757438f9727-kube-api-access-j97l5\" (UniqueName: \"kubernetes.io/projected/8e670068-38e4-4a0f-9313-4757438f9727-kube-api-access-j97l5\") pod \"8e670068-38e4-4a0f-9313-4757438f9727\" (UID: \"8e670068-38e4-4a0f-9313-4757438f9727\") " Nov 03 01:47:45.238030 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:45.237997 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e670068-38e4-4a0f-9313-4757438f9727-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/8e670068-38e4-4a0f-9313-4757438f9727-istio-csr-ca-configmap\") pod \"8e670068-38e4-4a0f-9313-4757438f9727\" (UID: \"8e670068-38e4-4a0f-9313-4757438f9727\") " Nov 03 01:47:45.238126 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:45.238036 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e670068-38e4-4a0f-9313-4757438f9727-cacerts\" (UniqueName: \"kubernetes.io/secret/8e670068-38e4-4a0f-9313-4757438f9727-cacerts\") pod \"8e670068-38e4-4a0f-9313-4757438f9727\" (UID: \"8e670068-38e4-4a0f-9313-4757438f9727\") " Nov 03 01:47:45.239728 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:45.239633 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e670068-38e4-4a0f-9313-4757438f9727-istio-csr-ca-configmap" pod "8e670068-38e4-4a0f-9313-4757438f9727" (UID: "8e670068-38e4-4a0f-9313-4757438f9727"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:47:45.241178 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:45.241150 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e670068-38e4-4a0f-9313-4757438f9727-cacerts" pod "8e670068-38e4-4a0f-9313-4757438f9727" (UID: "8e670068-38e4-4a0f-9313-4757438f9727"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:47:45.241427 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:45.241194 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e670068-38e4-4a0f-9313-4757438f9727-istio-token" pod "8e670068-38e4-4a0f-9313-4757438f9727" (UID: "8e670068-38e4-4a0f-9313-4757438f9727"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:47:45.241427 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:45.241328 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e670068-38e4-4a0f-9313-4757438f9727-local-certs" pod "8e670068-38e4-4a0f-9313-4757438f9727" (UID: "8e670068-38e4-4a0f-9313-4757438f9727"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:47:45.243209 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:45.243180 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e670068-38e4-4a0f-9313-4757438f9727-istio-csr-dns-cert" pod "8e670068-38e4-4a0f-9313-4757438f9727" (UID: "8e670068-38e4-4a0f-9313-4757438f9727"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:47:45.243323 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:45.243187 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e670068-38e4-4a0f-9313-4757438f9727-kube-api-access-j97l5" pod "8e670068-38e4-4a0f-9313-4757438f9727" (UID: "8e670068-38e4-4a0f-9313-4757438f9727"). InnerVolumeSpecName "kube-api-access-j97l5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:47:45.246370 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:45.246348 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e670068-38e4-4a0f-9313-4757438f9727-istio-kubeconfig" pod "8e670068-38e4-4a0f-9313-4757438f9727" (UID: "8e670068-38e4-4a0f-9313-4757438f9727"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:47:45.339522 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:45.339480 4421 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e670068-38e4-4a0f-9313-4757438f9727-istio-token\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:47:45.339522 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:45.339515 4421 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e670068-38e4-4a0f-9313-4757438f9727-local-certs\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:47:45.339522 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:45.339526 4421 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/8e670068-38e4-4a0f-9313-4757438f9727-istio-csr-dns-cert\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:47:45.339783 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:45.339560 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j97l5\" (UniqueName: \"kubernetes.io/projected/8e670068-38e4-4a0f-9313-4757438f9727-kube-api-access-j97l5\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:47:45.339783 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:45.339572 4421 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/8e670068-38e4-4a0f-9313-4757438f9727-istio-csr-ca-configmap\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:47:45.339783 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:45.339581 4421 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/8e670068-38e4-4a0f-9313-4757438f9727-cacerts\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:47:45.339783 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:45.339590 4421 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8e670068-38e4-4a0f-9313-4757438f9727-istio-kubeconfig\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:47:45.618865 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:45.618821 4421 patch_prober.go:29] interesting pod/e2e-worker-single-main-515c8f-openshift-default-67c66d4ffcdz4nw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.136:15021/healthz/ready\": dial tcp 10.134.1.136:15021: connect: connection refused" start-of-body= Nov 03 01:47:45.619188 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:45.619156 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-515c8f-openshift-default-67c66d4ffcdz4nw" podUID="17cc6835-b616-49f8-8dd6-e2cd0ede70a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.136:15021/healthz/ready\": dial tcp 10.134.1.136:15021: connect: connection refused" Nov 03 01:47:45.974473 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:45.974381 4421 generic.go:356] "Generic (PLEG): container finished" podID="8e670068-38e4-4a0f-9313-4757438f9727" containerID="be251286daa2cd0b2e68643a924b3c1cc424c165c6fdc84393671e7991f4d817" exitCode=0 Nov 03 01:47:45.974665 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:45.974509 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gw8mh" Nov 03 01:47:45.977484 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:45.977458 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gw8mh" event={"ID":"8e670068-38e4-4a0f-9313-4757438f9727","Type":"ContainerDied","Data":"be251286daa2cd0b2e68643a924b3c1cc424c165c6fdc84393671e7991f4d817"} Nov 03 01:47:45.977649 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:45.977502 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gw8mh" event={"ID":"8e670068-38e4-4a0f-9313-4757438f9727","Type":"ContainerDied","Data":"5776400ca28fb73de73f317b4da6d210da6a3c1295d911a09d22efd4ef6f7443"} Nov 03 01:47:45.977649 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:45.977532 4421 scope.go:122] "RemoveContainer" containerID="be251286daa2cd0b2e68643a924b3c1cc424c165c6fdc84393671e7991f4d817" Nov 03 01:47:45.992205 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:45.992186 4421 scope.go:122] "RemoveContainer" containerID="be251286daa2cd0b2e68643a924b3c1cc424c165c6fdc84393671e7991f4d817" Nov 03 01:47:45.992517 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:47:45.992494 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be251286daa2cd0b2e68643a924b3c1cc424c165c6fdc84393671e7991f4d817\": container with ID starting with be251286daa2cd0b2e68643a924b3c1cc424c165c6fdc84393671e7991f4d817 not found: ID does not exist" containerID="be251286daa2cd0b2e68643a924b3c1cc424c165c6fdc84393671e7991f4d817" Nov 03 01:47:45.992602 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:45.992528 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be251286daa2cd0b2e68643a924b3c1cc424c165c6fdc84393671e7991f4d817"} err="failed to get container status \"be251286daa2cd0b2e68643a924b3c1cc424c165c6fdc84393671e7991f4d817\": rpc error: code = NotFound desc = could not find container \"be251286daa2cd0b2e68643a924b3c1cc424c165c6fdc84393671e7991f4d817\": container with ID starting with be251286daa2cd0b2e68643a924b3c1cc424c165c6fdc84393671e7991f4d817 not found: ID does not exist" Nov 03 01:47:46.002764 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:46.002741 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gw8mh"] Nov 03 01:47:46.032019 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:46.031964 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gw8mh"] Nov 03 01:47:46.624324 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:46.618726 4421 patch_prober.go:29] interesting pod/e2e-worker-single-main-515c8f-openshift-default-67c66d4ffcdz4nw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.136:15021/healthz/ready\": dial tcp 10.134.1.136:15021: connect: connection refused" start-of-body= Nov 03 01:47:46.624324 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:46.618797 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-515c8f-openshift-default-67c66d4ffcdz4nw" podUID="17cc6835-b616-49f8-8dd6-e2cd0ede70a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.136:15021/healthz/ready\": dial tcp 10.134.1.136:15021: connect: connection refused" Nov 03 01:47:47.614774 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:47.614733 4421 patch_prober.go:29] interesting pod/e2e-worker-single-main-515c8f-openshift-default-67c66d4ffcdz4nw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.136:15021/healthz/ready\": dial tcp 10.134.1.136:15021: connect: connection refused" start-of-body= Nov 03 01:47:47.615160 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:47.615124 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-515c8f-openshift-default-67c66d4ffcdz4nw" podUID="17cc6835-b616-49f8-8dd6-e2cd0ede70a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.136:15021/healthz/ready\": dial tcp 10.134.1.136:15021: connect: connection refused" Nov 03 01:47:47.976987 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:47.976907 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e670068-38e4-4a0f-9313-4757438f9727" path="/var/lib/kubelet/pods/8e670068-38e4-4a0f-9313-4757438f9727/volumes" Nov 03 01:47:48.627975 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:48.627940 4421 patch_prober.go:29] interesting pod/e2e-worker-single-main-515c8f-openshift-default-67c66d4ffcdz4nw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.136:15021/healthz/ready\": dial tcp 10.134.1.136:15021: connect: connection refused" start-of-body= Nov 03 01:47:48.628302 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:48.628253 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-515c8f-openshift-default-67c66d4ffcdz4nw" podUID="17cc6835-b616-49f8-8dd6-e2cd0ede70a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.136:15021/healthz/ready\": dial tcp 10.134.1.136:15021: connect: connection refused" Nov 03 01:47:49.619801 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:49.619756 4421 patch_prober.go:29] interesting pod/e2e-worker-single-main-515c8f-openshift-default-67c66d4ffcdz4nw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.136:15021/healthz/ready\": dial tcp 10.134.1.136:15021: connect: connection refused" start-of-body= Nov 03 01:47:49.620465 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:49.620431 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-515c8f-openshift-default-67c66d4ffcdz4nw" podUID="17cc6835-b616-49f8-8dd6-e2cd0ede70a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.136:15021/healthz/ready\": dial tcp 10.134.1.136:15021: connect: connection refused" Nov 03 01:47:50.612884 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:50.612844 4421 patch_prober.go:29] interesting pod/e2e-worker-single-main-515c8f-openshift-default-67c66d4ffcdz4nw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.136:15021/healthz/ready\": dial tcp 10.134.1.136:15021: connect: connection refused" start-of-body= Nov 03 01:47:50.613127 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:50.612916 4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-515c8f-openshift-default-67c66d4ffcdz4nw" podUID="17cc6835-b616-49f8-8dd6-e2cd0ede70a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.136:15021/healthz/ready\": dial tcp 10.134.1.136:15021: connect: connection refused" Nov 03 01:47:51.617034 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:51.617004 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-515c8f-openshift-default-67c66d4ffcdz4nw" Nov 03 01:47:51.617588 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:51.617315 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-515c8f-openshift-default-67c66d4ffcdz4nw" Nov 03 01:47:51.617984 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:51.617966 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-515c8f-openshift-default-67c66d4ffcdz4nw" Nov 03 01:47:54.970286 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:54.970251 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-515c8f-5bc5b95bcx7mrg" Nov 03 01:47:54.970817 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:47:54.970436 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-515c8f-78658vzmt5" Nov 03 01:48:00.552296 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:48:00.552243 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-515c8f/simulator-single-main-515c8f-kserve-7488db5694-g8xpd"] Nov 03 01:48:00.553054 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:48:00.553033 4421 cpu_manager.go:414] "containerMap: removing container" podUID="8e670068-38e4-4a0f-9313-4757438f9727" containerName="discovery" Nov 03 01:48:00.553054 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:48:00.553052 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e670068-38e4-4a0f-9313-4757438f9727" containerName="discovery" Nov 03 01:48:00.553228 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:48:00.553147 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e670068-38e4-4a0f-9313-4757438f9727" containerName="discovery" Nov 03 01:48:00.557881 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:48:00.557860 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-515c8f/simulator-single-main-515c8f-kserve-7488db5694-g8xpd" Nov 03 01:48:00.563733 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:48:00.563707 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-515c8f\"/\"openshift-service-ca.crt\"" Nov 03 01:48:00.563871 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:48:00.563774 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-515c8f\"/\"kube-root-ca.crt\"" Nov 03 01:48:00.563871 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:48:00.563834 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-515c8f\"/\"simulator-single-main-515c8f-kserve-self-signed-certs\"" Nov 03 01:48:00.563871 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:48:00.563866 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-515c8f\"/\"default-dockercfg-ldfx8\"" Nov 03 01:48:00.566389 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:48:00.566359 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-515c8f/simulator-single-main-515c8f-kserve-7488db5694-g8xpd"] Nov 03 01:48:00.697476 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:48:00.697434 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64d6d02c-ec5c-4aaf-914f-ccd45c0deecb-tmp-dir\") pod \"simulator-single-main-515c8f-kserve-7488db5694-g8xpd\" (UID: \"64d6d02c-ec5c-4aaf-914f-ccd45c0deecb\") " pod="e2e-models-e2e-worker-single-main-515c8f/simulator-single-main-515c8f-kserve-7488db5694-g8xpd" Nov 03 01:48:00.697476 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:48:00.697477 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64d6d02c-ec5c-4aaf-914f-ccd45c0deecb-home\") pod \"simulator-single-main-515c8f-kserve-7488db5694-g8xpd\" (UID: \"64d6d02c-ec5c-4aaf-914f-ccd45c0deecb\") " pod="e2e-models-e2e-worker-single-main-515c8f/simulator-single-main-515c8f-kserve-7488db5694-g8xpd" Nov 03 01:48:00.697709 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:48:00.697497 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64d6d02c-ec5c-4aaf-914f-ccd45c0deecb-dshm\") pod \"simulator-single-main-515c8f-kserve-7488db5694-g8xpd\" (UID: \"64d6d02c-ec5c-4aaf-914f-ccd45c0deecb\") " pod="e2e-models-e2e-worker-single-main-515c8f/simulator-single-main-515c8f-kserve-7488db5694-g8xpd" Nov 03 01:48:00.697709 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:48:00.697584 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64d6d02c-ec5c-4aaf-914f-ccd45c0deecb-tls-certs\") pod \"simulator-single-main-515c8f-kserve-7488db5694-g8xpd\" (UID: \"64d6d02c-ec5c-4aaf-914f-ccd45c0deecb\") " pod="e2e-models-e2e-worker-single-main-515c8f/simulator-single-main-515c8f-kserve-7488db5694-g8xpd" Nov 03 01:48:00.697709 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:48:00.697634 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64d6d02c-ec5c-4aaf-914f-ccd45c0deecb-model-cache\") pod \"simulator-single-main-515c8f-kserve-7488db5694-g8xpd\" (UID: \"64d6d02c-ec5c-4aaf-914f-ccd45c0deecb\") " pod="e2e-models-e2e-worker-single-main-515c8f/simulator-single-main-515c8f-kserve-7488db5694-g8xpd" Nov 03 01:48:00.697843 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:48:00.697747 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qt9gb\" (UniqueName: \"kubernetes.io/projected/64d6d02c-ec5c-4aaf-914f-ccd45c0deecb-kube-api-access-qt9gb\") pod \"simulator-single-main-515c8f-kserve-7488db5694-g8xpd\" (UID: \"64d6d02c-ec5c-4aaf-914f-ccd45c0deecb\") " pod="e2e-models-e2e-worker-single-main-515c8f/simulator-single-main-515c8f-kserve-7488db5694-g8xpd" Nov 03 01:48:00.799263 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:48:00.799223 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qt9gb\" (UniqueName: \"kubernetes.io/projected/64d6d02c-ec5c-4aaf-914f-ccd45c0deecb-kube-api-access-qt9gb\") pod \"simulator-single-main-515c8f-kserve-7488db5694-g8xpd\" (UID: \"64d6d02c-ec5c-4aaf-914f-ccd45c0deecb\") " pod="e2e-models-e2e-worker-single-main-515c8f/simulator-single-main-515c8f-kserve-7488db5694-g8xpd" Nov 03 01:48:00.799494 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:48:00.799301 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64d6d02c-ec5c-4aaf-914f-ccd45c0deecb-tmp-dir\") pod \"simulator-single-main-515c8f-kserve-7488db5694-g8xpd\" (UID: \"64d6d02c-ec5c-4aaf-914f-ccd45c0deecb\") " pod="e2e-models-e2e-worker-single-main-515c8f/simulator-single-main-515c8f-kserve-7488db5694-g8xpd" Nov 03 01:48:00.799494 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:48:00.799331 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64d6d02c-ec5c-4aaf-914f-ccd45c0deecb-home\") pod \"simulator-single-main-515c8f-kserve-7488db5694-g8xpd\" (UID: \"64d6d02c-ec5c-4aaf-914f-ccd45c0deecb\") " pod="e2e-models-e2e-worker-single-main-515c8f/simulator-single-main-515c8f-kserve-7488db5694-g8xpd" Nov 03 01:48:00.799494 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:48:00.799356 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64d6d02c-ec5c-4aaf-914f-ccd45c0deecb-dshm\") pod \"simulator-single-main-515c8f-kserve-7488db5694-g8xpd\" (UID: \"64d6d02c-ec5c-4aaf-914f-ccd45c0deecb\") " pod="e2e-models-e2e-worker-single-main-515c8f/simulator-single-main-515c8f-kserve-7488db5694-g8xpd" Nov 03 01:48:00.799494 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:48:00.799404 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64d6d02c-ec5c-4aaf-914f-ccd45c0deecb-tls-certs\") pod \"simulator-single-main-515c8f-kserve-7488db5694-g8xpd\" (UID: \"64d6d02c-ec5c-4aaf-914f-ccd45c0deecb\") " pod="e2e-models-e2e-worker-single-main-515c8f/simulator-single-main-515c8f-kserve-7488db5694-g8xpd" Nov 03 01:48:00.799494 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:48:00.799484 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64d6d02c-ec5c-4aaf-914f-ccd45c0deecb-model-cache\") pod \"simulator-single-main-515c8f-kserve-7488db5694-g8xpd\" (UID: \"64d6d02c-ec5c-4aaf-914f-ccd45c0deecb\") " pod="e2e-models-e2e-worker-single-main-515c8f/simulator-single-main-515c8f-kserve-7488db5694-g8xpd" Nov 03 01:48:00.799812 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:48:00.799730 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64d6d02c-ec5c-4aaf-914f-ccd45c0deecb-tmp-dir\") pod \"simulator-single-main-515c8f-kserve-7488db5694-g8xpd\" (UID: \"64d6d02c-ec5c-4aaf-914f-ccd45c0deecb\") " pod="e2e-models-e2e-worker-single-main-515c8f/simulator-single-main-515c8f-kserve-7488db5694-g8xpd" Nov 03 01:48:00.799812 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:48:00.799756 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64d6d02c-ec5c-4aaf-914f-ccd45c0deecb-home\") pod \"simulator-single-main-515c8f-kserve-7488db5694-g8xpd\" (UID: \"64d6d02c-ec5c-4aaf-914f-ccd45c0deecb\") " pod="e2e-models-e2e-worker-single-main-515c8f/simulator-single-main-515c8f-kserve-7488db5694-g8xpd" Nov 03 01:48:00.799908 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:48:00.799842 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64d6d02c-ec5c-4aaf-914f-ccd45c0deecb-model-cache\") pod \"simulator-single-main-515c8f-kserve-7488db5694-g8xpd\" (UID: \"64d6d02c-ec5c-4aaf-914f-ccd45c0deecb\") " pod="e2e-models-e2e-worker-single-main-515c8f/simulator-single-main-515c8f-kserve-7488db5694-g8xpd" Nov 03 01:48:00.801800 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:48:00.801773 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64d6d02c-ec5c-4aaf-914f-ccd45c0deecb-dshm\") pod \"simulator-single-main-515c8f-kserve-7488db5694-g8xpd\" (UID: \"64d6d02c-ec5c-4aaf-914f-ccd45c0deecb\") " pod="e2e-models-e2e-worker-single-main-515c8f/simulator-single-main-515c8f-kserve-7488db5694-g8xpd" Nov 03 01:48:00.801932 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:48:00.801914 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64d6d02c-ec5c-4aaf-914f-ccd45c0deecb-tls-certs\") pod \"simulator-single-main-515c8f-kserve-7488db5694-g8xpd\" (UID: \"64d6d02c-ec5c-4aaf-914f-ccd45c0deecb\") " pod="e2e-models-e2e-worker-single-main-515c8f/simulator-single-main-515c8f-kserve-7488db5694-g8xpd" Nov 03 01:48:00.806880 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:48:00.806833 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qt9gb\" (UniqueName: \"kubernetes.io/projected/64d6d02c-ec5c-4aaf-914f-ccd45c0deecb-kube-api-access-qt9gb\") pod \"simulator-single-main-515c8f-kserve-7488db5694-g8xpd\" (UID: \"64d6d02c-ec5c-4aaf-914f-ccd45c0deecb\") " pod="e2e-models-e2e-worker-single-main-515c8f/simulator-single-main-515c8f-kserve-7488db5694-g8xpd" Nov 03 01:48:00.872453 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:48:00.872418 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-515c8f/simulator-single-main-515c8f-kserve-7488db5694-g8xpd" Nov 03 01:48:01.151662 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:48:01.151631 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-515c8f/simulator-single-main-515c8f-kserve-7488db5694-g8xpd"] Nov 03 01:48:01.154011 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:48:01.153980 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod64d6d02c_ec5c_4aaf_914f_ccd45c0deecb.slice/crio-e0c25acf4c5a67e32d9de5e1a331529adfb28527a8cf3f152d3dc9ef076728c8 WatchSource:0}: Error finding container e0c25acf4c5a67e32d9de5e1a331529adfb28527a8cf3f152d3dc9ef076728c8: Status 404 returned error can't find the container with id e0c25acf4c5a67e32d9de5e1a331529adfb28527a8cf3f152d3dc9ef076728c8 Nov 03 01:48:02.063445 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:48:02.063404 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-515c8f/simulator-single-main-515c8f-kserve-7488db5694-g8xpd" event={"ID":"64d6d02c-ec5c-4aaf-914f-ccd45c0deecb","Type":"ContainerStarted","Data":"a46d0ffa7ea46bbcc295ba6cfd69b7127bb5c92df8ba8c9682f3aa85590bc8e5"} Nov 03 01:48:02.063851 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:48:02.063456 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-515c8f/simulator-single-main-515c8f-kserve-7488db5694-g8xpd" event={"ID":"64d6d02c-ec5c-4aaf-914f-ccd45c0deecb","Type":"ContainerStarted","Data":"e0c25acf4c5a67e32d9de5e1a331529adfb28527a8cf3f152d3dc9ef076728c8"} Nov 03 01:48:02.081887 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:48:02.081837 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-515c8f/simulator-single-main-515c8f-kserve-7488db5694-g8xpd" podStartSLOduration=2.081822739 podStartE2EDuration="2.081822739s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:48:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:48:02.079892325 +0000 UTC m=+4024.250895419" watchObservedRunningTime="2026-11-03 01:48:02.081822739 +0000 UTC m=+4024.252825834" Nov 03 01:48:02.873439 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:48:02.873396 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-515c8f/simulator-single-main-515c8f-kserve-7488db5694-g8xpd" Nov 03 01:48:06.938959 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:48:06.938921 4421 scope.go:122] "RemoveContainer" containerID="fe07e5ecf5cce2ef4a8039ec70ccefba7d8e2c79c8a7313f99a7c0509f203719" Nov 03 01:48:06.948237 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:48:06.948214 4421 scope.go:122] "RemoveContainer" containerID="a51e848cbff6bcad6b457200ebc9a9d2ca36b5123f1e31645dfa7a9c4d888e86" Nov 03 01:48:06.956434 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:48:06.956409 4421 scope.go:122] "RemoveContainer" containerID="089638156edf368cf15cbddaf0799fcb496dda8db1fbe0050bf39deef21c5690" Nov 03 01:48:06.964568 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:48:06.964548 4421 scope.go:122] "RemoveContainer" containerID="f3b13ada8b5b3a8795ee0297827dfcdefa4aaf8317dc1a3f54d0e2d7fb8b88a1" Nov 03 01:48:06.972622 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:48:06.972596 4421 scope.go:122] "RemoveContainer" containerID="4665afafb4dfa9894c5f0af3f636e0a18b9cad983a6f0973af2e06a395fde60d" Nov 03 01:48:06.982426 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:48:06.982405 4421 scope.go:122] "RemoveContainer" containerID="6fb8ee4b10e193db76ebcb9a1d1c21d27b583b5d9f95f7f1d6063d181b8f5019" Nov 03 01:48:10.872766 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:48:10.872719 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-515c8f/simulator-single-main-515c8f-kserve-7488db5694-g8xpd" Nov 03 01:48:10.885615 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:48:10.885588 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-515c8f/simulator-single-main-515c8f-kserve-7488db5694-g8xpd" Nov 03 01:48:11.127269 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:48:11.127169 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-515c8f/simulator-single-main-515c8f-kserve-7488db5694-g8xpd" Nov 03 01:48:14.670064 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:48:14.670020 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 01:48:44.669142 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:48:44.669101 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 01:48:49.059124 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:48:49.059086 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-515c8f/simulator-single-main-515c8f-kserve-7488db5694-g8xpd"] Nov 03 01:48:50.033402 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:48:50.033370 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-515c8f-5bc5b95bcx7mrg"] Nov 03 01:48:50.039865 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:48:50.039833 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-515c8f-78658vzmt5"] Nov 03 01:48:55.034388 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:48:55.034321 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-515c8f-5bc5b95bcx7mrg" podUID="43919454-2a18-49a8-9582-5c0bebc68666" containerName="payload-processing" containerID="cri-o://385b7d8c7677acc7eada0ca323ae312f210551c48b63dd2930f5eff4f56de8dc" gracePeriod=25 Nov 03 01:48:55.040944 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:48:55.040906 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-515c8f-78658vzmt5" podUID="111bfc1a-cee1-4a78-99ea-25c769ed7c2e" containerName="payload-pre-processing" containerID="cri-o://ee4e1792eecd889a95c389697017cc55ae6ac2b74e1b803ac4f071ab35c4a3fa" gracePeriod=25 Nov 03 01:49:04.079700 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:04.079660 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-515c8f/simulator-single-main-515c8f-kserve-7488db5694-g8xpd" podUID="64d6d02c-ec5c-4aaf-914f-ccd45c0deecb" containerName="main" containerID="cri-o://a46d0ffa7ea46bbcc295ba6cfd69b7127bb5c92df8ba8c9682f3aa85590bc8e5" gracePeriod=45 Nov 03 01:49:04.115731 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:04.115693 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-single-main-515c8f/simulator-single-main-515c8f-kserve-7488db5694-g8xpd" podUID="64d6d02c-ec5c-4aaf-914f-ccd45c0deecb" containerName="main" probeResult="failure" output="Get \"https://10.134.1.139:8000/ready\": dial tcp 10.134.1.139:8000: connect: connection refused" Nov 03 01:49:04.336635 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:04.336568 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-515c8f/simulator-single-main-515c8f-kserve-7488db5694-g8xpd" Nov 03 01:49:04.360567 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:04.360535 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64d6d02c-ec5c-4aaf-914f-ccd45c0deecb-dshm\" (UniqueName: \"kubernetes.io/empty-dir/64d6d02c-ec5c-4aaf-914f-ccd45c0deecb-dshm\") pod \"64d6d02c-ec5c-4aaf-914f-ccd45c0deecb\" (UID: \"64d6d02c-ec5c-4aaf-914f-ccd45c0deecb\") " Nov 03 01:49:04.362629 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:04.360578 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/64d6d02c-ec5c-4aaf-914f-ccd45c0deecb-tls-certs\" (UniqueName: \"kubernetes.io/secret/64d6d02c-ec5c-4aaf-914f-ccd45c0deecb-tls-certs\") pod \"64d6d02c-ec5c-4aaf-914f-ccd45c0deecb\" (UID: \"64d6d02c-ec5c-4aaf-914f-ccd45c0deecb\") " Nov 03 01:49:04.362629 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:04.360627 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64d6d02c-ec5c-4aaf-914f-ccd45c0deecb-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64d6d02c-ec5c-4aaf-914f-ccd45c0deecb-model-cache\") pod \"64d6d02c-ec5c-4aaf-914f-ccd45c0deecb\" (UID: \"64d6d02c-ec5c-4aaf-914f-ccd45c0deecb\") " Nov 03 01:49:04.362629 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:04.360669 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64d6d02c-ec5c-4aaf-914f-ccd45c0deecb-kube-api-access-qt9gb\" (UniqueName: \"kubernetes.io/projected/64d6d02c-ec5c-4aaf-914f-ccd45c0deecb-kube-api-access-qt9gb\") pod \"64d6d02c-ec5c-4aaf-914f-ccd45c0deecb\" (UID: \"64d6d02c-ec5c-4aaf-914f-ccd45c0deecb\") " Nov 03 01:49:04.362629 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:04.361006 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64d6d02c-ec5c-4aaf-914f-ccd45c0deecb-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64d6d02c-ec5c-4aaf-914f-ccd45c0deecb-tmp-dir\") pod \"64d6d02c-ec5c-4aaf-914f-ccd45c0deecb\" (UID: \"64d6d02c-ec5c-4aaf-914f-ccd45c0deecb\") " Nov 03 01:49:04.362629 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:04.361085 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64d6d02c-ec5c-4aaf-914f-ccd45c0deecb-home\" (UniqueName: \"kubernetes.io/empty-dir/64d6d02c-ec5c-4aaf-914f-ccd45c0deecb-home\") pod \"64d6d02c-ec5c-4aaf-914f-ccd45c0deecb\" (UID: \"64d6d02c-ec5c-4aaf-914f-ccd45c0deecb\") " Nov 03 01:49:04.362629 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:04.361862 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64d6d02c-ec5c-4aaf-914f-ccd45c0deecb-model-cache" pod "64d6d02c-ec5c-4aaf-914f-ccd45c0deecb" (UID: "64d6d02c-ec5c-4aaf-914f-ccd45c0deecb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:49:04.362629 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:04.361924 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64d6d02c-ec5c-4aaf-914f-ccd45c0deecb-tmp-dir" pod "64d6d02c-ec5c-4aaf-914f-ccd45c0deecb" (UID: "64d6d02c-ec5c-4aaf-914f-ccd45c0deecb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:49:04.362629 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:04.362009 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64d6d02c-ec5c-4aaf-914f-ccd45c0deecb-home" pod "64d6d02c-ec5c-4aaf-914f-ccd45c0deecb" (UID: "64d6d02c-ec5c-4aaf-914f-ccd45c0deecb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:49:04.363616 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:04.363585 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64d6d02c-ec5c-4aaf-914f-ccd45c0deecb-dshm" pod "64d6d02c-ec5c-4aaf-914f-ccd45c0deecb" (UID: "64d6d02c-ec5c-4aaf-914f-ccd45c0deecb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:49:04.364071 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:04.364046 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64d6d02c-ec5c-4aaf-914f-ccd45c0deecb-tls-certs" pod "64d6d02c-ec5c-4aaf-914f-ccd45c0deecb" (UID: "64d6d02c-ec5c-4aaf-914f-ccd45c0deecb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:49:04.365434 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:04.365407 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64d6d02c-ec5c-4aaf-914f-ccd45c0deecb-kube-api-access-qt9gb" pod "64d6d02c-ec5c-4aaf-914f-ccd45c0deecb" (UID: "64d6d02c-ec5c-4aaf-914f-ccd45c0deecb"). InnerVolumeSpecName "kube-api-access-qt9gb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:49:04.386046 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:04.386017 4421 generic.go:356] "Generic (PLEG): container finished" podID="64d6d02c-ec5c-4aaf-914f-ccd45c0deecb" containerID="a46d0ffa7ea46bbcc295ba6cfd69b7127bb5c92df8ba8c9682f3aa85590bc8e5" exitCode=0 Nov 03 01:49:04.386179 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:04.386064 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-515c8f/simulator-single-main-515c8f-kserve-7488db5694-g8xpd" Nov 03 01:49:04.386179 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:04.386098 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-515c8f/simulator-single-main-515c8f-kserve-7488db5694-g8xpd" event={"ID":"64d6d02c-ec5c-4aaf-914f-ccd45c0deecb","Type":"ContainerDied","Data":"a46d0ffa7ea46bbcc295ba6cfd69b7127bb5c92df8ba8c9682f3aa85590bc8e5"} Nov 03 01:49:04.386179 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:04.386137 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-515c8f/simulator-single-main-515c8f-kserve-7488db5694-g8xpd" event={"ID":"64d6d02c-ec5c-4aaf-914f-ccd45c0deecb","Type":"ContainerDied","Data":"e0c25acf4c5a67e32d9de5e1a331529adfb28527a8cf3f152d3dc9ef076728c8"} Nov 03 01:49:04.386179 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:04.386159 4421 scope.go:122] "RemoveContainer" containerID="a46d0ffa7ea46bbcc295ba6cfd69b7127bb5c92df8ba8c9682f3aa85590bc8e5" Nov 03 01:49:04.396849 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:04.396826 4421 scope.go:122] "RemoveContainer" containerID="a46d0ffa7ea46bbcc295ba6cfd69b7127bb5c92df8ba8c9682f3aa85590bc8e5" Nov 03 01:49:04.397093 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:49:04.397075 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a46d0ffa7ea46bbcc295ba6cfd69b7127bb5c92df8ba8c9682f3aa85590bc8e5\": container with ID starting with a46d0ffa7ea46bbcc295ba6cfd69b7127bb5c92df8ba8c9682f3aa85590bc8e5 not found: ID does not exist" containerID="a46d0ffa7ea46bbcc295ba6cfd69b7127bb5c92df8ba8c9682f3aa85590bc8e5" Nov 03 01:49:04.397160 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:04.397101 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a46d0ffa7ea46bbcc295ba6cfd69b7127bb5c92df8ba8c9682f3aa85590bc8e5"} err="failed to get container status \"a46d0ffa7ea46bbcc295ba6cfd69b7127bb5c92df8ba8c9682f3aa85590bc8e5\": rpc error: code = NotFound desc = could not find container \"a46d0ffa7ea46bbcc295ba6cfd69b7127bb5c92df8ba8c9682f3aa85590bc8e5\": container with ID starting with a46d0ffa7ea46bbcc295ba6cfd69b7127bb5c92df8ba8c9682f3aa85590bc8e5 not found: ID does not exist" Nov 03 01:49:04.409063 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:04.409034 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-515c8f/simulator-single-main-515c8f-kserve-7488db5694-g8xpd"] Nov 03 01:49:04.413903 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:04.413879 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-515c8f/simulator-single-main-515c8f-kserve-7488db5694-g8xpd"] Nov 03 01:49:04.462987 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:04.462947 4421 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64d6d02c-ec5c-4aaf-914f-ccd45c0deecb-home\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:49:04.462987 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:04.462990 4421 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64d6d02c-ec5c-4aaf-914f-ccd45c0deecb-dshm\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:49:04.463319 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:04.463007 4421 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64d6d02c-ec5c-4aaf-914f-ccd45c0deecb-tls-certs\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:49:04.463319 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:04.463020 4421 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64d6d02c-ec5c-4aaf-914f-ccd45c0deecb-model-cache\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:49:04.463319 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:04.463040 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qt9gb\" (UniqueName: \"kubernetes.io/projected/64d6d02c-ec5c-4aaf-914f-ccd45c0deecb-kube-api-access-qt9gb\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:49:04.463319 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:04.463056 4421 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64d6d02c-ec5c-4aaf-914f-ccd45c0deecb-tmp-dir\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:49:04.969345 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:04.969268 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-515c8f-5bc5b95bcx7mrg" podUID="43919454-2a18-49a8-9582-5c0bebc68666" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:49:04.969570 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:04.969320 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-515c8f-78658vzmt5" podUID="111bfc1a-cee1-4a78-99ea-25c769ed7c2e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:49:05.977382 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:05.977350 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64d6d02c-ec5c-4aaf-914f-ccd45c0deecb" path="/var/lib/kubelet/pods/64d6d02c-ec5c-4aaf-914f-ccd45c0deecb/volumes" Nov 03 01:49:07.024619 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:07.024583 4421 scope.go:122] "RemoveContainer" containerID="7ad169dd1d7c974c53009b56cea55ee9158dfd0c5d75a8f92be67cb39c0b13d4" Nov 03 01:49:14.668487 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:49:14.668397 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 01:49:14.977316 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:14.977188 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-515c8f-78658vzmt5" podUID="111bfc1a-cee1-4a78-99ea-25c769ed7c2e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:49:14.977550 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:14.977348 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-515c8f-5bc5b95bcx7mrg" podUID="43919454-2a18-49a8-9582-5c0bebc68666" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:49:15.294955 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:15.294928 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-515c8f-5bc5b95bcx7mrg" Nov 03 01:49:15.327747 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:15.327716 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-515c8f-78658vzmt5" Nov 03 01:49:15.367532 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:15.367495 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/111bfc1a-cee1-4a78-99ea-25c769ed7c2e-kube-api-access-2jp9c\" (UniqueName: \"kubernetes.io/projected/111bfc1a-cee1-4a78-99ea-25c769ed7c2e-kube-api-access-2jp9c\") pod \"111bfc1a-cee1-4a78-99ea-25c769ed7c2e\" (UID: \"111bfc1a-cee1-4a78-99ea-25c769ed7c2e\") " Nov 03 01:49:15.367715 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:15.367566 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/111bfc1a-cee1-4a78-99ea-25c769ed7c2e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/111bfc1a-cee1-4a78-99ea-25c769ed7c2e-plugins-config-volume\") pod \"111bfc1a-cee1-4a78-99ea-25c769ed7c2e\" (UID: \"111bfc1a-cee1-4a78-99ea-25c769ed7c2e\") " Nov 03 01:49:15.367715 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:15.367675 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43919454-2a18-49a8-9582-5c0bebc68666-kube-api-access-pl55j\" (UniqueName: \"kubernetes.io/projected/43919454-2a18-49a8-9582-5c0bebc68666-kube-api-access-pl55j\") pod \"43919454-2a18-49a8-9582-5c0bebc68666\" (UID: \"43919454-2a18-49a8-9582-5c0bebc68666\") " Nov 03 01:49:15.367809 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:15.367750 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43919454-2a18-49a8-9582-5c0bebc68666-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43919454-2a18-49a8-9582-5c0bebc68666-plugins-config-volume\") pod \"43919454-2a18-49a8-9582-5c0bebc68666\" (UID: \"43919454-2a18-49a8-9582-5c0bebc68666\") " Nov 03 01:49:15.367975 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:15.367945 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/111bfc1a-cee1-4a78-99ea-25c769ed7c2e-plugins-config-volume" pod "111bfc1a-cee1-4a78-99ea-25c769ed7c2e" (UID: "111bfc1a-cee1-4a78-99ea-25c769ed7c2e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:49:15.368141 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:15.368092 4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/111bfc1a-cee1-4a78-99ea-25c769ed7c2e-plugins-config-volume\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:49:15.368228 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:15.368203 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43919454-2a18-49a8-9582-5c0bebc68666-plugins-config-volume" pod "43919454-2a18-49a8-9582-5c0bebc68666" (UID: "43919454-2a18-49a8-9582-5c0bebc68666"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:49:15.369847 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:15.369823 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/111bfc1a-cee1-4a78-99ea-25c769ed7c2e-kube-api-access-2jp9c" pod "111bfc1a-cee1-4a78-99ea-25c769ed7c2e" (UID: "111bfc1a-cee1-4a78-99ea-25c769ed7c2e"). InnerVolumeSpecName "kube-api-access-2jp9c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:49:15.370351 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:15.370330 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43919454-2a18-49a8-9582-5c0bebc68666-kube-api-access-pl55j" pod "43919454-2a18-49a8-9582-5c0bebc68666" (UID: "43919454-2a18-49a8-9582-5c0bebc68666"). InnerVolumeSpecName "kube-api-access-pl55j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:49:15.439592 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:15.439551 4421 generic.go:356] "Generic (PLEG): container finished" podID="43919454-2a18-49a8-9582-5c0bebc68666" containerID="385b7d8c7677acc7eada0ca323ae312f210551c48b63dd2930f5eff4f56de8dc" exitCode=0 Nov 03 01:49:15.439814 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:15.439602 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-515c8f-5bc5b95bcx7mrg" Nov 03 01:49:15.439814 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:15.439646 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-515c8f-5bc5b95bcx7mrg" event={"ID":"43919454-2a18-49a8-9582-5c0bebc68666","Type":"ContainerDied","Data":"385b7d8c7677acc7eada0ca323ae312f210551c48b63dd2930f5eff4f56de8dc"} Nov 03 01:49:15.439814 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:15.439718 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-515c8f-5bc5b95bcx7mrg" event={"ID":"43919454-2a18-49a8-9582-5c0bebc68666","Type":"ContainerDied","Data":"7b7237c5c4733044c958371ad0877eea92f3fa527432103798d5b0948da3f3fb"} Nov 03 01:49:15.439814 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:15.439750 4421 scope.go:122] "RemoveContainer" containerID="385b7d8c7677acc7eada0ca323ae312f210551c48b63dd2930f5eff4f56de8dc" Nov 03 01:49:15.441079 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:15.441058 4421 generic.go:356] "Generic (PLEG): container finished" podID="111bfc1a-cee1-4a78-99ea-25c769ed7c2e" containerID="ee4e1792eecd889a95c389697017cc55ae6ac2b74e1b803ac4f071ab35c4a3fa" exitCode=0 Nov 03 01:49:15.441220 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:15.441087 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-515c8f-78658vzmt5" event={"ID":"111bfc1a-cee1-4a78-99ea-25c769ed7c2e","Type":"ContainerDied","Data":"ee4e1792eecd889a95c389697017cc55ae6ac2b74e1b803ac4f071ab35c4a3fa"} Nov 03 01:49:15.441220 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:15.441101 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-515c8f-78658vzmt5" Nov 03 01:49:15.441220 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:15.441138 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-515c8f-78658vzmt5" event={"ID":"111bfc1a-cee1-4a78-99ea-25c769ed7c2e","Type":"ContainerDied","Data":"1d2c8d84001ddbe6a2f03a0b39f6a99b09b51a7a48bcdf5f887f9c8acc955091"} Nov 03 01:49:15.450404 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:15.450388 4421 scope.go:122] "RemoveContainer" containerID="385b7d8c7677acc7eada0ca323ae312f210551c48b63dd2930f5eff4f56de8dc" Nov 03 01:49:15.450669 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:49:15.450651 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"385b7d8c7677acc7eada0ca323ae312f210551c48b63dd2930f5eff4f56de8dc\": container with ID starting with 385b7d8c7677acc7eada0ca323ae312f210551c48b63dd2930f5eff4f56de8dc not found: ID does not exist" containerID="385b7d8c7677acc7eada0ca323ae312f210551c48b63dd2930f5eff4f56de8dc" Nov 03 01:49:15.450754 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:15.450676 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"385b7d8c7677acc7eada0ca323ae312f210551c48b63dd2930f5eff4f56de8dc"} err="failed to get container status \"385b7d8c7677acc7eada0ca323ae312f210551c48b63dd2930f5eff4f56de8dc\": rpc error: code = NotFound desc = could not find container \"385b7d8c7677acc7eada0ca323ae312f210551c48b63dd2930f5eff4f56de8dc\": container with ID starting with 385b7d8c7677acc7eada0ca323ae312f210551c48b63dd2930f5eff4f56de8dc not found: ID does not exist" Nov 03 01:49:15.450754 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:15.450692 4421 scope.go:122] "RemoveContainer" containerID="ee4e1792eecd889a95c389697017cc55ae6ac2b74e1b803ac4f071ab35c4a3fa" Nov 03 01:49:15.459088 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:15.459061 4421 scope.go:122] "RemoveContainer" containerID="ee4e1792eecd889a95c389697017cc55ae6ac2b74e1b803ac4f071ab35c4a3fa" Nov 03 01:49:15.459393 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:49:15.459360 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee4e1792eecd889a95c389697017cc55ae6ac2b74e1b803ac4f071ab35c4a3fa\": container with ID starting with ee4e1792eecd889a95c389697017cc55ae6ac2b74e1b803ac4f071ab35c4a3fa not found: ID does not exist" containerID="ee4e1792eecd889a95c389697017cc55ae6ac2b74e1b803ac4f071ab35c4a3fa" Nov 03 01:49:15.459518 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:15.459421 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee4e1792eecd889a95c389697017cc55ae6ac2b74e1b803ac4f071ab35c4a3fa"} err="failed to get container status \"ee4e1792eecd889a95c389697017cc55ae6ac2b74e1b803ac4f071ab35c4a3fa\": rpc error: code = NotFound desc = could not find container \"ee4e1792eecd889a95c389697017cc55ae6ac2b74e1b803ac4f071ab35c4a3fa\": container with ID starting with ee4e1792eecd889a95c389697017cc55ae6ac2b74e1b803ac4f071ab35c4a3fa not found: ID does not exist" Nov 03 01:49:15.462267 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:15.462244 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-515c8f-5bc5b95bcx7mrg"] Nov 03 01:49:15.467424 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:15.467402 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-515c8f-5bc5b95bcx7mrg"] Nov 03 01:49:15.468828 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:15.468808 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pl55j\" (UniqueName: \"kubernetes.io/projected/43919454-2a18-49a8-9582-5c0bebc68666-kube-api-access-pl55j\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:49:15.468884 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:15.468830 4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43919454-2a18-49a8-9582-5c0bebc68666-plugins-config-volume\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:49:15.468884 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:15.468840 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jp9c\" (UniqueName: \"kubernetes.io/projected/111bfc1a-cee1-4a78-99ea-25c769ed7c2e-kube-api-access-2jp9c\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:49:15.479741 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:15.479720 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-515c8f-78658vzmt5"] Nov 03 01:49:15.484084 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:15.484060 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-515c8f-78658vzmt5"] Nov 03 01:49:15.978882 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:15.978848 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="111bfc1a-cee1-4a78-99ea-25c769ed7c2e" path="/var/lib/kubelet/pods/111bfc1a-cee1-4a78-99ea-25c769ed7c2e/volumes" Nov 03 01:49:15.979303 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:15.979232 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43919454-2a18-49a8-9582-5c0bebc68666" path="/var/lib/kubelet/pods/43919454-2a18-49a8-9582-5c0bebc68666/volumes" Nov 03 01:49:21.733250 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:21.733208 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-515c8f-5bc5b95bcmfzbh"] Nov 03 01:49:21.733678 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:21.733662 4421 cpu_manager.go:414] "containerMap: removing container" podUID="111bfc1a-cee1-4a78-99ea-25c769ed7c2e" containerName="payload-pre-processing" Nov 03 01:49:21.733733 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:21.733680 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="111bfc1a-cee1-4a78-99ea-25c769ed7c2e" containerName="payload-pre-processing" Nov 03 01:49:21.733733 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:21.733692 4421 cpu_manager.go:414] "containerMap: removing container" podUID="43919454-2a18-49a8-9582-5c0bebc68666" containerName="payload-processing" Nov 03 01:49:21.733733 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:21.733697 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43919454-2a18-49a8-9582-5c0bebc68666" containerName="payload-processing" Nov 03 01:49:21.733733 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:21.733718 4421 cpu_manager.go:414] "containerMap: removing container" podUID="64d6d02c-ec5c-4aaf-914f-ccd45c0deecb" containerName="main" Nov 03 01:49:21.733733 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:21.733723 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64d6d02c-ec5c-4aaf-914f-ccd45c0deecb" containerName="main" Nov 03 01:49:21.733902 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:21.733795 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="111bfc1a-cee1-4a78-99ea-25c769ed7c2e" containerName="payload-pre-processing" Nov 03 01:49:21.733902 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:21.733804 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="43919454-2a18-49a8-9582-5c0bebc68666" containerName="payload-processing" Nov 03 01:49:21.733902 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:21.733813 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="64d6d02c-ec5c-4aaf-914f-ccd45c0deecb" containerName="main" Nov 03 01:49:21.738799 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:21.738778 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-515c8f-5bc5b95bcmfzbh" Nov 03 01:49:21.747798 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:21.747762 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-515c8f-5bc5b95bcmfzbh"] Nov 03 01:49:21.789894 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:21.789820 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-515c8f\"" Nov 03 01:49:21.790141 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:21.789923 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-7c3b72c5-dockercfg-9s2fr\"" Nov 03 01:49:21.832781 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:21.832740 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzgmw\" (UniqueName: \"kubernetes.io/projected/04f23065-f50c-4adb-9a84-6c3cb7798410-kube-api-access-bzgmw\") pod \"payload-processing-e2e-worker-single-main-515c8f-5bc5b95bcmfzbh\" (UID: \"04f23065-f50c-4adb-9a84-6c3cb7798410\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-515c8f-5bc5b95bcmfzbh" Nov 03 01:49:21.832781 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:21.832784 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04f23065-f50c-4adb-9a84-6c3cb7798410-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-515c8f-5bc5b95bcmfzbh\" (UID: \"04f23065-f50c-4adb-9a84-6c3cb7798410\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-515c8f-5bc5b95bcmfzbh" Nov 03 01:49:21.933524 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:21.933472 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzgmw\" (UniqueName: \"kubernetes.io/projected/04f23065-f50c-4adb-9a84-6c3cb7798410-kube-api-access-bzgmw\") pod \"payload-processing-e2e-worker-single-main-515c8f-5bc5b95bcmfzbh\" (UID: \"04f23065-f50c-4adb-9a84-6c3cb7798410\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-515c8f-5bc5b95bcmfzbh" Nov 03 01:49:21.933524 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:21.933529 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04f23065-f50c-4adb-9a84-6c3cb7798410-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-515c8f-5bc5b95bcmfzbh\" (UID: \"04f23065-f50c-4adb-9a84-6c3cb7798410\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-515c8f-5bc5b95bcmfzbh" Nov 03 01:49:21.934365 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:21.934334 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04f23065-f50c-4adb-9a84-6c3cb7798410-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-515c8f-5bc5b95bcmfzbh\" (UID: \"04f23065-f50c-4adb-9a84-6c3cb7798410\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-515c8f-5bc5b95bcmfzbh" Nov 03 01:49:21.941895 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:21.941861 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzgmw\" (UniqueName: \"kubernetes.io/projected/04f23065-f50c-4adb-9a84-6c3cb7798410-kube-api-access-bzgmw\") pod \"payload-processing-e2e-worker-single-main-515c8f-5bc5b95bcmfzbh\" (UID: \"04f23065-f50c-4adb-9a84-6c3cb7798410\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-515c8f-5bc5b95bcmfzbh" Nov 03 01:49:22.054016 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:22.053983 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-515c8f-5bc5b95bcmfzbh" Nov 03 01:49:22.331793 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:22.331758 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-515c8f-5bc5b95bcmfzbh"] Nov 03 01:49:22.334659 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:49:22.334632 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04f23065_f50c_4adb_9a84_6c3cb7798410.slice/crio-dff7dc93d318e2afb45abdad6924f287063ec5f76a517200a9532328fc3de3e8 WatchSource:0}: Error finding container dff7dc93d318e2afb45abdad6924f287063ec5f76a517200a9532328fc3de3e8: Status 404 returned error can't find the container with id dff7dc93d318e2afb45abdad6924f287063ec5f76a517200a9532328fc3de3e8 Nov 03 01:49:22.481146 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:22.481111 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-515c8f-5bc5b95bcmfzbh" event={"ID":"04f23065-f50c-4adb-9a84-6c3cb7798410","Type":"ContainerStarted","Data":"d95f386a18ee9f934031cd71a0f8a93ecfcfc4f7e388f8a8d088cdda21e0751d"} Nov 03 01:49:22.481400 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:22.481172 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-515c8f-5bc5b95bcmfzbh" event={"ID":"04f23065-f50c-4adb-9a84-6c3cb7798410","Type":"ContainerStarted","Data":"dff7dc93d318e2afb45abdad6924f287063ec5f76a517200a9532328fc3de3e8"} Nov 03 01:49:22.481400 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:22.481206 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-515c8f-5bc5b95bcmfzbh" Nov 03 01:49:22.497640 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:22.497592 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-515c8f-5bc5b95bcmfzbh" podStartSLOduration=1.497578433 podStartE2EDuration="1.497578433s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:49:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:49:22.495614353 +0000 UTC m=+4104.666617450" watchObservedRunningTime="2026-11-03 01:49:22.497578433 +0000 UTC m=+4104.668581636" Nov 03 01:49:30.248528 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:30.248490 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-515c8f-kjqq8"] Nov 03 01:49:30.252500 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:30.252473 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-515c8f-kjqq8" Nov 03 01:49:30.257182 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:30.257153 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:49:30.257338 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:30.257213 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-swfcr\"" Nov 03 01:49:30.259798 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:30.259776 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-515c8f-kjqq8"] Nov 03 01:49:30.318451 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:30.318420 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 01:49:30.318621 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:30.318456 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 01:49:30.322932 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:30.322901 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2583247d-a775-4c9b-99e2-0b3014584478-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-515c8f-kjqq8\" (UID: \"2583247d-a775-4c9b-99e2-0b3014584478\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-515c8f-kjqq8" Nov 03 01:49:30.323124 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:30.323061 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-splhf\" (UniqueName: \"kubernetes.io/projected/2583247d-a775-4c9b-99e2-0b3014584478-kube-api-access-splhf\") pod \"maas-api-revoke-keys-e2e-worker-single-main-515c8f-kjqq8\" (UID: \"2583247d-a775-4c9b-99e2-0b3014584478\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-515c8f-kjqq8" Nov 03 01:49:30.323201 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:30.323174 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2583247d-a775-4c9b-99e2-0b3014584478-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-515c8f-kjqq8\" (UID: \"2583247d-a775-4c9b-99e2-0b3014584478\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-515c8f-kjqq8" Nov 03 01:49:30.424128 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:30.424083 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-splhf\" (UniqueName: \"kubernetes.io/projected/2583247d-a775-4c9b-99e2-0b3014584478-kube-api-access-splhf\") pod \"maas-api-revoke-keys-e2e-worker-single-main-515c8f-kjqq8\" (UID: \"2583247d-a775-4c9b-99e2-0b3014584478\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-515c8f-kjqq8" Nov 03 01:49:30.424385 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:30.424193 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2583247d-a775-4c9b-99e2-0b3014584478-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-515c8f-kjqq8\" (UID: \"2583247d-a775-4c9b-99e2-0b3014584478\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-515c8f-kjqq8" Nov 03 01:49:30.424385 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:30.424230 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2583247d-a775-4c9b-99e2-0b3014584478-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-515c8f-kjqq8\" (UID: \"2583247d-a775-4c9b-99e2-0b3014584478\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-515c8f-kjqq8" Nov 03 01:49:30.424892 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:30.424860 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2583247d-a775-4c9b-99e2-0b3014584478-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-515c8f-kjqq8\" (UID: \"2583247d-a775-4c9b-99e2-0b3014584478\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-515c8f-kjqq8" Nov 03 01:49:30.426633 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:30.426614 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2583247d-a775-4c9b-99e2-0b3014584478-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-515c8f-kjqq8\" (UID: \"2583247d-a775-4c9b-99e2-0b3014584478\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-515c8f-kjqq8" Nov 03 01:49:30.431459 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:30.431439 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-splhf\" (UniqueName: \"kubernetes.io/projected/2583247d-a775-4c9b-99e2-0b3014584478-kube-api-access-splhf\") pod \"maas-api-revoke-keys-e2e-worker-single-main-515c8f-kjqq8\" (UID: \"2583247d-a775-4c9b-99e2-0b3014584478\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-515c8f-kjqq8" Nov 03 01:49:30.572494 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:30.572405 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-515c8f-kjqq8" Nov 03 01:49:30.847880 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:30.847852 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-515c8f-kjqq8"] Nov 03 01:49:30.850622 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:49:30.850587 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2583247d_a775_4c9b_99e2_0b3014584478.slice/crio-20fce4a4afa83a5a3bf0cef4338a13aa3afabe28f5bd5b260ce8da6ed161c520 WatchSource:0}: Error finding container 20fce4a4afa83a5a3bf0cef4338a13aa3afabe28f5bd5b260ce8da6ed161c520: Status 404 returned error can't find the container with id 20fce4a4afa83a5a3bf0cef4338a13aa3afabe28f5bd5b260ce8da6ed161c520 Nov 03 01:49:31.527353 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:31.527308 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-515c8f-kjqq8" event={"ID":"2583247d-a775-4c9b-99e2-0b3014584478","Type":"ContainerStarted","Data":"cf7b5bc2b438a13e0f3a081ffd892995a2af9bed004852cf9d28c45bea963c15"} Nov 03 01:49:31.527768 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:31.527374 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-515c8f-kjqq8" event={"ID":"2583247d-a775-4c9b-99e2-0b3014584478","Type":"ContainerStarted","Data":"20fce4a4afa83a5a3bf0cef4338a13aa3afabe28f5bd5b260ce8da6ed161c520"} Nov 03 01:49:31.544424 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:31.544373 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-515c8f-kjqq8" podStartSLOduration=1.5443596529999999 podStartE2EDuration="1.544359653s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:49:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:49:31.542340677 +0000 UTC m=+4113.713343774" watchObservedRunningTime="2026-11-03 01:49:31.544359653 +0000 UTC m=+4113.715362748" Nov 03 01:49:32.533897 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:32.533861 4421 generic.go:356] "Generic (PLEG): container finished" podID="2583247d-a775-4c9b-99e2-0b3014584478" containerID="cf7b5bc2b438a13e0f3a081ffd892995a2af9bed004852cf9d28c45bea963c15" exitCode=0 Nov 03 01:49:32.534344 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:32.533952 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-515c8f-kjqq8" event={"ID":"2583247d-a775-4c9b-99e2-0b3014584478","Type":"ContainerDied","Data":"cf7b5bc2b438a13e0f3a081ffd892995a2af9bed004852cf9d28c45bea963c15"} Nov 03 01:49:33.489234 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:33.489203 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-515c8f-5bc5b95bcmfzbh" Nov 03 01:49:33.682689 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:33.682661 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-515c8f-kjqq8" Nov 03 01:49:33.760781 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:33.760748 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2583247d-a775-4c9b-99e2-0b3014584478-kube-api-access-splhf\" (UniqueName: \"kubernetes.io/projected/2583247d-a775-4c9b-99e2-0b3014584478-kube-api-access-splhf\") pod \"2583247d-a775-4c9b-99e2-0b3014584478\" (UID: \"2583247d-a775-4c9b-99e2-0b3014584478\") " Nov 03 01:49:33.760951 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:33.760813 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2583247d-a775-4c9b-99e2-0b3014584478-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2583247d-a775-4c9b-99e2-0b3014584478-maas-api-xks-ca\") pod \"2583247d-a775-4c9b-99e2-0b3014584478\" (UID: \"2583247d-a775-4c9b-99e2-0b3014584478\") " Nov 03 01:49:33.760951 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:33.760890 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2583247d-a775-4c9b-99e2-0b3014584478-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2583247d-a775-4c9b-99e2-0b3014584478-maas-api-openshift-ca\") pod \"2583247d-a775-4c9b-99e2-0b3014584478\" (UID: \"2583247d-a775-4c9b-99e2-0b3014584478\") " Nov 03 01:49:33.761347 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:33.761319 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2583247d-a775-4c9b-99e2-0b3014584478-maas-api-openshift-ca" pod "2583247d-a775-4c9b-99e2-0b3014584478" (UID: "2583247d-a775-4c9b-99e2-0b3014584478"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:49:33.762997 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:33.762967 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2583247d-a775-4c9b-99e2-0b3014584478-maas-api-xks-ca" pod "2583247d-a775-4c9b-99e2-0b3014584478" (UID: "2583247d-a775-4c9b-99e2-0b3014584478"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:49:33.763067 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:33.763039 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2583247d-a775-4c9b-99e2-0b3014584478-kube-api-access-splhf" pod "2583247d-a775-4c9b-99e2-0b3014584478" (UID: "2583247d-a775-4c9b-99e2-0b3014584478"). InnerVolumeSpecName "kube-api-access-splhf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:49:33.861693 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:33.861656 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-splhf\" (UniqueName: \"kubernetes.io/projected/2583247d-a775-4c9b-99e2-0b3014584478-kube-api-access-splhf\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:49:33.861693 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:33.861690 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2583247d-a775-4c9b-99e2-0b3014584478-maas-api-xks-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:49:33.861917 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:33.861702 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2583247d-a775-4c9b-99e2-0b3014584478-maas-api-openshift-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:49:34.545637 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:34.545604 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-515c8f-kjqq8" Nov 03 01:49:34.545637 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:34.545623 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-515c8f-kjqq8" event={"ID":"2583247d-a775-4c9b-99e2-0b3014584478","Type":"ContainerDied","Data":"20fce4a4afa83a5a3bf0cef4338a13aa3afabe28f5bd5b260ce8da6ed161c520"} Nov 03 01:49:34.545878 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:34.545673 4421 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="20fce4a4afa83a5a3bf0cef4338a13aa3afabe28f5bd5b260ce8da6ed161c520" Nov 03 01:49:40.368821 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:40.368768 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-515c8f-5bc5b95bcmfzbh"] Nov 03 01:49:41.178009 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:41.177958 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-515c8f-kjqq8"] Nov 03 01:49:41.180392 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:41.180360 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-515c8f-kjqq8"] Nov 03 01:49:41.538793 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:41.538757 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-515c8f-openshift-default-67c66d4ffcdz4nw"] Nov 03 01:49:41.539265 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:41.539014 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-515c8f-openshift-default-67c66d4ffcdz4nw" podUID="17cc6835-b616-49f8-8dd6-e2cd0ede70a9" containerName="istio-proxy" containerID="cri-o://dbc8635100eb25d057488716f308494a7e48d73fb6e52172758669fa23eaeed7" gracePeriod=30 Nov 03 01:49:41.977910 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:41.977832 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2583247d-a775-4c9b-99e2-0b3014584478" path="/var/lib/kubelet/pods/2583247d-a775-4c9b-99e2-0b3014584478/volumes" Nov 03 01:49:44.668589 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:49:44.668549 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 01:49:45.370177 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:45.370133 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-515c8f-5bc5b95bcmfzbh" podUID="04f23065-f50c-4adb-9a84-6c3cb7798410" containerName="payload-processing" containerID="cri-o://d95f386a18ee9f934031cd71a0f8a93ecfcfc4f7e388f8a8d088cdda21e0751d" gracePeriod=25 Nov 03 01:49:45.599833 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:45.599795 4421 generic.go:356] "Generic (PLEG): container finished" podID="04f23065-f50c-4adb-9a84-6c3cb7798410" containerID="d95f386a18ee9f934031cd71a0f8a93ecfcfc4f7e388f8a8d088cdda21e0751d" exitCode=0 Nov 03 01:49:45.600090 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:45.599875 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-515c8f-5bc5b95bcmfzbh" event={"ID":"04f23065-f50c-4adb-9a84-6c3cb7798410","Type":"ContainerDied","Data":"d95f386a18ee9f934031cd71a0f8a93ecfcfc4f7e388f8a8d088cdda21e0751d"} Nov 03 01:49:45.628125 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:45.628060 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-515c8f-5bc5b95bcmfzbh" Nov 03 01:49:45.683533 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:45.683498 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04f23065-f50c-4adb-9a84-6c3cb7798410-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04f23065-f50c-4adb-9a84-6c3cb7798410-plugins-config-volume\") pod \"04f23065-f50c-4adb-9a84-6c3cb7798410\" (UID: \"04f23065-f50c-4adb-9a84-6c3cb7798410\") " Nov 03 01:49:45.684005 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:45.683543 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04f23065-f50c-4adb-9a84-6c3cb7798410-kube-api-access-bzgmw\" (UniqueName: \"kubernetes.io/projected/04f23065-f50c-4adb-9a84-6c3cb7798410-kube-api-access-bzgmw\") pod \"04f23065-f50c-4adb-9a84-6c3cb7798410\" (UID: \"04f23065-f50c-4adb-9a84-6c3cb7798410\") " Nov 03 01:49:45.684005 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:45.683836 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04f23065-f50c-4adb-9a84-6c3cb7798410-plugins-config-volume" pod "04f23065-f50c-4adb-9a84-6c3cb7798410" (UID: "04f23065-f50c-4adb-9a84-6c3cb7798410"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:49:45.685803 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:45.685771 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04f23065-f50c-4adb-9a84-6c3cb7798410-kube-api-access-bzgmw" pod "04f23065-f50c-4adb-9a84-6c3cb7798410" (UID: "04f23065-f50c-4adb-9a84-6c3cb7798410"). InnerVolumeSpecName "kube-api-access-bzgmw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:49:45.785259 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:45.785216 4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04f23065-f50c-4adb-9a84-6c3cb7798410-plugins-config-volume\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:49:45.785259 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:45.785252 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzgmw\" (UniqueName: \"kubernetes.io/projected/04f23065-f50c-4adb-9a84-6c3cb7798410-kube-api-access-bzgmw\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:49:46.606157 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:46.606122 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-515c8f-5bc5b95bcmfzbh" Nov 03 01:49:46.606364 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:46.606118 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-515c8f-5bc5b95bcmfzbh" event={"ID":"04f23065-f50c-4adb-9a84-6c3cb7798410","Type":"ContainerDied","Data":"dff7dc93d318e2afb45abdad6924f287063ec5f76a517200a9532328fc3de3e8"} Nov 03 01:49:46.606364 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:46.606259 4421 scope.go:122] "RemoveContainer" containerID="d95f386a18ee9f934031cd71a0f8a93ecfcfc4f7e388f8a8d088cdda21e0751d" Nov 03 01:49:46.632772 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:46.632736 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-515c8f-5bc5b95bcmfzbh"] Nov 03 01:49:46.635926 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:46.635896 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-515c8f-5bc5b95bcmfzbh"] Nov 03 01:49:46.798112 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:46.798086 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-515c8f-openshift-default-67c66d4ffcdz4nw" Nov 03 01:49:46.897683 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:46.897599 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17cc6835-b616-49f8-8dd6-e2cd0ede70a9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17cc6835-b616-49f8-8dd6-e2cd0ede70a9-istio-envoy\") pod \"17cc6835-b616-49f8-8dd6-e2cd0ede70a9\" (UID: \"17cc6835-b616-49f8-8dd6-e2cd0ede70a9\") " Nov 03 01:49:46.897683 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:46.897643 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17cc6835-b616-49f8-8dd6-e2cd0ede70a9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17cc6835-b616-49f8-8dd6-e2cd0ede70a9-credential-socket\") pod \"17cc6835-b616-49f8-8dd6-e2cd0ede70a9\" (UID: \"17cc6835-b616-49f8-8dd6-e2cd0ede70a9\") " Nov 03 01:49:46.897683 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:46.897674 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17cc6835-b616-49f8-8dd6-e2cd0ede70a9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17cc6835-b616-49f8-8dd6-e2cd0ede70a9-workload-socket\") pod \"17cc6835-b616-49f8-8dd6-e2cd0ede70a9\" (UID: \"17cc6835-b616-49f8-8dd6-e2cd0ede70a9\") " Nov 03 01:49:46.898019 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:46.897725 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17cc6835-b616-49f8-8dd6-e2cd0ede70a9-istio-token\" (UniqueName: \"kubernetes.io/projected/17cc6835-b616-49f8-8dd6-e2cd0ede70a9-istio-token\") pod \"17cc6835-b616-49f8-8dd6-e2cd0ede70a9\" (UID: \"17cc6835-b616-49f8-8dd6-e2cd0ede70a9\") " Nov 03 01:49:46.898019 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:46.897768 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17cc6835-b616-49f8-8dd6-e2cd0ede70a9-kube-api-access-qf6d8\" (UniqueName: \"kubernetes.io/projected/17cc6835-b616-49f8-8dd6-e2cd0ede70a9-kube-api-access-qf6d8\") pod \"17cc6835-b616-49f8-8dd6-e2cd0ede70a9\" (UID: \"17cc6835-b616-49f8-8dd6-e2cd0ede70a9\") " Nov 03 01:49:46.898019 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:46.897808 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17cc6835-b616-49f8-8dd6-e2cd0ede70a9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17cc6835-b616-49f8-8dd6-e2cd0ede70a9-workload-certs\") pod \"17cc6835-b616-49f8-8dd6-e2cd0ede70a9\" (UID: \"17cc6835-b616-49f8-8dd6-e2cd0ede70a9\") " Nov 03 01:49:46.898019 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:46.897887 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/17cc6835-b616-49f8-8dd6-e2cd0ede70a9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17cc6835-b616-49f8-8dd6-e2cd0ede70a9-istio-podinfo\") pod \"17cc6835-b616-49f8-8dd6-e2cd0ede70a9\" (UID: \"17cc6835-b616-49f8-8dd6-e2cd0ede70a9\") " Nov 03 01:49:46.898019 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:46.897925 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17cc6835-b616-49f8-8dd6-e2cd0ede70a9-workload-socket" pod "17cc6835-b616-49f8-8dd6-e2cd0ede70a9" (UID: "17cc6835-b616-49f8-8dd6-e2cd0ede70a9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:49:46.898019 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:46.897937 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17cc6835-b616-49f8-8dd6-e2cd0ede70a9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17cc6835-b616-49f8-8dd6-e2cd0ede70a9-istio-data\") pod \"17cc6835-b616-49f8-8dd6-e2cd0ede70a9\" (UID: \"17cc6835-b616-49f8-8dd6-e2cd0ede70a9\") " Nov 03 01:49:46.898019 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:46.897992 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17cc6835-b616-49f8-8dd6-e2cd0ede70a9-credential-socket" pod "17cc6835-b616-49f8-8dd6-e2cd0ede70a9" (UID: "17cc6835-b616-49f8-8dd6-e2cd0ede70a9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:49:46.898570 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:46.898022 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17cc6835-b616-49f8-8dd6-e2cd0ede70a9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17cc6835-b616-49f8-8dd6-e2cd0ede70a9-istiod-ca-cert\") pod \"17cc6835-b616-49f8-8dd6-e2cd0ede70a9\" (UID: \"17cc6835-b616-49f8-8dd6-e2cd0ede70a9\") " Nov 03 01:49:46.898570 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:46.898159 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17cc6835-b616-49f8-8dd6-e2cd0ede70a9-istio-data" pod "17cc6835-b616-49f8-8dd6-e2cd0ede70a9" (UID: "17cc6835-b616-49f8-8dd6-e2cd0ede70a9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:49:46.898570 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:46.898180 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17cc6835-b616-49f8-8dd6-e2cd0ede70a9-workload-certs" pod "17cc6835-b616-49f8-8dd6-e2cd0ede70a9" (UID: "17cc6835-b616-49f8-8dd6-e2cd0ede70a9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:49:46.898570 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:46.898429 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17cc6835-b616-49f8-8dd6-e2cd0ede70a9-istiod-ca-cert" pod "17cc6835-b616-49f8-8dd6-e2cd0ede70a9" (UID: "17cc6835-b616-49f8-8dd6-e2cd0ede70a9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:49:46.898570 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:46.898440 4421 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17cc6835-b616-49f8-8dd6-e2cd0ede70a9-istio-data\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:49:46.898570 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:46.898468 4421 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17cc6835-b616-49f8-8dd6-e2cd0ede70a9-credential-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:49:46.898570 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:46.898485 4421 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17cc6835-b616-49f8-8dd6-e2cd0ede70a9-workload-socket\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:49:46.898570 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:46.898526 4421 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17cc6835-b616-49f8-8dd6-e2cd0ede70a9-workload-certs\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:49:46.899962 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:46.899936 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17cc6835-b616-49f8-8dd6-e2cd0ede70a9-istio-envoy" pod "17cc6835-b616-49f8-8dd6-e2cd0ede70a9" (UID: "17cc6835-b616-49f8-8dd6-e2cd0ede70a9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:49:46.900070 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:46.900044 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/17cc6835-b616-49f8-8dd6-e2cd0ede70a9-istio-podinfo" pod "17cc6835-b616-49f8-8dd6-e2cd0ede70a9" (UID: "17cc6835-b616-49f8-8dd6-e2cd0ede70a9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:49:46.900150 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:46.900131 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17cc6835-b616-49f8-8dd6-e2cd0ede70a9-kube-api-access-qf6d8" pod "17cc6835-b616-49f8-8dd6-e2cd0ede70a9" (UID: "17cc6835-b616-49f8-8dd6-e2cd0ede70a9"). InnerVolumeSpecName "kube-api-access-qf6d8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:49:46.900195 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:46.900161 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17cc6835-b616-49f8-8dd6-e2cd0ede70a9-istio-token" pod "17cc6835-b616-49f8-8dd6-e2cd0ede70a9" (UID: "17cc6835-b616-49f8-8dd6-e2cd0ede70a9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:49:46.999498 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:46.999455 4421 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17cc6835-b616-49f8-8dd6-e2cd0ede70a9-istio-podinfo\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:49:46.999498 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:46.999493 4421 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17cc6835-b616-49f8-8dd6-e2cd0ede70a9-istiod-ca-cert\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:49:46.999498 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:46.999506 4421 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17cc6835-b616-49f8-8dd6-e2cd0ede70a9-istio-envoy\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:49:46.999759 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:46.999514 4421 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17cc6835-b616-49f8-8dd6-e2cd0ede70a9-istio-token\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:49:46.999759 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:46.999524 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qf6d8\" (UniqueName: \"kubernetes.io/projected/17cc6835-b616-49f8-8dd6-e2cd0ede70a9-kube-api-access-qf6d8\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:49:47.612743 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:47.612706 4421 generic.go:356] "Generic (PLEG): container finished" podID="17cc6835-b616-49f8-8dd6-e2cd0ede70a9" containerID="dbc8635100eb25d057488716f308494a7e48d73fb6e52172758669fa23eaeed7" exitCode=0 Nov 03 01:49:47.612935 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:47.612763 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-515c8f-openshift-default-67c66d4ffcdz4nw" Nov 03 01:49:47.612935 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:47.612765 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-515c8f-openshift-default-67c66d4ffcdz4nw" event={"ID":"17cc6835-b616-49f8-8dd6-e2cd0ede70a9","Type":"ContainerDied","Data":"dbc8635100eb25d057488716f308494a7e48d73fb6e52172758669fa23eaeed7"} Nov 03 01:49:47.612935 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:47.612820 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-515c8f-openshift-default-67c66d4ffcdz4nw" event={"ID":"17cc6835-b616-49f8-8dd6-e2cd0ede70a9","Type":"ContainerDied","Data":"5ea56e18e0091ad54ff16278893b0c809cbb9a657aa537753708bd42e1b64b8c"} Nov 03 01:49:47.612935 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:47.612853 4421 scope.go:122] "RemoveContainer" containerID="dbc8635100eb25d057488716f308494a7e48d73fb6e52172758669fa23eaeed7" Nov 03 01:49:47.622402 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:47.622377 4421 scope.go:122] "RemoveContainer" containerID="dbc8635100eb25d057488716f308494a7e48d73fb6e52172758669fa23eaeed7" Nov 03 01:49:47.622733 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:49:47.622707 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbc8635100eb25d057488716f308494a7e48d73fb6e52172758669fa23eaeed7\": container with ID starting with dbc8635100eb25d057488716f308494a7e48d73fb6e52172758669fa23eaeed7 not found: ID does not exist" containerID="dbc8635100eb25d057488716f308494a7e48d73fb6e52172758669fa23eaeed7" Nov 03 01:49:47.622787 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:47.622743 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbc8635100eb25d057488716f308494a7e48d73fb6e52172758669fa23eaeed7"} err="failed to get container status \"dbc8635100eb25d057488716f308494a7e48d73fb6e52172758669fa23eaeed7\": rpc error: code = NotFound desc = could not find container \"dbc8635100eb25d057488716f308494a7e48d73fb6e52172758669fa23eaeed7\": container with ID starting with dbc8635100eb25d057488716f308494a7e48d73fb6e52172758669fa23eaeed7 not found: ID does not exist" Nov 03 01:49:47.635119 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:47.635074 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-515c8f-openshift-default-67c66d4ffcdz4nw"] Nov 03 01:49:47.640712 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:47.640689 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-515c8f-openshift-default-67c66d4ffcdz4nw"] Nov 03 01:49:47.977618 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:47.977533 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04f23065-f50c-4adb-9a84-6c3cb7798410" path="/var/lib/kubelet/pods/04f23065-f50c-4adb-9a84-6c3cb7798410/volumes" Nov 03 01:49:47.978029 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:49:47.977976 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17cc6835-b616-49f8-8dd6-e2cd0ede70a9" path="/var/lib/kubelet/pods/17cc6835-b616-49f8-8dd6-e2cd0ede70a9/volumes" Nov 03 01:50:01.812156 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:01.812118 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-077b6796-kserve-58584d646f-45vkg"] Nov 03 01:50:01.812615 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:01.812520 4421 cpu_manager.go:414] "containerMap: removing container" podUID="04f23065-f50c-4adb-9a84-6c3cb7798410" containerName="payload-processing" Nov 03 01:50:01.812615 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:01.812530 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04f23065-f50c-4adb-9a84-6c3cb7798410" containerName="payload-processing" Nov 03 01:50:01.812615 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:01.812542 4421 cpu_manager.go:414] "containerMap: removing container" podUID="2583247d-a775-4c9b-99e2-0b3014584478" containerName="revoke-keys" Nov 03 01:50:01.812615 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:01.812547 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2583247d-a775-4c9b-99e2-0b3014584478" containerName="revoke-keys" Nov 03 01:50:01.812615 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:01.812558 4421 cpu_manager.go:414] "containerMap: removing container" podUID="17cc6835-b616-49f8-8dd6-e2cd0ede70a9" containerName="istio-proxy" Nov 03 01:50:01.812615 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:01.812563 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17cc6835-b616-49f8-8dd6-e2cd0ede70a9" containerName="istio-proxy" Nov 03 01:50:01.812856 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:01.812635 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="04f23065-f50c-4adb-9a84-6c3cb7798410" containerName="payload-processing" Nov 03 01:50:01.812856 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:01.812641 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="2583247d-a775-4c9b-99e2-0b3014584478" containerName="revoke-keys" Nov 03 01:50:01.812856 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:01.812649 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="17cc6835-b616-49f8-8dd6-e2cd0ede70a9" containerName="istio-proxy" Nov 03 01:50:01.817610 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:01.817585 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-077b6796-kserve-58584d646f-45vkg" Nov 03 01:50:01.825430 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:01.825393 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-077b6796-kserve-self-signed-certs\"" Nov 03 01:50:01.831469 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:01.831435 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-077b6796-kserve-58584d646f-45vkg"] Nov 03 01:50:01.876033 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:01.875991 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be23a06d-0c04-4004-b6ec-67b5f4f7af19-dshm\") pod \"e2e-same-id-b-077b6796-kserve-58584d646f-45vkg\" (UID: \"be23a06d-0c04-4004-b6ec-67b5f4f7af19\") " pod="llm/e2e-same-id-b-077b6796-kserve-58584d646f-45vkg" Nov 03 01:50:01.876216 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:01.876046 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be23a06d-0c04-4004-b6ec-67b5f4f7af19-tls-certs\") pod \"e2e-same-id-b-077b6796-kserve-58584d646f-45vkg\" (UID: \"be23a06d-0c04-4004-b6ec-67b5f4f7af19\") " pod="llm/e2e-same-id-b-077b6796-kserve-58584d646f-45vkg" Nov 03 01:50:01.876216 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:01.876139 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqqh6\" (UniqueName: \"kubernetes.io/projected/be23a06d-0c04-4004-b6ec-67b5f4f7af19-kube-api-access-lqqh6\") pod \"e2e-same-id-b-077b6796-kserve-58584d646f-45vkg\" (UID: \"be23a06d-0c04-4004-b6ec-67b5f4f7af19\") " pod="llm/e2e-same-id-b-077b6796-kserve-58584d646f-45vkg" Nov 03 01:50:01.876216 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:01.876174 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be23a06d-0c04-4004-b6ec-67b5f4f7af19-tmp-dir\") pod \"e2e-same-id-b-077b6796-kserve-58584d646f-45vkg\" (UID: \"be23a06d-0c04-4004-b6ec-67b5f4f7af19\") " pod="llm/e2e-same-id-b-077b6796-kserve-58584d646f-45vkg" Nov 03 01:50:01.876385 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:01.876266 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be23a06d-0c04-4004-b6ec-67b5f4f7af19-home\") pod \"e2e-same-id-b-077b6796-kserve-58584d646f-45vkg\" (UID: \"be23a06d-0c04-4004-b6ec-67b5f4f7af19\") " pod="llm/e2e-same-id-b-077b6796-kserve-58584d646f-45vkg" Nov 03 01:50:01.876385 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:01.876327 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be23a06d-0c04-4004-b6ec-67b5f4f7af19-model-cache\") pod \"e2e-same-id-b-077b6796-kserve-58584d646f-45vkg\" (UID: \"be23a06d-0c04-4004-b6ec-67b5f4f7af19\") " pod="llm/e2e-same-id-b-077b6796-kserve-58584d646f-45vkg" Nov 03 01:50:01.976804 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:01.976770 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be23a06d-0c04-4004-b6ec-67b5f4f7af19-home\") pod \"e2e-same-id-b-077b6796-kserve-58584d646f-45vkg\" (UID: \"be23a06d-0c04-4004-b6ec-67b5f4f7af19\") " pod="llm/e2e-same-id-b-077b6796-kserve-58584d646f-45vkg" Nov 03 01:50:01.977039 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:01.976812 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be23a06d-0c04-4004-b6ec-67b5f4f7af19-model-cache\") pod \"e2e-same-id-b-077b6796-kserve-58584d646f-45vkg\" (UID: \"be23a06d-0c04-4004-b6ec-67b5f4f7af19\") " pod="llm/e2e-same-id-b-077b6796-kserve-58584d646f-45vkg" Nov 03 01:50:01.977039 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:01.976860 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be23a06d-0c04-4004-b6ec-67b5f4f7af19-dshm\") pod \"e2e-same-id-b-077b6796-kserve-58584d646f-45vkg\" (UID: \"be23a06d-0c04-4004-b6ec-67b5f4f7af19\") " pod="llm/e2e-same-id-b-077b6796-kserve-58584d646f-45vkg" Nov 03 01:50:01.977039 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:01.976913 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be23a06d-0c04-4004-b6ec-67b5f4f7af19-tls-certs\") pod \"e2e-same-id-b-077b6796-kserve-58584d646f-45vkg\" (UID: \"be23a06d-0c04-4004-b6ec-67b5f4f7af19\") " pod="llm/e2e-same-id-b-077b6796-kserve-58584d646f-45vkg" Nov 03 01:50:01.977039 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:01.976952 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqqh6\" (UniqueName: \"kubernetes.io/projected/be23a06d-0c04-4004-b6ec-67b5f4f7af19-kube-api-access-lqqh6\") pod \"e2e-same-id-b-077b6796-kserve-58584d646f-45vkg\" (UID: \"be23a06d-0c04-4004-b6ec-67b5f4f7af19\") " pod="llm/e2e-same-id-b-077b6796-kserve-58584d646f-45vkg" Nov 03 01:50:01.977039 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:01.976986 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be23a06d-0c04-4004-b6ec-67b5f4f7af19-tmp-dir\") pod \"e2e-same-id-b-077b6796-kserve-58584d646f-45vkg\" (UID: \"be23a06d-0c04-4004-b6ec-67b5f4f7af19\") " pod="llm/e2e-same-id-b-077b6796-kserve-58584d646f-45vkg" Nov 03 01:50:01.977353 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:01.977223 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be23a06d-0c04-4004-b6ec-67b5f4f7af19-home\") pod \"e2e-same-id-b-077b6796-kserve-58584d646f-45vkg\" (UID: \"be23a06d-0c04-4004-b6ec-67b5f4f7af19\") " pod="llm/e2e-same-id-b-077b6796-kserve-58584d646f-45vkg" Nov 03 01:50:01.977826 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:01.977800 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be23a06d-0c04-4004-b6ec-67b5f4f7af19-model-cache\") pod \"e2e-same-id-b-077b6796-kserve-58584d646f-45vkg\" (UID: \"be23a06d-0c04-4004-b6ec-67b5f4f7af19\") " pod="llm/e2e-same-id-b-077b6796-kserve-58584d646f-45vkg" Nov 03 01:50:01.978410 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:01.978157 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be23a06d-0c04-4004-b6ec-67b5f4f7af19-tmp-dir\") pod \"e2e-same-id-b-077b6796-kserve-58584d646f-45vkg\" (UID: \"be23a06d-0c04-4004-b6ec-67b5f4f7af19\") " pod="llm/e2e-same-id-b-077b6796-kserve-58584d646f-45vkg" Nov 03 01:50:01.983101 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:01.983069 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be23a06d-0c04-4004-b6ec-67b5f4f7af19-dshm\") pod \"e2e-same-id-b-077b6796-kserve-58584d646f-45vkg\" (UID: \"be23a06d-0c04-4004-b6ec-67b5f4f7af19\") " pod="llm/e2e-same-id-b-077b6796-kserve-58584d646f-45vkg" Nov 03 01:50:01.986964 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:01.986932 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqqh6\" (UniqueName: \"kubernetes.io/projected/be23a06d-0c04-4004-b6ec-67b5f4f7af19-kube-api-access-lqqh6\") pod \"e2e-same-id-b-077b6796-kserve-58584d646f-45vkg\" (UID: \"be23a06d-0c04-4004-b6ec-67b5f4f7af19\") " pod="llm/e2e-same-id-b-077b6796-kserve-58584d646f-45vkg" Nov 03 01:50:01.990878 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:01.990847 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be23a06d-0c04-4004-b6ec-67b5f4f7af19-tls-certs\") pod \"e2e-same-id-b-077b6796-kserve-58584d646f-45vkg\" (UID: \"be23a06d-0c04-4004-b6ec-67b5f4f7af19\") " pod="llm/e2e-same-id-b-077b6796-kserve-58584d646f-45vkg" Nov 03 01:50:02.135784 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:02.135684 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-077b6796-kserve-58584d646f-45vkg" Nov 03 01:50:02.410298 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:02.410209 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-077b6796-kserve-58584d646f-45vkg"] Nov 03 01:50:02.412847 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:50:02.412813 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbe23a06d_0c04_4004_b6ec_67b5f4f7af19.slice/crio-ccb14f49b515717ec3996028d4bc869debaa6ce74def1e2dffbce8392710fc83 WatchSource:0}: Error finding container ccb14f49b515717ec3996028d4bc869debaa6ce74def1e2dffbce8392710fc83: Status 404 returned error can't find the container with id ccb14f49b515717ec3996028d4bc869debaa6ce74def1e2dffbce8392710fc83 Nov 03 01:50:02.708673 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:02.708587 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-077b6796-kserve-58584d646f-45vkg" event={"ID":"be23a06d-0c04-4004-b6ec-67b5f4f7af19","Type":"ContainerStarted","Data":"b2e674cdf582895b78524b506871bbb050feb8115cefb0154b93b68c785fe615"} Nov 03 01:50:02.708673 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:02.708630 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-077b6796-kserve-58584d646f-45vkg" event={"ID":"be23a06d-0c04-4004-b6ec-67b5f4f7af19","Type":"ContainerStarted","Data":"ccb14f49b515717ec3996028d4bc869debaa6ce74def1e2dffbce8392710fc83"} Nov 03 01:50:02.725668 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:02.725619 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-b-077b6796-kserve-58584d646f-45vkg" podStartSLOduration=1.725605894 podStartE2EDuration="1.725605894s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:50:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:50:02.72458399 +0000 UTC m=+4144.895587083" watchObservedRunningTime="2026-11-03 01:50:02.725605894 +0000 UTC m=+4144.896608984" Nov 03 01:50:03.136047 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:03.136003 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-077b6796-kserve-58584d646f-45vkg" Nov 03 01:50:07.048067 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:07.048033 4421 scope.go:122] "RemoveContainer" containerID="02775679b7026bdbe1cd47d9fb5106c6c5233da4ed159ab3215b092a841a762c" Nov 03 01:50:12.136427 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:12.136392 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-077b6796-kserve-58584d646f-45vkg" Nov 03 01:50:12.149241 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:12.149210 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-077b6796-kserve-58584d646f-45vkg" Nov 03 01:50:12.819832 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:12.819788 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-077b6796-kserve-58584d646f-45vkg" Nov 03 01:50:14.669266 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:50:14.669227 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 01:50:19.117165 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:19.117124 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-wcr2t"] Nov 03 01:50:19.122250 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:19.122224 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-wcr2t" Nov 03 01:50:19.126436 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:19.126413 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:50:19.127635 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:19.127604 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-wcr2t"] Nov 03 01:50:19.127772 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:19.127667 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-swfcr\"" Nov 03 01:50:19.127772 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:19.127611 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 01:50:19.127772 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:19.127762 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 01:50:19.196772 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:19.196724 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a6caccc-771d-442b-b010-6fe164cc4c71-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-wcr2t\" (UID: \"9a6caccc-771d-442b-b010-6fe164cc4c71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-wcr2t" Nov 03 01:50:19.196772 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:19.196775 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a6caccc-771d-442b-b010-6fe164cc4c71-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-wcr2t\" (UID: \"9a6caccc-771d-442b-b010-6fe164cc4c71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-wcr2t" Nov 03 01:50:19.197056 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:19.196957 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfxwn\" (UniqueName: \"kubernetes.io/projected/9a6caccc-771d-442b-b010-6fe164cc4c71-kube-api-access-mfxwn\") pod \"maas-api-revoke-sub-452b6d7b-wcr2t\" (UID: \"9a6caccc-771d-442b-b010-6fe164cc4c71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-wcr2t" Nov 03 01:50:19.298447 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:19.298402 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a6caccc-771d-442b-b010-6fe164cc4c71-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-wcr2t\" (UID: \"9a6caccc-771d-442b-b010-6fe164cc4c71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-wcr2t" Nov 03 01:50:19.298675 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:19.298459 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a6caccc-771d-442b-b010-6fe164cc4c71-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-wcr2t\" (UID: \"9a6caccc-771d-442b-b010-6fe164cc4c71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-wcr2t" Nov 03 01:50:19.298675 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:19.298565 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfxwn\" (UniqueName: \"kubernetes.io/projected/9a6caccc-771d-442b-b010-6fe164cc4c71-kube-api-access-mfxwn\") pod \"maas-api-revoke-sub-452b6d7b-wcr2t\" (UID: \"9a6caccc-771d-442b-b010-6fe164cc4c71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-wcr2t" Nov 03 01:50:19.299166 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:19.299133 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a6caccc-771d-442b-b010-6fe164cc4c71-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-wcr2t\" (UID: \"9a6caccc-771d-442b-b010-6fe164cc4c71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-wcr2t" Nov 03 01:50:19.301060 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:19.301030 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a6caccc-771d-442b-b010-6fe164cc4c71-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-wcr2t\" (UID: \"9a6caccc-771d-442b-b010-6fe164cc4c71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-wcr2t" Nov 03 01:50:19.306605 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:19.306569 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfxwn\" (UniqueName: \"kubernetes.io/projected/9a6caccc-771d-442b-b010-6fe164cc4c71-kube-api-access-mfxwn\") pod \"maas-api-revoke-sub-452b6d7b-wcr2t\" (UID: \"9a6caccc-771d-442b-b010-6fe164cc4c71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-wcr2t" Nov 03 01:50:19.440187 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:19.440101 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-wcr2t" Nov 03 01:50:19.713401 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:19.713356 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-wcr2t"] Nov 03 01:50:19.715716 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:50:19.715680 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a6caccc_771d_442b_b010_6fe164cc4c71.slice/crio-8e19ebec6166fbd2281d4f6f2efd540b5dc15007eb0e5b296e4ad2f7ae7a6443 WatchSource:0}: Error finding container 8e19ebec6166fbd2281d4f6f2efd540b5dc15007eb0e5b296e4ad2f7ae7a6443: Status 404 returned error can't find the container with id 8e19ebec6166fbd2281d4f6f2efd540b5dc15007eb0e5b296e4ad2f7ae7a6443 Nov 03 01:50:19.844385 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:19.844346 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-wcr2t" event={"ID":"9a6caccc-771d-442b-b010-6fe164cc4c71","Type":"ContainerStarted","Data":"8e19ebec6166fbd2281d4f6f2efd540b5dc15007eb0e5b296e4ad2f7ae7a6443"} Nov 03 01:50:20.853049 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:20.853011 4421 generic.go:356] "Generic (PLEG): container finished" podID="9a6caccc-771d-442b-b010-6fe164cc4c71" containerID="32b7b6b8272bd4c766f25ff7ebe97e9e434753850d807dc1d8f58f8daedb95c9" exitCode=0 Nov 03 01:50:20.853602 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:20.853082 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-wcr2t" event={"ID":"9a6caccc-771d-442b-b010-6fe164cc4c71","Type":"ContainerDied","Data":"32b7b6b8272bd4c766f25ff7ebe97e9e434753850d807dc1d8f58f8daedb95c9"} Nov 03 01:50:22.289423 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:22.289395 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-wcr2t" Nov 03 01:50:22.388129 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:22.388095 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a6caccc-771d-442b-b010-6fe164cc4c71-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a6caccc-771d-442b-b010-6fe164cc4c71-maas-api-openshift-ca\") pod \"9a6caccc-771d-442b-b010-6fe164cc4c71\" (UID: \"9a6caccc-771d-442b-b010-6fe164cc4c71\") " Nov 03 01:50:22.388364 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:22.388264 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a6caccc-771d-442b-b010-6fe164cc4c71-kube-api-access-mfxwn\" (UniqueName: \"kubernetes.io/projected/9a6caccc-771d-442b-b010-6fe164cc4c71-kube-api-access-mfxwn\") pod \"9a6caccc-771d-442b-b010-6fe164cc4c71\" (UID: \"9a6caccc-771d-442b-b010-6fe164cc4c71\") " Nov 03 01:50:22.388364 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:22.388331 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9a6caccc-771d-442b-b010-6fe164cc4c71-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a6caccc-771d-442b-b010-6fe164cc4c71-maas-api-xks-ca\") pod \"9a6caccc-771d-442b-b010-6fe164cc4c71\" (UID: \"9a6caccc-771d-442b-b010-6fe164cc4c71\") " Nov 03 01:50:22.388595 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:22.388562 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a6caccc-771d-442b-b010-6fe164cc4c71-maas-api-openshift-ca" pod "9a6caccc-771d-442b-b010-6fe164cc4c71" (UID: "9a6caccc-771d-442b-b010-6fe164cc4c71"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:50:22.388749 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:22.388723 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a6caccc-771d-442b-b010-6fe164cc4c71-maas-api-openshift-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:50:22.390714 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:22.390684 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a6caccc-771d-442b-b010-6fe164cc4c71-maas-api-xks-ca" pod "9a6caccc-771d-442b-b010-6fe164cc4c71" (UID: "9a6caccc-771d-442b-b010-6fe164cc4c71"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:50:22.391174 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:22.391154 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a6caccc-771d-442b-b010-6fe164cc4c71-kube-api-access-mfxwn" pod "9a6caccc-771d-442b-b010-6fe164cc4c71" (UID: "9a6caccc-771d-442b-b010-6fe164cc4c71"). InnerVolumeSpecName "kube-api-access-mfxwn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:50:22.489853 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:22.489741 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mfxwn\" (UniqueName: \"kubernetes.io/projected/9a6caccc-771d-442b-b010-6fe164cc4c71-kube-api-access-mfxwn\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:50:22.489853 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:22.489793 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a6caccc-771d-442b-b010-6fe164cc4c71-maas-api-xks-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:50:22.887913 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:22.887885 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-wcr2t" Nov 03 01:50:22.888112 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:22.887883 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-wcr2t" event={"ID":"9a6caccc-771d-442b-b010-6fe164cc4c71","Type":"ContainerDied","Data":"8e19ebec6166fbd2281d4f6f2efd540b5dc15007eb0e5b296e4ad2f7ae7a6443"} Nov 03 01:50:22.888112 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:22.888008 4421 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8e19ebec6166fbd2281d4f6f2efd540b5dc15007eb0e5b296e4ad2f7ae7a6443" Nov 03 01:50:31.461548 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:31.461505 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-077b6796-kserve-58584d646f-45vkg"] Nov 03 01:50:34.864649 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:34.864616 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-9hv84"] Nov 03 01:50:34.865074 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:34.865057 4421 cpu_manager.go:414] "containerMap: removing container" podUID="9a6caccc-771d-442b-b010-6fe164cc4c71" containerName="revoke-keys" Nov 03 01:50:34.865128 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:34.865075 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a6caccc-771d-442b-b010-6fe164cc4c71" containerName="revoke-keys" Nov 03 01:50:34.865170 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:34.865164 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a6caccc-771d-442b-b010-6fe164cc4c71" containerName="revoke-keys" Nov 03 01:50:34.868702 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:34.868685 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-9hv84" Nov 03 01:50:34.873915 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:34.873381 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 01:50:34.874701 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:34.874668 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:50:34.874890 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:34.874716 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 01:50:34.874890 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:34.874769 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-swfcr\"" Nov 03 01:50:34.877396 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:34.877380 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-9hv84"] Nov 03 01:50:34.952885 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:34.952846 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgs86\" (UniqueName: \"kubernetes.io/projected/80170d86-c913-4510-a93a-fecd93986897-kube-api-access-bgs86\") pod \"maas-api-revoke-sub-3e31a506-9hv84\" (UID: \"80170d86-c913-4510-a93a-fecd93986897\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-9hv84" Nov 03 01:50:34.953077 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:34.952899 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80170d86-c913-4510-a93a-fecd93986897-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-9hv84\" (UID: \"80170d86-c913-4510-a93a-fecd93986897\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-9hv84" Nov 03 01:50:34.953077 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:34.952979 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80170d86-c913-4510-a93a-fecd93986897-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-9hv84\" (UID: \"80170d86-c913-4510-a93a-fecd93986897\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-9hv84" Nov 03 01:50:35.054598 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:35.054551 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgs86\" (UniqueName: \"kubernetes.io/projected/80170d86-c913-4510-a93a-fecd93986897-kube-api-access-bgs86\") pod \"maas-api-revoke-sub-3e31a506-9hv84\" (UID: \"80170d86-c913-4510-a93a-fecd93986897\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-9hv84" Nov 03 01:50:35.054821 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:35.054613 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80170d86-c913-4510-a93a-fecd93986897-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-9hv84\" (UID: \"80170d86-c913-4510-a93a-fecd93986897\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-9hv84" Nov 03 01:50:35.054821 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:35.054637 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80170d86-c913-4510-a93a-fecd93986897-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-9hv84\" (UID: \"80170d86-c913-4510-a93a-fecd93986897\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-9hv84" Nov 03 01:50:35.055331 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:35.055310 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80170d86-c913-4510-a93a-fecd93986897-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-9hv84\" (UID: \"80170d86-c913-4510-a93a-fecd93986897\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-9hv84" Nov 03 01:50:35.057149 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:35.057125 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80170d86-c913-4510-a93a-fecd93986897-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-9hv84\" (UID: \"80170d86-c913-4510-a93a-fecd93986897\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-9hv84" Nov 03 01:50:35.063174 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:35.063151 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgs86\" (UniqueName: \"kubernetes.io/projected/80170d86-c913-4510-a93a-fecd93986897-kube-api-access-bgs86\") pod \"maas-api-revoke-sub-3e31a506-9hv84\" (UID: \"80170d86-c913-4510-a93a-fecd93986897\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-9hv84" Nov 03 01:50:35.184589 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:35.184495 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-9hv84" Nov 03 01:50:35.474195 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:35.474090 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-9hv84"] Nov 03 01:50:35.479071 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:50:35.479010 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80170d86_c913_4510_a93a_fecd93986897.slice/crio-0d0100d2797ad6dcd055dcb7e958109650699dfca0ec753ff909370cfed82111 WatchSource:0}: Error finding container 0d0100d2797ad6dcd055dcb7e958109650699dfca0ec753ff909370cfed82111: Status 404 returned error can't find the container with id 0d0100d2797ad6dcd055dcb7e958109650699dfca0ec753ff909370cfed82111 Nov 03 01:50:36.018369 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:36.017388 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-9hv84" event={"ID":"80170d86-c913-4510-a93a-fecd93986897","Type":"ContainerStarted","Data":"3ddc4cebbe0a7add21a6f997287e93e9002492641e516163c9fdc86cbee86290"} Nov 03 01:50:36.018369 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:36.017462 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-9hv84" event={"ID":"80170d86-c913-4510-a93a-fecd93986897","Type":"ContainerStarted","Data":"0d0100d2797ad6dcd055dcb7e958109650699dfca0ec753ff909370cfed82111"} Nov 03 01:50:36.045298 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:36.044377 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-9hv84" podStartSLOduration=2.044338589 podStartE2EDuration="2.044338589s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:50:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:50:36.042056717 +0000 UTC m=+4178.213059814" watchObservedRunningTime="2026-11-03 01:50:36.044338589 +0000 UTC m=+4178.215341687" Nov 03 01:50:37.072299 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:37.071253 4421 generic.go:356] "Generic (PLEG): container finished" podID="80170d86-c913-4510-a93a-fecd93986897" containerID="3ddc4cebbe0a7add21a6f997287e93e9002492641e516163c9fdc86cbee86290" exitCode=0 Nov 03 01:50:37.072299 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:37.071422 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-9hv84" event={"ID":"80170d86-c913-4510-a93a-fecd93986897","Type":"ContainerDied","Data":"3ddc4cebbe0a7add21a6f997287e93e9002492641e516163c9fdc86cbee86290"} Nov 03 01:50:38.217187 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:38.217157 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-9hv84" Nov 03 01:50:38.295195 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:38.295164 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80170d86-c913-4510-a93a-fecd93986897-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80170d86-c913-4510-a93a-fecd93986897-maas-api-openshift-ca\") pod \"80170d86-c913-4510-a93a-fecd93986897\" (UID: \"80170d86-c913-4510-a93a-fecd93986897\") " Nov 03 01:50:38.295408 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:38.295331 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80170d86-c913-4510-a93a-fecd93986897-kube-api-access-bgs86\" (UniqueName: \"kubernetes.io/projected/80170d86-c913-4510-a93a-fecd93986897-kube-api-access-bgs86\") pod \"80170d86-c913-4510-a93a-fecd93986897\" (UID: \"80170d86-c913-4510-a93a-fecd93986897\") " Nov 03 01:50:38.295408 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:38.295371 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/80170d86-c913-4510-a93a-fecd93986897-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80170d86-c913-4510-a93a-fecd93986897-maas-api-xks-ca\") pod \"80170d86-c913-4510-a93a-fecd93986897\" (UID: \"80170d86-c913-4510-a93a-fecd93986897\") " Nov 03 01:50:38.295590 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:38.295564 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80170d86-c913-4510-a93a-fecd93986897-maas-api-openshift-ca" pod "80170d86-c913-4510-a93a-fecd93986897" (UID: "80170d86-c913-4510-a93a-fecd93986897"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:50:38.297468 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:38.297434 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80170d86-c913-4510-a93a-fecd93986897-kube-api-access-bgs86" pod "80170d86-c913-4510-a93a-fecd93986897" (UID: "80170d86-c913-4510-a93a-fecd93986897"). InnerVolumeSpecName "kube-api-access-bgs86". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:50:38.297573 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:38.297461 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80170d86-c913-4510-a93a-fecd93986897-maas-api-xks-ca" pod "80170d86-c913-4510-a93a-fecd93986897" (UID: "80170d86-c913-4510-a93a-fecd93986897"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:50:38.396171 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:38.396081 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80170d86-c913-4510-a93a-fecd93986897-maas-api-xks-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:50:38.396171 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:38.396118 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80170d86-c913-4510-a93a-fecd93986897-maas-api-openshift-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:50:38.396171 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:38.396130 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bgs86\" (UniqueName: \"kubernetes.io/projected/80170d86-c913-4510-a93a-fecd93986897-kube-api-access-bgs86\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:50:39.082765 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:39.082738 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-9hv84" Nov 03 01:50:39.083051 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:39.082740 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-9hv84" event={"ID":"80170d86-c913-4510-a93a-fecd93986897","Type":"ContainerDied","Data":"0d0100d2797ad6dcd055dcb7e958109650699dfca0ec753ff909370cfed82111"} Nov 03 01:50:39.083193 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:39.083073 4421 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0d0100d2797ad6dcd055dcb7e958109650699dfca0ec753ff909370cfed82111" Nov 03 01:50:44.669249 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:50:44.669163 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 01:50:46.488711 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:46.488672 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-b-077b6796-kserve-58584d646f-45vkg" podUID="be23a06d-0c04-4004-b6ec-67b5f4f7af19" containerName="main" containerID="cri-o://b2e674cdf582895b78524b506871bbb050feb8115cefb0154b93b68c785fe615" gracePeriod=45 Nov 03 01:50:47.031930 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:47.031525 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-077b6796-kserve-58584d646f-45vkg" Nov 03 01:50:47.091663 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:47.091562 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be23a06d-0c04-4004-b6ec-67b5f4f7af19-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be23a06d-0c04-4004-b6ec-67b5f4f7af19-tmp-dir\") pod \"be23a06d-0c04-4004-b6ec-67b5f4f7af19\" (UID: \"be23a06d-0c04-4004-b6ec-67b5f4f7af19\") " Nov 03 01:50:47.091851 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:47.091700 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be23a06d-0c04-4004-b6ec-67b5f4f7af19-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be23a06d-0c04-4004-b6ec-67b5f4f7af19-model-cache\") pod \"be23a06d-0c04-4004-b6ec-67b5f4f7af19\" (UID: \"be23a06d-0c04-4004-b6ec-67b5f4f7af19\") " Nov 03 01:50:47.091851 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:47.091746 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/be23a06d-0c04-4004-b6ec-67b5f4f7af19-tls-certs\" (UniqueName: \"kubernetes.io/secret/be23a06d-0c04-4004-b6ec-67b5f4f7af19-tls-certs\") pod \"be23a06d-0c04-4004-b6ec-67b5f4f7af19\" (UID: \"be23a06d-0c04-4004-b6ec-67b5f4f7af19\") " Nov 03 01:50:47.091851 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:47.091792 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be23a06d-0c04-4004-b6ec-67b5f4f7af19-dshm\" (UniqueName: \"kubernetes.io/empty-dir/be23a06d-0c04-4004-b6ec-67b5f4f7af19-dshm\") pod \"be23a06d-0c04-4004-b6ec-67b5f4f7af19\" (UID: \"be23a06d-0c04-4004-b6ec-67b5f4f7af19\") " Nov 03 01:50:47.092060 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:47.091893 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be23a06d-0c04-4004-b6ec-67b5f4f7af19-home\" (UniqueName: \"kubernetes.io/empty-dir/be23a06d-0c04-4004-b6ec-67b5f4f7af19-home\") pod \"be23a06d-0c04-4004-b6ec-67b5f4f7af19\" (UID: \"be23a06d-0c04-4004-b6ec-67b5f4f7af19\") " Nov 03 01:50:47.092060 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:47.091896 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be23a06d-0c04-4004-b6ec-67b5f4f7af19-tmp-dir" pod "be23a06d-0c04-4004-b6ec-67b5f4f7af19" (UID: "be23a06d-0c04-4004-b6ec-67b5f4f7af19"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:50:47.092060 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:47.091942 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be23a06d-0c04-4004-b6ec-67b5f4f7af19-kube-api-access-lqqh6\" (UniqueName: \"kubernetes.io/projected/be23a06d-0c04-4004-b6ec-67b5f4f7af19-kube-api-access-lqqh6\") pod \"be23a06d-0c04-4004-b6ec-67b5f4f7af19\" (UID: \"be23a06d-0c04-4004-b6ec-67b5f4f7af19\") " Nov 03 01:50:47.092244 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:47.092231 4421 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be23a06d-0c04-4004-b6ec-67b5f4f7af19-tmp-dir\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:50:47.092893 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:47.092523 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be23a06d-0c04-4004-b6ec-67b5f4f7af19-model-cache" pod "be23a06d-0c04-4004-b6ec-67b5f4f7af19" (UID: "be23a06d-0c04-4004-b6ec-67b5f4f7af19"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:50:47.092893 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:47.092844 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be23a06d-0c04-4004-b6ec-67b5f4f7af19-home" pod "be23a06d-0c04-4004-b6ec-67b5f4f7af19" (UID: "be23a06d-0c04-4004-b6ec-67b5f4f7af19"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:50:47.094575 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:47.094550 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be23a06d-0c04-4004-b6ec-67b5f4f7af19-tls-certs" pod "be23a06d-0c04-4004-b6ec-67b5f4f7af19" (UID: "be23a06d-0c04-4004-b6ec-67b5f4f7af19"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:50:47.094908 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:47.094877 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be23a06d-0c04-4004-b6ec-67b5f4f7af19-kube-api-access-lqqh6" pod "be23a06d-0c04-4004-b6ec-67b5f4f7af19" (UID: "be23a06d-0c04-4004-b6ec-67b5f4f7af19"). InnerVolumeSpecName "kube-api-access-lqqh6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:50:47.095477 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:47.095447 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be23a06d-0c04-4004-b6ec-67b5f4f7af19-dshm" pod "be23a06d-0c04-4004-b6ec-67b5f4f7af19" (UID: "be23a06d-0c04-4004-b6ec-67b5f4f7af19"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:50:47.175157 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:47.175090 4421 generic.go:356] "Generic (PLEG): container finished" podID="be23a06d-0c04-4004-b6ec-67b5f4f7af19" containerID="b2e674cdf582895b78524b506871bbb050feb8115cefb0154b93b68c785fe615" exitCode=0 Nov 03 01:50:47.175391 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:47.175231 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-077b6796-kserve-58584d646f-45vkg" event={"ID":"be23a06d-0c04-4004-b6ec-67b5f4f7af19","Type":"ContainerDied","Data":"b2e674cdf582895b78524b506871bbb050feb8115cefb0154b93b68c785fe615"} Nov 03 01:50:47.175391 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:47.175288 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-077b6796-kserve-58584d646f-45vkg" event={"ID":"be23a06d-0c04-4004-b6ec-67b5f4f7af19","Type":"ContainerDied","Data":"ccb14f49b515717ec3996028d4bc869debaa6ce74def1e2dffbce8392710fc83"} Nov 03 01:50:47.175391 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:47.175316 4421 scope.go:122] "RemoveContainer" containerID="b2e674cdf582895b78524b506871bbb050feb8115cefb0154b93b68c785fe615" Nov 03 01:50:47.175602 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:47.175478 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-077b6796-kserve-58584d646f-45vkg" Nov 03 01:50:47.193263 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:47.193236 4421 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be23a06d-0c04-4004-b6ec-67b5f4f7af19-model-cache\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:50:47.193444 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:47.193263 4421 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be23a06d-0c04-4004-b6ec-67b5f4f7af19-tls-certs\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:50:47.193444 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:47.193345 4421 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be23a06d-0c04-4004-b6ec-67b5f4f7af19-dshm\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:50:47.193444 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:47.193359 4421 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be23a06d-0c04-4004-b6ec-67b5f4f7af19-home\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:50:47.193444 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:47.193373 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqqh6\" (UniqueName: \"kubernetes.io/projected/be23a06d-0c04-4004-b6ec-67b5f4f7af19-kube-api-access-lqqh6\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:50:47.201581 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:47.201561 4421 scope.go:122] "RemoveContainer" containerID="b2e674cdf582895b78524b506871bbb050feb8115cefb0154b93b68c785fe615" Nov 03 01:50:47.201903 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:50:47.201881 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2e674cdf582895b78524b506871bbb050feb8115cefb0154b93b68c785fe615\": container with ID starting with b2e674cdf582895b78524b506871bbb050feb8115cefb0154b93b68c785fe615 not found: ID does not exist" containerID="b2e674cdf582895b78524b506871bbb050feb8115cefb0154b93b68c785fe615" Nov 03 01:50:47.201986 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:47.201916 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2e674cdf582895b78524b506871bbb050feb8115cefb0154b93b68c785fe615"} err="failed to get container status \"b2e674cdf582895b78524b506871bbb050feb8115cefb0154b93b68c785fe615\": rpc error: code = NotFound desc = could not find container \"b2e674cdf582895b78524b506871bbb050feb8115cefb0154b93b68c785fe615\": container with ID starting with b2e674cdf582895b78524b506871bbb050feb8115cefb0154b93b68c785fe615 not found: ID does not exist" Nov 03 01:50:47.259407 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:47.259327 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-077b6796-kserve-58584d646f-45vkg"] Nov 03 01:50:47.263303 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:47.260105 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-077b6796-kserve-58584d646f-45vkg"] Nov 03 01:50:47.822033 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:47.821974 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-same-id-b-077b6796-kserve-58584d646f-45vkg" podUID="be23a06d-0c04-4004-b6ec-67b5f4f7af19" containerName="main" probeResult="failure" output="Get \"https://10.134.1.142:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:50:47.982323 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:47.982286 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be23a06d-0c04-4004-b6ec-67b5f4f7af19" path="/var/lib/kubelet/pods/be23a06d-0c04-4004-b6ec-67b5f4f7af19/volumes" Nov 03 01:50:51.720233 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:51.720195 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4823bfaf-p7fp5"] Nov 03 01:50:51.721133 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:51.721104 4421 cpu_manager.go:414] "containerMap: removing container" podUID="80170d86-c913-4510-a93a-fecd93986897" containerName="revoke-keys" Nov 03 01:50:51.721133 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:51.721133 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80170d86-c913-4510-a93a-fecd93986897" containerName="revoke-keys" Nov 03 01:50:51.721347 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:51.721234 4421 cpu_manager.go:414] "containerMap: removing container" podUID="be23a06d-0c04-4004-b6ec-67b5f4f7af19" containerName="main" Nov 03 01:50:51.721347 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:51.721243 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be23a06d-0c04-4004-b6ec-67b5f4f7af19" containerName="main" Nov 03 01:50:51.721576 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:51.721560 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="80170d86-c913-4510-a93a-fecd93986897" containerName="revoke-keys" Nov 03 01:50:51.721646 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:51.721599 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="be23a06d-0c04-4004-b6ec-67b5f4f7af19" containerName="main" Nov 03 01:50:51.731666 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:51.731630 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4823bfaf-p7fp5" Nov 03 01:50:51.732492 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:51.732465 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4823bfaf-p7fp5"] Nov 03 01:50:51.735700 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:51.735674 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:50:51.736696 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:51.736672 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 01:50:51.736831 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:51.736812 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 01:50:51.736831 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:51.736825 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-swfcr\"" Nov 03 01:50:51.785149 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:51.785110 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d649c869-44f4-42c3-b222-eec4e0375ba2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4823bfaf-p7fp5\" (UID: \"d649c869-44f4-42c3-b222-eec4e0375ba2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4823bfaf-p7fp5" Nov 03 01:50:51.785410 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:51.785158 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6btk\" (UniqueName: \"kubernetes.io/projected/d649c869-44f4-42c3-b222-eec4e0375ba2-kube-api-access-t6btk\") pod \"maas-api-revoke-sub-4823bfaf-p7fp5\" (UID: \"d649c869-44f4-42c3-b222-eec4e0375ba2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4823bfaf-p7fp5" Nov 03 01:50:51.785410 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:51.785312 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d649c869-44f4-42c3-b222-eec4e0375ba2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4823bfaf-p7fp5\" (UID: \"d649c869-44f4-42c3-b222-eec4e0375ba2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4823bfaf-p7fp5" Nov 03 01:50:51.886682 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:51.886647 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d649c869-44f4-42c3-b222-eec4e0375ba2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4823bfaf-p7fp5\" (UID: \"d649c869-44f4-42c3-b222-eec4e0375ba2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4823bfaf-p7fp5" Nov 03 01:50:51.886892 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:51.886728 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d649c869-44f4-42c3-b222-eec4e0375ba2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4823bfaf-p7fp5\" (UID: \"d649c869-44f4-42c3-b222-eec4e0375ba2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4823bfaf-p7fp5" Nov 03 01:50:51.886892 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:51.886761 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6btk\" (UniqueName: \"kubernetes.io/projected/d649c869-44f4-42c3-b222-eec4e0375ba2-kube-api-access-t6btk\") pod \"maas-api-revoke-sub-4823bfaf-p7fp5\" (UID: \"d649c869-44f4-42c3-b222-eec4e0375ba2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4823bfaf-p7fp5" Nov 03 01:50:51.887452 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:51.887423 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d649c869-44f4-42c3-b222-eec4e0375ba2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4823bfaf-p7fp5\" (UID: \"d649c869-44f4-42c3-b222-eec4e0375ba2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4823bfaf-p7fp5" Nov 03 01:50:51.889142 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:51.889115 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d649c869-44f4-42c3-b222-eec4e0375ba2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4823bfaf-p7fp5\" (UID: \"d649c869-44f4-42c3-b222-eec4e0375ba2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4823bfaf-p7fp5" Nov 03 01:50:51.894571 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:51.894548 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6btk\" (UniqueName: \"kubernetes.io/projected/d649c869-44f4-42c3-b222-eec4e0375ba2-kube-api-access-t6btk\") pod \"maas-api-revoke-sub-4823bfaf-p7fp5\" (UID: \"d649c869-44f4-42c3-b222-eec4e0375ba2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4823bfaf-p7fp5" Nov 03 01:50:52.047262 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:52.047226 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4823bfaf-p7fp5" Nov 03 01:50:52.321393 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:52.321366 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4823bfaf-p7fp5"] Nov 03 01:50:52.323490 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:50:52.323458 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd649c869_44f4_42c3_b222_eec4e0375ba2.slice/crio-48b6557a5c2885719fa8563e5fa59cb004c5c0f2c342917e99396f3f4d88e5ee WatchSource:0}: Error finding container 48b6557a5c2885719fa8563e5fa59cb004c5c0f2c342917e99396f3f4d88e5ee: Status 404 returned error can't find the container with id 48b6557a5c2885719fa8563e5fa59cb004c5c0f2c342917e99396f3f4d88e5ee Nov 03 01:50:53.212452 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:53.212415 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4823bfaf-p7fp5" event={"ID":"d649c869-44f4-42c3-b222-eec4e0375ba2","Type":"ContainerStarted","Data":"a02b926898878e5f8069cea00dabb7b39f1e4097e11f6f00ddfaffcef4496ba0"} Nov 03 01:50:53.212901 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:53.212471 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4823bfaf-p7fp5" event={"ID":"d649c869-44f4-42c3-b222-eec4e0375ba2","Type":"ContainerStarted","Data":"48b6557a5c2885719fa8563e5fa59cb004c5c0f2c342917e99396f3f4d88e5ee"} Nov 03 01:50:53.231659 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:53.231607 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4823bfaf-p7fp5" podStartSLOduration=2.231591817 podStartE2EDuration="2.231591817s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:50:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:50:53.229678161 +0000 UTC m=+4195.400681258" watchObservedRunningTime="2026-11-03 01:50:53.231591817 +0000 UTC m=+4195.402594912" Nov 03 01:50:54.219911 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:54.219873 4421 generic.go:356] "Generic (PLEG): container finished" podID="d649c869-44f4-42c3-b222-eec4e0375ba2" containerID="a02b926898878e5f8069cea00dabb7b39f1e4097e11f6f00ddfaffcef4496ba0" exitCode=0 Nov 03 01:50:54.220391 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:54.219949 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4823bfaf-p7fp5" event={"ID":"d649c869-44f4-42c3-b222-eec4e0375ba2","Type":"ContainerDied","Data":"a02b926898878e5f8069cea00dabb7b39f1e4097e11f6f00ddfaffcef4496ba0"} Nov 03 01:50:55.363937 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:55.363913 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4823bfaf-p7fp5" Nov 03 01:50:55.421751 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:55.421716 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d649c869-44f4-42c3-b222-eec4e0375ba2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d649c869-44f4-42c3-b222-eec4e0375ba2-maas-api-openshift-ca\") pod \"d649c869-44f4-42c3-b222-eec4e0375ba2\" (UID: \"d649c869-44f4-42c3-b222-eec4e0375ba2\") " Nov 03 01:50:55.421934 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:55.421797 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d649c869-44f4-42c3-b222-eec4e0375ba2-kube-api-access-t6btk\" (UniqueName: \"kubernetes.io/projected/d649c869-44f4-42c3-b222-eec4e0375ba2-kube-api-access-t6btk\") pod \"d649c869-44f4-42c3-b222-eec4e0375ba2\" (UID: \"d649c869-44f4-42c3-b222-eec4e0375ba2\") " Nov 03 01:50:55.421934 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:55.421865 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d649c869-44f4-42c3-b222-eec4e0375ba2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d649c869-44f4-42c3-b222-eec4e0375ba2-maas-api-xks-ca\") pod \"d649c869-44f4-42c3-b222-eec4e0375ba2\" (UID: \"d649c869-44f4-42c3-b222-eec4e0375ba2\") " Nov 03 01:50:55.422158 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:55.422131 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d649c869-44f4-42c3-b222-eec4e0375ba2-maas-api-openshift-ca" pod "d649c869-44f4-42c3-b222-eec4e0375ba2" (UID: "d649c869-44f4-42c3-b222-eec4e0375ba2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:50:55.424075 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:55.424049 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d649c869-44f4-42c3-b222-eec4e0375ba2-kube-api-access-t6btk" pod "d649c869-44f4-42c3-b222-eec4e0375ba2" (UID: "d649c869-44f4-42c3-b222-eec4e0375ba2"). InnerVolumeSpecName "kube-api-access-t6btk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:50:55.424075 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:55.424059 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d649c869-44f4-42c3-b222-eec4e0375ba2-maas-api-xks-ca" pod "d649c869-44f4-42c3-b222-eec4e0375ba2" (UID: "d649c869-44f4-42c3-b222-eec4e0375ba2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:50:55.523373 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:55.523341 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d649c869-44f4-42c3-b222-eec4e0375ba2-maas-api-xks-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:50:55.523373 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:55.523370 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d649c869-44f4-42c3-b222-eec4e0375ba2-maas-api-openshift-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:50:55.523373 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:55.523381 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6btk\" (UniqueName: \"kubernetes.io/projected/d649c869-44f4-42c3-b222-eec4e0375ba2-kube-api-access-t6btk\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:50:56.229766 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:56.229732 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4823bfaf-p7fp5" Nov 03 01:50:56.229987 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:56.229769 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4823bfaf-p7fp5" event={"ID":"d649c869-44f4-42c3-b222-eec4e0375ba2","Type":"ContainerDied","Data":"48b6557a5c2885719fa8563e5fa59cb004c5c0f2c342917e99396f3f4d88e5ee"} Nov 03 01:50:56.229987 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:50:56.229815 4421 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="48b6557a5c2885719fa8563e5fa59cb004c5c0f2c342917e99396f3f4d88e5ee" Nov 03 01:51:04.885017 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:51:04.884981 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-5f985d8646-z5mcb"] Nov 03 01:51:04.885503 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:51:04.885444 4421 cpu_manager.go:414] "containerMap: removing container" podUID="d649c869-44f4-42c3-b222-eec4e0375ba2" containerName="revoke-keys" Nov 03 01:51:04.885503 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:51:04.885461 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d649c869-44f4-42c3-b222-eec4e0375ba2" containerName="revoke-keys" Nov 03 01:51:04.885599 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:51:04.885538 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="d649c869-44f4-42c3-b222-eec4e0375ba2" containerName="revoke-keys" Nov 03 01:51:04.888830 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:51:04.888812 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-5f985d8646-z5mcb" Nov 03 01:51:04.895497 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:51:04.895467 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-5f985d8646-z5mcb"] Nov 03 01:51:04.895936 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:51:04.895914 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Nov 03 01:51:04.897165 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:51:04.897146 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-l45h2\"" Nov 03 01:51:05.019991 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:51:05.019949 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f944078-d792-4086-8096-ae6256e7d955-plugins-config-volume\") pod \"payload-pre-processing-5f985d8646-z5mcb\" (UID: \"5f944078-d792-4086-8096-ae6256e7d955\") " pod="openshift-ingress/payload-pre-processing-5f985d8646-z5mcb" Nov 03 01:51:05.019991 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:51:05.019991 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmzkk\" (UniqueName: \"kubernetes.io/projected/5f944078-d792-4086-8096-ae6256e7d955-kube-api-access-bmzkk\") pod \"payload-pre-processing-5f985d8646-z5mcb\" (UID: \"5f944078-d792-4086-8096-ae6256e7d955\") " pod="openshift-ingress/payload-pre-processing-5f985d8646-z5mcb" Nov 03 01:51:05.120680 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:51:05.120644 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f944078-d792-4086-8096-ae6256e7d955-plugins-config-volume\") pod \"payload-pre-processing-5f985d8646-z5mcb\" (UID: \"5f944078-d792-4086-8096-ae6256e7d955\") " pod="openshift-ingress/payload-pre-processing-5f985d8646-z5mcb" Nov 03 01:51:05.120680 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:51:05.120678 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmzkk\" (UniqueName: \"kubernetes.io/projected/5f944078-d792-4086-8096-ae6256e7d955-kube-api-access-bmzkk\") pod \"payload-pre-processing-5f985d8646-z5mcb\" (UID: \"5f944078-d792-4086-8096-ae6256e7d955\") " pod="openshift-ingress/payload-pre-processing-5f985d8646-z5mcb" Nov 03 01:51:05.121468 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:51:05.121445 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f944078-d792-4086-8096-ae6256e7d955-plugins-config-volume\") pod \"payload-pre-processing-5f985d8646-z5mcb\" (UID: \"5f944078-d792-4086-8096-ae6256e7d955\") " pod="openshift-ingress/payload-pre-processing-5f985d8646-z5mcb" Nov 03 01:51:05.130417 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:51:05.130381 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmzkk\" (UniqueName: \"kubernetes.io/projected/5f944078-d792-4086-8096-ae6256e7d955-kube-api-access-bmzkk\") pod \"payload-pre-processing-5f985d8646-z5mcb\" (UID: \"5f944078-d792-4086-8096-ae6256e7d955\") " pod="openshift-ingress/payload-pre-processing-5f985d8646-z5mcb" Nov 03 01:51:05.203701 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:51:05.203615 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-5f985d8646-z5mcb" Nov 03 01:51:05.482362 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:51:05.482266 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-5f985d8646-z5mcb"] Nov 03 01:51:05.486220 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:51:05.486194 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f944078_d792_4086_8096_ae6256e7d955.slice/crio-9300b3c456d23c84e87f7b9b3f74649b1f1f318c4624b1e8f4da708592b6917b WatchSource:0}: Error finding container 9300b3c456d23c84e87f7b9b3f74649b1f1f318c4624b1e8f4da708592b6917b: Status 404 returned error can't find the container with id 9300b3c456d23c84e87f7b9b3f74649b1f1f318c4624b1e8f4da708592b6917b Nov 03 01:51:06.284788 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:51:06.284752 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-5f985d8646-z5mcb" event={"ID":"5f944078-d792-4086-8096-ae6256e7d955","Type":"ContainerStarted","Data":"be33d971ec702a7c8edfbf1e51a69c6be2d1c0567dd634c1a756e7e771348ad3"} Nov 03 01:51:06.285229 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:51:06.284810 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-5f985d8646-z5mcb" event={"ID":"5f944078-d792-4086-8096-ae6256e7d955","Type":"ContainerStarted","Data":"9300b3c456d23c84e87f7b9b3f74649b1f1f318c4624b1e8f4da708592b6917b"} Nov 03 01:51:06.285229 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:51:06.284876 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-5f985d8646-z5mcb" Nov 03 01:51:06.301861 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:51:06.301806 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-5f985d8646-z5mcb" podStartSLOduration=2.30178754 podStartE2EDuration="2.30178754s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:51:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:51:06.299968881 +0000 UTC m=+4208.470971978" watchObservedRunningTime="2026-11-03 01:51:06.30178754 +0000 UTC m=+4208.472790637" Nov 03 01:51:14.669025 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:51:14.668915 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 01:51:17.293652 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:51:17.293618 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-5f985d8646-z5mcb" Nov 03 01:51:19.021680 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:51:19.021637 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-76c78d77d6-zpncn"] Nov 03 01:51:19.025258 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:51:19.025233 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-76c78d77d6-zpncn" Nov 03 01:51:19.032323 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:51:19.032294 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-76c78d77d6-zpncn"] Nov 03 01:51:19.163866 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:51:19.163827 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17a9712a-5861-4d51-a7bb-4f32c71e9e78-plugins-config-volume\") pod \"payload-processing-76c78d77d6-zpncn\" (UID: \"17a9712a-5861-4d51-a7bb-4f32c71e9e78\") " pod="openshift-ingress/payload-processing-76c78d77d6-zpncn" Nov 03 01:51:19.164061 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:51:19.163893 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwltb\" (UniqueName: \"kubernetes.io/projected/17a9712a-5861-4d51-a7bb-4f32c71e9e78-kube-api-access-rwltb\") pod \"payload-processing-76c78d77d6-zpncn\" (UID: \"17a9712a-5861-4d51-a7bb-4f32c71e9e78\") " pod="openshift-ingress/payload-processing-76c78d77d6-zpncn" Nov 03 01:51:19.264654 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:51:19.264596 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17a9712a-5861-4d51-a7bb-4f32c71e9e78-plugins-config-volume\") pod \"payload-processing-76c78d77d6-zpncn\" (UID: \"17a9712a-5861-4d51-a7bb-4f32c71e9e78\") " pod="openshift-ingress/payload-processing-76c78d77d6-zpncn" Nov 03 01:51:19.264866 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:51:19.264691 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwltb\" (UniqueName: \"kubernetes.io/projected/17a9712a-5861-4d51-a7bb-4f32c71e9e78-kube-api-access-rwltb\") pod \"payload-processing-76c78d77d6-zpncn\" (UID: \"17a9712a-5861-4d51-a7bb-4f32c71e9e78\") " pod="openshift-ingress/payload-processing-76c78d77d6-zpncn" Nov 03 01:51:19.265389 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:51:19.265364 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17a9712a-5861-4d51-a7bb-4f32c71e9e78-plugins-config-volume\") pod \"payload-processing-76c78d77d6-zpncn\" (UID: \"17a9712a-5861-4d51-a7bb-4f32c71e9e78\") " pod="openshift-ingress/payload-processing-76c78d77d6-zpncn" Nov 03 01:51:19.272950 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:51:19.272888 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwltb\" (UniqueName: \"kubernetes.io/projected/17a9712a-5861-4d51-a7bb-4f32c71e9e78-kube-api-access-rwltb\") pod \"payload-processing-76c78d77d6-zpncn\" (UID: \"17a9712a-5861-4d51-a7bb-4f32c71e9e78\") " pod="openshift-ingress/payload-processing-76c78d77d6-zpncn" Nov 03 01:51:19.343078 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:51:19.343042 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-76c78d77d6-zpncn" Nov 03 01:51:19.617518 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:51:19.617492 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-76c78d77d6-zpncn"] Nov 03 01:51:19.619766 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:51:19.619732 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17a9712a_5861_4d51_a7bb_4f32c71e9e78.slice/crio-4eb614a4acb84fa38220f2ffd84beb6cae2ab04b3287eb8ea59aab6612da3e1f WatchSource:0}: Error finding container 4eb614a4acb84fa38220f2ffd84beb6cae2ab04b3287eb8ea59aab6612da3e1f: Status 404 returned error can't find the container with id 4eb614a4acb84fa38220f2ffd84beb6cae2ab04b3287eb8ea59aab6612da3e1f Nov 03 01:51:20.354924 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:51:20.354887 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-76c78d77d6-zpncn" event={"ID":"17a9712a-5861-4d51-a7bb-4f32c71e9e78","Type":"ContainerStarted","Data":"2bce2c120bb796cfa04e25f5c2065f0531c268338ba6a4ee98fae1f8d98ed9b0"} Nov 03 01:51:20.355362 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:51:20.354939 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-76c78d77d6-zpncn" event={"ID":"17a9712a-5861-4d51-a7bb-4f32c71e9e78","Type":"ContainerStarted","Data":"4eb614a4acb84fa38220f2ffd84beb6cae2ab04b3287eb8ea59aab6612da3e1f"} Nov 03 01:51:20.355362 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:51:20.354975 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-76c78d77d6-zpncn" Nov 03 01:51:20.372624 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:51:20.372571 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-76c78d77d6-zpncn" podStartSLOduration=1.372557543 podStartE2EDuration="1.372557543s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:51:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:51:20.370547182 +0000 UTC m=+4222.541550278" watchObservedRunningTime="2026-11-03 01:51:20.372557543 +0000 UTC m=+4222.543560640" Nov 03 01:51:31.362388 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:51:31.362355 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-76c78d77d6-zpncn" Nov 03 01:51:44.669020 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:51:44.668982 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 01:51:45.840843 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:51:45.840804 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-5f985d8646-z5mcb"] Nov 03 01:51:45.855532 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:51:45.855495 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-76c78d77d6-zpncn"] Nov 03 01:51:50.841511 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:51:50.841449 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-5f985d8646-z5mcb" podUID="5f944078-d792-4086-8096-ae6256e7d955" containerName="payload-pre-processing" containerID="cri-o://be33d971ec702a7c8edfbf1e51a69c6be2d1c0567dd634c1a756e7e771348ad3" gracePeriod=25 Nov 03 01:51:50.856181 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:51:50.856130 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-76c78d77d6-zpncn" podUID="17a9712a-5861-4d51-a7bb-4f32c71e9e78" containerName="payload-processing" containerID="cri-o://2bce2c120bb796cfa04e25f5c2065f0531c268338ba6a4ee98fae1f8d98ed9b0" gracePeriod=25 Nov 03 01:51:51.449358 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:51:51.448709 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-5f985d8646-z5mcb" Nov 03 01:51:51.525316 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:51:51.506407 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-76c78d77d6-zpncn" Nov 03 01:51:51.588620 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:51:51.588582 4421 generic.go:356] "Generic (PLEG): container finished" podID="5f944078-d792-4086-8096-ae6256e7d955" containerID="be33d971ec702a7c8edfbf1e51a69c6be2d1c0567dd634c1a756e7e771348ad3" exitCode=0 Nov 03 01:51:51.588980 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:51:51.588959 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-5f985d8646-z5mcb" event={"ID":"5f944078-d792-4086-8096-ae6256e7d955","Type":"ContainerDied","Data":"be33d971ec702a7c8edfbf1e51a69c6be2d1c0567dd634c1a756e7e771348ad3"} Nov 03 01:51:51.589126 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:51:51.589113 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-5f985d8646-z5mcb" event={"ID":"5f944078-d792-4086-8096-ae6256e7d955","Type":"ContainerDied","Data":"9300b3c456d23c84e87f7b9b3f74649b1f1f318c4624b1e8f4da708592b6917b"} Nov 03 01:51:51.595799 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:51:51.595771 4421 scope.go:122] "RemoveContainer" containerID="be33d971ec702a7c8edfbf1e51a69c6be2d1c0567dd634c1a756e7e771348ad3" Nov 03 01:51:51.603649 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:51:51.603387 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f944078-d792-4086-8096-ae6256e7d955-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f944078-d792-4086-8096-ae6256e7d955-plugins-config-volume\") pod \"5f944078-d792-4086-8096-ae6256e7d955\" (UID: \"5f944078-d792-4086-8096-ae6256e7d955\") " Nov 03 01:51:51.603916 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:51:51.603426 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-5f985d8646-z5mcb" Nov 03 01:51:51.604820 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:51:51.603777 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f944078-d792-4086-8096-ae6256e7d955-plugins-config-volume" pod "5f944078-d792-4086-8096-ae6256e7d955" (UID: "5f944078-d792-4086-8096-ae6256e7d955"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:51:51.604820 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:51:51.603900 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f944078-d792-4086-8096-ae6256e7d955-kube-api-access-bmzkk\" (UniqueName: \"kubernetes.io/projected/5f944078-d792-4086-8096-ae6256e7d955-kube-api-access-bmzkk\") pod \"5f944078-d792-4086-8096-ae6256e7d955\" (UID: \"5f944078-d792-4086-8096-ae6256e7d955\") " Nov 03 01:51:51.604820 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:51:51.604764 4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f944078-d792-4086-8096-ae6256e7d955-plugins-config-volume\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:51:51.648047 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:51:51.646290 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f944078-d792-4086-8096-ae6256e7d955-kube-api-access-bmzkk" pod "5f944078-d792-4086-8096-ae6256e7d955" (UID: "5f944078-d792-4086-8096-ae6256e7d955"). InnerVolumeSpecName "kube-api-access-bmzkk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:51:51.651429 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:51:51.650925 4421 generic.go:356] "Generic (PLEG): container finished" podID="17a9712a-5861-4d51-a7bb-4f32c71e9e78" containerID="2bce2c120bb796cfa04e25f5c2065f0531c268338ba6a4ee98fae1f8d98ed9b0" exitCode=0 Nov 03 01:51:51.651429 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:51:51.651080 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-76c78d77d6-zpncn" event={"ID":"17a9712a-5861-4d51-a7bb-4f32c71e9e78","Type":"ContainerDied","Data":"2bce2c120bb796cfa04e25f5c2065f0531c268338ba6a4ee98fae1f8d98ed9b0"} Nov 03 01:51:51.651429 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:51:51.651132 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-76c78d77d6-zpncn" event={"ID":"17a9712a-5861-4d51-a7bb-4f32c71e9e78","Type":"ContainerDied","Data":"4eb614a4acb84fa38220f2ffd84beb6cae2ab04b3287eb8ea59aab6612da3e1f"} Nov 03 01:51:51.651429 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:51:51.651215 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-76c78d77d6-zpncn" Nov 03 01:51:51.696491 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:51:51.696464 4421 scope.go:122] "RemoveContainer" containerID="be33d971ec702a7c8edfbf1e51a69c6be2d1c0567dd634c1a756e7e771348ad3" Nov 03 01:51:51.696885 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:51:51.696854 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be33d971ec702a7c8edfbf1e51a69c6be2d1c0567dd634c1a756e7e771348ad3\": container with ID starting with be33d971ec702a7c8edfbf1e51a69c6be2d1c0567dd634c1a756e7e771348ad3 not found: ID does not exist" containerID="be33d971ec702a7c8edfbf1e51a69c6be2d1c0567dd634c1a756e7e771348ad3" Nov 03 01:51:51.697001 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:51:51.696900 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be33d971ec702a7c8edfbf1e51a69c6be2d1c0567dd634c1a756e7e771348ad3"} err="failed to get container status \"be33d971ec702a7c8edfbf1e51a69c6be2d1c0567dd634c1a756e7e771348ad3\": rpc error: code = NotFound desc = could not find container \"be33d971ec702a7c8edfbf1e51a69c6be2d1c0567dd634c1a756e7e771348ad3\": container with ID starting with be33d971ec702a7c8edfbf1e51a69c6be2d1c0567dd634c1a756e7e771348ad3 not found: ID does not exist" Nov 03 01:51:51.697001 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:51:51.696927 4421 scope.go:122] "RemoveContainer" containerID="2bce2c120bb796cfa04e25f5c2065f0531c268338ba6a4ee98fae1f8d98ed9b0" Nov 03 01:51:51.718843 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:51:51.714470 4421 scope.go:122] "RemoveContainer" containerID="2bce2c120bb796cfa04e25f5c2065f0531c268338ba6a4ee98fae1f8d98ed9b0" Nov 03 01:51:51.718843 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:51:51.715542 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17a9712a-5861-4d51-a7bb-4f32c71e9e78-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17a9712a-5861-4d51-a7bb-4f32c71e9e78-plugins-config-volume\") pod \"17a9712a-5861-4d51-a7bb-4f32c71e9e78\" (UID: \"17a9712a-5861-4d51-a7bb-4f32c71e9e78\") " Nov 03 01:51:51.718843 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:51:51.715656 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17a9712a-5861-4d51-a7bb-4f32c71e9e78-kube-api-access-rwltb\" (UniqueName: \"kubernetes.io/projected/17a9712a-5861-4d51-a7bb-4f32c71e9e78-kube-api-access-rwltb\") pod \"17a9712a-5861-4d51-a7bb-4f32c71e9e78\" (UID: \"17a9712a-5861-4d51-a7bb-4f32c71e9e78\") " Nov 03 01:51:51.718843 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:51:51.715961 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmzkk\" (UniqueName: \"kubernetes.io/projected/5f944078-d792-4086-8096-ae6256e7d955-kube-api-access-bmzkk\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:51:51.734305 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:51:51.728421 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2bce2c120bb796cfa04e25f5c2065f0531c268338ba6a4ee98fae1f8d98ed9b0\": container with ID starting with 2bce2c120bb796cfa04e25f5c2065f0531c268338ba6a4ee98fae1f8d98ed9b0 not found: ID does not exist" containerID="2bce2c120bb796cfa04e25f5c2065f0531c268338ba6a4ee98fae1f8d98ed9b0" Nov 03 01:51:51.734305 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:51:51.728473 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bce2c120bb796cfa04e25f5c2065f0531c268338ba6a4ee98fae1f8d98ed9b0"} err="failed to get container status \"2bce2c120bb796cfa04e25f5c2065f0531c268338ba6a4ee98fae1f8d98ed9b0\": rpc error: code = NotFound desc = could not find container \"2bce2c120bb796cfa04e25f5c2065f0531c268338ba6a4ee98fae1f8d98ed9b0\": container with ID starting with 2bce2c120bb796cfa04e25f5c2065f0531c268338ba6a4ee98fae1f8d98ed9b0 not found: ID does not exist" Nov 03 01:51:51.734305 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:51:51.728913 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17a9712a-5861-4d51-a7bb-4f32c71e9e78-plugins-config-volume" pod "17a9712a-5861-4d51-a7bb-4f32c71e9e78" (UID: "17a9712a-5861-4d51-a7bb-4f32c71e9e78"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:51:51.765467 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:51:51.765404 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17a9712a-5861-4d51-a7bb-4f32c71e9e78-kube-api-access-rwltb" pod "17a9712a-5861-4d51-a7bb-4f32c71e9e78" (UID: "17a9712a-5861-4d51-a7bb-4f32c71e9e78"). InnerVolumeSpecName "kube-api-access-rwltb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:51:51.820196 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:51:51.819914 4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17a9712a-5861-4d51-a7bb-4f32c71e9e78-plugins-config-volume\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:51:51.820196 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:51:51.819962 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwltb\" (UniqueName: \"kubernetes.io/projected/17a9712a-5861-4d51-a7bb-4f32c71e9e78-kube-api-access-rwltb\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:51:51.997893 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:51:51.997800 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-5f985d8646-z5mcb"] Nov 03 01:51:52.001848 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:51:52.001821 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-5f985d8646-z5mcb"] Nov 03 01:51:52.026399 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:51:52.026324 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-76c78d77d6-zpncn"] Nov 03 01:51:52.049302 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:51:52.042344 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-76c78d77d6-zpncn"] Nov 03 01:51:52.364298 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:51:52.363457 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-76c78d77d6-zpncn" podUID="17a9712a-5861-4d51-a7bb-4f32c71e9e78" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.1.147:50052\" within 1s: context deadline exceeded" Nov 03 01:51:52.364298 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:51:52.364046 4421 logging.go:55] [core] [Channel #17464 SubChannel #17465]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.1.147:50052", ServerName: "10.134.1.147:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.1.147:50052: operation was canceled" Nov 03 01:51:53.977130 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:51:53.977095 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17a9712a-5861-4d51-a7bb-4f32c71e9e78" path="/var/lib/kubelet/pods/17a9712a-5861-4d51-a7bb-4f32c71e9e78/volumes" Nov 03 01:51:53.977641 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:51:53.977529 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f944078-d792-4086-8096-ae6256e7d955" path="/var/lib/kubelet/pods/5f944078-d792-4086-8096-ae6256e7d955/volumes" Nov 03 01:52:13.026070 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:52:13.025964 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fabdcaee-grtwc"] Nov 03 01:52:13.030545 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:52:13.030501 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fabdcaee-grtwc"] Nov 03 01:52:13.977421 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:52:13.977380 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a94bb737-6938-44c2-8949-794c4702a4cf" path="/var/lib/kubelet/pods/a94bb737-6938-44c2-8949-794c4702a4cf/volumes" Nov 03 01:52:14.668495 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:52:14.668453 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 01:52:29.020923 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:52:29.020885 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2539e5d6-scqdz"] Nov 03 01:52:29.023747 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:52:29.023718 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2539e5d6-scqdz"] Nov 03 01:52:29.976652 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:52:29.976611 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0143cfca-8bd8-4380-9eda-ff603b205172" path="/var/lib/kubelet/pods/0143cfca-8bd8-4380-9eda-ff603b205172/volumes" Nov 03 01:52:44.668769 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:52:44.668727 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 01:53:14.668503 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:53:14.668457 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 01:53:17.468831 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:53:17.468791 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-jqhsg"] Nov 03 01:53:17.469427 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:53:17.469405 4421 cpu_manager.go:414] "containerMap: removing container" podUID="5f944078-d792-4086-8096-ae6256e7d955" containerName="payload-pre-processing" Nov 03 01:53:17.469524 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:53:17.469429 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f944078-d792-4086-8096-ae6256e7d955" containerName="payload-pre-processing" Nov 03 01:53:17.469524 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:53:17.469457 4421 cpu_manager.go:414] "containerMap: removing container" podUID="17a9712a-5861-4d51-a7bb-4f32c71e9e78" containerName="payload-processing" Nov 03 01:53:17.469524 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:53:17.469465 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17a9712a-5861-4d51-a7bb-4f32c71e9e78" containerName="payload-processing" Nov 03 01:53:17.469704 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:53:17.469591 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="17a9712a-5861-4d51-a7bb-4f32c71e9e78" containerName="payload-processing" Nov 03 01:53:17.469704 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:53:17.469614 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f944078-d792-4086-8096-ae6256e7d955" containerName="payload-pre-processing" Nov 03 01:53:17.472904 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:53:17.472879 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-jqhsg" Nov 03 01:53:17.487920 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:53:17.487881 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:53:17.489188 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:53:17.489162 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 01:53:17.489386 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:53:17.489320 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 01:53:17.489469 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:53:17.489383 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-swfcr\"" Nov 03 01:53:17.491713 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:53:17.491691 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-jqhsg"] Nov 03 01:53:17.570029 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:53:17.569988 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cqn9\" (UniqueName: \"kubernetes.io/projected/d267a793-0377-46a3-8ce5-1e693dd96410-kube-api-access-2cqn9\") pod \"maas-api-revoke-sub-e2547622-jqhsg\" (UID: \"d267a793-0377-46a3-8ce5-1e693dd96410\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-jqhsg" Nov 03 01:53:17.570259 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:53:17.570055 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d267a793-0377-46a3-8ce5-1e693dd96410-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2547622-jqhsg\" (UID: \"d267a793-0377-46a3-8ce5-1e693dd96410\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-jqhsg" Nov 03 01:53:17.570259 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:53:17.570165 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d267a793-0377-46a3-8ce5-1e693dd96410-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2547622-jqhsg\" (UID: \"d267a793-0377-46a3-8ce5-1e693dd96410\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-jqhsg" Nov 03 01:53:17.671487 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:53:17.671452 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d267a793-0377-46a3-8ce5-1e693dd96410-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2547622-jqhsg\" (UID: \"d267a793-0377-46a3-8ce5-1e693dd96410\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-jqhsg" Nov 03 01:53:17.671686 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:53:17.671502 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d267a793-0377-46a3-8ce5-1e693dd96410-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2547622-jqhsg\" (UID: \"d267a793-0377-46a3-8ce5-1e693dd96410\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-jqhsg" Nov 03 01:53:17.671686 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:53:17.671585 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cqn9\" (UniqueName: \"kubernetes.io/projected/d267a793-0377-46a3-8ce5-1e693dd96410-kube-api-access-2cqn9\") pod \"maas-api-revoke-sub-e2547622-jqhsg\" (UID: \"d267a793-0377-46a3-8ce5-1e693dd96410\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-jqhsg" Nov 03 01:53:17.672122 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:53:17.672099 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d267a793-0377-46a3-8ce5-1e693dd96410-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2547622-jqhsg\" (UID: \"d267a793-0377-46a3-8ce5-1e693dd96410\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-jqhsg" Nov 03 01:53:17.674067 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:53:17.674040 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d267a793-0377-46a3-8ce5-1e693dd96410-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2547622-jqhsg\" (UID: \"d267a793-0377-46a3-8ce5-1e693dd96410\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-jqhsg" Nov 03 01:53:17.679978 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:53:17.679949 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cqn9\" (UniqueName: \"kubernetes.io/projected/d267a793-0377-46a3-8ce5-1e693dd96410-kube-api-access-2cqn9\") pod \"maas-api-revoke-sub-e2547622-jqhsg\" (UID: \"d267a793-0377-46a3-8ce5-1e693dd96410\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-jqhsg" Nov 03 01:53:17.786220 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:53:17.786180 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-jqhsg" Nov 03 01:53:18.061610 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:53:18.061583 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-jqhsg"] Nov 03 01:53:18.066291 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:53:18.065861 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd267a793_0377_46a3_8ce5_1e693dd96410.slice/crio-950c0a6b4d88bd579ff45fec7f089574ade697aecda1d59e6cd0b93fc66d3b4b WatchSource:0}: Error finding container 950c0a6b4d88bd579ff45fec7f089574ade697aecda1d59e6cd0b93fc66d3b4b: Status 404 returned error can't find the container with id 950c0a6b4d88bd579ff45fec7f089574ade697aecda1d59e6cd0b93fc66d3b4b Nov 03 01:53:19.078141 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:53:19.078101 4421 generic.go:356] "Generic (PLEG): container finished" podID="d267a793-0377-46a3-8ce5-1e693dd96410" containerID="d291667c50d3fc9ebee497acee01536d77ade76419c838f23c832993425926a2" exitCode=0 Nov 03 01:53:19.078641 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:53:19.078201 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-jqhsg" event={"ID":"d267a793-0377-46a3-8ce5-1e693dd96410","Type":"ContainerDied","Data":"d291667c50d3fc9ebee497acee01536d77ade76419c838f23c832993425926a2"} Nov 03 01:53:19.078641 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:53:19.078335 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-jqhsg" event={"ID":"d267a793-0377-46a3-8ce5-1e693dd96410","Type":"ContainerStarted","Data":"950c0a6b4d88bd579ff45fec7f089574ade697aecda1d59e6cd0b93fc66d3b4b"} Nov 03 01:53:20.221950 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:53:20.221924 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-jqhsg" Nov 03 01:53:20.293361 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:53:20.293321 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d267a793-0377-46a3-8ce5-1e693dd96410-kube-api-access-2cqn9\" (UniqueName: \"kubernetes.io/projected/d267a793-0377-46a3-8ce5-1e693dd96410-kube-api-access-2cqn9\") pod \"d267a793-0377-46a3-8ce5-1e693dd96410\" (UID: \"d267a793-0377-46a3-8ce5-1e693dd96410\") " Nov 03 01:53:20.293361 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:53:20.293365 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d267a793-0377-46a3-8ce5-1e693dd96410-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d267a793-0377-46a3-8ce5-1e693dd96410-maas-api-xks-ca\") pod \"d267a793-0377-46a3-8ce5-1e693dd96410\" (UID: \"d267a793-0377-46a3-8ce5-1e693dd96410\") " Nov 03 01:53:20.293584 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:53:20.293522 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d267a793-0377-46a3-8ce5-1e693dd96410-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d267a793-0377-46a3-8ce5-1e693dd96410-maas-api-openshift-ca\") pod \"d267a793-0377-46a3-8ce5-1e693dd96410\" (UID: \"d267a793-0377-46a3-8ce5-1e693dd96410\") " Nov 03 01:53:20.293905 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:53:20.293875 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d267a793-0377-46a3-8ce5-1e693dd96410-maas-api-openshift-ca" pod "d267a793-0377-46a3-8ce5-1e693dd96410" (UID: "d267a793-0377-46a3-8ce5-1e693dd96410"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:53:20.295493 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:53:20.295474 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d267a793-0377-46a3-8ce5-1e693dd96410-kube-api-access-2cqn9" pod "d267a793-0377-46a3-8ce5-1e693dd96410" (UID: "d267a793-0377-46a3-8ce5-1e693dd96410"). InnerVolumeSpecName "kube-api-access-2cqn9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:53:20.295600 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:53:20.295496 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d267a793-0377-46a3-8ce5-1e693dd96410-maas-api-xks-ca" pod "d267a793-0377-46a3-8ce5-1e693dd96410" (UID: "d267a793-0377-46a3-8ce5-1e693dd96410"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:53:20.394566 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:53:20.394485 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d267a793-0377-46a3-8ce5-1e693dd96410-maas-api-openshift-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:53:20.394566 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:53:20.394515 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2cqn9\" (UniqueName: \"kubernetes.io/projected/d267a793-0377-46a3-8ce5-1e693dd96410-kube-api-access-2cqn9\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:53:20.394566 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:53:20.394531 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d267a793-0377-46a3-8ce5-1e693dd96410-maas-api-xks-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:53:21.089996 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:53:21.089954 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-jqhsg" event={"ID":"d267a793-0377-46a3-8ce5-1e693dd96410","Type":"ContainerDied","Data":"950c0a6b4d88bd579ff45fec7f089574ade697aecda1d59e6cd0b93fc66d3b4b"} Nov 03 01:53:21.090204 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:53:21.090020 4421 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="950c0a6b4d88bd579ff45fec7f089574ade697aecda1d59e6cd0b93fc66d3b4b" Nov 03 01:53:21.090204 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:53:21.090092 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-jqhsg" Nov 03 01:53:35.501135 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:53:35.501082 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-kws96"] Nov 03 01:53:35.501721 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:53:35.501531 4421 cpu_manager.go:414] "containerMap: removing container" podUID="d267a793-0377-46a3-8ce5-1e693dd96410" containerName="revoke-keys" Nov 03 01:53:35.501721 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:53:35.501547 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d267a793-0377-46a3-8ce5-1e693dd96410" containerName="revoke-keys" Nov 03 01:53:35.501721 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:53:35.501656 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="d267a793-0377-46a3-8ce5-1e693dd96410" containerName="revoke-keys" Nov 03 01:53:35.505083 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:53:35.505059 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-kws96" Nov 03 01:53:35.513564 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:53:35.513534 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 01:53:35.520859 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:53:35.513752 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-swfcr\"" Nov 03 01:53:35.521067 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:53:35.513833 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 01:53:35.521176 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:53:35.515456 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:53:35.521535 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:53:35.521510 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-kws96"] Nov 03 01:53:35.590236 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:53:35.589922 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7w9s\" (UniqueName: \"kubernetes.io/projected/3a177155-0959-45a2-a9c4-e5802e34891f-kube-api-access-s7w9s\") pod \"maas-api-revoke-sub-c1e271a9-kws96\" (UID: \"3a177155-0959-45a2-a9c4-e5802e34891f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-kws96" Nov 03 01:53:35.590236 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:53:35.590005 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a177155-0959-45a2-a9c4-e5802e34891f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-kws96\" (UID: \"3a177155-0959-45a2-a9c4-e5802e34891f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-kws96" Nov 03 01:53:35.590236 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:53:35.590055 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a177155-0959-45a2-a9c4-e5802e34891f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-kws96\" (UID: \"3a177155-0959-45a2-a9c4-e5802e34891f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-kws96" Nov 03 01:53:35.690984 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:53:35.690937 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a177155-0959-45a2-a9c4-e5802e34891f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-kws96\" (UID: \"3a177155-0959-45a2-a9c4-e5802e34891f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-kws96" Nov 03 01:53:35.691199 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:53:35.691038 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7w9s\" (UniqueName: \"kubernetes.io/projected/3a177155-0959-45a2-a9c4-e5802e34891f-kube-api-access-s7w9s\") pod \"maas-api-revoke-sub-c1e271a9-kws96\" (UID: \"3a177155-0959-45a2-a9c4-e5802e34891f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-kws96" Nov 03 01:53:35.691199 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:53:35.691077 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a177155-0959-45a2-a9c4-e5802e34891f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-kws96\" (UID: \"3a177155-0959-45a2-a9c4-e5802e34891f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-kws96" Nov 03 01:53:35.692234 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:53:35.692201 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a177155-0959-45a2-a9c4-e5802e34891f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-kws96\" (UID: \"3a177155-0959-45a2-a9c4-e5802e34891f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-kws96" Nov 03 01:53:35.695092 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:53:35.695061 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a177155-0959-45a2-a9c4-e5802e34891f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-kws96\" (UID: \"3a177155-0959-45a2-a9c4-e5802e34891f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-kws96" Nov 03 01:53:35.699833 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:53:35.699798 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7w9s\" (UniqueName: \"kubernetes.io/projected/3a177155-0959-45a2-a9c4-e5802e34891f-kube-api-access-s7w9s\") pod \"maas-api-revoke-sub-c1e271a9-kws96\" (UID: \"3a177155-0959-45a2-a9c4-e5802e34891f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-kws96" Nov 03 01:53:35.884968 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:53:35.884859 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-kws96" Nov 03 01:53:36.278619 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:53:36.278516 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-kws96"] Nov 03 01:53:37.296086 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:53:37.296035 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-kws96" event={"ID":"3a177155-0959-45a2-a9c4-e5802e34891f","Type":"ContainerStarted","Data":"2aea7dae110b026663e11754c5e98d0b6b019ef25d8d8923e9efa43b15f5f8f0"} Nov 03 01:53:37.296549 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:53:37.296105 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-kws96" event={"ID":"3a177155-0959-45a2-a9c4-e5802e34891f","Type":"ContainerStarted","Data":"6c8e3c611178951e0cff3ddce92fc698a582193a2d9dfee8906fa614fa72b2bd"} Nov 03 01:53:37.312419 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:53:37.312357 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-kws96" podStartSLOduration=2.312341832 podStartE2EDuration="2.312341832s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:53:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:53:37.311919194 +0000 UTC m=+4359.482922294" watchObservedRunningTime="2026-11-03 01:53:37.312341832 +0000 UTC m=+4359.483344928" Nov 03 01:53:38.302853 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:53:38.302820 4421 generic.go:356] "Generic (PLEG): container finished" podID="3a177155-0959-45a2-a9c4-e5802e34891f" containerID="2aea7dae110b026663e11754c5e98d0b6b019ef25d8d8923e9efa43b15f5f8f0" exitCode=0 Nov 03 01:53:38.303296 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:53:38.302898 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-kws96" event={"ID":"3a177155-0959-45a2-a9c4-e5802e34891f","Type":"ContainerDied","Data":"2aea7dae110b026663e11754c5e98d0b6b019ef25d8d8923e9efa43b15f5f8f0"} Nov 03 01:53:39.448200 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:53:39.448171 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-kws96" Nov 03 01:53:39.541191 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:53:39.541150 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a177155-0959-45a2-a9c4-e5802e34891f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a177155-0959-45a2-a9c4-e5802e34891f-maas-api-xks-ca\") pod \"3a177155-0959-45a2-a9c4-e5802e34891f\" (UID: \"3a177155-0959-45a2-a9c4-e5802e34891f\") " Nov 03 01:53:39.541416 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:53:39.541244 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a177155-0959-45a2-a9c4-e5802e34891f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a177155-0959-45a2-a9c4-e5802e34891f-maas-api-openshift-ca\") pod \"3a177155-0959-45a2-a9c4-e5802e34891f\" (UID: \"3a177155-0959-45a2-a9c4-e5802e34891f\") " Nov 03 01:53:39.541416 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:53:39.541371 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a177155-0959-45a2-a9c4-e5802e34891f-kube-api-access-s7w9s\" (UniqueName: \"kubernetes.io/projected/3a177155-0959-45a2-a9c4-e5802e34891f-kube-api-access-s7w9s\") pod \"3a177155-0959-45a2-a9c4-e5802e34891f\" (UID: \"3a177155-0959-45a2-a9c4-e5802e34891f\") " Nov 03 01:53:39.541619 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:53:39.541595 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a177155-0959-45a2-a9c4-e5802e34891f-maas-api-openshift-ca" pod "3a177155-0959-45a2-a9c4-e5802e34891f" (UID: "3a177155-0959-45a2-a9c4-e5802e34891f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:53:39.541759 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:53:39.541738 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a177155-0959-45a2-a9c4-e5802e34891f-maas-api-openshift-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:53:39.543460 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:53:39.543435 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a177155-0959-45a2-a9c4-e5802e34891f-maas-api-xks-ca" pod "3a177155-0959-45a2-a9c4-e5802e34891f" (UID: "3a177155-0959-45a2-a9c4-e5802e34891f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:53:39.543662 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:53:39.543638 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a177155-0959-45a2-a9c4-e5802e34891f-kube-api-access-s7w9s" pod "3a177155-0959-45a2-a9c4-e5802e34891f" (UID: "3a177155-0959-45a2-a9c4-e5802e34891f"). InnerVolumeSpecName "kube-api-access-s7w9s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:53:39.642388 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:53:39.642256 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a177155-0959-45a2-a9c4-e5802e34891f-maas-api-xks-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:53:39.642388 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:53:39.642331 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7w9s\" (UniqueName: \"kubernetes.io/projected/3a177155-0959-45a2-a9c4-e5802e34891f-kube-api-access-s7w9s\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:53:40.314574 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:53:40.314539 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-kws96" Nov 03 01:53:40.314574 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:53:40.314558 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-kws96" event={"ID":"3a177155-0959-45a2-a9c4-e5802e34891f","Type":"ContainerDied","Data":"6c8e3c611178951e0cff3ddce92fc698a582193a2d9dfee8906fa614fa72b2bd"} Nov 03 01:53:40.314807 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:53:40.314614 4421 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6c8e3c611178951e0cff3ddce92fc698a582193a2d9dfee8906fa614fa72b2bd" Nov 03 01:53:44.669865 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:53:44.669828 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 01:53:51.874803 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:53:51.874763 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-nrpfp"] Nov 03 01:53:51.875412 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:53:51.875188 4421 cpu_manager.go:414] "containerMap: removing container" podUID="3a177155-0959-45a2-a9c4-e5802e34891f" containerName="revoke-keys" Nov 03 01:53:51.875412 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:53:51.875200 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a177155-0959-45a2-a9c4-e5802e34891f" containerName="revoke-keys" Nov 03 01:53:51.875412 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:53:51.875267 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a177155-0959-45a2-a9c4-e5802e34891f" containerName="revoke-keys" Nov 03 01:53:51.878792 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:53:51.878769 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-nrpfp" Nov 03 01:53:51.886885 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:53:51.886860 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:53:51.886975 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:53:51.886953 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-swfcr\"" Nov 03 01:53:51.886975 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:53:51.886958 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 01:53:51.887089 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:53:51.886981 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 01:53:51.887708 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:53:51.887675 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-nrpfp"] Nov 03 01:53:51.998364 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:53:51.998327 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tljtj\" (UniqueName: \"kubernetes.io/projected/f4260979-c90d-4671-9c5b-1bc1e9481b30-kube-api-access-tljtj\") pod \"maas-api-revoke-sub-2e1f5b5f-nrpfp\" (UID: \"f4260979-c90d-4671-9c5b-1bc1e9481b30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-nrpfp" Nov 03 01:53:51.998575 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:53:51.998388 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4260979-c90d-4671-9c5b-1bc1e9481b30-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-nrpfp\" (UID: \"f4260979-c90d-4671-9c5b-1bc1e9481b30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-nrpfp" Nov 03 01:53:51.998575 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:53:51.998456 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4260979-c90d-4671-9c5b-1bc1e9481b30-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-nrpfp\" (UID: \"f4260979-c90d-4671-9c5b-1bc1e9481b30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-nrpfp" Nov 03 01:53:52.099481 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:53:52.099435 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tljtj\" (UniqueName: \"kubernetes.io/projected/f4260979-c90d-4671-9c5b-1bc1e9481b30-kube-api-access-tljtj\") pod \"maas-api-revoke-sub-2e1f5b5f-nrpfp\" (UID: \"f4260979-c90d-4671-9c5b-1bc1e9481b30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-nrpfp" Nov 03 01:53:52.099710 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:53:52.099526 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4260979-c90d-4671-9c5b-1bc1e9481b30-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-nrpfp\" (UID: \"f4260979-c90d-4671-9c5b-1bc1e9481b30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-nrpfp" Nov 03 01:53:52.099710 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:53:52.099552 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4260979-c90d-4671-9c5b-1bc1e9481b30-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-nrpfp\" (UID: \"f4260979-c90d-4671-9c5b-1bc1e9481b30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-nrpfp" Nov 03 01:53:52.100386 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:53:52.100353 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4260979-c90d-4671-9c5b-1bc1e9481b30-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-nrpfp\" (UID: \"f4260979-c90d-4671-9c5b-1bc1e9481b30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-nrpfp" Nov 03 01:53:52.102121 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:53:52.102096 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4260979-c90d-4671-9c5b-1bc1e9481b30-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-nrpfp\" (UID: \"f4260979-c90d-4671-9c5b-1bc1e9481b30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-nrpfp" Nov 03 01:53:52.108166 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:53:52.108137 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tljtj\" (UniqueName: \"kubernetes.io/projected/f4260979-c90d-4671-9c5b-1bc1e9481b30-kube-api-access-tljtj\") pod \"maas-api-revoke-sub-2e1f5b5f-nrpfp\" (UID: \"f4260979-c90d-4671-9c5b-1bc1e9481b30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-nrpfp" Nov 03 01:53:52.195793 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:53:52.195687 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-nrpfp" Nov 03 01:53:52.471202 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:53:52.471167 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-nrpfp"] Nov 03 01:53:52.474220 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:53:52.474192 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4260979_c90d_4671_9c5b_1bc1e9481b30.slice/crio-4933a5a16e156d3ea19bf14d2a6d677b118ffd2d8ebfecc4ce9061d1d72d2e6a WatchSource:0}: Error finding container 4933a5a16e156d3ea19bf14d2a6d677b118ffd2d8ebfecc4ce9061d1d72d2e6a: Status 404 returned error can't find the container with id 4933a5a16e156d3ea19bf14d2a6d677b118ffd2d8ebfecc4ce9061d1d72d2e6a Nov 03 01:53:53.394142 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:53:53.394106 4421 generic.go:356] "Generic (PLEG): container finished" podID="f4260979-c90d-4671-9c5b-1bc1e9481b30" containerID="d09130e07e680fe84116b1af0469f3c7e5af408ef46e370ae49838bba0ace23c" exitCode=0 Nov 03 01:53:53.394597 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:53:53.394190 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-nrpfp" event={"ID":"f4260979-c90d-4671-9c5b-1bc1e9481b30","Type":"ContainerDied","Data":"d09130e07e680fe84116b1af0469f3c7e5af408ef46e370ae49838bba0ace23c"} Nov 03 01:53:53.394597 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:53:53.394239 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-nrpfp" event={"ID":"f4260979-c90d-4671-9c5b-1bc1e9481b30","Type":"ContainerStarted","Data":"4933a5a16e156d3ea19bf14d2a6d677b118ffd2d8ebfecc4ce9061d1d72d2e6a"} Nov 03 01:53:54.537838 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:53:54.537809 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-nrpfp" Nov 03 01:53:54.626460 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:53:54.626420 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f4260979-c90d-4671-9c5b-1bc1e9481b30-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4260979-c90d-4671-9c5b-1bc1e9481b30-maas-api-xks-ca\") pod \"f4260979-c90d-4671-9c5b-1bc1e9481b30\" (UID: \"f4260979-c90d-4671-9c5b-1bc1e9481b30\") " Nov 03 01:53:54.626660 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:53:54.626469 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4260979-c90d-4671-9c5b-1bc1e9481b30-kube-api-access-tljtj\" (UniqueName: \"kubernetes.io/projected/f4260979-c90d-4671-9c5b-1bc1e9481b30-kube-api-access-tljtj\") pod \"f4260979-c90d-4671-9c5b-1bc1e9481b30\" (UID: \"f4260979-c90d-4671-9c5b-1bc1e9481b30\") " Nov 03 01:53:54.626660 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:53:54.626560 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4260979-c90d-4671-9c5b-1bc1e9481b30-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4260979-c90d-4671-9c5b-1bc1e9481b30-maas-api-openshift-ca\") pod \"f4260979-c90d-4671-9c5b-1bc1e9481b30\" (UID: \"f4260979-c90d-4671-9c5b-1bc1e9481b30\") " Nov 03 01:53:54.626968 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:53:54.626943 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4260979-c90d-4671-9c5b-1bc1e9481b30-maas-api-openshift-ca" pod "f4260979-c90d-4671-9c5b-1bc1e9481b30" (UID: "f4260979-c90d-4671-9c5b-1bc1e9481b30"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:53:54.628634 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:53:54.628607 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4260979-c90d-4671-9c5b-1bc1e9481b30-kube-api-access-tljtj" pod "f4260979-c90d-4671-9c5b-1bc1e9481b30" (UID: "f4260979-c90d-4671-9c5b-1bc1e9481b30"). InnerVolumeSpecName "kube-api-access-tljtj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:53:54.628724 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:53:54.628705 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4260979-c90d-4671-9c5b-1bc1e9481b30-maas-api-xks-ca" pod "f4260979-c90d-4671-9c5b-1bc1e9481b30" (UID: "f4260979-c90d-4671-9c5b-1bc1e9481b30"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:53:54.728137 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:53:54.728045 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tljtj\" (UniqueName: \"kubernetes.io/projected/f4260979-c90d-4671-9c5b-1bc1e9481b30-kube-api-access-tljtj\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:53:54.728137 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:53:54.728078 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4260979-c90d-4671-9c5b-1bc1e9481b30-maas-api-openshift-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:53:54.728137 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:53:54.728088 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4260979-c90d-4671-9c5b-1bc1e9481b30-maas-api-xks-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:53:55.405497 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:53:55.405456 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-nrpfp" event={"ID":"f4260979-c90d-4671-9c5b-1bc1e9481b30","Type":"ContainerDied","Data":"4933a5a16e156d3ea19bf14d2a6d677b118ffd2d8ebfecc4ce9061d1d72d2e6a"} Nov 03 01:53:55.405497 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:53:55.405480 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-nrpfp" Nov 03 01:53:55.405733 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:53:55.405516 4421 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4933a5a16e156d3ea19bf14d2a6d677b118ffd2d8ebfecc4ce9061d1d72d2e6a" Nov 03 01:54:02.901918 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:54:02.901871 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-txdgp"] Nov 03 01:54:02.902493 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:54:02.902318 4421 cpu_manager.go:414] "containerMap: removing container" podUID="f4260979-c90d-4671-9c5b-1bc1e9481b30" containerName="revoke-keys" Nov 03 01:54:02.902493 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:54:02.902335 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4260979-c90d-4671-9c5b-1bc1e9481b30" containerName="revoke-keys" Nov 03 01:54:02.902493 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:54:02.902435 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4260979-c90d-4671-9c5b-1bc1e9481b30" containerName="revoke-keys" Nov 03 01:54:02.905746 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:54:02.905720 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-txdgp" Nov 03 01:54:02.911394 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:54:02.911354 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:54:02.911394 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:54:02.911354 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 01:54:02.911604 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:54:02.911420 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-swfcr\"" Nov 03 01:54:02.911604 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:54:02.911473 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 01:54:02.914676 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:54:02.914651 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-txdgp"] Nov 03 01:54:03.007901 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:54:03.007852 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1749c24-bedc-4256-b826-21a75c3604f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-txdgp\" (UID: \"e1749c24-bedc-4256-b826-21a75c3604f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-txdgp" Nov 03 01:54:03.007901 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:54:03.007903 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1749c24-bedc-4256-b826-21a75c3604f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-txdgp\" (UID: \"e1749c24-bedc-4256-b826-21a75c3604f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-txdgp" Nov 03 01:54:03.008148 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:54:03.008066 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvwf6\" (UniqueName: \"kubernetes.io/projected/e1749c24-bedc-4256-b826-21a75c3604f4-kube-api-access-gvwf6\") pod \"maas-api-revoke-sub-ec843018-txdgp\" (UID: \"e1749c24-bedc-4256-b826-21a75c3604f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-txdgp" Nov 03 01:54:03.109291 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:54:03.109239 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1749c24-bedc-4256-b826-21a75c3604f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-txdgp\" (UID: \"e1749c24-bedc-4256-b826-21a75c3604f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-txdgp" Nov 03 01:54:03.109557 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:54:03.109304 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1749c24-bedc-4256-b826-21a75c3604f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-txdgp\" (UID: \"e1749c24-bedc-4256-b826-21a75c3604f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-txdgp" Nov 03 01:54:03.109557 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:54:03.109406 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvwf6\" (UniqueName: \"kubernetes.io/projected/e1749c24-bedc-4256-b826-21a75c3604f4-kube-api-access-gvwf6\") pod \"maas-api-revoke-sub-ec843018-txdgp\" (UID: \"e1749c24-bedc-4256-b826-21a75c3604f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-txdgp" Nov 03 01:54:03.110095 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:54:03.110065 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1749c24-bedc-4256-b826-21a75c3604f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-txdgp\" (UID: \"e1749c24-bedc-4256-b826-21a75c3604f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-txdgp" Nov 03 01:54:03.111756 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:54:03.111723 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1749c24-bedc-4256-b826-21a75c3604f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-txdgp\" (UID: \"e1749c24-bedc-4256-b826-21a75c3604f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-txdgp" Nov 03 01:54:03.117546 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:54:03.117523 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvwf6\" (UniqueName: \"kubernetes.io/projected/e1749c24-bedc-4256-b826-21a75c3604f4-kube-api-access-gvwf6\") pod \"maas-api-revoke-sub-ec843018-txdgp\" (UID: \"e1749c24-bedc-4256-b826-21a75c3604f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-txdgp" Nov 03 01:54:03.221157 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:54:03.221070 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-txdgp" Nov 03 01:54:03.515773 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:54:03.515730 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-txdgp"] Nov 03 01:54:03.518588 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:54:03.518557 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1749c24_bedc_4256_b826_21a75c3604f4.slice/crio-e62cb0c6ed85fa33fdcfec3b0b5b5e15129ce070a70ed5caa618b07ad09dc82b WatchSource:0}: Error finding container e62cb0c6ed85fa33fdcfec3b0b5b5e15129ce070a70ed5caa618b07ad09dc82b: Status 404 returned error can't find the container with id e62cb0c6ed85fa33fdcfec3b0b5b5e15129ce070a70ed5caa618b07ad09dc82b Nov 03 01:54:04.451342 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:54:04.451303 4421 generic.go:356] "Generic (PLEG): container finished" podID="e1749c24-bedc-4256-b826-21a75c3604f4" containerID="c169e086d6832d2ff9e6c97eb66de6005fcbd8b10be463ce65268a8ec5783dc4" exitCode=0 Nov 03 01:54:04.451758 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:54:04.451360 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-txdgp" event={"ID":"e1749c24-bedc-4256-b826-21a75c3604f4","Type":"ContainerDied","Data":"c169e086d6832d2ff9e6c97eb66de6005fcbd8b10be463ce65268a8ec5783dc4"} Nov 03 01:54:04.451758 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:54:04.451407 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-txdgp" event={"ID":"e1749c24-bedc-4256-b826-21a75c3604f4","Type":"ContainerStarted","Data":"e62cb0c6ed85fa33fdcfec3b0b5b5e15129ce070a70ed5caa618b07ad09dc82b"} Nov 03 01:54:05.640882 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:54:05.640854 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-txdgp" Nov 03 01:54:05.735132 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:54:05.735027 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e1749c24-bedc-4256-b826-21a75c3604f4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1749c24-bedc-4256-b826-21a75c3604f4-maas-api-xks-ca\") pod \"e1749c24-bedc-4256-b826-21a75c3604f4\" (UID: \"e1749c24-bedc-4256-b826-21a75c3604f4\") " Nov 03 01:54:05.735348 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:54:05.735131 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1749c24-bedc-4256-b826-21a75c3604f4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1749c24-bedc-4256-b826-21a75c3604f4-maas-api-openshift-ca\") pod \"e1749c24-bedc-4256-b826-21a75c3604f4\" (UID: \"e1749c24-bedc-4256-b826-21a75c3604f4\") " Nov 03 01:54:05.735348 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:54:05.735200 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1749c24-bedc-4256-b826-21a75c3604f4-kube-api-access-gvwf6\" (UniqueName: \"kubernetes.io/projected/e1749c24-bedc-4256-b826-21a75c3604f4-kube-api-access-gvwf6\") pod \"e1749c24-bedc-4256-b826-21a75c3604f4\" (UID: \"e1749c24-bedc-4256-b826-21a75c3604f4\") " Nov 03 01:54:05.735858 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:54:05.735813 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1749c24-bedc-4256-b826-21a75c3604f4-maas-api-openshift-ca" pod "e1749c24-bedc-4256-b826-21a75c3604f4" (UID: "e1749c24-bedc-4256-b826-21a75c3604f4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:54:05.737795 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:54:05.737723 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1749c24-bedc-4256-b826-21a75c3604f4-kube-api-access-gvwf6" pod "e1749c24-bedc-4256-b826-21a75c3604f4" (UID: "e1749c24-bedc-4256-b826-21a75c3604f4"). InnerVolumeSpecName "kube-api-access-gvwf6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:54:05.738647 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:54:05.738625 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1749c24-bedc-4256-b826-21a75c3604f4-maas-api-xks-ca" pod "e1749c24-bedc-4256-b826-21a75c3604f4" (UID: "e1749c24-bedc-4256-b826-21a75c3604f4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:54:05.841297 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:54:05.840924 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1749c24-bedc-4256-b826-21a75c3604f4-maas-api-xks-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:54:05.841297 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:54:05.840972 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1749c24-bedc-4256-b826-21a75c3604f4-maas-api-openshift-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:54:05.841297 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:54:05.840992 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvwf6\" (UniqueName: \"kubernetes.io/projected/e1749c24-bedc-4256-b826-21a75c3604f4-kube-api-access-gvwf6\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:54:06.477427 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:54:06.477385 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-txdgp" event={"ID":"e1749c24-bedc-4256-b826-21a75c3604f4","Type":"ContainerDied","Data":"e62cb0c6ed85fa33fdcfec3b0b5b5e15129ce070a70ed5caa618b07ad09dc82b"} Nov 03 01:54:06.477676 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:54:06.477660 4421 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e62cb0c6ed85fa33fdcfec3b0b5b5e15129ce070a70ed5caa618b07ad09dc82b" Nov 03 01:54:06.477881 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:54:06.477867 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-txdgp" Nov 03 01:54:07.132590 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:54:07.132556 4421 scope.go:122] "RemoveContainer" containerID="834980a7c4244550679703c544e140f1873d945934b08bb1446ff5a7369c591b" Nov 03 01:54:07.183345 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:54:07.181230 4421 scope.go:122] "RemoveContainer" containerID="b27e94badf4be2dada2012713ae3ba095938cbc4f420e9160cd630235bcc2458" Nov 03 01:54:14.672050 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:54:14.672008 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 01:54:29.109780 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:54:29.109740 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-p67d6"] Nov 03 01:54:29.110381 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:54:29.110157 4421 cpu_manager.go:414] "containerMap: removing container" podUID="e1749c24-bedc-4256-b826-21a75c3604f4" containerName="revoke-keys" Nov 03 01:54:29.110381 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:54:29.110170 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1749c24-bedc-4256-b826-21a75c3604f4" containerName="revoke-keys" Nov 03 01:54:29.110381 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:54:29.110238 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1749c24-bedc-4256-b826-21a75c3604f4" containerName="revoke-keys" Nov 03 01:54:29.113717 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:54:29.113693 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-p67d6" Nov 03 01:54:29.119074 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:54:29.119001 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:54:29.119242 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:54:29.119106 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 01:54:29.119242 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:54:29.119110 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 01:54:29.123860 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:54:29.123818 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-p67d6"] Nov 03 01:54:29.124046 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:54:29.124027 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-swfcr\"" Nov 03 01:54:29.208320 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:54:29.208261 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vz4tr\" (UniqueName: \"kubernetes.io/projected/8963e105-fbc7-4218-944c-8f67b2444ffe-kube-api-access-vz4tr\") pod \"maas-api-revoke-sub-046e1343-p67d6\" (UID: \"8963e105-fbc7-4218-944c-8f67b2444ffe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-p67d6" Nov 03 01:54:29.208494 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:54:29.208335 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8963e105-fbc7-4218-944c-8f67b2444ffe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-p67d6\" (UID: \"8963e105-fbc7-4218-944c-8f67b2444ffe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-p67d6" Nov 03 01:54:29.208541 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:54:29.208494 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8963e105-fbc7-4218-944c-8f67b2444ffe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-p67d6\" (UID: \"8963e105-fbc7-4218-944c-8f67b2444ffe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-p67d6" Nov 03 01:54:29.309349 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:54:29.309312 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vz4tr\" (UniqueName: \"kubernetes.io/projected/8963e105-fbc7-4218-944c-8f67b2444ffe-kube-api-access-vz4tr\") pod \"maas-api-revoke-sub-046e1343-p67d6\" (UID: \"8963e105-fbc7-4218-944c-8f67b2444ffe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-p67d6" Nov 03 01:54:29.309535 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:54:29.309418 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8963e105-fbc7-4218-944c-8f67b2444ffe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-p67d6\" (UID: \"8963e105-fbc7-4218-944c-8f67b2444ffe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-p67d6" Nov 03 01:54:29.309535 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:54:29.309526 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8963e105-fbc7-4218-944c-8f67b2444ffe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-p67d6\" (UID: \"8963e105-fbc7-4218-944c-8f67b2444ffe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-p67d6" Nov 03 01:54:29.310188 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:54:29.310164 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8963e105-fbc7-4218-944c-8f67b2444ffe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-p67d6\" (UID: \"8963e105-fbc7-4218-944c-8f67b2444ffe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-p67d6" Nov 03 01:54:29.311898 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:54:29.311867 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8963e105-fbc7-4218-944c-8f67b2444ffe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-p67d6\" (UID: \"8963e105-fbc7-4218-944c-8f67b2444ffe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-p67d6" Nov 03 01:54:29.317922 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:54:29.317900 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vz4tr\" (UniqueName: \"kubernetes.io/projected/8963e105-fbc7-4218-944c-8f67b2444ffe-kube-api-access-vz4tr\") pod \"maas-api-revoke-sub-046e1343-p67d6\" (UID: \"8963e105-fbc7-4218-944c-8f67b2444ffe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-p67d6" Nov 03 01:54:29.429235 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:54:29.429144 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-p67d6" Nov 03 01:54:29.702806 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:54:29.702582 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-p67d6"] Nov 03 01:54:29.705821 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:54:29.705790 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8963e105_fbc7_4218_944c_8f67b2444ffe.slice/crio-b54628e9e4d2d72af2dbffca9a2ae51debb309e821d3701cfd64259c1a80d064 WatchSource:0}: Error finding container b54628e9e4d2d72af2dbffca9a2ae51debb309e821d3701cfd64259c1a80d064: Status 404 returned error can't find the container with id b54628e9e4d2d72af2dbffca9a2ae51debb309e821d3701cfd64259c1a80d064 Nov 03 01:54:30.611766 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:54:30.611732 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-p67d6" event={"ID":"8963e105-fbc7-4218-944c-8f67b2444ffe","Type":"ContainerStarted","Data":"bda0f63a91fc43a277504c7bcd516bea28915efb13f71c0bdb103539ca1b99f7"} Nov 03 01:54:30.612161 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:54:30.611817 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-p67d6" event={"ID":"8963e105-fbc7-4218-944c-8f67b2444ffe","Type":"ContainerStarted","Data":"b54628e9e4d2d72af2dbffca9a2ae51debb309e821d3701cfd64259c1a80d064"} Nov 03 01:54:30.629687 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:54:30.629641 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-p67d6" podStartSLOduration=1.629628636 podStartE2EDuration="1.629628636s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:54:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:54:30.629243036 +0000 UTC m=+4412.800246133" watchObservedRunningTime="2026-11-03 01:54:30.629628636 +0000 UTC m=+4412.800631773" Nov 03 01:54:31.618222 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:54:31.618183 4421 generic.go:356] "Generic (PLEG): container finished" podID="8963e105-fbc7-4218-944c-8f67b2444ffe" containerID="bda0f63a91fc43a277504c7bcd516bea28915efb13f71c0bdb103539ca1b99f7" exitCode=0 Nov 03 01:54:31.618680 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:54:31.618240 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-p67d6" event={"ID":"8963e105-fbc7-4218-944c-8f67b2444ffe","Type":"ContainerDied","Data":"bda0f63a91fc43a277504c7bcd516bea28915efb13f71c0bdb103539ca1b99f7"} Nov 03 01:54:32.769545 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:54:32.769519 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-p67d6" Nov 03 01:54:32.845223 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:54:32.845185 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8963e105-fbc7-4218-944c-8f67b2444ffe-kube-api-access-vz4tr\" (UniqueName: \"kubernetes.io/projected/8963e105-fbc7-4218-944c-8f67b2444ffe-kube-api-access-vz4tr\") pod \"8963e105-fbc7-4218-944c-8f67b2444ffe\" (UID: \"8963e105-fbc7-4218-944c-8f67b2444ffe\") " Nov 03 01:54:32.845473 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:54:32.845249 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8963e105-fbc7-4218-944c-8f67b2444ffe-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8963e105-fbc7-4218-944c-8f67b2444ffe-maas-api-xks-ca\") pod \"8963e105-fbc7-4218-944c-8f67b2444ffe\" (UID: \"8963e105-fbc7-4218-944c-8f67b2444ffe\") " Nov 03 01:54:32.845473 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:54:32.845351 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8963e105-fbc7-4218-944c-8f67b2444ffe-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8963e105-fbc7-4218-944c-8f67b2444ffe-maas-api-openshift-ca\") pod \"8963e105-fbc7-4218-944c-8f67b2444ffe\" (UID: \"8963e105-fbc7-4218-944c-8f67b2444ffe\") " Nov 03 01:54:32.845702 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:54:32.845676 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8963e105-fbc7-4218-944c-8f67b2444ffe-maas-api-openshift-ca" pod "8963e105-fbc7-4218-944c-8f67b2444ffe" (UID: "8963e105-fbc7-4218-944c-8f67b2444ffe"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:54:32.847465 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:54:32.847432 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8963e105-fbc7-4218-944c-8f67b2444ffe-maas-api-xks-ca" pod "8963e105-fbc7-4218-944c-8f67b2444ffe" (UID: "8963e105-fbc7-4218-944c-8f67b2444ffe"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:54:32.847569 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:54:32.847470 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8963e105-fbc7-4218-944c-8f67b2444ffe-kube-api-access-vz4tr" pod "8963e105-fbc7-4218-944c-8f67b2444ffe" (UID: "8963e105-fbc7-4218-944c-8f67b2444ffe"). InnerVolumeSpecName "kube-api-access-vz4tr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:54:32.946864 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:54:32.946779 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vz4tr\" (UniqueName: \"kubernetes.io/projected/8963e105-fbc7-4218-944c-8f67b2444ffe-kube-api-access-vz4tr\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:54:32.946864 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:54:32.946811 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8963e105-fbc7-4218-944c-8f67b2444ffe-maas-api-xks-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:54:32.946864 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:54:32.946821 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8963e105-fbc7-4218-944c-8f67b2444ffe-maas-api-openshift-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:54:33.630011 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:54:33.629975 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-p67d6" event={"ID":"8963e105-fbc7-4218-944c-8f67b2444ffe","Type":"ContainerDied","Data":"b54628e9e4d2d72af2dbffca9a2ae51debb309e821d3701cfd64259c1a80d064"} Nov 03 01:54:33.630210 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:54:33.630036 4421 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b54628e9e4d2d72af2dbffca9a2ae51debb309e821d3701cfd64259c1a80d064" Nov 03 01:54:33.630210 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:54:33.629992 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-p67d6" Nov 03 01:54:39.317212 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:54:39.317179 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8wz5k"] Nov 03 01:54:39.317650 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:54:39.317588 4421 cpu_manager.go:414] "containerMap: removing container" podUID="8963e105-fbc7-4218-944c-8f67b2444ffe" containerName="revoke-keys" Nov 03 01:54:39.317650 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:54:39.317599 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8963e105-fbc7-4218-944c-8f67b2444ffe" containerName="revoke-keys" Nov 03 01:54:39.317743 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:54:39.317674 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="8963e105-fbc7-4218-944c-8f67b2444ffe" containerName="revoke-keys" Nov 03 01:54:39.320757 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:54:39.320737 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8wz5k" Nov 03 01:54:39.327201 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:54:39.327162 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:54:39.327201 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:54:39.327183 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 01:54:39.327527 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:54:39.327206 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 01:54:39.327527 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:54:39.327352 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-swfcr\"" Nov 03 01:54:39.330381 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:54:39.330358 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8wz5k"] Nov 03 01:54:39.410504 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:54:39.410471 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c00a507-1d4e-4a5e-b87b-0e0de7419052-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-8wz5k\" (UID: \"7c00a507-1d4e-4a5e-b87b-0e0de7419052\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8wz5k" Nov 03 01:54:39.410706 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:54:39.410532 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c00a507-1d4e-4a5e-b87b-0e0de7419052-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-8wz5k\" (UID: \"7c00a507-1d4e-4a5e-b87b-0e0de7419052\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8wz5k" Nov 03 01:54:39.410706 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:54:39.410644 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlbz8\" (UniqueName: \"kubernetes.io/projected/7c00a507-1d4e-4a5e-b87b-0e0de7419052-kube-api-access-wlbz8\") pod \"maas-api-revoke-sub-d3fba056-8wz5k\" (UID: \"7c00a507-1d4e-4a5e-b87b-0e0de7419052\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8wz5k" Nov 03 01:54:39.511759 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:54:39.511723 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c00a507-1d4e-4a5e-b87b-0e0de7419052-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-8wz5k\" (UID: \"7c00a507-1d4e-4a5e-b87b-0e0de7419052\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8wz5k" Nov 03 01:54:39.511949 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:54:39.511774 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlbz8\" (UniqueName: \"kubernetes.io/projected/7c00a507-1d4e-4a5e-b87b-0e0de7419052-kube-api-access-wlbz8\") pod \"maas-api-revoke-sub-d3fba056-8wz5k\" (UID: \"7c00a507-1d4e-4a5e-b87b-0e0de7419052\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8wz5k" Nov 03 01:54:39.511949 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:54:39.511849 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c00a507-1d4e-4a5e-b87b-0e0de7419052-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-8wz5k\" (UID: \"7c00a507-1d4e-4a5e-b87b-0e0de7419052\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8wz5k" Nov 03 01:54:39.512488 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:54:39.512465 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c00a507-1d4e-4a5e-b87b-0e0de7419052-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-8wz5k\" (UID: \"7c00a507-1d4e-4a5e-b87b-0e0de7419052\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8wz5k" Nov 03 01:54:39.514247 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:54:39.514226 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c00a507-1d4e-4a5e-b87b-0e0de7419052-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-8wz5k\" (UID: \"7c00a507-1d4e-4a5e-b87b-0e0de7419052\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8wz5k" Nov 03 01:54:39.520128 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:54:39.520101 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlbz8\" (UniqueName: \"kubernetes.io/projected/7c00a507-1d4e-4a5e-b87b-0e0de7419052-kube-api-access-wlbz8\") pod \"maas-api-revoke-sub-d3fba056-8wz5k\" (UID: \"7c00a507-1d4e-4a5e-b87b-0e0de7419052\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8wz5k" Nov 03 01:54:39.635894 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:54:39.635793 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8wz5k" Nov 03 01:54:39.913226 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:54:39.913081 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8wz5k"] Nov 03 01:54:39.917610 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:54:39.917580 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c00a507_1d4e_4a5e_b87b_0e0de7419052.slice/crio-c33ab884a91be0a62c50a1cac407a776c314fec59af42a4980af611a995e6932 WatchSource:0}: Error finding container c33ab884a91be0a62c50a1cac407a776c314fec59af42a4980af611a995e6932: Status 404 returned error can't find the container with id c33ab884a91be0a62c50a1cac407a776c314fec59af42a4980af611a995e6932 Nov 03 01:54:40.669629 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:54:40.669577 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8wz5k" event={"ID":"7c00a507-1d4e-4a5e-b87b-0e0de7419052","Type":"ContainerStarted","Data":"773851ea7f158baecddec6a07fd6bfe9c93d46a62b7407a75c9204a40c79563c"} Nov 03 01:54:40.670099 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:54:40.669665 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8wz5k" event={"ID":"7c00a507-1d4e-4a5e-b87b-0e0de7419052","Type":"ContainerStarted","Data":"c33ab884a91be0a62c50a1cac407a776c314fec59af42a4980af611a995e6932"} Nov 03 01:54:40.686665 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:54:40.686597 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8wz5k" podStartSLOduration=1.686574735 podStartE2EDuration="1.686574735s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:54:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:54:40.685182895 +0000 UTC m=+4422.856185994" watchObservedRunningTime="2026-11-03 01:54:40.686574735 +0000 UTC m=+4422.857577833" Nov 03 01:54:41.681142 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:54:41.681094 4421 generic.go:356] "Generic (PLEG): container finished" podID="7c00a507-1d4e-4a5e-b87b-0e0de7419052" containerID="773851ea7f158baecddec6a07fd6bfe9c93d46a62b7407a75c9204a40c79563c" exitCode=0 Nov 03 01:54:41.681764 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:54:41.681442 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8wz5k" event={"ID":"7c00a507-1d4e-4a5e-b87b-0e0de7419052","Type":"ContainerDied","Data":"773851ea7f158baecddec6a07fd6bfe9c93d46a62b7407a75c9204a40c79563c"} Nov 03 01:54:42.325298 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:54:42.321401 4421 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-q5n9x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 01:54:42.325298 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:54:42.321473 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-q5n9x" podUID="e11a97cb-0980-45d1-842f-ded2fb31b03f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 01:54:42.880124 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:54:42.880089 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8wz5k" Nov 03 01:54:42.984231 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:54:42.984098 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c00a507-1d4e-4a5e-b87b-0e0de7419052-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c00a507-1d4e-4a5e-b87b-0e0de7419052-maas-api-openshift-ca\") pod \"7c00a507-1d4e-4a5e-b87b-0e0de7419052\" (UID: \"7c00a507-1d4e-4a5e-b87b-0e0de7419052\") " Nov 03 01:54:42.984455 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:54:42.984247 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c00a507-1d4e-4a5e-b87b-0e0de7419052-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c00a507-1d4e-4a5e-b87b-0e0de7419052-maas-api-xks-ca\") pod \"7c00a507-1d4e-4a5e-b87b-0e0de7419052\" (UID: \"7c00a507-1d4e-4a5e-b87b-0e0de7419052\") " Nov 03 01:54:42.984455 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:54:42.984309 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c00a507-1d4e-4a5e-b87b-0e0de7419052-kube-api-access-wlbz8\" (UniqueName: \"kubernetes.io/projected/7c00a507-1d4e-4a5e-b87b-0e0de7419052-kube-api-access-wlbz8\") pod \"7c00a507-1d4e-4a5e-b87b-0e0de7419052\" (UID: \"7c00a507-1d4e-4a5e-b87b-0e0de7419052\") " Nov 03 01:54:42.984455 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:54:42.984428 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c00a507-1d4e-4a5e-b87b-0e0de7419052-maas-api-openshift-ca" pod "7c00a507-1d4e-4a5e-b87b-0e0de7419052" (UID: "7c00a507-1d4e-4a5e-b87b-0e0de7419052"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:54:42.984654 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:54:42.984582 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c00a507-1d4e-4a5e-b87b-0e0de7419052-maas-api-openshift-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:54:42.986476 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:54:42.986445 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c00a507-1d4e-4a5e-b87b-0e0de7419052-maas-api-xks-ca" pod "7c00a507-1d4e-4a5e-b87b-0e0de7419052" (UID: "7c00a507-1d4e-4a5e-b87b-0e0de7419052"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:54:42.986476 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:54:42.986467 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c00a507-1d4e-4a5e-b87b-0e0de7419052-kube-api-access-wlbz8" pod "7c00a507-1d4e-4a5e-b87b-0e0de7419052" (UID: "7c00a507-1d4e-4a5e-b87b-0e0de7419052"). InnerVolumeSpecName "kube-api-access-wlbz8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:54:43.086097 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:54:43.086055 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c00a507-1d4e-4a5e-b87b-0e0de7419052-maas-api-xks-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:54:43.086097 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:54:43.086089 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wlbz8\" (UniqueName: \"kubernetes.io/projected/7c00a507-1d4e-4a5e-b87b-0e0de7419052-kube-api-access-wlbz8\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:54:43.694545 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:54:43.694513 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8wz5k" Nov 03 01:54:43.694545 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:54:43.694531 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8wz5k" event={"ID":"7c00a507-1d4e-4a5e-b87b-0e0de7419052","Type":"ContainerDied","Data":"c33ab884a91be0a62c50a1cac407a776c314fec59af42a4980af611a995e6932"} Nov 03 01:54:43.694811 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:54:43.694624 4421 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c33ab884a91be0a62c50a1cac407a776c314fec59af42a4980af611a995e6932" Nov 03 01:54:44.668790 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:54:44.668749 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 01:55:00.008170 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:00.008136 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-jbdmp"] Nov 03 01:55:00.008736 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:00.008540 4421 cpu_manager.go:414] "containerMap: removing container" podUID="7c00a507-1d4e-4a5e-b87b-0e0de7419052" containerName="revoke-keys" Nov 03 01:55:00.008736 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:00.008554 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c00a507-1d4e-4a5e-b87b-0e0de7419052" containerName="revoke-keys" Nov 03 01:55:00.008736 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:00.008644 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c00a507-1d4e-4a5e-b87b-0e0de7419052" containerName="revoke-keys" Nov 03 01:55:00.011946 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:00.011926 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-jbdmp" Nov 03 01:55:00.016965 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:00.016873 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:55:00.021632 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:00.018029 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 01:55:00.021632 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:00.018165 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-swfcr\"" Nov 03 01:55:00.021632 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:00.018166 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 01:55:00.021632 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:00.021564 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-jbdmp"] Nov 03 01:55:00.057718 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:00.057677 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfhm4\" (UniqueName: \"kubernetes.io/projected/5051a056-68ce-45b5-85e1-cfd8485097ea-kube-api-access-lfhm4\") pod \"maas-api-revoke-sub-ae594979-jbdmp\" (UID: \"5051a056-68ce-45b5-85e1-cfd8485097ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-jbdmp" Nov 03 01:55:00.058093 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:00.058061 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5051a056-68ce-45b5-85e1-cfd8485097ea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-jbdmp\" (UID: \"5051a056-68ce-45b5-85e1-cfd8485097ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-jbdmp" Nov 03 01:55:00.058376 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:00.058349 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5051a056-68ce-45b5-85e1-cfd8485097ea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-jbdmp\" (UID: \"5051a056-68ce-45b5-85e1-cfd8485097ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-jbdmp" Nov 03 01:55:00.159213 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:00.159165 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5051a056-68ce-45b5-85e1-cfd8485097ea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-jbdmp\" (UID: \"5051a056-68ce-45b5-85e1-cfd8485097ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-jbdmp" Nov 03 01:55:00.159468 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:00.159311 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfhm4\" (UniqueName: \"kubernetes.io/projected/5051a056-68ce-45b5-85e1-cfd8485097ea-kube-api-access-lfhm4\") pod \"maas-api-revoke-sub-ae594979-jbdmp\" (UID: \"5051a056-68ce-45b5-85e1-cfd8485097ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-jbdmp" Nov 03 01:55:00.159468 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:00.159392 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5051a056-68ce-45b5-85e1-cfd8485097ea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-jbdmp\" (UID: \"5051a056-68ce-45b5-85e1-cfd8485097ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-jbdmp" Nov 03 01:55:00.160013 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:00.159983 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5051a056-68ce-45b5-85e1-cfd8485097ea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-jbdmp\" (UID: \"5051a056-68ce-45b5-85e1-cfd8485097ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-jbdmp" Nov 03 01:55:00.161676 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:00.161648 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5051a056-68ce-45b5-85e1-cfd8485097ea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-jbdmp\" (UID: \"5051a056-68ce-45b5-85e1-cfd8485097ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-jbdmp" Nov 03 01:55:00.168828 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:00.168806 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfhm4\" (UniqueName: \"kubernetes.io/projected/5051a056-68ce-45b5-85e1-cfd8485097ea-kube-api-access-lfhm4\") pod \"maas-api-revoke-sub-ae594979-jbdmp\" (UID: \"5051a056-68ce-45b5-85e1-cfd8485097ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-jbdmp" Nov 03 01:55:00.328100 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:00.328011 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-jbdmp" Nov 03 01:55:00.616905 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:00.616878 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-jbdmp"] Nov 03 01:55:00.618726 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:55:00.618695 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5051a056_68ce_45b5_85e1_cfd8485097ea.slice/crio-448ab021896cac0b23e3ce4c6632789a688b321cbebfee4a70cba9ec66995c6a WatchSource:0}: Error finding container 448ab021896cac0b23e3ce4c6632789a688b321cbebfee4a70cba9ec66995c6a: Status 404 returned error can't find the container with id 448ab021896cac0b23e3ce4c6632789a688b321cbebfee4a70cba9ec66995c6a Nov 03 01:55:00.794253 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:00.794219 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-jbdmp" event={"ID":"5051a056-68ce-45b5-85e1-cfd8485097ea","Type":"ContainerStarted","Data":"448ab021896cac0b23e3ce4c6632789a688b321cbebfee4a70cba9ec66995c6a"} Nov 03 01:55:01.804294 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:01.804242 4421 generic.go:356] "Generic (PLEG): container finished" podID="5051a056-68ce-45b5-85e1-cfd8485097ea" containerID="3b2769defbdc86b810e913735765fc6308047c0753a260e106d109e24f5b9dc8" exitCode=0 Nov 03 01:55:01.804898 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:01.804321 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-jbdmp" event={"ID":"5051a056-68ce-45b5-85e1-cfd8485097ea","Type":"ContainerDied","Data":"3b2769defbdc86b810e913735765fc6308047c0753a260e106d109e24f5b9dc8"} Nov 03 01:55:02.962223 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:02.962201 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-jbdmp" Nov 03 01:55:03.086707 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:03.086612 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5051a056-68ce-45b5-85e1-cfd8485097ea-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5051a056-68ce-45b5-85e1-cfd8485097ea-maas-api-openshift-ca\") pod \"5051a056-68ce-45b5-85e1-cfd8485097ea\" (UID: \"5051a056-68ce-45b5-85e1-cfd8485097ea\") " Nov 03 01:55:03.086707 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:03.086698 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5051a056-68ce-45b5-85e1-cfd8485097ea-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5051a056-68ce-45b5-85e1-cfd8485097ea-maas-api-xks-ca\") pod \"5051a056-68ce-45b5-85e1-cfd8485097ea\" (UID: \"5051a056-68ce-45b5-85e1-cfd8485097ea\") " Nov 03 01:55:03.086914 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:03.086741 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5051a056-68ce-45b5-85e1-cfd8485097ea-kube-api-access-lfhm4\" (UniqueName: \"kubernetes.io/projected/5051a056-68ce-45b5-85e1-cfd8485097ea-kube-api-access-lfhm4\") pod \"5051a056-68ce-45b5-85e1-cfd8485097ea\" (UID: \"5051a056-68ce-45b5-85e1-cfd8485097ea\") " Nov 03 01:55:03.087012 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:03.086991 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5051a056-68ce-45b5-85e1-cfd8485097ea-maas-api-openshift-ca" pod "5051a056-68ce-45b5-85e1-cfd8485097ea" (UID: "5051a056-68ce-45b5-85e1-cfd8485097ea"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:55:03.088831 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:03.088801 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5051a056-68ce-45b5-85e1-cfd8485097ea-maas-api-xks-ca" pod "5051a056-68ce-45b5-85e1-cfd8485097ea" (UID: "5051a056-68ce-45b5-85e1-cfd8485097ea"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:55:03.088964 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:03.088950 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5051a056-68ce-45b5-85e1-cfd8485097ea-kube-api-access-lfhm4" pod "5051a056-68ce-45b5-85e1-cfd8485097ea" (UID: "5051a056-68ce-45b5-85e1-cfd8485097ea"). InnerVolumeSpecName "kube-api-access-lfhm4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:55:03.187969 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:03.187932 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5051a056-68ce-45b5-85e1-cfd8485097ea-maas-api-xks-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:55:03.187969 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:03.187967 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lfhm4\" (UniqueName: \"kubernetes.io/projected/5051a056-68ce-45b5-85e1-cfd8485097ea-kube-api-access-lfhm4\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:55:03.187969 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:03.187978 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5051a056-68ce-45b5-85e1-cfd8485097ea-maas-api-openshift-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:55:03.817501 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:03.817470 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-jbdmp" Nov 03 01:55:03.817671 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:03.817496 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-jbdmp" event={"ID":"5051a056-68ce-45b5-85e1-cfd8485097ea","Type":"ContainerDied","Data":"448ab021896cac0b23e3ce4c6632789a688b321cbebfee4a70cba9ec66995c6a"} Nov 03 01:55:03.817671 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:03.817553 4421 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="448ab021896cac0b23e3ce4c6632789a688b321cbebfee4a70cba9ec66995c6a" Nov 03 01:55:14.703284 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:55:14.703194 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 01:55:16.218025 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:16.217990 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-kqt48"] Nov 03 01:55:16.218445 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:16.218437 4421 cpu_manager.go:414] "containerMap: removing container" podUID="5051a056-68ce-45b5-85e1-cfd8485097ea" containerName="revoke-keys" Nov 03 01:55:16.218490 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:16.218452 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5051a056-68ce-45b5-85e1-cfd8485097ea" containerName="revoke-keys" Nov 03 01:55:16.218573 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:16.218558 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="5051a056-68ce-45b5-85e1-cfd8485097ea" containerName="revoke-keys" Nov 03 01:55:16.221839 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:16.221817 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-kqt48" Nov 03 01:55:16.229407 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:16.229370 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-kqt48"] Nov 03 01:55:16.230218 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:16.230194 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 01:55:16.230402 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:16.230357 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-swfcr\"" Nov 03 01:55:16.230521 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:16.230412 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:55:16.230521 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:16.230481 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 01:55:16.383585 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:16.383546 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t94gt\" (UniqueName: \"kubernetes.io/projected/fcad2130-1e0c-47dd-8368-50effdd4d69d-kube-api-access-t94gt\") pod \"maas-api-revoke-sub-a4aef7d7-kqt48\" (UID: \"fcad2130-1e0c-47dd-8368-50effdd4d69d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-kqt48" Nov 03 01:55:16.383765 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:16.383620 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcad2130-1e0c-47dd-8368-50effdd4d69d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-kqt48\" (UID: \"fcad2130-1e0c-47dd-8368-50effdd4d69d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-kqt48" Nov 03 01:55:16.383765 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:16.383712 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcad2130-1e0c-47dd-8368-50effdd4d69d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-kqt48\" (UID: \"fcad2130-1e0c-47dd-8368-50effdd4d69d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-kqt48" Nov 03 01:55:16.485245 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:16.485151 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t94gt\" (UniqueName: \"kubernetes.io/projected/fcad2130-1e0c-47dd-8368-50effdd4d69d-kube-api-access-t94gt\") pod \"maas-api-revoke-sub-a4aef7d7-kqt48\" (UID: \"fcad2130-1e0c-47dd-8368-50effdd4d69d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-kqt48" Nov 03 01:55:16.485245 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:16.485235 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcad2130-1e0c-47dd-8368-50effdd4d69d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-kqt48\" (UID: \"fcad2130-1e0c-47dd-8368-50effdd4d69d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-kqt48" Nov 03 01:55:16.485479 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:16.485292 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcad2130-1e0c-47dd-8368-50effdd4d69d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-kqt48\" (UID: \"fcad2130-1e0c-47dd-8368-50effdd4d69d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-kqt48" Nov 03 01:55:16.485958 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:16.485932 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcad2130-1e0c-47dd-8368-50effdd4d69d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-kqt48\" (UID: \"fcad2130-1e0c-47dd-8368-50effdd4d69d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-kqt48" Nov 03 01:55:16.487700 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:16.487680 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcad2130-1e0c-47dd-8368-50effdd4d69d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-kqt48\" (UID: \"fcad2130-1e0c-47dd-8368-50effdd4d69d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-kqt48" Nov 03 01:55:16.495560 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:16.495535 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t94gt\" (UniqueName: \"kubernetes.io/projected/fcad2130-1e0c-47dd-8368-50effdd4d69d-kube-api-access-t94gt\") pod \"maas-api-revoke-sub-a4aef7d7-kqt48\" (UID: \"fcad2130-1e0c-47dd-8368-50effdd4d69d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-kqt48" Nov 03 01:55:16.541776 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:16.541740 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-kqt48" Nov 03 01:55:16.818249 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:16.818214 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-kqt48"] Nov 03 01:55:16.820624 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:55:16.820577 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcad2130_1e0c_47dd_8368_50effdd4d69d.slice/crio-ab9bea9a57a9a7049efdf13a3cd36737c90f4fd910cc75baded0dfd4333846e3 WatchSource:0}: Error finding container ab9bea9a57a9a7049efdf13a3cd36737c90f4fd910cc75baded0dfd4333846e3: Status 404 returned error can't find the container with id ab9bea9a57a9a7049efdf13a3cd36737c90f4fd910cc75baded0dfd4333846e3 Nov 03 01:55:16.907529 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:16.907491 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-kqt48" event={"ID":"fcad2130-1e0c-47dd-8368-50effdd4d69d","Type":"ContainerStarted","Data":"ab9bea9a57a9a7049efdf13a3cd36737c90f4fd910cc75baded0dfd4333846e3"} Nov 03 01:55:17.914435 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:17.914395 4421 generic.go:356] "Generic (PLEG): container finished" podID="fcad2130-1e0c-47dd-8368-50effdd4d69d" containerID="4325af5af7db8b4cd10682157d0401f9e94c413a68bba1d4e2873bff48bc8c54" exitCode=0 Nov 03 01:55:17.914943 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:17.914473 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-kqt48" event={"ID":"fcad2130-1e0c-47dd-8368-50effdd4d69d","Type":"ContainerDied","Data":"4325af5af7db8b4cd10682157d0401f9e94c413a68bba1d4e2873bff48bc8c54"} Nov 03 01:55:19.055800 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:19.055771 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-kqt48" Nov 03 01:55:19.111388 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:19.111359 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fcad2130-1e0c-47dd-8368-50effdd4d69d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcad2130-1e0c-47dd-8368-50effdd4d69d-maas-api-xks-ca\") pod \"fcad2130-1e0c-47dd-8368-50effdd4d69d\" (UID: \"fcad2130-1e0c-47dd-8368-50effdd4d69d\") " Nov 03 01:55:19.111589 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:19.111411 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fcad2130-1e0c-47dd-8368-50effdd4d69d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcad2130-1e0c-47dd-8368-50effdd4d69d-maas-api-openshift-ca\") pod \"fcad2130-1e0c-47dd-8368-50effdd4d69d\" (UID: \"fcad2130-1e0c-47dd-8368-50effdd4d69d\") " Nov 03 01:55:19.111589 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:19.111518 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fcad2130-1e0c-47dd-8368-50effdd4d69d-kube-api-access-t94gt\" (UniqueName: \"kubernetes.io/projected/fcad2130-1e0c-47dd-8368-50effdd4d69d-kube-api-access-t94gt\") pod \"fcad2130-1e0c-47dd-8368-50effdd4d69d\" (UID: \"fcad2130-1e0c-47dd-8368-50effdd4d69d\") " Nov 03 01:55:19.111885 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:19.111849 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcad2130-1e0c-47dd-8368-50effdd4d69d-maas-api-openshift-ca" pod "fcad2130-1e0c-47dd-8368-50effdd4d69d" (UID: "fcad2130-1e0c-47dd-8368-50effdd4d69d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:55:19.113687 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:19.113659 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fcad2130-1e0c-47dd-8368-50effdd4d69d-maas-api-xks-ca" pod "fcad2130-1e0c-47dd-8368-50effdd4d69d" (UID: "fcad2130-1e0c-47dd-8368-50effdd4d69d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:55:19.113786 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:19.113728 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcad2130-1e0c-47dd-8368-50effdd4d69d-kube-api-access-t94gt" pod "fcad2130-1e0c-47dd-8368-50effdd4d69d" (UID: "fcad2130-1e0c-47dd-8368-50effdd4d69d"). InnerVolumeSpecName "kube-api-access-t94gt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:55:19.213147 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:19.213054 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t94gt\" (UniqueName: \"kubernetes.io/projected/fcad2130-1e0c-47dd-8368-50effdd4d69d-kube-api-access-t94gt\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:55:19.213147 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:19.213090 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcad2130-1e0c-47dd-8368-50effdd4d69d-maas-api-xks-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:55:19.213147 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:19.213101 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcad2130-1e0c-47dd-8368-50effdd4d69d-maas-api-openshift-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:55:19.926057 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:19.926028 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-kqt48" Nov 03 01:55:19.926057 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:19.926047 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-kqt48" event={"ID":"fcad2130-1e0c-47dd-8368-50effdd4d69d","Type":"ContainerDied","Data":"ab9bea9a57a9a7049efdf13a3cd36737c90f4fd910cc75baded0dfd4333846e3"} Nov 03 01:55:19.926322 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:19.926097 4421 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ab9bea9a57a9a7049efdf13a3cd36737c90f4fd910cc75baded0dfd4333846e3" Nov 03 01:55:23.023213 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:23.023175 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-wcr2t"] Nov 03 01:55:23.025626 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:23.025592 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-wcr2t"] Nov 03 01:55:23.976564 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:23.976524 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a6caccc-771d-442b-b010-6fe164cc4c71" path="/var/lib/kubelet/pods/9a6caccc-771d-442b-b010-6fe164cc4c71/volumes" Nov 03 01:55:31.675541 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:31.675501 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-dmff7"] Nov 03 01:55:31.676962 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:31.676931 4421 cpu_manager.go:414] "containerMap: removing container" podUID="fcad2130-1e0c-47dd-8368-50effdd4d69d" containerName="revoke-keys" Nov 03 01:55:31.677172 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:31.677154 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fcad2130-1e0c-47dd-8368-50effdd4d69d" containerName="revoke-keys" Nov 03 01:55:31.689137 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:31.687047 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="fcad2130-1e0c-47dd-8368-50effdd4d69d" containerName="revoke-keys" Nov 03 01:55:31.691846 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:31.691822 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-dmff7"] Nov 03 01:55:31.691982 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:31.691941 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-dmff7" Nov 03 01:55:31.695577 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:31.695548 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 01:55:31.695713 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:31.695664 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 01:55:31.695854 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:31.695837 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-swfcr\"" Nov 03 01:55:31.695929 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:31.695889 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:55:31.790172 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:31.790126 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76b9d8cd-40e1-40da-9bf2-014af7b1a15d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-dmff7\" (UID: \"76b9d8cd-40e1-40da-9bf2-014af7b1a15d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-dmff7" Nov 03 01:55:31.790381 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:31.790246 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrds9\" (UniqueName: \"kubernetes.io/projected/76b9d8cd-40e1-40da-9bf2-014af7b1a15d-kube-api-access-mrds9\") pod \"maas-api-revoke-sub-79805ca9-dmff7\" (UID: \"76b9d8cd-40e1-40da-9bf2-014af7b1a15d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-dmff7" Nov 03 01:55:31.790381 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:31.790297 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76b9d8cd-40e1-40da-9bf2-014af7b1a15d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-dmff7\" (UID: \"76b9d8cd-40e1-40da-9bf2-014af7b1a15d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-dmff7" Nov 03 01:55:31.891797 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:31.891756 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76b9d8cd-40e1-40da-9bf2-014af7b1a15d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-dmff7\" (UID: \"76b9d8cd-40e1-40da-9bf2-014af7b1a15d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-dmff7" Nov 03 01:55:31.892010 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:31.891828 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrds9\" (UniqueName: \"kubernetes.io/projected/76b9d8cd-40e1-40da-9bf2-014af7b1a15d-kube-api-access-mrds9\") pod \"maas-api-revoke-sub-79805ca9-dmff7\" (UID: \"76b9d8cd-40e1-40da-9bf2-014af7b1a15d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-dmff7" Nov 03 01:55:31.892010 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:31.891849 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76b9d8cd-40e1-40da-9bf2-014af7b1a15d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-dmff7\" (UID: \"76b9d8cd-40e1-40da-9bf2-014af7b1a15d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-dmff7" Nov 03 01:55:31.892593 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:31.892571 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76b9d8cd-40e1-40da-9bf2-014af7b1a15d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-dmff7\" (UID: \"76b9d8cd-40e1-40da-9bf2-014af7b1a15d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-dmff7" Nov 03 01:55:31.894389 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:31.894366 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76b9d8cd-40e1-40da-9bf2-014af7b1a15d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-dmff7\" (UID: \"76b9d8cd-40e1-40da-9bf2-014af7b1a15d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-dmff7" Nov 03 01:55:31.899647 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:31.899624 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrds9\" (UniqueName: \"kubernetes.io/projected/76b9d8cd-40e1-40da-9bf2-014af7b1a15d-kube-api-access-mrds9\") pod \"maas-api-revoke-sub-79805ca9-dmff7\" (UID: \"76b9d8cd-40e1-40da-9bf2-014af7b1a15d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-dmff7" Nov 03 01:55:32.027318 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:32.025470 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-dmff7" Nov 03 01:55:32.358733 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:32.358705 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-dmff7"] Nov 03 01:55:32.361127 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:55:32.361088 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76b9d8cd_40e1_40da_9bf2_014af7b1a15d.slice/crio-9731946a0584c32599ba0b029beba48c2dbaa6e509e7d463042fac748ce75038 WatchSource:0}: Error finding container 9731946a0584c32599ba0b029beba48c2dbaa6e509e7d463042fac748ce75038: Status 404 returned error can't find the container with id 9731946a0584c32599ba0b029beba48c2dbaa6e509e7d463042fac748ce75038 Nov 03 01:55:32.998036 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:32.998001 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-dmff7" event={"ID":"76b9d8cd-40e1-40da-9bf2-014af7b1a15d","Type":"ContainerStarted","Data":"19794d0c4e31326c3a656ddd139a264f16f30be8aa9c8ab60b871a57bea8e9fe"} Nov 03 01:55:32.998536 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:32.998057 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-dmff7" event={"ID":"76b9d8cd-40e1-40da-9bf2-014af7b1a15d","Type":"ContainerStarted","Data":"9731946a0584c32599ba0b029beba48c2dbaa6e509e7d463042fac748ce75038"} Nov 03 01:55:33.018138 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:33.018070 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-dmff7" podStartSLOduration=2.018050805 podStartE2EDuration="2.018050805s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:55:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:55:33.014708513 +0000 UTC m=+4475.185711610" watchObservedRunningTime="2026-11-03 01:55:33.018050805 +0000 UTC m=+4475.189053903" Nov 03 01:55:34.004526 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:34.004487 4421 generic.go:356] "Generic (PLEG): container finished" podID="76b9d8cd-40e1-40da-9bf2-014af7b1a15d" containerID="19794d0c4e31326c3a656ddd139a264f16f30be8aa9c8ab60b871a57bea8e9fe" exitCode=0 Nov 03 01:55:34.004987 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:34.004578 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-dmff7" event={"ID":"76b9d8cd-40e1-40da-9bf2-014af7b1a15d","Type":"ContainerDied","Data":"19794d0c4e31326c3a656ddd139a264f16f30be8aa9c8ab60b871a57bea8e9fe"} Nov 03 01:55:35.162483 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:35.162455 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-dmff7" Nov 03 01:55:35.228853 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:35.228820 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76b9d8cd-40e1-40da-9bf2-014af7b1a15d-kube-api-access-mrds9\" (UniqueName: \"kubernetes.io/projected/76b9d8cd-40e1-40da-9bf2-014af7b1a15d-kube-api-access-mrds9\") pod \"76b9d8cd-40e1-40da-9bf2-014af7b1a15d\" (UID: \"76b9d8cd-40e1-40da-9bf2-014af7b1a15d\") " Nov 03 01:55:35.230937 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:35.230909 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76b9d8cd-40e1-40da-9bf2-014af7b1a15d-kube-api-access-mrds9" pod "76b9d8cd-40e1-40da-9bf2-014af7b1a15d" (UID: "76b9d8cd-40e1-40da-9bf2-014af7b1a15d"). InnerVolumeSpecName "kube-api-access-mrds9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:55:35.330199 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:35.330112 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76b9d8cd-40e1-40da-9bf2-014af7b1a15d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76b9d8cd-40e1-40da-9bf2-014af7b1a15d-maas-api-openshift-ca\") pod \"76b9d8cd-40e1-40da-9bf2-014af7b1a15d\" (UID: \"76b9d8cd-40e1-40da-9bf2-014af7b1a15d\") " Nov 03 01:55:35.330199 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:35.330163 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/76b9d8cd-40e1-40da-9bf2-014af7b1a15d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76b9d8cd-40e1-40da-9bf2-014af7b1a15d-maas-api-xks-ca\") pod \"76b9d8cd-40e1-40da-9bf2-014af7b1a15d\" (UID: \"76b9d8cd-40e1-40da-9bf2-014af7b1a15d\") " Nov 03 01:55:35.330558 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:35.330531 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76b9d8cd-40e1-40da-9bf2-014af7b1a15d-maas-api-openshift-ca" pod "76b9d8cd-40e1-40da-9bf2-014af7b1a15d" (UID: "76b9d8cd-40e1-40da-9bf2-014af7b1a15d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:55:35.330667 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:35.330654 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76b9d8cd-40e1-40da-9bf2-014af7b1a15d-maas-api-openshift-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:55:35.330710 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:35.330670 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mrds9\" (UniqueName: \"kubernetes.io/projected/76b9d8cd-40e1-40da-9bf2-014af7b1a15d-kube-api-access-mrds9\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:55:35.332260 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:35.332235 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76b9d8cd-40e1-40da-9bf2-014af7b1a15d-maas-api-xks-ca" pod "76b9d8cd-40e1-40da-9bf2-014af7b1a15d" (UID: "76b9d8cd-40e1-40da-9bf2-014af7b1a15d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:55:35.431361 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:35.431320 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76b9d8cd-40e1-40da-9bf2-014af7b1a15d-maas-api-xks-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:55:36.015385 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:36.015350 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-dmff7" Nov 03 01:55:36.015541 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:36.015389 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-dmff7" event={"ID":"76b9d8cd-40e1-40da-9bf2-014af7b1a15d","Type":"ContainerDied","Data":"9731946a0584c32599ba0b029beba48c2dbaa6e509e7d463042fac748ce75038"} Nov 03 01:55:36.015541 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:36.015443 4421 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9731946a0584c32599ba0b029beba48c2dbaa6e509e7d463042fac748ce75038" Nov 03 01:55:39.021353 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:39.021314 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-9hv84"] Nov 03 01:55:39.023737 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:39.023711 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-9hv84"] Nov 03 01:55:39.977375 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:39.977337 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80170d86-c913-4510-a93a-fecd93986897" path="/var/lib/kubelet/pods/80170d86-c913-4510-a93a-fecd93986897/volumes" Nov 03 01:55:44.700250 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:55:44.700210 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 01:55:46.334953 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:46.334913 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-xv8hh"] Nov 03 01:55:46.335507 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:46.335325 4421 cpu_manager.go:414] "containerMap: removing container" podUID="76b9d8cd-40e1-40da-9bf2-014af7b1a15d" containerName="revoke-keys" Nov 03 01:55:46.335507 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:46.335338 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76b9d8cd-40e1-40da-9bf2-014af7b1a15d" containerName="revoke-keys" Nov 03 01:55:46.335507 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:46.335420 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="76b9d8cd-40e1-40da-9bf2-014af7b1a15d" containerName="revoke-keys" Nov 03 01:55:46.338741 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:46.338719 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-xv8hh" Nov 03 01:55:46.344439 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:46.344415 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:55:46.344686 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:46.344669 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 01:55:46.344902 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:46.344887 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-swfcr\"" Nov 03 01:55:46.345088 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:46.345076 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 01:55:46.349921 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:46.349897 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-xv8hh"] Nov 03 01:55:46.352381 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:46.350954 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwxvm\" (UniqueName: \"kubernetes.io/projected/f44d783a-0886-43e6-bace-2e3b4a166664-kube-api-access-dwxvm\") pod \"maas-api-revoke-sub-0b7e8edb-xv8hh\" (UID: \"f44d783a-0886-43e6-bace-2e3b4a166664\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-xv8hh" Nov 03 01:55:46.352566 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:46.352547 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f44d783a-0886-43e6-bace-2e3b4a166664-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-xv8hh\" (UID: \"f44d783a-0886-43e6-bace-2e3b4a166664\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-xv8hh" Nov 03 01:55:46.352701 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:46.352681 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f44d783a-0886-43e6-bace-2e3b4a166664-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-xv8hh\" (UID: \"f44d783a-0886-43e6-bace-2e3b4a166664\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-xv8hh" Nov 03 01:55:46.453756 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:46.453713 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwxvm\" (UniqueName: \"kubernetes.io/projected/f44d783a-0886-43e6-bace-2e3b4a166664-kube-api-access-dwxvm\") pod \"maas-api-revoke-sub-0b7e8edb-xv8hh\" (UID: \"f44d783a-0886-43e6-bace-2e3b4a166664\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-xv8hh" Nov 03 01:55:46.453965 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:46.453786 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f44d783a-0886-43e6-bace-2e3b4a166664-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-xv8hh\" (UID: \"f44d783a-0886-43e6-bace-2e3b4a166664\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-xv8hh" Nov 03 01:55:46.453965 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:46.453846 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f44d783a-0886-43e6-bace-2e3b4a166664-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-xv8hh\" (UID: \"f44d783a-0886-43e6-bace-2e3b4a166664\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-xv8hh" Nov 03 01:55:46.454563 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:46.454532 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f44d783a-0886-43e6-bace-2e3b4a166664-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-xv8hh\" (UID: \"f44d783a-0886-43e6-bace-2e3b4a166664\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-xv8hh" Nov 03 01:55:46.456341 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:46.456313 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f44d783a-0886-43e6-bace-2e3b4a166664-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-xv8hh\" (UID: \"f44d783a-0886-43e6-bace-2e3b4a166664\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-xv8hh" Nov 03 01:55:46.462551 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:46.462524 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwxvm\" (UniqueName: \"kubernetes.io/projected/f44d783a-0886-43e6-bace-2e3b4a166664-kube-api-access-dwxvm\") pod \"maas-api-revoke-sub-0b7e8edb-xv8hh\" (UID: \"f44d783a-0886-43e6-bace-2e3b4a166664\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-xv8hh" Nov 03 01:55:46.664494 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:46.664399 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-xv8hh" Nov 03 01:55:46.938937 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:46.938906 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-xv8hh"] Nov 03 01:55:46.940941 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:55:46.940912 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf44d783a_0886_43e6_bace_2e3b4a166664.slice/crio-15f8b4230f5801a40e1920ea93f51a1f849a63d835fec5334922cf3c49b4c329 WatchSource:0}: Error finding container 15f8b4230f5801a40e1920ea93f51a1f849a63d835fec5334922cf3c49b4c329: Status 404 returned error can't find the container with id 15f8b4230f5801a40e1920ea93f51a1f849a63d835fec5334922cf3c49b4c329 Nov 03 01:55:47.092248 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:47.092202 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-xv8hh" event={"ID":"f44d783a-0886-43e6-bace-2e3b4a166664","Type":"ContainerStarted","Data":"15f8b4230f5801a40e1920ea93f51a1f849a63d835fec5334922cf3c49b4c329"} Nov 03 01:55:48.098954 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:48.098921 4421 generic.go:356] "Generic (PLEG): container finished" podID="f44d783a-0886-43e6-bace-2e3b4a166664" containerID="1b82dbacf13933f35bc07b9cf416eed024b15ccc1f13c88c4dce2a118b1d219e" exitCode=0 Nov 03 01:55:48.098954 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:48.098960 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-xv8hh" event={"ID":"f44d783a-0886-43e6-bace-2e3b4a166664","Type":"ContainerDied","Data":"1b82dbacf13933f35bc07b9cf416eed024b15ccc1f13c88c4dce2a118b1d219e"} Nov 03 01:55:49.237712 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:49.237689 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-xv8hh" Nov 03 01:55:49.280227 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:49.280187 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f44d783a-0886-43e6-bace-2e3b4a166664-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f44d783a-0886-43e6-bace-2e3b4a166664-maas-api-openshift-ca\") pod \"f44d783a-0886-43e6-bace-2e3b4a166664\" (UID: \"f44d783a-0886-43e6-bace-2e3b4a166664\") " Nov 03 01:55:49.280447 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:49.280266 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f44d783a-0886-43e6-bace-2e3b4a166664-kube-api-access-dwxvm\" (UniqueName: \"kubernetes.io/projected/f44d783a-0886-43e6-bace-2e3b4a166664-kube-api-access-dwxvm\") pod \"f44d783a-0886-43e6-bace-2e3b4a166664\" (UID: \"f44d783a-0886-43e6-bace-2e3b4a166664\") " Nov 03 01:55:49.280447 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:49.280395 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f44d783a-0886-43e6-bace-2e3b4a166664-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f44d783a-0886-43e6-bace-2e3b4a166664-maas-api-xks-ca\") pod \"f44d783a-0886-43e6-bace-2e3b4a166664\" (UID: \"f44d783a-0886-43e6-bace-2e3b4a166664\") " Nov 03 01:55:49.280645 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:49.280621 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f44d783a-0886-43e6-bace-2e3b4a166664-maas-api-openshift-ca" pod "f44d783a-0886-43e6-bace-2e3b4a166664" (UID: "f44d783a-0886-43e6-bace-2e3b4a166664"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:55:49.280781 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:49.280761 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f44d783a-0886-43e6-bace-2e3b4a166664-maas-api-openshift-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:55:49.282599 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:49.282559 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f44d783a-0886-43e6-bace-2e3b4a166664-maas-api-xks-ca" pod "f44d783a-0886-43e6-bace-2e3b4a166664" (UID: "f44d783a-0886-43e6-bace-2e3b4a166664"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:55:49.282744 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:49.282713 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f44d783a-0886-43e6-bace-2e3b4a166664-kube-api-access-dwxvm" pod "f44d783a-0886-43e6-bace-2e3b4a166664" (UID: "f44d783a-0886-43e6-bace-2e3b4a166664"). InnerVolumeSpecName "kube-api-access-dwxvm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:55:49.381345 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:49.381227 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwxvm\" (UniqueName: \"kubernetes.io/projected/f44d783a-0886-43e6-bace-2e3b4a166664-kube-api-access-dwxvm\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:55:49.381345 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:49.381261 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f44d783a-0886-43e6-bace-2e3b4a166664-maas-api-xks-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:55:50.111440 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:50.111408 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-xv8hh" Nov 03 01:55:50.111624 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:50.111411 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-xv8hh" event={"ID":"f44d783a-0886-43e6-bace-2e3b4a166664","Type":"ContainerDied","Data":"15f8b4230f5801a40e1920ea93f51a1f849a63d835fec5334922cf3c49b4c329"} Nov 03 01:55:50.111624 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:50.111540 4421 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="15f8b4230f5801a40e1920ea93f51a1f849a63d835fec5334922cf3c49b4c329" Nov 03 01:55:56.022821 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:56.022782 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4823bfaf-p7fp5"] Nov 03 01:55:56.025123 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:56.025093 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4823bfaf-p7fp5"] Nov 03 01:55:56.949784 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:56.949745 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kwzx5"] Nov 03 01:55:56.950153 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:56.950140 4421 cpu_manager.go:414] "containerMap: removing container" podUID="f44d783a-0886-43e6-bace-2e3b4a166664" containerName="revoke-keys" Nov 03 01:55:56.950203 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:56.950154 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f44d783a-0886-43e6-bace-2e3b4a166664" containerName="revoke-keys" Nov 03 01:55:56.950248 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:56.950238 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="f44d783a-0886-43e6-bace-2e3b4a166664" containerName="revoke-keys" Nov 03 01:55:56.953508 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:56.953491 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kwzx5" Nov 03 01:55:56.959450 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:56.959420 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:55:56.959633 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:56.959460 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 01:55:56.959633 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:56.959517 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 01:55:56.959633 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:56.959568 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-swfcr\"" Nov 03 01:55:56.962629 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:56.962606 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kwzx5"] Nov 03 01:55:57.057117 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:57.057084 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbae01c1-e316-4efd-8de1-43f66fe35e1a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-kwzx5\" (UID: \"cbae01c1-e316-4efd-8de1-43f66fe35e1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kwzx5" Nov 03 01:55:57.057555 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:57.057124 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldd42\" (UniqueName: \"kubernetes.io/projected/cbae01c1-e316-4efd-8de1-43f66fe35e1a-kube-api-access-ldd42\") pod \"maas-api-revoke-sub-3d4156f1-kwzx5\" (UID: \"cbae01c1-e316-4efd-8de1-43f66fe35e1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kwzx5" Nov 03 01:55:57.057555 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:57.057216 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbae01c1-e316-4efd-8de1-43f66fe35e1a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-kwzx5\" (UID: \"cbae01c1-e316-4efd-8de1-43f66fe35e1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kwzx5" Nov 03 01:55:57.158411 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:57.158378 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbae01c1-e316-4efd-8de1-43f66fe35e1a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-kwzx5\" (UID: \"cbae01c1-e316-4efd-8de1-43f66fe35e1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kwzx5" Nov 03 01:55:57.158604 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:57.158417 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldd42\" (UniqueName: \"kubernetes.io/projected/cbae01c1-e316-4efd-8de1-43f66fe35e1a-kube-api-access-ldd42\") pod \"maas-api-revoke-sub-3d4156f1-kwzx5\" (UID: \"cbae01c1-e316-4efd-8de1-43f66fe35e1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kwzx5" Nov 03 01:55:57.158604 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:57.158456 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbae01c1-e316-4efd-8de1-43f66fe35e1a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-kwzx5\" (UID: \"cbae01c1-e316-4efd-8de1-43f66fe35e1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kwzx5" Nov 03 01:55:57.159257 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:57.159229 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbae01c1-e316-4efd-8de1-43f66fe35e1a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-kwzx5\" (UID: \"cbae01c1-e316-4efd-8de1-43f66fe35e1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kwzx5" Nov 03 01:55:57.160848 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:57.160821 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbae01c1-e316-4efd-8de1-43f66fe35e1a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-kwzx5\" (UID: \"cbae01c1-e316-4efd-8de1-43f66fe35e1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kwzx5" Nov 03 01:55:57.166594 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:57.166567 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldd42\" (UniqueName: \"kubernetes.io/projected/cbae01c1-e316-4efd-8de1-43f66fe35e1a-kube-api-access-ldd42\") pod \"maas-api-revoke-sub-3d4156f1-kwzx5\" (UID: \"cbae01c1-e316-4efd-8de1-43f66fe35e1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kwzx5" Nov 03 01:55:57.269746 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:57.269714 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kwzx5" Nov 03 01:55:57.545875 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:57.545831 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kwzx5"] Nov 03 01:55:57.547649 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:55:57.547605 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbae01c1_e316_4efd_8de1_43f66fe35e1a.slice/crio-be546a9f2931e280a71cdb00bdf481bf7c473b2965b6792d78d3d23b7f9cc295 WatchSource:0}: Error finding container be546a9f2931e280a71cdb00bdf481bf7c473b2965b6792d78d3d23b7f9cc295: Status 404 returned error can't find the container with id be546a9f2931e280a71cdb00bdf481bf7c473b2965b6792d78d3d23b7f9cc295 Nov 03 01:55:57.977038 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:57.977002 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d649c869-44f4-42c3-b222-eec4e0375ba2" path="/var/lib/kubelet/pods/d649c869-44f4-42c3-b222-eec4e0375ba2/volumes" Nov 03 01:55:58.154767 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:58.154675 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kwzx5" event={"ID":"cbae01c1-e316-4efd-8de1-43f66fe35e1a","Type":"ContainerStarted","Data":"50c7fc4e06a7bbc896244f17fc315d834a8057ba73e3b201f35890a2a48acf24"} Nov 03 01:55:58.155141 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:58.154776 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kwzx5" event={"ID":"cbae01c1-e316-4efd-8de1-43f66fe35e1a","Type":"ContainerStarted","Data":"be546a9f2931e280a71cdb00bdf481bf7c473b2965b6792d78d3d23b7f9cc295"} Nov 03 01:55:58.173346 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:58.173262 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kwzx5" podStartSLOduration=2.173243703 podStartE2EDuration="2.173243703s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:55:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:55:58.173182242 +0000 UTC m=+4500.344185339" watchObservedRunningTime="2026-11-03 01:55:58.173243703 +0000 UTC m=+4500.344246800" Nov 03 01:55:59.161076 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:59.161040 4421 generic.go:356] "Generic (PLEG): container finished" podID="cbae01c1-e316-4efd-8de1-43f66fe35e1a" containerID="50c7fc4e06a7bbc896244f17fc315d834a8057ba73e3b201f35890a2a48acf24" exitCode=0 Nov 03 01:55:59.161531 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:55:59.161133 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kwzx5" event={"ID":"cbae01c1-e316-4efd-8de1-43f66fe35e1a","Type":"ContainerDied","Data":"50c7fc4e06a7bbc896244f17fc315d834a8057ba73e3b201f35890a2a48acf24"} Nov 03 01:56:00.312508 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:56:00.312479 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kwzx5" Nov 03 01:56:00.387121 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:56:00.387083 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbae01c1-e316-4efd-8de1-43f66fe35e1a-kube-api-access-ldd42\" (UniqueName: \"kubernetes.io/projected/cbae01c1-e316-4efd-8de1-43f66fe35e1a-kube-api-access-ldd42\") pod \"cbae01c1-e316-4efd-8de1-43f66fe35e1a\" (UID: \"cbae01c1-e316-4efd-8de1-43f66fe35e1a\") " Nov 03 01:56:00.387347 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:56:00.387156 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cbae01c1-e316-4efd-8de1-43f66fe35e1a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbae01c1-e316-4efd-8de1-43f66fe35e1a-maas-api-xks-ca\") pod \"cbae01c1-e316-4efd-8de1-43f66fe35e1a\" (UID: \"cbae01c1-e316-4efd-8de1-43f66fe35e1a\") " Nov 03 01:56:00.387347 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:56:00.387203 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cbae01c1-e316-4efd-8de1-43f66fe35e1a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbae01c1-e316-4efd-8de1-43f66fe35e1a-maas-api-openshift-ca\") pod \"cbae01c1-e316-4efd-8de1-43f66fe35e1a\" (UID: \"cbae01c1-e316-4efd-8de1-43f66fe35e1a\") " Nov 03 01:56:00.387657 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:56:00.387634 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbae01c1-e316-4efd-8de1-43f66fe35e1a-maas-api-openshift-ca" pod "cbae01c1-e316-4efd-8de1-43f66fe35e1a" (UID: "cbae01c1-e316-4efd-8de1-43f66fe35e1a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:56:00.389374 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:56:00.389348 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cbae01c1-e316-4efd-8de1-43f66fe35e1a-maas-api-xks-ca" pod "cbae01c1-e316-4efd-8de1-43f66fe35e1a" (UID: "cbae01c1-e316-4efd-8de1-43f66fe35e1a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:56:00.389511 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:56:00.389487 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbae01c1-e316-4efd-8de1-43f66fe35e1a-kube-api-access-ldd42" pod "cbae01c1-e316-4efd-8de1-43f66fe35e1a" (UID: "cbae01c1-e316-4efd-8de1-43f66fe35e1a"). InnerVolumeSpecName "kube-api-access-ldd42". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:56:00.488433 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:56:00.488346 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ldd42\" (UniqueName: \"kubernetes.io/projected/cbae01c1-e316-4efd-8de1-43f66fe35e1a-kube-api-access-ldd42\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:56:00.488433 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:56:00.488380 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbae01c1-e316-4efd-8de1-43f66fe35e1a-maas-api-xks-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:56:00.488433 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:56:00.488389 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbae01c1-e316-4efd-8de1-43f66fe35e1a-maas-api-openshift-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:56:01.172292 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:56:01.172253 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kwzx5" Nov 03 01:56:01.172484 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:56:01.172251 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kwzx5" event={"ID":"cbae01c1-e316-4efd-8de1-43f66fe35e1a","Type":"ContainerDied","Data":"be546a9f2931e280a71cdb00bdf481bf7c473b2965b6792d78d3d23b7f9cc295"} Nov 03 01:56:01.172484 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:56:01.172387 4421 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="be546a9f2931e280a71cdb00bdf481bf7c473b2965b6792d78d3d23b7f9cc295" Nov 03 01:56:07.231544 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:56:07.231507 4421 scope.go:122] "RemoveContainer" containerID="cf7b5bc2b438a13e0f3a081ffd892995a2af9bed004852cf9d28c45bea963c15" Nov 03 01:56:13.621261 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:56:13.621224 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kwzx5"] Nov 03 01:56:13.623459 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:56:13.623421 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kwzx5"] Nov 03 01:56:13.637599 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:56:13.637562 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-h4fl6"] Nov 03 01:56:13.637975 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:56:13.637959 4421 cpu_manager.go:414] "containerMap: removing container" podUID="cbae01c1-e316-4efd-8de1-43f66fe35e1a" containerName="revoke-keys" Nov 03 01:56:13.637975 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:56:13.637972 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbae01c1-e316-4efd-8de1-43f66fe35e1a" containerName="revoke-keys" Nov 03 01:56:13.638215 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:56:13.638053 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbae01c1-e316-4efd-8de1-43f66fe35e1a" containerName="revoke-keys" Nov 03 01:56:13.641368 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:56:13.641343 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-h4fl6" Nov 03 01:56:13.649824 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:56:13.649780 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:56:13.649824 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:56:13.649790 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 01:56:13.650010 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:56:13.649829 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 01:56:13.650403 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:56:13.650356 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-swfcr\"" Nov 03 01:56:13.653689 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:56:13.653662 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-h4fl6"] Nov 03 01:56:13.784443 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:56:13.784397 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/532cfac6-5118-42ca-959a-7eeaf46f63f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-h4fl6\" (UID: \"532cfac6-5118-42ca-959a-7eeaf46f63f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-h4fl6" Nov 03 01:56:13.784628 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:56:13.784602 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/532cfac6-5118-42ca-959a-7eeaf46f63f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-h4fl6\" (UID: \"532cfac6-5118-42ca-959a-7eeaf46f63f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-h4fl6" Nov 03 01:56:13.784677 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:56:13.784638 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bllr6\" (UniqueName: \"kubernetes.io/projected/532cfac6-5118-42ca-959a-7eeaf46f63f4-kube-api-access-bllr6\") pod \"maas-api-revoke-sub-3d4156f1-h4fl6\" (UID: \"532cfac6-5118-42ca-959a-7eeaf46f63f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-h4fl6" Nov 03 01:56:13.885260 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:56:13.885165 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/532cfac6-5118-42ca-959a-7eeaf46f63f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-h4fl6\" (UID: \"532cfac6-5118-42ca-959a-7eeaf46f63f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-h4fl6" Nov 03 01:56:13.885260 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:56:13.885251 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/532cfac6-5118-42ca-959a-7eeaf46f63f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-h4fl6\" (UID: \"532cfac6-5118-42ca-959a-7eeaf46f63f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-h4fl6" Nov 03 01:56:13.885491 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:56:13.885292 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bllr6\" (UniqueName: \"kubernetes.io/projected/532cfac6-5118-42ca-959a-7eeaf46f63f4-kube-api-access-bllr6\") pod \"maas-api-revoke-sub-3d4156f1-h4fl6\" (UID: \"532cfac6-5118-42ca-959a-7eeaf46f63f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-h4fl6" Nov 03 01:56:13.885917 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:56:13.885894 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/532cfac6-5118-42ca-959a-7eeaf46f63f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-h4fl6\" (UID: \"532cfac6-5118-42ca-959a-7eeaf46f63f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-h4fl6" Nov 03 01:56:13.888310 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:56:13.888260 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/532cfac6-5118-42ca-959a-7eeaf46f63f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-h4fl6\" (UID: \"532cfac6-5118-42ca-959a-7eeaf46f63f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-h4fl6" Nov 03 01:56:13.893879 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:56:13.893850 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bllr6\" (UniqueName: \"kubernetes.io/projected/532cfac6-5118-42ca-959a-7eeaf46f63f4-kube-api-access-bllr6\") pod \"maas-api-revoke-sub-3d4156f1-h4fl6\" (UID: \"532cfac6-5118-42ca-959a-7eeaf46f63f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-h4fl6" Nov 03 01:56:13.960426 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:56:13.960394 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-h4fl6" Nov 03 01:56:13.979582 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:56:13.979557 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cbae01c1-e316-4efd-8de1-43f66fe35e1a" path="/var/lib/kubelet/pods/cbae01c1-e316-4efd-8de1-43f66fe35e1a/volumes" Nov 03 01:56:14.241735 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:56:14.241705 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-h4fl6"] Nov 03 01:56:14.243920 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:56:14.243890 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod532cfac6_5118_42ca_959a_7eeaf46f63f4.slice/crio-cf38e7cceae2689138272d79f065507a772b43eca825ec6e80bfb888c104f804 WatchSource:0}: Error finding container cf38e7cceae2689138272d79f065507a772b43eca825ec6e80bfb888c104f804: Status 404 returned error can't find the container with id cf38e7cceae2689138272d79f065507a772b43eca825ec6e80bfb888c104f804 Nov 03 01:56:14.298486 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:56:14.298451 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-h4fl6" event={"ID":"532cfac6-5118-42ca-959a-7eeaf46f63f4","Type":"ContainerStarted","Data":"cf38e7cceae2689138272d79f065507a772b43eca825ec6e80bfb888c104f804"} Nov 03 01:56:14.669755 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:56:14.669643 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 01:56:15.305221 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:56:15.305130 4421 generic.go:356] "Generic (PLEG): container finished" podID="532cfac6-5118-42ca-959a-7eeaf46f63f4" containerID="fec631eb7450e0c74ff11ecf57f64e497340f110d31133579f6cd2435984ae7c" exitCode=0 Nov 03 01:56:15.305405 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:56:15.305216 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-h4fl6" event={"ID":"532cfac6-5118-42ca-959a-7eeaf46f63f4","Type":"ContainerDied","Data":"fec631eb7450e0c74ff11ecf57f64e497340f110d31133579f6cd2435984ae7c"} Nov 03 01:56:16.449240 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:56:16.449213 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-h4fl6" Nov 03 01:56:16.613664 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:56:16.613569 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/532cfac6-5118-42ca-959a-7eeaf46f63f4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/532cfac6-5118-42ca-959a-7eeaf46f63f4-maas-api-xks-ca\") pod \"532cfac6-5118-42ca-959a-7eeaf46f63f4\" (UID: \"532cfac6-5118-42ca-959a-7eeaf46f63f4\") " Nov 03 01:56:16.613664 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:56:16.613653 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/532cfac6-5118-42ca-959a-7eeaf46f63f4-kube-api-access-bllr6\" (UniqueName: \"kubernetes.io/projected/532cfac6-5118-42ca-959a-7eeaf46f63f4-kube-api-access-bllr6\") pod \"532cfac6-5118-42ca-959a-7eeaf46f63f4\" (UID: \"532cfac6-5118-42ca-959a-7eeaf46f63f4\") " Nov 03 01:56:16.613916 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:56:16.613758 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/532cfac6-5118-42ca-959a-7eeaf46f63f4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/532cfac6-5118-42ca-959a-7eeaf46f63f4-maas-api-openshift-ca\") pod \"532cfac6-5118-42ca-959a-7eeaf46f63f4\" (UID: \"532cfac6-5118-42ca-959a-7eeaf46f63f4\") " Nov 03 01:56:16.614120 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:56:16.614098 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/532cfac6-5118-42ca-959a-7eeaf46f63f4-maas-api-openshift-ca" pod "532cfac6-5118-42ca-959a-7eeaf46f63f4" (UID: "532cfac6-5118-42ca-959a-7eeaf46f63f4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:56:16.615776 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:56:16.615743 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/532cfac6-5118-42ca-959a-7eeaf46f63f4-kube-api-access-bllr6" pod "532cfac6-5118-42ca-959a-7eeaf46f63f4" (UID: "532cfac6-5118-42ca-959a-7eeaf46f63f4"). InnerVolumeSpecName "kube-api-access-bllr6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:56:16.615776 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:56:16.615747 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/532cfac6-5118-42ca-959a-7eeaf46f63f4-maas-api-xks-ca" pod "532cfac6-5118-42ca-959a-7eeaf46f63f4" (UID: "532cfac6-5118-42ca-959a-7eeaf46f63f4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:56:16.715224 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:56:16.715176 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bllr6\" (UniqueName: \"kubernetes.io/projected/532cfac6-5118-42ca-959a-7eeaf46f63f4-kube-api-access-bllr6\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:56:16.715224 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:56:16.715212 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/532cfac6-5118-42ca-959a-7eeaf46f63f4-maas-api-openshift-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:56:16.715224 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:56:16.715225 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/532cfac6-5118-42ca-959a-7eeaf46f63f4-maas-api-xks-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:56:17.318785 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:56:17.318750 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-h4fl6" Nov 03 01:56:17.319004 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:56:17.318777 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-h4fl6" event={"ID":"532cfac6-5118-42ca-959a-7eeaf46f63f4","Type":"ContainerDied","Data":"cf38e7cceae2689138272d79f065507a772b43eca825ec6e80bfb888c104f804"} Nov 03 01:56:17.319004 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:56:17.318838 4421 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cf38e7cceae2689138272d79f065507a772b43eca825ec6e80bfb888c104f804" Nov 03 01:56:38.998778 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:56:38.998737 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-wzcbf"] Nov 03 01:56:38.999459 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:56:38.999439 4421 cpu_manager.go:414] "containerMap: removing container" podUID="532cfac6-5118-42ca-959a-7eeaf46f63f4" containerName="revoke-keys" Nov 03 01:56:38.999521 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:56:38.999462 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="532cfac6-5118-42ca-959a-7eeaf46f63f4" containerName="revoke-keys" Nov 03 01:56:38.999578 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:56:38.999566 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="532cfac6-5118-42ca-959a-7eeaf46f63f4" containerName="revoke-keys" Nov 03 01:56:39.003408 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:56:39.003391 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-wzcbf" Nov 03 01:56:39.007295 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:56:39.007247 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 01:56:39.008371 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:56:39.008347 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:56:39.008506 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:56:39.008428 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 01:56:39.008506 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:56:39.008462 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-swfcr\"" Nov 03 01:56:39.009808 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:56:39.009779 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-wzcbf"] Nov 03 01:56:39.041881 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:56:39.041846 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49b5b322-05f4-4f73-9548-5b9e9c4e2831-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-wzcbf\" (UID: \"49b5b322-05f4-4f73-9548-5b9e9c4e2831\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-wzcbf" Nov 03 01:56:39.042083 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:56:39.041897 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49b5b322-05f4-4f73-9548-5b9e9c4e2831-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-wzcbf\" (UID: \"49b5b322-05f4-4f73-9548-5b9e9c4e2831\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-wzcbf" Nov 03 01:56:39.042174 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:56:39.042079 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnnhc\" (UniqueName: \"kubernetes.io/projected/49b5b322-05f4-4f73-9548-5b9e9c4e2831-kube-api-access-cnnhc\") pod \"maas-api-revoke-sub-e2fc9c5b-wzcbf\" (UID: \"49b5b322-05f4-4f73-9548-5b9e9c4e2831\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-wzcbf" Nov 03 01:56:39.142650 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:56:39.142613 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnnhc\" (UniqueName: \"kubernetes.io/projected/49b5b322-05f4-4f73-9548-5b9e9c4e2831-kube-api-access-cnnhc\") pod \"maas-api-revoke-sub-e2fc9c5b-wzcbf\" (UID: \"49b5b322-05f4-4f73-9548-5b9e9c4e2831\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-wzcbf" Nov 03 01:56:39.142840 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:56:39.142680 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49b5b322-05f4-4f73-9548-5b9e9c4e2831-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-wzcbf\" (UID: \"49b5b322-05f4-4f73-9548-5b9e9c4e2831\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-wzcbf" Nov 03 01:56:39.142840 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:56:39.142721 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49b5b322-05f4-4f73-9548-5b9e9c4e2831-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-wzcbf\" (UID: \"49b5b322-05f4-4f73-9548-5b9e9c4e2831\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-wzcbf" Nov 03 01:56:39.143491 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:56:39.143462 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49b5b322-05f4-4f73-9548-5b9e9c4e2831-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-wzcbf\" (UID: \"49b5b322-05f4-4f73-9548-5b9e9c4e2831\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-wzcbf" Nov 03 01:56:39.145177 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:56:39.145157 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49b5b322-05f4-4f73-9548-5b9e9c4e2831-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-wzcbf\" (UID: \"49b5b322-05f4-4f73-9548-5b9e9c4e2831\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-wzcbf" Nov 03 01:56:39.151813 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:56:39.151793 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnnhc\" (UniqueName: \"kubernetes.io/projected/49b5b322-05f4-4f73-9548-5b9e9c4e2831-kube-api-access-cnnhc\") pod \"maas-api-revoke-sub-e2fc9c5b-wzcbf\" (UID: \"49b5b322-05f4-4f73-9548-5b9e9c4e2831\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-wzcbf" Nov 03 01:56:39.320172 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:56:39.320084 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-wzcbf" Nov 03 01:56:39.597035 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:56:39.596937 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-wzcbf"] Nov 03 01:56:39.600264 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:56:39.600231 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49b5b322_05f4_4f73_9548_5b9e9c4e2831.slice/crio-effbeb9ad775683fb6cf1900749c8e573bcf6426fda2d1a2cc8acb7707b18862 WatchSource:0}: Error finding container effbeb9ad775683fb6cf1900749c8e573bcf6426fda2d1a2cc8acb7707b18862: Status 404 returned error can't find the container with id effbeb9ad775683fb6cf1900749c8e573bcf6426fda2d1a2cc8acb7707b18862 Nov 03 01:56:40.442665 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:56:40.442628 4421 generic.go:356] "Generic (PLEG): container finished" podID="49b5b322-05f4-4f73-9548-5b9e9c4e2831" containerID="e8e8e6a37d40b7f5ead1068794eb58e1b853c58b65cb3f5c0e880e1d338f7821" exitCode=0 Nov 03 01:56:40.443104 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:56:40.442708 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-wzcbf" event={"ID":"49b5b322-05f4-4f73-9548-5b9e9c4e2831","Type":"ContainerDied","Data":"e8e8e6a37d40b7f5ead1068794eb58e1b853c58b65cb3f5c0e880e1d338f7821"} Nov 03 01:56:40.443104 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:56:40.442770 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-wzcbf" event={"ID":"49b5b322-05f4-4f73-9548-5b9e9c4e2831","Type":"ContainerStarted","Data":"effbeb9ad775683fb6cf1900749c8e573bcf6426fda2d1a2cc8acb7707b18862"} Nov 03 01:56:41.580747 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:56:41.580722 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-wzcbf" Nov 03 01:56:41.666597 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:56:41.666549 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/49b5b322-05f4-4f73-9548-5b9e9c4e2831-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49b5b322-05f4-4f73-9548-5b9e9c4e2831-maas-api-xks-ca\") pod \"49b5b322-05f4-4f73-9548-5b9e9c4e2831\" (UID: \"49b5b322-05f4-4f73-9548-5b9e9c4e2831\") " Nov 03 01:56:41.666792 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:56:41.666646 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49b5b322-05f4-4f73-9548-5b9e9c4e2831-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49b5b322-05f4-4f73-9548-5b9e9c4e2831-maas-api-openshift-ca\") pod \"49b5b322-05f4-4f73-9548-5b9e9c4e2831\" (UID: \"49b5b322-05f4-4f73-9548-5b9e9c4e2831\") " Nov 03 01:56:41.666792 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:56:41.666682 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49b5b322-05f4-4f73-9548-5b9e9c4e2831-kube-api-access-cnnhc\" (UniqueName: \"kubernetes.io/projected/49b5b322-05f4-4f73-9548-5b9e9c4e2831-kube-api-access-cnnhc\") pod \"49b5b322-05f4-4f73-9548-5b9e9c4e2831\" (UID: \"49b5b322-05f4-4f73-9548-5b9e9c4e2831\") " Nov 03 01:56:41.667064 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:56:41.667041 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49b5b322-05f4-4f73-9548-5b9e9c4e2831-maas-api-openshift-ca" pod "49b5b322-05f4-4f73-9548-5b9e9c4e2831" (UID: "49b5b322-05f4-4f73-9548-5b9e9c4e2831"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:56:41.668764 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:56:41.668740 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49b5b322-05f4-4f73-9548-5b9e9c4e2831-kube-api-access-cnnhc" pod "49b5b322-05f4-4f73-9548-5b9e9c4e2831" (UID: "49b5b322-05f4-4f73-9548-5b9e9c4e2831"). InnerVolumeSpecName "kube-api-access-cnnhc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:56:41.668848 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:56:41.668823 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49b5b322-05f4-4f73-9548-5b9e9c4e2831-maas-api-xks-ca" pod "49b5b322-05f4-4f73-9548-5b9e9c4e2831" (UID: "49b5b322-05f4-4f73-9548-5b9e9c4e2831"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:56:41.767797 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:56:41.767756 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49b5b322-05f4-4f73-9548-5b9e9c4e2831-maas-api-xks-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:56:41.767797 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:56:41.767794 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49b5b322-05f4-4f73-9548-5b9e9c4e2831-maas-api-openshift-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:56:41.768041 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:56:41.767810 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cnnhc\" (UniqueName: \"kubernetes.io/projected/49b5b322-05f4-4f73-9548-5b9e9c4e2831-kube-api-access-cnnhc\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:56:42.455540 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:56:42.455505 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-wzcbf" event={"ID":"49b5b322-05f4-4f73-9548-5b9e9c4e2831","Type":"ContainerDied","Data":"effbeb9ad775683fb6cf1900749c8e573bcf6426fda2d1a2cc8acb7707b18862"} Nov 03 01:56:42.455540 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:56:42.455540 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-wzcbf" Nov 03 01:56:42.455767 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:56:42.455556 4421 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="effbeb9ad775683fb6cf1900749c8e573bcf6426fda2d1a2cc8acb7707b18862" Nov 03 01:56:44.669409 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:56:44.669369 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 01:56:49.211384 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:56:49.211344 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-496js"] Nov 03 01:56:49.211951 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:56:49.211742 4421 cpu_manager.go:414] "containerMap: removing container" podUID="49b5b322-05f4-4f73-9548-5b9e9c4e2831" containerName="revoke-keys" Nov 03 01:56:49.211951 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:56:49.211754 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49b5b322-05f4-4f73-9548-5b9e9c4e2831" containerName="revoke-keys" Nov 03 01:56:49.211951 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:56:49.211845 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="49b5b322-05f4-4f73-9548-5b9e9c4e2831" containerName="revoke-keys" Nov 03 01:56:49.215146 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:56:49.215124 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-496js" Nov 03 01:56:49.220748 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:56:49.220723 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:56:49.220932 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:56:49.220769 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 01:56:49.220932 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:56:49.220801 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-swfcr\"" Nov 03 01:56:49.220932 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:56:49.220869 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 01:56:49.223446 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:56:49.223424 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-496js"] Nov 03 01:56:49.341345 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:56:49.341307 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f1e75eb-34b8-4483-bb5f-82408c09f09e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-496js\" (UID: \"8f1e75eb-34b8-4483-bb5f-82408c09f09e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-496js" Nov 03 01:56:49.341535 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:56:49.341444 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f1e75eb-34b8-4483-bb5f-82408c09f09e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-496js\" (UID: \"8f1e75eb-34b8-4483-bb5f-82408c09f09e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-496js" Nov 03 01:56:49.341535 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:56:49.341500 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnptk\" (UniqueName: \"kubernetes.io/projected/8f1e75eb-34b8-4483-bb5f-82408c09f09e-kube-api-access-tnptk\") pod \"maas-api-revoke-sub-5a29efb3-496js\" (UID: \"8f1e75eb-34b8-4483-bb5f-82408c09f09e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-496js" Nov 03 01:56:49.441981 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:56:49.441946 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f1e75eb-34b8-4483-bb5f-82408c09f09e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-496js\" (UID: \"8f1e75eb-34b8-4483-bb5f-82408c09f09e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-496js" Nov 03 01:56:49.442187 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:56:49.441994 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnptk\" (UniqueName: \"kubernetes.io/projected/8f1e75eb-34b8-4483-bb5f-82408c09f09e-kube-api-access-tnptk\") pod \"maas-api-revoke-sub-5a29efb3-496js\" (UID: \"8f1e75eb-34b8-4483-bb5f-82408c09f09e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-496js" Nov 03 01:56:49.442187 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:56:49.442068 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f1e75eb-34b8-4483-bb5f-82408c09f09e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-496js\" (UID: \"8f1e75eb-34b8-4483-bb5f-82408c09f09e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-496js" Nov 03 01:56:49.442764 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:56:49.442742 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f1e75eb-34b8-4483-bb5f-82408c09f09e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-496js\" (UID: \"8f1e75eb-34b8-4483-bb5f-82408c09f09e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-496js" Nov 03 01:56:49.444797 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:56:49.444773 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f1e75eb-34b8-4483-bb5f-82408c09f09e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-496js\" (UID: \"8f1e75eb-34b8-4483-bb5f-82408c09f09e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-496js" Nov 03 01:56:49.450527 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:56:49.450493 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnptk\" (UniqueName: \"kubernetes.io/projected/8f1e75eb-34b8-4483-bb5f-82408c09f09e-kube-api-access-tnptk\") pod \"maas-api-revoke-sub-5a29efb3-496js\" (UID: \"8f1e75eb-34b8-4483-bb5f-82408c09f09e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-496js" Nov 03 01:56:49.530965 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:56:49.530930 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-496js" Nov 03 01:56:49.810137 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:56:49.810108 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-496js"] Nov 03 01:56:49.812324 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:56:49.812293 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f1e75eb_34b8_4483_bb5f_82408c09f09e.slice/crio-01241c311a1bea6f130d778642e01fa19db5fb7eae794ee8706a43f04306591c WatchSource:0}: Error finding container 01241c311a1bea6f130d778642e01fa19db5fb7eae794ee8706a43f04306591c: Status 404 returned error can't find the container with id 01241c311a1bea6f130d778642e01fa19db5fb7eae794ee8706a43f04306591c Nov 03 01:56:50.500403 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:56:50.500366 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-496js" event={"ID":"8f1e75eb-34b8-4483-bb5f-82408c09f09e","Type":"ContainerStarted","Data":"dac4e8f822975f87e1588b360c977c20c2424e4dee7db0c79cebd3447b013e70"} Nov 03 01:56:50.501051 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:56:50.500432 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-496js" event={"ID":"8f1e75eb-34b8-4483-bb5f-82408c09f09e","Type":"ContainerStarted","Data":"01241c311a1bea6f130d778642e01fa19db5fb7eae794ee8706a43f04306591c"} Nov 03 01:56:50.533926 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:56:50.533847 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-496js" podStartSLOduration=1.533826197 podStartE2EDuration="1.533826197s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:56:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:56:50.533446778 +0000 UTC m=+4552.704449874" watchObservedRunningTime="2026-11-03 01:56:50.533826197 +0000 UTC m=+4552.704829294" Nov 03 01:56:51.516672 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:56:51.516620 4421 generic.go:356] "Generic (PLEG): container finished" podID="8f1e75eb-34b8-4483-bb5f-82408c09f09e" containerID="dac4e8f822975f87e1588b360c977c20c2424e4dee7db0c79cebd3447b013e70" exitCode=0 Nov 03 01:56:51.517472 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:56:51.517441 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-496js" event={"ID":"8f1e75eb-34b8-4483-bb5f-82408c09f09e","Type":"ContainerDied","Data":"dac4e8f822975f87e1588b360c977c20c2424e4dee7db0c79cebd3447b013e70"} Nov 03 01:56:52.766022 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:56:52.765991 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-496js" Nov 03 01:56:52.892395 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:56:52.892309 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f1e75eb-34b8-4483-bb5f-82408c09f09e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f1e75eb-34b8-4483-bb5f-82408c09f09e-maas-api-openshift-ca\") pod \"8f1e75eb-34b8-4483-bb5f-82408c09f09e\" (UID: \"8f1e75eb-34b8-4483-bb5f-82408c09f09e\") " Nov 03 01:56:52.892395 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:56:52.892385 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f1e75eb-34b8-4483-bb5f-82408c09f09e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f1e75eb-34b8-4483-bb5f-82408c09f09e-maas-api-xks-ca\") pod \"8f1e75eb-34b8-4483-bb5f-82408c09f09e\" (UID: \"8f1e75eb-34b8-4483-bb5f-82408c09f09e\") " Nov 03 01:56:52.892650 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:56:52.892410 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f1e75eb-34b8-4483-bb5f-82408c09f09e-kube-api-access-tnptk\" (UniqueName: \"kubernetes.io/projected/8f1e75eb-34b8-4483-bb5f-82408c09f09e-kube-api-access-tnptk\") pod \"8f1e75eb-34b8-4483-bb5f-82408c09f09e\" (UID: \"8f1e75eb-34b8-4483-bb5f-82408c09f09e\") " Nov 03 01:56:52.892650 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:56:52.892583 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f1e75eb-34b8-4483-bb5f-82408c09f09e-maas-api-openshift-ca" pod "8f1e75eb-34b8-4483-bb5f-82408c09f09e" (UID: "8f1e75eb-34b8-4483-bb5f-82408c09f09e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:56:52.892836 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:56:52.892816 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f1e75eb-34b8-4483-bb5f-82408c09f09e-maas-api-openshift-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:56:52.894577 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:56:52.894555 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f1e75eb-34b8-4483-bb5f-82408c09f09e-kube-api-access-tnptk" pod "8f1e75eb-34b8-4483-bb5f-82408c09f09e" (UID: "8f1e75eb-34b8-4483-bb5f-82408c09f09e"). InnerVolumeSpecName "kube-api-access-tnptk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:56:52.894659 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:56:52.894620 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f1e75eb-34b8-4483-bb5f-82408c09f09e-maas-api-xks-ca" pod "8f1e75eb-34b8-4483-bb5f-82408c09f09e" (UID: "8f1e75eb-34b8-4483-bb5f-82408c09f09e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:56:52.994194 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:56:52.994143 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f1e75eb-34b8-4483-bb5f-82408c09f09e-maas-api-xks-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:56:52.994194 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:56:52.994191 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tnptk\" (UniqueName: \"kubernetes.io/projected/8f1e75eb-34b8-4483-bb5f-82408c09f09e-kube-api-access-tnptk\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:56:53.567924 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:56:53.567887 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-496js" event={"ID":"8f1e75eb-34b8-4483-bb5f-82408c09f09e","Type":"ContainerDied","Data":"01241c311a1bea6f130d778642e01fa19db5fb7eae794ee8706a43f04306591c"} Nov 03 01:56:53.567924 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:56:53.567914 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-496js" Nov 03 01:56:53.568155 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:56:53.567940 4421 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="01241c311a1bea6f130d778642e01fa19db5fb7eae794ee8706a43f04306591c" Nov 03 01:56:56.585484 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:56:56.585451 4421 generic.go:356] "Generic (PLEG): container finished" podID="4d579ceb-59fa-499a-b241-7f4e940976c3" containerID="2514591708bc044c70e2e2e96c4736520edf271ef0bfec5ab01134ecd9417785" exitCode=1 Nov 03 01:56:56.585938 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:56:56.585502 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-6f98fffc9c-l2ptp" event={"ID":"4d579ceb-59fa-499a-b241-7f4e940976c3","Type":"ContainerDied","Data":"2514591708bc044c70e2e2e96c4736520edf271ef0bfec5ab01134ecd9417785"} Nov 03 01:56:56.585938 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:56:56.585877 4421 scope.go:122] "RemoveContainer" containerID="2514591708bc044c70e2e2e96c4736520edf271ef0bfec5ab01134ecd9417785" Nov 03 01:56:57.592949 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:56:57.592910 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-6f98fffc9c-l2ptp" event={"ID":"4d579ceb-59fa-499a-b241-7f4e940976c3","Type":"ContainerStarted","Data":"9478f412acef3d715737b3eea75a1c0c7d6e97124696b2e1412688f3adfce071"} Nov 03 01:56:57.593371 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:56:57.593120 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-6f98fffc9c-l2ptp" Nov 03 01:57:07.256448 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:57:07.256405 4421 scope.go:122] "RemoveContainer" containerID="a02b926898878e5f8069cea00dabb7b39f1e4097e11f6f00ddfaffcef4496ba0" Nov 03 01:57:07.266082 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:57:07.266062 4421 scope.go:122] "RemoveContainer" containerID="3ddc4cebbe0a7add21a6f997287e93e9002492641e516163c9fdc86cbee86290" Nov 03 01:57:07.274440 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:57:07.274421 4421 scope.go:122] "RemoveContainer" containerID="32b7b6b8272bd4c766f25ff7ebe97e9e434753850d807dc1d8f58f8daedb95c9" Nov 03 01:57:07.647405 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:57:07.647315 4421 generic.go:356] "Generic (PLEG): container finished" podID="4d579ceb-59fa-499a-b241-7f4e940976c3" containerID="9478f412acef3d715737b3eea75a1c0c7d6e97124696b2e1412688f3adfce071" exitCode=1 Nov 03 01:57:07.647554 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:57:07.647388 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-6f98fffc9c-l2ptp" event={"ID":"4d579ceb-59fa-499a-b241-7f4e940976c3","Type":"ContainerDied","Data":"9478f412acef3d715737b3eea75a1c0c7d6e97124696b2e1412688f3adfce071"} Nov 03 01:57:07.647554 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:57:07.647467 4421 scope.go:122] "RemoveContainer" containerID="2514591708bc044c70e2e2e96c4736520edf271ef0bfec5ab01134ecd9417785" Nov 03 01:57:07.647890 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:57:07.647871 4421 scope.go:122] "RemoveContainer" containerID="9478f412acef3d715737b3eea75a1c0c7d6e97124696b2e1412688f3adfce071" Nov 03 01:57:07.648168 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:57:07.648143 4421 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 10s restarting failed container=manager pod=jobset-controller-manager-6f98fffc9c-l2ptp_openshift-jobset-operator(4d579ceb-59fa-499a-b241-7f4e940976c3)\"" pod="openshift-jobset-operator/jobset-controller-manager-6f98fffc9c-l2ptp" podUID="4d579ceb-59fa-499a-b241-7f4e940976c3" Nov 03 01:57:14.668464 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:57:14.668417 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 01:57:18.971473 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:57:18.971433 4421 scope.go:122] "RemoveContainer" containerID="9478f412acef3d715737b3eea75a1c0c7d6e97124696b2e1412688f3adfce071" Nov 03 01:57:19.715107 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:57:19.715070 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-6f98fffc9c-l2ptp" event={"ID":"4d579ceb-59fa-499a-b241-7f4e940976c3","Type":"ContainerStarted","Data":"11413678b2fb24ff03a8543a6001854e2863e1cef3beef5f3f7aa7c4c57ea3b4"} Nov 03 01:57:19.715333 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:57:19.715289 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-6f98fffc9c-l2ptp" Nov 03 01:57:30.334962 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:57:30.334927 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7fde87-7bf9f9f857-45v2f"] Nov 03 01:57:30.335507 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:57:30.335428 4421 cpu_manager.go:414] "containerMap: removing container" podUID="8f1e75eb-34b8-4483-bb5f-82408c09f09e" containerName="revoke-keys" Nov 03 01:57:30.335507 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:57:30.335446 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f1e75eb-34b8-4483-bb5f-82408c09f09e" containerName="revoke-keys" Nov 03 01:57:30.335657 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:57:30.335572 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f1e75eb-34b8-4483-bb5f-82408c09f09e" containerName="revoke-keys" Nov 03 01:57:30.339251 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:57:30.339225 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7fde87-7bf9f9f857-45v2f" Nov 03 01:57:30.352609 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:57:30.351811 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 01:57:30.352609 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:57:30.352028 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 01:57:30.352609 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:57:30.352039 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-7fde87-serving-cert\"" Nov 03 01:57:30.352609 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:57:30.352075 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-nfcxl\"" Nov 03 01:57:30.352609 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:57:30.352225 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 03 01:57:30.355755 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:57:30.355728 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7fde87-7bf9f9f857-45v2f"] Nov 03 01:57:30.361951 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:57:30.361921 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d46a8aac-1c79-4df3-b53f-413706517a91-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-7fde87-7bf9f9f857-45v2f\" (UID: \"d46a8aac-1c79-4df3-b53f-413706517a91\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7fde87-7bf9f9f857-45v2f" Nov 03 01:57:30.362316 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:57:30.362266 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d46a8aac-1c79-4df3-b53f-413706517a91-metrics-certs\") pod \"maas-api-e2e-worker-single-main-7fde87-7bf9f9f857-45v2f\" (UID: \"d46a8aac-1c79-4df3-b53f-413706517a91\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7fde87-7bf9f9f857-45v2f" Nov 03 01:57:30.362953 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:57:30.362930 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67r4g\" (UniqueName: \"kubernetes.io/projected/d46a8aac-1c79-4df3-b53f-413706517a91-kube-api-access-67r4g\") pod \"maas-api-e2e-worker-single-main-7fde87-7bf9f9f857-45v2f\" (UID: \"d46a8aac-1c79-4df3-b53f-413706517a91\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7fde87-7bf9f9f857-45v2f" Nov 03 01:57:30.363183 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:57:30.363163 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d46a8aac-1c79-4df3-b53f-413706517a91-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-7fde87-7bf9f9f857-45v2f\" (UID: \"d46a8aac-1c79-4df3-b53f-413706517a91\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7fde87-7bf9f9f857-45v2f" Nov 03 01:57:30.464410 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:57:30.464374 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d46a8aac-1c79-4df3-b53f-413706517a91-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-7fde87-7bf9f9f857-45v2f\" (UID: \"d46a8aac-1c79-4df3-b53f-413706517a91\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7fde87-7bf9f9f857-45v2f" Nov 03 01:57:30.464590 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:57:30.464426 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d46a8aac-1c79-4df3-b53f-413706517a91-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-7fde87-7bf9f9f857-45v2f\" (UID: \"d46a8aac-1c79-4df3-b53f-413706517a91\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7fde87-7bf9f9f857-45v2f" Nov 03 01:57:30.464590 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:57:30.464483 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d46a8aac-1c79-4df3-b53f-413706517a91-metrics-certs\") pod \"maas-api-e2e-worker-single-main-7fde87-7bf9f9f857-45v2f\" (UID: \"d46a8aac-1c79-4df3-b53f-413706517a91\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7fde87-7bf9f9f857-45v2f" Nov 03 01:57:30.464590 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:57:30.464521 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67r4g\" (UniqueName: \"kubernetes.io/projected/d46a8aac-1c79-4df3-b53f-413706517a91-kube-api-access-67r4g\") pod \"maas-api-e2e-worker-single-main-7fde87-7bf9f9f857-45v2f\" (UID: \"d46a8aac-1c79-4df3-b53f-413706517a91\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7fde87-7bf9f9f857-45v2f" Nov 03 01:57:30.465199 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:57:30.465170 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d46a8aac-1c79-4df3-b53f-413706517a91-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-7fde87-7bf9f9f857-45v2f\" (UID: \"d46a8aac-1c79-4df3-b53f-413706517a91\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7fde87-7bf9f9f857-45v2f" Nov 03 01:57:30.467127 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:57:30.467099 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d46a8aac-1c79-4df3-b53f-413706517a91-metrics-certs\") pod \"maas-api-e2e-worker-single-main-7fde87-7bf9f9f857-45v2f\" (UID: \"d46a8aac-1c79-4df3-b53f-413706517a91\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7fde87-7bf9f9f857-45v2f" Nov 03 01:57:30.467259 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:57:30.467111 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d46a8aac-1c79-4df3-b53f-413706517a91-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-7fde87-7bf9f9f857-45v2f\" (UID: \"d46a8aac-1c79-4df3-b53f-413706517a91\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7fde87-7bf9f9f857-45v2f" Nov 03 01:57:30.473369 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:57:30.473344 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-7fde87-6b96867c8jxctp"] Nov 03 01:57:30.474993 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:57:30.474968 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67r4g\" (UniqueName: \"kubernetes.io/projected/d46a8aac-1c79-4df3-b53f-413706517a91-kube-api-access-67r4g\") pod \"maas-api-e2e-worker-single-main-7fde87-7bf9f9f857-45v2f\" (UID: \"d46a8aac-1c79-4df3-b53f-413706517a91\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7fde87-7bf9f9f857-45v2f" Nov 03 01:57:30.478002 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:57:30.477978 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-7fde87-6b96867c8jxctp" Nov 03 01:57:30.488864 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:57:30.488829 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-7fde87-6b96867c8jxctp"] Nov 03 01:57:30.565971 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:57:30.565939 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9w6sn\" (UniqueName: \"kubernetes.io/projected/bfaa1793-a630-4d73-b226-6943e7db3357-kube-api-access-9w6sn\") pod \"payload-processing-e2e-worker-single-main-7fde87-6b96867c8jxctp\" (UID: \"bfaa1793-a630-4d73-b226-6943e7db3357\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-7fde87-6b96867c8jxctp" Nov 03 01:57:30.566162 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:57:30.565989 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfaa1793-a630-4d73-b226-6943e7db3357-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-7fde87-6b96867c8jxctp\" (UID: \"bfaa1793-a630-4d73-b226-6943e7db3357\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-7fde87-6b96867c8jxctp" Nov 03 01:57:30.569713 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:57:30.569686 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-7fde87\"" Nov 03 01:57:30.569713 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:57:30.569686 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-bf3a2bea-dockercfg-d2pfg\"" Nov 03 01:57:30.654107 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:57:30.654010 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7fde87-7bf9f9f857-45v2f" Nov 03 01:57:30.667598 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:57:30.667563 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9w6sn\" (UniqueName: \"kubernetes.io/projected/bfaa1793-a630-4d73-b226-6943e7db3357-kube-api-access-9w6sn\") pod \"payload-processing-e2e-worker-single-main-7fde87-6b96867c8jxctp\" (UID: \"bfaa1793-a630-4d73-b226-6943e7db3357\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-7fde87-6b96867c8jxctp" Nov 03 01:57:30.667721 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:57:30.667652 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfaa1793-a630-4d73-b226-6943e7db3357-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-7fde87-6b96867c8jxctp\" (UID: \"bfaa1793-a630-4d73-b226-6943e7db3357\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-7fde87-6b96867c8jxctp" Nov 03 01:57:30.668345 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:57:30.668318 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfaa1793-a630-4d73-b226-6943e7db3357-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-7fde87-6b96867c8jxctp\" (UID: \"bfaa1793-a630-4d73-b226-6943e7db3357\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-7fde87-6b96867c8jxctp" Nov 03 01:57:30.675731 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:57:30.675708 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9w6sn\" (UniqueName: \"kubernetes.io/projected/bfaa1793-a630-4d73-b226-6943e7db3357-kube-api-access-9w6sn\") pod \"payload-processing-e2e-worker-single-main-7fde87-6b96867c8jxctp\" (UID: \"bfaa1793-a630-4d73-b226-6943e7db3357\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-7fde87-6b96867c8jxctp" Nov 03 01:57:30.727914 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:57:30.727886 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-6f98fffc9c-l2ptp" Nov 03 01:57:30.804866 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:57:30.804830 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-7fde87-6b96867c8jxctp" Nov 03 01:57:30.936464 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:57:30.936436 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7fde87-7bf9f9f857-45v2f"] Nov 03 01:57:30.937995 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:57:30.937974 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd46a8aac_1c79_4df3_b53f_413706517a91.slice/crio-59467f4a3d9e583ee982f1ce4eceed8be5858db6bd56990bb291399d4761eec4 WatchSource:0}: Error finding container 59467f4a3d9e583ee982f1ce4eceed8be5858db6bd56990bb291399d4761eec4: Status 404 returned error can't find the container with id 59467f4a3d9e583ee982f1ce4eceed8be5858db6bd56990bb291399d4761eec4 Nov 03 01:57:30.939444 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:57:30.939423 4421 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 01:57:31.083679 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:57:31.083652 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-7fde87-6b96867c8jxctp"] Nov 03 01:57:31.085440 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:57:31.085408 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfaa1793_a630_4d73_b226_6943e7db3357.slice/crio-b651f6c9820abfb88eedf42fd6eedb8be34da4ef7299b997f5869077496cdfe2 WatchSource:0}: Error finding container b651f6c9820abfb88eedf42fd6eedb8be34da4ef7299b997f5869077496cdfe2: Status 404 returned error can't find the container with id b651f6c9820abfb88eedf42fd6eedb8be34da4ef7299b997f5869077496cdfe2 Nov 03 01:57:31.782461 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:57:31.782422 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-7fde87-6b96867c8jxctp" event={"ID":"bfaa1793-a630-4d73-b226-6943e7db3357","Type":"ContainerStarted","Data":"a50b05dae259bb4264ab32b07eb5c5b82bc3f5332badbcdcba3498fca530c430"} Nov 03 01:57:31.783015 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:57:31.782484 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-7fde87-6b96867c8jxctp" event={"ID":"bfaa1793-a630-4d73-b226-6943e7db3357","Type":"ContainerStarted","Data":"b651f6c9820abfb88eedf42fd6eedb8be34da4ef7299b997f5869077496cdfe2"} Nov 03 01:57:31.783015 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:57:31.782625 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-7fde87-6b96867c8jxctp" Nov 03 01:57:31.784091 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:57:31.784066 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7fde87-7bf9f9f857-45v2f" event={"ID":"d46a8aac-1c79-4df3-b53f-413706517a91","Type":"ContainerStarted","Data":"58161848ebff66f2d4097ce3e1129beb030ee154ef77ca77d7c47106e0613a6e"} Nov 03 01:57:31.784211 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:57:31.784112 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7fde87-7bf9f9f857-45v2f" Nov 03 01:57:31.784211 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:57:31.784130 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7fde87-7bf9f9f857-45v2f" event={"ID":"d46a8aac-1c79-4df3-b53f-413706517a91","Type":"ContainerStarted","Data":"59467f4a3d9e583ee982f1ce4eceed8be5858db6bd56990bb291399d4761eec4"} Nov 03 01:57:31.824143 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:57:31.824087 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7fde87-7bf9f9f857-45v2f" podStartSLOduration=1.166034631 podStartE2EDuration="1.824073285s" totalImagesPullingTime="658.038654ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:57:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:57:31.823009918 +0000 UTC m=+4593.994013015" watchObservedRunningTime="2026-11-03 01:57:31.824073285 +0000 UTC m=+4593.995076381" Nov 03 01:57:31.825486 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:57:31.825454 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-7fde87-6b96867c8jxctp" podStartSLOduration=1.825444487 podStartE2EDuration="1.825444487s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:57:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:57:31.799693391 +0000 UTC m=+4593.970696489" watchObservedRunningTime="2026-11-03 01:57:31.825444487 +0000 UTC m=+4593.996447582" Nov 03 01:57:37.795247 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:57:37.795203 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7fde87-7bf9f9f857-45v2f" Nov 03 01:57:42.793294 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:57:42.793247 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-7fde87-6b96867c8jxctp" Nov 03 01:57:44.668357 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:57:44.668320 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 01:57:46.380531 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:57:46.380490 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-7fde87/simulator-single-main-7fde87-kserve-5db8b76fdf-x74kx"] Nov 03 01:57:46.384556 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:57:46.384530 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-7fde87/simulator-single-main-7fde87-kserve-5db8b76fdf-x74kx" Nov 03 01:57:46.394113 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:57:46.394078 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-7fde87\"/\"openshift-service-ca.crt\"" Nov 03 01:57:46.394320 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:57:46.394294 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-7fde87\"/\"simulator-single-main-7fde87-kserve-self-signed-certs\"" Nov 03 01:57:46.394422 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:57:46.394350 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-7fde87\"/\"kube-root-ca.crt\"" Nov 03 01:57:46.394498 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:57:46.394431 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-7fde87\"/\"default-dockercfg-j6bkr\"" Nov 03 01:57:46.395427 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:57:46.395399 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-7fde87/simulator-single-main-7fde87-kserve-5db8b76fdf-x74kx"] Nov 03 01:57:46.533893 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:57:46.533853 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f2bc993-f2ac-48bd-b2d1-e00be41d9fbf-home\") pod \"simulator-single-main-7fde87-kserve-5db8b76fdf-x74kx\" (UID: \"2f2bc993-f2ac-48bd-b2d1-e00be41d9fbf\") " pod="e2e-models-e2e-worker-single-main-7fde87/simulator-single-main-7fde87-kserve-5db8b76fdf-x74kx" Nov 03 01:57:46.534110 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:57:46.533984 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f2bc993-f2ac-48bd-b2d1-e00be41d9fbf-model-cache\") pod \"simulator-single-main-7fde87-kserve-5db8b76fdf-x74kx\" (UID: \"2f2bc993-f2ac-48bd-b2d1-e00be41d9fbf\") " pod="e2e-models-e2e-worker-single-main-7fde87/simulator-single-main-7fde87-kserve-5db8b76fdf-x74kx" Nov 03 01:57:46.534110 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:57:46.534084 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f2bc993-f2ac-48bd-b2d1-e00be41d9fbf-dshm\") pod \"simulator-single-main-7fde87-kserve-5db8b76fdf-x74kx\" (UID: \"2f2bc993-f2ac-48bd-b2d1-e00be41d9fbf\") " pod="e2e-models-e2e-worker-single-main-7fde87/simulator-single-main-7fde87-kserve-5db8b76fdf-x74kx" Nov 03 01:57:46.534251 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:57:46.534172 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f2bc993-f2ac-48bd-b2d1-e00be41d9fbf-tmp-dir\") pod \"simulator-single-main-7fde87-kserve-5db8b76fdf-x74kx\" (UID: \"2f2bc993-f2ac-48bd-b2d1-e00be41d9fbf\") " pod="e2e-models-e2e-worker-single-main-7fde87/simulator-single-main-7fde87-kserve-5db8b76fdf-x74kx" Nov 03 01:57:46.534251 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:57:46.534194 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f2bc993-f2ac-48bd-b2d1-e00be41d9fbf-tls-certs\") pod \"simulator-single-main-7fde87-kserve-5db8b76fdf-x74kx\" (UID: \"2f2bc993-f2ac-48bd-b2d1-e00be41d9fbf\") " pod="e2e-models-e2e-worker-single-main-7fde87/simulator-single-main-7fde87-kserve-5db8b76fdf-x74kx" Nov 03 01:57:46.534419 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:57:46.534261 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cngtb\" (UniqueName: \"kubernetes.io/projected/2f2bc993-f2ac-48bd-b2d1-e00be41d9fbf-kube-api-access-cngtb\") pod \"simulator-single-main-7fde87-kserve-5db8b76fdf-x74kx\" (UID: \"2f2bc993-f2ac-48bd-b2d1-e00be41d9fbf\") " pod="e2e-models-e2e-worker-single-main-7fde87/simulator-single-main-7fde87-kserve-5db8b76fdf-x74kx" Nov 03 01:57:46.635681 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:57:46.635587 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f2bc993-f2ac-48bd-b2d1-e00be41d9fbf-home\") pod \"simulator-single-main-7fde87-kserve-5db8b76fdf-x74kx\" (UID: \"2f2bc993-f2ac-48bd-b2d1-e00be41d9fbf\") " pod="e2e-models-e2e-worker-single-main-7fde87/simulator-single-main-7fde87-kserve-5db8b76fdf-x74kx" Nov 03 01:57:46.635681 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:57:46.635637 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f2bc993-f2ac-48bd-b2d1-e00be41d9fbf-model-cache\") pod \"simulator-single-main-7fde87-kserve-5db8b76fdf-x74kx\" (UID: \"2f2bc993-f2ac-48bd-b2d1-e00be41d9fbf\") " pod="e2e-models-e2e-worker-single-main-7fde87/simulator-single-main-7fde87-kserve-5db8b76fdf-x74kx" Nov 03 01:57:46.635681 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:57:46.635665 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f2bc993-f2ac-48bd-b2d1-e00be41d9fbf-dshm\") pod \"simulator-single-main-7fde87-kserve-5db8b76fdf-x74kx\" (UID: \"2f2bc993-f2ac-48bd-b2d1-e00be41d9fbf\") " pod="e2e-models-e2e-worker-single-main-7fde87/simulator-single-main-7fde87-kserve-5db8b76fdf-x74kx" Nov 03 01:57:46.635978 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:57:46.635704 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f2bc993-f2ac-48bd-b2d1-e00be41d9fbf-tmp-dir\") pod \"simulator-single-main-7fde87-kserve-5db8b76fdf-x74kx\" (UID: \"2f2bc993-f2ac-48bd-b2d1-e00be41d9fbf\") " pod="e2e-models-e2e-worker-single-main-7fde87/simulator-single-main-7fde87-kserve-5db8b76fdf-x74kx" Nov 03 01:57:46.635978 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:57:46.635723 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f2bc993-f2ac-48bd-b2d1-e00be41d9fbf-tls-certs\") pod \"simulator-single-main-7fde87-kserve-5db8b76fdf-x74kx\" (UID: \"2f2bc993-f2ac-48bd-b2d1-e00be41d9fbf\") " pod="e2e-models-e2e-worker-single-main-7fde87/simulator-single-main-7fde87-kserve-5db8b76fdf-x74kx" Nov 03 01:57:46.635978 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:57:46.635748 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cngtb\" (UniqueName: \"kubernetes.io/projected/2f2bc993-f2ac-48bd-b2d1-e00be41d9fbf-kube-api-access-cngtb\") pod \"simulator-single-main-7fde87-kserve-5db8b76fdf-x74kx\" (UID: \"2f2bc993-f2ac-48bd-b2d1-e00be41d9fbf\") " pod="e2e-models-e2e-worker-single-main-7fde87/simulator-single-main-7fde87-kserve-5db8b76fdf-x74kx" Nov 03 01:57:46.636176 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:57:46.636011 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f2bc993-f2ac-48bd-b2d1-e00be41d9fbf-home\") pod \"simulator-single-main-7fde87-kserve-5db8b76fdf-x74kx\" (UID: \"2f2bc993-f2ac-48bd-b2d1-e00be41d9fbf\") " pod="e2e-models-e2e-worker-single-main-7fde87/simulator-single-main-7fde87-kserve-5db8b76fdf-x74kx" Nov 03 01:57:46.636176 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:57:46.636159 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f2bc993-f2ac-48bd-b2d1-e00be41d9fbf-tmp-dir\") pod \"simulator-single-main-7fde87-kserve-5db8b76fdf-x74kx\" (UID: \"2f2bc993-f2ac-48bd-b2d1-e00be41d9fbf\") " pod="e2e-models-e2e-worker-single-main-7fde87/simulator-single-main-7fde87-kserve-5db8b76fdf-x74kx" Nov 03 01:57:46.636289 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:57:46.636195 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f2bc993-f2ac-48bd-b2d1-e00be41d9fbf-model-cache\") pod \"simulator-single-main-7fde87-kserve-5db8b76fdf-x74kx\" (UID: \"2f2bc993-f2ac-48bd-b2d1-e00be41d9fbf\") " pod="e2e-models-e2e-worker-single-main-7fde87/simulator-single-main-7fde87-kserve-5db8b76fdf-x74kx" Nov 03 01:57:46.638103 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:57:46.638079 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f2bc993-f2ac-48bd-b2d1-e00be41d9fbf-dshm\") pod \"simulator-single-main-7fde87-kserve-5db8b76fdf-x74kx\" (UID: \"2f2bc993-f2ac-48bd-b2d1-e00be41d9fbf\") " pod="e2e-models-e2e-worker-single-main-7fde87/simulator-single-main-7fde87-kserve-5db8b76fdf-x74kx" Nov 03 01:57:46.638252 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:57:46.638235 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f2bc993-f2ac-48bd-b2d1-e00be41d9fbf-tls-certs\") pod \"simulator-single-main-7fde87-kserve-5db8b76fdf-x74kx\" (UID: \"2f2bc993-f2ac-48bd-b2d1-e00be41d9fbf\") " pod="e2e-models-e2e-worker-single-main-7fde87/simulator-single-main-7fde87-kserve-5db8b76fdf-x74kx" Nov 03 01:57:46.643974 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:57:46.643949 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cngtb\" (UniqueName: \"kubernetes.io/projected/2f2bc993-f2ac-48bd-b2d1-e00be41d9fbf-kube-api-access-cngtb\") pod \"simulator-single-main-7fde87-kserve-5db8b76fdf-x74kx\" (UID: \"2f2bc993-f2ac-48bd-b2d1-e00be41d9fbf\") " pod="e2e-models-e2e-worker-single-main-7fde87/simulator-single-main-7fde87-kserve-5db8b76fdf-x74kx" Nov 03 01:57:46.698475 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:57:46.698436 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-7fde87/simulator-single-main-7fde87-kserve-5db8b76fdf-x74kx" Nov 03 01:57:46.976226 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:57:46.976188 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-7fde87/simulator-single-main-7fde87-kserve-5db8b76fdf-x74kx"] Nov 03 01:57:46.978234 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:57:46.978204 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2f2bc993_f2ac_48bd_b2d1_e00be41d9fbf.slice/crio-88c16d94780d3ce210c81d873d2f13657a873ba355f198c17b1dce35d2c95672 WatchSource:0}: Error finding container 88c16d94780d3ce210c81d873d2f13657a873ba355f198c17b1dce35d2c95672: Status 404 returned error can't find the container with id 88c16d94780d3ce210c81d873d2f13657a873ba355f198c17b1dce35d2c95672 Nov 03 01:57:47.874301 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:57:47.873075 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-7fde87/simulator-single-main-7fde87-kserve-5db8b76fdf-x74kx" event={"ID":"2f2bc993-f2ac-48bd-b2d1-e00be41d9fbf","Type":"ContainerStarted","Data":"9c6069785c718ba015fefea527fcc31ce7e55dea5a51a3769539e4e3f710800c"} Nov 03 01:57:47.874301 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:57:47.873127 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-7fde87/simulator-single-main-7fde87-kserve-5db8b76fdf-x74kx" event={"ID":"2f2bc993-f2ac-48bd-b2d1-e00be41d9fbf","Type":"ContainerStarted","Data":"88c16d94780d3ce210c81d873d2f13657a873ba355f198c17b1dce35d2c95672"} Nov 03 01:57:47.890678 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:57:47.890627 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-7fde87/simulator-single-main-7fde87-kserve-5db8b76fdf-x74kx" podStartSLOduration=1.890613411 podStartE2EDuration="1.890613411s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:57:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:57:47.890329867 +0000 UTC m=+4610.061332987" watchObservedRunningTime="2026-11-03 01:57:47.890613411 +0000 UTC m=+4610.061616507" Nov 03 01:57:48.699172 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:57:48.699112 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-7fde87/simulator-single-main-7fde87-kserve-5db8b76fdf-x74kx" Nov 03 01:57:56.699558 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:57:56.699520 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-7fde87/simulator-single-main-7fde87-kserve-5db8b76fdf-x74kx" Nov 03 01:57:56.712645 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:57:56.712617 4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-7fde87/simulator-single-main-7fde87-kserve-5db8b76fdf-x74kx" Nov 03 01:57:56.930806 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:57:56.930774 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-7fde87/simulator-single-main-7fde87-kserve-5db8b76fdf-x74kx" Nov 03 01:58:14.669410 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:58:14.669328 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 01:58:21.022434 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:58:21.022396 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-jqhsg"] Nov 03 01:58:21.024556 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:58:21.024533 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-jqhsg"] Nov 03 01:58:21.977701 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:58:21.977658 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d267a793-0377-46a3-8ce5-1e693dd96410" path="/var/lib/kubelet/pods/d267a793-0377-46a3-8ce5-1e693dd96410/volumes" Nov 03 01:58:31.762432 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:58:31.762384 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-7fde87/simulator-single-main-7fde87-kserve-5db8b76fdf-x74kx"] Nov 03 01:58:32.347425 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:58:32.347387 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-7fde87-6b96867c8jxctp"] Nov 03 01:58:33.554760 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:58:33.554724 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7fde87-7bf9f9f857-45v2f"] Nov 03 01:58:33.555208 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:58:33.554986 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7fde87-7bf9f9f857-45v2f" podUID="d46a8aac-1c79-4df3-b53f-413706517a91" containerName="maas-api" containerID="cri-o://58161848ebff66f2d4097ce3e1129beb030ee154ef77ca77d7c47106e0613a6e" gracePeriod=30 Nov 03 01:58:33.812009 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:58:33.811947 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7fde87-7bf9f9f857-45v2f" Nov 03 01:58:33.915940 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:58:33.915879 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d46a8aac-1c79-4df3-b53f-413706517a91-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d46a8aac-1c79-4df3-b53f-413706517a91-openshift-service-ca\") pod \"d46a8aac-1c79-4df3-b53f-413706517a91\" (UID: \"d46a8aac-1c79-4df3-b53f-413706517a91\") " Nov 03 01:58:33.916175 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:58:33.915971 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d46a8aac-1c79-4df3-b53f-413706517a91-kube-api-access-67r4g\" (UniqueName: \"kubernetes.io/projected/d46a8aac-1c79-4df3-b53f-413706517a91-kube-api-access-67r4g\") pod \"d46a8aac-1c79-4df3-b53f-413706517a91\" (UID: \"d46a8aac-1c79-4df3-b53f-413706517a91\") " Nov 03 01:58:33.916175 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:58:33.915995 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d46a8aac-1c79-4df3-b53f-413706517a91-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d46a8aac-1c79-4df3-b53f-413706517a91-metrics-certs\") pod \"d46a8aac-1c79-4df3-b53f-413706517a91\" (UID: \"d46a8aac-1c79-4df3-b53f-413706517a91\") " Nov 03 01:58:33.916175 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:58:33.916044 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d46a8aac-1c79-4df3-b53f-413706517a91-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d46a8aac-1c79-4df3-b53f-413706517a91-maas-api-tls\") pod \"d46a8aac-1c79-4df3-b53f-413706517a91\" (UID: \"d46a8aac-1c79-4df3-b53f-413706517a91\") " Nov 03 01:58:33.916436 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:58:33.916404 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d46a8aac-1c79-4df3-b53f-413706517a91-openshift-service-ca" pod "d46a8aac-1c79-4df3-b53f-413706517a91" (UID: "d46a8aac-1c79-4df3-b53f-413706517a91"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:58:33.918254 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:58:33.918231 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d46a8aac-1c79-4df3-b53f-413706517a91-metrics-certs" pod "d46a8aac-1c79-4df3-b53f-413706517a91" (UID: "d46a8aac-1c79-4df3-b53f-413706517a91"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:58:33.918350 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:58:33.918302 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d46a8aac-1c79-4df3-b53f-413706517a91-kube-api-access-67r4g" pod "d46a8aac-1c79-4df3-b53f-413706517a91" (UID: "d46a8aac-1c79-4df3-b53f-413706517a91"). InnerVolumeSpecName "kube-api-access-67r4g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:58:33.918442 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:58:33.918422 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d46a8aac-1c79-4df3-b53f-413706517a91-maas-api-tls" pod "d46a8aac-1c79-4df3-b53f-413706517a91" (UID: "d46a8aac-1c79-4df3-b53f-413706517a91"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:58:34.016930 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:58:34.016894 4421 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d46a8aac-1c79-4df3-b53f-413706517a91-openshift-service-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:58:34.016930 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:58:34.016923 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67r4g\" (UniqueName: \"kubernetes.io/projected/d46a8aac-1c79-4df3-b53f-413706517a91-kube-api-access-67r4g\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:58:34.017285 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:58:34.016938 4421 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d46a8aac-1c79-4df3-b53f-413706517a91-metrics-certs\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:58:34.017285 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:58:34.016952 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d46a8aac-1c79-4df3-b53f-413706517a91-maas-api-tls\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:58:34.126372 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:58:34.126261 4421 generic.go:356] "Generic (PLEG): container finished" podID="d46a8aac-1c79-4df3-b53f-413706517a91" containerID="58161848ebff66f2d4097ce3e1129beb030ee154ef77ca77d7c47106e0613a6e" exitCode=0 Nov 03 01:58:34.126372 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:58:34.126326 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7fde87-7bf9f9f857-45v2f" event={"ID":"d46a8aac-1c79-4df3-b53f-413706517a91","Type":"ContainerDied","Data":"58161848ebff66f2d4097ce3e1129beb030ee154ef77ca77d7c47106e0613a6e"} Nov 03 01:58:34.126372 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:58:34.126346 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7fde87-7bf9f9f857-45v2f" Nov 03 01:58:34.126627 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:58:34.126395 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7fde87-7bf9f9f857-45v2f" event={"ID":"d46a8aac-1c79-4df3-b53f-413706517a91","Type":"ContainerDied","Data":"59467f4a3d9e583ee982f1ce4eceed8be5858db6bd56990bb291399d4761eec4"} Nov 03 01:58:34.126627 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:58:34.126425 4421 scope.go:122] "RemoveContainer" containerID="58161848ebff66f2d4097ce3e1129beb030ee154ef77ca77d7c47106e0613a6e" Nov 03 01:58:34.136934 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:58:34.136913 4421 scope.go:122] "RemoveContainer" containerID="58161848ebff66f2d4097ce3e1129beb030ee154ef77ca77d7c47106e0613a6e" Nov 03 01:58:34.137227 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:58:34.137208 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58161848ebff66f2d4097ce3e1129beb030ee154ef77ca77d7c47106e0613a6e\": container with ID starting with 58161848ebff66f2d4097ce3e1129beb030ee154ef77ca77d7c47106e0613a6e not found: ID does not exist" containerID="58161848ebff66f2d4097ce3e1129beb030ee154ef77ca77d7c47106e0613a6e" Nov 03 01:58:34.137326 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:58:34.137242 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58161848ebff66f2d4097ce3e1129beb030ee154ef77ca77d7c47106e0613a6e"} err="failed to get container status \"58161848ebff66f2d4097ce3e1129beb030ee154ef77ca77d7c47106e0613a6e\": rpc error: code = NotFound desc = could not find container \"58161848ebff66f2d4097ce3e1129beb030ee154ef77ca77d7c47106e0613a6e\": container with ID starting with 58161848ebff66f2d4097ce3e1129beb030ee154ef77ca77d7c47106e0613a6e not found: ID does not exist" Nov 03 01:58:34.143755 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:58:34.143728 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7fde87-7bf9f9f857-45v2f"] Nov 03 01:58:34.147508 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:58:34.147474 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7fde87-7bf9f9f857-45v2f"] Nov 03 01:58:35.977563 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:58:35.977525 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d46a8aac-1c79-4df3-b53f-413706517a91" path="/var/lib/kubelet/pods/d46a8aac-1c79-4df3-b53f-413706517a91/volumes" Nov 03 01:58:37.347929 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:58:37.347886 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-7fde87-6b96867c8jxctp" podUID="bfaa1793-a630-4d73-b226-6943e7db3357" containerName="payload-processing" containerID="cri-o://a50b05dae259bb4264ab32b07eb5c5b82bc3f5332badbcdcba3498fca530c430" gracePeriod=25 Nov 03 01:58:40.028047 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:58:40.028010 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-kws96"] Nov 03 01:58:40.031543 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:58:40.031513 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-kws96"] Nov 03 01:58:41.977494 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:58:41.977460 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a177155-0959-45a2-a9c4-e5802e34891f" path="/var/lib/kubelet/pods/3a177155-0959-45a2-a9c4-e5802e34891f/volumes" Nov 03 01:58:42.791334 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:58:42.791295 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-7fde87-6b96867c8jxctp" podUID="bfaa1793-a630-4d73-b226-6943e7db3357" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:58:44.668869 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:58:44.668827 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 01:58:46.783501 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:58:46.783460 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-7fde87/simulator-single-main-7fde87-kserve-5db8b76fdf-x74kx" podUID="2f2bc993-f2ac-48bd-b2d1-e00be41d9fbf" containerName="main" containerID="cri-o://9c6069785c718ba015fefea527fcc31ce7e55dea5a51a3769539e4e3f710800c" gracePeriod=45 Nov 03 01:58:46.920264 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:58:46.920219 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-single-main-7fde87/simulator-single-main-7fde87-kserve-5db8b76fdf-x74kx" podUID="2f2bc993-f2ac-48bd-b2d1-e00be41d9fbf" containerName="main" probeResult="failure" output="Get \"https://10.134.1.164:8000/ready\": dial tcp 10.134.1.164:8000: connect: connection refused" Nov 03 01:58:47.035176 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:58:47.035114 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-7fde87/simulator-single-main-7fde87-kserve-5db8b76fdf-x74kx" Nov 03 01:58:47.132804 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:58:47.132771 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f2bc993-f2ac-48bd-b2d1-e00be41d9fbf-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f2bc993-f2ac-48bd-b2d1-e00be41d9fbf-tmp-dir\") pod \"2f2bc993-f2ac-48bd-b2d1-e00be41d9fbf\" (UID: \"2f2bc993-f2ac-48bd-b2d1-e00be41d9fbf\") " Nov 03 01:58:47.133017 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:58:47.132859 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f2bc993-f2ac-48bd-b2d1-e00be41d9fbf-home\" (UniqueName: \"kubernetes.io/empty-dir/2f2bc993-f2ac-48bd-b2d1-e00be41d9fbf-home\") pod \"2f2bc993-f2ac-48bd-b2d1-e00be41d9fbf\" (UID: \"2f2bc993-f2ac-48bd-b2d1-e00be41d9fbf\") " Nov 03 01:58:47.133017 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:58:47.132905 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f2bc993-f2ac-48bd-b2d1-e00be41d9fbf-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f2bc993-f2ac-48bd-b2d1-e00be41d9fbf-model-cache\") pod \"2f2bc993-f2ac-48bd-b2d1-e00be41d9fbf\" (UID: \"2f2bc993-f2ac-48bd-b2d1-e00be41d9fbf\") " Nov 03 01:58:47.133017 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:58:47.132938 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f2bc993-f2ac-48bd-b2d1-e00be41d9fbf-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f2bc993-f2ac-48bd-b2d1-e00be41d9fbf-dshm\") pod \"2f2bc993-f2ac-48bd-b2d1-e00be41d9fbf\" (UID: \"2f2bc993-f2ac-48bd-b2d1-e00be41d9fbf\") " Nov 03 01:58:47.133017 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:58:47.133000 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f2bc993-f2ac-48bd-b2d1-e00be41d9fbf-kube-api-access-cngtb\" (UniqueName: \"kubernetes.io/projected/2f2bc993-f2ac-48bd-b2d1-e00be41d9fbf-kube-api-access-cngtb\") pod \"2f2bc993-f2ac-48bd-b2d1-e00be41d9fbf\" (UID: \"2f2bc993-f2ac-48bd-b2d1-e00be41d9fbf\") " Nov 03 01:58:47.133250 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:58:47.133043 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f2bc993-f2ac-48bd-b2d1-e00be41d9fbf-tls-certs\" (UniqueName: \"kubernetes.io/secret/2f2bc993-f2ac-48bd-b2d1-e00be41d9fbf-tls-certs\") pod \"2f2bc993-f2ac-48bd-b2d1-e00be41d9fbf\" (UID: \"2f2bc993-f2ac-48bd-b2d1-e00be41d9fbf\") " Nov 03 01:58:47.133250 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:58:47.133045 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f2bc993-f2ac-48bd-b2d1-e00be41d9fbf-tmp-dir" pod "2f2bc993-f2ac-48bd-b2d1-e00be41d9fbf" (UID: "2f2bc993-f2ac-48bd-b2d1-e00be41d9fbf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:58:47.133250 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:58:47.133122 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f2bc993-f2ac-48bd-b2d1-e00be41d9fbf-home" pod "2f2bc993-f2ac-48bd-b2d1-e00be41d9fbf" (UID: "2f2bc993-f2ac-48bd-b2d1-e00be41d9fbf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:58:47.133250 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:58:47.133221 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f2bc993-f2ac-48bd-b2d1-e00be41d9fbf-model-cache" pod "2f2bc993-f2ac-48bd-b2d1-e00be41d9fbf" (UID: "2f2bc993-f2ac-48bd-b2d1-e00be41d9fbf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:58:47.133486 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:58:47.133414 4421 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f2bc993-f2ac-48bd-b2d1-e00be41d9fbf-tmp-dir\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:58:47.133486 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:58:47.133432 4421 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f2bc993-f2ac-48bd-b2d1-e00be41d9fbf-home\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:58:47.133486 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:58:47.133445 4421 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f2bc993-f2ac-48bd-b2d1-e00be41d9fbf-model-cache\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:58:47.135505 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:58:47.135474 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f2bc993-f2ac-48bd-b2d1-e00be41d9fbf-tls-certs" pod "2f2bc993-f2ac-48bd-b2d1-e00be41d9fbf" (UID: "2f2bc993-f2ac-48bd-b2d1-e00be41d9fbf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:58:47.135660 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:58:47.135623 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f2bc993-f2ac-48bd-b2d1-e00be41d9fbf-dshm" pod "2f2bc993-f2ac-48bd-b2d1-e00be41d9fbf" (UID: "2f2bc993-f2ac-48bd-b2d1-e00be41d9fbf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:58:47.136411 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:58:47.136389 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f2bc993-f2ac-48bd-b2d1-e00be41d9fbf-kube-api-access-cngtb" pod "2f2bc993-f2ac-48bd-b2d1-e00be41d9fbf" (UID: "2f2bc993-f2ac-48bd-b2d1-e00be41d9fbf"). InnerVolumeSpecName "kube-api-access-cngtb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:58:47.196333 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:58:47.196298 4421 generic.go:356] "Generic (PLEG): container finished" podID="2f2bc993-f2ac-48bd-b2d1-e00be41d9fbf" containerID="9c6069785c718ba015fefea527fcc31ce7e55dea5a51a3769539e4e3f710800c" exitCode=0 Nov 03 01:58:47.196539 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:58:47.196348 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-7fde87/simulator-single-main-7fde87-kserve-5db8b76fdf-x74kx" Nov 03 01:58:47.196539 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:58:47.196420 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-7fde87/simulator-single-main-7fde87-kserve-5db8b76fdf-x74kx" event={"ID":"2f2bc993-f2ac-48bd-b2d1-e00be41d9fbf","Type":"ContainerDied","Data":"9c6069785c718ba015fefea527fcc31ce7e55dea5a51a3769539e4e3f710800c"} Nov 03 01:58:47.196539 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:58:47.196458 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-7fde87/simulator-single-main-7fde87-kserve-5db8b76fdf-x74kx" event={"ID":"2f2bc993-f2ac-48bd-b2d1-e00be41d9fbf","Type":"ContainerDied","Data":"88c16d94780d3ce210c81d873d2f13657a873ba355f198c17b1dce35d2c95672"} Nov 03 01:58:47.196539 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:58:47.196480 4421 scope.go:122] "RemoveContainer" containerID="9c6069785c718ba015fefea527fcc31ce7e55dea5a51a3769539e4e3f710800c" Nov 03 01:58:47.208375 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:58:47.208354 4421 scope.go:122] "RemoveContainer" containerID="9c6069785c718ba015fefea527fcc31ce7e55dea5a51a3769539e4e3f710800c" Nov 03 01:58:47.208738 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:58:47.208710 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c6069785c718ba015fefea527fcc31ce7e55dea5a51a3769539e4e3f710800c\": container with ID starting with 9c6069785c718ba015fefea527fcc31ce7e55dea5a51a3769539e4e3f710800c not found: ID does not exist" containerID="9c6069785c718ba015fefea527fcc31ce7e55dea5a51a3769539e4e3f710800c" Nov 03 01:58:47.208856 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:58:47.208750 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c6069785c718ba015fefea527fcc31ce7e55dea5a51a3769539e4e3f710800c"} err="failed to get container status \"9c6069785c718ba015fefea527fcc31ce7e55dea5a51a3769539e4e3f710800c\": rpc error: code = NotFound desc = could not find container \"9c6069785c718ba015fefea527fcc31ce7e55dea5a51a3769539e4e3f710800c\": container with ID starting with 9c6069785c718ba015fefea527fcc31ce7e55dea5a51a3769539e4e3f710800c not found: ID does not exist" Nov 03 01:58:47.216414 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:58:47.216383 4421 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2f2bc993_f2ac_48bd_b2d1_e00be41d9fbf.slice\": RecentStats: unable to find data in memory cache]" Nov 03 01:58:47.227056 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:58:47.227024 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-7fde87/simulator-single-main-7fde87-kserve-5db8b76fdf-x74kx"] Nov 03 01:58:47.230313 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:58:47.230267 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-7fde87/simulator-single-main-7fde87-kserve-5db8b76fdf-x74kx"] Nov 03 01:58:47.234411 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:58:47.234394 4421 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f2bc993-f2ac-48bd-b2d1-e00be41d9fbf-dshm\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:58:47.234468 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:58:47.234414 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cngtb\" (UniqueName: \"kubernetes.io/projected/2f2bc993-f2ac-48bd-b2d1-e00be41d9fbf-kube-api-access-cngtb\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:58:47.234468 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:58:47.234425 4421 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f2bc993-f2ac-48bd-b2d1-e00be41d9fbf-tls-certs\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:58:47.979515 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:58:47.979479 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f2bc993-f2ac-48bd-b2d1-e00be41d9fbf" path="/var/lib/kubelet/pods/2f2bc993-f2ac-48bd-b2d1-e00be41d9fbf/volumes" Nov 03 01:58:52.791818 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:58:52.791780 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-7fde87-6b96867c8jxctp" podUID="bfaa1793-a630-4d73-b226-6943e7db3357" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:58:55.022566 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:58:55.022517 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-nrpfp"] Nov 03 01:58:55.024740 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:58:55.024712 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-nrpfp"] Nov 03 01:58:55.977376 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:58:55.977339 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4260979-c90d-4671-9c5b-1bc1e9481b30" path="/var/lib/kubelet/pods/f4260979-c90d-4671-9c5b-1bc1e9481b30/volumes" Nov 03 01:58:57.604225 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:58:57.604152 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-7fde87-6b96867c8jxctp" Nov 03 01:58:57.634362 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:58:57.634334 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bfaa1793-a630-4d73-b226-6943e7db3357-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfaa1793-a630-4d73-b226-6943e7db3357-plugins-config-volume\") pod \"bfaa1793-a630-4d73-b226-6943e7db3357\" (UID: \"bfaa1793-a630-4d73-b226-6943e7db3357\") " Nov 03 01:58:57.634527 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:58:57.634405 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfaa1793-a630-4d73-b226-6943e7db3357-kube-api-access-9w6sn\" (UniqueName: \"kubernetes.io/projected/bfaa1793-a630-4d73-b226-6943e7db3357-kube-api-access-9w6sn\") pod \"bfaa1793-a630-4d73-b226-6943e7db3357\" (UID: \"bfaa1793-a630-4d73-b226-6943e7db3357\") " Nov 03 01:58:57.634714 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:58:57.634685 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfaa1793-a630-4d73-b226-6943e7db3357-plugins-config-volume" pod "bfaa1793-a630-4d73-b226-6943e7db3357" (UID: "bfaa1793-a630-4d73-b226-6943e7db3357"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:58:57.634875 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:58:57.634787 4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfaa1793-a630-4d73-b226-6943e7db3357-plugins-config-volume\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:58:57.636521 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:58:57.636499 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfaa1793-a630-4d73-b226-6943e7db3357-kube-api-access-9w6sn" pod "bfaa1793-a630-4d73-b226-6943e7db3357" (UID: "bfaa1793-a630-4d73-b226-6943e7db3357"). InnerVolumeSpecName "kube-api-access-9w6sn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:58:57.736196 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:58:57.736152 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9w6sn\" (UniqueName: \"kubernetes.io/projected/bfaa1793-a630-4d73-b226-6943e7db3357-kube-api-access-9w6sn\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:58:58.257675 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:58:58.257639 4421 generic.go:356] "Generic (PLEG): container finished" podID="bfaa1793-a630-4d73-b226-6943e7db3357" containerID="a50b05dae259bb4264ab32b07eb5c5b82bc3f5332badbcdcba3498fca530c430" exitCode=0 Nov 03 01:58:58.257881 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:58:58.257687 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-7fde87-6b96867c8jxctp" event={"ID":"bfaa1793-a630-4d73-b226-6943e7db3357","Type":"ContainerDied","Data":"a50b05dae259bb4264ab32b07eb5c5b82bc3f5332badbcdcba3498fca530c430"} Nov 03 01:58:58.257881 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:58:58.257702 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-7fde87-6b96867c8jxctp" Nov 03 01:58:58.257881 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:58:58.257767 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-7fde87-6b96867c8jxctp" event={"ID":"bfaa1793-a630-4d73-b226-6943e7db3357","Type":"ContainerDied","Data":"b651f6c9820abfb88eedf42fd6eedb8be34da4ef7299b997f5869077496cdfe2"} Nov 03 01:58:58.257881 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:58:58.257796 4421 scope.go:122] "RemoveContainer" containerID="a50b05dae259bb4264ab32b07eb5c5b82bc3f5332badbcdcba3498fca530c430" Nov 03 01:58:58.267083 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:58:58.267065 4421 scope.go:122] "RemoveContainer" containerID="a50b05dae259bb4264ab32b07eb5c5b82bc3f5332badbcdcba3498fca530c430" Nov 03 01:58:58.267388 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:58:58.267368 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a50b05dae259bb4264ab32b07eb5c5b82bc3f5332badbcdcba3498fca530c430\": container with ID starting with a50b05dae259bb4264ab32b07eb5c5b82bc3f5332badbcdcba3498fca530c430 not found: ID does not exist" containerID="a50b05dae259bb4264ab32b07eb5c5b82bc3f5332badbcdcba3498fca530c430" Nov 03 01:58:58.267460 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:58:58.267399 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a50b05dae259bb4264ab32b07eb5c5b82bc3f5332badbcdcba3498fca530c430"} err="failed to get container status \"a50b05dae259bb4264ab32b07eb5c5b82bc3f5332badbcdcba3498fca530c430\": rpc error: code = NotFound desc = could not find container \"a50b05dae259bb4264ab32b07eb5c5b82bc3f5332badbcdcba3498fca530c430\": container with ID starting with a50b05dae259bb4264ab32b07eb5c5b82bc3f5332badbcdcba3498fca530c430 not found: ID does not exist" Nov 03 01:58:58.275225 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:58:58.275202 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-7fde87-6b96867c8jxctp"] Nov 03 01:58:58.280702 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:58:58.280681 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-7fde87-6b96867c8jxctp"] Nov 03 01:58:59.976893 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:58:59.976862 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bfaa1793-a630-4d73-b226-6943e7db3357" path="/var/lib/kubelet/pods/bfaa1793-a630-4d73-b226-6943e7db3357/volumes" Nov 03 01:59:04.970245 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:59:04.970200 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-7fde87-6b96867c8lcmpd"] Nov 03 01:59:04.970806 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:59:04.970786 4421 cpu_manager.go:414] "containerMap: removing container" podUID="bfaa1793-a630-4d73-b226-6943e7db3357" containerName="payload-processing" Nov 03 01:59:04.970876 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:59:04.970809 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfaa1793-a630-4d73-b226-6943e7db3357" containerName="payload-processing" Nov 03 01:59:04.970876 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:59:04.970844 4421 cpu_manager.go:414] "containerMap: removing container" podUID="2f2bc993-f2ac-48bd-b2d1-e00be41d9fbf" containerName="main" Nov 03 01:59:04.970876 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:59:04.970853 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f2bc993-f2ac-48bd-b2d1-e00be41d9fbf" containerName="main" Nov 03 01:59:04.971061 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:59:04.970879 4421 cpu_manager.go:414] "containerMap: removing container" podUID="d46a8aac-1c79-4df3-b53f-413706517a91" containerName="maas-api" Nov 03 01:59:04.971061 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:59:04.970889 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d46a8aac-1c79-4df3-b53f-413706517a91" containerName="maas-api" Nov 03 01:59:04.971061 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:59:04.970996 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="d46a8aac-1c79-4df3-b53f-413706517a91" containerName="maas-api" Nov 03 01:59:04.971061 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:59:04.971011 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfaa1793-a630-4d73-b226-6943e7db3357" containerName="payload-processing" Nov 03 01:59:04.971061 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:59:04.971020 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f2bc993-f2ac-48bd-b2d1-e00be41d9fbf" containerName="main" Nov 03 01:59:04.975884 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:59:04.975858 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-7fde87-6b96867c8lcmpd" Nov 03 01:59:04.986107 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:59:04.986079 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-7fde87\"" Nov 03 01:59:04.988538 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:59:04.988498 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7fde87-7bf9f9f857-tcrkj"] Nov 03 01:59:04.992851 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:59:04.992825 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-7fde87-6b96867c8lcmpd"] Nov 03 01:59:04.992968 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:59:04.992950 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7fde87-7bf9f9f857-tcrkj" Nov 03 01:59:05.005218 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:59:05.005165 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-nfcxl\"" Nov 03 01:59:05.005450 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:59:05.005430 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 03 01:59:05.005562 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:59:05.005474 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-7fde87-serving-cert\"" Nov 03 01:59:05.014553 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:59:05.014493 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7fde87-7bf9f9f857-tcrkj"] Nov 03 01:59:05.084672 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:59:05.084638 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 01:59:05.084672 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:59:05.084642 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 01:59:05.116699 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:59:05.116656 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f751dabb-190f-4f7a-8fd3-ed5a1b268f3d-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-7fde87-6b96867c8lcmpd\" (UID: \"f751dabb-190f-4f7a-8fd3-ed5a1b268f3d\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-7fde87-6b96867c8lcmpd" Nov 03 01:59:05.116960 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:59:05.116737 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/22400262-f9f4-42aa-9f98-cebb1b519b91-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-7fde87-7bf9f9f857-tcrkj\" (UID: \"22400262-f9f4-42aa-9f98-cebb1b519b91\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7fde87-7bf9f9f857-tcrkj" Nov 03 01:59:05.116960 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:59:05.116874 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/22400262-f9f4-42aa-9f98-cebb1b519b91-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-7fde87-7bf9f9f857-tcrkj\" (UID: \"22400262-f9f4-42aa-9f98-cebb1b519b91\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7fde87-7bf9f9f857-tcrkj" Nov 03 01:59:05.116960 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:59:05.116911 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/22400262-f9f4-42aa-9f98-cebb1b519b91-metrics-certs\") pod \"maas-api-e2e-worker-single-main-7fde87-7bf9f9f857-tcrkj\" (UID: \"22400262-f9f4-42aa-9f98-cebb1b519b91\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7fde87-7bf9f9f857-tcrkj" Nov 03 01:59:05.117150 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:59:05.116981 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jqxh\" (UniqueName: \"kubernetes.io/projected/f751dabb-190f-4f7a-8fd3-ed5a1b268f3d-kube-api-access-2jqxh\") pod \"payload-processing-e2e-worker-single-main-7fde87-6b96867c8lcmpd\" (UID: \"f751dabb-190f-4f7a-8fd3-ed5a1b268f3d\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-7fde87-6b96867c8lcmpd" Nov 03 01:59:05.117150 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:59:05.117028 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nc97n\" (UniqueName: \"kubernetes.io/projected/22400262-f9f4-42aa-9f98-cebb1b519b91-kube-api-access-nc97n\") pod \"maas-api-e2e-worker-single-main-7fde87-7bf9f9f857-tcrkj\" (UID: \"22400262-f9f4-42aa-9f98-cebb1b519b91\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7fde87-7bf9f9f857-tcrkj" Nov 03 01:59:05.217886 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:59:05.217839 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jqxh\" (UniqueName: \"kubernetes.io/projected/f751dabb-190f-4f7a-8fd3-ed5a1b268f3d-kube-api-access-2jqxh\") pod \"payload-processing-e2e-worker-single-main-7fde87-6b96867c8lcmpd\" (UID: \"f751dabb-190f-4f7a-8fd3-ed5a1b268f3d\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-7fde87-6b96867c8lcmpd" Nov 03 01:59:05.217886 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:59:05.217894 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nc97n\" (UniqueName: \"kubernetes.io/projected/22400262-f9f4-42aa-9f98-cebb1b519b91-kube-api-access-nc97n\") pod \"maas-api-e2e-worker-single-main-7fde87-7bf9f9f857-tcrkj\" (UID: \"22400262-f9f4-42aa-9f98-cebb1b519b91\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7fde87-7bf9f9f857-tcrkj" Nov 03 01:59:05.218174 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:59:05.217939 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f751dabb-190f-4f7a-8fd3-ed5a1b268f3d-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-7fde87-6b96867c8lcmpd\" (UID: \"f751dabb-190f-4f7a-8fd3-ed5a1b268f3d\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-7fde87-6b96867c8lcmpd" Nov 03 01:59:05.218174 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:59:05.217976 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/22400262-f9f4-42aa-9f98-cebb1b519b91-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-7fde87-7bf9f9f857-tcrkj\" (UID: \"22400262-f9f4-42aa-9f98-cebb1b519b91\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7fde87-7bf9f9f857-tcrkj" Nov 03 01:59:05.218174 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:59:05.218060 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/22400262-f9f4-42aa-9f98-cebb1b519b91-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-7fde87-7bf9f9f857-tcrkj\" (UID: \"22400262-f9f4-42aa-9f98-cebb1b519b91\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7fde87-7bf9f9f857-tcrkj" Nov 03 01:59:05.218415 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:59:05.218192 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/22400262-f9f4-42aa-9f98-cebb1b519b91-metrics-certs\") pod \"maas-api-e2e-worker-single-main-7fde87-7bf9f9f857-tcrkj\" (UID: \"22400262-f9f4-42aa-9f98-cebb1b519b91\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7fde87-7bf9f9f857-tcrkj" Nov 03 01:59:05.218841 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:59:05.218811 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/22400262-f9f4-42aa-9f98-cebb1b519b91-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-7fde87-7bf9f9f857-tcrkj\" (UID: \"22400262-f9f4-42aa-9f98-cebb1b519b91\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7fde87-7bf9f9f857-tcrkj" Nov 03 01:59:05.218968 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:59:05.218857 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f751dabb-190f-4f7a-8fd3-ed5a1b268f3d-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-7fde87-6b96867c8lcmpd\" (UID: \"f751dabb-190f-4f7a-8fd3-ed5a1b268f3d\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-7fde87-6b96867c8lcmpd" Nov 03 01:59:05.220666 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:59:05.220621 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/22400262-f9f4-42aa-9f98-cebb1b519b91-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-7fde87-7bf9f9f857-tcrkj\" (UID: \"22400262-f9f4-42aa-9f98-cebb1b519b91\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7fde87-7bf9f9f857-tcrkj" Nov 03 01:59:05.220794 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:59:05.220774 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/22400262-f9f4-42aa-9f98-cebb1b519b91-metrics-certs\") pod \"maas-api-e2e-worker-single-main-7fde87-7bf9f9f857-tcrkj\" (UID: \"22400262-f9f4-42aa-9f98-cebb1b519b91\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7fde87-7bf9f9f857-tcrkj" Nov 03 01:59:05.226218 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:59:05.226188 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nc97n\" (UniqueName: \"kubernetes.io/projected/22400262-f9f4-42aa-9f98-cebb1b519b91-kube-api-access-nc97n\") pod \"maas-api-e2e-worker-single-main-7fde87-7bf9f9f857-tcrkj\" (UID: \"22400262-f9f4-42aa-9f98-cebb1b519b91\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7fde87-7bf9f9f857-tcrkj" Nov 03 01:59:05.226569 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:59:05.226535 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jqxh\" (UniqueName: \"kubernetes.io/projected/f751dabb-190f-4f7a-8fd3-ed5a1b268f3d-kube-api-access-2jqxh\") pod \"payload-processing-e2e-worker-single-main-7fde87-6b96867c8lcmpd\" (UID: \"f751dabb-190f-4f7a-8fd3-ed5a1b268f3d\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-7fde87-6b96867c8lcmpd" Nov 03 01:59:05.293897 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:59:05.293862 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-7fde87-6b96867c8lcmpd" Nov 03 01:59:05.318146 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:59:05.318115 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7fde87-7bf9f9f857-tcrkj" Nov 03 01:59:05.581953 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:59:05.581803 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-7fde87-6b96867c8lcmpd"] Nov 03 01:59:05.585577 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:59:05.585540 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf751dabb_190f_4f7a_8fd3_ed5a1b268f3d.slice/crio-195b9e3b500d542cb0e1d8cd0a1c6567d962eacffdb90dc3ea4fd47223902410 WatchSource:0}: Error finding container 195b9e3b500d542cb0e1d8cd0a1c6567d962eacffdb90dc3ea4fd47223902410: Status 404 returned error can't find the container with id 195b9e3b500d542cb0e1d8cd0a1c6567d962eacffdb90dc3ea4fd47223902410 Nov 03 01:59:05.604597 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:59:05.604572 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7fde87-7bf9f9f857-tcrkj"] Nov 03 01:59:05.606932 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:59:05.606903 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22400262_f9f4_42aa_9f98_cebb1b519b91.slice/crio-55415de893b17e271deb51017a86ecb7c89f81a1e28d9a27a085a4b452961cf8 WatchSource:0}: Error finding container 55415de893b17e271deb51017a86ecb7c89f81a1e28d9a27a085a4b452961cf8: Status 404 returned error can't find the container with id 55415de893b17e271deb51017a86ecb7c89f81a1e28d9a27a085a4b452961cf8 Nov 03 01:59:06.022294 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:59:06.022247 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-txdgp"] Nov 03 01:59:06.026990 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:59:06.026958 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-txdgp"] Nov 03 01:59:06.302219 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:59:06.302104 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7fde87-7bf9f9f857-tcrkj" event={"ID":"22400262-f9f4-42aa-9f98-cebb1b519b91","Type":"ContainerStarted","Data":"82bf1ab3009355b5f8a4aa89075eb0ecdcad17fb63ea4804f6a2816f36e9ea27"} Nov 03 01:59:06.302219 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:59:06.302176 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7fde87-7bf9f9f857-tcrkj" event={"ID":"22400262-f9f4-42aa-9f98-cebb1b519b91","Type":"ContainerStarted","Data":"55415de893b17e271deb51017a86ecb7c89f81a1e28d9a27a085a4b452961cf8"} Nov 03 01:59:06.302219 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:59:06.302205 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7fde87-7bf9f9f857-tcrkj" Nov 03 01:59:06.303787 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:59:06.303754 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-7fde87-6b96867c8lcmpd" event={"ID":"f751dabb-190f-4f7a-8fd3-ed5a1b268f3d","Type":"ContainerStarted","Data":"71a8d7ffe0ff9778c83d31e2549985deadfc306f55ef1124b0a5ba9443b6402c"} Nov 03 01:59:06.303919 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:59:06.303811 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-7fde87-6b96867c8lcmpd" event={"ID":"f751dabb-190f-4f7a-8fd3-ed5a1b268f3d","Type":"ContainerStarted","Data":"195b9e3b500d542cb0e1d8cd0a1c6567d962eacffdb90dc3ea4fd47223902410"} Nov 03 01:59:06.303919 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:59:06.303842 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-7fde87-6b96867c8lcmpd" Nov 03 01:59:06.319672 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:59:06.319615 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7fde87-7bf9f9f857-tcrkj" podStartSLOduration=1.795476018 podStartE2EDuration="2.319594905s" totalImagesPullingTime="524.118887ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:59:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:59:06.318064131 +0000 UTC m=+4688.489067238" watchObservedRunningTime="2026-11-03 01:59:06.319594905 +0000 UTC m=+4688.490598001" Nov 03 01:59:06.335915 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:59:06.335860 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-7fde87-6b96867c8lcmpd" podStartSLOduration=2.335845259 podStartE2EDuration="2.335845259s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:59:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:59:06.333448354 +0000 UTC m=+4688.504451454" watchObservedRunningTime="2026-11-03 01:59:06.335845259 +0000 UTC m=+4688.506848355" Nov 03 01:59:07.977881 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:59:07.977844 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1749c24-bedc-4256-b826-21a75c3604f4" path="/var/lib/kubelet/pods/e1749c24-bedc-4256-b826-21a75c3604f4/volumes" Nov 03 01:59:12.317355 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:59:12.317325 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7fde87-7bf9f9f857-tcrkj" Nov 03 01:59:14.669405 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:59:14.669363 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 01:59:17.315068 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:59:17.315031 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-7fde87-6b96867c8lcmpd" Nov 03 01:59:22.772619 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:59:22.772581 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-7fde87-6b96867c8lcmpd"] Nov 03 01:59:23.844475 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:59:23.844437 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7fde87-7bf9f9f857-tcrkj"] Nov 03 01:59:23.845024 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:59:23.844721 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7fde87-7bf9f9f857-tcrkj" podUID="22400262-f9f4-42aa-9f98-cebb1b519b91" containerName="maas-api" containerID="cri-o://82bf1ab3009355b5f8a4aa89075eb0ecdcad17fb63ea4804f6a2816f36e9ea27" gracePeriod=30 Nov 03 01:59:24.111905 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:59:24.111877 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7fde87-7bf9f9f857-tcrkj" Nov 03 01:59:24.310841 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:59:24.310802 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22400262-f9f4-42aa-9f98-cebb1b519b91-kube-api-access-nc97n\" (UniqueName: \"kubernetes.io/projected/22400262-f9f4-42aa-9f98-cebb1b519b91-kube-api-access-nc97n\") pod \"22400262-f9f4-42aa-9f98-cebb1b519b91\" (UID: \"22400262-f9f4-42aa-9f98-cebb1b519b91\") " Nov 03 01:59:24.311023 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:59:24.310857 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22400262-f9f4-42aa-9f98-cebb1b519b91-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/22400262-f9f4-42aa-9f98-cebb1b519b91-openshift-service-ca\") pod \"22400262-f9f4-42aa-9f98-cebb1b519b91\" (UID: \"22400262-f9f4-42aa-9f98-cebb1b519b91\") " Nov 03 01:59:24.311023 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:59:24.310892 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/22400262-f9f4-42aa-9f98-cebb1b519b91-metrics-certs\" (UniqueName: \"kubernetes.io/secret/22400262-f9f4-42aa-9f98-cebb1b519b91-metrics-certs\") pod \"22400262-f9f4-42aa-9f98-cebb1b519b91\" (UID: \"22400262-f9f4-42aa-9f98-cebb1b519b91\") " Nov 03 01:59:24.311023 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:59:24.310954 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/22400262-f9f4-42aa-9f98-cebb1b519b91-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/22400262-f9f4-42aa-9f98-cebb1b519b91-maas-api-tls\") pod \"22400262-f9f4-42aa-9f98-cebb1b519b91\" (UID: \"22400262-f9f4-42aa-9f98-cebb1b519b91\") " Nov 03 01:59:24.311666 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:59:24.311300 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22400262-f9f4-42aa-9f98-cebb1b519b91-openshift-service-ca" pod "22400262-f9f4-42aa-9f98-cebb1b519b91" (UID: "22400262-f9f4-42aa-9f98-cebb1b519b91"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:59:24.311666 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:59:24.311494 4421 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/22400262-f9f4-42aa-9f98-cebb1b519b91-openshift-service-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:59:24.313149 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:59:24.313120 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22400262-f9f4-42aa-9f98-cebb1b519b91-metrics-certs" pod "22400262-f9f4-42aa-9f98-cebb1b519b91" (UID: "22400262-f9f4-42aa-9f98-cebb1b519b91"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:59:24.313319 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:59:24.313191 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22400262-f9f4-42aa-9f98-cebb1b519b91-maas-api-tls" pod "22400262-f9f4-42aa-9f98-cebb1b519b91" (UID: "22400262-f9f4-42aa-9f98-cebb1b519b91"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:59:24.313384 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:59:24.313319 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22400262-f9f4-42aa-9f98-cebb1b519b91-kube-api-access-nc97n" pod "22400262-f9f4-42aa-9f98-cebb1b519b91" (UID: "22400262-f9f4-42aa-9f98-cebb1b519b91"). InnerVolumeSpecName "kube-api-access-nc97n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:59:24.412444 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:59:24.412348 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nc97n\" (UniqueName: \"kubernetes.io/projected/22400262-f9f4-42aa-9f98-cebb1b519b91-kube-api-access-nc97n\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:59:24.412444 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:59:24.412377 4421 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/22400262-f9f4-42aa-9f98-cebb1b519b91-metrics-certs\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:59:24.412444 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:59:24.412387 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/22400262-f9f4-42aa-9f98-cebb1b519b91-maas-api-tls\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:59:24.421217 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:59:24.421181 4421 generic.go:356] "Generic (PLEG): container finished" podID="22400262-f9f4-42aa-9f98-cebb1b519b91" containerID="82bf1ab3009355b5f8a4aa89075eb0ecdcad17fb63ea4804f6a2816f36e9ea27" exitCode=0 Nov 03 01:59:24.421424 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:59:24.421237 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7fde87-7bf9f9f857-tcrkj" Nov 03 01:59:24.421424 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:59:24.421268 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7fde87-7bf9f9f857-tcrkj" event={"ID":"22400262-f9f4-42aa-9f98-cebb1b519b91","Type":"ContainerDied","Data":"82bf1ab3009355b5f8a4aa89075eb0ecdcad17fb63ea4804f6a2816f36e9ea27"} Nov 03 01:59:24.421424 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:59:24.421353 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7fde87-7bf9f9f857-tcrkj" event={"ID":"22400262-f9f4-42aa-9f98-cebb1b519b91","Type":"ContainerDied","Data":"55415de893b17e271deb51017a86ecb7c89f81a1e28d9a27a085a4b452961cf8"} Nov 03 01:59:24.421424 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:59:24.421376 4421 scope.go:122] "RemoveContainer" containerID="82bf1ab3009355b5f8a4aa89075eb0ecdcad17fb63ea4804f6a2816f36e9ea27" Nov 03 01:59:24.431245 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:59:24.431226 4421 scope.go:122] "RemoveContainer" containerID="82bf1ab3009355b5f8a4aa89075eb0ecdcad17fb63ea4804f6a2816f36e9ea27" Nov 03 01:59:24.431565 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:59:24.431542 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82bf1ab3009355b5f8a4aa89075eb0ecdcad17fb63ea4804f6a2816f36e9ea27\": container with ID starting with 82bf1ab3009355b5f8a4aa89075eb0ecdcad17fb63ea4804f6a2816f36e9ea27 not found: ID does not exist" containerID="82bf1ab3009355b5f8a4aa89075eb0ecdcad17fb63ea4804f6a2816f36e9ea27" Nov 03 01:59:24.431615 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:59:24.431574 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82bf1ab3009355b5f8a4aa89075eb0ecdcad17fb63ea4804f6a2816f36e9ea27"} err="failed to get container status \"82bf1ab3009355b5f8a4aa89075eb0ecdcad17fb63ea4804f6a2816f36e9ea27\": rpc error: code = NotFound desc = could not find container \"82bf1ab3009355b5f8a4aa89075eb0ecdcad17fb63ea4804f6a2816f36e9ea27\": container with ID starting with 82bf1ab3009355b5f8a4aa89075eb0ecdcad17fb63ea4804f6a2816f36e9ea27 not found: ID does not exist" Nov 03 01:59:24.448362 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:59:24.448340 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7fde87-7bf9f9f857-tcrkj"] Nov 03 01:59:24.458940 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:59:24.458905 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7fde87-7bf9f9f857-tcrkj"] Nov 03 01:59:25.626490 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:59:25.626456 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-h4fl6"] Nov 03 01:59:25.628486 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:59:25.628463 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-h4fl6"] Nov 03 01:59:25.650496 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:59:25.650461 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f9qrl"] Nov 03 01:59:25.650924 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:59:25.650909 4421 cpu_manager.go:414] "containerMap: removing container" podUID="22400262-f9f4-42aa-9f98-cebb1b519b91" containerName="maas-api" Nov 03 01:59:25.650979 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:59:25.650926 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22400262-f9f4-42aa-9f98-cebb1b519b91" containerName="maas-api" Nov 03 01:59:25.651028 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:59:25.651010 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="22400262-f9f4-42aa-9f98-cebb1b519b91" containerName="maas-api" Nov 03 01:59:25.655646 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:59:25.655630 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f9qrl" Nov 03 01:59:25.661442 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:59:25.661416 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f9qrl"] Nov 03 01:59:25.661647 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:59:25.661480 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 01:59:25.661738 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:59:25.661480 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:59:25.661738 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:59:25.661527 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 01:59:25.662779 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:59:25.662757 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-swfcr\"" Nov 03 01:59:25.725420 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:59:25.725373 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d48403bd-83fd-4dee-9e78-63530260621b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-f9qrl\" (UID: \"d48403bd-83fd-4dee-9e78-63530260621b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f9qrl" Nov 03 01:59:25.725690 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:59:25.725501 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d48403bd-83fd-4dee-9e78-63530260621b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-f9qrl\" (UID: \"d48403bd-83fd-4dee-9e78-63530260621b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f9qrl" Nov 03 01:59:25.725690 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:59:25.725599 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bg7g\" (UniqueName: \"kubernetes.io/projected/d48403bd-83fd-4dee-9e78-63530260621b-kube-api-access-4bg7g\") pod \"maas-api-revoke-sub-3d4156f1-f9qrl\" (UID: \"d48403bd-83fd-4dee-9e78-63530260621b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f9qrl" Nov 03 01:59:25.826854 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:59:25.826813 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d48403bd-83fd-4dee-9e78-63530260621b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-f9qrl\" (UID: \"d48403bd-83fd-4dee-9e78-63530260621b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f9qrl" Nov 03 01:59:25.826854 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:59:25.826855 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bg7g\" (UniqueName: \"kubernetes.io/projected/d48403bd-83fd-4dee-9e78-63530260621b-kube-api-access-4bg7g\") pod \"maas-api-revoke-sub-3d4156f1-f9qrl\" (UID: \"d48403bd-83fd-4dee-9e78-63530260621b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f9qrl" Nov 03 01:59:25.827151 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:59:25.826921 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d48403bd-83fd-4dee-9e78-63530260621b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-f9qrl\" (UID: \"d48403bd-83fd-4dee-9e78-63530260621b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f9qrl" Nov 03 01:59:25.827604 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:59:25.827580 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d48403bd-83fd-4dee-9e78-63530260621b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-f9qrl\" (UID: \"d48403bd-83fd-4dee-9e78-63530260621b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f9qrl" Nov 03 01:59:25.829350 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:59:25.829325 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d48403bd-83fd-4dee-9e78-63530260621b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-f9qrl\" (UID: \"d48403bd-83fd-4dee-9e78-63530260621b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f9qrl" Nov 03 01:59:25.834795 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:59:25.834777 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bg7g\" (UniqueName: \"kubernetes.io/projected/d48403bd-83fd-4dee-9e78-63530260621b-kube-api-access-4bg7g\") pod \"maas-api-revoke-sub-3d4156f1-f9qrl\" (UID: \"d48403bd-83fd-4dee-9e78-63530260621b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f9qrl" Nov 03 01:59:25.971086 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:59:25.970988 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f9qrl" Nov 03 01:59:25.977959 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:59:25.977922 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22400262-f9f4-42aa-9f98-cebb1b519b91" path="/var/lib/kubelet/pods/22400262-f9f4-42aa-9f98-cebb1b519b91/volumes" Nov 03 01:59:25.978498 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:59:25.978480 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="532cfac6-5118-42ca-959a-7eeaf46f63f4" path="/var/lib/kubelet/pods/532cfac6-5118-42ca-959a-7eeaf46f63f4/volumes" Nov 03 01:59:26.249884 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:59:26.249852 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f9qrl"] Nov 03 01:59:26.250918 ip-10-0-133-217 kubenswrapper[4421]: W1103 01:59:26.250893 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd48403bd_83fd_4dee_9e78_63530260621b.slice/crio-3afe6904f06fe2b41191d0376e546c56714f3cc6717628f13f609f00d7de76e5 WatchSource:0}: Error finding container 3afe6904f06fe2b41191d0376e546c56714f3cc6717628f13f609f00d7de76e5: Status 404 returned error can't find the container with id 3afe6904f06fe2b41191d0376e546c56714f3cc6717628f13f609f00d7de76e5 Nov 03 01:59:26.435935 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:59:26.435904 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f9qrl" event={"ID":"d48403bd-83fd-4dee-9e78-63530260621b","Type":"ContainerStarted","Data":"3afe6904f06fe2b41191d0376e546c56714f3cc6717628f13f609f00d7de76e5"} Nov 03 01:59:27.443077 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:59:27.442989 4421 generic.go:356] "Generic (PLEG): container finished" podID="d48403bd-83fd-4dee-9e78-63530260621b" containerID="70354a6af58e58e5e8a4deca98f384c226236456d4c51f07457e90ed1c4c8dbd" exitCode=0 Nov 03 01:59:27.443486 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:59:27.443075 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f9qrl" event={"ID":"d48403bd-83fd-4dee-9e78-63530260621b","Type":"ContainerDied","Data":"70354a6af58e58e5e8a4deca98f384c226236456d4c51f07457e90ed1c4c8dbd"} Nov 03 01:59:27.773138 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:59:27.773094 4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-7fde87-6b96867c8lcmpd" podUID="f751dabb-190f-4f7a-8fd3-ed5a1b268f3d" containerName="payload-processing" containerID="cri-o://71a8d7ffe0ff9778c83d31e2549985deadfc306f55ef1124b0a5ba9443b6402c" gracePeriod=25 Nov 03 01:59:28.033428 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:59:28.033360 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-7fde87-6b96867c8lcmpd" Nov 03 01:59:28.046448 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:59:28.046419 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f751dabb-190f-4f7a-8fd3-ed5a1b268f3d-kube-api-access-2jqxh\" (UniqueName: \"kubernetes.io/projected/f751dabb-190f-4f7a-8fd3-ed5a1b268f3d-kube-api-access-2jqxh\") pod \"f751dabb-190f-4f7a-8fd3-ed5a1b268f3d\" (UID: \"f751dabb-190f-4f7a-8fd3-ed5a1b268f3d\") " Nov 03 01:59:28.046600 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:59:28.046467 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f751dabb-190f-4f7a-8fd3-ed5a1b268f3d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f751dabb-190f-4f7a-8fd3-ed5a1b268f3d-plugins-config-volume\") pod \"f751dabb-190f-4f7a-8fd3-ed5a1b268f3d\" (UID: \"f751dabb-190f-4f7a-8fd3-ed5a1b268f3d\") " Nov 03 01:59:28.046890 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:59:28.046867 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f751dabb-190f-4f7a-8fd3-ed5a1b268f3d-plugins-config-volume" pod "f751dabb-190f-4f7a-8fd3-ed5a1b268f3d" (UID: "f751dabb-190f-4f7a-8fd3-ed5a1b268f3d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:59:28.048733 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:59:28.048686 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f751dabb-190f-4f7a-8fd3-ed5a1b268f3d-kube-api-access-2jqxh" pod "f751dabb-190f-4f7a-8fd3-ed5a1b268f3d" (UID: "f751dabb-190f-4f7a-8fd3-ed5a1b268f3d"). InnerVolumeSpecName "kube-api-access-2jqxh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:59:28.148248 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:59:28.148205 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jqxh\" (UniqueName: \"kubernetes.io/projected/f751dabb-190f-4f7a-8fd3-ed5a1b268f3d-kube-api-access-2jqxh\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:59:28.148248 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:59:28.148240 4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f751dabb-190f-4f7a-8fd3-ed5a1b268f3d-plugins-config-volume\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:59:28.449792 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:59:28.449699 4421 generic.go:356] "Generic (PLEG): container finished" podID="f751dabb-190f-4f7a-8fd3-ed5a1b268f3d" containerID="71a8d7ffe0ff9778c83d31e2549985deadfc306f55ef1124b0a5ba9443b6402c" exitCode=0 Nov 03 01:59:28.449792 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:59:28.449745 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-7fde87-6b96867c8lcmpd" Nov 03 01:59:28.450375 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:59:28.449782 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-7fde87-6b96867c8lcmpd" event={"ID":"f751dabb-190f-4f7a-8fd3-ed5a1b268f3d","Type":"ContainerDied","Data":"71a8d7ffe0ff9778c83d31e2549985deadfc306f55ef1124b0a5ba9443b6402c"} Nov 03 01:59:28.450375 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:59:28.449838 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-7fde87-6b96867c8lcmpd" event={"ID":"f751dabb-190f-4f7a-8fd3-ed5a1b268f3d","Type":"ContainerDied","Data":"195b9e3b500d542cb0e1d8cd0a1c6567d962eacffdb90dc3ea4fd47223902410"} Nov 03 01:59:28.450375 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:59:28.449864 4421 scope.go:122] "RemoveContainer" containerID="71a8d7ffe0ff9778c83d31e2549985deadfc306f55ef1124b0a5ba9443b6402c" Nov 03 01:59:28.459851 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:59:28.459832 4421 scope.go:122] "RemoveContainer" containerID="71a8d7ffe0ff9778c83d31e2549985deadfc306f55ef1124b0a5ba9443b6402c" Nov 03 01:59:28.460100 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:59:28.460079 4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71a8d7ffe0ff9778c83d31e2549985deadfc306f55ef1124b0a5ba9443b6402c\": container with ID starting with 71a8d7ffe0ff9778c83d31e2549985deadfc306f55ef1124b0a5ba9443b6402c not found: ID does not exist" containerID="71a8d7ffe0ff9778c83d31e2549985deadfc306f55ef1124b0a5ba9443b6402c" Nov 03 01:59:28.460176 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:59:28.460107 4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71a8d7ffe0ff9778c83d31e2549985deadfc306f55ef1124b0a5ba9443b6402c"} err="failed to get container status \"71a8d7ffe0ff9778c83d31e2549985deadfc306f55ef1124b0a5ba9443b6402c\": rpc error: code = NotFound desc = could not find container \"71a8d7ffe0ff9778c83d31e2549985deadfc306f55ef1124b0a5ba9443b6402c\": container with ID starting with 71a8d7ffe0ff9778c83d31e2549985deadfc306f55ef1124b0a5ba9443b6402c not found: ID does not exist" Nov 03 01:59:28.472324 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:59:28.472300 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-7fde87-6b96867c8lcmpd"] Nov 03 01:59:28.476348 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:59:28.476321 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-7fde87-6b96867c8lcmpd"] Nov 03 01:59:28.599328 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:59:28.599302 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f9qrl" Nov 03 01:59:28.652332 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:59:28.652295 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d48403bd-83fd-4dee-9e78-63530260621b-kube-api-access-4bg7g\" (UniqueName: \"kubernetes.io/projected/d48403bd-83fd-4dee-9e78-63530260621b-kube-api-access-4bg7g\") pod \"d48403bd-83fd-4dee-9e78-63530260621b\" (UID: \"d48403bd-83fd-4dee-9e78-63530260621b\") " Nov 03 01:59:28.652544 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:59:28.652400 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d48403bd-83fd-4dee-9e78-63530260621b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d48403bd-83fd-4dee-9e78-63530260621b-maas-api-xks-ca\") pod \"d48403bd-83fd-4dee-9e78-63530260621b\" (UID: \"d48403bd-83fd-4dee-9e78-63530260621b\") " Nov 03 01:59:28.652544 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:59:28.652474 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d48403bd-83fd-4dee-9e78-63530260621b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d48403bd-83fd-4dee-9e78-63530260621b-maas-api-openshift-ca\") pod \"d48403bd-83fd-4dee-9e78-63530260621b\" (UID: \"d48403bd-83fd-4dee-9e78-63530260621b\") " Nov 03 01:59:28.652831 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:59:28.652806 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d48403bd-83fd-4dee-9e78-63530260621b-maas-api-openshift-ca" pod "d48403bd-83fd-4dee-9e78-63530260621b" (UID: "d48403bd-83fd-4dee-9e78-63530260621b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:59:28.654524 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:59:28.654500 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d48403bd-83fd-4dee-9e78-63530260621b-maas-api-xks-ca" pod "d48403bd-83fd-4dee-9e78-63530260621b" (UID: "d48403bd-83fd-4dee-9e78-63530260621b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:59:28.654626 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:59:28.654532 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d48403bd-83fd-4dee-9e78-63530260621b-kube-api-access-4bg7g" pod "d48403bd-83fd-4dee-9e78-63530260621b" (UID: "d48403bd-83fd-4dee-9e78-63530260621b"). InnerVolumeSpecName "kube-api-access-4bg7g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:59:28.754335 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:59:28.754298 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4bg7g\" (UniqueName: \"kubernetes.io/projected/d48403bd-83fd-4dee-9e78-63530260621b-kube-api-access-4bg7g\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:59:28.754335 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:59:28.754332 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d48403bd-83fd-4dee-9e78-63530260621b-maas-api-xks-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:59:28.754335 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:59:28.754342 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d48403bd-83fd-4dee-9e78-63530260621b-maas-api-openshift-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:59:29.456235 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:59:29.456141 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f9qrl" Nov 03 01:59:29.456713 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:59:29.456145 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f9qrl" event={"ID":"d48403bd-83fd-4dee-9e78-63530260621b","Type":"ContainerDied","Data":"3afe6904f06fe2b41191d0376e546c56714f3cc6717628f13f609f00d7de76e5"} Nov 03 01:59:29.456713 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:59:29.456269 4421 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3afe6904f06fe2b41191d0376e546c56714f3cc6717628f13f609f00d7de76e5" Nov 03 01:59:29.977360 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:59:29.977329 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f751dabb-190f-4f7a-8fd3-ed5a1b268f3d" path="/var/lib/kubelet/pods/f751dabb-190f-4f7a-8fd3-ed5a1b268f3d/volumes" Nov 03 01:59:33.021507 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:59:33.021471 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-p67d6"] Nov 03 01:59:33.026028 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:59:33.026000 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-p67d6"] Nov 03 01:59:33.977500 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:59:33.977460 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8963e105-fbc7-4218-944c-8f67b2444ffe" path="/var/lib/kubelet/pods/8963e105-fbc7-4218-944c-8f67b2444ffe/volumes" Nov 03 01:59:43.029403 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:59:43.029304 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8wz5k"] Nov 03 01:59:43.031164 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:59:43.031139 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8wz5k"] Nov 03 01:59:43.977186 ip-10-0-133-217 kubenswrapper[4421]: I1103 01:59:43.977146 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c00a507-1d4e-4a5e-b87b-0e0de7419052" path="/var/lib/kubelet/pods/7c00a507-1d4e-4a5e-b87b-0e0de7419052/volumes" Nov 03 01:59:44.668760 ip-10-0-133-217 kubenswrapper[4421]: E1103 01:59:44.668721 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 02:00:00.146260 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:00:00.146219 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894520-qdkv5"] Nov 03 02:00:00.146816 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:00:00.146660 4421 cpu_manager.go:414] "containerMap: removing container" podUID="f751dabb-190f-4f7a-8fd3-ed5a1b268f3d" containerName="payload-processing" Nov 03 02:00:00.146816 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:00:00.146673 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f751dabb-190f-4f7a-8fd3-ed5a1b268f3d" containerName="payload-processing" Nov 03 02:00:00.146816 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:00:00.146683 4421 cpu_manager.go:414] "containerMap: removing container" podUID="d48403bd-83fd-4dee-9e78-63530260621b" containerName="revoke-keys" Nov 03 02:00:00.146816 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:00:00.146690 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d48403bd-83fd-4dee-9e78-63530260621b" containerName="revoke-keys" Nov 03 02:00:00.146816 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:00:00.146755 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="f751dabb-190f-4f7a-8fd3-ed5a1b268f3d" containerName="payload-processing" Nov 03 02:00:00.146816 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:00:00.146763 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="d48403bd-83fd-4dee-9e78-63530260621b" containerName="revoke-keys" Nov 03 02:00:00.150014 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:00:00.149996 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894520-qdkv5" Nov 03 02:00:00.156107 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:00:00.156070 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 02:00:00.158032 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:00:00.157353 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-swfcr\"" Nov 03 02:00:00.158032 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:00:00.157447 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 02:00:00.160154 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:00:00.160130 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894520-qdkv5"] Nov 03 02:00:00.305418 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:00:00.305374 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hs4kq\" (UniqueName: \"kubernetes.io/projected/a515ad6e-ccc5-4cf2-bb75-e9881b9d9077-kube-api-access-hs4kq\") pod \"maas-api-key-cleanup-29894520-qdkv5\" (UID: \"a515ad6e-ccc5-4cf2-bb75-e9881b9d9077\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894520-qdkv5" Nov 03 02:00:00.305418 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:00:00.305425 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a515ad6e-ccc5-4cf2-bb75-e9881b9d9077-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894520-qdkv5\" (UID: \"a515ad6e-ccc5-4cf2-bb75-e9881b9d9077\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894520-qdkv5" Nov 03 02:00:00.407116 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:00:00.407014 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hs4kq\" (UniqueName: \"kubernetes.io/projected/a515ad6e-ccc5-4cf2-bb75-e9881b9d9077-kube-api-access-hs4kq\") pod \"maas-api-key-cleanup-29894520-qdkv5\" (UID: \"a515ad6e-ccc5-4cf2-bb75-e9881b9d9077\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894520-qdkv5" Nov 03 02:00:00.407116 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:00:00.407065 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a515ad6e-ccc5-4cf2-bb75-e9881b9d9077-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894520-qdkv5\" (UID: \"a515ad6e-ccc5-4cf2-bb75-e9881b9d9077\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894520-qdkv5" Nov 03 02:00:00.407733 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:00:00.407714 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a515ad6e-ccc5-4cf2-bb75-e9881b9d9077-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894520-qdkv5\" (UID: \"a515ad6e-ccc5-4cf2-bb75-e9881b9d9077\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894520-qdkv5" Nov 03 02:00:00.417179 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:00:00.417159 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hs4kq\" (UniqueName: \"kubernetes.io/projected/a515ad6e-ccc5-4cf2-bb75-e9881b9d9077-kube-api-access-hs4kq\") pod \"maas-api-key-cleanup-29894520-qdkv5\" (UID: \"a515ad6e-ccc5-4cf2-bb75-e9881b9d9077\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894520-qdkv5" Nov 03 02:00:00.465062 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:00:00.465022 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894520-qdkv5" Nov 03 02:00:00.737970 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:00:00.737945 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894520-qdkv5"] Nov 03 02:00:00.740831 ip-10-0-133-217 kubenswrapper[4421]: W1103 02:00:00.740799 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda515ad6e_ccc5_4cf2_bb75_e9881b9d9077.slice/crio-ccf75b0479e006ba035c1a5cd0d102ee72dad8e6f58ddc5cee349d427b0e3675 WatchSource:0}: Error finding container ccf75b0479e006ba035c1a5cd0d102ee72dad8e6f58ddc5cee349d427b0e3675: Status 404 returned error can't find the container with id ccf75b0479e006ba035c1a5cd0d102ee72dad8e6f58ddc5cee349d427b0e3675 Nov 03 02:00:01.623269 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:00:01.623229 4421 generic.go:356] "Generic (PLEG): container finished" podID="a515ad6e-ccc5-4cf2-bb75-e9881b9d9077" containerID="2438f2889e44f3ebfa6f0b1c599b94f33868f18060b18baa16ebd2e877cbad39" exitCode=60 Nov 03 02:00:01.623720 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:00:01.623312 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894520-qdkv5" event={"ID":"a515ad6e-ccc5-4cf2-bb75-e9881b9d9077","Type":"ContainerDied","Data":"2438f2889e44f3ebfa6f0b1c599b94f33868f18060b18baa16ebd2e877cbad39"} Nov 03 02:00:01.623720 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:00:01.623366 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894520-qdkv5" event={"ID":"a515ad6e-ccc5-4cf2-bb75-e9881b9d9077","Type":"ContainerStarted","Data":"ccf75b0479e006ba035c1a5cd0d102ee72dad8e6f58ddc5cee349d427b0e3675"} Nov 03 02:00:01.623720 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:00:01.623620 4421 scope.go:122] "RemoveContainer" containerID="2438f2889e44f3ebfa6f0b1c599b94f33868f18060b18baa16ebd2e877cbad39" Nov 03 02:00:02.642126 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:00:02.642031 4421 generic.go:356] "Generic (PLEG): container finished" podID="a515ad6e-ccc5-4cf2-bb75-e9881b9d9077" containerID="134b3442c3217a867a1f182a5da20b0e9287846f2311bf1534c288e04b28aa7c" exitCode=60 Nov 03 02:00:02.642126 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:00:02.642094 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894520-qdkv5" event={"ID":"a515ad6e-ccc5-4cf2-bb75-e9881b9d9077","Type":"ContainerDied","Data":"134b3442c3217a867a1f182a5da20b0e9287846f2311bf1534c288e04b28aa7c"} Nov 03 02:00:02.642703 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:00:02.642198 4421 scope.go:122] "RemoveContainer" containerID="2438f2889e44f3ebfa6f0b1c599b94f33868f18060b18baa16ebd2e877cbad39" Nov 03 02:00:02.642703 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:00:02.642487 4421 scope.go:122] "RemoveContainer" containerID="134b3442c3217a867a1f182a5da20b0e9287846f2311bf1534c288e04b28aa7c" Nov 03 02:00:02.642783 ip-10-0-133-217 kubenswrapper[4421]: E1103 02:00:02.642743 4421 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894520-qdkv5_odh-ai-gateway-infra(a515ad6e-ccc5-4cf2-bb75-e9881b9d9077)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894520-qdkv5" podUID="a515ad6e-ccc5-4cf2-bb75-e9881b9d9077" Nov 03 02:00:03.024257 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:00:03.024189 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-jbdmp"] Nov 03 02:00:03.026475 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:00:03.026447 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-jbdmp"] Nov 03 02:00:03.649110 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:00:03.649075 4421 scope.go:122] "RemoveContainer" containerID="134b3442c3217a867a1f182a5da20b0e9287846f2311bf1534c288e04b28aa7c" Nov 03 02:00:03.649549 ip-10-0-133-217 kubenswrapper[4421]: E1103 02:00:03.649334 4421 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894520-qdkv5_odh-ai-gateway-infra(a515ad6e-ccc5-4cf2-bb75-e9881b9d9077)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894520-qdkv5" podUID="a515ad6e-ccc5-4cf2-bb75-e9881b9d9077" Nov 03 02:00:03.977642 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:00:03.977565 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5051a056-68ce-45b5-85e1-cfd8485097ea" path="/var/lib/kubelet/pods/5051a056-68ce-45b5-85e1-cfd8485097ea/volumes" Nov 03 02:00:07.334029 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:00:07.333991 4421 scope.go:122] "RemoveContainer" containerID="c169e086d6832d2ff9e6c97eb66de6005fcbd8b10be463ce65268a8ec5783dc4" Nov 03 02:00:07.343695 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:00:07.343670 4421 scope.go:122] "RemoveContainer" containerID="d09130e07e680fe84116b1af0469f3c7e5af408ef46e370ae49838bba0ace23c" Nov 03 02:00:07.351928 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:00:07.351908 4421 scope.go:122] "RemoveContainer" containerID="2aea7dae110b026663e11754c5e98d0b6b019ef25d8d8923e9efa43b15f5f8f0" Nov 03 02:00:07.359989 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:00:07.359968 4421 scope.go:122] "RemoveContainer" containerID="d291667c50d3fc9ebee497acee01536d77ade76419c838f23c832993425926a2" Nov 03 02:00:13.971336 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:00:13.971299 4421 scope.go:122] "RemoveContainer" containerID="134b3442c3217a867a1f182a5da20b0e9287846f2311bf1534c288e04b28aa7c" Nov 03 02:00:14.669454 ip-10-0-133-217 kubenswrapper[4421]: E1103 02:00:14.669367 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 02:00:14.704286 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:00:14.704250 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894520-qdkv5" event={"ID":"a515ad6e-ccc5-4cf2-bb75-e9881b9d9077","Type":"ContainerStarted","Data":"f5a58e49f7167667751fa5d70499c82a9a26254bffff2c7c59dea443e6ac5a4c"} Nov 03 02:00:14.721681 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:00:14.721626 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894520-qdkv5" podStartSLOduration=14.721613607 podStartE2EDuration="14.721613607s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:00:14.71994704 +0000 UTC m=+4756.890950135" watchObservedRunningTime="2026-11-03 02:00:14.721613607 +0000 UTC m=+4756.892616727" Nov 03 02:00:15.711970 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:00:15.711937 4421 generic.go:356] "Generic (PLEG): container finished" podID="a515ad6e-ccc5-4cf2-bb75-e9881b9d9077" containerID="f5a58e49f7167667751fa5d70499c82a9a26254bffff2c7c59dea443e6ac5a4c" exitCode=60 Nov 03 02:00:15.712445 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:00:15.712000 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894520-qdkv5" event={"ID":"a515ad6e-ccc5-4cf2-bb75-e9881b9d9077","Type":"ContainerDied","Data":"f5a58e49f7167667751fa5d70499c82a9a26254bffff2c7c59dea443e6ac5a4c"} Nov 03 02:00:15.712445 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:00:15.712052 4421 scope.go:122] "RemoveContainer" containerID="134b3442c3217a867a1f182a5da20b0e9287846f2311bf1534c288e04b28aa7c" Nov 03 02:00:15.712445 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:00:15.712406 4421 scope.go:122] "RemoveContainer" containerID="f5a58e49f7167667751fa5d70499c82a9a26254bffff2c7c59dea443e6ac5a4c" Nov 03 02:00:15.712669 ip-10-0-133-217 kubenswrapper[4421]: E1103 02:00:15.712649 4421 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29894520-qdkv5_odh-ai-gateway-infra(a515ad6e-ccc5-4cf2-bb75-e9881b9d9077)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894520-qdkv5" podUID="a515ad6e-ccc5-4cf2-bb75-e9881b9d9077" Nov 03 02:00:15.729916 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:00:15.729884 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894520-qdkv5"] Nov 03 02:00:16.864494 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:00:16.864464 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894520-qdkv5" Nov 03 02:00:16.960683 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:00:16.960647 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a515ad6e-ccc5-4cf2-bb75-e9881b9d9077-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a515ad6e-ccc5-4cf2-bb75-e9881b9d9077-maas-api-service-ca\") pod \"a515ad6e-ccc5-4cf2-bb75-e9881b9d9077\" (UID: \"a515ad6e-ccc5-4cf2-bb75-e9881b9d9077\") " Nov 03 02:00:16.960865 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:00:16.960698 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a515ad6e-ccc5-4cf2-bb75-e9881b9d9077-kube-api-access-hs4kq\" (UniqueName: \"kubernetes.io/projected/a515ad6e-ccc5-4cf2-bb75-e9881b9d9077-kube-api-access-hs4kq\") pod \"a515ad6e-ccc5-4cf2-bb75-e9881b9d9077\" (UID: \"a515ad6e-ccc5-4cf2-bb75-e9881b9d9077\") " Nov 03 02:00:16.961045 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:00:16.961022 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a515ad6e-ccc5-4cf2-bb75-e9881b9d9077-maas-api-service-ca" pod "a515ad6e-ccc5-4cf2-bb75-e9881b9d9077" (UID: "a515ad6e-ccc5-4cf2-bb75-e9881b9d9077"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:00:16.962853 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:00:16.962823 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a515ad6e-ccc5-4cf2-bb75-e9881b9d9077-kube-api-access-hs4kq" pod "a515ad6e-ccc5-4cf2-bb75-e9881b9d9077" (UID: "a515ad6e-ccc5-4cf2-bb75-e9881b9d9077"). InnerVolumeSpecName "kube-api-access-hs4kq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:00:17.062015 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:00:17.061977 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a515ad6e-ccc5-4cf2-bb75-e9881b9d9077-maas-api-service-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 02:00:17.062015 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:00:17.062010 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hs4kq\" (UniqueName: \"kubernetes.io/projected/a515ad6e-ccc5-4cf2-bb75-e9881b9d9077-kube-api-access-hs4kq\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 02:00:17.725103 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:00:17.725073 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894520-qdkv5" Nov 03 02:00:17.725103 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:00:17.725090 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894520-qdkv5" event={"ID":"a515ad6e-ccc5-4cf2-bb75-e9881b9d9077","Type":"ContainerDied","Data":"ccf75b0479e006ba035c1a5cd0d102ee72dad8e6f58ddc5cee349d427b0e3675"} Nov 03 02:00:17.725420 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:00:17.725144 4421 scope.go:122] "RemoveContainer" containerID="f5a58e49f7167667751fa5d70499c82a9a26254bffff2c7c59dea443e6ac5a4c" Nov 03 02:00:17.756469 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:00:17.756438 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894520-qdkv5"] Nov 03 02:00:17.760946 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:00:17.760918 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894520-qdkv5"] Nov 03 02:00:17.978457 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:00:17.978378 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a515ad6e-ccc5-4cf2-bb75-e9881b9d9077" path="/var/lib/kubelet/pods/a515ad6e-ccc5-4cf2-bb75-e9881b9d9077/volumes" Nov 03 02:00:20.022434 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:00:20.022397 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-kqt48"] Nov 03 02:00:20.029422 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:00:20.029396 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-kqt48"] Nov 03 02:00:21.976555 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:00:21.976519 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fcad2130-1e0c-47dd-8368-50effdd4d69d" path="/var/lib/kubelet/pods/fcad2130-1e0c-47dd-8368-50effdd4d69d/volumes" Nov 03 02:00:36.027375 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:00:36.027325 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-dmff7"] Nov 03 02:00:36.029975 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:00:36.029951 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-dmff7"] Nov 03 02:00:37.985130 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:00:37.985094 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76b9d8cd-40e1-40da-9bf2-014af7b1a15d" path="/var/lib/kubelet/pods/76b9d8cd-40e1-40da-9bf2-014af7b1a15d/volumes" Nov 03 02:00:44.669976 ip-10-0-133-217 kubenswrapper[4421]: E1103 02:00:44.669932 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 02:00:50.024770 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:00:50.024728 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-xv8hh"] Nov 03 02:00:50.026490 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:00:50.026463 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-xv8hh"] Nov 03 02:00:51.977161 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:00:51.977113 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f44d783a-0886-43e6-bace-2e3b4a166664" path="/var/lib/kubelet/pods/f44d783a-0886-43e6-bace-2e3b4a166664/volumes" Nov 03 02:01:06.491976 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:01:06.491928 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-mldb6"] Nov 03 02:01:06.492680 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:01:06.492541 4421 cpu_manager.go:414] "containerMap: removing container" podUID="a515ad6e-ccc5-4cf2-bb75-e9881b9d9077" containerName="cleanup" Nov 03 02:01:06.492680 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:01:06.492563 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a515ad6e-ccc5-4cf2-bb75-e9881b9d9077" containerName="cleanup" Nov 03 02:01:06.492680 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:01:06.492619 4421 cpu_manager.go:414] "containerMap: removing container" podUID="a515ad6e-ccc5-4cf2-bb75-e9881b9d9077" containerName="cleanup" Nov 03 02:01:06.492680 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:01:06.492627 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a515ad6e-ccc5-4cf2-bb75-e9881b9d9077" containerName="cleanup" Nov 03 02:01:06.492680 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:01:06.492642 4421 cpu_manager.go:414] "containerMap: removing container" podUID="a515ad6e-ccc5-4cf2-bb75-e9881b9d9077" containerName="cleanup" Nov 03 02:01:06.492680 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:01:06.492651 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a515ad6e-ccc5-4cf2-bb75-e9881b9d9077" containerName="cleanup" Nov 03 02:01:06.493047 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:01:06.492731 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="a515ad6e-ccc5-4cf2-bb75-e9881b9d9077" containerName="cleanup" Nov 03 02:01:06.493047 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:01:06.492748 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="a515ad6e-ccc5-4cf2-bb75-e9881b9d9077" containerName="cleanup" Nov 03 02:01:06.496365 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:01:06.496342 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-mldb6" Nov 03 02:01:06.501590 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:01:06.501565 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 02:01:06.501746 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:01:06.501599 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 02:01:06.502956 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:01:06.502658 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-mldb6"] Nov 03 02:01:06.571517 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:01:06.571474 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 02:01:06.571517 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:01:06.571505 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-swfcr\"" Nov 03 02:01:06.584968 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:01:06.584937 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwv7s\" (UniqueName: \"kubernetes.io/projected/00f785dd-eed8-4121-87fb-f7f88bb416b8-kube-api-access-qwv7s\") pod \"maas-api-revoke-sub-d576e68b-mldb6\" (UID: \"00f785dd-eed8-4121-87fb-f7f88bb416b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-mldb6" Nov 03 02:01:06.585064 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:01:06.584973 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00f785dd-eed8-4121-87fb-f7f88bb416b8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-mldb6\" (UID: \"00f785dd-eed8-4121-87fb-f7f88bb416b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-mldb6" Nov 03 02:01:06.585064 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:01:06.584996 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00f785dd-eed8-4121-87fb-f7f88bb416b8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-mldb6\" (UID: \"00f785dd-eed8-4121-87fb-f7f88bb416b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-mldb6" Nov 03 02:01:06.686057 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:01:06.686010 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwv7s\" (UniqueName: \"kubernetes.io/projected/00f785dd-eed8-4121-87fb-f7f88bb416b8-kube-api-access-qwv7s\") pod \"maas-api-revoke-sub-d576e68b-mldb6\" (UID: \"00f785dd-eed8-4121-87fb-f7f88bb416b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-mldb6" Nov 03 02:01:06.686292 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:01:06.686061 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00f785dd-eed8-4121-87fb-f7f88bb416b8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-mldb6\" (UID: \"00f785dd-eed8-4121-87fb-f7f88bb416b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-mldb6" Nov 03 02:01:06.686292 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:01:06.686202 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00f785dd-eed8-4121-87fb-f7f88bb416b8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-mldb6\" (UID: \"00f785dd-eed8-4121-87fb-f7f88bb416b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-mldb6" Nov 03 02:01:06.686795 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:01:06.686774 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00f785dd-eed8-4121-87fb-f7f88bb416b8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-mldb6\" (UID: \"00f785dd-eed8-4121-87fb-f7f88bb416b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-mldb6" Nov 03 02:01:06.688634 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:01:06.688604 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00f785dd-eed8-4121-87fb-f7f88bb416b8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-mldb6\" (UID: \"00f785dd-eed8-4121-87fb-f7f88bb416b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-mldb6" Nov 03 02:01:06.695624 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:01:06.695596 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwv7s\" (UniqueName: \"kubernetes.io/projected/00f785dd-eed8-4121-87fb-f7f88bb416b8-kube-api-access-qwv7s\") pod \"maas-api-revoke-sub-d576e68b-mldb6\" (UID: \"00f785dd-eed8-4121-87fb-f7f88bb416b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-mldb6" Nov 03 02:01:06.811127 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:01:06.811092 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-mldb6" Nov 03 02:01:07.090223 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:01:07.090012 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-mldb6"] Nov 03 02:01:07.092809 ip-10-0-133-217 kubenswrapper[4421]: W1103 02:01:07.092783 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00f785dd_eed8_4121_87fb_f7f88bb416b8.slice/crio-398865471d0bac0dd8d4f4edfda3895814390238163d4015a6014ba4d9153a1e WatchSource:0}: Error finding container 398865471d0bac0dd8d4f4edfda3895814390238163d4015a6014ba4d9153a1e: Status 404 returned error can't find the container with id 398865471d0bac0dd8d4f4edfda3895814390238163d4015a6014ba4d9153a1e Nov 03 02:01:07.399684 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:01:07.399599 4421 scope.go:122] "RemoveContainer" containerID="3b2769defbdc86b810e913735765fc6308047c0753a260e106d109e24f5b9dc8" Nov 03 02:01:07.408774 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:01:07.408750 4421 scope.go:122] "RemoveContainer" containerID="773851ea7f158baecddec6a07fd6bfe9c93d46a62b7407a75c9204a40c79563c" Nov 03 02:01:07.417650 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:01:07.417624 4421 scope.go:122] "RemoveContainer" containerID="bda0f63a91fc43a277504c7bcd516bea28915efb13f71c0bdb103539ca1b99f7" Nov 03 02:01:07.998796 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:01:07.998758 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-mldb6" event={"ID":"00f785dd-eed8-4121-87fb-f7f88bb416b8","Type":"ContainerStarted","Data":"ab6327d7f57b31336e5cc9e26b74609e0a87810db5e3ba5e3757efa8a55b68a6"} Nov 03 02:01:07.999265 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:01:07.998816 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-mldb6" event={"ID":"00f785dd-eed8-4121-87fb-f7f88bb416b8","Type":"ContainerStarted","Data":"398865471d0bac0dd8d4f4edfda3895814390238163d4015a6014ba4d9153a1e"} Nov 03 02:01:08.017050 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:01:08.017005 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-mldb6" podStartSLOduration=2.016989327 podStartE2EDuration="2.016989327s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:01:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:01:08.01575006 +0000 UTC m=+4810.186753156" watchObservedRunningTime="2026-11-03 02:01:08.016989327 +0000 UTC m=+4810.187992423" Nov 03 02:01:09.004704 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:01:09.004662 4421 generic.go:356] "Generic (PLEG): container finished" podID="00f785dd-eed8-4121-87fb-f7f88bb416b8" containerID="ab6327d7f57b31336e5cc9e26b74609e0a87810db5e3ba5e3757efa8a55b68a6" exitCode=0 Nov 03 02:01:09.005151 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:01:09.004707 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-mldb6" event={"ID":"00f785dd-eed8-4121-87fb-f7f88bb416b8","Type":"ContainerDied","Data":"ab6327d7f57b31336e5cc9e26b74609e0a87810db5e3ba5e3757efa8a55b68a6"} Nov 03 02:01:10.155562 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:01:10.155530 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-mldb6" Nov 03 02:01:10.323592 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:01:10.323498 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00f785dd-eed8-4121-87fb-f7f88bb416b8-kube-api-access-qwv7s\" (UniqueName: \"kubernetes.io/projected/00f785dd-eed8-4121-87fb-f7f88bb416b8-kube-api-access-qwv7s\") pod \"00f785dd-eed8-4121-87fb-f7f88bb416b8\" (UID: \"00f785dd-eed8-4121-87fb-f7f88bb416b8\") " Nov 03 02:01:10.323592 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:01:10.323582 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/00f785dd-eed8-4121-87fb-f7f88bb416b8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00f785dd-eed8-4121-87fb-f7f88bb416b8-maas-api-xks-ca\") pod \"00f785dd-eed8-4121-87fb-f7f88bb416b8\" (UID: \"00f785dd-eed8-4121-87fb-f7f88bb416b8\") " Nov 03 02:01:10.323855 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:01:10.323651 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00f785dd-eed8-4121-87fb-f7f88bb416b8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00f785dd-eed8-4121-87fb-f7f88bb416b8-maas-api-openshift-ca\") pod \"00f785dd-eed8-4121-87fb-f7f88bb416b8\" (UID: \"00f785dd-eed8-4121-87fb-f7f88bb416b8\") " Nov 03 02:01:10.324027 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:01:10.324000 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00f785dd-eed8-4121-87fb-f7f88bb416b8-maas-api-openshift-ca" pod "00f785dd-eed8-4121-87fb-f7f88bb416b8" (UID: "00f785dd-eed8-4121-87fb-f7f88bb416b8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:01:10.325730 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:01:10.325700 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00f785dd-eed8-4121-87fb-f7f88bb416b8-maas-api-xks-ca" pod "00f785dd-eed8-4121-87fb-f7f88bb416b8" (UID: "00f785dd-eed8-4121-87fb-f7f88bb416b8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:01:10.325887 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:01:10.325866 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00f785dd-eed8-4121-87fb-f7f88bb416b8-kube-api-access-qwv7s" pod "00f785dd-eed8-4121-87fb-f7f88bb416b8" (UID: "00f785dd-eed8-4121-87fb-f7f88bb416b8"). InnerVolumeSpecName "kube-api-access-qwv7s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:01:10.424543 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:01:10.424501 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwv7s\" (UniqueName: \"kubernetes.io/projected/00f785dd-eed8-4121-87fb-f7f88bb416b8-kube-api-access-qwv7s\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 02:01:10.424543 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:01:10.424535 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00f785dd-eed8-4121-87fb-f7f88bb416b8-maas-api-xks-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 02:01:10.424543 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:01:10.424546 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00f785dd-eed8-4121-87fb-f7f88bb416b8-maas-api-openshift-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 02:01:11.020028 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:01:11.019996 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-mldb6" Nov 03 02:01:11.020219 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:01:11.020038 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-mldb6" event={"ID":"00f785dd-eed8-4121-87fb-f7f88bb416b8","Type":"ContainerDied","Data":"398865471d0bac0dd8d4f4edfda3895814390238163d4015a6014ba4d9153a1e"} Nov 03 02:01:11.020219 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:01:11.020088 4421 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="398865471d0bac0dd8d4f4edfda3895814390238163d4015a6014ba4d9153a1e" Nov 03 02:01:14.669318 ip-10-0-133-217 kubenswrapper[4421]: E1103 02:01:14.669257 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 02:01:22.744070 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:01:22.744030 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ngqgb"] Nov 03 02:01:22.746460 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:01:22.744680 4421 cpu_manager.go:414] "containerMap: removing container" podUID="00f785dd-eed8-4121-87fb-f7f88bb416b8" containerName="revoke-keys" Nov 03 02:01:22.746460 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:01:22.744700 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00f785dd-eed8-4121-87fb-f7f88bb416b8" containerName="revoke-keys" Nov 03 02:01:22.746460 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:01:22.744828 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="00f785dd-eed8-4121-87fb-f7f88bb416b8" containerName="revoke-keys" Nov 03 02:01:22.746460 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:01:22.744841 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="a515ad6e-ccc5-4cf2-bb75-e9881b9d9077" containerName="cleanup" Nov 03 02:01:22.748175 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:01:22.748159 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ngqgb" Nov 03 02:01:22.754646 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:01:22.754621 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ngqgb"] Nov 03 02:01:22.755555 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:01:22.755533 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-swfcr\"" Nov 03 02:01:22.755699 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:01:22.755533 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 02:01:22.755699 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:01:22.755535 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 02:01:22.755917 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:01:22.755540 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 02:01:22.783533 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:01:22.783494 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d06f23d7-efb5-4ad5-9586-e8cd430a6229-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-ngqgb\" (UID: \"d06f23d7-efb5-4ad5-9586-e8cd430a6229\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ngqgb" Nov 03 02:01:22.783746 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:01:22.783556 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4s9wb\" (UniqueName: \"kubernetes.io/projected/d06f23d7-efb5-4ad5-9586-e8cd430a6229-kube-api-access-4s9wb\") pod \"maas-api-revoke-sub-d2bc93bc-ngqgb\" (UID: \"d06f23d7-efb5-4ad5-9586-e8cd430a6229\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ngqgb" Nov 03 02:01:22.783746 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:01:22.783698 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d06f23d7-efb5-4ad5-9586-e8cd430a6229-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-ngqgb\" (UID: \"d06f23d7-efb5-4ad5-9586-e8cd430a6229\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ngqgb" Nov 03 02:01:22.884637 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:01:22.884603 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d06f23d7-efb5-4ad5-9586-e8cd430a6229-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-ngqgb\" (UID: \"d06f23d7-efb5-4ad5-9586-e8cd430a6229\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ngqgb" Nov 03 02:01:22.884820 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:01:22.884667 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4s9wb\" (UniqueName: \"kubernetes.io/projected/d06f23d7-efb5-4ad5-9586-e8cd430a6229-kube-api-access-4s9wb\") pod \"maas-api-revoke-sub-d2bc93bc-ngqgb\" (UID: \"d06f23d7-efb5-4ad5-9586-e8cd430a6229\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ngqgb" Nov 03 02:01:22.884820 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:01:22.884720 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d06f23d7-efb5-4ad5-9586-e8cd430a6229-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-ngqgb\" (UID: \"d06f23d7-efb5-4ad5-9586-e8cd430a6229\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ngqgb" Nov 03 02:01:22.885449 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:01:22.885422 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d06f23d7-efb5-4ad5-9586-e8cd430a6229-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-ngqgb\" (UID: \"d06f23d7-efb5-4ad5-9586-e8cd430a6229\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ngqgb" Nov 03 02:01:22.886973 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:01:22.886944 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d06f23d7-efb5-4ad5-9586-e8cd430a6229-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-ngqgb\" (UID: \"d06f23d7-efb5-4ad5-9586-e8cd430a6229\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ngqgb" Nov 03 02:01:22.893055 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:01:22.893034 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4s9wb\" (UniqueName: \"kubernetes.io/projected/d06f23d7-efb5-4ad5-9586-e8cd430a6229-kube-api-access-4s9wb\") pod \"maas-api-revoke-sub-d2bc93bc-ngqgb\" (UID: \"d06f23d7-efb5-4ad5-9586-e8cd430a6229\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ngqgb" Nov 03 02:01:23.065502 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:01:23.065455 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ngqgb" Nov 03 02:01:23.346554 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:01:23.346517 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ngqgb"] Nov 03 02:01:23.349163 ip-10-0-133-217 kubenswrapper[4421]: W1103 02:01:23.349133 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd06f23d7_efb5_4ad5_9586_e8cd430a6229.slice/crio-aedc4872113c1af0345066858d19e153c5444bc2990125f0f652f890f6ed7f08 WatchSource:0}: Error finding container aedc4872113c1af0345066858d19e153c5444bc2990125f0f652f890f6ed7f08: Status 404 returned error can't find the container with id aedc4872113c1af0345066858d19e153c5444bc2990125f0f652f890f6ed7f08 Nov 03 02:01:24.097735 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:01:24.097692 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ngqgb" event={"ID":"d06f23d7-efb5-4ad5-9586-e8cd430a6229","Type":"ContainerStarted","Data":"6394b99dc1679be7910433d15688f817a4796cfd57ccef834bea4c77252d7491"} Nov 03 02:01:24.098163 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:01:24.097752 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ngqgb" event={"ID":"d06f23d7-efb5-4ad5-9586-e8cd430a6229","Type":"ContainerStarted","Data":"aedc4872113c1af0345066858d19e153c5444bc2990125f0f652f890f6ed7f08"} Nov 03 02:01:24.114425 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:01:24.114371 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ngqgb" podStartSLOduration=2.114356218 podStartE2EDuration="2.114356218s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:01:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:01:24.112680989 +0000 UTC m=+4826.283684087" watchObservedRunningTime="2026-11-03 02:01:24.114356218 +0000 UTC m=+4826.285359315" Nov 03 02:01:25.104817 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:01:25.104783 4421 generic.go:356] "Generic (PLEG): container finished" podID="d06f23d7-efb5-4ad5-9586-e8cd430a6229" containerID="6394b99dc1679be7910433d15688f817a4796cfd57ccef834bea4c77252d7491" exitCode=0 Nov 03 02:01:25.105338 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:01:25.104870 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ngqgb" event={"ID":"d06f23d7-efb5-4ad5-9586-e8cd430a6229","Type":"ContainerDied","Data":"6394b99dc1679be7910433d15688f817a4796cfd57ccef834bea4c77252d7491"} Nov 03 02:01:26.244353 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:01:26.244324 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ngqgb" Nov 03 02:01:26.322841 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:01:26.322801 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d06f23d7-efb5-4ad5-9586-e8cd430a6229-kube-api-access-4s9wb\" (UniqueName: \"kubernetes.io/projected/d06f23d7-efb5-4ad5-9586-e8cd430a6229-kube-api-access-4s9wb\") pod \"d06f23d7-efb5-4ad5-9586-e8cd430a6229\" (UID: \"d06f23d7-efb5-4ad5-9586-e8cd430a6229\") " Nov 03 02:01:26.322841 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:01:26.322848 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d06f23d7-efb5-4ad5-9586-e8cd430a6229-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d06f23d7-efb5-4ad5-9586-e8cd430a6229-maas-api-xks-ca\") pod \"d06f23d7-efb5-4ad5-9586-e8cd430a6229\" (UID: \"d06f23d7-efb5-4ad5-9586-e8cd430a6229\") " Nov 03 02:01:26.323094 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:01:26.322926 4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d06f23d7-efb5-4ad5-9586-e8cd430a6229-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d06f23d7-efb5-4ad5-9586-e8cd430a6229-maas-api-openshift-ca\") pod \"d06f23d7-efb5-4ad5-9586-e8cd430a6229\" (UID: \"d06f23d7-efb5-4ad5-9586-e8cd430a6229\") " Nov 03 02:01:26.323379 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:01:26.323352 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d06f23d7-efb5-4ad5-9586-e8cd430a6229-maas-api-openshift-ca" pod "d06f23d7-efb5-4ad5-9586-e8cd430a6229" (UID: "d06f23d7-efb5-4ad5-9586-e8cd430a6229"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:01:26.325042 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:01:26.325008 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d06f23d7-efb5-4ad5-9586-e8cd430a6229-maas-api-xks-ca" pod "d06f23d7-efb5-4ad5-9586-e8cd430a6229" (UID: "d06f23d7-efb5-4ad5-9586-e8cd430a6229"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:01:26.325042 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:01:26.325013 4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d06f23d7-efb5-4ad5-9586-e8cd430a6229-kube-api-access-4s9wb" pod "d06f23d7-efb5-4ad5-9586-e8cd430a6229" (UID: "d06f23d7-efb5-4ad5-9586-e8cd430a6229"). InnerVolumeSpecName "kube-api-access-4s9wb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:01:26.424611 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:01:26.424514 4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4s9wb\" (UniqueName: \"kubernetes.io/projected/d06f23d7-efb5-4ad5-9586-e8cd430a6229-kube-api-access-4s9wb\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 02:01:26.424611 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:01:26.424548 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d06f23d7-efb5-4ad5-9586-e8cd430a6229-maas-api-xks-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 02:01:26.424611 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:01:26.424561 4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d06f23d7-efb5-4ad5-9586-e8cd430a6229-maas-api-openshift-ca\") on node \"ip-10-0-133-217.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 02:01:27.116077 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:01:27.116045 4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ngqgb" Nov 03 02:01:27.116299 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:01:27.116079 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ngqgb" event={"ID":"d06f23d7-efb5-4ad5-9586-e8cd430a6229","Type":"ContainerDied","Data":"aedc4872113c1af0345066858d19e153c5444bc2990125f0f652f890f6ed7f08"} Nov 03 02:01:27.116299 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:01:27.116138 4421 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aedc4872113c1af0345066858d19e153c5444bc2990125f0f652f890f6ed7f08" Nov 03 02:01:34.742190 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:01:34.742145 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-7b97bffd64-84pv6"] Nov 03 02:01:34.742753 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:01:34.742733 4421 cpu_manager.go:414] "containerMap: removing container" podUID="d06f23d7-efb5-4ad5-9586-e8cd430a6229" containerName="revoke-keys" Nov 03 02:01:34.742907 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:01:34.742756 4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d06f23d7-efb5-4ad5-9586-e8cd430a6229" containerName="revoke-keys" Nov 03 02:01:34.742907 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:01:34.742896 4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="d06f23d7-efb5-4ad5-9586-e8cd430a6229" containerName="revoke-keys" Nov 03 02:01:34.746288 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:01:34.746251 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7b97bffd64-84pv6" Nov 03 02:01:34.757495 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:01:34.757461 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7b97bffd64-84pv6"] Nov 03 02:01:34.834900 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:01:34.834868 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-nfcxl\"" Nov 03 02:01:34.835119 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:01:34.834938 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 03 02:01:34.835119 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:01:34.835047 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Nov 03 02:01:34.839719 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:01:34.839700 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 02:01:34.839843 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:01:34.839744 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 02:01:34.912446 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:01:34.912398 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/806d0039-8de4-40a2-ae57-59889bc895e1-openshift-service-ca\") pod \"maas-api-7b97bffd64-84pv6\" (UID: \"806d0039-8de4-40a2-ae57-59889bc895e1\") " pod="odh-ai-gateway-infra/maas-api-7b97bffd64-84pv6" Nov 03 02:01:34.912644 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:01:34.912477 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cr75z\" (UniqueName: \"kubernetes.io/projected/806d0039-8de4-40a2-ae57-59889bc895e1-kube-api-access-cr75z\") pod \"maas-api-7b97bffd64-84pv6\" (UID: \"806d0039-8de4-40a2-ae57-59889bc895e1\") " pod="odh-ai-gateway-infra/maas-api-7b97bffd64-84pv6" Nov 03 02:01:34.912644 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:01:34.912514 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/806d0039-8de4-40a2-ae57-59889bc895e1-metrics-certs\") pod \"maas-api-7b97bffd64-84pv6\" (UID: \"806d0039-8de4-40a2-ae57-59889bc895e1\") " pod="odh-ai-gateway-infra/maas-api-7b97bffd64-84pv6" Nov 03 02:01:34.912644 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:01:34.912577 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/806d0039-8de4-40a2-ae57-59889bc895e1-maas-api-tls\") pod \"maas-api-7b97bffd64-84pv6\" (UID: \"806d0039-8de4-40a2-ae57-59889bc895e1\") " pod="odh-ai-gateway-infra/maas-api-7b97bffd64-84pv6" Nov 03 02:01:35.014006 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:01:35.013962 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cr75z\" (UniqueName: \"kubernetes.io/projected/806d0039-8de4-40a2-ae57-59889bc895e1-kube-api-access-cr75z\") pod \"maas-api-7b97bffd64-84pv6\" (UID: \"806d0039-8de4-40a2-ae57-59889bc895e1\") " pod="odh-ai-gateway-infra/maas-api-7b97bffd64-84pv6" Nov 03 02:01:35.014215 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:01:35.014010 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/806d0039-8de4-40a2-ae57-59889bc895e1-metrics-certs\") pod \"maas-api-7b97bffd64-84pv6\" (UID: \"806d0039-8de4-40a2-ae57-59889bc895e1\") " pod="odh-ai-gateway-infra/maas-api-7b97bffd64-84pv6" Nov 03 02:01:35.014215 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:01:35.014041 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/806d0039-8de4-40a2-ae57-59889bc895e1-maas-api-tls\") pod \"maas-api-7b97bffd64-84pv6\" (UID: \"806d0039-8de4-40a2-ae57-59889bc895e1\") " pod="odh-ai-gateway-infra/maas-api-7b97bffd64-84pv6" Nov 03 02:01:35.014215 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:01:35.014176 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/806d0039-8de4-40a2-ae57-59889bc895e1-openshift-service-ca\") pod \"maas-api-7b97bffd64-84pv6\" (UID: \"806d0039-8de4-40a2-ae57-59889bc895e1\") " pod="odh-ai-gateway-infra/maas-api-7b97bffd64-84pv6" Nov 03 02:01:35.014925 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:01:35.014898 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/806d0039-8de4-40a2-ae57-59889bc895e1-openshift-service-ca\") pod \"maas-api-7b97bffd64-84pv6\" (UID: \"806d0039-8de4-40a2-ae57-59889bc895e1\") " pod="odh-ai-gateway-infra/maas-api-7b97bffd64-84pv6" Nov 03 02:01:35.016692 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:01:35.016671 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/806d0039-8de4-40a2-ae57-59889bc895e1-metrics-certs\") pod \"maas-api-7b97bffd64-84pv6\" (UID: \"806d0039-8de4-40a2-ae57-59889bc895e1\") " pod="odh-ai-gateway-infra/maas-api-7b97bffd64-84pv6" Nov 03 02:01:35.016881 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:01:35.016859 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/806d0039-8de4-40a2-ae57-59889bc895e1-maas-api-tls\") pod \"maas-api-7b97bffd64-84pv6\" (UID: \"806d0039-8de4-40a2-ae57-59889bc895e1\") " pod="odh-ai-gateway-infra/maas-api-7b97bffd64-84pv6" Nov 03 02:01:35.021975 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:01:35.021954 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cr75z\" (UniqueName: \"kubernetes.io/projected/806d0039-8de4-40a2-ae57-59889bc895e1-kube-api-access-cr75z\") pod \"maas-api-7b97bffd64-84pv6\" (UID: \"806d0039-8de4-40a2-ae57-59889bc895e1\") " pod="odh-ai-gateway-infra/maas-api-7b97bffd64-84pv6" Nov 03 02:01:35.065171 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:01:35.065130 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7b97bffd64-84pv6" Nov 03 02:01:35.343073 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:01:35.343023 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7b97bffd64-84pv6"] Nov 03 02:01:35.345716 ip-10-0-133-217 kubenswrapper[4421]: W1103 02:01:35.345686 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod806d0039_8de4_40a2_ae57_59889bc895e1.slice/crio-b918702852fb7594da74665f5cea5dabe5912652cbd3bb0ca50ba6f9568dbcec WatchSource:0}: Error finding container b918702852fb7594da74665f5cea5dabe5912652cbd3bb0ca50ba6f9568dbcec: Status 404 returned error can't find the container with id b918702852fb7594da74665f5cea5dabe5912652cbd3bb0ca50ba6f9568dbcec Nov 03 02:01:36.164548 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:01:36.164499 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7b97bffd64-84pv6" event={"ID":"806d0039-8de4-40a2-ae57-59889bc895e1","Type":"ContainerStarted","Data":"d9ed29cefa9dada3b68961488d65b8892210499f01ff2f7c20dadd99a7ad461b"} Nov 03 02:01:36.165056 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:01:36.164568 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7b97bffd64-84pv6" event={"ID":"806d0039-8de4-40a2-ae57-59889bc895e1","Type":"ContainerStarted","Data":"b918702852fb7594da74665f5cea5dabe5912652cbd3bb0ca50ba6f9568dbcec"} Nov 03 02:01:36.165056 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:01:36.164686 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-7b97bffd64-84pv6" Nov 03 02:01:36.182806 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:01:36.182761 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-7b97bffd64-84pv6" podStartSLOduration=1.57522709 podStartE2EDuration="2.182746614s" totalImagesPullingTime="607.519524ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:01:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:01:36.181610498 +0000 UTC m=+4838.352613593" watchObservedRunningTime="2026-11-03 02:01:36.182746614 +0000 UTC m=+4838.353749727" Nov 03 02:01:42.039840 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:01:42.039799 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-wzcbf"] Nov 03 02:01:42.041660 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:01:42.041626 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-wzcbf"] Nov 03 02:01:42.174539 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:01:42.174496 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-7b97bffd64-84pv6" Nov 03 02:01:43.979887 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:01:43.979847 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49b5b322-05f4-4f73-9548-5b9e9c4e2831" path="/var/lib/kubelet/pods/49b5b322-05f4-4f73-9548-5b9e9c4e2831/volumes" Nov 03 02:01:44.700176 ip-10-0-133-217 kubenswrapper[4421]: E1103 02:01:44.700133 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 02:01:53.021693 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:01:53.021653 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-496js"] Nov 03 02:01:53.024630 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:01:53.024603 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-496js"] Nov 03 02:01:53.976912 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:01:53.976869 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f1e75eb-34b8-4483-bb5f-82408c09f09e" path="/var/lib/kubelet/pods/8f1e75eb-34b8-4483-bb5f-82408c09f09e/volumes" Nov 03 02:02:07.448403 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:02:07.448369 4421 scope.go:122] "RemoveContainer" containerID="50c7fc4e06a7bbc896244f17fc315d834a8057ba73e3b201f35890a2a48acf24" Nov 03 02:02:07.457605 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:02:07.457581 4421 scope.go:122] "RemoveContainer" containerID="1b82dbacf13933f35bc07b9cf416eed024b15ccc1f13c88c4dce2a118b1d219e" Nov 03 02:02:07.465494 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:02:07.465475 4421 scope.go:122] "RemoveContainer" containerID="19794d0c4e31326c3a656ddd139a264f16f30be8aa9c8ab60b871a57bea8e9fe" Nov 03 02:02:07.472840 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:02:07.472826 4421 scope.go:122] "RemoveContainer" containerID="4325af5af7db8b4cd10682157d0401f9e94c413a68bba1d4e2873bff48bc8c54" Nov 03 02:02:14.668948 ip-10-0-133-217 kubenswrapper[4421]: E1103 02:02:14.668908 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 02:02:44.684783 ip-10-0-133-217 kubenswrapper[4421]: E1103 02:02:44.684675 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 02:03:07.504538 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:03:07.504506 4421 scope.go:122] "RemoveContainer" containerID="dac4e8f822975f87e1588b360c977c20c2424e4dee7db0c79cebd3447b013e70" Nov 03 02:03:07.513508 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:03:07.513485 4421 scope.go:122] "RemoveContainer" containerID="e8e8e6a37d40b7f5ead1068794eb58e1b853c58b65cb3f5c0e880e1d338f7821" Nov 03 02:03:07.521638 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:03:07.521619 4421 scope.go:122] "RemoveContainer" containerID="fec631eb7450e0c74ff11ecf57f64e497340f110d31133579f6cd2435984ae7c" Nov 03 02:03:14.700045 ip-10-0-133-217 kubenswrapper[4421]: E1103 02:03:14.700003 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 02:03:38.927418 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:03:38.927368 4421 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-s4dc8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 02:03:38.928205 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:03:38.928166 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-s4dc8" podUID="ab5414b5-9d47-4c9f-9d0e-82242b9d6db2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 02:03:44.669125 ip-10-0-133-217 kubenswrapper[4421]: E1103 02:03:44.669085 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 02:04:14.670126 ip-10-0-133-217 kubenswrapper[4421]: E1103 02:04:14.670041 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 02:04:29.021167 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:04:29.021128 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f9qrl"] Nov 03 02:04:29.024089 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:04:29.024059 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f9qrl"] Nov 03 02:04:29.977706 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:04:29.977671 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d48403bd-83fd-4dee-9e78-63530260621b" path="/var/lib/kubelet/pods/d48403bd-83fd-4dee-9e78-63530260621b/volumes" Nov 03 02:04:44.669392 ip-10-0-133-217 kubenswrapper[4421]: E1103 02:04:44.669351 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 02:04:57.320446 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:04:57.320399 4421 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-q5n9x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 02:04:57.321175 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:04:57.321137 4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-q5n9x" podUID="e11a97cb-0980-45d1-842f-ded2fb31b03f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 02:05:14.669325 ip-10-0-133-217 kubenswrapper[4421]: E1103 02:05:14.669267 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 02:05:44.668422 ip-10-0-133-217 kubenswrapper[4421]: E1103 02:05:44.668332 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 02:06:07.583028 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:06:07.582986 4421 scope.go:122] "RemoveContainer" containerID="70354a6af58e58e5e8a4deca98f384c226236456d4c51f07457e90ed1c4c8dbd" Nov 03 02:06:11.041145 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:06:11.041110 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-mldb6"] Nov 03 02:06:11.043897 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:06:11.043869 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-mldb6"] Nov 03 02:06:11.994305 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:06:11.991571 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00f785dd-eed8-4121-87fb-f7f88bb416b8" path="/var/lib/kubelet/pods/00f785dd-eed8-4121-87fb-f7f88bb416b8/volumes" Nov 03 02:06:14.669158 ip-10-0-133-217 kubenswrapper[4421]: E1103 02:06:14.669112 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 02:06:15.744543 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:06:15.744510 4421 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-597cdd5495-wq4hg_5f9699ce-0019-4989-a2df-3043231dd93e/authorino/0.log" Nov 03 02:06:27.020651 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:06:27.020613 4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ngqgb"] Nov 03 02:06:27.023051 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:06:27.023021 4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ngqgb"] Nov 03 02:06:27.977550 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:06:27.977515 4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d06f23d7-efb5-4ad5-9586-e8cd430a6229" path="/var/lib/kubelet/pods/d06f23d7-efb5-4ad5-9586-e8cd430a6229/volumes" Nov 03 02:06:29.894783 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:06:29.894751 4421 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-7b97bffd64-84pv6_806d0039-8de4-40a2-ae57-59889bc895e1/maas-api/0.log" Nov 03 02:06:35.659468 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:06:35.659430 4421 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-597cdd5495-wq4hg_5f9699ce-0019-4989-a2df-3043231dd93e/authorino/0.log" Nov 03 02:06:36.205020 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:06:36.204984 4421 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-4222l_c2040caa-0454-46bb-bf81-a7063657d4ad/kuadrant-console-plugin/0.log" Nov 03 02:06:38.231293 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:06:38.231247 4421 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-594b68584c-jw4kg_4fa1295e-cc54-460b-aff5-a8e91aeb537d/kube-auth-proxy/0.log" Nov 03 02:06:39.386875 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:06:39.386842 4421 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-596dc9545f-kr275_3e73ad9e-cd24-4d53-82f0-2e521cae7445/istio-proxy/0.log" Nov 03 02:06:39.755407 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:06:39.755367 4421 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-596dc9545f-q5n9x_e11a97cb-0980-45d1-842f-ded2fb31b03f/istio-proxy/0.log" Nov 03 02:06:39.952158 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:06:39.952123 4421 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-596dc9545f-s4dc8_ab5414b5-9d47-4c9f-9d0e-82242b9d6db2/istio-proxy/0.log" Nov 03 02:06:42.200008 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:06:42.199974 4421 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-tmxmj_1b62c15b-bbbf-4926-b4c3-27d12277743b/main/0.log" Nov 03 02:06:42.560064 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:06:42.560031 4421 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-lc7tp_42b27a10-3a97-4e19-8b8a-d5588e9936ef/main/0.log" Nov 03 02:06:42.959230 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:06:42.959148 4421 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-5dqn2_0a54c4af-a899-4f16-83a8-8718b0c7eef5/main/0.log" Nov 03 02:06:43.335089 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:06:43.335057 4421 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-8gm8n_2d367e12-e46f-4a68-9825-9b1c934d4d12/main/0.log" Nov 03 02:06:44.668318 ip-10-0-133-217 kubenswrapper[4421]: E1103 02:06:44.668266 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 02:06:53.707368 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:06:53.707327 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-grd2d/must-gather-rfrzx"] Nov 03 02:06:53.711469 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:06:53.711446 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-grd2d/must-gather-rfrzx" Nov 03 02:06:53.718551 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:06:53.718310 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-grd2d\"/\"openshift-service-ca.crt\"" Nov 03 02:06:53.718551 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:06:53.718396 4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-grd2d\"/\"kube-root-ca.crt\"" Nov 03 02:06:53.718551 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:06:53.718489 4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-grd2d\"/\"default-dockercfg-wj6sf\"" Nov 03 02:06:53.721138 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:06:53.721115 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-grd2d/must-gather-rfrzx"] Nov 03 02:06:53.772216 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:06:53.772178 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jktl4\" (UniqueName: \"kubernetes.io/projected/50d13a77-8559-43c6-bd22-9b7e78052c01-kube-api-access-jktl4\") pod \"must-gather-rfrzx\" (UID: \"50d13a77-8559-43c6-bd22-9b7e78052c01\") " pod="openshift-must-gather-grd2d/must-gather-rfrzx" Nov 03 02:06:53.772216 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:06:53.772216 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/50d13a77-8559-43c6-bd22-9b7e78052c01-must-gather-output\") pod \"must-gather-rfrzx\" (UID: \"50d13a77-8559-43c6-bd22-9b7e78052c01\") " pod="openshift-must-gather-grd2d/must-gather-rfrzx" Nov 03 02:06:53.873063 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:06:53.873025 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jktl4\" (UniqueName: \"kubernetes.io/projected/50d13a77-8559-43c6-bd22-9b7e78052c01-kube-api-access-jktl4\") pod \"must-gather-rfrzx\" (UID: \"50d13a77-8559-43c6-bd22-9b7e78052c01\") " pod="openshift-must-gather-grd2d/must-gather-rfrzx" Nov 03 02:06:53.873063 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:06:53.873065 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/50d13a77-8559-43c6-bd22-9b7e78052c01-must-gather-output\") pod \"must-gather-rfrzx\" (UID: \"50d13a77-8559-43c6-bd22-9b7e78052c01\") " pod="openshift-must-gather-grd2d/must-gather-rfrzx" Nov 03 02:06:53.873512 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:06:53.873489 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/50d13a77-8559-43c6-bd22-9b7e78052c01-must-gather-output\") pod \"must-gather-rfrzx\" (UID: \"50d13a77-8559-43c6-bd22-9b7e78052c01\") " pod="openshift-must-gather-grd2d/must-gather-rfrzx" Nov 03 02:06:53.881209 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:06:53.881186 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jktl4\" (UniqueName: \"kubernetes.io/projected/50d13a77-8559-43c6-bd22-9b7e78052c01-kube-api-access-jktl4\") pod \"must-gather-rfrzx\" (UID: \"50d13a77-8559-43c6-bd22-9b7e78052c01\") " pod="openshift-must-gather-grd2d/must-gather-rfrzx" Nov 03 02:06:54.026251 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:06:54.026216 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-grd2d/must-gather-rfrzx" Nov 03 02:06:54.304477 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:06:54.304338 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-grd2d/must-gather-rfrzx"] Nov 03 02:06:54.307475 ip-10-0-133-217 kubenswrapper[4421]: W1103 02:06:54.307440 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod50d13a77_8559_43c6_bd22_9b7e78052c01.slice/crio-a173506f4095d94a8a37777811040cada868f6032fb5717408e8051872388236 WatchSource:0}: Error finding container a173506f4095d94a8a37777811040cada868f6032fb5717408e8051872388236: Status 404 returned error can't find the container with id a173506f4095d94a8a37777811040cada868f6032fb5717408e8051872388236 Nov 03 02:06:54.309420 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:06:54.309402 4421 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 02:06:55.205340 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:06:55.205301 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-grd2d/must-gather-rfrzx" event={"ID":"50d13a77-8559-43c6-bd22-9b7e78052c01","Type":"ContainerStarted","Data":"a173506f4095d94a8a37777811040cada868f6032fb5717408e8051872388236"} Nov 03 02:06:56.222072 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:06:56.221984 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-grd2d/must-gather-rfrzx" event={"ID":"50d13a77-8559-43c6-bd22-9b7e78052c01","Type":"ContainerStarted","Data":"0dc775c784e51fe7bbca50a821821db5bb061a6b6c7e01a2b1b441293aa29309"} Nov 03 02:06:56.222626 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:06:56.222598 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-grd2d/must-gather-rfrzx" event={"ID":"50d13a77-8559-43c6-bd22-9b7e78052c01","Type":"ContainerStarted","Data":"6fc0a9230d681408577c89984ca04651d1b4df03884f87064810c08cc921edde"} Nov 03 02:06:56.240425 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:06:56.240377 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-grd2d/must-gather-rfrzx" podStartSLOduration=2.153790057 podStartE2EDuration="3.240364781s" totalImagesPullingTime="1.086574724s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:06:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:06:56.237659528 +0000 UTC m=+5158.408662627" watchObservedRunningTime="2026-11-03 02:06:56.240364781 +0000 UTC m=+5158.411367877" Nov 03 02:06:57.579602 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:06:57.579395 4421 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-262mm_92341144-23c4-4cf0-a0c8-26e1891397e9/global-pull-secret-syncer/0.log" Nov 03 02:06:57.937069 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:06:57.936992 4421 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-jltt2_cd47e545-4089-44ad-bc4d-0545061c7b11/konnectivity-agent/0.log" Nov 03 02:06:57.963030 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:06:57.963004 4421 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-133-217.us-east-2.compute.internal_a1171317f26f09b75b7b789f796f1384/haproxy/0.log" Nov 03 02:07:05.337424 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:07:05.337377 4421 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-597cdd5495-wq4hg_5f9699ce-0019-4989-a2df-3043231dd93e/authorino/0.log" Nov 03 02:07:05.846453 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:07:05.846414 4421 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-4222l_c2040caa-0454-46bb-bf81-a7063657d4ad/kuadrant-console-plugin/0.log" Nov 03 02:07:07.608900 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:07:07.608863 4421 scope.go:122] "RemoveContainer" containerID="ab6327d7f57b31336e5cc9e26b74609e0a87810db5e3ba5e3757efa8a55b68a6" Nov 03 02:07:08.253605 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:07:08.253570 4421 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-4xbpw_ab520521-bd60-489a-b3bb-c7ee861809cf/monitoring-plugin/0.log" Nov 03 02:07:08.291807 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:07:08.291755 4421 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6ddbp_f5bb206e-db1f-4ed5-91ae-08adef206263/node-exporter/0.log" Nov 03 02:07:08.319332 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:07:08.319250 4421 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6ddbp_f5bb206e-db1f-4ed5-91ae-08adef206263/kube-rbac-proxy/0.log" Nov 03 02:07:08.343790 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:07:08.343735 4421 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6ddbp_f5bb206e-db1f-4ed5-91ae-08adef206263/init-textfile/0.log" Nov 03 02:07:08.581145 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:07:08.581052 4421 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-vb7lq_9aba6c65-0e46-4eea-b9cf-f4cc8b0d5ef8/kube-rbac-proxy-main/0.log" Nov 03 02:07:08.606718 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:07:08.606655 4421 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-vb7lq_9aba6c65-0e46-4eea-b9cf-f4cc8b0d5ef8/kube-rbac-proxy-self/0.log" Nov 03 02:07:08.647217 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:07:08.646998 4421 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-vb7lq_9aba6c65-0e46-4eea-b9cf-f4cc8b0d5ef8/openshift-state-metrics/0.log" Nov 03 02:07:08.692345 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:07:08.692231 4421 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_252de3ff-2d31-4c35-961f-a93acc666093/prometheus/0.log" Nov 03 02:07:08.721498 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:07:08.721468 4421 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_252de3ff-2d31-4c35-961f-a93acc666093/config-reloader/0.log" Nov 03 02:07:08.746808 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:07:08.746753 4421 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_252de3ff-2d31-4c35-961f-a93acc666093/thanos-sidecar/0.log" Nov 03 02:07:08.784112 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:07:08.784075 4421 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_252de3ff-2d31-4c35-961f-a93acc666093/kube-rbac-proxy-web/0.log" Nov 03 02:07:08.809569 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:07:08.809538 4421 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_252de3ff-2d31-4c35-961f-a93acc666093/kube-rbac-proxy/0.log" Nov 03 02:07:08.835551 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:07:08.835469 4421 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_252de3ff-2d31-4c35-961f-a93acc666093/kube-rbac-proxy-thanos/0.log" Nov 03 02:07:08.866269 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:07:08.866239 4421 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_252de3ff-2d31-4c35-961f-a93acc666093/init-config-reloader/0.log" Nov 03 02:07:08.916727 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:07:08.916692 4421 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-ft4j8_b694def6-001a-4d24-b091-eb4495bbb2d6/prometheus-operator/0.log" Nov 03 02:07:08.936155 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:07:08.936071 4421 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-ft4j8_b694def6-001a-4d24-b091-eb4495bbb2d6/kube-rbac-proxy/0.log" Nov 03 02:07:08.978283 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:07:08.976522 4421 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-2496n_e906bfeb-04c0-4465-9ec6-6c18aad596ae/prometheus-operator-admission-webhook/0.log" Nov 03 02:07:08.982107 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:07:08.982075 4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-grd2d/perf-node-gather-daemonset-8fc55"] Nov 03 02:07:08.990286 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:07:08.990246 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-grd2d/perf-node-gather-daemonset-8fc55" Nov 03 02:07:08.997642 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:07:08.997615 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-grd2d/perf-node-gather-daemonset-8fc55"] Nov 03 02:07:09.101371 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:07:09.101220 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c2d7dbde-57e0-44d6-b35f-99872b00d700-sys\") pod \"perf-node-gather-daemonset-8fc55\" (UID: \"c2d7dbde-57e0-44d6-b35f-99872b00d700\") " pod="openshift-must-gather-grd2d/perf-node-gather-daemonset-8fc55" Nov 03 02:07:09.101371 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:07:09.101323 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c2d7dbde-57e0-44d6-b35f-99872b00d700-proc\") pod \"perf-node-gather-daemonset-8fc55\" (UID: \"c2d7dbde-57e0-44d6-b35f-99872b00d700\") " pod="openshift-must-gather-grd2d/perf-node-gather-daemonset-8fc55" Nov 03 02:07:09.101756 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:07:09.101396 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c2d7dbde-57e0-44d6-b35f-99872b00d700-podres\") pod \"perf-node-gather-daemonset-8fc55\" (UID: \"c2d7dbde-57e0-44d6-b35f-99872b00d700\") " pod="openshift-must-gather-grd2d/perf-node-gather-daemonset-8fc55" Nov 03 02:07:09.101756 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:07:09.101449 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6j2w9\" (UniqueName: \"kubernetes.io/projected/c2d7dbde-57e0-44d6-b35f-99872b00d700-kube-api-access-6j2w9\") pod \"perf-node-gather-daemonset-8fc55\" (UID: \"c2d7dbde-57e0-44d6-b35f-99872b00d700\") " pod="openshift-must-gather-grd2d/perf-node-gather-daemonset-8fc55" Nov 03 02:07:09.101756 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:07:09.101522 4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c2d7dbde-57e0-44d6-b35f-99872b00d700-lib-modules\") pod \"perf-node-gather-daemonset-8fc55\" (UID: \"c2d7dbde-57e0-44d6-b35f-99872b00d700\") " pod="openshift-must-gather-grd2d/perf-node-gather-daemonset-8fc55" Nov 03 02:07:09.136793 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:07:09.136757 4421 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-68bd88d8d8-jjhxr_9f251a8f-5101-4fa3-a215-b1ee53bb822d/thanos-query/0.log" Nov 03 02:07:09.160862 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:07:09.160830 4421 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-68bd88d8d8-jjhxr_9f251a8f-5101-4fa3-a215-b1ee53bb822d/kube-rbac-proxy-web/0.log" Nov 03 02:07:09.193676 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:07:09.193610 4421 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-68bd88d8d8-jjhxr_9f251a8f-5101-4fa3-a215-b1ee53bb822d/kube-rbac-proxy/0.log" Nov 03 02:07:09.205314 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:07:09.205262 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c2d7dbde-57e0-44d6-b35f-99872b00d700-sys\") pod \"perf-node-gather-daemonset-8fc55\" (UID: \"c2d7dbde-57e0-44d6-b35f-99872b00d700\") " pod="openshift-must-gather-grd2d/perf-node-gather-daemonset-8fc55" Nov 03 02:07:09.206321 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:07:09.205758 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c2d7dbde-57e0-44d6-b35f-99872b00d700-sys\") pod \"perf-node-gather-daemonset-8fc55\" (UID: \"c2d7dbde-57e0-44d6-b35f-99872b00d700\") " pod="openshift-must-gather-grd2d/perf-node-gather-daemonset-8fc55" Nov 03 02:07:09.206558 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:07:09.206335 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c2d7dbde-57e0-44d6-b35f-99872b00d700-proc\") pod \"perf-node-gather-daemonset-8fc55\" (UID: \"c2d7dbde-57e0-44d6-b35f-99872b00d700\") " pod="openshift-must-gather-grd2d/perf-node-gather-daemonset-8fc55" Nov 03 02:07:09.206558 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:07:09.206437 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c2d7dbde-57e0-44d6-b35f-99872b00d700-podres\") pod \"perf-node-gather-daemonset-8fc55\" (UID: \"c2d7dbde-57e0-44d6-b35f-99872b00d700\") " pod="openshift-must-gather-grd2d/perf-node-gather-daemonset-8fc55" Nov 03 02:07:09.206558 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:07:09.206500 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6j2w9\" (UniqueName: \"kubernetes.io/projected/c2d7dbde-57e0-44d6-b35f-99872b00d700-kube-api-access-6j2w9\") pod \"perf-node-gather-daemonset-8fc55\" (UID: \"c2d7dbde-57e0-44d6-b35f-99872b00d700\") " pod="openshift-must-gather-grd2d/perf-node-gather-daemonset-8fc55" Nov 03 02:07:09.206858 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:07:09.206554 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c2d7dbde-57e0-44d6-b35f-99872b00d700-podres\") pod \"perf-node-gather-daemonset-8fc55\" (UID: \"c2d7dbde-57e0-44d6-b35f-99872b00d700\") " pod="openshift-must-gather-grd2d/perf-node-gather-daemonset-8fc55" Nov 03 02:07:09.206858 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:07:09.206583 4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c2d7dbde-57e0-44d6-b35f-99872b00d700-lib-modules\") pod \"perf-node-gather-daemonset-8fc55\" (UID: \"c2d7dbde-57e0-44d6-b35f-99872b00d700\") " pod="openshift-must-gather-grd2d/perf-node-gather-daemonset-8fc55" Nov 03 02:07:09.206858 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:07:09.206768 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c2d7dbde-57e0-44d6-b35f-99872b00d700-lib-modules\") pod \"perf-node-gather-daemonset-8fc55\" (UID: \"c2d7dbde-57e0-44d6-b35f-99872b00d700\") " pod="openshift-must-gather-grd2d/perf-node-gather-daemonset-8fc55" Nov 03 02:07:09.207243 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:07:09.207211 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c2d7dbde-57e0-44d6-b35f-99872b00d700-proc\") pod \"perf-node-gather-daemonset-8fc55\" (UID: \"c2d7dbde-57e0-44d6-b35f-99872b00d700\") " pod="openshift-must-gather-grd2d/perf-node-gather-daemonset-8fc55" Nov 03 02:07:09.215841 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:07:09.215813 4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6j2w9\" (UniqueName: \"kubernetes.io/projected/c2d7dbde-57e0-44d6-b35f-99872b00d700-kube-api-access-6j2w9\") pod \"perf-node-gather-daemonset-8fc55\" (UID: \"c2d7dbde-57e0-44d6-b35f-99872b00d700\") " pod="openshift-must-gather-grd2d/perf-node-gather-daemonset-8fc55" Nov 03 02:07:09.219564 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:07:09.219527 4421 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-68bd88d8d8-jjhxr_9f251a8f-5101-4fa3-a215-b1ee53bb822d/prom-label-proxy/0.log" Nov 03 02:07:09.243805 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:07:09.243766 4421 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-68bd88d8d8-jjhxr_9f251a8f-5101-4fa3-a215-b1ee53bb822d/kube-rbac-proxy-rules/0.log" Nov 03 02:07:09.270566 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:07:09.270509 4421 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-68bd88d8d8-jjhxr_9f251a8f-5101-4fa3-a215-b1ee53bb822d/kube-rbac-proxy-metrics/0.log" Nov 03 02:07:09.322668 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:07:09.322631 4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-grd2d/perf-node-gather-daemonset-8fc55" Nov 03 02:07:09.670014 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:07:09.669984 4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-grd2d/perf-node-gather-daemonset-8fc55"] Nov 03 02:07:09.680127 ip-10-0-133-217 kubenswrapper[4421]: W1103 02:07:09.680092 4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podc2d7dbde_57e0_44d6_b35f_99872b00d700.slice/crio-4c451ce1074ed19bdeb2ebe02ecd8e1e48ae30a51d360892128fb9db3567fb5a WatchSource:0}: Error finding container 4c451ce1074ed19bdeb2ebe02ecd8e1e48ae30a51d360892128fb9db3567fb5a: Status 404 returned error can't find the container with id 4c451ce1074ed19bdeb2ebe02ecd8e1e48ae30a51d360892128fb9db3567fb5a Nov 03 02:07:10.409467 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:07:10.409364 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-grd2d/perf-node-gather-daemonset-8fc55" event={"ID":"c2d7dbde-57e0-44d6-b35f-99872b00d700","Type":"ContainerStarted","Data":"46aaaf9c34a49f04d34f4726a389ac3605731534a6f33dbe42b449022c0619c5"} Nov 03 02:07:10.409467 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:07:10.409433 4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-grd2d/perf-node-gather-daemonset-8fc55" event={"ID":"c2d7dbde-57e0-44d6-b35f-99872b00d700","Type":"ContainerStarted","Data":"4c451ce1074ed19bdeb2ebe02ecd8e1e48ae30a51d360892128fb9db3567fb5a"} Nov 03 02:07:10.409744 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:07:10.409596 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-grd2d/perf-node-gather-daemonset-8fc55" Nov 03 02:07:10.428185 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:07:10.427912 4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-grd2d/perf-node-gather-daemonset-8fc55" podStartSLOduration=2.42789137 podStartE2EDuration="2.42789137s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 02:07:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:07:10.425094794 +0000 UTC m=+5172.596097891" watchObservedRunningTime="2026-11-03 02:07:10.42789137 +0000 UTC m=+5172.598894467" Nov 03 02:07:11.777817 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:07:11.777785 4421 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-594b68584c-jw4kg_4fa1295e-cc54-460b-aff5-a8e91aeb537d/kube-auth-proxy/0.log" Nov 03 02:07:11.876259 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:07:11.876221 4421 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6dd789c8b7-bzcwg_a145389b-418b-4ef0-9c66-29c9d7786081/download-server/0.log" Nov 03 02:07:11.886656 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:07:11.886601 4421 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-596dc9545f-kr275_3e73ad9e-cd24-4d53-82f0-2e521cae7445/istio-proxy/0.log" Nov 03 02:07:11.926170 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:07:11.926134 4421 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-596dc9545f-q5n9x_e11a97cb-0980-45d1-842f-ded2fb31b03f/istio-proxy/0.log" Nov 03 02:07:11.945023 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:07:11.944984 4421 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-596dc9545f-s4dc8_ab5414b5-9d47-4c9f-9d0e-82242b9d6db2/istio-proxy/0.log" Nov 03 02:07:13.745494 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:07:13.745458 4421 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-rx2xs_9c505674-e66b-4573-bbf5-7b6acf6684e6/dns/0.log" Nov 03 02:07:13.768498 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:07:13.768463 4421 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-rx2xs_9c505674-e66b-4573-bbf5-7b6acf6684e6/kube-rbac-proxy/0.log" Nov 03 02:07:13.828628 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:07:13.828597 4421 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-cphks_e7f1faa5-fc0e-4f46-9208-b1e58f50fcf9/dns-node-resolver/0.log" Nov 03 02:07:14.535854 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:07:14.535812 4421 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-5f895c599b-f8f2x_46039de5-8eaf-4029-9cb6-ca23508092f8/registry/0.log" Nov 03 02:07:14.622734 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:07:14.622692 4421 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-w8k5t_5844f274-ad38-42a3-9900-b051b01c9953/node-ca/0.log" Nov 03 02:07:14.685304 ip-10-0-133-217 kubenswrapper[4421]: E1103 02:07:14.684851 4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zskmb" containerName="extractor" Nov 03 02:07:15.958003 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:07:15.957930 4421 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-597cdd5495-wq4hg_5f9699ce-0019-4989-a2df-3043231dd93e/authorino/0.log" Nov 03 02:07:16.002303 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:07:15.999705 4421 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-4222l_c2040caa-0454-46bb-bf81-a7063657d4ad/kuadrant-console-plugin/0.log" Nov 03 02:07:16.432315 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:07:16.430841 4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-grd2d/perf-node-gather-daemonset-8fc55" Nov 03 02:07:16.663824 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:07:16.663790 4421 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-594b68584c-jw4kg_4fa1295e-cc54-460b-aff5-a8e91aeb537d/kube-auth-proxy/0.log" Nov 03 02:07:16.860290 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:07:16.860236 4421 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-596dc9545f-kr275_3e73ad9e-cd24-4d53-82f0-2e521cae7445/istio-proxy/0.log" Nov 03 02:07:16.922757 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:07:16.922724 4421 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-596dc9545f-q5n9x_e11a97cb-0980-45d1-842f-ded2fb31b03f/istio-proxy/0.log" Nov 03 02:07:16.958750 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:07:16.958716 4421 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-596dc9545f-s4dc8_ab5414b5-9d47-4c9f-9d0e-82242b9d6db2/istio-proxy/0.log" Nov 03 02:07:20.570836 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:07:20.570807 4421 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-whbrp_76ccef27-d056-402e-a7e1-ee524d63bf35/serve-healthcheck-canary/0.log" Nov 03 02:07:21.352439 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:07:21.352408 4421 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-594b68584c-jw4kg_4fa1295e-cc54-460b-aff5-a8e91aeb537d/kube-auth-proxy/0.log" Nov 03 02:07:21.391130 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:07:21.391098 4421 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-zskmb_bac578ef-a88b-4a74-9a65-bebc8ea1d00e/kube-rbac-proxy/0.log" Nov 03 02:07:21.413564 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:07:21.413530 4421 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-zskmb_bac578ef-a88b-4a74-9a65-bebc8ea1d00e/exporter/0.log" Nov 03 02:07:21.441750 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:07:21.441717 4421 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-zskmb_bac578ef-a88b-4a74-9a65-bebc8ea1d00e/extractor/0.log" Nov 03 02:07:21.472161 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:07:21.472113 4421 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-596dc9545f-kr275_3e73ad9e-cd24-4d53-82f0-2e521cae7445/istio-proxy/0.log" Nov 03 02:07:21.510327 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:07:21.510295 4421 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-596dc9545f-q5n9x_e11a97cb-0980-45d1-842f-ded2fb31b03f/istio-proxy/0.log" Nov 03 02:07:21.534177 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:07:21.534146 4421 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-596dc9545f-s4dc8_ab5414b5-9d47-4c9f-9d0e-82242b9d6db2/istio-proxy/0.log" Nov 03 02:07:23.969030 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:07:23.968999 4421 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-7cc9488979-bj5xx_816475cf-e48b-4d1a-a20f-16c62118bb0f/manager/0.log" Nov 03 02:07:23.990514 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:07:23.990482 4421 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-7cc9488979-bj5xx_816475cf-e48b-4d1a-a20f-16c62118bb0f/copy-manifests/0.log" Nov 03 02:07:25.264704 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:07:25.264672 4421 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-jobset-operator_jobset-controller-manager-6f98fffc9c-l2ptp_4d579ceb-59fa-499a-b241-7f4e940976c3/manager/1.log" Nov 03 02:07:25.265186 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:07:25.265171 4421 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-jobset-operator_jobset-controller-manager-6f98fffc9c-l2ptp_4d579ceb-59fa-499a-b241-7f4e940976c3/manager/2.log" Nov 03 02:07:26.441390 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:07:26.441360 4421 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-594b68584c-jw4kg_4fa1295e-cc54-460b-aff5-a8e91aeb537d/kube-auth-proxy/0.log" Nov 03 02:07:26.578697 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:07:26.578663 4421 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-596dc9545f-kr275_3e73ad9e-cd24-4d53-82f0-2e521cae7445/istio-proxy/0.log" Nov 03 02:07:26.620453 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:07:26.620418 4421 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-596dc9545f-q5n9x_e11a97cb-0980-45d1-842f-ded2fb31b03f/istio-proxy/0.log" Nov 03 02:07:26.643869 ip-10-0-133-217 kubenswrapper[4421]: I1103 02:07:26.643834 4421 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-596dc9545f-s4dc8_ab5414b5-9d47-4c9f-9d0e-82242b9d6db2/istio-proxy/0.log"